deepfabric: Generate and Evaluate Synthetic Training Data

Overview

DeepFabric is a Python toolkit for creating synthetic datasets that teach language models to reason, use tools, and follow structured output requirements. It addresses a common dataset-generation problem: producing varied examples that remain relevant to a domain rather than repeating similar samples.

It also supports evaluating fine-tuned models on held-out tool-use tasks. The project is aimed at developers and ML practitioners who need a pipeline from dataset creation through training preparation and evaluation, rather than a standalone text generator.

Key Features

• Builds topic trees or graphs to guide sample coverage and reduce redundancy.
• Generates datasets through a CLI, Python library, or YAML configuration.
• Supports multiple hosted and local LLM providers, including OpenAI, Anthropic, Gemini, Ollama, and OpenRouter.
• Creates tool-use conversations with schema constraints and optional execution in isolated WebAssembly environments through Spin.
• Exports datasets for Hugging Face and common training workflows such as TRL, Unsloth, and Axolotl.
• Evaluates tool selection, parameter accuracy, execution success, and response quality.
• Supports evaluation with Transformers or Ollama, including LoRA adapters with the Transformers backend.

Use Cases

• ML teams can generate domain-focused instruction data when curated examples are limited or need broader topic coverage.
• Agent developers can create tool-calling examples based on executed tool results instead of relying only on simulated outputs.
• Fine-tuning practitioners can prepare Hugging Face datasets for TRL, Unsloth, or Axolotl workflows.
• Model evaluators can measure tool-use behavior on held-out tasks after fine-tuning.

Getting Started

Install the package with:

pip install deepfabric

The CLI requires an LLM provider configuration, such as an OPENAI_API_KEY for the README's OpenAI example. See the README for generation, configuration, training, and evaluation instructions.

Alternatives

• Ragas: Ragas generates test data and evaluates LLM applications, with a stronger focus on RAG quality metrics than domain-specific training datasets and agent traces.
• Verifiers: Verifiers builds customizable environments, datasets, and reward functions for LLM training and evaluation, rather than focusing on synthetic dataset generation.

Considerations

• Generating data with hosted providers requires access credentials and may incur provider costs. Local generation is also supported through Ollama.
• Spin-based tool execution requires a running Spin service; the README provides a Docker image and local setup instructions.
• Evaluation requires a dataset and an inference backend. Running local models through Transformers may require suitable compute resources, depending on the model.
• Synthetic data and evaluation scores do not by themselves establish that a model will perform reliably in production. Review generated samples and test against the intended tasks.