{"name":"DeepFabric: High-Quality Synthetic Data for Agentic AI Systems","description":"DeepFabric is an open-source Python library designed to generate high-quality synthetic training data for language models and agent evaluations. It excels at creating domain-specific datasets that teach models to think, plan, and act effectively, including correct tool usage and adherence to schema structures. This comprehensive pipeline also integrates training and evaluation capabilities, ensuring robust model development.","github":"https://github.com/stacklok/promptwright","url":"https://osrepos.com/repo/stacklok-promptwright","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/stacklok-promptwright","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/stacklok-promptwright.md","json":"https://osrepos.com/repo/stacklok-promptwright.json","topics":["python","ai","machine-learning","synthetic-data","agents","huggingface","evaluation","open-source"],"keywords":["python","ai","machine-learning","synthetic-data","agents","huggingface","evaluation","open-source"],"stars":null,"summary":"DeepFabric is an open-source Python library designed to generate high-quality synthetic training data for language models and agent evaluations. It excels at creating domain-specific datasets that teach models to think, plan, and act effectively, including correct tool usage and adherence to schema structures. This comprehensive pipeline also integrates training and evaluation capabilities, ensuring robust model development.","content":"## Introduction\nDeepFabric is a powerful open-source Python library that streamlines the process of generating synthetic training data for language models and evaluating agentic systems. It provides a complete pipeline to create high-quality, domain-specific datasets, train models, and rigorously assess their performance, particularly in tool-calling scenarios. By focusing on realistic reasoning traces and tool-calling patterns, DeepFabric helps develop models that can think, plan, and act effectively.\n\n## Installation\nGetting started with DeepFabric is straightforward. You can install it using pip:\n\nbash\npip install deepfabric\n\n\n## Examples\nDeepFabric can be used via its CLI, as a library, or with YAML configurations. Here's a quick example using the CLI to generate a dataset:\n\nbash\nexport OPENAI_API_KEY=\"your-api-key\"\n\ndeepfabric generate \\\n  --topic-prompt \"Python programming fundamentals\" \\\n  --generation-system-prompt \"You are a Python expert\" \\\n  --mode graph \\\n  --depth 3 \\\n  --degree 3 \\\n  --num-samples 9 \\\n  --batch-size 3 \\\n  --provider openai \\\n  --model gpt-4o \\\n  --output-save-as dataset.jsonl\n\n\nThis command generates a topic graph and creates 27 unique nodes, then generates 27 training samples saved to `dataset.jsonl`, ensuring 100% topic coverage.\n\nFor evaluation, after training your model, you can use DeepFabric's built-in evaluator:\n\npython\nfrom deepfabric.evaluation import Evaluator, EvaluatorConfig, InferenceConfig\nfrom datasets import load_dataset\n\n# Load your evaluation dataset\ndataset = load_dataset(\"your-username/your-dataset\", split=\"test\")\n\nconfig = EvaluatorConfig(\n    inference_config=InferenceConfig(\n        model_path=\"./output/checkpoint-final\",  # Local path or HF Hub ID\n        backend=\"transformers\",\n    ),\n)\n\nevaluator = Evaluator(config)\nresults = evaluator.evaluate(dataset=dataset)\n\nprint(f\"Overall Score: {results.metrics.overall_score:.2%}\")\n\n\n## Why Use It\nDeepFabric stands out by generating synthetic data that ensures high diversity while maintaining domain-anchored relevance, thanks to its unique topic graph generation algorithms. This approach prevents model overfit, a common issue with other tools. A key differentiator is its support for real tool execution using the Spin Framework, allowing agents to interact with isolated WebAssembly sandboxes. This produces authentic training data where decisions are based on actual observations, rather than simulated outputs. The platform also offers robust evaluation metrics, including tool selection accuracy, parameter accuracy, and execution success rate, providing a comprehensive view of model performance.\n\n## Links\n*   GitHub Repository: [https://github.com/nolabs-ai/deepfabric](https://github.com/nolabs-ai/deepfabric){:target=\"_blank\"}\n*   Documentation: [https://always-further.github.io/deepfabric/](https://always-further.github.io/deepfabric/){:target=\"_blank\"}\n*   Discord: [https://discord.gg/pPcjYzGvbS](https://discord.gg/pPcjYzGvbS){:target=\"_blank\"}\n*   Issues: [https://github.com/always-further/deepfabric/issues](https://github.com/always-further/deepfabric/issues){:target=\"_blank\"}","metrics":{"detailViews":2,"githubClicks":1},"dates":{"published":null,"modified":"2026-07-02T15:04:28.000Z"}}