Repository History
311 repositories tagged with Python
DataDreamer: Streamlining Synthetic Data Generation and LLM Workflows
DataDreamer is an open-source Python library designed for efficient prompting, synthetic data generation, and model training workflows. It simplifies the process of creating complex LLM workflows, generating high-quality synthetic datasets, and aligning or fine-tuning models. Built to be simple, efficient, and research-grade, DataDreamer empowers users to build reproducible and shareable AI solutions.
EasyInstruct: An Easy-to-Use Instruction Processing Framework for LLMs
EasyInstruct is an open-source Python framework designed to simplify instruction processing for Large Language Models (LLMs). Accepted at ACL 2024, it offers modularized components for instruction generation, selection, and prompting, supporting various LLMs like GPT-4 and LLaMA. This framework is ideal for researchers and developers working on LLM-based experiments and applications.
LazyLLM: Low-Code Development for Multi-Agent LLM Applications
LazyLLM offers a low-code development tool designed for building multi-agent LLM applications with ease. It simplifies the creation of complex AI applications, providing a streamlined workflow for rapid prototyping, data feedback, and iterative optimization. Developers can leverage its extensive features for deployment, cross-platform compatibility, and efficient model fine-tuning.
ChatArena: Multi-Agent Language Game Environments for LLMs
ChatArena is a Python library designed to provide multi-agent language game environments for Large Language Models (LLMs), aiming to foster the development of communication and collaboration capabilities in AI. It offers a flexible framework for defining players, environments, and interactions based on Markov Decision Processes. Please note that as of August 11, 2025, this project has been deprecated due to a lack of widespread community use and is no longer receiving updates or support.
Agentarium: A Python Framework for AI Agent Simulations
Agentarium is an open-source Python framework designed for creating and managing simulations with AI-powered agents. It offers an intuitive platform for designing complex, interactive environments where agents can act, learn, and evolve. This powerful tool simplifies the orchestration of multiple AI agents and their interactions.
Lighteval: Your All-in-One Toolkit for LLM Evaluation
Lighteval is a comprehensive toolkit from Hugging Face for evaluating Large Language Models (LLMs) across various backends. It enables users to dive deep into model performance by saving detailed, sample-by-sample results and supports over 1000 evaluation tasks. The framework offers extensive customization options, allowing users to create custom tasks and metrics tailored to their specific needs.
PromptBench: A Unified Framework for LLM Evaluation and Robustness
PromptBench is a comprehensive Python library designed for the evaluation and understanding of Large Language Models (LLMs). It provides a unified framework for assessing model performance, exploring various prompt engineering techniques, and evaluating robustness against adversarial attacks. This tool empowers researchers to conduct in-depth analyses of LLMs across diverse datasets and models.
LangTest: A Comprehensive Library for Safe & Effective Language Models
LangTest is an open-source Python library dedicated to ensuring the safety and effectiveness of language models. It offers a comprehensive framework for testing model quality, covering robustness, bias, fairness, and accuracy across various NLP tasks and LLM providers. With LangTest, developers can generate and execute over 60 distinct test types with just one line of code, promoting responsible AI development.
EvalPlus: Rigorous Evaluation for LLM-Synthesized Code
EvalPlus is a robust framework designed for the rigorous evaluation of code generated by Large Language Models (LLMs). It extends standard benchmarks like HumanEval and MBPP with significantly more tests, offering precise assessment of code correctness and efficiency. This tool is crucial for developers and researchers aiming to thoroughly validate LLM-synthesized code.
AgentEvals: Robust Evaluation Tools for LLM Agent Trajectories
AgentEvals is a powerful open-source package from LangChain designed to simplify the evaluation of agentic applications. It provides a collection of ready-made evaluators and utilities, with a particular focus on analyzing agent trajectories, the intermediate steps an agent takes to solve problems. This helps developers understand and improve the reliability and performance of their LLM agents.
Phoenix: AI Observability and Evaluation Platform for LLMs
Phoenix is an open-source AI observability platform from Arize AI, designed for comprehensive experimentation, evaluation, and troubleshooting of LLM applications. It provides robust features including OpenTelemetry-based tracing, LLM evaluation, and systematic prompt management. This platform helps developers optimize and debug their AI models effectively across various environments.
Observers: A Lightweight Library for AI Observability in Python
Observers is a Python library designed for AI observability, enabling developers to track and store interactions with generative AI APIs. It provides a flexible framework with various observers for popular LLM providers and multiple storage backends. This tool helps in monitoring, debugging, and analyzing AI model behavior effectively.