Repository History
5 repositories tagged with Evaluation
RAGChecker: A Fine-grained Framework for Diagnosing RAG Systems
RAGChecker is an advanced automatic evaluation framework developed by Amazon Science, specifically designed to assess and diagnose Retrieval-Augmented Generation (RAG) systems. It offers a comprehensive suite of metrics and tools for in-depth analysis of RAG performance. This framework empowers developers and researchers to thoroughly evaluate and enhance their RAG systems with precision.
AgentEvals: Robust Evaluation Tools for LLM Agent Trajectories
AgentEvals is a powerful open-source package from LangChain designed to simplify the evaluation of agentic applications. It provides a collection of ready-made evaluators and utilities, with a particular focus on analyzing agent trajectories, the intermediate steps an agent takes to solve problems. This helps developers understand and improve the reliability and performance of their LLM agents.
LLMBox: A Comprehensive Python Library for LLM Training and Evaluation
LLMBox is a comprehensive Python library designed for implementing Large Language Models, offering a unified training pipeline and extensive model evaluation capabilities. It provides a one-stop solution for both training and utilizing LLMs, emphasizing flexibility and efficiency. Developers can leverage its diverse training strategies and blazingly fast inference for their LLM projects.
Verifiers: Environments for LLM Reinforcement Learning and Evaluation
Verifiers is a Python library by Prime Intellect AI for building environments to train and evaluate Large Language Models (LLMs). It enables the creation of custom environments with datasets, model harnesses, and reward functions, supporting reinforcement learning, capability evaluation, and synthetic data generation. This library is tightly integrated with the Prime Intellect ecosystem, including their Environments Hub and training framework.
Weave by Weights & Biases: A Toolkit for AI-Powered Applications
Weave is an open-source toolkit developed by Weights & Biases designed for building and managing AI-powered applications. It provides robust features for logging, debugging, and evaluating language model inputs and outputs, streamlining the development workflow for generative AI. Weave aims to bring rigor and best practices to the experimental process of AI software development.