Repository History
5 repositories tagged with LLM Evaluation
Ragas: Supercharge Your LLM Application Evaluations
Ragas is an ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. It offers objective metrics, intelligent test generation, and data-driven insights to move beyond subjective assessments. This framework helps developers build feedback loops and continuously improve their LLM applications.
PromptBench: A Unified Framework for LLM Evaluation and Robustness
PromptBench is a comprehensive Python library designed for the evaluation and understanding of Large Language Models (LLMs). It provides a unified framework for assessing model performance, exploring various prompt engineering techniques, and evaluating robustness against adversarial attacks. This tool empowers researchers to conduct in-depth analyses of LLMs across diverse datasets and models.
Phoenix: AI Observability and Evaluation Platform for LLMs
Phoenix is an open-source AI observability platform from Arize AI, designed for comprehensive experimentation, evaluation, and troubleshooting of LLM applications. It provides robust features including OpenTelemetry-based tracing, LLM evaluation, and systematic prompt management. This platform helps developers optimize and debug their AI models effectively across various environments.
JailbreakEval: An Integrated Toolkit for Evaluating LLM Jailbreak Attempts
JailbreakEval is an award-winning collection of automated evaluators designed to assess jailbreak attempts against large language models. It addresses the impracticality of manual inspection for large-scale analysis by unifying various evaluation tools. This toolkit is invaluable for both jailbreak researchers and evaluator developers, offering a robust framework for creating and benchmarking new evaluators.
Opik: Open-Source LLM Observability, Evaluation, and Optimization
Opik is an open-source platform by Comet designed to streamline the lifecycle of LLM applications. It provides comprehensive tools for debugging, evaluating, and monitoring RAG systems and agentic workflows. Developers can leverage its tracing, automated evaluations, and production-ready dashboards to build and optimize generative AI applications.