Repository History
351 repositories tagged with Python
EvalPlus: Rigorous Evaluation for LLM-Synthesized Code
EvalPlus is a robust framework designed for the rigorous evaluation of code generated by Large Language Models (LLMs). It extends standard benchmarks like HumanEval and MBPP with significantly more tests, offering precise assessment of code correctness and efficiency. This tool is crucial for developers and researchers aiming to thoroughly validate LLM-synthesized code.
AgentEvals: Robust Evaluation Tools for LLM Agent Trajectories
AgentEvals is a powerful open-source package from LangChain designed to simplify the evaluation of agentic applications. It provides a collection of ready-made evaluators and utilities, with a particular focus on analyzing agent trajectories, the intermediate steps an agent takes to solve problems. This helps developers understand and improve the reliability and performance of their LLM agents.
Phoenix: AI Observability and Evaluation Platform for LLMs
Phoenix is an open-source AI observability platform from Arize AI, designed for comprehensive experimentation, evaluation, and troubleshooting of LLM applications. It provides robust features including OpenTelemetry-based tracing, LLM evaluation, and systematic prompt management. This platform helps developers optimize and debug their AI models effectively across various environments.
Observers: A Lightweight Library for AI Observability in Python
Observers is a Python library designed for AI observability, enabling developers to track and store interactions with generative AI APIs. It provides a flexible framework with various observers for popular LLM providers and multiple storage backends. This tool helps in monitoring, debugging, and analyzing AI model behavior effectively.
Jsonformer: Bulletproof Structured JSON Generation from Language Models
Jsonformer is a powerful library designed to generate syntactically correct and schema-conforming JSON from language models. It addresses the common challenge of unreliable JSON output by focusing on generating only content tokens, making the process more efficient and robust. This approach ensures bulletproof structured data generation for various applications.
JailbreakEval: An Integrated Toolkit for Evaluating LLM Jailbreak Attempts
JailbreakEval is an award-winning collection of automated evaluators designed to assess jailbreak attempts against large language models. It addresses the impracticality of manual inspection for large-scale analysis by unifying various evaluation tools. This toolkit is invaluable for both jailbreak researchers and evaluator developers, offering a robust framework for creating and benchmarking new evaluators.
EasyJailbreak: A Python Framework for Adversarial LLM Jailbreak Prompts
EasyJailbreak is an intuitive Python framework designed for generating adversarial jailbreak prompts for Large Language Models (LLMs). It provides a structured approach to decompose the jailbreaking process into iterative steps, offering components for mutation, attack, and evaluation. This tool is ideal for researchers and developers focused on LLM security and understanding model vulnerabilities.
Guardrails: Enhancing LLM Reliability and Structured Data Generation
Guardrails is a Python framework designed to build reliable AI applications by adding guardrails to large language models. It helps detect, quantify, and mitigate risks in LLM inputs/outputs, and facilitates the generation of structured data. This framework ensures more predictable and safer interactions with AI models.
Hiring Agent: An AI Agent for Resume Evaluation and Scoring
Hiring Agent is an open-source AI agent designed to evaluate and score resumes objectively. It extracts structured data from PDF resumes, enriches it with GitHub profile signals, and provides a fair, explainable evaluation with detailed scores and evidence. This tool supports both local LLMs via Ollama and cloud-based options like Google Gemini.
LLM Guard: The Security Toolkit for LLM Interactions
LLM Guard is an open-source security toolkit developed by Protect AI, designed to fortify the safety of Large Language Models. It offers comprehensive protection against various threats, including prompt injection, data leakage, and harmful language, ensuring secure and reliable LLM interactions.
AuditNLG: Auditing Generative AI for Trustworthiness
AuditNLG is an open-source library from Salesforce designed to enhance the trustworthiness of generative AI language models. It provides state-of-the-art techniques to detect and improve factualness, safety, and constraint adherence in AI-generated text. This library simplifies the process of auditing AI outputs, offering explanations and alternative suggestions for problematic content.
Odysseus: A Comprehensive Self-Hosted AI Workspace for Productivity
Odysseus is a powerful self-hosted AI workspace designed to integrate various AI-powered tools into a single platform. It offers functionalities for chat, agents, deep research, document management, email, and calendar, supporting both local and API models. This comprehensive solution aims to enhance productivity and streamline AI workflows in a private environment.