{"name":"Evidently: Open-Source ML and LLM Observability Framework","description":"Evidently is an open-source Python library designed for evaluating, testing, and monitoring machine learning and large language model systems. It provides over 100 built-in metrics for various tasks, from data drift detection to LLM judges, supporting both tabular and text data. This framework helps ensure the quality and performance of AI-powered systems throughout their lifecycle.","github":"https://github.com/evidentlyai/evidently","url":"https://osrepos.com/repo/evidentlyai-evidently","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/evidentlyai-evidently","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/evidentlyai-evidently.md","json":"https://osrepos.com/repo/evidentlyai-evidently.json","topics":["data-science","machine-learning","llm","mlops","data-drift","observability","jupyter-notebook","generative-ai"],"keywords":["data-science","machine-learning","llm","mlops","data-drift","observability","jupyter-notebook","generative-ai"],"stars":null,"summary":"Evidently is an open-source Python library designed for evaluating, testing, and monitoring machine learning and large language model systems. It provides over 100 built-in metrics for various tasks, from data drift detection to LLM judges, supporting both tabular and text data. This framework helps ensure the quality and performance of AI-powered systems throughout their lifecycle.","content":"## Introduction\nEvidently is a powerful open-source Python library that serves as an ML and LLM observability framework. It enables users to evaluate, test, and monitor any AI-powered system or data pipeline, from tabular data to Generative AI applications. With over 100 built-in metrics, Evidently supports both offline evaluations and live monitoring, offering a modular architecture for various use cases.\n\n## Installation\nTo get started with Evidently, you can install it using pip or Conda.\n\nsh\npip install evidently\n\n\nAlternatively, for Conda users:\n\nsh\nconda install -c conda-forge evidently\n\n\nTo run the Evidently UI with demo projects, you can use `uv` or a standard virtual environment:\n\nsh\nuv run --with evidently evidently ui --demo-projects all\n\n\nIf `uv` is not installed, set up a virtual environment:\n\nsh\npip install virtualenv\nvirtualenv venv\nsource venv/bin/activate\npip install evidently\nevidently ui --demo-projects all\n\nThen, visit `localhost:8000` in your browser.\n\n## Examples\nEvidently offers comprehensive tools for both LLM and traditional ML/data evaluations, along with a monitoring dashboard.\n\n### LLM Evaluations\nHere's a quick example for LLM evaluations, checking sentiment, text length, and specific word presence in responses.\n\npython\nimport pandas as pd\nfrom evidently import Report\nfrom evidently import Dataset, DataDefinition\nfrom evidently.descriptors import Sentiment, TextLength, Contains\nfrom evidently.presets import TextEvals\n\neval_df = pd.DataFrame([\n    [\"What is the capital of Japan?\", \"The capital of Japan is Tokyo.\"],\n    [\"Who painted the Mona Lisa?\", \"Leonardo da Vinci.\"],\n    [\"Can you write an essay?\", \"I'm sorry, but I can't assist with homework.\"]],\n                       columns=[\"question\", \"answer\"])\n\neval_dataset = Dataset.from_pandas(pd.DataFrame(eval_df),\ndata_definition=DataDefinition(),\ndescriptors=[\n    Sentiment(\"answer\", alias=\"Sentiment\"),\n    TextLength(\"answer\", alias=\"Length\"),\n    Contains(\"answer\", items=['sorry', 'apologize'], mode=\"any\", alias=\"Denials\")\n])\n\nreport = Report([\n    TextEvals()\n])\n\nmy_eval = report.run(eval_dataset)\nmy_eval\n\n\n### Data and ML Evaluations\nFor data and ML evaluations, Evidently can detect data drift using various statistical methods.\n\npython\nimport pandas as pd\nfrom sklearn import datasets\n\nfrom evidently import Report\nfrom evidently.presets import DataDriftPreset\n\niris_data = datasets.load_iris(as_frame=True)\niris_frame = iris_data.frame\n\nreport = Report([\n    DataDriftPreset(method=\"psi\")\n],\ninclude_tests=\"True\")\nmy_eval = report.run(iris_frame.iloc[:60], iris_frame.iloc[60:])\nmy_eval\n\nYou can also save reports as HTML files using `my_eval.save_html(\"file.html\")`.\n\n### Monitoring Dashboard\nEvidently also provides a Monitoring UI service to visualize metrics and test results over time. You can self-host the open-source version or use [Evidently Cloud](https://www.evidentlyai.com/register){:target=\"_blank\"} for additional features like dataset management, alerting, and no-code evaluations.\n\n## Why Use Evidently\nEvidently offers a comprehensive suite of tools for evaluating various aspects of AI systems, making it invaluable for maintaining model quality and reliability. With over 100 built-in evaluations, and the ability to add custom ones, it covers a wide range of needs. It works with tabular and text data, supports evaluations for predictive and generative tasks, and provides both offline evaluations and live monitoring.\n\nKey evaluation capabilities include:\n*   **Text descriptors**: Length, sentiment, toxicity, language, special symbols, regular expression matches.\n*   **LLM outputs**: Semantic similarity, retrieval relevance, summarization quality, using model- and LLM-based evaluations.\n*   **Data quality**: Missing values, duplicates, min-max ranges, new categorical values, correlations.\n*   **Data distribution drift**: Over 20 statistical tests and distance metrics to compare shifts in data distribution.\n*   **Classification**: Accuracy, precision, recall, ROC AUC, confusion matrix, bias.\n*   **Regression**: MAE, ME, RMSE, error distribution, error normality, error bias.\n*   **Ranking (including RAG)**: NDCG, MAP, MRR, Hit Rate.\n*   **Recommendations**: Serendipity, novelty, diversity, popularity bias.\n\n## Links\n*   [Official Documentation](https://docs.evidentlyai.com){:target=\"_blank\"}\n*   [API Reference](https://evidentlyai.github.io/evidently/api-reference){:target=\"_blank\"}\n*   [Discord Community](https://discord.gg/xZjKRaNp8b){:target=\"_blank\"}\n*   [GitHub Repository](https://github.com/evidentlyai/evidently){:target=\"_blank\"}","metrics":{"detailViews":2,"githubClicks":1},"dates":{"published":null,"modified":"2026-06-30T07:09:54.000Z"}}