# Evidently: Open-Source ML and LLM Observability Framework

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/evidentlyai-evidently
Generated for open source discovery and AI-assisted research.

Evidently is an open-source Python library designed for evaluating, testing, and monitoring machine learning and large language model systems. It provides over 100 built-in metrics for various tasks, from data drift detection to LLM judges, supporting both tabular and text data. This framework helps ensure the quality and performance of AI-powered systems throughout their lifecycle.

GitHub: https://github.com/evidentlyai/evidently
OSRepos URL: https://osrepos.com/repo/evidentlyai-evidently

## Summary

Evidently is an open-source Python library designed for evaluating, testing, and monitoring machine learning and large language model systems. It provides over 100 built-in metrics for various tasks, from data drift detection to LLM judges, supporting both tabular and text data. This framework helps ensure the quality and performance of AI-powered systems throughout their lifecycle.

## Topics

- data-science
- machine-learning
- llm
- mlops
- data-drift
- observability
- jupyter-notebook
- generative-ai

## Repository Information

Last analyzed by OSRepos: Tue Jun 30 2026 08:09:54 GMT+0100 (Western European Summer Time)
Detail views: 2
GitHub clicks: 1

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction
Evidently is a powerful open-source Python library that serves as an ML and LLM observability framework. It enables users to evaluate, test, and monitor any AI-powered system or data pipeline, from tabular data to Generative AI applications. With over 100 built-in metrics, Evidently supports both offline evaluations and live monitoring, offering a modular architecture for various use cases.

## Installation
To get started with Evidently, you can install it using pip or Conda.

sh
pip install evidently


Alternatively, for Conda users:

sh
conda install -c conda-forge evidently


To run the Evidently UI with demo projects, you can use `uv` or a standard virtual environment:

sh
uv run --with evidently evidently ui --demo-projects all


If `uv` is not installed, set up a virtual environment:

sh
pip install virtualenv
virtualenv venv
source venv/bin/activate
pip install evidently
evidently ui --demo-projects all

Then, visit `localhost:8000` in your browser.

## Examples
Evidently offers comprehensive tools for both LLM and traditional ML/data evaluations, along with a monitoring dashboard.

### LLM Evaluations
Here's a quick example for LLM evaluations, checking sentiment, text length, and specific word presence in responses.

python
import pandas as pd
from evidently import Report
from evidently import Dataset, DataDefinition
from evidently.descriptors import Sentiment, TextLength, Contains
from evidently.presets import TextEvals

eval_df = pd.DataFrame([
    ["What is the capital of Japan?", "The capital of Japan is Tokyo."],
    ["Who painted the Mona Lisa?", "Leonardo da Vinci."],
    ["Can you write an essay?", "I'm sorry, but I can't assist with homework."]],
                       columns=["question", "answer"])

eval_dataset = Dataset.from_pandas(pd.DataFrame(eval_df),
data_definition=DataDefinition(),
descriptors=[
    Sentiment("answer", alias="Sentiment"),
    TextLength("answer", alias="Length"),
    Contains("answer", items=['sorry', 'apologize'], mode="any", alias="Denials")
])

report = Report([
    TextEvals()
])

my_eval = report.run(eval_dataset)
my_eval


### Data and ML Evaluations
For data and ML evaluations, Evidently can detect data drift using various statistical methods.

python
import pandas as pd
from sklearn import datasets

from evidently import Report
from evidently.presets import DataDriftPreset

iris_data = datasets.load_iris(as_frame=True)
iris_frame = iris_data.frame

report = Report([
    DataDriftPreset(method="psi")
],
include_tests="True")
my_eval = report.run(iris_frame.iloc[:60], iris_frame.iloc[60:])
my_eval

You can also save reports as HTML files using `my_eval.save_html("file.html")`.

### Monitoring Dashboard
Evidently also provides a Monitoring UI service to visualize metrics and test results over time. You can self-host the open-source version or use [Evidently Cloud](https://www.evidentlyai.com/register){:target="_blank"} for additional features like dataset management, alerting, and no-code evaluations.

## Why Use Evidently
Evidently offers a comprehensive suite of tools for evaluating various aspects of AI systems, making it invaluable for maintaining model quality and reliability. With over 100 built-in evaluations, and the ability to add custom ones, it covers a wide range of needs. It works with tabular and text data, supports evaluations for predictive and generative tasks, and provides both offline evaluations and live monitoring.

Key evaluation capabilities include:
*   **Text descriptors**: Length, sentiment, toxicity, language, special symbols, regular expression matches.
*   **LLM outputs**: Semantic similarity, retrieval relevance, summarization quality, using model- and LLM-based evaluations.
*   **Data quality**: Missing values, duplicates, min-max ranges, new categorical values, correlations.
*   **Data distribution drift**: Over 20 statistical tests and distance metrics to compare shifts in data distribution.
*   **Classification**: Accuracy, precision, recall, ROC AUC, confusion matrix, bias.
*   **Regression**: MAE, ME, RMSE, error distribution, error normality, error bias.
*   **Ranking (including RAG)**: NDCG, MAP, MRR, Hit Rate.
*   **Recommendations**: Serendipity, novelty, diversity, popularity bias.

## Links
*   [Official Documentation](https://docs.evidentlyai.com){:target="_blank"}
*   [API Reference](https://evidentlyai.github.io/evidently/api-reference){:target="_blank"}
*   [Discord Community](https://discord.gg/xZjKRaNp8b){:target="_blank"}
*   [GitHub Repository](https://github.com/evidentlyai/evidently){:target="_blank"}