langtest: Test Language Models for Safety and Quality

Overview

LangTest is a Python library for assessing whether language models and NLP systems behave safely and effectively. It helps teams test for issues such as robustness failures, bias, fairness concerns, and accuracy problems, rather than relying on a single performance score. It supports NLP workflows using Spark NLP and Hugging Face Transformers, as well as testing LLMs from services including OpenAI, Cohere, AI21, Hugging Face Inference API, and Azure OpenAI.

The library generates and runs test cases through a Harness interface, and can use test results to augment training data for select models. It is aimed at NLP practitioners and teams building or evaluating language-model applications who need repeatable checks across tasks such as named entity recognition, translation, text classification, and question answering.

Key Features

• Generate and run more than 60 types of language-model and NLP tests.
• Assess robustness, bias, representation, fairness, and accuracy.
• Test NLP models used for named entity recognition, translation, and text classification.
• Connect to Spark NLP and Hugging Face Transformers workflows.
• Test LLMs through supported providers and APIs, including OpenAI and Azure OpenAI.
• Apply automated training-data augmentation based on test results for select models.
• Use benchmark datasets for evaluation scenarios including question answering and summarization.

Use Cases

• NLP engineers can probe named entity recognition models for robustness before deploying them in production pipelines.
• Responsible-AI teams can evaluate language models for demographic or stereotype-related bias.
• LLM developers can test question-answering or summarization systems against relevant benchmark datasets.
• Model owners can use test results to guide data augmentation for models supported by that workflow.

Getting Started

Install the Transformers integration and run a test harness:

pip install 'langtest[transformers]'

See the README and documentation for model setup, test configuration, and examples.

Alternatives

• AuditNLG: AuditNLG focuses on generated-text factuality, safety, and instruction-following, while langtest covers broader model testing such as robustness, bias, and fairness.

Considerations

LangTest is an evaluation and testing library, not a replacement for model hosting or a complete model-development platform. Its integrations cover a range of NLP frameworks and LLM services, but teams should confirm that their model, task, and provider are supported before adopting it. A Python runtime is required, and some workflows may depend on the external model services or frameworks they use. The project is actively maintained, with recent commits and a release in the past year; work is shared across multiple contributors rather than being concentrated in one. The Apache-2.0 license permits commercial use, modification, and distribution, subject to its terms.