AuditNLG: Check and Improve Trust in Generated Text

Overview

AuditNLG is a Python toolkit for checking whether generated text is factually grounded, safe, and compliant with explicit or implicit instructions. It brings multiple evaluation methods behind a shared interface, helping researchers and developers assess text without wiring each evaluator into their workflow separately.

When a result needs improvement, the toolkit can ask a language model to revise the text and provide explanations for detected issues. It is an evaluation and inference layer, not a toolkit for training or tuning models.

Key Features

• Scores factual consistency, including against supplied knowledge.
• Checks for unsafe content using model-based and service-based methods.
• Evaluates whether output follows instructions and constraints such as format, style, or audience.
• Offers alternative factualness, safety, and constraint evaluators, including local Hugging Face models and external services.
• Generates candidate revisions through configurable prompt-refinement strategies.
• Produces explanations for detected trust issues.
• Accepts JSON samples and can use shared knowledge across samples.

Use Cases

• Researchers comparing factual-consistency methods on generated summaries or other text, especially when they need to try multiple evaluators.
• Product teams screening model responses for unsafe language before review or deployment, with the understanding that scores are not a substitute for policy review.
• Developers checking whether generated responses follow required formats, styles, or audience instructions.
• Teams investigating a problematic output and using explanations or suggested revisions to guide human review.

Getting Started

Install the package:

pip install auditnlg

Some evaluators and revision methods require API credentials. See the README for method options, input format, and examples.

Alternatives

• langtest: LangTest generates and runs broader model-quality tests for robustness, bias, fairness, and accuracy, rather than focusing on factuality, safety, and instruction adherence in generated text.

Considerations

The project is marked quiet: it has had no commits in the last 90 days and no release in over a year. Four contributors are recorded, and contribution is not flagged as concentrated, but the repository reports no CI or tests, so adopters should independently verify behavior and compatibility.

Evaluation quality depends on the selected models, services, and training data. Several methods need vendor API tokens, while local-model options may require downloading and running model resources; CUDA is optional in the documented command-line example. The README also notes that the toolkit does not provide model training or tuning and disclaims responsibility for third-party model outputs.