LuminosoInsight

python-ftfy: Repair Mojibake and Unicode Text

ftfy is a Python library that repairs mojibake and other common Unicode glitches in text. It suits developers cleaning text from mixed or unreliable sources, with conservative fixes designed to avoid changing text that is already valid.

PythonOther licenseLibraryText ProcessingNLP
View on GitHub Compare
Stars
4.1k
Forks
127
Stars, last 30 days
+0

Our take

Health score

30/100

How it is scored
Activity
19
Community
59
Issues
21
Pull requests
20

+0 stars in the last 30 days

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

4,068

Forks

131

Change since Nov 18, 2026: +1 stars·0 forks

Overview

ftfy repairs text that has been decoded or transformed incorrectly, especially mojibake caused by encoding mix-ups. It can also address related glitches such as HTML entities appearing in plain text and punctuation changes that interfere with decoding.

The library is for Python applications that need to clean text after it has been collected or processed. Its approach is deliberately conservative: it aims to leave sensible, correctly decoded text unchanged rather than guess at every unusual string. See the repository and documentation for details and configuration options.

Key Features

  • Repairs common UTF-8 mojibake patterns in Unicode strings.
  • Can correct multiple layers of mojibake in one pass.
  • Handles cases where curly quotes or altered spaces complicate recovery.
  • Decodes HTML entities found outside HTML, including some incorrectly capitalized forms.
  • Avoids applying a repair when text appears sensible, reducing false positives.
  • Offers text-fixing functions, configuration, and command-line usage documented by the project.

Use Cases

  • Clean text ingested from legacy databases or exports where encoding mistakes have occurred.
  • Prepare multilingual text for search, display, or analysis in a Python data pipeline.
  • Repair corrupted user-submitted or copied text before presenting it in an application.
  • Normalize scraped or aggregated content containing stray HTML entities or encoding artifacts.

Getting Started

Install from PyPI:

pip install ftfy

See the README and documentation for usage and configuration details.

Considerations

  • The project targets Python 3.
  • Repairs rely on heuristics and are intentionally conservative. They are not a general-purpose encoding detector, so ambiguous or correctly decoded text may be left unchanged.
  • The repository metadata reports the license as NOASSERTION. Review the project’s license information before using or distributing it.

Found this useful?

Share it with someone who would like python-ftfy.