Open Source Text Processing Tools
Text processing covers methods for transforming, analyzing, and formatting written data. It helps address tasks such as cleaning inconsistent input, detecting character encodings, splitting text into meaningful units, extracting structured information, comparing strings, and converting between formats. These operations support applications ranging from content management and search to language analysis and everyday command-line workflows.
Open source tools in this area include libraries for parsing, segmentation, normalization, markup conversion, and string comparison, as well as utilities for editing text in files or streams. When choosing a tool, consider its license, maintenance activity, documentation, language and runtime requirements, performance, and compatibility with existing systems. These tools are useful for developers, data practitioners, content teams, and anyone building workflows that handle text.
26 repositories · updated October 17, 2026
mistune: Parse Markdown in Python
Mistune is a Python library for converting Markdown to HTML, with configurable renderers and plugins. It suits applications that need Markdown processing in Python and want to extend parsing or output behavior.
ActiveHealth 46/100
chardet: Detect Text Encodings in Python
chardet identifies character encodings in byte data, including through incremental processing of large files and streams. Its Python API and CLI suit applications that need to interpret text of unknown origin, with language and MIME-type information also available.
ActiveHealth 59/100
fuzzywuzzy: Match Similar Strings in Python
FuzzyWuzzy helps Python applications compare strings that differ in spelling, formatting, or word order. The repository is archived and points users to its renamed successor, TheFuzz.
ArchivedHealth 34/100
python-Levenshtein: Fast String Distance and Similarity in Python
A C extension for Python that computes Levenshtein edit distance, string similarity, and related string operations. It suits developers who need these calculations in Python or as a standalone C library, with maintenance and GPL licensing caveats to consider.
InactiveHealth 27/100
pangu.py: Add Spacing Between CJK Text and Latin Characters
pangu.py is a Python library and command-line tool that inserts spaces between Chinese, Japanese, or Korean text and Latin letters, numbers, and symbols. It suits workflows that need consistent mixed-script typography in strings or text files.
ActiveHealth 37/100
pyfiglet: Render Text as ASCII Art in Python
pyfiglet renders text as FIGlet-style ASCII art using Python, either as a library or a command-line tool. It suits Python applications and terminal workflows that need decorative text without invoking an external FIGlet program.
MaintainedHealth 47/100
python-pinyin: Convert Chinese Characters to Pinyin
A Python library and command-line tool for converting Chinese characters to pinyin. It handles polyphonic characters, offers multiple pronunciation styles, and can use custom dictionaries to adjust readings.
QuietHealth 25/100
awesome-slugify: Create Configurable Slugs from Text
A Python library for turning text into configurable, readable slugs for URLs, filenames, and other identifiers. It supports transliteration, Unicode-preserving output, custom replacements, and unique slug generation.
InactiveHealth 24/100
python-slugify: Create Unicode-Aware Slugs in Python
python-slugify turns text into URL-style slugs, with options for transliteration, Unicode retention, and output rules. It suits Python applications that need readable identifiers, especially when teams can manage algorithm and backend changes carefully.
ActiveHealth 54/100
unicode-slugify: Create Unicode-Aware URL Slugs
unicode-slugify is a Python library for turning text into URL-friendly slugs while preserving Unicode characters. It also supports ASCII transliteration and configurable character rules for different slug formats.
InactiveHealth 31/100
pyparsing: Build Text Parsers in Python
pyparsing is a Python library for defining parsing-expression grammars directly in code. Use it to turn structured text into accessible parse results without writing a separate grammar file or relying only on regular expressions.
ActiveHealth 45/100
python-nameparser: Parse Human Names into Components
A Python library that splits human names into structured fields such as given name, family name, title, suffix, and nickname. It suits applications that need to organize, format, or compare names while accounting for varied naming conventions.
ActiveHealth 70/100