{"name":"python-ftfy: Effortlessly Fixing Mojibake and Unicode Glitches","description":"ftfy is a powerful Python library designed to automatically correct \"mojibake\" and other common glitches in Unicode text. It intelligently detects and fixes encoding mix-ups, transforming unreadable characters into their intended form. This tool is essential for developers and data scientists working with messy text data, ensuring readability and data integrity.","github":"https://github.com/LuminosoInsight/python-ftfy","url":"https://osrepos.com/repo/luminosoinsight-python-ftfy","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/luminosoinsight-python-ftfy","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/luminosoinsight-python-ftfy.md","json":"https://osrepos.com/repo/luminosoinsight-python-ftfy.json","topics":["Python","Text Processing","Unicode","Mojibake","Data Cleaning","NLP","Open Source"],"keywords":["Python","Text Processing","Unicode","Mojibake","Data Cleaning","NLP","Open Source"],"stars":null,"summary":"ftfy is a powerful Python library designed to automatically correct \"mojibake\" and other common glitches in Unicode text. It intelligently detects and fixes encoding mix-ups, transforming unreadable characters into their intended form. This tool is essential for developers and data scientists working with messy text data, ensuring readability and data integrity.","content":"## Introduction\n\n`ftfy`, short for \"fixes text for you\", is a robust Python library developed by Robyn Speer, designed to effortlessly correct \"mojibake\" and other common glitches in Unicode text. It intelligently detects patterns of characters that were clearly meant to be UTF-8 but were decoded incorrectly, transforming unreadable text into its intended, comprehensible form. This tool is invaluable for anyone working with diverse and potentially corrupted text data.\n\n## Installation\n\nGetting started with `ftfy` is straightforward. It is a Python 3 package and can be installed using `pip`:\n\nbash\npip install ftfy\n\n\nIf you have both Python 2 and 3 installed, you might need to use `pip3`:\n\nbash\npip3 install ftfy\n\n\n## Examples\n\n`ftfy` excels at fixing a variety of text corruption issues. Here are some real-world examples of its capabilities:\n\n**Fixing basic mojibake (encoding mix-ups):**\n\npython\nimport ftfy\nprint(ftfy.fix_text('✔ No problems'))\n# Expected output: '? No problems'\n\n\n**Correcting multiple layers of mojibake simultaneously:**\n\npython\nimport ftfy\nprint(ftfy.fix_text('The Mona Lisa doesn’t have eyebrows.'))\n# Expected output: \"The Mona Lisa doesn't have eyebrows.\"\n\n\n**Handling mojibake with \"curly quotes\" applied on top:**\n\npython\nimport ftfy\nprint(ftfy.fix_text(\"l’humanité\"))\n# Expected output: \"l'humanité\"\n\n\n**Decoding HTML entities outside of HTML, even with incorrect capitalization:**\n\npython\nimport ftfy\nprint(ftfy.fix_text('P&EACUTE;REZ'))\n# Expected output: 'PÉREZ'\n\n\nA key principle of `ftfy` is to avoid false positives. It will not alter text that is already sensible, ensuring data integrity:\n\npython\nimport ftfy\nprint(ftfy.fix_text('IL Y MARQUÉ…'))\n# Expected output: 'IL Y MARQUÉ…' (unchanged)\n\n\n## Why Use ftfy?\n\nDealing with corrupted or improperly encoded text data can be a significant challenge, consuming valuable development time and leading to frustrating debugging sessions. `ftfy` automates the complex process of identifying and correcting these errors, making it a \"handy piece of magic\" as described by users. Its ability to recover original strings from seemingly impossible mojibake patterns is a testament to its robust design.\n\n`ftfy` has been widely adopted and cited in major Natural Language Processing (NLP) research, proving its reliability and effectiveness in crucial data processing steps. Testimonials highlight its ability to save \"a large amount of frustrating dev work\" and make life \"livable again\" for developers.\n\n## Further Resources\n\nTo learn more about `ftfy` and its extensive capabilities, explore the following resources:\n\n*   **GitHub Repository:** <a href=\"https://github.com/rspeer/python-ftfy\" target=\"_blank\">https://github.com/rspeer/python-ftfy</a>\n*   **Official Documentation:** <a href=\"https://ftfy.readthedocs.org/en/latest/\" target=\"_blank\">https://ftfy.readthedocs.io/en/latest/</a>\n*   **Zenodo Citation:** <a href=\"http://doi.org/10.5281/zenodo.2591652\" target=\"_blank\">http://doi.org/10.5281/zenodo.2591652</a>","metrics":{"detailViews":2,"githubClicks":4},"dates":{"published":null,"modified":"2025-10-21T07:00:47.000Z"}}