{"name":"chardet: A Fast and Accurate Python Character Encoding Detector","description":"chardet is a powerful Python library designed for universal character encoding detection. The recently rewritten version 7 offers significant improvements in accuracy and speed, making it a robust solution for identifying text encodings. It provides features like language and MIME type detection, along with a flexible API for various use cases.","github":"https://github.com/chardet/chardet","url":"https://osrepos.com/repo/chardet-chardet","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/chardet-chardet","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/chardet-chardet.md","json":"https://osrepos.com/repo/chardet-chardet.json","topics":["Python","character encoding","encoding detection","text processing","open source","utility","library"],"keywords":["Python","character encoding","encoding detection","text processing","open source","utility","library"],"stars":null,"summary":"chardet is a powerful Python library designed for universal character encoding detection. The recently rewritten version 7 offers significant improvements in accuracy and speed, making it a robust solution for identifying text encodings. It provides features like language and MIME type detection, along with a flexible API for various use cases.","content":"## Introduction\n\nThe `chardet` library is a powerful Python tool for automatically detecting character encodings. It's essential when dealing with text data from unknown sources, preventing mojibake and ensuring proper display and processing. The latest iteration, `chardet` version 7, represents a complete rewrite, delivering unparalleled performance and accuracy compared to its predecessors and alternatives.\n\n## Installation\n\nTo install `chardet`, use pip:\n\nbash\npip install chardet\n\n\n## Examples\n\n### Quick Start\n\npython\nimport chardet\n\nchardet.detect(b\"Python is a great programming language for beginners and experts alike.\")\n# {'encoding': 'ascii', 'confidence': 1.0, 'language': 'en', 'mime_type': 'text/plain'}\n\n# UTF-8 English with accented characters\nchardet.detect(\"The naïve approach doesn't always work in complex systems.\".encode(\"utf-8\"))\n# {'encoding': 'utf-8', 'confidence': 0.84, 'language': 'en', 'mime_type': 'text/plain'}\n\n# Japanese EUC-JP\nchardet.detect(\"????????????????????????EUC-JP????????????????????????????\".encode(\"euc-jp\"))\n# {'encoding': 'EUC-JP', 'confidence': 1.0, 'language': 'ja', 'mime_type': 'text/plain'}\n\n# Get all candidate encodings ranked by confidence\ntext = \"Le café est une boisson très populaire en France et dans le monde entier.\"\nresults = chardet.detect_all(text.encode(\"windows-1252\"))\nfor r in results[:4]:\n    print(r[\"encoding\"], round(r[\"confidence\"], 2))\n# Windows-1252 0.32\n# iso8859-15 0.32\n# ISO-8859-1 0.32\n# MacRoman 0.31\n\n\n### Streaming Detection\n\nFor large files or network streams, use `UniversalDetector` to feed data incrementally:\n\npython\nfrom chardet import UniversalDetector\n\ndetector = UniversalDetector()\nwith open(\"unknown.txt\", \"rb\") as f:\n    for line in f:\n        detector.feed(line)\n        if detector.done:\n            break\nresult = detector.close()\nprint(result)\n\n\n### Encoding Era Filtering\n\nRestrict detection to specific encoding eras to reduce false positives:\n\npython\nfrom chardet import detect_all\nfrom chardet.enums import EncodingEra\n\ndata = \"?????? ???????? ???????? ?????????? ????????? ? ?????????? ??????? ??????.\".encode(\"windows-1251\")\n\n# All encoding eras are considered by default, 4 candidates across eras\nfor r in detect_all(data):\n    print(r[\"encoding\"], round(r[\"confidence\"], 2))\n# Windows-1251 0.46\n# MacCyrillic 0.42\n# KZ1048 0.2\n# ptcp154 0.2\n\n# Restrict to modern web encodings, 1 confident result\nfor r in detect_all(data, encoding_era=EncodingEra.MODERN_WEB):\n    print(r[\"encoding\"], round(r[\"confidence\"], 2))\n# Windows-1251 0.46\n\n\n### Encoding Filters\n\nRestrict detection to specific encodings, or exclude encodings you don't want:\n\npython\n# Only consider UTF-8 and Windows-1252\nchardet.detect(data, include_encodings=[\"utf-8\", \"windows-1252\"])\n\n# Consider everything except EBCDIC\nchardet.detect(data, exclude_encodings=[\"cp037\", \"cp500\"])\n\n\n## Why Use chardet 7?\n\n`chardet` version 7 stands out with remarkable improvements, making it a superior choice for character encoding detection:\n\n*   **Exceptional Accuracy**: Achieves 99.3% accuracy on a diverse test set, significantly outperforming previous versions and other libraries like `charset-normalizer`.\n*   **Blazing Fast Performance**: It is 47 times faster than `chardet` 6.0.0 with mypyc, and 1.5 times faster than `charset-normalizer` 3.4.6.\n*   **Comprehensive Detection**: Beyond encoding, it offers 95.7% accurate language detection across 49 languages and identifies over 40 binary file formats via MIME type detection.\n*   **Flexible Filtering**: Includes parameters like `include_encodings` and `exclude_encodings` to fine-tune detection, along with `encoding_era` filtering for modern web encodings.\n*   **Broad Encoding Support**: Supports 99 encodings, including EBCDIC, Mac, DOS, and various European families.\n*   **Modern Licensing**: Released under the permissive 0BSD license, a change from the previous LGPL.\n*   **Thread-Safe API**: Designed for concurrent calls, scaling well on free-threaded Python.\n*   **Familiar API**: Maintains the same public API (e.g., `detect()`, `detect_all()`, `UniversalDetector`, and the `chardetect` CLI) for an easy drop-in replacement.\n\n## Links\n\n*   **GitHub Repository**: [https://github.com/chardet/chardet](https://github.com/chardet/chardet)\n*   **Official Documentation**: [https://chardet.readthedocs.io](https://chardet.readthedocs.io)","metrics":{"detailViews":0,"githubClicks":0},"dates":{"published":null,"modified":"2026-08-02T00:19:56.000Z"}}