chardet: A Fast and Accurate Python Character Encoding Detector

Introduction

The chardet library is a powerful Python tool for automatically detecting character encodings. It's essential when dealing with text data from unknown sources, preventing mojibake and ensuring proper display and processing. The latest iteration, chardet version 7, represents a complete rewrite, delivering unparalleled performance and accuracy compared to its predecessors and alternatives.

Installation

To install chardet, use pip:

pip install chardet

Examples

Quick Start

import chardet

chardet.detect(b"Python is a great programming language for beginners and experts alike.")
# {'encoding': 'ascii', 'confidence': 1.0, 'language': 'en', 'mime_type': 'text/plain'}

# UTF-8 English with accented characters
chardet.detect("The naïve approach doesn't always work in complex systems.".encode("utf-8"))
# {'encoding': 'utf-8', 'confidence': 0.84, 'language': 'en', 'mime_type': 'text/plain'}

# Japanese EUC-JP
chardet.detect("????????????????????????EUC-JP????????????????????????????".encode("euc-jp"))
# {'encoding': 'EUC-JP', 'confidence': 1.0, 'language': 'ja', 'mime_type': 'text/plain'}

# Get all candidate encodings ranked by confidence
text = "Le café est une boisson très populaire en France et dans le monde entier."
results = chardet.detect_all(text.encode("windows-1252"))
for r in results[:4]:
    print(r["encoding"], round(r["confidence"], 2))
# Windows-1252 0.32
# iso8859-15 0.32
# ISO-8859-1 0.32
# MacRoman 0.31

Streaming Detection

For large files or network streams, use UniversalDetector to feed data incrementally:

from chardet import UniversalDetector

detector = UniversalDetector()
with open("unknown.txt", "rb") as f:
    for line in f:
        detector.feed(line)
        if detector.done:
            break
result = detector.close()
print(result)

Encoding Era Filtering

Restrict detection to specific encoding eras to reduce false positives:

from chardet import detect_all
from chardet.enums import EncodingEra

data = "?????? ???????? ???????? ?????????? ????????? ? ?????????? ??????? ??????.".encode("windows-1251")

# All encoding eras are considered by default, 4 candidates across eras
for r in detect_all(data):
    print(r["encoding"], round(r["confidence"], 2))
# Windows-1251 0.46
# MacCyrillic 0.42
# KZ1048 0.2
# ptcp154 0.2

# Restrict to modern web encodings, 1 confident result
for r in detect_all(data, encoding_era=EncodingEra.MODERN_WEB):
    print(r["encoding"], round(r["confidence"], 2))
# Windows-1251 0.46

Encoding Filters

Restrict detection to specific encodings, or exclude encodings you don't want:

# Only consider UTF-8 and Windows-1252
chardet.detect(data, include_encodings=["utf-8", "windows-1252"])

# Consider everything except EBCDIC
chardet.detect(data, exclude_encodings=["cp037", "cp500"])

Why Use chardet 7?

chardet version 7 stands out with remarkable improvements, making it a superior choice for character encoding detection:

Exceptional Accuracy: Achieves 99.3% accuracy on a diverse test set, significantly outperforming previous versions and other libraries like charset-normalizer.
Blazing Fast Performance: It is 47 times faster than chardet 6.0.0 with mypyc, and 1.5 times faster than charset-normalizer 3.4.6.
Comprehensive Detection: Beyond encoding, it offers 95.7% accurate language detection across 49 languages and identifies over 40 binary file formats via MIME type detection.
Flexible Filtering: Includes parameters like include_encodings and exclude_encodings to fine-tune detection, along with encoding_era filtering for modern web encodings.
Broad Encoding Support: Supports 99 encodings, including EBCDIC, Mac, DOS, and various European families.
Modern Licensing: Released under the permissive 0BSD license, a change from the previous LGPL.
Thread-Safe API: Designed for concurrent calls, scaling well on free-threaded Python.
Familiar API: Maintains the same public API (e.g., detect(), detect_all(), UniversalDetector, and the chardetect CLI) for an easy drop-in replacement.