chardet 7.7.0

on Python PyPI

Big release: raising max_bytes to cover a huge input no longer makes detect() slow, every answer now decodes the window it was detected from, and markup declarations are read the way browsers read them. A lot of this came from new contributors, so thank you all!

Performance

• Raising max_bytes on large inputs no longer gets slow. On compiled wheels, detect(data, max_bytes=len(data)) on 272 MiB takes about 0.2s for cp1252 and 0.5s for Shift_JIS, down from several seconds. UTF-8 validation and a final decode of the winner still read the whole window, and the other stages stop at a 256 KiB evidence cap (ADR-0006). The default 200 KB cap is still the fastest path, and default calls are unchanged.
• The pure-Python wheel on CPython scores bigram models through operator.itemgetter instead of a Python loop, making detect() about 1.25x faster and detect_all() about 1.35x faster. (#404, thanks @deadstrobe5)
• Fixed three scans that went quadratic on hostile input: the UTF-7 + scan on line-wrapped base64 (128 KiB went from 5.4s to 1.2ms), unclosed <meta and <?xml tags in the markup scan, and the EBCDIC charset-declaration regex. (#403, thanks @Samin061)

Bug Fixes

• Every answer now decodes the examined window under the name you get back. BOMs, BOM-less UTF-16/32 null patterns, ISO-2022/HZ escapes, and <meta charset> declarations used to be trusted without decoding the rest of the window, so a confident answer could still raise further in. The 5.x/6.x display names (EUC-JP, SHIFT_JIS, Big5, ISO-2022-JP) are only used when those narrower codecs decode it too. (#394 and #396, thanks @Samin061)
• Markup declarations follow the WHATWG prescan: declarations inside comments are ignored, a charset= inside another attribute (like a refresh URL) no longer beats a real <meta charset>, quoted or spaced content charsets work, and an ASCII page declaring utf-16le no longer comes back as UTF-16-LE. (#389 and #398, thanks @SashaMIT and @xiehuanyi)
• PEP 263 cookies resolve the way tokenize.detect_encoding resolves them, including only counting a line-2 cookie when line 1 is blank or a comment. (#391, thanks @xiehuanyi)
• Fixed BOM-bearing UTF-32 coming back as UTF-16 when max_bytes cuts a code unit. (#405, thanks @AryanRG)
• Fixed language labels on Unicode text tilting toward accent-heavy languages, so a French paragraph with a few accents no longer comes back as Hungarian. Mislabels on the test corpus's Unicode files dropped from 140 to 104.
• Fixed a mostly-ASCII Windows-1252 file with a single Ö detecting as HP-Roman8, and English with a lone É detecting as MacRoman. (#383, thanks @aadsm)
• Fixed null-separated records like find -print0 output coming back as UTF-16-LE with Chinese as the language.
• prefer_superset=True no longer swaps in a code page that can't decode the data, and now respects include_encodings/exclude_encodings. (#393, thanks @00200200)
• chardetect and UniversalDetector no longer mistake input cut exactly at the byte cap for the whole stream. (#392, thanks @00200200)
• lookup_encoding's cache is capped at 1024 entries, since charset names come from untrusted markup. (#390, thanks @Samin061)
• Plus smaller fixes to detect_all() ranking, EBCDIC XML MIME types, and the deprecated chardet.equivalences shim. The changelog has the full list.

Improvements

• New -m/--mime-type flag for chardetect shows the MIME type next to the encoding, like file -i. (#384, thanks @smi11 for the request)
• Visual-order Hebrew is now detected, and cp864 Arabic is trained on contextually shaped text in visual order, the way real cp864 systems stored it.
• EncodingInfo.is_multibyte is now is_multibyte_cjk. The old name still works with a DeprecationWarning and goes away in 8.0.
• pipeline/markup.py is now compiled by mypyc (15 modules total), and CI enforces 100% test coverage.

Full Changelog: 7.6.0...7.7.0