Big release: a Cython scoring kernel joins mypyc in compiled wheels, every model retrained on a deduplicated corpus, UTF-7 fixed in both directions, and a guarantee that detect() never returns an encoding that can't decode your complete input.
Performance
- Compiled wheels now score bigram profiles through a small Cython kernel alongside mypyc, and the pair is 4.7x faster than the pure wheel on CPython 3.14.
_kernel.pystays plain Python (PyPy and pure wheels run it interpreted, unchanged), and detection output is bit-identical. The kernel declares itself safe without the GIL, so free-threaded CPython scales instead of silently re-enabling the GIL on import: 3.14t runs the whole suite in ~340ms across 8 threads, the fastest configuration measured. Compiled builds now need both hooks:HATCH_BUILD_HOOK_ENABLE_MYPYC=true HATCH_BUILD_HOOK_ENABLE_CUSTOM=true. - Added support for CPython 3.15, including the free-threaded build. No code changes were needed.
Bug Fixes
detect()no longer returns an encoding that cannot decode the input it was given (#380, thanks @yarikoptic). When the whole input has been examined and the winner's only multi-byte evidence is an incomplete trailing sequence, the best candidate that decodes the input completely wins instead. Genuinely truncated data keeps its answer.- Fixed delimited ASCII data like
|NAME,+LAY|misdetecting as UTF-7 (#371 follow-up, thanks @agreenburg). The whole buffer must now actually decode as UTF-7, and a lone shifted character must land in a plausible script range. - Signed UTF-7 no longer reads as ASCII: the BOM stage recognizes the four UTF-7 signature prefixes when the rest of the buffer decodes as UTF-7.
- Fixed short apostrophe-heavy English being labeled Scottish Gaelic or Breton: a rare-language label on an input under 128 bytes now needs a 0.03 lead over the best mainstream language (ADR-0005).
- Fixed Hungarian text losing to a Czech reading in confusion rescoring; tied pairs are compared only under language models both encodings have.
- Fixed space-padded text matching a degenerate Serbian model at high confidence; statistical scoring now skips repeated-whitespace bigrams. This also fixes windows-1251 logs misdetecting as windows-1250 (#379).
- Fixed EBCDIC text being invisible to the early pipeline stages, and the last two EBCDIC sibling misdetections.
- Fixed training normalization gaps that starved ISO-8859-16 and the 26 pre-euro encodings at exactly their distinguishing bytes.
Improvements
- Retrained every bigram model on a refreshed, deduplicated corpus with training provenance now recorded per model.
- New ANSI-art model for cp437, trained on 16,621 text-mode art files from 16colo.rs.
- Rare-language arbitration (ADR-0005): low-confidence statistical winners from languages with no documented legacy-encoding population yield to near-tied mainstream candidates.
- Confusion-group resolution is context-aware: per-occurrence votes, word-shape demotions, art-model exemption.
- Statistical dead heats no longer resolve by candidate enumeration order.
- Training pipeline hardening after a cache-loss post-mortem.
Full Changelog: 7.5.1...7.6.0