tokenizers 0.22.0

v0.22.0

on Python PyPI

What's Changed

• Bump on-headers and compression in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #1827
• Implement from_bytes and read_bytes Methods in WordPiece Tokenizer for WebAssembly Compatibility by @sondalex in #1758
• fix: use AHashMap to fix compile error by @b00f in #1840
• New stream by @ArthurZucker in #1856
• [docs] Add more decoders by @pcuenca in #1849
• Fix missing parenthesis in EncodingVisualizer.calculate_label_colors by @Liam-DeVoe in #1853
• Update quicktour.mdx re: Issue #1625 by @WilliamPLaCroix in #1846
• remove stray comment by @sanderland in #1831
• Fix typo in README by @aisk in #1808
• RUSTSEC-2024-0436 - replace paste with pastey by @nystromjd in #1834
• Tokenizer: Add native async bindings, via py03-async-runtimes. by @michaelfeil in #1843

New Contributors

• @b00f made their first contribution in #1840
• @pcuenca made their first contribution in #1849
• @Liam-DeVoe made their first contribution in #1853
• @WilliamPLaCroix made their first contribution in #1846
• @sanderland made their first contribution in #1831
• @aisk made their first contribution in #1808
• @nystromjd made their first contribution in #1834
• @michaelfeil made their first contribution in #1843

Full Changelog: v0.21.3...v0.22.0rc0