tokenizers 0.20.1

Release v0.20.1

on Python PyPI

What's Changed

The most awaited offset issue with Llama is fixed 🥳

• Update README.md by @ArthurZucker in #1608
• fix benchmark file link by @152334H in #1610
• Bump actions/download-artifact from 3 to 4.1.7 in /.github/workflows by @dependabot in #1626
• [ignore_merges] Fix offsets by @ArthurZucker in #1640
• Bump body-parser and express in /tokenizers/examples/unstable_wasm/www by @dependabot in #1629
• Bump serve-static and express in /tokenizers/examples/unstable_wasm/www by @dependabot in #1630
• Bump send and express in /tokenizers/examples/unstable_wasm/www by @dependabot in #1631
• Bump webpack from 5.76.0 to 5.95.0 in /tokenizers/examples/unstable_wasm/www by @dependabot in #1641
• Fix documentation build by @ArthurZucker in #1642
• style: simplify string formatting for readability by @hamirmahal in #1632

New Contributors

• @152334H made their first contribution in #1610
• @hamirmahal made their first contribution in #1632

Full Changelog: v0.20.0...v0.20.1