huggingface/tokenizers v0.15.1

on GitHub

What's Changed

• udpate to version = "0.15.1-dev0" by @ArthurZucker in #1390
• Derive Clone on Tokenizer, add Encoding.into_tokens() method by @epwalsh in #1381
• Stale bot. by @Narsil in #1404
• Fix doc links in readme by @Pierrci in #1367
• Faster HF dataset iteration in docs by @mariosasko in #1414
• Add quick doc to byte_level.rs by @steventrouble in #1420
• Fix make bench. by @Narsil in #1428
• Bump follow-redirects from 1.15.1 to 1.15.4 in /tokenizers/examples/unstable_wasm/www by @dependabot in #1430
• pyo3: update to 0.20 by @mikelui in #1386
• Encode special tokens by @ArthurZucker in #1437
• Update release for python3.12 windows by @ArthurZucker in #1438

New Contributors

• @steventrouble made their first contribution in #1420

Full Changelog: v0.15.0...v0.15.1