huggingface/tokenizers python-v0.10.1

Python v0.10.1

on GitHub

Fixed

• [#616]: Fix SentencePiece tokenizers conversion
• [#617]: Fix offsets produced by Precompiled Normalizer (used by tokenizers converted from SPM)
• [#618]: Fix Normalizer.normalize with PyNormalizedStringRefMut
• [#620]: Fix serialization/deserialization for overlapping models
• [#621]: Fix ByteLevel instantiation from a previously saved state (using __getstate__())