huggingface/tokenizers python-v0.9.3

Python v0.9.3

on GitHub

Fixed

• [#470]: Fix hanging error when training with custom component
• [#476]: TemplateProcessing serialization is now deterministic
• [#481]: Fix SentencePieceBPETokenizer.from_files

Added

• [#477]: UnicodeScripts PreTokenizer to avoid merges between various scripts
• [#480]: Unigram now accepts an initial_alphabet and handles special_tokens correctly