huggingface/tokenizers python-v0.13.0

Python v0.13.0

on GitHub

[0.13.0]

• [#956] PyO3 version upgrade
• [#1055] M1 automated builds
• [#1008] Decoder is now a composable trait, but without being backward incompatible
• [#1047, #1051, #1052] Processor is now a composable trait, but without being backward incompatible

Both trait changes warrant a "major" number since, despite best efforts to not break backward
compatibility, the code is different enough that we cannot be exactly sure.