huggingface/tokenizers v0.13.0

Rust v0.13.0

on GitHub

[0.13.0]

• [#1009] unstable_wasm feature to support building on Wasm (it's unstable !)
• [#1008] Decoder is now a composable trait, but without being backward incompatible
• [#1047, #1051, #1052] Processor is now a composable trait, but without being backward incompatible

Both trait changes warrant a "major" number since, despite best efforts to not break backward
compatibility, the code is different enough that we cannot be exactly sure.