Changes: Big improvements in speed for BPE (Both training and tokenization) (#165) Fixes: Do not open all files directly while training (#163) There was a bug in ByteLevel PreTokenizer that caused offsets to be wrong if a char got split up in multiple bytes. (cf #156) The LongestFirst truncation strategy had a bug (#174)
huggingface/