What's Changed
- pyo3: update to 0.19 by @mikelui in #1322
- Add
expect()for disabling truncation by @boyleconnor in #1316 - Re-using scritpts from safetensors. by @Narsil in #1328
- Reduce number of different revisions by 1 by @Narsil in #1329
- Python 38 arm by @Narsil in #1330
- Move to maturing mimicking move for
safetensors. + Rewritten node bindings. by @Narsil in #1331 - Updating the docs with the new command. by @Narsil in #1333
- Update added tokens by @ArthurZucker in #1335
- update package version for dev by @ArthurZucker in #1339
- Added ability to inspect a 'Sequence' pre-tokenizer. by @eaplatanios in #1341
- Let's allow hf_hub < 1.0 by @ArthurZucker in #1344
- Fixing the progressbar. by @Narsil in #1353
- Preparing release. by @Narsil in #1355
- fix a clerical error in the comment by @tiandiweizun in #1356
- fix: remove useless token by @rtrompier in #1371
- Bump @babel/traverse from 7.22.11 to 7.23.2 in /bindings/node by @dependabot in #1370
- Allow hf_hub 0.18 by @mariosasko in #1383
- Allow
huggingface_hub<1.0by @Wauplin in #1385 - [
pre_tokenizers] Fix sentencepiece based Metaspace by @ArthurZucker in #1357 - udpate to version = "0.15.1-dev0" by @ArthurZucker in #1390
- Derive
CloneonTokenizer, addEncoding.into_tokens()method by @epwalsh in #1381 - Stale bot. by @Narsil in #1404
- Fix doc links in readme by @Pierrci in #1367
- Faster HF dataset iteration in docs by @mariosasko in #1414
- Add quick doc to byte_level.rs by @steventrouble in #1420
- Fix make bench. by @Narsil in #1428
- Bump follow-redirects from 1.15.1 to 1.15.4 in /tokenizers/examples/unstable_wasm/www by @dependabot in #1430
- pyo3: update to 0.20 by @mikelui in #1386
New Contributors
- @mikelui made their first contribution in #1322
- @eaplatanios made their first contribution in #1341
- @tiandiweizun made their first contribution in #1356
- @rtrompier made their first contribution in #1371
- @mariosasko made their first contribution in #1383
- @Wauplin made their first contribution in #1385
- @steventrouble made their first contribution in #1420
Full Changelog: v0.13.4.rc2...v0.15.1.rc0