This is the last v0 release, we are moving to v1!!
More details coming soon 👀
What's Changed
- Update daachorse from 1.0.1 to 3.0.0 by @musicinmybrain in #2050
- Ci test fixes by @sebpop in #2079
- Avoid full vocab clone in get_vocab_size() by @eunseo9311 in #2074
- ci: pin benchmark jobs to a larger dedicated runner by @ArthurZucker in #2084
- ci: target benchmark runners by group, not nonexistent label by @ArthurZucker in #2090
- Bindings: acquire model read-lock once per call instead of per pre-token by @sebpop in #2072
- ci: split audit in a separate job by @SBrandeis in #2103
- ci: install libcairo on benchmark runners by @SBrandeis in #2100
- feat(ci): enable npm's trusted publisher for releases by @McPatate in #2111
- chore: bump pyo3 to version 0.29 by @SBrandeis in #2115
- perf: add benchmarks for the
decodeAPI by @SBrandeis in #2099 - ci: cache HF test data and authenticate Hub downloads to avoid rate limits by @SBrandeis in #2102
- Implementing Parity-aware BPE by @cimeister in #1974
- Make doc builds faster by @mishig25 in #2158
- chore(deps-dev): bump websocket-driver from 0.7.4 to 0.7.5 in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #2208
- chore(deps-dev): bump shell-quote from 1.8.3 to 1.10.0 in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #2244
- chore: remove stale examples (kills 50% of dependabot traffic) by @ArthurZucker in #2285
- chore(node): drop 8 unused devDependencies by @ArthurZucker in #2286
- ci: pin every action to one SHA, drop stale audit suppressions by @ArthurZucker in #2291
- chore(node): take node security alerts from 33 to 0 by @ArthurZucker in #2287
- fix(python): pin the ruff rule set so a ruff release can't redden main by @ArthurZucker in #2292
- Add a ParityBpeTrainer example and list it in the trainers API docs by @cimeister in #2217
New Contributors
- @eunseo9311 made their first contribution in #2074
- @cimeister made their first contribution in #1974
Full Changelog: v0.23.1...v0.23.2