github huggingface/tokenizers v0.19.0rc0

latest releases: v0.20.1, v0.20.1rc1, v0.20.0...
pre-release6 months ago

Bumping 3 versions because of this: https://github.com/huggingface/transformers/blob/60dea593edd0b94ee15dc3917900b26e3acfbbee/setup.py#L177

What's Changed

  • chore: Remove CLI - this was originally intended for local development by @bryantbiggs in #1442
  • [remove black] And use ruff by @ArthurZucker in #1436
  • Bump ip from 2.0.0 to 2.0.1 in /bindings/node by @dependabot in #1456
  • Added ability to inspect a 'Sequence' decoder and the AddedVocabulary. by @eaplatanios in #1443
  • 🚨🚨 BREAKING CHANGE 🚨🚨: (add_prefix_space dropped everything is using prepend_scheme enum instead) Refactor metaspace by @ArthurZucker in #1476
  • Add more support for tiktoken based tokenizers by @ArthurZucker in #1493
  • PyO3 0.21. by @Narsil in #1494
  • Remove 3.13 (potential undefined behavior.) by @Narsil in #1497
  • Bumping all versions 3 times (ty transformers :) ) by @Narsil in #1498

Full Changelog: v0.15.2...v0.19.0rc0

Don't miss a new tokenizers release

NewReleases is sending notifications on new releases.