pypi tokenizers 0.15.2
v0.15.2

latest releases: 1.0.0rc3, 0.23.3, 1.0.0rc2...
2 years ago

What's Changed

Big shoutout to @rlrs for the fast replace normalizers PR. This boosts the performances of the tokenizers:
image

New Contributors

Full Changelog: v0.15.1...v0.15.2rc1

Don't miss a new tokenizers release

NewReleases is sending notifications on new releases.