elixir-nx/tokenizers v0.3.0

on GitHub

Added

• Add option to use cache when downloading pretrained files. We check the ETAG of
  the file before trying to download it. This introduces the :use_cache and :cache_dir
  options to the Tokenizers.from_pretrained/2 function.

• Support adding special tokens when creating a tokenizer. This allows a pretrained
  tokenizer to be loaded with additional special tokens.

  This change adds the :additional_special_tokens option to the Tokenizers.from_pretrained/2
  function.

• Add support for the riscv64gc-unknown-linux-gnu target, which is useful for Nerves
  projects running on 64 bits RISC-V computers.
  This means that we are precompiling the project to run on those machines.

Changed

• Change minimum required version of Rustler Precompiled to ~> 0.6. With this, we have
  the aarch64-unknown-linux-musl and riscv64gc-unknown-linux-gnu as default targets.
  But we also drop support for the NIF version 2.14.

Pull requests

• Add option to use cache when download pretrained files by @philss in #23
• Fix typos by @kianmeng in #24
• Update rustler_precompiled to v0.5.5 by @philss in #25
• Feature/add special tokens by @travismorton in #26
• Add riscv64gc-unknown-linux-gnu for RISC-V Nerves devices by @fhunleth in #27
• Simplify release workflow by @philss in #29
• Allow usage of castore 1.0 by @edennis in #30
• Updates to release v0.3.0 by @seanmor5 in #31

New Contributors

• @kianmeng made their first contribution in #24
• @travismorton made their first contribution in #26
• @fhunleth made their first contribution in #27
• @edennis made their first contribution in #30

Full Changelog: v0.2.0...v0.3.0
Official changelog: https://github.com/elixir-nx/tokenizers/blob/main/CHANGELOG.md

Checksums

Here is the list of SHA256 checksums of the precompiled files:

e73178ccbea2e63b7b86afcbcff1a01a10e2b69901f424e703e8a38ff74c1dcf  ex_tokenizers-v0.3.0-nif-2.15-x86_64-pc-windows-gnu.dll.tar.gz
a562cac8feb8b3964860a461897be64d27a2a999f8cb237334f552ad1e14ff8a  ex_tokenizers-v0.3.0-nif-2.15-x86_64-pc-windows-msvc.dll.tar.gz
da32956b0346021376fd14e2c484c1150b1ed197577d5c8f99193b3b1c815ae2  ex_tokenizers-v0.3.0-nif-2.16-x86_64-pc-windows-gnu.dll.tar.gz
aaf9fbd3ffbfced33e7871f3f036067156433fd935f0b42778f47982aeee4717  ex_tokenizers-v0.3.0-nif-2.16-x86_64-pc-windows-msvc.dll.tar.gz
9937b4a50fbd03e48483484b09aa95446a4b4a67eb07e74f480193eaac73087a  libex_tokenizers-v0.3.0-nif-2.15-aarch64-apple-darwin.so.tar.gz
56038e1045c674c3a321a5bade5467c0cb599ede1a58df4fd1428f9eefe979b9  libex_tokenizers-v0.3.0-nif-2.15-aarch64-unknown-linux-gnu.so.tar.gz
bd1b27a026f3f8f5b0d60a7a2415bb19823fc5a726b173da24ca8fc1534e49f4  libex_tokenizers-v0.3.0-nif-2.15-aarch64-unknown-linux-musl.so.tar.gz
52e527a66d2806321c2297362fbac27429fb8e60f29386637c66ce78e8dadcf3  libex_tokenizers-v0.3.0-nif-2.15-arm-unknown-linux-gnueabihf.so.tar.gz
f43190147eafbc812607b61c9459a6c47aa526c492c13e2157cd613e382ba35e  libex_tokenizers-v0.3.0-nif-2.15-riscv64gc-unknown-linux-gnu.so.tar.gz
36a6d6691a3b3fa6d56ddf37fa2f696ae79bf2b991c17ee7cccbb2b9dda1719b  libex_tokenizers-v0.3.0-nif-2.15-x86_64-apple-darwin.so.tar.gz
f48f95b1a5373f75a555a78899916acb48d1c003795a02b0ee0001b7c92c8a51  libex_tokenizers-v0.3.0-nif-2.15-x86_64-unknown-linux-gnu.so.tar.gz
e79bac2f303dafdf7b8c7946c77a5f7be2e90db40f3102d1bdd3929d309fc949  libex_tokenizers-v0.3.0-nif-2.15-x86_64-unknown-linux-musl.so.tar.gz
5fdc9b12dcdc0eaf6ac7ba8a3608af33abe902299f81ec953fd7eac6de8477de  libex_tokenizers-v0.3.0-nif-2.16-aarch64-apple-darwin.so.tar.gz
0e40ed777f14a41df64526b50224392723eabd83188fa79910c6baceca4c72f1  libex_tokenizers-v0.3.0-nif-2.16-aarch64-unknown-linux-gnu.so.tar.gz
287d5896f09562c25527105998f1468f97649fb7a8fec60a04ad53d113a5d9df  libex_tokenizers-v0.3.0-nif-2.16-aarch64-unknown-linux-musl.so.tar.gz
dc8fdaa04935d32ab6a0cc00d06c1f3b747eb08eb09ab8b9d2143157c2de436e  libex_tokenizers-v0.3.0-nif-2.16-arm-unknown-linux-gnueabihf.so.tar.gz
db79c2f522a4b39940d98550b29b47e9f8ba45c1d3f3a00a19278a3b977c65b7  libex_tokenizers-v0.3.0-nif-2.16-riscv64gc-unknown-linux-gnu.so.tar.gz
b343d1dd3e2467e4e54624bbd8ff9dcefe48dd23ddd59f8ab54e808cc19a5865  libex_tokenizers-v0.3.0-nif-2.16-x86_64-apple-darwin.so.tar.gz
3850d19c6b2e635e46475a0ee0cc6c3dc62b0325a8c232fdf1137cc98e48af9d  libex_tokenizers-v0.3.0-nif-2.16-x86_64-unknown-linux-gnu.so.tar.gz
bdfeb7816dec218b04a024d2163f1fe7080eaca6e9b18823c3ac8f9cef3ecfbc  libex_tokenizers-v0.3.0-nif-2.16-x86_64-unknown-linux-musl.so.tar.gz