elixir-nx/tokenizers v0.3.1

on GitHub

Added

• Add binary variants for accessing encoding data. This way we can convert encoding
  data to tensors without additional allocations. The following functions were added:

  • get_u32_ids/1
  • get_u32_attention_mask/1
  • get_u32_type_ids/1
  • get_u32_special_tokens_mask/1

Pull requests

• Add binary variants for accessing encoding data by @jonatanklosko in #32
• Use updated GitHub Actions by @philss in #33

Full Changelog: v0.3.0...v0.3.1

Checksums

Here is the SHA256 checksums of each precompiled file:

d27eed0e8395f4065721f970a7a6f109d3f54d1b36c850d0a14844082ab90c8d  ex_tokenizers-v0.3.1-nif-2.15-x86_64-pc-windows-gnu.dll.tar.gz
0f5981b8affa087be4384f0fc9de444271556662b289333748e7a4d25f2d00c7  ex_tokenizers-v0.3.1-nif-2.15-x86_64-pc-windows-msvc.dll.tar.gz
d110ebf8a1c7e43487eb9c224dae456e72596618c0c614eb617990a47f203804  ex_tokenizers-v0.3.1-nif-2.16-x86_64-pc-windows-gnu.dll.tar.gz
7d140750c9b927434b179ddf33efd8bb77428f17556ebd8d1d8d07212c19c76a  ex_tokenizers-v0.3.1-nif-2.16-x86_64-pc-windows-msvc.dll.tar.gz
6f15a001864f3ad5911c9a75b178ead798ed66c2325fb825588bde3b36ea5c72  libex_tokenizers-v0.3.1-nif-2.15-aarch64-apple-darwin.so.tar.gz
5ba97d7fa56763b5d5235a77bfc26169baabb5aa3c7a04e2583f9b1ec92e84a3  libex_tokenizers-v0.3.1-nif-2.15-aarch64-unknown-linux-gnu.so.tar.gz
c86df80018eef4fd74174bc484401a71a06a50bacdd26478129a51d1750555f4  libex_tokenizers-v0.3.1-nif-2.15-aarch64-unknown-linux-musl.so.tar.gz
e5b7f21328f5c203fa9da3c140f10e34e034cc623f1d6c681193fd5b7724738a  libex_tokenizers-v0.3.1-nif-2.15-arm-unknown-linux-gnueabihf.so.tar.gz
9de202cf8ee208ab6f43f7474e8614918fd2a434b725444ec1741ded4a8e78bc  libex_tokenizers-v0.3.1-nif-2.15-riscv64gc-unknown-linux-gnu.so.tar.gz
58288eb84284a6c692df5789e3ca238f89a3f7858ddce33095c0a3cee47e2060  libex_tokenizers-v0.3.1-nif-2.15-x86_64-apple-darwin.so.tar.gz
64ce2cda9b2631968ba3aeee86574e62f2e33077cd8aa65137a700d6a03de6de  libex_tokenizers-v0.3.1-nif-2.15-x86_64-unknown-linux-gnu.so.tar.gz
32334ac268196e708c79dea0d5ee4c4adbc59716693454fe5d88fb302b5bfd30  libex_tokenizers-v0.3.1-nif-2.15-x86_64-unknown-linux-musl.so.tar.gz
2fefb0ef7cc98438332a8effd7788b258f245f7f2e7e9dfe13c4f61b5ce23e01  libex_tokenizers-v0.3.1-nif-2.16-aarch64-apple-darwin.so.tar.gz
a0761c7e6dc035180ea42fb15d4a6ad957bd33e5ae363a4ef04ed444bc523bb8  libex_tokenizers-v0.3.1-nif-2.16-aarch64-unknown-linux-gnu.so.tar.gz
6eb0686cf9dc77df45dbd284d7b63be4b62798b2dd930b53cc7ca0723da883aa  libex_tokenizers-v0.3.1-nif-2.16-aarch64-unknown-linux-musl.so.tar.gz
4aebed75ee0ad016877d8c64f1c841b25d9e473132dc7fa80842b01a3a42b8fc  libex_tokenizers-v0.3.1-nif-2.16-arm-unknown-linux-gnueabihf.so.tar.gz
615099e92676e1470f9b88aff8aa00f21fee1c568c80df57c26a934b55d042db  libex_tokenizers-v0.3.1-nif-2.16-riscv64gc-unknown-linux-gnu.so.tar.gz
0b3b366e06b87cc1252d90f66af73482d89e2fa4c1a7faad235ac342d83490e3  libex_tokenizers-v0.3.1-nif-2.16-x86_64-apple-darwin.so.tar.gz
b1dd8be7b57f1929ada728b1049e652b2c1ab5c100f766ae53dcd30093926a3b  libex_tokenizers-v0.3.1-nif-2.16-x86_64-unknown-linux-gnu.so.tar.gz
a0d3ed11d8287b674279695dd9ea61f10f61e9661eaa5d5837b90bce6efed612  libex_tokenizers-v0.3.1-nif-2.16-x86_64-unknown-linux-musl.so.tar.gz