pypi sentencepiece 0.2.3
v0.2.3
on Python PyPI

3 hours ago

Release Notes v0.2.3

This release introduces SentencePiece Lite, a zero-dependency C++20 runtime, new vocabulary-selection algorithms for training (Auto-character Coverage and Sparse L1 Unigram pruning), a Bazel (Bzlmod) build, a large number of security and robustness fixes found by OSS-Fuzz and community reports, and a cleanup of long-deprecated APIs and flags.

1. SentencePiece Lite (New Zero-Dependency Runtime)

  • New lite/ runtime: A compact (~45 KB stripped, < 2k LOC) C++20 tokenizer engine that runs directly on memory-mapped FlatBuffers model binaries (.spm.fb) with no runtime dependency on Protobuf or Abseil (#1330, Commit a2cb4d8). See lite/README.md.

2. Training: New Vocabulary Selection Algorithms

  • Auto-character Coverage (--auto_character_coverage): New TrainerSpec.auto_character_coverage option for both BPE and Unigram that globally optimizes the allocation between single characters and multi-character subwords within the vocabulary budget, while keeping lossless round-trip via byte fallback (Commits e23ece8, df18339, d34a6ce). See doc/auto_character_coverage.md.
  • Sparse L1 Unigram pruning (--use_sparse_pruning): Optional L1-regularized EM vocabulary selection with Post-Lasso debiased refit for the Unigram model, replacing the discrete heuristic pruning (#1295, Commit fd53b5c). Required when --auto_character_coverage is used with Unigram. See doc/sparse_l1_unigram.md.
  • PruneUnreachableSentencePieces: Remove pieces that can never appear in a Viterbi path during Unigram training, and fix the discrete pruning loop order (Commits b292ce5, 59c3b80).
  • Custom pretokenizer callback: Trainer now accepts a std::function<std::vector<std::string>(absl::string_view)> pretokenizer, exposed to Python via the pretokenizer= training argument; pretokenization_delimiter and callback are mutually exclusive (#1287, Commit d1b6696).
  • unk_id = -1 with byte_fallback: Training with --unk_id=-1 is now allowed when --byte_fallback=true; null-byte pieces are treated as UNUSED when byte fallback is enabled (fixes #1308, Commit c2fbf86).
  • spm_eval metrics: Added Bytes/Tok, Model PPL, Empirical PPL, and Coverage metrics (Commit 01305c9).

3. Python Wrapper

  • Pickling: SentencePieceProcessor now preserves encode options and the full instance dict (including subclasses) when pickled (#1349, Commits 4e66c53, 52b1a47).
  • Normalizer fixes:
    • Preserve add_dummy_prefix / escape_whitespaces / remove_extra_whitespaces flags loaded from a model or spec instead of silently forcing them to False (#1337, fixes #1334).
    • Bind SentencePieceNormalizer.LoadFromSerializedNormalizerSpec, fixing AttributeError with normalizer_spec= (#1333).
  • Decode / Encode edge cases:
    • Accept the legacy out_type alias in DecodeIds / DecodePieces (#1336).
    • Fix Decode() of empty input for non-str return types (#1289).
    • Fix encode() with an empty batch and out_type='offset_mapping' (#1320).
    • Fix decoding a batch whose first sequence is empty (#1321).
  • Buffer protocol & integer inputs: Accept 8/16/32/64-bit signed/unsigned integer buffers with saturation clamping and zero-copy lifetime safety; accept general sequences (including tuples) in decode APIs (Commit b3717da).
  • Free-threaded Python: Fix a pretokenizer callback race on free-threaded builds (cp314t) and GIL-safety issues in _TrainFromMap (Commits ca0f388, ee1d3dd). Free-threaded wheels are now tested on 3.14t / 3.15t.
  • Python version support: Requires Python >= 3.10; added Python 3.15 classifier and cibuildwheel 4.3.1 (Commits fa40c16, ab49d75).
  • AIX: Fixed the Python build on AIX (#1359).

4. Performance Optimizations

  • BPE training: Replaced the per-symbol position btree_set with a vector and parallelized initial indexing; removed the 65,535-character sentence length limit by replacing 16-bit packed positions with a Position struct (Commits a44ce92, b3717da).
  • Sentence preprocessing: Parallelized TrainerInterface::LoadSentences and SplitSentencesByWhitespace, with RunBatch-based dynamic load balancing for normalization (Commits a44ce92, d1b6696).
  • Decode: Avoid intermediate allocations and skip redundant StrReplaceAll in Decode() (Commit a44ce92).
  • Unigram: Avoid unnecessary allocations in Lattice::NBest / Lattice::Sample, heap-free score precomputation in BoundedPriorityQueue::Gc, and deduplicated corpus byte counting (Commit ee1d3dd).
  • ParallelEncode / Normalizer: Skip unused SentencePieceText allocations in ParallelEncode overloads and skip alignment bookkeeping when not requested (Commit ee1d3dd).

5. Security & Hardening

  • OSS-Fuzz findings: Bounded hypothesis allocations, agenda shrink rounds, and periodic timeout checks in unigram::Lattice::NBest to fix timeouts/OOM; added piece-ID bounds checks in IdToPiece, GetScore, IsControl, IsUnknown, IsUnused, IsByte (fixes #1342, #1344, #1347, Commit 2e432ca).
  • Lattice bounds: Reject invalid positions/lengths in Lattice::Insert, verify EOS/begin nodes before Viterbi backtrace, and guard against null prev pointers and cycles (Commit fa40c16).
  • Empty-piece guards: Guard empty pieces in PrefixMatcher::PrefixMatch and Unigram PieceToId (#1326, #1325).
  • Charsmap validation: Validate leaf node offsets and safely slice values in DecompileCharsMap; fix Builder::BuildNFC_CFMap and regenerate nfc_cf.tsv (#1329, #1327).
  • Trainer guards: RET_CHECK on zero character/word counts, clamp thread pool size to 1024, fix swap-and-pop bug when removing empty normalized sentences, and fix NaN in Gumbel sampling / thread-local RNG re-seeding (Commits b3717da, 70ed5fc, a44ce92).
  • Lite hardening: Alignment and bounds checks on FlatBuffers models, direct-mapping validation, and endian-swap overflow protection (Commit b3717da).
  • General input validation: Additional edge-case hardening across C++, Lite, and Python (Commit 5d21631).

6. Normalization

  • nmt_nfkc: Normalize U+061C (ARABIC LETTER MARK) to space (#1331).

7. Deprecations & Removals

  • Removed deprecated C++ / Python APIs (#1286, Commit 442a7d7):
    • *AsSerializedProto / *AsImmutableProto variants, ImmutableSentencePieceText, ImmutableNBestSentencePieceText.
    • SampleEncodeAndScore*, CalculateEntropy.
    • SetVocabulary / ResetVocabulary / LoadVocabulary.
    • Python: OverrideNormalizerSpec, __getstate_for_pickle__.
  • Deprecated TrainerSpec fields (now marked [deprecated = true] and removed from spm_train CLI flags): accept_language, self_test_sample_size, mining_sentence_size, training_sentence_size, train_extremely_large_corpus, enable_differential_privacy, differential_privacy_noise_level, differential_privacy_clipping_threshold (Commits d29cf00, 470b7d5).
  • Removed SelfTestData from sentencepiece_model.proto (Commit 470b7d5).
  • Removed contrib/nlcodec fast BPE trainer and other dead code / unused build dependencies (Commits b6910fd, c91fbef).

8. Build System & Packaging

  • Bazel (Bzlmod) support: New MODULE.bazel / BUILD.bazel covering the runtime, trainer, CLI binaries, shared libraries, and test suites; darts-clone, libsais, and abseil-cpp are taken from the Bazel Central Registry (#1310, #1311, #1312, #1313, closes #1280).
  • C++20 baseline: Raised the required C++ standard to C++20 across CMake, Bazel, and the Python extension, including MSVC (#1316, Commit c0232a9).
  • Dependency upgrades: Protobuf v36.2, Abseil 20260817.0, FlatBuffers 25.12.19, GoogleTest 1.18.0, Google Benchmark 1.10.0; removed legacy protobuf-lite and builtin_pb (#1296, Commit 443fedc).
  • GoogleTest migration: Replaced the custom testharness with GoogleTest / GoogleMock via CMake FetchContent (Commits 87ab362, c5ae4a9).
  • CMake modernization: Modernized configuration and target export, protoc cross-compilation support (CMAKE_CROSSCOMPILING_EMULATOR), and self-contained sdist (Commits 6bcf11d, e9e7600, 16e057b, 4e5215e).

9. Documentation

  • GitHub Pages site: Added an MkDocs Material documentation site with integrated navigation (Commits a24b577, a705d0a).
  • New guides: doc/auto_character_coverage.md, doc/sparse_l1_unigram.md, doc/bazel.md, and lite/README.md.
  • Updated references: Reorganized options.md, special_symbols.md, C++ / normalization / CMake docs, and the tokenizer comparison cheat sheet for 0.2.3; README examples now use return_type (Commits ee1d3dd, 70ed5fc, cc5aab4).
Check out latest releases or
releases around sentencepiece 0.2.3

Don't miss a new sentencepiece release

NewReleases is sending notifications on new releases.

Get notifications