drumih/turbo-fieldfare 0.4.2

TurboFieldfare 0.4.2 — Lossless Streaming Decode

on GitHub

TurboFieldfare 0.4.2 replaces streaming detokenization with a lossless, O(1)-per-token pipeline shared by batch and streaming decode, pins byte-fallback handling to the reference decoder's semantics, and hardens tokenizer loading against malformed configurations.

Tokenization and decoding

• Replaced O(n²) streaming detokenization with per-token fragments; batch decode is now a push-loop over the same pipeline, so the two agree by construction.
• Removed the clean_up_tokenization_spaces pass that rewrote model output and silently dropped punctuation after whitespace runs from the stream.
• Byte-fallback runs commit with the reference HF decoder's semantics: runs fuse across skipped special tokens, invalid runs decode to one replacement character per byte, and trailing runs are flushed instead of dropped.
• Channel and tool markers act as detokenizer barriers, so text held in a byte-fallback run can no longer surface after a marker and route into the wrong channel.
• Flush tails and control-token deltas go through the structured decoder's channel state, so generations cut off inside the thought channel no longer leak held-back text into visible content.

Tokenizer loading

• The decoder pipeline is verified structurally against the tokenizer.json declaration — Sequence[Replace, ByteFallback, Fuse] — instead of behavioral probes that a dependency change could break.
• Special-token IDs come from added_tokens directly, replacing per-generation probe caches that ran two full library decodes per distinct token ID.
• Missing special markers and out-of-range token IDs are rejected with typed errors instead of silently binding to <unk> or crashing at load.

Community

• Expanded community benchmark coverage with M1 Air, M4 Pro, and M5 reports.

Validation

• CI passed on the release commit, including release builds and the complete serial test suite.
• Decode behavior is pinned from three directions: a differential test against swift-transformers, streaming-equals-batch round-trips over arbitrary token streams, and property tests against a reimplementation of the reference run semantics.

This is a source-only release. It does not include model weights or application binaries.

Full changelog: 0.4.1...0.4.2

Implemented in #117 and #118.

Thanks to everyone who submitted benchmark reports.