hermes-lcm v0.21.0-rc2
Lossless Context Management for Hermes: a release candidate limited to the merged CJK-safe trajectory-state token chunking correction.
Highlights
- Preserves UTF-8 character boundaries when the optional
tiktokenpath splits oversized trajectory-state documents. - Keeps each decoded chunk within the requested token budget and fails explicitly when that budget cannot contain one complete Unicode character.
Changes
- #492 corrects fixed-width token-ID slicing that could split a multibyte character and introduce U+FFFD replacement characters when chunks were decoded independently.