github activeing123/mcptoon v0.3.0
v0.3.0 — tiktoken-verified TOON

latest releases: v0.8.40, v0.8.39, v0.8.38...
3 months ago

What's new

Fixed: TOON scalar substitution honesty

A Hacker News commenter pointed out that some TOON scalar substitutions (null → ∅, newline → ↲) actually cost more tokens, not fewer. They were right.

We ran tiktoken verification (o200k_base + cl100k_base):

Substitution Original TOON Verdict
null → ∅ 1 token 2 tokens ❌ Worse
`
→ ↲` 1 token 2 tokens ❌ Worse
true → T 1 token 1 token = Same
false → F 1 token 1 token = Same

Fix: Removed all scalar substitutions that don't save tokens. null stays null, true/false stay as-is. Token savings now come from structural compression only (removing JSON braces, quotes, brackets).

Changed

  • _toon_scalar(): honest implementation, no token-wasting substitutions
  • README: all benchmark numbers updated with tiktoken verification, tokenizer version noted
  • TOON savings: 56% → 20-40% for results (honest), 61% for schemas, 97% for discovery
  • CHANGELOG dates fixed (2025 → 2026)

Verified with tiktoken

{"content":[{"type":"text","text":"hello"}]}  →  hello         (85% savings)
{"type":"object","properties":{"q":{"type":"string"}}}  →  q:s*        (83% savings)
{"name":"search","count":3,"cached":true,"error":null}  →  name:search|count:3|cached:true|error:null  (24% savings)

Full Changelog: v0.2.3...v0.3.0

Don't miss a new mcptoon release

NewReleases is sending notifications on new releases.