What's new
Fixed: TOON scalar substitution honesty
A Hacker News commenter pointed out that some TOON scalar substitutions (null → ∅, newline → ↲) actually cost more tokens, not fewer. They were right.
We ran tiktoken verification (o200k_base + cl100k_base):
| Substitution | Original | TOON | Verdict |
|---|---|---|---|
null → ∅
| 1 token | 2 tokens | ❌ Worse |
| ` | |||
| → ↲` | 1 token | 2 tokens | ❌ Worse |
true → T
| 1 token | 1 token | = Same |
false → F
| 1 token | 1 token | = Same |
Fix: Removed all scalar substitutions that don't save tokens. null stays null, true/false stay as-is. Token savings now come from structural compression only (removing JSON braces, quotes, brackets).
Changed
_toon_scalar(): honest implementation, no token-wasting substitutions- README: all benchmark numbers updated with tiktoken verification, tokenizer version noted
- TOON savings: 56% → 20-40% for results (honest), 61% for schemas, 97% for discovery
- CHANGELOG dates fixed (2025 → 2026)
Verified with tiktoken
{"content":[{"type":"text","text":"hello"}]} → hello (85% savings)
{"type":"object","properties":{"q":{"type":"string"}}} → q:s* (83% savings)
{"name":"search","count":3,"cached":true,"error":null} → name:search|count:3|cached:true|error:null (24% savings)
Full Changelog: v0.2.3...v0.3.0