Scientific Compression Engine (10 Information-Theoretic Optimizations)
Major release adding a scientifically-grounded compression engine — 10 optimizations derived from Shannon entropy, Kolmogorov complexity, Bayesian inference, and rate-distortion theory.
New Features
- BPE Token-Aware Entropy — Shannon entropy on BPE token distributions, precisely matching LLM tokenizer behavior
- N-Gram Jaccard + Minhash — Order-sensitive bigram deduplication with O(1) Minhash approximation
- Cross-File Dedup v2 — Shared block detection across cached files, replaced with
[= Fn:L1-L2]references - Bayesian Mode Predictor — Learns optimal read mode per file signature from historical outcomes
- Adaptive LITM Profiles — Model-specific Lost-In-The-Middle weights (Claude/GPT/Gemini)
- Boltzmann Cache Eviction — Thermodynamic-inspired eviction scoring with configurable token budget
- Information Density Metric — Semantic tokens per output token, integrated into quality scoring
- Auto-Delta Encoding — Automatic diffs for changed files (98.9% savings for 1-line edits)
- Huffman Instruction Templates — Short codes replacing verbose instructions (52-60% shorter)
- Kolmogorov Complexity Proxy — Gzip-ratio file classification guiding mode selection
Benchmarks (on lean-ctx's own codebase)
| Scenario | Savings |
|---|---|
| Cache re-read | 99% (~8 tokens vs thousands) |
| Map mode (server.rs) | 97.6% (8,684 → 206 tokens) |
| Auto-delta (1-line edit) | 98.9% (3,325 → 38 tokens) |
| Typical 40-read session | 69.0% (149,695 → 46,332 tokens) |
Install
cargo install lean-ctxOr update: cargo install lean-ctx --force
Full changelog: https://github.com/yvgude/lean-ctx/blob/main/CHANGELOG.md
Full Changelog: v2.2.0...v2.3.0