KV streaming: faster long contexts.
At 64K context and more, the KV cache (the model's memory of the conversation) now lives in RAM. Only the part the attention is actually reading stays on the GPU. The VRAM this frees holds more experts, so decoding is faster:
- Q2_0 at 262K: 50.9 -> 62.6 tokens/s. Experts in VRAM go from 1,589 to 3,872.
- Q2_0 at 128K: about +6%.
- IQ3_S at 128K: about +1% (its experts are bigger, so the freed VRAM holds fewer of them).
Measured on an RTX 5070 12 GB, Ryzen 5 7600, 64 GB.
- It doesn't change what the model computes. The attention reads exactly the same values; only where the KV lives changes. With the same experts on the GPU, the output is identical token for token. All needle tests pass (1K to 262K).
- It costs RAM: ~13.7 KB per context token, so 1.7 GB at 128K and 3.4 GB at 262K. Setup only turns it on when that fits next to the model.
- Prompt reading: about 1% slower at 262K.
Turning it on: new installs get it automatically at 64K+ context. For an existing install, run START-HERE.bat --setup once and pick the same model and context; nothing is downloaded again. You can also add --kv-resident 32768 to the args in your strata-*.json.
Also in this engine: an opt-in suffix drafter (--suffix-draft). It proposes drafts when the reply repeats text from the context. It's off by default.
Updating: get the latest files (git pull or download the repo again), then run START-HERE.bat with the model window closed. It updates the engine by itself before starting. On Linux, ./setup.sh builds the engine from source as before.
The ready-made Strata engine for Windows (RTX 30 / 40 / 50: sm_86, sm_89, sm_120 + PTX), CUDA 13.0.
You don't need to download this yourself: START-HERE.bat fetches it (and NVIDIA's cuBLAS from pip), so no compiler or CUDA Toolkit is needed, only an NVIDIA driver 580 or newer.
Contents: strata.exe (the engine), strata-vision.exe (the optional image encoder), BUILD.json.