Niko1221/Strata v0.1.8

Strata v0.1.8

on GitHub

A new browser app with a live Monitor, an optional 4-bit KV cache, and a clear answer on the expert cache.

• The Strata app in the browser (#22, thanks @code-martin for the idea). http://127.0.0.1:8080 is now a proper app with three tabs:

  • Chat: streaming answers, the model's thinking (folded away once it answers), Markdown with code copy buttons, pictures when images are on, sampling and thinking-level settings, export and new chat. Chats stay in your browser.
  • Monitor: what the model is doing (reading the prompt, with progress, or writing at how many tokens/s); GPU load, VRAM, temperature, power and PCIe traffic; CPU, RAM and disk; the context in use; the last requests. Each has a minute of history.
  • About: the model and engine settings, your PC, and the addresses for other apps.

  Everything is served by the model's own server (no internet needed). The data behind the Monitor is also at GET /metrics. http://127.0.0.1:8080/?q=your question opens a new chat already asking.

• 4-bit KV cache, optional (#21, thanks @code-martin). --kv q4_0 stores the long-context attention cache in 4 bits after a Hadamard rotation, which spreads out the outliers. It uses about half the RAM of the default 8-bit cache (0.83 vs 1.52 GiB at 128K) and is slightly faster at very long context, at a small quality cost (document perplexity +8% at 1K, +12% at 8K; 5/5 needles found). It works with KV streaming and the MTP drafter. The default stays 8-bit. START-HERE.bat --setup asks above 8K context.

• The expert-cache warning is gone (#23). The start-up warning about the expert cache was left over from a bug fixed long ago. Measured on 2,557 tokens: the cache on and off predict the actual next token equally well (perplexity difference -0.005 +- 0.005 nats per token). The GPU and CPU round slightly differently, so their text can diverge after a while, but neither is less accurate. The engine now prints one accurate line instead.

Updating: get the latest files (git pull or download the repo again), then run START-HERE.bat with the model window closed. It updates the engine by itself before starting. This version needs engine 0.1.8, and setup installs one new Python package (psutil, for the Monitor). On Linux, ./setup.sh builds the engine from source as before.

The ready-made Strata engine for Windows (RTX 30 / 40 / 50: sm_86, sm_89, sm_120 + PTX), CUDA 13.0.

You don't need to download this yourself: START-HERE.bat fetches it (and NVIDIA's cuBLAS from pip), so no compiler or CUDA Toolkit is needed, only an NVIDIA driver 580 or newer.

Contents: strata.exe (the engine), strata-vision.exe (the optional image encoder), BUILD.json.