Niko1221/Strata v0.1.14

Strata v0.1.14

on GitHub

Fixed: the remaining stall with the IQ models, and a lot for Linux.

• Fixed: generation could stop mid-answer with the IQ models (IQ2_XS, IQ3_XXS, IQ3_S) (#31, #29, thanks @Zauberio, @marcosandresacevedo, @Lufen34). The thread dumps from 0.1.13's stall reports showed the engine waiting forever inside the NVIDIA driver while it copied experts to the GPU in the middle of a verify window. The experts are now copied by a GPU kernel instead, which needs no driver call at that point (Q2_0 always worked this way). About 1% slower output with IQ3_S. --pcie-mode dma restores the old way.
• Linux: 0.1.13 did not compile (std::isfinite, #49, thanks @pdsmike, @siganos, @Lufen34). Fixed.
• Linux: an engine that stopped on an error could hang instead of exiting, so the server never started it again. Fixed; errors in the middle of an answer now also reach the client as an error instead of an empty answer.
• Linux setup: ./setup.sh runs on a fresh clone again; Python without venv support is detected; Arch and other distributions work when CUDA is installed (/opt/cuda is found, #46, thanks @siganos); after a failed update compile, the old engine starts again instead of failing on a setting it cannot read (#49); a failed build is retried once (#45, thanks @pipeob0).
• Linux: one expert-pool worker per physical core instead of per hyper-thread (#40, thanks @jugeeeemu, who measured +16% on a 13700K).
• WSL works, with KV streaming off there (the driver pins only about 1 GB of RAM); the engine now says so at start instead of failing at the first request.
• 24-32 GB cards fill their VRAM with experts (#46): the expert profile ranks all 24,576 experts (the cache stopped at 8,000 before). tools/make_profile.py builds one from your own prompts.
• AVX2-only CPUs (Ryzen 5000, X3D, Ice Lake...) decode the IQ models' rows once per verify window instead of once per token (#43, thanks @pipeob0): about 5% faster there.
• Windows large pages actually work for accounts with "Lock pages in memory" (#42, thanks @pipeob0), and the log says why when they are refused.
• --port works for existing installs (#32, thanks @uvhat); the Monitor keeps the last 500 requests with totals and a Show all button (#35, thanks @XeonG); the server window shows how far a long prompt is (N of M tokens).

Updating: get the latest files (git pull or download the repo again), then run START-HERE.bat with the model window closed: it updates the engine to 0.1.14 by itself. On Linux, run ./setup.sh: it compiles the new engine.

The ready-made Strata engine for Windows (RTX 30 / 40 / 50: sm_86, sm_89, sm_120 + PTX), CUDA 13.0.

You don't need to download this yourself: START-HERE.bat fetches it (and NVIDIA's cuBLAS from pip), so no compiler or CUDA Toolkit is needed, only an NVIDIA driver 580 or newer.

Contents: strata.exe (the engine), strata-vision.exe (the optional image encoder), BUILD.json.