Niko1221/Strata v0.1.7

Strata v0.1.7

on GitHub

Faster code edits (prompt lookup), temperature and sampling that actually work, and three fixes.

• Prompt lookup, on by default. When the reply repeats text from the conversation (an agent rewriting a file, quoting code or a document), the engine now drafts up to 5 tokens from the earlier copy. It only does this where it has measured that it pays: code edits are 6-11% faster (e.g. 101 -> 112 tokens/s), and other text is unchanged. Drafts are checked by the model like the MTP's, so the output is the same. Turn it off with --suffix-draft 0 in your config's args.
• Per-request sampling (#19, thanks @j-luwierski). temperature, top_p, top_k, min_p, seed, repetition_penalty, frequency_penalty and presence_penalty from the request are honored. Before this, every request decoded greedily whatever the client sent. You can set defaults in an optional sampling block in your strata-*.json. Greedy requests (no temperature, or 0) are exactly as before, at the same speed. The same seed gives the same text. Image requests still decode greedily.
• Replies no longer stop at 1,024 tokens (#18, thanks @coolio986; fixes #17). A missing, 0 or -1 max_tokens now means "the rest of the context".
• Setup on a short path (#20). Installing to a folder like G:\Strata crashed step 6 with IndexError: 1; fixed.
• Setup on too little RAM (#20). Setup now says at step 1 when the PC has less RAM than the smallest model needs. All of the model's experts live in RAM (34-50 GB); the GPU holds a copy of the most-used ones, so a bigger GPU does not lower the RAM needed.

Updating: get the latest files (git pull or download the repo again), then run START-HERE.bat with the model window closed. It updates the engine by itself before starting. This version's server needs engine 0.1.7, and setup takes care of that. On Linux, ./setup.sh builds the engine from source as before.

The ready-made Strata engine for Windows (RTX 30 / 40 / 50: sm_86, sm_89, sm_120 + PTX), CUDA 13.0.

You don't need to download this yourself: START-HERE.bat fetches it (and NVIDIA's cuBLAS from pip), so no compiler or CUDA Toolkit is needed, only an NVIDIA driver 580 or newer.

Contents: strata.exe (the engine), strata-vision.exe (the optional image encoder), BUILD.json.