Faster sampled answers, pictures in Claude Code's tool results, and a build fix for PCs with two CUDA toolkits.
Update: the dashboard's About > Update, or git pull, then ./setup.sh (Windows: START-MAYA.bat). The engine recompiles; nothing is downloaded again.
What's new
- Sampled answers decode faster (#92 by @0xPreDa). The sampler now picks the top_k tokens in one pass instead of k scans of the vocabulary, and runs the draw chain on one thread: a sample takes about 0.07 ms instead of 1.5 ms at top_k 20 on an RTX 4090 (11 ms at top_k 64, which
top_k0 also means). Sampled decode on 4x RTX 4090: +13%. The picks are the same, token for token - checked on NVIDIA and AMD. - Pictures in tool results reach the model (#120 by @CoolJohn-lab, fixes #119). A picture inside an Anthropic
tool_result- how Claude Code's Read tool returns an image - was dropped before the image encoder; now the model sees it. - A second CUDA toolkit (#124 by @fuad00): with
--nvccpointing at a toolkit other than the system's, the engine and the image encoder now link that toolkit's runtime and cuBLAS; they took the system's before. A build that already linked the wrong ones needs itsbuild/folder removed once. - README: a 2x RTX 4090 row (#123 by @fuad00).
Checked
- The sampler: the same picks as v1.1.1, token for token, in three sampled settings on a Tesla V100 and on a Strix Halo (AMD, Windows), and its parity test on both.
- Pictures in tool results: the server tests, and the model's own chat template renders the picture.
- The toolkit fix: the engine and the image encoder build on Windows; Ubuntu's own CUDA package builds as before.
- The engine builds on Linux, Windows and AMD; the GitHub checks pass.