mw00/project-maya v1.1.8

Project Maya v1.1.8

on GitHub

Contributors' fixes and additions: speculative sampling on Turing cards, AMD's tuning table for the installed library, prompt-reading progress on the stream, and tool calls with templates that open thinking only when asked.

Update: the dashboard's About > Update, or git pull, then ./setup.sh (Windows: START-MAYA.bat). The engine recompiles; nothing is downloaded again.

What's new

• Speculative sampling on Turing cards (#161 by @sociolog). With STRATA_GLM_SPEC_SAMPLE=1, its draft kernel needed more registers than a Turing card (RTX 20, T4) has for a block of 1024 threads, and the launch failed wherever the MTP block sat on such a card. The kernel is now bounded to fit; other cards run it as before.
• AMD: the tuning table for the installed hipBLASLt (#162 by @Tsury, #158). The setup passes every prompt tuning table made for the card's architecture, and the engine takes the one made for the installed hipBLASLt version. Before, it passed only the newest, and a PC with another library version fell back to plain hipBLAS. A Strix Halo (gfx1151) table for hipBLASLt 100401 is added: on @Tsury's Linux PC with that library, a tool-calling session's prompts took 29 s instead of 52 s. Run ./maya.sh --setup once to get the new list into your config.
• Prompt progress on the stream (#163 by @eliaskg). A streamed chat request with "return_progress": true gets prompt_progress chunks (total, cache, processed, time_ms) while a long prompt is read, so a client can draw a progress bar. Nothing changes unless a request asks; non-streamed requests and the Anthropic API are as before.
• Tool calls with templates that open thinking only when asked (#148 by @beverm2391). With Max thinking, a model whose chat template opens <think> only when enable_thinking is true returned its tool calls inside the reasoning text. The template now gets the thinking default the reply parser assumes. Maya's GLM template renders as before.
• Docs: the CPU image encoder with the AMD engine on Linux, set up by hand (#159 by @Tsury); what to install when the CUDA profiler or NVTX headers are missing (#160 by @Tsury).

Checked

• The same tokens as v1.1.7 on two Tesla V100s (a 6K prompt, the CPU lane on and off), prompts at 8K and 32K and decode - greedy, and sampled with speculative sampling - at the same speed.
• The prompt progress end to end: a 41K-token prompt streamed 10 progress chunks, processed rising to total; none without the flag; a non-streamed request answers as before.
• Maya's GLM template renders exactly as before at every thinking level, with and without tools.
• The server end to end on two GPUs, the parity tests, and the GitHub checks.