mw00/project-maya v1.1.10

Project Maya v1.1.10

on GitHub

The Strix Halo and Gorgon Halo release: faster answers and faster prompts on the Ryzen AI Max APUs, a faster cold start from a hard disk, and prompts through both links of an NVLink pair.

Update: the dashboard's About > Update, or git pull, then ./setup.sh (Windows: START-MAYA.bat). The engine recompiles; nothing is downloaded again.

What's new

• Strix Halo and Gorgon Halo: faster answers. On the Ryzen AI Max APUs (Radeon 8050S / 8060S / 8065S, gfx115x), answers come faster with the same text. On a Gorgon Halo (Radeon 8065S, Ryzen AI Max+ PRO 495 with 192 GB, 160 GB of it for the GPU, which holds all of Maya-S; Windows 11), 256-token answers to short prompts run at 28.4-29.6 tokens/s instead of 19.7, and at 23.2-23.5 instead of 18.4-18.5 after 3K- and 8K-token prompts; longer answers (about 600 tokens) hold 26.3 and 23.4. It is on by default on these APUs only; every other GPU decodes as in v1.1.9. A 128 GB Strix Halo, whose GPU holds less of the model, is not measured yet. STRATA_GLM_MTP_DECODE=0 turns it off, 1 tries it on one GPU elsewhere (usually slower where experts stream from RAM); it stays off with --parallel or a DFlash2 drafter. Thanks to @boxwrench for testing on a Strix Halo.
• Every expert stays in GPU memory where it fits (#101). Where the GPU's budget holds every expert plus the room a prompt borrows (a 192 GB Gorgon Halo under Windows), the start loads all of them, and a long prompt no longer moves experts out to read them back from the SSD: answers after an 8K-token prompt +7%. On by default on AMD (STRATA_GLM_COMPLETE_POOL, STRATA_GLM_RESIDENT_SKIP); NVIDIA keeps its pool unless they are set to 1.
• Faster prompts on Strix Halo and Gorgon Halo (#98 and #113 by @boxwrench). A prompt's routed experts run in one fused int8 pass, and the wave32 WMMA attention is the default on gfx1151: on the Radeon 8065S, 3K- and 8K-token prompts read at 271 and 278 tokens/s instead of 233 and 253. STRATA_GLM_PREFILL_FUSED=0 and STRATA_GLM_PREFILL_ATTN=f16q give the old paths.
• A faster cold start from a hard disk (#182 by @hmartiniano). The start reads a GGUF pack's experts in the order they lie in its files, as it already did for EXL3, instead of the order the routes rank them, so a hard disk reads them in one pass instead of seeking from one to the next; how much faster depends on the disk. Which experts go to VRAM and RAM is unchanged.
• Two GPUs over NVLink: prompts through both links (#121 by @q8atnight). With STRATA_GLM_DUAL_PREFILL=1, in a prompt of one chunk the waiting GPU stages every second expert over its own PCIe link and NVLink (Maya-M on two RTX 3090 with NVLink: 1K- and 4K-token prompts +28% and +24%). The start checks that peer copies arrive intact and keeps it off on a machine whose GPUs report peer access but deliver wrong bytes.

NVIDIA GPUs answer as in v1.1.9: the same tokens and speed.

Checked

• Radeon 8065S (Gorgon Halo, 192 GB, Windows 11, HIP SDK 7.2), Maya-S: the same greedy answers with the faster decode on and off over four prompts (3K and 8K-token documents, code, a short question) and 256- to 616-token answers; the GEMV parity tests; the server end to end; two requests at once (--parallel 2).
• Two Tesla V100s: the same tokens as v1.1.9; the engine's tests; the server end to end.
• The engine builds on Linux, Windows (MSVC) and Windows + AMD; the GitHub checks.
• Not measured yet: a 128 GB Strix Halo (its GPU holds less of the model), Strix Halo under Linux with this release, and the Radeon 8050S / 8060S. A ./maya.sh --bench report in an issue helps.