mw00/project-maya v1.1.3

Project Maya v1.1.3

on GitHub

DFlash2 kept only where it is faster (a speed calibration on your PC), an expert profile that fills VRAM well from the first answer, an opt-in skip of the experts that would come from the SSD, and the EXL3 quant back in the Windows setup.

Update: the dashboard's About > Update, or git pull, then ./setup.sh (Windows: START-MAYA.bat). The engine recompiles; nothing is downloaded again. A setup that drafts with DFlash2 runs the speed calibration once at its first start (5-20 minutes, every step shown).

What's new

• The speed calibration (DFlash2). DFlash2 makes some PCs faster and others slower: it depends on how much of the model fits in the GPUs' memory, and on the CPU and the RAM. When the setup drafts with DFlash2 - and once, at the first start after this update, for a config that already does - Maya measures it on your PC against the MTP block (one GPU: against no drafting) and keeps the faster. It loads the model twice; each time it reads a long document, writes about 1,500 tokens of answers after a warm-up (sampled as the dashboard samples, the same questions both times) and compares the sustained speeds; DFlash2 stays only when it is more than 5% faster. ./maya.sh --speed-calibration measures again; --draft dflash2 --no-speed-calibration keeps DFlash2 without measuring.
• An expert profile for every GLM pack (#88 by @needmorevram). A start fills VRAM by a profile built from Maya-L's routing - code, math, eight languages, long documents and tool calls, with MTP drafting on - instead of expert order or one machine's usage file: on 9 GPUs a fresh start decodes 35.4 -> 40.7 tok/s. The draft layer keeps at least its card's equal share of slots. Learning across starts is opt-in (--expert-profile-save FILE), and the usage file is read only when STRATA_GLM_USAGE names it. An installed config gets the profile at its next start.
• Skip the experts that would come from the SSD (#109 by @Ajay9o9; opt-in, changes the answers). STRATA_GLM_DISK_SKIP_W=0.12: where the model does not fit in VRAM + RAM, a routed expert held by neither and with less than 12% of the route's weight is left out of the token instead of read from the SSD - an RTX 3090 with 60 GB of RAM decoded 62% faster, an RX 7900 XTX 54%. The answers move away from the full model's, more the less RAM there is: KL 0.12 with 60 GB of RAM (the top token the same 88% of the time), 0.19-0.23 with 24-40 GB (75-78%). Unset, nothing changes.
• The experimental EXL3 quant is offered on Windows again, as v1.1.0 released it; v1.1.1 had hidden it there after one crash report (#129), which we are still looking into with its reporter.
• The setup's DFlash2 choice says what it gives again - the same answers, faster - now that the speed calibration checks it on every PC.

Checked

• The speed calibration end to end on one and two Tesla V100s, where DFlash2 is slower: it kept no drafting and the MTP block there, wrote the result into the config, and a later start did not measure again; it leaves the learned profile, the usage file and the kept conversations untouched. Both outcomes are also tested with stand-in engines.
• The expert profile: on two V100s a fresh start decodes as fast as with that PC's own usage file on new topics, and 8% faster than with neither; on one V100, as fast as v1.1.2. It changes no token's computation: the one-token decode path gives the same output as v1.1.2's (KL 0) on NVIDIA and AMD. Its parity and profile tests pass.
• Skipping disk experts: unset, the same tokens as v1.1.2 on NVIDIA (one GPU and two) and AMD; set, on a V100 with part of the model on the SSD, decode 42-150% faster.
• The server end to end on two GPUs (answers, pictures, context reloads, a clean stop); the setup's tests; the engine builds on Linux, Windows and AMD; the GitHub checks pass.