github mw00/project-maya v1.1.71

one hour ago

Hy4-Flash decodes faster where the CPU computes experts and on Radeon, reads long prompts faster on Radeon, runs faster on busy PCs and reads its first prompt at full speed. It also picks the fastest drafting for each PC, can keep a long context's cache on an SSD, and there are fixes for GLM, DeepSeek, Hy4-Flash and the setup.

Measured on our test PCs (Hy4-Flash Maya-XS, v1.1.70 -> v1.1.71, tokens/s):

PC Decode (./maya.sh --bench) Prompt reading, 8K Prompt reading, 32K
1x Tesla V100 32 GB + Xeon Gold 6138 (the CPU computes the experts that don't fit) 41.0 -> 53.1 (+30%) 630 -> 630 580 -> 588 (+1%)
Radeon 8065S (Gorgon Halo) 40.9 -> 43.7 (+7%) 421 -> 422 373 -> 430 (+15%)

The first start after installing or updating measures your PC once (about 1-4 minutes, depending on the PC); the setup and the server say so before it starts, and the next starts skip it.

  • Faster Hy4-Flash decoding where the CPU computes experts. The experts kept in VRAM now follow the ones your answers use, and the CPU side no longer stalls while other programs keep the PC busy (#1082). Decode against v1.1.70: about +30% on one Tesla V100 with the CPU computing the rest and +7% on a Radeon 8065S (Gorgon Halo). HY4_HOT=0 turns the expert following off.
  • Faster long prompts on Radeon. Hy4-Flash reads 32K-token prompts about 15% faster on a Radeon 8065S (RDNA 3 / 3.5 cards in general).
  • The first prompt reads at full speed. The first start measures the PC before it says ready, so the first prompt no longer reads slowly while the PC is being measured.
  • Repeatable on several GPUs. The first answers match their repeats and the same start gives the same prompt route on one GPU, the layer split and tensor parallel.
  • The fastest drafting for each PC (--drafting auto). The first start tries the drafting choices and keeps the fastest for that PC; --speed-calibration measures again, and --drafting off|on|mtp|dfly2 still picks by hand. On an 8 GB card --mtp no longer stops the start.
  • A long context's cache on an SSD (--kv-spill auto). Where the RAM cannot hold a long context, the older positions go to a file on a drive (--kv-spill-dir). HY4_KV_MORE=1 (opt-in) streams the cache also where it fits but the freed VRAM holds more experts.
  • Windows with several NVIDIA cards. The VRAM Windows keeps for itself is measured for each card, and --room-report shows the cards side by side.
  • Opt-ins. "web_ui": false in the config (or MAYA_WEB_UI=0) serves the API alone, with no web page and no browser opening at the start (#1089); --gpu-fetch / HY4_FETCH lets the GPU fetch the experts it needs from RAM itself, auto timing the options once.
  • Fix: GLM-5.3 with drafting no longer stalls (#860). A regression since v1.1.53 that could end in an error is gone.
  • Fix: DeepSeek no longer hangs on two AMD GPUs (#1086, #1027). --no-expert-cache is no longer needed.
  • Fix: DeepSeek after a refused unbuffered read (#1076). The file keeps being read unbuffered, and the refusal is logged once.
  • Fix: Hy4-Flash after an agent client rewinds a chat (#1091). Wrong answers after rewinds to different depths are gone (#1092).
  • Fix: running the setup again keeps every setting (#1089). Your GPUs, port, host, API key, model management, idle unload, loop guard and the rest of the server settings stay; --port / --host / --api-key given to the setup win, and a copy of each changed config is kept.
  • Fix: --rebuild rebuilds every engine the PC uses, not only the first model's.
  • Fix: AMD read-only model folders (#1090). The start says once that the GEMM timings cannot be kept, and HY4_LT_STORE=<file> keeps them elsewhere.
  • DeepSeek: the CPU side's copies stay on their own NUMA node on Linux PCs with two nodes or more (#1083).
  • Credits. The CPU-side fix for busy PCs (#1082) was contributed by @lastbl0c, the DeepSeek NUMA copies (#1083) by @needmorevram, the AMD GEMM timings fix (#1090) by @Antero128 and the fix for Hy4-Flash after rewinds (#1092) by @lij768423-svg. The rest of this release is by @mw00 (peasantsmith).

Full list: CHANGELOG.md.

Supported by @wjHunter, @os-groot, @davedgd, @jeanluchayes, @sugattor, @vikcious, @ThatSimpleTech, @router0mail and @davidfernengel.

Maya is free; if it helps you, you can sponsor it: https://github.com/sponsors/mw00

Don't miss a new project-maya release

NewReleases is sending notifications on new releases.