Niko1221/Strata v0.1.45

Strata 0.1.45

on GitHub

0.1.45: Qwen3.6-35B-A3B reads pictures, serves several requests at once and runs up to 40% faster; switches for multi-user servers; Mac and setup fixes. Answers of the Qwen3.8-Flash-Next models are the same as in 0.1.44.1 (byte-identical in our identity tests on every GPU machine we tested: CUDA sm_60 / sm_86, HIP gfx1201, Intel A750, Mac M4). Update with UPDATE.bat (Linux: ./update.sh); setup replaces the engine with 0.1.45.

Qwen3.6-35B-A3B

• Pictures. Setup asks "Do you want images?" for Qwen3.6, downloads the pinned mmproj (0.9 GB) and writes --vision. It is the same projector family as Flash-Next's, so the same image encoder reads it. Checked against llama.cpp on the same pictures, same prompt ids, teacher-forced on llama.cpp's own answer: first token the same in 5 of 6 cases (the sixth, two pictures in one request, differs at a near tie), mean KL 0.0005 to 0.035 against 0.002 to 0.056 between llama.cpp's own CPU and GPU runs. Answers on the test pictures are the same text llama.cpp gives.
• Several requests at once (--parallel N at setup, "parallel": N in the config, up to 8). Every window carries one token of each conversation, so the dense weights and every expert two conversations share are read once. Every slot decodes the tokens it decodes alone (10 prompts x 128 greedy, 2 / 4 / 8 slots, slots reused, a 2.5K-token prompt read beside decoding slots). An R9700 32 GB with every expert in VRAM, 128 new tokens per request, five interleaved pairs against llama.cpp's llama-server -np N -fa on: 1 / 2 / 4 / 8 requests 83.9 / 173.5 / 252 / 325 tok/s against 73.8 / 107 / 165 / 208 (1.14x to 1.61x, 5 of 5 pairs each). --batch-mtp (opt-in) lets every slot verify draft tokens: one request 1.6x, two 1.2x, four 1.1x.
• Setup offers it, with the price. Where every expert stays in VRAM, setup suggests up to 3-4 slots. On a card that does not hold every expert the slots take VRAM from the expert cache: a request alone runs 4-11% slower, and next to one request at a time WITH the draft layer, N together are only 1.0-1.15x faster (RTX 3060 12 GB at 8K context, 4 requests: 1.15x, alone -4%; RTX A4000 / Tesla P100 16 GB at 32K, 2 requests: 1.0x, alone -7 to -11%). Setup says that, offers the count the card can spare at the chosen context (12 GB: 4 slots at 8K, none at 32K; 16 GB: 2 at 32K), asks "How many requests at once?" (Enter keeps one at a time) and keeps any number you give. It is worth it for several users or agents at once, not for one chat.
• Faster. New decode kernels with the old kernels' bits (checked on the real activations: 0 mismatches in 18,360 comparisons), a prompt delta rule that loads one token ahead, and a bigger prompt chunk by default (4096 / 2048 / 1024 tokens, whatever fits; --prefill 512 brings the old one back). R9700 32 GB, UD-Q4_K_XL: decode 83.2 to 117.8 tok/s (1.42x, 5 of 5 pairs; llama.cpp 82), 4K prompt 3,453 to 6,220 tok/s (1.80x), with --mtp 2 167.7 to 183.7. UD-Q6_K_XL: decode 80.4 to 106.9. RTX 3060 12 GB (CPU-bound in the hybrid mode): decode +2%, prompt +3%. Text from the old chunk size differs a little on long prompts (mean KL 0.004 against chunk 512, the same size as the llama.cpp CPU-GPU floor); prompts shorter than the chunk are unchanged.
• Checked on: R9700 32 GB (UD-Q4_K_XL, UD-Q6_K_XL, Q8_0), RTX 3060 12 GB, Tesla P100 16 GB and RTX A4000 16 GB (UD-Q4_K_XL and UD-Q6_K_XL), with --batch 2 / 4 / 8, --batch-mtp, sampled requests and pictures: every slot equals its run alone. Not run: Windows hardware (the Windows zips contain the engine and start; Qwen3.6 on Windows AMD got a fix, see below, but is untested), Intel Arc (SYCL still refuses the family), the RTX 5070.
• A start whose buffers do not fit now says so with the numbers instead of a bare allocation failure.

Multi-user servers: opt-in switches (#2364, thanks havok2-htwo)

Everything here is opt-in; defaults are unchanged. For a server with several cards and many users:

• --batch above 8 on a layer split (16 slots on 4 cards = 4 groups of 4).
• STRATA_SLOT_COPY_DIRECT=1 and STRATA_SLOT_COPY_BATCHSYNC=1 (admission copy device to device), STRATA_CKPT_POOL=1, STRATA_CKPT_THREADS=1 (checkpoints), STRATA_ADMIT_MERGE_FIRST=1, STRATA_PIPE_BYIELD_NODRAIN=1, STRATA_BATCH_SELF_COMMIT=1, STRATA_QSA_SLOTS=1, STRATA_HANDOFF_PIPE=P, the server's STRATA_ADMIT_SHORT_FIRST=<tokens>, and STRATA_TRACE for admission phases.
• CUDA only, a build option: -DSTRATA_MMQ_TAB=ON with STRATA_MMQ_DIRECT=1 and STRATA_MMQ_GROUP_SORT=1 (pointer-table MMQ, resident experts read in place). It is off in the release builds and costs 36-38 MiB per GPU.
• Measured on 4x R9700 (HIP, IQ3_S with every expert in VRAM, layer split, three interleaved rounds): 16 users 207 to 324 tok/s (1.56x, 3 of 3 rounds), time to first token p50 20.6 s to 2.0 s; 8 users 1.20x; 16 users with 1.2K-token prompts 1.41x; one user unchanged. The slot count alone gives about 40% of the 16-user gain, the copy and checkpoint switches the rest. Every slot of a 16 and a 32 slot batch decodes the tokens it decodes alone. On small cards (2 cards, 16 GB) the gain is smaller and the memory cost real; see docs/BATCHING.md.
• STRATA_SLOT_CKPT_MOVE=1 is experimental: do not use it. With it, tools/batch_groups_cache_test.py fails (the slot reuse pattern changes); all other switches pass it.

Faster, from your pull requests

We went through every open pull request with a speed claim, built each on the cards we have and ran the answer check against 0.1.44.1 before changing a default. What is on by default now gives the same answers as before (10 prompts and 4K / 20K exact prompt checks, identical on RTX 3060, Tesla P100, R9700 and Arc A750); each has an environment switch to turn it off. Everything that changes a number or is not faster everywhere is opt-in.

• Long prompts on AMD (#1749 vlupilin, #1943 2jztricks). The top-k kernel for very long contexts and an exact scorer for long prefill: on an R9700 a 131K-token prompt reads 8.7% faster, 230K 16%, 500K 79%; answers identical.
• Intel Arc (#2340 leadgtr7, #2207 aslater3). Q6_K decode no longer limited by ALU work and wide kernels in large-register mode: on an Arc Pro B70 decode is 3.5% (short prompt) to 4.2% (4K prompt) faster, same answers; the A750 kernel timings improve 1.7x at 4 columns.
• A resident-RAM copy keeps the prompt path's lend region (#1896 elightcap). With room in the RAM budget, decode went 7% and prompt reading 2.5% faster on an RTX 3060 (5 of 5 pairs).
• NVIDIA prefill and CPU (#1368 sergqwer, #1107 brenoperucchi, #2016 libratechw, #1949 maxious). Small, each 0.6-1.4% on an RTX 3060 / RTX A4000, and the AVX2 dot kernels keep their accumulators in registers (2-9% on the router micro-benchmark).
• Server (#1542, #1557, #1852, #1891, #1163, #1164). Tool-call and logit-bias parsing, prompt tokenisation from the last shared prefix and conversation-cache copies cost fewer milliseconds per request; no engine change.
• Opt-in, measured: STRATA_HIP_WMMA=1 on an R9700 (+44% prompt, +3.7% decode), the five decode switches of the CUDA series (+7% at 30K context on RTX 3060 and A4000), STRATA_EXCHANGE_ROTATE=1 (+5% decode on a 3060 with the resident RAM copy; do not use it with the resident copy under --expert-cache 1500 yet: the engine did not start on one 3060 box), STRATA_PCIE_BALANCE=1 (+5.6% on two A4000, slightly slower on single 3060 / P100), plus the Turing (sm_75) kernel series behind STRATA_SM75_PERF=1. We have no sm_75 card: those paths build, run bit-for-bit on a 3060 in emulation, and carry the author's speed numbers only. All switches are in docs/DETAILS.md and the backend pages.
• Not for you yet: STRATA_SIDE_MIXER=2 and STRATA_SH_STREAM=1 are slower on gfx1201 (0.41x and 0.44x decode); STRATA_QSA_WARP=1 crashes a Tesla P100. Leave them off.
• Moved to 0.1.46: the Volta/Pascal series. Together it made default decode on an RTX 3060 about 4% slower after a long prompt; we are finding the cause before it ships. The contributors' commits are kept.
• Thanks to the many people who sent speed work this cycle, in particular Rafa (the Turing series), vlupilin, 2jztricks, leadgtr7 and elightcap.

Setup

• Images on a card under 24 GB (#2406, thanks olituks). "Do you want images? yes" now gives the CPU encoder on an NVIDIA card under 24 GB: a picture takes 4-6 s and is read at up to 300 tokens (--vision-tokens 1024 for more detail), and nothing is taken from the GPU. The GPU encoder keeps ~1.4 GB of VRAM that the experts would use: 3-5% of text speed on 16 GB cards (measured, Linux), and a report of 35-60% on an RTX 4080 SUPER under Windows that we could not reproduce. --vision gpu is kept, with a note. Existing configs are untouched.
• UD-Q5_K_XL no longer fails its size check (#2346). The metadata-only first shard ends 6 alignment bytes before its aligned data start; setup counted them as missing.
• Qwen3.6 on Windows AMD (#2372, thanks for the exact diagnosis). Setup ran strata.exe --families without the engine's rocm\bin on PATH, so Windows showed two "dll not found" dialogs and setup said the engine did not know Qwen3.6. Every engine program setup starts now gets the same PATH.
• NVFP4 on a CPU without AVX-512 (#2347). It already runs there (the AVX2 rows are bit-equal to ggml's and as fast as the AVX-512 rows when memory-bound); the message in the report is the canonical Q2_0 pack's check, reached when --pack points at a folder without native_experts.txt. The engine now says which folder is missing the file and where one sits, and setup notes what the CPU rows are on an AVX2 CPU.

Fixes

• Models whose attention.compress_ratios is all zeros (#2396, thanks roverdrubber, who sent the patch). 0.1.44 refused OrcaRouter's Qwen3.8-Flash-Next-Uncensored GGUFs with "compress_ratios[3] = 0, but full_attention_interval = 4". A list of only zeros is a converter's placeholder: it is treated as absent, with one line on stderr. A list with any non-zero entry is still checked in full.
• --kv q4_0 with KV streaming and MTP (#2403). A Windows RX 6900 XT faulted in the first verify window. We could not reproduce it (R9700, RTX A4000, Tesla P100). The q4_0 verify window took a token-by-token path the other formats do not (about 650 more graph nodes); it now takes the batched window by default (STRATA_DF_QB_Q4=0 for the old path), with identical tokens on three GPUs. We do not know that this fixes the fault; the reporter's retest will tell.
• reasoning_effort: false against a configured default (PR #2352, thanks Tsury) keeps its meaning (thinking off).
• Prometheus gauges (#2341, PR #2359, thanks nekomario28). llamacpp:prompt_tokens_seconds and predicted_tokens_seconds are averaged over the window between scrapes, as in llama.cpp, so an idle server reads 0.
• strata_bench polls the address the config binds (#2330, PR #2362, thanks nekomario28).

Mac

All Mac checks of this release ran on an M4 Mac mini; there is no M5 in the test set any more, so the 128 GB numbers in the Mac docs are from the 0.1.44 M5 Max runs.

• Small-memory Macs (PR #2345, thanks kingdonb). When no chunk of the expert arena is big enough for the prompt's buffers (an arena that shrank to 0.8-1.0 GB chunks on a 24 GB Mac), the buffers get their own allocation instead of "no arena region for the prompt buffers". Forced on the M4 (5 GB arena in 1.2 GB chunks) the old build stops at that message and the new one runs the 4K prompt with the same output.
• M1 Max (PRs #2348, #2349, #2350, thanks demetree). The prompt kernels qsa_attn_mm and qsa_flash no longer hold a device load in a select's arm (which lost the threadgroup on an M1 Max); the numbers are unchanged for every row that was used, and in isolation the two kernels run about 2x faster on the M4. moe_mm_parity runs its default tile again. The docs record two opt-ins an M1 Max cannot run. The M1 Max faults are the contributors' measurements; we have no M1.
• The Monitor tab shows the Mac's GPU load and memory (PR #2360, thanks demetree), from the IOKit registry and the Metal working set; temperature and power are not readable without root and say so.
• setup --source works on a Mac (#2363, PR #2365, thanks nekomario28).

Strata is free and open source. If it runs well on your PC, a coffee keeps the work on it going: