Niko1221/Strata v0.1.43.1

Strata v0.1.43.1

on GitHub

A hotfix for 0.1.43: Linux NVIDIA decode is fast again on desktops (the adaptive overlap is opt-in again), long prompts no longer stall on Windows and SATA SSDs, two-card batch serving is about 35% faster again, RX 6000 prompts are fast again, and fixes for AMD multi-slot serving, Intel cards, hangs and the server. Answers are the same as 0.1.43 (byte-identical in our identity tests on every machine we tested); only a user message that quotes </tool_call> is now read as text (#2026). Update with UPDATE.bat (Linux: ./update.sh); setup replaces the engine with 0.1.43.1.

Fixes

• The adaptive overlap is opt-in again, on every machine (#2010; the change is Krypto-Whitehat's PR #2011). 0.1.43 turned STRATA_ADAPT_OVERLAP on by default on Linux NVIDIA. It helped on the boxes we measured (RTX A4000 +5-10%, Tesla P100 +4-6%) and cost decode on others: an RTX 5070 Ti desktop -21% (68.4 vs 86.4 tok/s with it off, the same build), an RTX 5060 Ti -19%, and an RTX 5090 Laptop (WSL2) -21% (84.6 vs 106.6). One RTX 5090 desktop gained 1-2%. We cannot test every machine, so the default is now off everywhere: STRATA_ADAPT_OVERLAP=1 turns it on, and the start-up log says whether it is on or off (and why it cannot run). Answers are the same either way. The two fixes below remove the collapse that the overlap's callbacks could cause on desktops, so =1 is safe to try again; measure it with your own A/B before you keep it. Plain 0.1.43 users who lost speed: STRATA_ADAPT_OVERLAP=0 was the workaround, and no longer has to be set.
• Linux NVIDIA: decode collapsed on some desktops with 0.1.43's adaptive overlap (#1964, #1975; found by oal in #1983).
  • Cause: the engine keeps its decode thread on one CPU core. CUDA starts its host-callback thread only when the overlap first uses it, and that thread then inherited the same single core, so each of the overlap's per-layer callbacks waited about 1 ms for the core instead of microseconds.
    • The reports: an RTX 4090 with a Ryzen 9800X3D decoded at 19-37 tok/s instead of 102-116, and an RTX 5090 lost 26%.
  • Fix, part 1: the callback thread now starts before the decode thread is pinned (oal's warm-up, now on by default; STRATA_ADAPT_CALLBACK_WARMUP=0 turns it off).
  • Fix, part 2: the overlap no longer needs that thread. Each layer's "copies landed" signal is a 4-byte copy into pinned memory, in stream order (STRATA_ADAPT_SIGNAL=host brings the callback back).
  • Measured on an RTX 3060 (Linux, Core Ultra 7 265) with the reporters' adaptive settings, 5 interleaved pairs: 36.5 -> 56.5 tok/s (5/5). With the overlap now as fast as with it off, the overlap keeps its gain where it helps:
    • RTX A4000: +5-10% over overlap off, the same speed as 0.1.43;
    • Tesla P100: +6%.
  • Windows was not affected (the overlap was opt-in there). Since the overlap is now off by default everywhere, this fix matters to those who set STRATA_ADAPT_OVERLAP=1.
• Linux SATA SSD: long prompts were about 7x slower than in 0.1.41 (#2040). The start-up check for the n-gram table's direct reads switched to mmap on a SATA SSD (see the next item: it no longer switches by itself).
• Windows and Linux: a long prompt could stop at "layer 1 of the prompt chunk" and restart the engine (#1984, and very likely #1971).
  • Cause: when the start-up check finds --ple-io direct slow, 0.1.42 switched to --ple-io mmap. A prompt's first chunk then read its n-gram rows as one page fault after another from a cold 27 GB table, with no progress mark, and the 60 s watchdog stopped the engine.
  • Fix, part 1: on every platform the check now only warns and keeps direct reads (--ple-io mmap or --ple-io ram choose the others; STRATA_PLE_AUTO_MMAP=1 brings back the 0.1.42 switch). Its 15,000 rows/s bar was set on NVMe drives; a SATA SSD tops out at 10-14K rows/s with direct reads and the switch to mmap made its prompts about 7x slower (#2040: 77.8K-token prompt, 884 tok/s direct against 76-154 mapped).
  • Fix, part 2: the mapped mode itself asks the OS for the next 4,096 rows ahead (a cold 32K prompt on Linux, mapped: 269 -> 301 tok/s, now equal to direct).
  • Fix, part 3: the watchdog's allowance for slow disk reads now measures on Windows too, and holds while the disk was busy in the last 10 seconds instead of the last one, so a slow but working prompt read is no longer stopped. A real hang is still stopped after 60 s.
• AMD (RX 7900 XTX, multi-slot serving with pictures): a memory access fault after about 30 minutes of load (#1996, mantovaniluca91, whose analysis this follows). 0.1.43 moved the prompt chunk's step records and token ids to pinned memory (#1712). Every other host-to-GPU copy of the prompt path whose source was ordinary (pageable) memory now goes through a small ring of pinned buffers too. That covers the picture embedding rows, the draft step's bounds and tokens, the fallback n-gram rows, and the routing tables of the non-mapped path. Same answers, the same prompt speed (within 0.2%), and a 65-minute 6-client stress with pictures, long prompts and batch slots ran with no error on an R9700. We have no RX 7000 card, so the reporter's own 4-hour run will tell.
• Batch slots: a request could get HTTP 400 "BGEN: no such free slot", and its conversation stayed broken (#1994). The server now takes or waits for another slot; after repeated refusals the request gets a retryable 503 instead of a permanent error, and no slot stays held.
• Two servers sharing a GPU: a failing before_load hook was ignored and both models loaded onto one card (#1997).
  • When the hook fails or times out, the server now checks the card's free VRAM (min_free_vram_mib): with room it loads and logs a warning, without room it answers 503 with Retry-After instead of crashing.
  • "before_load_strict": true always refuses on a failed hook, and "before_load_ignore_failure": true keeps the old behaviour.
  • "before_load_timeout_s" sets the hook's timeout.
  • POST /unload?wait=1 waits for the running turn before unloading.
• Intel: a two-card layer split lost the device at the first answer when experts were missing from VRAM (#1980). With STRATA_VERIFY_NO_HOST=1, only the first card's missing experts had a host copy. The engine now stops at start with a clear message instead, naming the fixes. Setup writes STRATA_STAGE_TRIM=1 for two Intel cards, which holds every expert in VRAM in the reporter's setup. We have no two-card Intel machine; the reporter's run will confirm.
• ./update.sh / UPDATE.bat: a build that failed could leave the old engine running for good (#2054, MrKop-git). After a compiler upgrade made the engine build fail, BUILD.json could carry the new source's stamp beside the old binary, and every later update said "engine already built for this PC". Setup now compares the version inside the engine file with the source before it trusts the stamp, compiles again when they differ, and stops without stamping or installing when a build did not produce the new engine. (The rolling-distro cause, a compiler newer than the CUDA toolkit accepts: use CC=gcc-14 CXX=g++-14 CUDAHOSTCXX=g++-14.)
• Two NVIDIA cards with --batch and --batch-mtp: several requests at once ran about 30% slower than in 0.1.41 (#2038). Since 0.1.42 --batch-mtp was accepted on a layer split, and the automatic batch groups then dropped to one serial group, so the two cards took turns. On a layer split the default pipelined groups now win (the start-up log says so); --batch-groups 1 or STRATA_BATCH_MTP_SERIAL=1 keeps the serial draft windows. 2x RTX A4000, 5 interleaved pairs: 4 requests 62.3 -> 85.2 tok/s together, 2 requests 55.9 -> 69.6, one request unchanged.
• RX 6000 cards (gfx103x) on Windows: prompts were 16-20% slower since 0.1.40.2 (#2055; the fix is did-technomancer's PR #2056, who also found the cause). The draft layer's side stream is created again on gfx103x (STRATA_MTP_SIDE_STREAM=0/1 overrides). RX 6800, a 7.9K prompt: 584-596 -> 798-808 tok/s in their runs. Other AMD cards are unchanged.
• Intel: with --resident-experts and the adaptive tier, a swapped-out expert's RAM copy was lost (#2014). The SYCL port had dropped the copy that saves the evicted expert, so a later CPU read of it used wrong weights. It is copied back again, as on CUDA; a byte check of every swap against the model file (STRATA_CHECK_SWAPS=1) found 0 differences over about 15,000 swaps on an Arc Pro B70 and an A750. Setup's default for the B70 (STRATA_VERIFY_NO_HOST=1) did not reach this path.
• Windows: a request could be stopped by the watchdog while the previous conversation was being parked (#2049), or hang in the draft step with almost no VRAM left (#2065). Parking a conversation now reports progress as it copies, so a slow copy of paged-out memory is no longer mistaken for a hang; every request checks the free VRAM and warns, with the --vram-reserve-mib that would leave room, when it is very low (Windows then falls back to slow shared memory); and the stall report names the draft wait. These were not 0.1.43 changes: both happened on 0.1.41 too.
• A short prompt right after start could end the engine with "lending the prompt path its slots failed" (#2090). A chunk smaller than the start-up chunk could need more of the borrowed cache than was set aside. The loan now covers what the request's chunk needs, and a chunk that still does not fit is read in smaller pieces instead of stopping the engine.
• The server: an answer could come back empty when the user's own text quoted </tool_call> (#2026); a tool call written as fenced JSON inside its tags is recovered (#2037); a client that leaves while its pictures are prepared is cancelled at once (#1999).
• Setup and tools: --calibrate --model X tunes that installed model instead of running a fresh setup (#2036); a first CMake configure registers all native-expert tests (#2027); BENCH.bat / ./bench.sh names only the GPUs the engine uses, one by one when they differ (#1985, #2062, #2001), and a server the benchmark starts no longer saves over your expert profile (#2088).
• Strix Halo on Windows (#2013): a diagnostic. STRATA_TRACE=2 waits for the GPU at each start-up step and names the first one after which it reports an error.

Strata is free and open source. If it runs well on your PC, a coffee keeps the work on it going: