Infernix windows-2026.11.18
AI disclaimer: Infernix was developed with AI (mostly Claude Opus 5.5). It is hobby work and
is likely to be neither complete nor entirely accurate.
Highlights
- Documents and PDFs, with Anthropic citations. Anthropic
documentblocks (plain text, custom
content, base64 or URL PDFs) andsearch_resultblocks, in a user message or a tool result, used
to be a 400 that ended the conversation, since the block stays in its history. They now enter the
prompt, and the OpenAI routes take PDFs as Chatfileand Responsesinput_fileparts. A PDF is
read with PDFium (now in the release zip): each page's text and, with--vision, an image of the
page. Withcitations: {"enabled": true}the reply's claims come back as Anthropic citation blocks
(character, page, content-block and search-result locations), streamed too. A request's PDFs may
hold up to 1,000 pages; what limits them is the context. On the 27B NVFP4, a 120-page excerpt
(89,185 tokens) answered a cited question from page 28 correctly in 12.8 s (22.4 s with vision), and
a second question in 2.4 s (2.1 s) from the prefix cache. Count Tokens counts exactly what Messages
runs. Not tried yet: real-world PDFs with unusual fonts or embedded images, URL and encrypted PDFs
served, OpenAI file inputs served. - NVIDIA's Qwen3.8-Flash-Next-NVFP4-QAD, now the recommended Flash-Next model. NVIDIA's
quantization-aware distillation of its NVFP4 release also quantizes the shared experts to NVFP4 and
the GDN, attention and indexer projections to MXFP8. Infernix runs each layer's shared expert as the
last job of the routed experts' call, in the routed experts' arithmetic. Two conversions are published:
the bit-exact one and Dense8 (hyper-connection mixers, PLE projections, final mixer andlm_headin
8 bits, as in Dense8). Perplexity on the quick corpus (BF16 KV,--reproducible, one build): QAD
Dense8 3.4385 and QAD bit-exact 3.4422, against 3.5095 for NVIDIA NVFP4 Dense8 and 3.5066 for
FP4-Local at the same 4-bit expert arithmetic. The better quality costs some speed with DFlash2:
against NVIDIA NVFP4 Dense8 (INT8 KV, one request, three interleaved runs), QAD Dense8 decodes 3.1 %
slower with DFlash2 (271.0-272.8 against 280.1-281.3 tok/s; acceptance length 7.53 against 7.63),
3.5 % faster with MTP on average (244.3-258.8 against 242.8-250.8), and prefills level at 8,192
tokens. The bit-exact QAD conversion decodes 2.9 % behind FP4-Local's 4-bit route with DFlash2
(279.8-280.7 against 288.0-289.2 tok/s) and level with MTP. Neither repository carries an n-gram
volume: use NVIDIA's FP8 volume. - Long conversations under KV pressure resume from their cache. With two agent conversations
that do not fit the KV cache together, continuing turns on the 27B took up to 142 s to the first
token with most of their history cached in RAM. A Qwen3.5 pause now publishes its state, a resume
is matched over the request's whole cached history, the cache is no longer emptied to make room for
a binding, and Host restores are issued as batched copies. On the official 27B NVFP4, two concurrent
conversations (186K and 96K tokens, six turns each): continuing-turn time to first token mean 20.0
to 2.9 s, max 72.3 to 18.3 s, replayed tokens 347K to 21.5K, wall time 219 to 59 s; in an agentic
replay at the same pressure, mean time to first token 2.81 to 1.18 s (one run each; a counter of
cache entries lost without a backing copy rose from 0 to 13 there). Under pressure, requests now
take 30-second turns instead of one-second ones, so two long requests no longer pause each other in
turn (31-35 pauses to 1 in the test): at the same wall time the first of the two finished sooner and
their mean completion came 13 % sooner, but a paused request now waits up to 30 s of the other's
service. - Memory sized as measured, and shown.
--max-context autosizes one request to the whole KV
cache on the Qwen3.5-family models whose experts stay in VRAM (256,896 tokens for the NVIDIA NVFP4
27B on a headless RTX 5090 with INT8 KV, two lanes and DFlash2; the launcher now starts the 27Bs this
way). The Host prefix cache is sized automatically by default (--host-context-mib auto), follows
the RAM other programs need while requests run, and grows back past its startup size when RAM frees
(27B: 61.5 to 70.5 GiB within 15 s of 10 GiB freeing). The RAM plans now count the kernel images the
CUDA driver reads, which are loaded at CUDA initialization, and kernels are built without line
information: the driver reads 1,467 MiB of RAM instead of 4,596 MiB, and CUDA initialization takes
0.85 s instead of 2.8 s. A smaller CUDA Graph allowance (27B NVFP4 at 131K context: 228,352 to
250,496 tokens of KV) and one Device snapshot slot by default with a Host tier (14,272 more tokens on
the 27B at four lanes) give the KV cache more VRAM. The launcher's new Memory panel shows what VRAM,
RAM and the SSD hold, part by part. - Recovering instead of stopping. Running out of device memory (another program taking the VRAM)
aborted the server; now the running requests fail with HTTP 503 ("out of device memory (...); the
request can be retried"), queued ones carry on, and only eight recoveries in a row with no successful
unit stop the Engine. An Engine that holds requests and makes no progress logs a warning after 30 s
and heals itself after two minutes: the running and paused requests fail and the queued ones are
admitted again, where before every queued request waited for a restart. And a crash under long
multi-turn Flash-Next sessions is fixed (kernels could read an expert's frame before its copy landed,
or write over a resident expert, giving an illegal address or NaN logits): with the reporter's flags,
windows-2026.11.07 died after 9 minutes; with the fix the same traffic ran 557 requests in 35 minutes
without a failure. - Sooner first tokens for agents.
- Prompts that arrive together are prefilled shortest-remaining first (aged), instead of in
rotation: four equal 5,000-word prompts on the official 27B NVFP4 reached their first tokens 35 %
sooner on average (2.219 to 1.440 s), the last one level; a mix of 10,000, 600 and 5,000 words 18 %
sooner on average (1.177 to 0.961 s), the longest level. - A new tool set's grammar is compiled beside the prefill instead of before admission (the first
turn 0.794-0.889 s sooner on FP4-Local, 0.303-0.787 s on the 27B NVFP4 with DFlash2), on up to 8
threads (the 20-tool grammar's compile 0.795 / 0.774 s on one thread, 0.139 / 0.138 s on eight, on
the CPU alone; the served first token was not measured). - Chat templates render 6.6-8.4× faster (Qwen3.8's template at 1,600 messages: 610.7 to 85.9 ms a
request; the served time to first token was not measured). - The prefix-cache tap a turn takes 2-8 tokens before its prompt's end no longer costs a second pass
on the default route: on the 27B NVFP4 such turns reach their first token in 21.8 / 23.2 ms instead
of 37.3 / 38.0 ms (drafting off); Flash-Next turns gain 8.6-12.3 ms (drafting off). - Messages and Responses streams still queued after 5 s start their stream and send keep-alives, so
clients that give up on a silent connection after 120 s keep it.
- Prompts that arrive together are prefilled shortest-remaining first (aged), instead of in
- Faster sampled speculation on
--reproducible. The drafters now draw a sampled request's drafts
with the same Gumbel keys the route verifies them with, so more of them are accepted; the output
stays plain sampling's text. Sampled at 0.7: the official 27B NVFP4 with DFlash2 163.0 to 186.3 tok/s
(+14.3 %), FP4-Local with DFlash2 200.7 to 206.3 (+2.8 %), the NVFP4 27B with--spec mtp+8.9 % (every
MTP step coupled on dense models), the 35B-A3B +1.08 % (its first step). On the default route the
same draw is a fast path,gumbel-sampling, registered only for the NVFP4 27B's MTP rounds with INT8
KV (+8.5 % sampled decode there; a seeded sampled request's text changes, its distribution does not,
by a distribution test of low power). With DFlash2 on the 27B it measured 3.6 % slower and on the
35B-A3B's MTP 0.6 % slower, so it is not registered there.
Key improvements
- Qwen3.8-Flash-Next speed (the same output, unless marked):
- narrow decode and verification calls: FP4-Local's W4A16 experts of up to three columns run on
register-resident row pairs (plain decode 122.61 to 124.91 tok/s; with DFlash2 structured output
+0.2 %), the W4A4 banks of Dense8 and recipe A the same way (Dense8 plain decode ×1.008-1.032; with
DFlash2 level, since its verification calls are wider), and every narrow down pass waits for its own
job instead of the whole gate/up grid (FP4-Local with DFlash2: conversation 213.6 to 215.4 tok/s,
structured output 474.8 to 477.7; IQ3_S structured output 420.5 to 428.6); - DFlash2 heads without a fused head (FP4-Local's BF16 head without
--lm-head-draft, Huihui,
recipes A and C, IQ3_S, FP8) pick their candidates in 40.7 instead of 193.3 µs a round (FP4-Local
conversation +0.9 to +1.8 %, structured output +1.0 to +1.1 %), and--reproduciblestarts the
shared expert early, as the default route does (FP4-Local conversation +0.7 to +1.4 %, structured
output +0.7 to +0.9 %, same bits); - decode after a long prompt: each wide prefill call borrows the expert-cache memory its own plan
needs instead of a formula's 2.0-3.9× too much, so fewer experts are evicted. FP4-Local (plain
decode, INT8 KV): the loan at 16,384 tokens 7,792 to 3,944 MiB; a 32,768-token prompt +0.5 % and the
256 tokens after it 104.5 to 112.3 tok/s (+7.4 %); a 4,096-token prompt +2.1 % and the decode after
it +1.8 %. IQ3_S's prefill chunk now fits at 16,384 tokens on a 32 GB card (the formula cut it to
13,440 in that run): a 32,768-token prompt +0.8 % and the decode after it +3.6 %; 4,096 tokens +1.8 %
and +0.9 %; - prefill: the last layer's MoE runs only over the columns the prompt reads (FP4-Local +1.66 to
+1.97 % at 512 tokens, +1.77 to +1.83 % at 2,048, +0.49 to +0.63 % at 16,384, +0.68 to +0.72 % at
32,768), the copy-ahead guesses the next layer's experts better and copies only those of more than
T / 64 columns (FP4-Local +4.38 to +4.61 % at 1,024 tokens, +3.30 to +3.52 % at 2,048, +0.10 to
+0.22 % at 4,096), and the IQ3_S import's tiles read a BF16 plane (prefill time from a cold expert
cache −3.7 % at 16,384 tokens, level at 4,096) and its GDN snapshot is read withldmatrix(+0.14 to
+0.40 %); - conversation turns: a short turn's last two prompt calls run as one (server time to first token
−8.6 to −12.3 ms with drafting off, −2.8 to −11.5 ms with DFlash2; default route only, where the
first token can change), and a client that does not echo its replies gets a snapshot at each turn's
opener (request time over turns 1-5: FP4-Local and Dense8 −2.6 %, IQ3_S −1.7 %, recipe C level; a
turn that captures pays +13 to +20 ms of time to first token); - a fresh prompt beside a decoding request is walked in 4,096-token steps, which halved the decoding
requests' longest pause (1.63 to 0.74 s) at the same total time (FP4-Local, one workload); - startup: the SSD expert tier page-locks its slots while it reads the experts (FP4-Local with
--host-context-mib 16384ready in 20.5 / 20.7 instead of 27.5 / 27.6 s; starts whose experts all
fit as pinned banks do not take this path), the CPU team's rate readings stop early when they agree
with a stored reference (banks 19.0 / 18.8 / 19.1 against 23.0 / 23.1 / 22.8 s; SSD tier 16.6 / 16.6
/ 16.4 against 24.7 / 25.1 / 27.2 s), and the least-used seeds' RAM copies are kept instead of read
again (that step 1.11-1.15 instead of 1.83-1.88 s); - CPU-served W4A4 misses (Dense8, recipe A, FP4-Local's 4-bit route) walk each row group once when
gate and up share their input scales: the CPU team serves 50-60 % more expert columns a second, and
Dense8 decodes 0.5 % faster plainly and 2.7 % faster with--spec auto(DFlash2), but prefills
0.37 % slower at 2,048 tokens (the cause is not established yet); - the CPU miss service no longer spins a core between requests (PR #69 by dwalthour measured 37-41 %
of a core on a 24-core Linux desktop), and a decoding round no longer pays a system call to wake
it; --expert-csf(new, opt-in, same output): the offloaded NVFP4 experts' RAM copies keep their block
scales compressed, about 3 GiB less RAM on Dense8, FP4-Local and the 122B. Prompts of 1,024 / 2,048 /
4,096 tokens get faster (Dense8 +2.0-2.4 / +3.1-3.4 / +3.9-4.0 %, FP4-Local +0.8-2.2 / +3.0-3.2 /
+4.0-4.3 %, the 122B +4.6-5.1 / +4.4-5.0 / +1.7-3.8 %) but decode slower (0.973-0.995 of its speed),
hence off by default.
- narrow decode and verification calls: FP4-Local's W4A16 experts of up to three columns run on
- Qwen3.5 family (27B, 35B-A3B, 122B-A10B):
- 27B NVFP4 prefill: partly filled tiles skip their empty work and the projections take better tiles
at 129-1,100 columns (prefill time −7.0 to −9.9 % at 129-255 tokens, −2.8 to −7.0 % at 264-1,100,
level at 40-104 and 2,048-8,192; then −9.7 to −10.6 % more at 200-255 tokens from the down
projection's tiles), the weight maps ask for L2 promotion (official NVFP4 −1.2 to −3.0 % at 129-255
tokens, 0.0 to −0.8 % at 264-511, −0.7 to −1.6 % at 520-8,192), and a remainder that would add a wave
runs on MMA tiles (−3.84 % at 1,025 tokens). The MLP's norm, quantizers and SwiGLU are fused for the
4-bit prompt route (NVIDIA NVFP4 +0.09 to +0.24 % at 1,024 tokens, +1.15 to +1.25 % at 4,096, +1.14
to +1.21 % at 16,384), the last layer computes its outputs only over the columns it needs (official
NVFP4 +1.32 to +1.36 % at 7,680 tokens, +1.11 to +1.13 % at 32,768), and the DFlash2 drafter's
context is projected only where it is read (official NVFP4 +1.00 to +1.10 % at 16,384 tokens, +0.78
to +0.83 % at 65,536); - 27B decode: the DFlash2 drafter's proposal head is screened by a 2-bit copy (357.6 MB of VRAM) in
rounds of up to 32 columns:--reproducible+2.87 to +2.92 % (official NVFP4), +2.76 to +2.80 %
(NVIDIA NVFP4), +3.71 to +3.72 % (groupwise) with the same text; on the default route the drafts,
and so the text, can change (NVIDIA +7.2 %, groupwise +3.9 %, official −0.4 % on texts that
differed). The A4-residual verification MLP is fused too (NVIDIA NVFP4 decode +0.26 to +0.31 %); - the groupwise formats (Q4, Q5, Q6, Q8) prefill on a staging ring: the groupwise 27B 9.2-9.5 % faster
from 1,024 to 16,384 tokens, and shorter prompts on re-derived width tables (prefill time −4.2 to
−10.4 % at 100-1,032 tokens, level at 3,592 and 4,096); the 35B-A3B's routed experts and Q8
projections on the ring (prefill time −0.8 to −1.2 % at 4,096 tokens, −4.3 to −4.6 % at 16,384), its
Q8 shared expert at more CTAs an SM (−1.8 % at 1,024 and 4,096, −0.2 % at 16,384); - 122B: 6-8-column CPU expert calls run as two column groups (a 255-token prompt 328 against 320
tok/s, other widths within 0.4 %), the copy-ahead takes experts from T / 64 rows as on Flash-Next
(prefill time −2.59 % at 1,024 tokens, −0.85 % at 2,048, −0.26 % at 4,096, level at 512), and the
last layer's offloaded MoE is trimmed (−1.4 to −1.7 % at 512-4,096 tokens; the first token can
change on the default route); - the prefix-cache tap 2-8 tokens before a prompt's end runs inside the prompt's last pass (server
time to first token on such turns, two sets of passes: 27B NVFP4 21.8 / 23.2 against 37.3 / 38.0 ms
without drafting and 30.4 / 31.3 against 45.0 / 46.3 ms with--spec auto, 35B-A3B 23.5 / 24.0
against 28.7 / 29.1 ms, 122B 377.8 / 301.5 against 402.9 / 329.2 ms). Default route only; the next
turn's text can change (13-21 of 24 replies equal the two-pass form's); - more accurate chunked GDN prefill, on every Qwen3.5-family and Flash-Next artifact: the worst state
error against FP64 falls from 3.56e-3 to 6.0e-5. It costs the Op 1.41-1.43× its time and FP4-Local
1.03-1.10 % of prefill at 32,768 tokens (level at 2,048 and 16,384). It is the reproducible route's
arithmetic everywhere, and the default route's wheregdn-f16-accumulatedoes not run (FP4-Local,
the 35B-A3B, Dense8, recipe C).
- 27B NVFP4 prefill: partly filled tiles skip their empty work and the projections take better tiles
- Serving:
- Anthropic: mid-conversation tool changes (
tool_addition/tool_removalblocks in a system
message, as Claude Code 2.1.325 sends claude.ai connector tools) are applied instead of refused;
max_tokens: 0pre-warms the prefix cache, as in Anthropic's API;--claude-code-usage-compat
(opt-in, PR #63 by Hundsbuah) works around
Claude Code counting a fully cached prompt twice (checked on Claude Code 2.1.309 and 2.1.322); cache
creation is summed across tool-loop rounds
(PR #64 by Hundsbuah), as are cached block
tokens; - Chat Completions:
--restore-reasoning(renamed from--restore-tool-reasoning) also gives a plain
answer sent back without reasoning the reasoning this server generated for it; a past reply is
encoded apart from its generation opener, so a reply with empty reasoning can serve the next turn
from its endpoint; a stream withtimings_per_tokenaccepts a timing that arrives before its first
chunk (PR #65 by escovercole); - Responses: a stored request whose input alone cannot fit the response store is refused before it
runs, instead of after its whole generation; - tool calling:
--tolerant-tool-callsrecovers values whose closers do not balance, values cut
inside their closer, arrays or objects with unescaped quotes between word characters, and Python's
None; a call whose only parameter is an undeclaredargumentsobject takes that object as its
arguments on every route; quoted calls are recognized inside Markdown block quotes and list items;
--tool-parameter-names declared(new, opt-in) lets a constrained call to a non-strict tool name
only its declared parameters, so a quoted tool call inside an edit stays text; - constrained decoding: xgrammar's fixes #1003 and #942 are backported: a token that runs from a
literal into free text is no longer masked out, and an evicted token mask is freed at once; --reasoning-loopalso catches counting loops whose lines change only a number (over two stored
corpora of 4,256 and 5,814 replies the new rule added no firing);/v1/modelsalso lists its entries under a top-levelmodels(Codex CLI reads it), in the server
and the launcher;GET /monitorlists each request the Engine holds;/metricsgains the
no-progress, device-memory recovery, grammar compile and wait, tool-call unwrap and SSD-promotion
counters and amemoryobject;- logging: a stalled console (a text selection with QuickEdit, a pipe nobody reads) no longer blocks
the threads that log, and--log-filekeeps every record; CUDA failures reach--log-filebefore
the abort (PR #61 by Sugaroverdose); - prefix cache: with
--prefix-cache-file, the SSD tier now survives a restart (each slab's hash is
saved and checked at its first read), a promotion waits for a Host write in flight instead of
falling back to a much shorter source (after a restart, the 27B with a 480 MiB Host tier resumed at
1,024 tokens instead of prefilling from scratch), a tap at the end of a prompt's last image or clip
lets a new conversation about the same media resume past it without a Vision pass (default route;
on FP4-Local 1,028 prompt tokens were reused), and the cache
file is flushed to the drive before it replaces the previous one (+0.8 / +1.0 s on a 16 GiB save,
+10-12 %).
- Anthropic: mid-conversation tool changes (
- Launcher:
- dashboard: simpler dials that turn amber or red past their limits (CPU 85 / 95 °C, GPU 80 / 90 °C,
under 4 / 2 GiB of free RAM), one Endpoints menu, the Memory panel, and warnings when the server
has made no progress for 30 s, with counts of healed stalls, preemptions and replayed tokens; - Configure: every option the server lists now has a control (29 had none), in sections that open
and close, with a badge of the options set and a list control for--fast-path/--no-fast-path;
the launch preview warns about an option given without the one it needs; - the 27Bs start with
--max-context auto(Context length offers Auto), the launcher no longer
writes a Host prefix cache size, the gauges and sizing follow the NVIDIA GPU with the most VRAM, and
the catalog lists both NVFP4-QAD conversions, QAD Dense8 as recommended.
- dashboard: simpler dials that turn amber or red past their limits (CPU 85 / 95 °C, GPU 80 / 90 °C,
- Fixes:
- VRAM other programs hold (#68): Windows' VRAM
budget and free figure leave out memory other processes hold, so startup sized for an empty card.
Flash-Next and the offloaded 122B no longer warn "N MiB of the expert frames were placed in system
memory": the expert cache reads the free VRAM before each 64 MiB step and stops cleanly (FP4-Local
beside a 3 GiB program: 7,815 of 9,108 frames, no warning, 152 MiB free after startup; a quiet card
keeps 9,108; startup time unchanged, frame growth 0.67 against 0.71 s). A resident Qwen3.5 (27B,
35B-A3B) with--kv-capacity autoor--max-context autono longer takes the other program's VRAM
(Windows evicted it to fit the KV cache): startup maps the KV space in 64 MiB steps, stops where
Windows lowers the budget and sizes the KV cache to that (the 27B NVFP4 beside a 3 GiB program:
262,016 tokens instead of 370,304, and the program kept its 3,504 MiB instead of being pushed down
to 416; decode unchanged, 86.0 against 85.9-86.0 tok/s; a quiet card is unchanged). The reporter's
own setup (a display attached) has not been retested yet; - Flash-Next and offloaded experts: a paused request that resumed without replay could run its first
round before its Host restore landed and read stale KV and state; cancelling a long prompt
mid-prefill and retrying could give NaN logits for every request and then an illegal address (with
the fix, 117 cancel-and-retry cycles over 40 minutes passed); the wide NVFP4 passes faulted on an
expert they could not serve instead of failing one request
(PR #62 by Sugaroverdose); a ring slot could be
recycled while the CPU still read it (silently wrong expert output); a request short of KV now binds
between a long prompt's walk spans instead of waiting out the whole prompt (it could expire in the
queue behind a cold prompt of more than 30 s); a binding no longer waits for unrelated device work;
forced tokens (a thinking budget's close) count toward presence and frequency penalties, so a paused
and resumed request equals its unpaused run; the idle KV shrink released nothing (FP4-Local's expert
cache now regrows after a conversation: 7,987 to 8,542 frames); an expert's down pass that waits 10 s
without progress fails its call as a recoverable error instead of hanging until Windows resets the
GPU; with packed BF16 KV, n-gram tree rounds at 7 or 8 lanes wrote past a flag array; and Flash-Next
no longer warns falsely about CUDA Graph memory
(#68); - Qwen3.5: a draft-tree round of two or more rows threw on an artifact whose GDN input projection is
NVFP4; a 122B prompt paused while it prefilled kept the expert frames lent to its stream while it
waited; a resume from a snapshot is counted as a snapshot restore in the statistics (every resume
counted as a replay restore before); on Windows and WSL2 the CUDA Graph memory figure (the startup
warning,memoryin/metrics) counts the server's own device memory, so another program allocating
during startup no longer reads as graph memory; - prefix cache: a promotion whose SSD read could not be submitted dropped the entry's SSD copies, and
with them an SSD-only snapshot and every prefix resuming through it; a flipped bit in the saved
file's tables could restore a block under another identity (the tables now carry a hash); the bytes
past an SSD stream's end are written as zeros; the SSD write budget no longer freezes after the clock
is set back; and a family of error paths in both prefix caches and the expert leases that could leave
a pin, slot, image or lease taken until restart now give it back (found by review; most are not
reached by any test); - CUDA kernels: router logits that were all NaN killed the CUDA context; with packed BF16 KV past 16,384
keys, the prompt-split fast path could mix stale workspace into the first query tiles; an n-gram tree
chain of 17 or more drafts with penalties overran its penalty array; a resident MoE call of more than
65,535 entries (a drafter call of 8,192 or more columns at top-8) failed to launch (the wrapped path is
not reached by a test); and about twenty kernels gained operand and overlap checks, alignment
declarations and ordering fixes (asynchronous copies, programmatic launches) with no symptom seen;
the lossless route's three fixes keep the bits of every valid input; - Engine: a device failure while a request is released completes it and recovers instead of stopping
the Engine; a request that finishes beside a Flash-Next image encode completes at once instead of
when the encode ends; the throughput reporter's first sample could end the server if the Engine
failed at that moment; request ids follow the queue order; - serving and protocols: a PNG with an illegal bit depth crashed the server (a division by zero); PNGs
decode a scanline at a time (an 8192 × 8192 16-bit image held 537 + 537 + 201 MB at once) and FFmpeg
decoders check the pixel budget before they allocate; a JSON schema whose shared references nest 40
deep never finished compiling, and a body of 100,000 inexact numbers held a handler thread for
minutes; history tool-call arguments that are not a JSON object answer 400 instead of 500 (as NInfer
#446 reported); a cancelled non-streamed Anthropic answer is a 499 instead of a 500, and a cancelled
Response is not stored; request-log records with text that is not UTF-8 are written instead of
thrown, and a rotation while another program holds the full file no longer deletes a kept file; the
console panel no longer holds up/metricswhile the console is stalled, and counts cancelled
requests as cancelled; the tool-call fallback line in the log carries no control bytes; a host such as
127.example.comis no longer taken for loopback (the open-server warning now shows); a huge
--prefix-cache-save-minsoverflowed, and output files could name the model artifact or one another; - launcher: settings are written durably, with
infernix-launcher.json.bakread when the file is torn
(a crash right after a save could lose every setting, configuration, the downloads queue and the API
keys); Unload, or loading another model, cancels a load in progress; "Delete model files" can no
longer reach a file outside the model's folder that an artifact header names; an answer the server
cut off is a 502 or ends with the protocol's error event instead of passing as complete; an update
zip without a digest is checked against the release's size; update checks stop when the launcher
does; a restart within the same second keeps the earlier server log; a chat save no longer fails
while the chat is read; readiness is taken only from the launcher's own server; on Linux the server
dies with the launcher; and the telemetry, autoload, settings saves, download errors, watchdog and
exit no longer stall or end the launcher on an odd reading or a slow disk; - Linux: builds with FFmpeg 6.1 (an animated WebP is then refused, with a message naming FFmpeg 8), and
WSL2 reports the server's own VRAM use; - Windows: the working-set privilege is enabled before the working set is raised
(#68), and an Engine holds a 1 ms timer
resolution while it runs (a 1 ms wait took 15.74 ms median before).
- VRAM other programs hold (#68): Windows' VRAM
- Build: CUDA 13.4 is the tested toolkit (later ones are untested; a fork measured 13.5's device
link at over 46 GB of RAM). Visual Studio 2022's MSVC 14.44 no longer rejects a lambda capture in nine
translation units (reported by the fork nmorgowicz). PDF support needs PDFium:
tools/deps/fetch_pdfium.py(Python 3.11 or later) downloads the pinned release, and a build without
it succeeds and refuses PDFs; release packaging requires it.
CUDA line information is off by default (-DINFERNIX_CUDA_LINEINFO=ONrestores it; the machine code
is byte-identical), and a full build prints no warnings. - And many more smaller fixes from code reviews and a code audit, new tests (a SASS check of
what kernels load before their programmatic wait, a check that every server option has a launcher
control, real-model tests of out-of-memory recovery, the no-progress heal and a served shutdown),
benchmark tools (a decode-path probe, output hashes in the Op benches) and documentation updates,
among them FP4-Local in the README's Quality
section (its default decode path at 0.90× honest rounding).
Supported models
Each repository's model card gives its files, requirements, recommended command, quality and
speed; the README's Models section lists them
all.
- Qwen3.8-Flash-Next (experts offloaded to RAM or the SSD):
NVIDIA NVFP4-QAD Dense8
(new, recommended), NVIDIA NVFP4-QAD bit-exact (new),
NVIDIA NVFP4 Dense8
(the one benchmarked), NVIDIA NVFP4 bit-exact,
GSQ-RCO IQ3_S,
FP8,
NVIDIA FP4-Local,
Huihui abliterated FP4-Local,
Uncensored NVFP4
and Swift-1.5,
each with the DFlash2 drafter
and any of the four PLE n-gram volumes (FP8 recommended; the two FP4-Local conversions recommend
FP4-Local's own). The NVFP4-QAD conversions need this release or later. - Qwen3.5-122B-A10B (experts offloaded): Ornith-1.5-122B-NVFP4
and Ornith-1.5-122B-UNCENSORED-NVFP4. - Qwen3.8-27B: NVIDIA NVFP4 (recommended),
Quasar NVFP4,
Uncensored NVFP4,
Swift NVFP4, and NInfer's
official NVFP4 and
groupwise artifacts. - Qwen3.6: NInfer's official 27B,
27B NVFP4 and
35B-A3B artifacts. - Your own checkpoints of these architectures convert with
python -m tools.convert(Hugging
Face, NVIDIA ModelOpt including block-FP8 and per-row-scaled FP8, Quasar and GGUF sources). New
recipesqwen3_8_flash_next_nvfp4_qadandqwen3_8_flash_next_nvfp4_qad_dense8import NVIDIA's
NVFP4-QAD checkpoint bit-exactly.
Benchmarks
The README's speed tables are from 2026-10-30/31 (one RTX 5090 at PCIe Gen5 x8, on builds of that cycle).
They were not repeated for this release, so they include neither the speedups above nor the costs noted
with them. The figures in these notes are each change's own measurement, on the build and artifact it
names.
Behaviour notes
-
Output bits on the default and
--reproducibleroutes change for:- the more accurate chunked GDN prefill (the reproducible route on every Qwen3.5-family and Flash-Next
artifact; the default route on FP4-Local, the 35B-A3B, Dense8 and recipe C); - the first token of a Qwen3.5 generation prompt (its last layer now computes its outputs only over
the last columns); - the 27B NVFP4's FP8 projections at a few prompt widths that had a split-K tail, and the groupwise
Q4 gate/up at the re-routed widths (prefill only); - IQ3_S on Flash-Next, whose prefill chunk now fits at 16,384 tokens, which moves its call grid;
- a Flash-Next request with presence or frequency penalties that was paused after forced tokens (now
equal to its unpaused run); - prompts whose history holds an echoed reply with empty reasoning (its token ids now match what was
generated); - constrained requests, whose masks no longer exclude a valid token (xgrammar #1003).
Default route only: the Flash-Next and Qwen3.5 single-pass opener taps, the tap at a prompt's last
media item, the 122B's trimmed last layer, the 27B's screened DFlash2 proposal head, a replayed
request (it now waits for its Host restore and runs the execution schedule), andgumbel-samplingon
the NVFP4 27B with--spec mtp. Both routes stay deterministic and request-invariant. - the more accurate chunked GDN prefill (the reproducible route on every Qwen3.5-family and Flash-Next
-
--lossless: no lossless kernel's arithmetic changed (the audit's three lossless fixes keep the
bits of every valid input). Its output can still change for constrained requests, whose masks the
xgrammar fix changes on every route,--losslessincluded, and for histories with an echoed
empty-reasoning reply, whose prompt tokens change. -
Changed defaults:
--host-context-mibisauto(8192 MiB before): the Host prefix cache takes all the RAM the
model's own plan leaves beyond--ram-free-mib, in 256 MiB steps and at least 1 GiB, so its size
depends on the RAM free at startup;--ram-free-mibis 4096 (8192 before);--device-snapshot-slotsis 1 when there is a Host tier (max-concurrency + 1before);--ram-resize autoalso acts while requests run: the tiers give memory back once less than max(R / 2,
2 GiB) is available (R =--ram-free-mib) and grow back after R + 1 GiB has stayed free for 10 s;- under KV pressure a service turn lasts 30 s (1 s before);
- prefill chunks go to the least remaining prompt, aged, instead of in rotation;
- grammars compile on up to a quarter of the hardware threads, at most 8;
- the 27B's DFlash2 drafter keeps a 2-bit screen of its proposal head, 357.6 MB of VRAM beside the
KV cache's share; - a tool call whose only parameter is an undeclared
argumentsobject takes that object as its
arguments on the strict, tolerant and constrained routes; - the launcher starts the 27Bs with
--max-context autoand no longer writes--host-context-mib; - CUDA line information is off in builds.
-
VRAM other programs hold: while another program holds VRAM, the offloaded experts' cache keeps the
--vram-headroom-mibheadroom instead of eating into it, so it can start a little smaller (97 frames
fewer in the FP4-Local test). A resident Qwen3.5's automatic KV capacity (--kv-capacity auto,
--max-context auto) can now differ between launches, with what other programs hold at startup; it
is fixed once the server has started. An explicit--max-contextstill keeps one whole request's
context, and below that Windows pages the other program out as before; an explicit--kv-capacity
and Flash-Next's KV are unchanged.--plancannot allocate, so it shows the budget's larger figure
while other programs hold VRAM. -
New, renamed and removed options:
--restore-tool-reasoningis now--restore-reasoning, and the old name is refused: a saved
launcher configuration or TOML file that uses it must be renamed before the server starts;- new:
--expert-csf,--tool-parameter-names open|declared(defaultopen),
--claude-code-usage-compat, and--max-context auto(refused with offloaded experts and on
Flash-Next; with it--kv-capacitydefaults toautoand an explicit value is its ceiling); --vram-past-budgetnow also sizes a resident model's KV cache;--prefix-cache-save-minsaccepts 1-525,600;--log-file,--request-log-jsonl,--route-trace
and--prefix-cache-filemay not name the model artifact or one another (the server refuses to
start);- a new fast path,
gumbel-sampling, runs on the NVFP4 27B's MTP rounds with INT8 KV;
--no-fast-path gumbel-samplingleaves it out.
-
Files and formats:
- the
--prefix-cache-fileformat is now 7 for both families (4 on Qwen3.5 and 2 on Flash-Next in
the previous release), with a hash for each slab and for the tables: a file saved by the previous
release is ignored once and replaced at shutdown; - with
--prefix-cache-fileand an SSD tier, the SSD file stays in--ssd-context-dirafter exit
(infernix-context-kept-<hash>.bin, named after the cache file's path); kept files of other paths
are listed in the startup log, not removed; <expert state>.cpu-ratesbeside--expert-statekeeps the CPU team's settled rates;- the launcher keeps
infernix-launcher.json.bakand names server logs
serve-<time>-<run>-<model>.log; - the serve request log stays at schema 31 and adds
reasoning_missingandreasoning_restoredto
request_start,arguments_unwrappedtotool_call_parse,cuda_graph_excess_absorbedto
server_start.memory, and the grammar compile and wait times; the throughput record adds
taps_retriedandpromotion_room_waits; infernix_chat_reasoning_missing_totalandinfernix_chat_reasoning_restored_totalgain a
messagelabel, and Prometheuslebounds print their shortest round-trip text.
- the
-
Protocol behaviour:
- out of device memory: HTTP 503 for the running requests; after two minutes without progress the
Engine fails its running and paused requests; - Anthropic:
documentandsearch_resultblocks,max_tokens: 0andtool_addition/
tool_removalare accepted (each was a 400); a cancelled non-streamed answer is 499
request_cancelled(500 before); a hidden-reasoning signature that does not decode to UTF-8 is a
400; - Responses: a cancelled response is not stored;
- history tool-call arguments that are not a JSON object (
nullincluded) are a 400; - Messages and Responses streams still queued after 5 s start their stream, and a later queue answer
arrives as the protocol's error event; a Chat Completions stream starts at admission; - PDFs: at most 1,000 pages across a request's PDFs; a scanned PDF without
--vision, an encrypted
PDF and unreadable bytes are 400s with their own codes.
- out of device memory: HTTP 503 for the running requests; after two minutes without progress the
-
A custom chat template with
breakorcontinueoutside a loop now fails at startup; inside an
ifthey now work as in Jinja2. -
On Linux with an FFmpeg older than 8, an animated WebP is refused.
-
Building from source: CUDA 13.4 is the tested toolkit; PDF support needs the PDFium download.
Downloads
infernix-2026.11.18-windows-x64.zip:infernix-serve.exeandinfernix-launcher.exewith their
FFmpeg, curl, zlib and libjpeg-turbo DLLs,pdfium.dll(new: PDF documents; PDFium's build without
V8 and XFA, release chromium/8152), the maintained chat templates, the licence of every component
they contain or load (licenses/, including PDFium's underlicenses/pdfiumand every npm package
bundled in the launcher's web UI) andTHIRD_PARTY_NOTICES.md. Unpack it to a folder and run
infernix-launcher.exe, orinfernix-serve.exedirectly. An earlier launcher can install it from
Settings → Updates, which keepsinfernix-launcher.json, logs and chats.ffmpeg-9.0.1-source.zip: not needed to run Infernix. It is provided for licence
compliance: the FFmpeg DLLs in the release zip are LGPL-2.1-or-later, and this is their
corresponding source (unchanged since the previous release).
The zip is 239 MB instead of 590 MB: infernix-serve.exe is 244 MB instead of 603 MB now that CUDA line
information is off in builds (the machine code is the same). Against the previous zip it adds
pdfium.dll and PDFium's licences and removes nothing.
Requirements: an RTX 50-series GPU (sm_120a), an NVIDIA driver of 580 or later (CUDA 13), and the
Visual C++ Redistributable for Visual Studio 2015-2026. No CUDA toolkit is needed. Linux users build
from source (see the README).
Verification and known gaps
Everything below ran on one RTX 5090 (PCIe Gen5 x8, driver 621.24) under Windows 11, built with
CUDA 13.4.
-
The full test suite, on the release commit, passed with no failures. The 323 non-real CTest entries
passed (33 min, 8 at a time); one of them, a tokenizer check, skips without a tokenizer directory passed by
hand. The 103 real-model entries passed (97 min, one at a time) on the 27B NVFP4 (NInfer's official
artifact), Flash-Next NVFP4-QAD Dense8 with the DFlash2 drafter and the n-gram volume, the 35B-A3B and
Ornith-1.5-122B. Eight of them skip by design: five tools that run only on inputs passed by hand, and the
three 4-bit-activation wide-plan scenarios, which need an artifact that offers them. The 48 Flash-Next entries
passed again on NVIDIA's NVFP4 Dense8 (60 min), whose shared expert takes the dense path QAD's does not. The six
entries FP4-Local changes most passed on NVIDIA FP4-Local, the three wide-plan scenarios among them. The Python
suites passed (488 tests, 1 skipped because it needs a transformers 5.5 environment, plus the reproducibility
digest's 3), and the launcher's end-to-end check passed all 116 of its checks. -
The release's first full run found these failures, and each is fixed:
- a CPU expert kernel overran its stack buffer on the test's vision shapes (more than 192 blocks of K; no
model's experts are that wide): the CPU kernels now refuse such a call, and the test's oracle slices K; - the stack-size check found two lossless kernels 16 and 32 bytes over their recorded frames: the
allowlist now records them (their machine code was reviewed; three other kernels' frames shrank); - a Flash-Next test waited without bound for output from a request that had ended at a stop token: the
test now turns stop tokens off and bounds the wait; - QAD Dense8's out-of-memory recovery test injected its failure into work the Engine absorbs (a binding
it gives back), so no request saw it: it now picks its failure points from the run itself; - QAD Dense8's decide test diverged because
screened-lm-headchose a token 0.1 nats below the best
one (see the known gaps): the test's exact chain now runs without that fast path; - a prefix-cache test read the Engine's statistics before their publication caught up with a result:
it now polls them; - the 27B DFlash2 tree tests counted other programs' memory as their CUDA Graphs': the server now counts
its own device memory (the Qwen3.5 family's fixes above); - the 4-bit-activation wide-plan scenarios ran on artifacts without 4-bit-activation experts: they skip
there, and run on FP4-Local in the suite; - the n-gram graph planning test expected an older allowance.
The non-real entries' time fell from 44 to 33 min: one test ran its idle modes on every shape, and now runs
them only on each route's decode shape (920 s to 305 s). - a CPU expert kernel overran its stack buffer on the test's vision shapes (more than 192 blocks of K; no
-
Linux: the release commit built in WSL Ubuntu (CUDA 13.4) without warnings. The CLI ran the 27B NVFP4
with DFlash2, and the server answered an OpenAI Chat request ("Bonjour") and an Anthropic Messages request
with a PDF, finding its code word. -
The release zip, unpacked with only System32 on PATH: its
infernix-serve.exe(build
windows-2026.11.18, the bytes the suite tested) served Flash-Next NVFP4-QAD Dense8. It answered an OpenAI Chat
request (which spent its 64-token limit thinking), an Anthropic Messages request ("Bonjour") and an Anthropic
request with a PDF, finding its code word throughpdfium.dll. Itsinfernix-launcher.exereported the same
build and served its page. -
Not checked for this release: updating to this release from an earlier launcher, and the README's
speed tables (see Benchmarks).
Known gaps:
screened-lm-head, a default fast path on QAD Dense8 (decode 1.4 % faster, measured for this
release), reads the output head through a 2-bit screen. In one test's synthetic prompt it chose a token
0.1 nats less likely than the best one;--no-fast-path screened-lm-heador--reproduciblereads the
exact head.- On the FP8 Flash-Next artifact, the preemption test still fails three of its checks, one each for plain
decode, MTP and DFlash2: each requires a resumed request to replay at least its 1,024-token prompt, and
there the replays covered 313-335 tokens. Every paused request completed all its tokens. - The CPU-served W4A4 misses' single walk makes Dense8's prefill 0.37 % slower at 2,048 tokens, a cost
whose cause is not established (see Key improvements). - out-of-memory recovery was tested by injection only: other processes taking up to 76 GiB made Windows
page instead of failing, and a shortage on a helper thread or inside anoexceptfunction still aborts; - the
gumbel-samplingregistration rests on a distribution test of low power; - PDFs: real-world PDFs, URL and encrypted PDFs and OpenAI file inputs were not tried served;
- Anthropic tool changes were not checked against a real Claude Code 2.1.325 capture;
--restore-reasoningwas not checked with real clients;- the launcher's GPU choice was not checked on a machine with two GPUs;
- in one series of FP4-Local starts on the build that keeps the seeds' RAM copies, the CPU team read
lower rates (~16.5K against ~20K experts/s, 5 of 6 starts); 18 later starts read 19.6-21.0K, and the
cause was not found.
Support
If you would like to help pay for the Claude Code subscription Infernix is developed with, you can
sponsor me on GitHub. Please also consider supporting
Neroued, who created NInfer.