github Wallawalla47/Infernix windows-2026.11.07
Infernix windows-2026.11.07
on GitHub

one hour ago

Infernix windows-2026.11.07

AI disclaimer: Infernix was developed with AI (mostly Claude Opus 5.5). It is hobby work and
is likely to be neither complete nor entirely accurate.

Highlights

  • An SSD tier for the prefix cache (opt-in). --ssd-context-gib N --ssd-context-dir DIR keeps
    the context the RAM tier of the prefix cache gives up in a scratch file on an SSD and reads it back
    instead of recomputing it, for every model. On FP4-Local, a 100K-token conversation pushed out of a
    4 GiB RAM tier resumed in 0.42 s from a Samsung 990 PRO, against 10.8 s to recompute it. In an
    agentic coding replay the average time to first token fell 21.6 % (4.3 to 3.4 s) and the 90th
    percentile 43 % (9.7 to 5.5 s). --ssd-context-gib-per-hour caps the writes (8 GiB an hour by
    default), snapshots are compressed losslessly and the drafter's ring is left out. It is off by
    default because it writes to the SSD: 10 GB in the 13-minute replay at the default budget.
  • Lossless packing: more context and more cached experts at the same output.
    • --kv-dtype bf16p stores BF16 KV in 23 % fewer bytes with bit-identical output, on Qwen3.5/3.6
      and Qwen3.8-Flash-Next: the 27B's longest start on a 32 GB GPU grows from 160K to 208K tokens.
      A request whose KV cannot be packed fails alone. The launcher's KV menu offers it.
    • Qwen3.8-Flash-Next packs its BF16 weights to 12 bits at load, losslessly. On NVIDIA's NVFP4
      bit-exact conversion, 8,782 MiB of BF16 weights now take 5,689 MiB of VRAM plus 910 MiB of RAM,
      and the freed VRAM caches more experts: plain decode +10.4 %, structured output +8.3 %.
  • N-gram trees (--ngram-tree-nodes N, off by default). When the text being copied occurs
    several times with different continuations (two versions of a file, say), one round verifies
    every occurrence as a tree. On the NVFP4 27B, a prompt with two versions of a file decodes 20 %
    faster at 32 nodes, and one-source prompts stay within 0.3 %. On Flash-Next, n-gram copies and trees
    are now chosen by measured round time: beside MTP, copies gain 8 % on two-version prompts; beside
    DFlash2 they still cost 1-3 %, so keep them off there (docs/ngram.md).
  • Fair service under KV pressure. When requests contend for KV, each now gets bounded service
    turns (about a second of new output, or up to 4 s of recovery) in rotation, instead of the oldest
    request taking all the capacity, so a long request can no longer leave another silent until it
    finishes. The design is NInfer's.
  • RAM given back under memory pressure (--ram-resize auto, the server's default). While less
    than 2 GiB of RAM is available, the prefix cache's RAM tier and the offloaded experts' RAM shrink
    between requests, and grow back once the memory is free again. On Flash-Next IQ3_S it gave back
    43.5 GiB and took all of it back within about 10 s of the memory coming free.
  • Output quality measured against the exact model. The README's new
    Quality section and
    docs/quality.md compare
    Infernix, NInfer, vLLM, TensorSharp, llama.cpp, Strata and Basalt on the same weights, using
    llama.cpp's KL-divergence statistics against Infernix's --lossless route, with honest rounding
    as the floor. On the 27B NVFP4, Infernix's decode path is 2.76× honest rounding (NInfer 2.74×,
    TensorSharp 9.4×, vLLM 11.1×). On Flash-Next IQ3_S it is 0.90× (llama.cpp 0.95×, Basalt 1.20×,
    Strata 1.45×). The tooling is in the repository (tools/quality/engine_compare.py).
  • No more slowdown at 256-token prompts on offloaded models. Prompts just over 255 tokens used
    to prefill more slowly than 255-token ones. The CPU now helps with prompt calls up to 575 tokens on
    every Flash-Next conversion and the 122B: IQ3_S prefills 19-23 % faster at 256-320 tokens. The FP8
    conversion takes CPU help at every prompt length, 12-26 % faster from 640 to 1,536 tokens.

Key improvements

  • Qwen3.8-Flash-Next speed (the same output, unless marked lossy):
    • verification rounds keep more of their RAM-expert reads on the PCIe link: structured output
      +3.6 % on FP4-Local's 4-bit route, +2.5 % on its 16-bit route and +4.3 % on Dense8, and the
      same rule on the FP8 conversion;
    • the shared expert starts early on every default route (+0.7-1.0 %);
    • faster DFlash2 rounds: a one-kernel drafter prepare, faster drafter feature projection and
      top-16 selection, GGUF embedding rows read 16 bytes at a time (16.2 to 7.4 µs a call), and the
      expert-cache update and the commit moved off the critical path;
    • prefill: MXFP8 GEMMs from 16,384 tokens keep their weights in L2 (FP4-Local +2 % at 32K tokens),
      FP4-Local's 16-bit prompt calls run on tensor cores, a 9-31 % faster general BF16 GEMM, and
      width-matched Q8 hyper-connection tiles;
    • QSA decode block selection in shorter steps (11-17 % faster at 8K tokens), a one-trip
      combine-inject, more kernels launched as programmatic dependents, faster NVFP4-KV and BF16-KV
      decode attention, and a radix top-16 for the DFlash2 head (490 to 244 µs);
    • the adaptive expert cache swaps at a margin of one route instead of 1.5, measured faster on IQ3_S;
    • the fast paths measured faster with INT8 KV now also run with FP8, NVFP4, K8V4, VQ2 and K4V2 KV,
      and FP4-Local's 4-bit route (--expert-a4-activations) gets the screened LM head (plain decode
      +6.6 %) and FP16-accumulate prefill (+5.7 % at 32K tokens);
    • --a8-activations (lossy, opt-in) on GGUF imports: the hyper-connection matrices are repacked
      to 8 bits and the 8-bit kernels are faster (IQ3_S with DFlash2, conversation prompt: 236.7 to
      263.7 tok/s).
  • Qwen3.5 family (27B, 35B-A3B, 122B-A10B):
    • the 122B resumes a continuing turn from the prefix cache (time to first token after a
      15,000-token turn: 1.65 s to 0.35-0.44 s), preempts under pressure (two queued requests' first
      tokens 2.3 and 2.8 s sooner), and runs the 8-bit PV prompt attention (+0.7-2.7 % at 64K tokens);
    • the decode graphs launch their serialized kernels as programmatic dependents;
    • an automatic KV size leaves 256 MiB of VRAM free on a GPU that drives no display (1 GiB before);
    • replay records and verification buffers are sized for the widest round that actually runs,
      which frees VRAM.
  • Serving:
    • --end-in-thinking refuse (the default): a reply can neither end nor open a tool call while its
      thinking is still open, which stops empty answers and turns that wrote tool calls inside their
      thinking until the output limit;
    • the reasoning-loop guard also stops a thinking run-on of hyphen-joined words, seen on long
      Flash-Next agent requests;
    • tool calling: several <function=...> blocks in one <tool_call> are read as separate calls;
      tools whose parameters are a root oneOf/anyOf get typed arguments; a stray backtick no longer
      turns the next line's call into text; Claude Code's deferred MCP tools can be called directly;
    • with vision off, an image inside a tool result (Claude Code reading an image file, Codex's
      view_image) becomes a short text note instead of failing that request and every later one in
      the session;
    • streams: each protocol's own heartbeat event during a long prefill (Codex keeps the stream), a
      Chat stream opens with its first real event (an honest time to first token), a /v1/responses
      WebSocket upgrade answers 426 so Codex switches to HTTP at once, and a paused streaming
      reader keeps its request for 60 s (the HTTP write timeout was 5 s);
    • Anthropic: thinking.display: "updates", thinking type between_tools (it thinks only when
      continuing after tool results) and an effort with thinking disabled are accepted (each was a
      400);
    • with --tolerant-tool-calls, a cut-off tool call is kept only when the output limit, the
      context window or a cancellation cut the generation; one a stop token or stop string cut comes
      back as text;
    • HTTP keep-alive: the window is 360 s by default (45 s before), advertised in Keep-Alive: and
      closed 10 s later, so pooled clients such as Bun's fetch (opencode) never write onto a closed
      socket; --max-http-clients sets the open-connection ceiling (64);
    • request bodies, and history tool-call arguments, nested deeper than 256 levels get a 400 instead
      of overflowing the server's stack;
    • images: a transparent PNG drops its alpha, as llama.cpp, Ollama and transformers do, and lossy
      WebP decodes to Pillow's pixels;
    • --no-fast-path NAME leaves one default fast path out, to measure it against the default route.
  • Launcher: the KV menu offers BF16 packed, and the configure page sets the SSD tier and the RAM
    resize. A server that cannot start for a missing DLL is named as such. Also fixed: .bat quoting,
    restart and log-rotation races, download checks, update checks against a release without notes,
    and console log writes that could stall the server's output pipe. Chat images are shown as links,
    so nothing is fetched.
  • Fixes:
    • under the new service turns, a resumed request could drop its own cached state because it did
      not fit beside the other request at that moment, and replay instead: on Qwen3.5 it recomputed
      from the start, on Flash-Next its output could change with when it was paused. A resume now
      takes its deepest cached state and the Engine makes room for it, and a Flash-Next pause always
      publishes its own resume point;
    • with the SSD tier on and a small RAM tier, Flash-Next could evict a just-finished conversation's
      endpoint before it reached the SSD, so the next turn recomputed from an earlier point. A capture
      that needs room now writes unwritten snapshots back to the SSD first, and waits for them;
    • the prefix cache file keeps superseded snapshots, so a prompt re-sent after a restart resumes
      instead of prefilling from scratch (#59);
    • an Engine failure now logs its cause at Error ("…: the Engine failed; all requests failed"),
      from PR #60 by Sugaroverdose;
    • the prefix cache could end the Engine under RAM-tier pressure while a snapshot was still being
      written (27B, 35B and 122B);
    • a tool call written straight after </tool_call> could end the Engine on the default tool route;
      a call the decoder cannot read now fails only its own request;
    • the 122B failed prompts whose first prefill pass was wider than 65,535 tokens (prompts of about
      140K tokens), and several Flash-Next kernels had the same limit;
    • the offloaded 122B fits its prefill chunk to what its expert cache can lend, so a long prompt no
      longer fails after admission on a full card, and --plan now reports what a start allocates;
    • attention keeps a row's history weight at exactly 1 while its maximum holds: before, weights on
      long prompts drifted slightly between key tiles;
    • Flash-Next's first token no longer depends on the width of the call that ends the prompt;
    • an out-of-range token id in an embedding gather fails its request instead of reading outside
      the table;
    • a GPU other than compute capability 12.0 is refused at startup by name, before anything loads;
    • Linux: the locked-memory soft limit is raised before locking, and a lock that still fails names
      the remedy.
  • Build: building from source now requires CUDA 13.4 (CUDA 13.2 miscompiles some sm_120a
    kernels), and the startup log records the nvcc version beside the build id.
  • And many more smaller fixes from code reviews, new tests (a two-process reproducibility check,
    tools/reproducible_digest.py, and a build that exposes reads before programmatic waits) and
    documentation updates.

Supported models

Each repository's model card gives its files, requirements, recommended command, quality and
speed; the README's Models section lists them
all.

Benchmarks

The README's speed tables are the previous release's (one RTX 5090 at PCIe Gen5 x8, 2026-10-30/31,
on builds of that cycle). They were not repeated for this release, so they do not include the
speedups above. The new Quality section was measured 2026-11-05/06 on the same PC.

Behaviour notes

  • Output bits change on the default and --reproducible routes: the attention fix, Flash-Next's
    first-token fix and the thinking rules below. Both routes remain deterministic and
    request-invariant; --lossless is unchanged.
  • --end-in-thinking refuse is the default: a reply cannot end, nor open a tool call when tools
    are declared, while its thinking is open. --end-in-thinking allow restores the old behaviour.
  • HTTP keep-alive defaults to 360 s (45 s before). --max-http-clients (64) is new, and
    --max-pending-requests now accepts 1-1024.
  • --ram-resize auto is the server's default; --ram-resize off keeps the RAM tiers at their
    start size.
  • Under KV pressure requests take bounded turns rather than following age order: an older request
    can now be paused too, and a paused request resumes sooner.
  • With vision off, an image inside a tool result becomes a text note; an image anywhere else still
    gets vision_disabled.
  • A Chat stream's first chunk now arrives with its first generated event, not when the request is
    admitted.
  • Transparent PNGs give different pixels on Windows (alpha dropped, not composited over white).
  • Building from source requires CUDA 13.4 or later.
  • The --prefix-cache-file format is now 4 on Qwen3.5 and 2 on Flash-Next: a file saved by the
    previous release is ignored once and replaced at shutdown. Saved files are larger, since they
    keep superseded snapshots (379 MB against 262 MB in the Flash-Next test). Their fingerprint also
    names --expert-tail-skip and the GPU.
  • The serve request log stays at schema 31 and adds prefix_cache_block_tokens; a request's done
    line says how many prompt tokens were in cached blocks with no state to resume from.

Downloads

  • infernix-2026.11.07-windows-x64.zip: infernix-serve.exe and infernix-launcher.exe with
    their FFmpeg, curl, zlib and libjpeg-turbo DLLs, the maintained chat templates, the licence of
    every component they contain or load (licenses/, including every npm package bundled in the
    launcher's web UI) and THIRD_PARTY_NOTICES.md. Unpack it to a folder and run
    infernix-launcher.exe, or infernix-serve.exe directly. An earlier launcher can install it
    from Settings → Updates, which keeps infernix-launcher.json, logs and chats.
  • ffmpeg-9.0.1-source.zip: not needed to run Infernix. It is provided for licence
    compliance: the FFmpeg DLLs in the release zip are LGPL-2.1-or-later, and this is their
    corresponding source (unchanged since the previous release).

Requirements: an RTX 50-series GPU (sm_120a), an NVIDIA driver of 580 or later (CUDA 13), and the
Visual C++ Redistributable for Visual Studio 2015-2026. No CUDA toolkit is needed. Linux users build
from source (see the README).

Verification and known gaps

Everything below ran on one RTX 5090 (PCIe Gen5 x8, driver 621.01) under Windows 11, built with
CUDA 13.4.

  • The full test suite, on the release tree before its last fix: all 301 non-real CTest entries
    passed (27 min, 8 at a time), and 72 of the 73 real-model entries passed (94 min, one at a time) on
    the 27B NVFP4 (NInfer's official artifact), Flash-Next NVFP4 Dense8 with the DFlash2 drafter and
    the n-gram volume, the 35B-A3B and Ornith-1.5-122B. Five real entries skip by design: they are tools that run only
    on inputs passed by hand (an artifact argument, a token list or reference file, a tokenizer
    directory). The Python suites passed (476
    tests, 1 skipped because it needs a transformers 5.5 environment, plus the reproducibility
    digest's 3).
  • The one failure was Flash-Next's SSD-tier endpoint resume in
    infernix_qwen4_exp_prefix_cache_real_test, fixed above. It failed only under CTest; run directly,
    the same build passed, which is how my first fix for it looked right and wasn't. After the second
    fix the whole test passed under CTest on Dense8 and FP4-Local (2-4 % slower, from waiting on the
    extra write-backs). I did not re-run the rest of the suite after that fix: it changes only
    Flash-Next captures with the SSD tier on, which no other test uses.
  • The release's first full run had 13 failures (1 non-real, 12 real), and every one is fixed. The
    service-turn resumes and the SSD tier were product bugs (above). The rest were the tests' own: the
    bench-table test expected the old table, the n-gram stop test could no longer reach the case it
    checks, and the 35B and 122B tests read the dense model's variable (they now have their own). Two
    test criteria changed:
    • infernix_qwen3_5_resume_cold_real_test allows a resumed decode up to max(0.5, 4×) its
      calibration from a cold one, up from max(0.1, 4×). On the NVFP4 27B, whose prompt path uses
      4-bit activations and decode 16-bit, correct resumes of ten texts measured 0.019-0.185; a stale
      state measures 3.99.
    • Flash-Next's n-gram tree test on Dense8 now counts a tree arm's tokens only from its first
      difference, and holds its misses to the worst of the run's chain arms plus two. Dense8's own
      chain arms are already 5.15 nats apart, which was more than the test allowed.
  • The release zip, unpacked with only System32 on PATH: its infernix-serve.exe (build
    windows-2026.11.07) served Flash-Next FP4-Local with DFlash2 and answered an OpenAI Chat request
    (which spent its 64-token limit thinking) and an Anthropic Messages request ("Bonjour"). Its
    infernix-launcher.exe reported the same build and served its page.
  • Not checked for this release: Linux (not built), updating to this release from the
    windows-2026.11.01 launcher, and the README's speed tables (see Benchmarks). The SSD tier's
    agentic-replay figures in the Highlights were measured before the SSD fix. When the RAM tier is
    short of room, the fix writes more to the SSD (2,076 MB against 1,356 MB in the test's SSD
    sections, which set no write budget), and a capture waits for those writes.

Support

If you would like to help pay for the Claude Code subscription Infernix is developed with, you can
sponsor me on GitHub. Please also consider supporting
Neroued, who created NInfer.

Check out latest releases or
releases around Wallawalla47/Infernix windows-2026.11.07

Don't miss a new Infernix release

NewReleases is sending notifications on new releases.

Get notifications