Infernix windows-2026.10.18

AI disclaimer: Infernix was developed with AI (mostly Claude Opus 5.5). It is hobby work and
is likely to be neither complete nor entirely accurate.

Highlights

• Two new Qwen3.8-Flash-Next conversions (both computed in exact integer arithmetic with 16-bit
  activations, so expert placement and batch width never change a bit):

  • GSQ-RCO IQ3_S:
    ISTA-DASLab's IQ3_S GGUF imported bit for bit, with an MTP drafter and vision. The fastest
    decode of the conversions (146 tok/s plain and 260 with MTP against Dense8's 110 and 143 in the
    same session), with the experts in 46.9 GiB of RAM.
  • FP8: Qwen's FP8 checkpoint
    bit for bit with Qwen's BF16 n-gram table, the most accurate conversion.

• Any n-gram table with any Flash-Next model. The PLE n-gram table comes in three volumes (FP8,
  IQ4_NL and BF16), and every Flash-Next conversion now runs with any of them (--ngram-volume).
  The published FP8 volume works with the IQ3_S and FP8 conversions as downloaded. With Dense8 the
  perplexity is 3.5102 with its own FP8 table, 3.5101 with BF16 and 3.5140 with IQ4_NL.

• A numerical acceptance policy for fast paths.

  • --disable-fast-paths runs the exact route, which is bit-exact from release to release and, on
    every Flash-Next artifact and the NVFP4/FP8 27B ones, request-invariant: other requests,
    concurrency, cache contents and memory pressure do not change a request's output.
  • By default the engine runs every faster path qualified for the artifact's kernels and KV format
    under one fixed budget: at most 1.5e-3 excess KL, 0.5 points of top-1 and 4e-3 TV against a
    16-bit reference, judged at 95 % confidence. The README's new Quality section lists it and the
    other engines' defaults that failed it.
  • --fast-path NAME runs a further path for qualification.

• Faster Qwen3.8-Flash-Next. In the agentic replay on Dense8 (three seeds, the README's
  workload), measured on main earlier on 2026-10-18 against the README's published runs:

  • decode rounds per second with one request decoding +19 % (output tok/s +10 %, +13 % at the run's
    own batching);
  • prefill of requests with no cache hit +8 %;
  • median time to first token −16 % and the 90th percentile −6 %; the average +1 %;
  • workload wall time −5 %.

  Among the changes: route, dispatch and the CPU plan fused and launched as programmatic dependents;
  the expert cache seeded from a calibrated expert order at a cold start and from each prompt's own
  routes after it; MTP draft steps past the confidence cut stopped on the device; and a 16K prefill
  chunk.

• Flash-Next options:

  • --draft-tokens auto (MTP draft length per round, or the fastest fixed length where adaptive
    lengths are not qualified);
  • --spec-min-p (the drafts' confidence cut);
  • --expert-tail-skip R (opt-in, approximate: skips some experts missing from VRAM for 12-33 %
    faster decode at a small quality cost; Strata does this by default at rank 7);
  • --ngram-residency auto|ram|volume, --expert-stripe, --expert-pin-mib;
  • --route-trace.

• Serving and the prefix cache:

  • After a reply that ends in a tool call, the conversation's cached state is held for 30 s
    (--cache-tool-hold-secs), since the agent returns once its tool has run.
  • Snapshots are taken where a prompt leaves its cached path, so a re-sent transcript with content
    inserted before its end resumes at the branch.
  • --cache-host-state fp24|bf16 packs the Host tier's recurrent state; --config FILE reads
    options from JSON.
  • --claude-code-prompt as-sent|local.
  • A thinking request with no budget now leaves a quarter of max_tokens for the answer
    (--no-thinking-answer-room turns that off).
  • Anthropic Messages responses carry llama-server-style timings.

• Fixes:

  • Windows page-locks memory only up to the GPU's shared system memory: experts past that stay in RAM
    unpinned instead of failing startup ("SSD tier: cannot pin ... expert slots").
  • JSON-constrained output admits bounded whitespace (it returned {"":"ok"} on Qwen3.8 without
    thinking).
  • Qwen3.5 keeps serving after a capacity contract error and with paused requests (it answered every
    later request with 503).
  • A paused MTP request resuming from the prefix cache no longer fails its replay.
  • #19: the reasoning-loop guard splits words
    by code point, as Strata does.
  • #21: Flash-Next is converted with the
    maintained Qwen3.8 chat template, which accepts System messages after the first turn. Artifacts
    converted before should be started with --chat-template.
  • #29: hybrid prefix-cache taps and endpoints
    undo their supersession when no Device slot or publication follows, and Device-only snapshots
    rotate out of the last slot instead of holding every slot for good.
  • #24: Flash-Next's reclaim returns its stream
    and wide-frame leases and grows KV (found by reading the code; the reported scenario passed both
    before and after, so the issue stays open).
  • Fixes from an audit of this cycle's changes, among them a stale expert copy in the RAM tier's
    read-ahead and an under-sized Flash-Next prefill workspace.
  • A startup link probe and CPU-rate probe that could misread on a busy machine.
  • The release zip ships the maintained chat templates.
  • The remaining MSVC, GCC and nvcc warnings are fixed.

Benchmarks

The README's benchmark tables were measured on 23d310d6 (the previous release's build) and are
being re-measured for this release, against the latest Strata, strata-nvfp4 and upstream NInfer;
the README will be updated when that is done. On the 27B the agentic replay measured this cycle's
build 3.7 % fewer decode rounds per second than the previous release (output tok/s −2 %), because the
4- and 8-bit decode activations it used fail the new quality gate and are no longer the default.

Behaviour notes

• Breaking: --prefill-8bit-pv and --no-prefill-8bit-pv are removed; the 8-bit P×V prompt
  attention is now the prompt-pv8 fast path, on by default where it is qualified.
• The serve request log is schema 28 (spec_min_p in server_start).
• The default output of an artifact can differ slightly from the previous release where a fast path
  is now qualified for it; --disable-fast-paths gives the exact route.
• The 35B-A3B's Q8 shared-expert prefill is more precise (3-5× lower error against an FP64 oracle;
  2.3-2.6 % slower prefill), which changes its exact-route prefill bits once.

Downloads

• infernix-2026.10.18-windows-x64.zip: infernix-serve.exe with its FFmpeg, curl and zlib
  DLLs, the maintained chat templates, the licence of every component it contains or loads
  (licenses/) and THIRD_PARTY_NOTICES.md.
• ffmpeg-9.0.1-source.zip: not needed to run Infernix. It is provided for licence
  compliance: the FFmpeg DLLs in the release zip are LGPL-2.1-or-later, and this is their
  corresponding source.

Requirements: an RTX 50-series GPU (sm_120a), an NVIDIA driver of 580 or later (CUDA 13), and the
Visual C++ Redistributable for Visual Studio 2015-2026. No CUDA toolkit is needed. Linux users build
from source (see the README).

Verification and known gaps

Verified on the tagged build (RTX 5090 at PCIe Gen5 x8, Windows 11, VS 2026 / CUDA 13.4):

• The unit suite: 228 of 228 pass.
• Qwen3.8-Flash-Next real tests on Dense8: all 9 that run pass (expert store, VRAM pressure,
  expert-read fault, prefix state, prefix cache, walk interleave, preemption, sampled drafts and
  /v1/decide); the forward, scoring and route-trace tests skip without their extra inputs.
• Qwen3.8-27B real tests on the official NVFP4 artifact: 21 of 23 pass. The two failures are not new
  (below).
• The release zip, unpacked with only System32 on PATH, serves Dense8: models, chat, streaming,
  token counting, an image, Anthropic Messages and Responses.
• During the cycle: every Flash-Next conversion started and generated with each of the three n-gram
  tables; on the 27B with DFlash2, the default and exact routes gave the same text as the build
  before the final fixes, at the same decode speed.

Known gaps:

• Three real tests fail and also failed on the previous release's code:
  • infernix_qwen3_5_dflash_real_test needs an artifact with the original DFlash drafter, which the
    official artifact does not carry.
  • infernix_qwen3_5_dflash_prefill_real_test builds its options without the hybrid cache's
    normalization (a test fault).
  • infernix_qwen3_5_moe_real_test (35B-A3B) expects a 6-token prompt's continuation to resume 10
    tokens from the prefix cache and resumes none; the windows-2026.10.11 build fails it the same
    way. Not investigated further for this release.
• The benchmarks above were measured on main before the final audit fixes; the README's tables are
  being re-measured.
• Linux was not built or tested for this release.

Support

If you would like to help pay for the Claude Code subscription Infernix is developed with, you can
sponsor me on GitHub. Please also consider supporting
Neroued, who created NInfer.