JustVugg/colibri v1.12.1 on GitHub

96 pull requests since v1.12.0, 80 of them from contributors. Two tokenizers
brought back to the reference, brio on the ninth engine, coli chat working
again at the default context on two families, and a placement decision that
is now measured on the card in front of it instead of predicted.

Tokenizers, measured against the reference

#1654: qwen36 tokenized differently from HF tokenizers in two ways.
An added token right after punctuation was encoded as text (X.<|im_end|>
was 7 tokens instead of 3, every chat turn ending in punctuation paid +4,
#1653), and a whitespace run followed by a non-space was one piece where
the regex's \s+(?!\S) leaves the last char to the next one, so every
indented line of code tokenized differently. Measured on the real
vocabulary: 2,803 lines and blocks of code, Markdown, Chinese and
Japanese went from 757 identical to 2,803, with 5.4% fewer tokens.
#1656: OLMoE's tokenizer.json has no Split, a bare ByteLevel with
use_regex, for which HF runs the original GPT-2 pattern; tok.h applied
cl100k. A GPT-2 family in tok.h: 1,560/1,708 identical before, 1,708/1,708
after. The same measurement on GLM-5.2/5.3/5.3-Flash, DeepSeek V4 and V4.1,
Inkling and Qwen3.8 came back identical on every case.

Brio and the serve contract

#1662: POST /v1/systemone, the request and the reply of TypeSafe's
Jev API, served by the brio channel: a client written for it points at
colibri and changes the base URL. noul is a yes/no question, choice
scores the labels with their descriptions in the text, score the level
numbers with the expected value and the legend; confidence by their
documented formula. Any model name is accepted on that route. Measured on
the real Qwen3.6: the three-question example of the docs in 1m46 with the
state read once.
#1655: the DeepSeek V4 engine speaks the numeric channel (logprobs=k,
pin=1, max_tokens=0), so /v1/brio works on the ninth engine instead
of answering 500 (#1648). The head that used to keep only its argmax now
returns the whole row; ECHO per prompt position during prefill, the
prompt-end scores kept with a state snapshot on pin, and a logprob tail
on every DATA frame during generation. The tiny fixture pins that the
best ECHO token equals the greedy token from the same prefix, and that
the pinned predictor equals a cold prefill's.
#1659: qwen36 and qwen38 refused a request when prompt + max_tokens
exceeded the context, and the gateway's default budget for these two
families is 8192, the whole default context: every request without
max_tokens and every coli chat message answered 400 on a two-token
prompt (#1641). max_tokens is now a ceiling, clamped to the room the
prompt leaves, as GLM and DeepSeek V4 already did; only a prompt that does
not fit is refused. docs/api.md states the rule, docs/qwen38.md names
Q38_MAXT as the variable --ctx becomes.

The dense trunk in VRAM, measured before it is placed

#1657: qwen36 offers the rest of its dense trunk to the VRAM placer:
the DeltaNet out_proj (dnout), the attention q/k/v/o (attnproj) and
the shared expert (shexp), about 650 MB more of int8 on the 35B beside
lmhead and dnproj. Measured on four Tesla M10 by the reporter of
#1652, every placed component ran slower than the CPU (lm_head 68.8 ms
against 41.7), so the engine now times one GEMV both ways at startup and
withdraws the whole automatic placement when the GPU loses, giving the
VRAM back to the experts: auto equals off on that box, byte-identical
output. A hand-written COLI_PLACE stands; COLI_TRUNK_PROBE=0 trusts
the placer.

Performance

#1664: qwen36's dense trunk and routed experts multiply with integer
dot products. The activation is quantized to int8 once per call and the
weights, int8 rows or int4 planar blocks, meet it with maddubs / vpdpbusd
instead of a float conversion per weight; the integer kernels move from
quant.h into idot.h, shared by every engine. Measured on the 35B, 8
threads, every expert resident: decode 6.71 to 8.23 tok/s (+22.6%), lm_head
12.6 to 10.1 ms/token, the expert compute 22.7 to 15.6, for +1.3%
perplexity on 4 x 512 tokens. Both are the default (COLI_DENSE_IDOT=0,
QWEN_EXPERT_ACT=f32 restore the f32 kernels). COLI_DENSE_BITS=4 with
COLI_DENSE_INT4=<components> stores part of the trunk as int4 in blocks
of 64: opt-in, with the perplexity it costs per component in the docs
(lm_head alone +2.4%, everything +10%).
#1668: qwen38's dense trunk (553 matrices, 3.6 G weights, 8 GiB of
BF16 read on every token, more than the ten routed experts) is kept on the
CPU as int8 rows with the BF16 copy released, and multiplied with the same
integer kernels; the routed experts' e4m3 blocks are decoded eight at a
time in registers and multiplied with FMA instead of one table lookup per
weight. Measured on the released Qwen3.8-Flash-Next-FP8, 8 threads, RAM
LRU 96 per layer: decode 0.61 to 1.42 tok/s, the trunk 434 to 85 ms/token,
lm_head 76 to 13, the expert GEMVs 388 to 140, peak RSS 32.2 to 28.5 GB;
prefill of 512 tokens 495 to 149 s. Perplexity on 4 x 512 tokens +0.5%
(two chunks lower, two higher); the vector FP8 kernel alone reproduces the
BF16 run to four decimals. Both are the default (Q38_TRUNK_CPU_INT8=0
keeps the BF16 trunk, Q38_FP8_KERNEL=scalar the table kernel).

Performance, from contributors

#1606: the K1b grouped int4 family gets a multi-row tile and AVX-512
and AMX arms, and is no longer switched off on AVX-512 builds; exact on
all eight engines on a 16-core AVX-512 host.
#1239: an SSE4.1 tier for the olmoe and qwen36 int8 GEMV, for hosts
with SSE4.1 but no AVX2; on the Sandy Bridge of #1652 decode went from
2.45 to 3.54 tok/s.
#1313: matmul_fp8 computes four output rows per pass under clang,
where the contraction makes it bit-exact; GCC keeps the one-row kernel.
#1612: qwen36 gains the GLM engine's CACHE_ROUTE lever, with the
VRAM tier as the first residency level, opt-in.
#906: DEGRADE_ZERO, an opt-in policy that zero-fills a missed
expert slot below a gate-weight threshold instead of blocking on the
load (#865).
#1677: qwen36 projects a prompt's DeltaNet inputs (qkv and z) on the
card in blocks of up to 256 rows instead of one row at a time; a 259-row
prefill makes 2 projection calls instead of 259, with the convolution
history and the recurrent state checked against the CPU run. The paired
microbenchmark on an RTX 4070 read 4 to 10x per projection.
#1674: qwen36's attention projections the tier placed in VRAM answer
a whole prompt batch with one call per matrix; a failed call turns only
that handle off and the prompt continues on the CPU.
#1676: Kimi K3's streaming CUDA expert keeps the gate, up and SiTU
intermediates on the device and applies down there (one fused entry
point, optional in the DLL: an older backend keeps the three-call path).
#1673: the streaming MXFP4 matmul reuses one grow-only device scratch
per card instead of allocating and freeing weights and scales on every
call.
#1559 (kreuzzelg): convert_qwen36.py --down-bits 8 writes the mixed
expert layout, int4 gs64 gate/up and int8 down in one slab (5.7 bits per
weight against gs64's 4.5); the engine tells it apart by size and reads
each matrix in its own format on the CPU path, and refuses the VRAM tier
with a line. It is the knob behind the #1370 numbers: on wikitext-2 the
int8 down alone recovers a quarter of the gap between gs64 and all-int8,
the rest sits in gate/up. A measurement tool and a middle step, not the
answer to the gap.
#1730 (mfethe1): DeepSeek V4's FP4 expert kernels, the prefill batch
and the decode matvec, get a NEON arm; arm64 used to take the scalar
arm, which is why Apple Silicon prefilled at decode speed (#1696). Bit
identical to the scalar arm, and the ARM CI job now checks that on every
change; 17 to 22x on the kernel at the V4 expert shapes on an M-series
Mac, as measured by the author.
#1716 (jtinbergen): qwen36 quantizes its dense weights to int8 while
loading instead of keeping an f32 copy first, and converts f16/bf16 with
SIMD. On the 35B the resident set after load goes from 9.2 to 4.8 GB;
the generated text and the perplexity are identical to before.
#1286 (cameron): the grouped int4 GEMV and the fused gate/up GEMV get
an SSE4.1 arm for CPUs without AVX2 (Ivy Bridge and older). It vectorizes
across output rows, so each lane runs the scalar row's exact sequence and
the result is bit-identical; forced-SSE4.1 tests at -O1, -O3 and without
FP contraction pin it. 2.2 to 2.7x on the isolated kernel on a dual
E5-2680 v2.
#1686 (DebugSultan): qwen38's prefill chunk (Q38_PREFILL_BATCH_ROWS)
and workspace (Q38_PREFILL_WORKSPACE_MIB) are runtime knobs, the expert
load batch is no longer capped at top-k, and the QSA ranking and
attention run per position in parallel at prefill. Measured on the
released checkpoint on top of the int8 trunk: output byte-identical at
every chunk width, no speed change on our 16-core server; the knobs are
there for hardware where the chunk binds.

Fixed

#1650 (bokiko): a Qwen3.8 pin snapshots the recurrent and PLE state
but reuses the live attention and indexer rows; after an unrelated prompt
overwrote those rows, returning to the pin could change brio logprobs
without a warning. The engine now records the token identity of the live
rows (kv_prefix.h) and refuses a stale pin or prefix restore; image rows
are tainted. Wire regressions run on the BF16 and FP8 fixtures.
#1626: SNAP is the model directory for every non-GLM engine, so
coli run stops handing them a leftover environment (#1600).
#1604: glm53 honours Mat.resident in the Vulkan gate and frees the
Vulkan handle in mat_release.
#1321: glm53 sizes its expert cache around the model rather than
around MemAvailable, which the page cache had been inflating.
#1588: qwen36 refuses loudly on a failed encode-buffer realloc instead
of writing through NULL.
#1546: coli convert routes OLMoE to convert_olmoe_merged.py.
#1630: olmoe emits the ROUTE_TRACE records it announced; the stream
used to be a zero-byte file.
#1610: v41_dsml.py is staged during installation (and the nix flake
bumped).
#1511: the GPU test suite builds under HIP on gfx1151.
#1658: test_mem_available compared two reads of available memory
with == and failed on a busy Windows runner; a quarter of a GB of
tolerance.
#1670: DeepSeek V4.1 read only its argv cache cap, so RAM_GB=120
on a 128 GB box left the engine at eight expert slots per layer and
23.8 GB of RSS (#1666). With --cap omitted, coli chat, coli serve
and coli web now size the cache from the resource plan, with RAM_GB
or --ram as the budget; an explicit --cap, a measured profile and an
auto-tier plan keep precedence.
#1671: DeepSeek V4.1 treats max_tokens as a ceiling like the other
engines (#1641): a fitting prompt with a large request generates what
the context leaves, a score-only prompt may fill the context, and a
prompt one token over it is refused instead of silently truncated.
#1675: resident MXFP4 tensors on CUDA carried O float scales where
the kernel reads O x ceil(I/32) exponent bytes: short buffers were
over-read and long ones truncated. One format-aware size for upload,
refresh, accounting and release.
#1678, #1679, #1680, #1682, #1683, #1684: the
Qwen CUDA tier's lifecycle, end to end. Shutdown releases every resident
expert, projection handle and host table after parked callers resume;
a failed gate, up or down upload frees what it had already allocated;
the expert budget charges the three scale buffers at their own sizes
(two experts used to be admitted where one fit); a failed result
collection stops inference instead of publishing a partial MoE sum; a
failed or explicitly disabled tier start unwinds its storage and
synchronization objects, and a second init cannot overwrite a running
tier; the CUDA backend validates the whole device list before touching
state and keeps live contexts on a repeated init. Fault-injected on the
fake backend, then run together on an RTX 4070 under compute-sanitizer
with zero errors and zero bytes leaked.
#1669: test_systemone_api imports its scoring engine relative to
its package, so an installed tests package no longer breaks discovery.
#1697 (kevin9327): the dashboard redesign had dropped the reasoning
stream: thinking tokens arrived on delta.reasoning_content and vanished,
the bubble stayed empty until the answer and a stop during thinking lost
the turn. The stream is read again, rendered as its own folding block,
counted in the rate and the time to first token, and a unit test pins the
split.
#1693 (namespaceMarcello): with PILOT on, OLMoE could read the same
expert twice, once from the prefetcher and once from the forward pass,
into two slots; a slot being read now keeps a reservation in the index
(the colibri.c pattern) and the second caller waits for the first read
to publish. Three model-free scenarios pin it.
#1695 (namespaceMarcello): the prefill echo state and serve_echo sit
under the same QWEN36_NO_MAIN guard, so the segment build no longer
warns about a function it never gets; the full build is byte-identical.
#1724 (GenericRikka): Qwen3.6 decoded <think>, </think> and the
tool tags to nothing, because they live only in the tokenizer's
added_tokens; with thinking on, the closing tag never reached the
gateway and the whole answer came back as reasoning_content. The
non-special added tokens are decoded now; special ones such as
<|im_start|> still decode to nothing.
#1734 (tarazum): stopping coli serve closes the engine's stdin and
waits for it to exit on its own before the hard-stop ladder, so the
engine's teardown runs; qwen36 never saved its HEAT_FILE under coli serve (#1733). On Windows the gateway handles SIGBREAK and the engine
runs in its own process group.
#1726 (kevin9327): Inkling measured no RAM on Windows and sized its
expert cache to 16 per layer; it uses the shared probe now, which on
Linux and macOS reads the same numbers as before.
#1735: on GNU Make 3.81, the system make on macOS, .build-config
was never written and every build relinked (#1732).
#1731 (bokiko): the DeepSeek V4 CUDA object rebuilds when the nvcc
command changes, so a new CUDA_ARCH no longer links the old object.
#1728 (crichalchemist): make test-c VK=1 built 43 test binaries
without the Vulkan object; they link it now.
#1712 (kevin9327): Kimi K3, Inkling and OLMoE now treat max_tokens
as a ceiling like the other engines; coli chat's default of 16384
answered 400 on every Kimi and Inkling message against their 8192-token
window.
#1713 (kevin9327): coli plan, doctor and --auto-tier export the
variable that actually sizes the expert cache on Kimi K3
(K3_EXPERT_GB) and GLM-5.3 (GLM53_EXPERT_GB); only RAM_GB was
exported, which neither engine reads as the cache size.
#1711 (kevin9327): on Windows, a Kimi K3 CUDA_DLL build and a HIP
host were refused by --gpu as CPU-only; the probe reads the backend DLL
name the host was built with, and the launcher maps --gpu onto Kimi's
K3_CUDA.
#1710 (kevin9327): a tools[] entry whose function is not an object
answered HTTP 500 from the GLM and DeepSeek renderers; it is the 400 that
generation_options already had.
#1721 (monotophic): every frame the gateway writes to the engine is
checked, short writes are completed, and a failed CANCEL or STOP
drops the request's pending entry and answers a named 500 instead of a
silent close.
#1714, #1719 (benmaster82): the brio options form pins the shared
state, so per-question requests on one document read it once; the web
page can stop a scoring run, and duplicate options are removed on both
clients.
#1709 (kevin9327): regenerating a turn with pictures sends them again
and leaves the composer alone.
#1646 (Stamina9): qwen38 says once, on stderr, why the parallel expert
read path is not taken (disabled, cache smaller than the route, no FP8
scale bank, repeated expert, converted layout).
#1708 (wittchen): every VK=1 build of glm53 failed to compile on a
misplaced parenthesis.
#1707 (namespaceMarcello): the DeepSeek V4 unit objects rebuild when
the build flags change, so a CUDA engine build followed by make test-c
no longer links the wrong objects (#1702).
#1715 (namespaceMarcello): seven GLM-5.3 harnesses matched the
unittest glob and counted as zero tests; they are renamed, a skip exits 2,
the two tiny oracles run in CI, and a discovery test catches the next
empty module (#1700).
#1622: DeepSeek V4's REAP checkpoints store each expert as six
per-matrix records; the engine read them through buffered pread and
counted every one as a direct-I/O fallback (36% of expert reads on the
150B, #1615). Each segment now goes through the aligned direct window,
with a regression on a generated per-matrix fixture.
#1597: a replayed tool call whose arguments parsed as JSON but was
not an object ("[1, 2]", "5") answered HTTP 500 from the GLM renderers
before the engine was asked anything; it renders the call without
arguments, as every other renderer already did.
#1624: the five gcc 13 warnings left in make check are gone, and
st_index_load refuses an index path that would not fit its buffer
instead of opening a truncated one, with a long-path case in the tests.
#1651: a pyflakes pass over the launcher, autotune, the family
registry and the qwen36 converter: a measure() defined twice, a
readline import without a fallback, a stray f-string, dead variables.
#1580: make qwen36 CUDA_DLL=1 on Windows reached GNU make's implicit
rule and built a CPU-only binary; a bare qwen36 alias, a .build-config
prerequisite so a CUDA_DLL change rebuilds, a loader-against-header parity
test, and the Windows CUDA tier documented.
#1556: coli plan on macOS said "no supported GPU detected" on every
Mac; it now lists the Metal device by name, as identity only, without
pretending unified memory is a VRAM budget.
#1691: the installed launcher invoked as /bin/coli or /sbin/coli
on a merged-/usr system derived /libexec/colibri instead of
/usr/libexec/colibri, because abspath kept the alias (#1689,
florin65's patch): realpath first. A test runs the launcher through
such an alias, and another checks that every root module the launcher
reaches is in the make install list, the gap #1610 closed by hand.

Tools and the gateway

#1425: a general GGUF reader, pure stdlib, and a converter from GGUF
OLMoE checkpoints to a colibri container, with the numerical evidence in
its own CI job.
#1497: opt-in prompt-injected tool calling for the families without
native tool tokens (OLMoE, Qwen3.6), behind COLI_TOOL_FALLBACK=1, with a
two-turn end-to-end test.
#1355, #1357: durable per-request results and strict stdout
classification in the eval harness, and the logprob-gap check gated on
the engine preamble.
#1687: GET /metrics in Prometheus text format, behind the API key:
four gauges, six outcome counters and four histograms (queue wait, slot
occupancy, first output, engine call), no request labels, no new
dependency. The admission scheduler distinguishes completion, failure
and cancellation, lets a request use a free slot that no earlier waiter
reserved, and joins the keepalive pump before the slot is released.
#1717 (enitimeago): the web chat offers Continue on the last
assistant message when it stopped at the token limit, by hand or on an
error, and only when /health says the server continues assistant
turns (#1699).
#1402 (enitimeago): a request whose last message is a non-empty
assistant turn continues that turn instead of answering in a new one, on
/v1/chat/completions and /v1/messages, for all nine families (Kimi K3
frames the open turn engine-side); the prompt ends inside the turn as the
official template renders it without a generation cue. On by default,
COLI_CONTINUE_ASSISTANT=0 restores the old behaviour; refused together
with tools or a turn ending in whitespace, with a 400 that says why. Each
renderer is pinned against the vendored template (#1401).
#1102 (monotophic): checkpoint-faithful FP8 containers that store
kv_b_proj as fmt=8 could load but not decode attention; the absorb path
now decodes fmt=8 on CPU (bit-exact against the reference) and CUDA
(within the documented tolerance), and the kv_b sharding refuses by name
the formats it cannot serve, which also closes two silent misreads of
fmt=5 and fmt=6.
#1395 (rybruscoe): COLI_EXACT_VERIFY=1 makes the speculative verify
batch token-exact against sequential decode, at a measured cost on the
dot itself; off by default, the default path is unchanged.
#1720 (monotophic): a request carrying seed is accepted and the seed
ignored, as docs/api.md now says, instead of a 400; no determinism is
implied.
#1605: ORACLE_STRICT=1 makes a GLM oracle comparison exit non-zero
when it fails, token-exact by default with ORACLE_TF_MAX_MISMATCHES for
the documented teacher-forcing allowance; references are validated before
the comparison and non-finite logits cannot pass. Both oracle CI jobs run
real-process regressions against it.
#1705: tools/benchmark_baseline.py, a collection protocol on top of
the HTTP harness for a repeated three-engine serving baseline: one frozen
manifest (hardware, model and template identity, per-engine launch
settings, cache and speculation policy), a rotating plan over a
concurrency matrix, one collector per engine and round that manages no
server, and a comparison that keeps failed and missing cells visible and
distinguishes matched artifacts from deployment comparisons. No results
are bundled and no ranking is emitted.
#1688: tools/benchmark_http_serving.py, a stdlib HTTP streaming
benchmark over fixed JSONL conversations: closed-loop or paced arrivals
(periodic or Poisson, seeded), warmup separated from measurement,
first-output and duration SLOs, latency percentiles and usage-based
token throughput; a truncated or malformed stream is a failure, not a
sample.

Docs

#1639, #1644: the README shows brio mode and the dashboard as it
is: the workspace, the Brain page (the measured expert atlas as a cortex,
and a region inside it) and the Profiling page, in four languages;
docs/api.md describes the four pages instead of the old console.
#1492, #1643: a Japanese README, and its banner at the shipping
version, which the banner test now checks in every language.
#1634: the multi-disk guide states measured gains and limits instead
of "twice the bandwidth", with Bash and PowerShell examples.
#1617: connecting the pi coding agent to coli serve.
#1619: expected_bytes identity versus physical extent for
int4-rans256-g0 (#1273).
#1618 (bherald): docs/qwen38.md no longer calls the engine text-only;
the vision tower and the gateway image path shipped in 1.12.0.
#1649, #1647 (Suraj2105-1): the musl CI job runs on Alpine 3.24
and the release pipeline on Node.js 22, ahead of the 3.21 and Node 20
end of life.
#1568 (Yoruxyv): an Indonesian translation of the dashboard.
#1681 (XBold): the README and docs/qwen38.md no longer say Qwen3.8
has no GPU backend; the CUDA VRAM expert tier and the int8 trunk in VRAM
shipped in 1.12.0.

JustVugg/colibri v1.12.1 colibri 1.12.1 on GitHub