JustVugg/colibri v1.8.0 on GitHub

60 pull requests since v1.7.0. The cycle's spine: every engine now finds a
resident expert in one probe instead of a scan, prefill stopped re-reading
bytes it already had, and the DeepSeek V4 CUDA floor dropped from Ampere to
cards from 2016. Everything risky is opt-in; every default path is covered by
token-exact oracles.

Expert lookups: O(1) on every family

#1164, #1174 — DeepSeek V4 resident-expert lookup made O(1), and miss
cost made independent of the configured cache size.
#1193, #1215 — the same expert-to-slot index extended to Inkling and
Kimi K3, then to GLM, OLMoE and Qwen3.6: a resident hit is 1 probe where
the legacy worst case was 44 to 219, validated by model-free tests with a
deterministic probe counter. Every lookup revalidates the slot identity, so
stale bookkeeping can only become a miss, never wrong weights.
#1180 — V4 prefill reuses the expert cache across layers and batches
shared-expert work; the LRU used to re-read the same experts several times
per layer under prefill's expert-major sweeps.
#1162, #1166, #1171, #1178 (@TheOptik) — duplicate concurrent expert
loads coalesced; FP4 expert matmuls batched for prefill and specialized per
phase; fp8 attention weights decoded with SIMD instead of per-element
gathers.
#1197, #1200 — Inkling and Qwen3.6 batch dense and shared-expert prefill.

DeepSeek V4: a wider GPU tier

#1181 (@8PotatoChip8) — CUDA_ARCH=portable-pre-ampere NO_TC=1 brings
the whole CUDA tier to Pascal and Turing (GTX 10 / RTX 20 series); the
loader floor drops accordingly.
#1202, #1203 — opt-in DSV4_HYBRID=1: on a VRAM-tier miss the missing
experts are split between the GPU fill branch and the CPU branch, sized by
bandwidths measured at runtime, with the fill DMA overlapped against host
compute and host slabs pinned on Linux.
#1213 — opt-in COLI_CUDA_MOE_DOUBLE=1: a second full-layer expert
bank is prefetched on a dedicated stream while the current layer computes,
and the banks swap at the layer switch. Falls back to single-bank when VRAM
is short.
#1211 — the shard index is built once per engine open instead of six
times, the config parsed once instead of three, and the fp8 row transpose
vectorized bit-identically; a v4_open stderr line reports the split.
#1141 (@dcutugno) — measured guidance for V4_LOADER_LANES with and
without the GPU tier.

Both DSV4_HYBRID and COLI_CUDA_MOE_DOUBLE are experimental: defaults are
byte-identical with the envs unset, and they are looking for community
numbers on real cards.

Kimi K3: checkpoints, a vendor oracle, tool calling

#1188 — a tiny independent oracle validates the engine's KDA and MLA
paths token-exact against Moonshot's reference implementation, in CI,
without torch or network.
#1192, #1194 — opt-in recurrent-state checkpoints (COLI_K3_CKPT=N,
disk-parked via COLI_K3_CKPT_DIR): an edited or follow-up prompt restores
the deepest surviving checkpoint and re-prefills only the tail instead of
replaying the whole conversation through the SSM layers.
#1208 — K3_VK_UP=auto sizes the Vulkan expert-tier upload from
measured bandwidth instead of a fixed guess.
#1144 (@ZacharyZcR) — tool calling wired up with XTML rendering per
Moonshot's reference renderer.

KV cache quantization (@ZacharyZcR)

#1151 — CPU KV cache quantization lands: KV8 (fp8 e4m3) and KV_TQ.
#1155, #1207 — the flash and gather kvb arms became KV8/KV_TQ aware,
and KV8_GS adds grouped latent scales following FlashMLA's fp8 cache
geometry.
#1142 — prefill k/v reconstruction tiled past a size ceiling
(cause 2 of #768).

coli tune: measured, for every engine

#1196 — tune measures every engine through the serve protocol, not
just GLM.
#1209 — tune measures safe RAM and cache caps on the machine instead
of deriving them from totals.
#1195, #1201, #1175 — cap resolved from the plan when --cap is
absent; rotating workloads persisted and V4 loaders tuned; Metal reported
correctly in the plan label on Apple Silicon.
#1198 (@Mr-Neutr0n) — an explicit RAM_GB is clamped to the memory
actually available (#759).

Fixes across the fleet

#1220, #1222 — since the V4 CUDA tier landed, the launcher's --gpu and
--vram check only knew the Windows DLL and refused every valid Linux
make deepseek-v4 CUDA=1 build (#1219, a 1.7.0 regression); detection now
reads the binary's actual CUDA linkage, rejects lookalikes, and names the
right build per platform. Reported by @ANBAL534.
#1134 (@tonnthuir) — RSS guard measurement and reclamation fixed under
Linux.
#1167 — io_uring expert reads now honor the mirror replica (#1165).
#1152, #1153, #1156 — Inkling Metal MoE qgs at the callsites; the web
UI renders the reasoning stream instead of dropping it (#1148).
#1186, #1187 (@waizuichougou) — Qwen3.6 discovers visible CUDA devices
for the default tier and skips the discarded final decode.
#1170 (@jerome-benoit) — make install builds qwen36 first.
#1133 (@QQSHI13) — compress_ratios longer than num_hidden_layers
accepted.
#1185 — the GLM banner labels bit widths as compute, not storage.

Tools, bench, CI, docs

#1105, #1107 (@monotophic) — full-family FP8 e4m3 container mint with
content-validated resume; a per-token logprob channel, echo prefill
read-out, and token-ID prompt intake.
#1205, #1206, #1131, #1132, #1138, #1145 (@ZacharyZcR) — per-record
layout census for rANS containers; datapoint fixes for Windows 11 labeling
and win32 RAM probing; the tok/s bound split out of the determinism test;
V4 docs cross-references.
#1139 (@SyedYousufFaizan) — datapoint.py parses DeepSeek V4 logs.
#1182 (@Unknown-Findout) — datapoint checks temp free space before the
eviction write.
#1184, #1161, #1127 (@view321) — datapoint survives slow engine
teardown; persistent rotating-prompt bench datapoints; iobench sub-MB
blocks and a Windows [threads] fix.
#1160 — macOS CI compiles every Metal-capable engine with METAL=1.
#1216 — the README caught up with what already runs: the V4 CUDA floor,
the six families, K3 checkpoints, and the experimental V4 GPU levers.

JustVugg/colibri v1.8.0 colibri v1.8.0 on GitHub