JustVugg/colibri v1.1.0

colibri v1.1.0

on GitHub

A community release. 27 pull requests from more than 20 contributors, 216 commits since
v1.0.0. Most of what follows was found, measured, or fixed by people who do not work on
this project and had nothing to gain from it.

Added

• AMD GPU support (HIP/ROCm) (#339) — single-source backend_gpu_compat.h with a WMMA
  dispatch gate, so one codebase builds for CUDA and HIP. Validated on an RX 9070 XT
  (RDNA4, ROCm 7.2): token-exact against CPU on a real fmt=4 gs64 container, with resident
  dense and with routed experts in VRAM, plus a fail-injection control proving the GPU
  actually executed the work.
• Dual-SSD streaming (COLI_MODEL_MIRROR, #421) — read the model from two drives at
  once, roughly doubling streaming bandwidth on a disk-bound host.
• N-drive shard split (COLI_MODEL_DIRS, #469) — capacity aggregation: run a container
  no single drive can hold, spread across several with no duplication.
• fmt=5 (int3-g64) (#168) — 3-bit weights with per-64 group scales: measured 3.3x lower
  outlier-row error than per-row int4 at 25% fewer bytes.
• fmt=6 (E8/IQ3 lattice) (#465) — CPU decode kernel and dispatch; index codec tooling (#458).
• tools/try_tool_calling.py — dependency-free two-turn tool-calling probe that doubles
  as a smoke test.

Fixed

• Tool calling in coding clients (#401), root cause found, in two parts:
  • #506 — the engine capped prompt encoding at CTX-2, and the tokenizer stops dead at
    its limit without reporting anything. A prompt longer than the context was therefore
    silently truncated to its first CTX-2 tokens and answered anyway. With the 4096 default
    that is 4094 — exactly the prefill 4094 in the field report. The dropped tail was the
    tool instructions and the user's actual turn, so the model emitted a bare < and stopped;
    and because clients append to the end while truncation keeps the head, every retry
    re-sent a byte-identical prompt. Now refused with a 400 context_length_exceeded.
  • #505 — a tool call whose closing </tool_call> never arrived was dropped whole,
    because the parser required both tags. Now recovered when unambiguous, on both the
    streamed and non-streamed paths.
  • #437 — non-EOS role markers were armed as hard stops in serve mode and cut generation
    the instant a tool block started.
• Grouped-int4 (fmt=4) produced garbage output on CUDA with CUDA_DENSE=1 (#298) — the
  dense and attention kernels applied per-group scales as if they were per-row. Hardware-verified.
• OpenMP tuning re-exec preserved the CPU affinity mask (#476), jailing every thread onto
  one core when OMP_PROC_BIND/OMP_PLACES were set: roughly a 20x slowdown.
• Pilot eviction guard dropped ~100% of speculations once the cache filled (#497),
  collapsing PILOT_REAL to a hint-only path.
• Silent budget clamp capped the CUDA expert tier at ~109 experts regardless of
  CUDA_EXPERT_GB (#495).
• fmt=4 guard at the per-row-only CUDA entry points (#464/#470); COLI_CUDA_MTP=1 and
  COLI_CUDA=0 are now honoured over implicit defaults (#468).

Security

Threat model: model files come from mirrors that are not trusted.

• #368 — server hardening, JSON and tokenizer parser hardening, build flags, downloader
  and dependency pinning.
• #413 — the quant layout is resolved and validated against the on-disk byte counts
  (unknown layouts are refused rather than falling through to int2), shape-product overflow
  is rejected, and the olmoe dtype-3 path no longer trusts a crafted nbytes (heap overflow).

Performance — all byte-identical

• #481 +4.7x on the MLA-absorb score and value-mix reductions
• #477 +13% decode on AVX-512 (qt_addrow / qt_matvec_rows)
• #475 +11.6% with opt-in XEXP=1 (one OpenMP region per expert block at S=1 full residency)
• #473 +5.5% int4 IDOT at S=1 on AVX-512 VNNI

Changed

• glm.c is now colibri.c plus header modules (#391); make glm remains as an alias.
• Serve stage 2 (#192): response_format, per-request grammars, grammar-forced drafts.

Upgrade notes

• CTX still defaults to 4096. Coding clients send far more than that in a single system
  prompt. Use CTX=32768. Before this release an over-long prompt was silently truncated;
  now you get a clear 400 instead.