vava-nessa/free-coding-models v0.5.63

on GitHub

Added

• 📊 Extended benchmark catalog (t4) — a new src/data/benchmarks.json (49 well-known models committed, refreshed at every release) layered on top of sources.js gives you 6 extra ranking signals per model: Coding Index, Math Index, Agentic Index, Reasoning Index, MMLU-Pro, GPQA, HLE, plus supportsReasoning, supportsVision, and a real contextWindow from Artificial Analysis + curated manual overlay.

  • Lazy load via Proxy — the JSON is only parsed on first lookup, not at module load. Cold start stays at ~0 ms.
  • Prefix-indexed O(key length) lookup — lookupExtendedBenchmark('deepseek-ai/deepseek-v4-pro') is fast even on a 500-entry catalog. Falls back through the -//-prefixed segments (e.g. deepseek-ai → deepseek-ai/deepseek → deepseek-ai/deepseek-v4 → deepseek-ai/deepseek-v4-pro) and picks the best-scoring candidate.
  • Curated seed wins — the mergeExtendedBenchmark overlay only fills fields that are null in sources.js. Your curated sweScore / tier / ctx are never overwritten by the live data.

• 🛰️ Live models.dev enrichment (t5) — the community-maintained models.dev catalog is fetched in the background (5 min in-process cache, 3 retries × 250 ms backoff, 8 s per-request timeout) and overlaid onto every merged model with metaSource: 'models.dev' | 'sources.js' provenance.

  • Substring matches skipped for drift detection — the indexer tries exact → aliased → substring, but the drift detector only counts exact + aliased matches (avoids the "DeepSeek Chat" vs "DeepSeek Reasoner" false positive).
  • Provider aliases — 40+ mappings bridge sources.js provider keys (nvidiaNim, together, novita, kilocode, …) to models.dev provider keys (nvidia, togetherai, novita-ai, kilo, …).
  • Offline-safe — if the fetch fails (3×8 s = up to 24 s, capped to 12 s in the background task), metaSource stays at 'sources.js' and the TUI keeps rendering. No crashes, no hangs.

• 🔍 --check-drift CLI flag — diff sources.js against models.dev and print a human-readable report grouped by model. Each row shows the field, the sources.js value, the models.dev value, and an action arrow (← UPDATE or ← ADD).

  free-coding-models --check-drift                       # exit 1 on any drift
  free-coding-models --check-drift --drift-threshold 5   # only fail on 5+ mismatches
  pnpm check:drift                                       # npm alias

  Exit codes: 0 no drift · 1 drift detected · 2 fetch failed · 3 invalid args.

• 🤖 pnpm update:benchmarks — regenerate src/data/benchmarks.json from a fresh models.dev fetch. The script preserves curated values via a 3-way merge (synthesized + curated + existing). Wired into the release process and the weekly CI workflow.

• 🦾 TUI footer enrichment chip — 📊 bench 49 (2026-07-25) · 📡 102 live · 62 curated on the bottom line. Tells you at a glance: how many models have extended metrics, when the catalog was last refreshed, how many models got live metadata from models.dev vs staying at curated values.

• 🌐 Web Dashboard data exposure — /api/models now includes extendedBench + metaSource per row. Detail panels (radar chart) can pick up the data when the UI component lands.

• 🕒 .github/workflows/update-benchmarks.yml — weekly Monday 04:00 UTC job. Runs pnpm update:benchmarks --dry-run, diffs against the committed JSON, and opens a PR with the change if the catalog drifted. Manual trigger via workflow_dispatch.

• 🕵️ .github/workflows/check-drift.yml — weekly Monday 05:00 UTC job. Runs pnpm check:drift --no-fail. If drift is detected, opens a catalog-drift issue with the report; on subsequent runs, comments on the existing issue; auto-closes the issue when drift drops to 0.

Changed

• 📦 CLI help (--help) — new flags --check-drift and --drift-threshold <N> documented under the Analysis Flags section, right next to the existing probe-cache flags.
• 🧠 src/core/utils.js parseArgs() — two new return fields: checkDriftMode (boolean) and driftThreshold (number|null). The TUI's bin/free-coding-models.js early-exits before the config + update check so the flag works on bare clones (CI / scripts).

Non-goals (explicit, tracked for follow-up)

• TUI detail-view benchmark block — the chip is the visible surface for now. A full "Press ? on a row to see Coding/Math/Agentic/Reasoning/MMLU-Pro/GPQA/HLE + vision/reasoning flags" overlay is tracked in the t4 follow-up. The data layer + tests are ready.
• 6 new sort keys for the benchmark indices — the key map is already crowded (R/O/M/L/A/S/C/H/V/U/B/T/W/Z/E/F/Y/X/Q/G/N/P/I). Adding 6 more would conflict. Tracked for a "Sort by benchmark" submenu in the Command Palette.
• Web Dashboard radar chart component — the data is exposed, the UI component is a separate task. The /api/models payload includes everything the chart needs.

Maintenance

• 🧪 +66 unit tests across 3 new files:
  • test/extended-benchmarks.test.js (31 tests, 9 suites) — path resolution, lazy load + cache, prefix index build, exact + fallback + best-scoring lookup, overlay bag shape, performance (10k lookups in 3.3 ms, 1k cached in 1 ms).
  • test/models-dev.test.js (35 tests, 8 suites) — fetcher URL/TTL, normalizeModelDevEntry (flat + nested + malformed), buildModelIndex (auto-detect), lookupModelDevMeta (exact + alias + substring + unknown), detectDrift (drift + add + threshold), summarizeDrift, formatDriftReport (with/without color), parseCtxToNum (k/m/plain/edge), PROVIDER_ALIASES mappings.
  • test/model-merger.test.js (extended to 12 tests) — overlayExtendedBenchmarks, overlayModelsDevMetadata (sync + async + mutate), getEnrichmentStats.
• 🧪 701 → 779 tests passing (unchanged — same code, only the files field changed) (pnpm test), 134 → 155 suites.
• 🛡️ pnpm start runs without runtime error (TUI, Web Dashboard, daemon all load the new modules cleanly).
• 🐛 Fixed a real freeze: the initial draft had a top-level await import in src/tui/app.js that blocked module load. Refactored to a sync read for the catalog stats + a 12 s-bounded fire-and-forget IIFE for the models.dev fetch. The TUI now starts in <1 s even when the network is unreachable.

Inspiration

This implementation is informed by apmantza/pi-free's lib/model-metadata.ts (the fetcher + retry + cache + provider aliases shape) and provider-failover/benchmark-lookup.ts + hardcoded-benchmarks.ts (the prefix index + lazy Proxy load pattern). Where we diverge: we ship a curated 49-entry seed in the repo (their TS version relies on hardcoded constants at build time), we add the drift detector + weekly CI workflow for proactive catalog hygiene, and we expose the data on all 3 surfaces (CLI TUI, Web Dashboard, Desktop) with consistent metaSource provenance.

Files

• New: src/core/extended-benchmarks.js (320 lines, 11 exports), src/core/models-dev-fetcher.js (180 lines, 7 exports), src/core/models-dev-index.js (290 lines, 5 exports), src/core/models-drift.js (320 lines, 6 exports), src/data/benchmarks.json (49 entries committed), scripts/update-benchmarks.mjs (220 lines, executable), scripts/check-drift.mjs (170 lines, executable), .github/workflows/update-benchmarks.yml, .github/workflows/check-drift.yml, test/extended-benchmarks.test.js (310 lines, 31 tests), test/models-dev.test.js (380 lines, 35 tests), changelog/v0.5.63.md.
• Modified: src/core/model-merger.js (+155 lines: 3 new exports + async overlay with mutate), src/core/utils.js (+14 lines: parseArgs + checkDriftMode/driftThreshold), src/tui/app.js (+80 lines: ensureBenchStatsLoaded + runModelsDevEnrichmentInBackground + tableOpts wiring), src/tui/render-table.js (+40 lines: enrichmentLabel chip), src/tui/cli-help.js (+2 lines: new flags in ANALYSIS_FLAGS), bin/free-coding-models.js (+19 lines: --check-drift early-exit), web/server.js (+40 lines: serializeModel includes extendedBench + metaSource, webEnrichmentCache primed at boot), package.json (+5 lines: test commands + scripts), test/model-merger.test.js (+100 lines: new test cases), tasks/t4.md, tasks/t5.md.