vava-nessa/free-coding-models v0.5.60

on GitHub

Added

• 📈 Runtime telemetry: real-world scores (t3) — every routed request through the daemon now feeds a persistent per-model telemetry file with real success rate, throughput, and recent calls. FCM stops showing you only what models claim on SWE-bench and starts showing you what they actually do on free tiers.

  • Composite Real score (0..100) — successRate × 0.6 + sigmoid01(tok/s) × 0.25 + recency × 0.15. Null when below MIN_CALLS_FOR_SCORE = 5 (no penalty for new models).
  • Real success rate — (successCalls / totalCalls). Updated on every routed request, success AND failure paths alike (auth_error / 5xx / 429 all count).
  • Real throughput — avgTokensPerSecond from completion tokens / total latency. What your code actually streams at.
  • Recent calls (last 50) — FIFO trim per model. Useful for debugging "why did this just 500?" without rebuilding state.
  • Local-only by default — the file lives at ~/.free-coding-models/runtime-telemetry.json (0600 perms). Nothing leaves your machine unless you opt in to a future aggregate leaderboard. Metadata only — no prompts, no responses.

• 🆕 New sort key W in the TUI: sorts by Real score descending. Models with insufficient data (null score) sink to the bottom in both directions.

• 📊 New TUI Real column: shows the composite score inline per row, or – when there's not enough signal yet.

• 🔥 Shift+W Runtime Report overlay — per-model breakdown (success rate, avg latency, avg tok/s, recent calls list). Same pattern as Shift+R Router Dashboard / Shift+T Token Usage. Scroll with j/k/up/down/pageup/pagedown/home/end, close with Escape.

• 🌐 Web Dashboard Runtime Telemetry section — per-model cards with animated success-rate bars (red ≤ 50% / amber 50-80% / green ≥ 80%). Updated every /api/router/stats poll (5 s). Models with < 5 calls are excluded so the section only shows real signal.

• 🛰️ New daemon endpoint /stats/runtime — returns { ok, stats: { modelsTracked, totalCalls, modelsWithSignal }, models: { [key]: ModelTelemetry } }. Same shape as the in-TUI dashboard digest.

• 🧪 recordModelCall(providerKey, modelId, callResult, opts?) — drop-in hook for any future surface (Tauri Desktop, scripts) that wants to feed runtime telemetry.

• 🛠️ --clear-runtime CLI flag — wipe the telemetry file before launching any surface. Useful when the user wants to reset the baseline.

Changed

• 🩺 Daemon reverse proxy records every successful + failed routed outcome via recordRuntimeCall(). A debounced 5 s flush keeps the file in sync without thrashing the disk. The flush timer is cleared in shutdown() so no orphan writes after exit.

• 📋 Daemon statsPayload now exposes runtimeTelemetry: { stats, models } so the existing /stats consumers get the digest for free.

• 🔧 Internal — added sigmoid01(x) (logistic curve so 50 tok/s → 0.5) and recencyDecay(lastUpdatedMs) (1.0 today → 0.0 at 30 d) to the real-world score formula. Both are pure functions with no dependencies.

Non-goals (explicit)

• Streaming token extraction — the known gap from the PRD stays. The runtime telemetry only counts requests where the upstream returned usage (i.e. non-streaming, or streaming providers that emit usage on the final chunk).
• Sharing / leaderboard — no aggregation server yet. When there is one, opt-in.

Maintenance

• 🧪 +30 unit tests (test/runtime-telemetry.test.js, new file): path resolution (XDG-aware), constants sanity, recordModelCall write/accumulate/drop garbage/FIFO cap, tokensPerSecond computation, derived fields, getRealWorldScore null below threshold / healthy → high / error-heavy → mid / age decay, getCacheStats aggregates + modelsWithSignal threshold, pruneStaleEntries drops old, persistence round-trip + corrupt recovery + missing file + clear + read-merge-write concurrency safety.
• 🧪 671 → 701 tests passing (pnpm test), 125 → 134 suites.
• 🧹 vite build succeeds.
• 📖 README updated with a new 📈 Runtime telemetry: real-world scores section (where it shows up, privacy, --clear-runtime).

Inspiration

This implementation is informed by apmantza/pi-free's lib/telemetry.ts (which credits us in their other direction too). pi-free's turn_end hook is the daemon-side equivalent of our recordRuntimeCall; the schema, the 50-call FIFO, and the local-only file mirror their approach. Where we diverge: we expose the composite Real score as a first-class ranking signal (not just a debugging surface), ship it in 3 user-facing surfaces (CLI TUI / Web Dashboard / daemon /stats), and add the success×0.6 / speed×0.25 / recency×0.15 weights so the user sees a single number per model.

Files

• New: src/core/runtime-telemetry.js (377 lines, 11 exports), test/runtime-telemetry.test.js (390 lines, 30 tests), changelog/v0.5.60.md.
• Modified: src/core/router-daemon.js (record on every routed outcome + /stats/runtime + runtimeTelemetry in statsPayload), src/core/utils.js ('realworld' sort case), src/tui/app.js (boot-time score computation per result), src/tui/tui-state.js (5 new state fields for the overlay), src/tui/key-handler.js (Shift+W + overlay keyboard handling + command palette action), bin/free-coding-models.js (--clear-runtime flag plumbing), web/src/components/router/RouterView.jsx (Runtime Telemetry section), README.md, tasks/t3.md, package.json (test script).