CoreScope v3.8.1 — Route, Refined

Released 2026-05-27. Previous: v3.7.2 (2026-05-06). Supersedes v3.8.0 (released earlier today, before this perf fix landed).

⚡ Patch in v3.8.1: /api/nodes TTL fix

Community contribution from @efiten (#1425). The repeaterEnrichTTL was 15 seconds but the background recomputer runs every 5 minutes — so the cache was stale for 4m45s out of every 5min. On dense hop-graph deployments that meant 18s /api/nodes cold-miss responses. TTL is now derived as 2 * recomputer interval so the cache stays warm between ticks. Measured 18.6s → ~0.5s on the contributor's prod mesh (analyzer.on8ar.eu).

If you already deployed v3.8.0, just upgrade to v3.8.1 — same image semantics, drop-in.

Three weeks, 135 merged PRs. The headline is the Route Viewer redesign — packets and the paths they travel, finally rendered as one coherent story. Underneath: startup-window improvements for large DBs, a near-complete mobile chrome overhaul.

🗺️ NEW: Route Viewer

The new route viewer (route-view.js + route-view.css) puts packet context, observer paths, and resolved hops in one sidebar — and reflows cleanly to a bottom sheet on mobile.

• Packet context block in the sidebar header — per-type fact list (ADVERT name/role/sig/pubkey, DM src→dst with encryption state, GRP_TXT channel + decrypted preview, TRACE official-vs-observed hops, PATH src→dst with payload-source chips). Merges pkt.decoded_json + obs.decoded_json and falls back to byte-level raw_hex parsing for encrypted DMs.
• Multi-path picker — chip per unique observer-path (<count>/<total> + hex hop string). Click to isolate; "All" renders an edge-deduplicated UNION view (each unique edge once, stroke weight = observer count).
• Deep-link URLs — #/map?packet=<hash>&obs=<id>. Bookmarkable, shareable, single source of truth. sessionStorage flow removed.
• Hop resolution priority chain — server resolved_path → shared HopResolver (observer-IATA-aware, same as packets page) → raw prefix. Kills a whole class of "route view named hops differently than packet detail" bugs.
• Markers — uniform 22 px filled circle with seq number inside, hollow endpoint ring for SRC/DST, double concentric ring on SRC=DST loops, spider-fan for sub-14 px collisions debounced to zoomend.
• Colorblind-preset live colors — routeRamp per preset (viridis / plasma / pure-luminance) written to --mc-rt-ramp-0..4 CSS vars and hot-recoloured on cb-preset-changed / theme-changed.
• Desktop chrome — drag-to-resize sidebar persisted to localStorage, Material/Drive-style collapse chevron centred on the right edge.
• Mobile bottom sheet — anchored above bottom-nav + safe-area inset, thin summary line when collapsed (TYPE · N hops · X km · M obs), expands to ~75 vh. All three legacy mobile detail panels closed on route entry.

Closes: #1418, #1419, #1422 · PR #1423

Related map work that landed this cycle: role-aware marker shapes + outline rings (#1334), WCAG 2.2 AA pass on cluster bubbles + role pills + multi-byte labels (#1357), thinner always-on marker outline (#1347).

⚡ Startup

• Hot startup window — new packetStore.hotStartupHours config loads only N hours of packet history synchronously and fills the rest in background daily chunks. Large DBs (multi-GB) no longer block the HTTP listener on a full-retention load before serving traffic (#1187).
• Ingestor now owns the neighbor-graph and all schema migrations; the server is read-only. Removes a class of startup races where two processes raced the same migration (#1286, #1289).

🔬 NEW: Protocol decoder coverage

The MeshCore decoder now handles every documented payload type, with the legend updated end-to-end so operators see real names instead of Type 6.

• GRP_DATA + MULTIPART wire-format decoded (#1280) — group-data sensor payloads + multipart fragmentation are no longer opaque blobs.
• CONTROL flags + advertRole fix — accurate role attribution on ADVERTs from edge cases that were previously misclassified (#1280).
• payloadTypeNames table centralised, exposed at the API + UI; new TransportCodes / Feat1 / Feat2 enums; RAW_CUSTOM type plumbed through; sensor-payload docs published (#1291).

🆕 NEW: Operator features

• Repeater "usefulness score" (bridge axis) — second of four planned ranking dimensions for how much a repeater is structurally necessary to mesh connectivity (#1275, #672 axis 2/4).
• Area-based visual node filter — define GPS polygons in config; attribute and filter packets by transmitter location across nodes/packets/map/live/analytics (#839).
• Scoped vs unscoped transport-route statistics — per-region HMAC slicing of route stats; mirrors the existing hashChannels pattern (#915).
• Sortable Scope column on the Nodes list (#1195).
• Observer IATA badges on packets, with new filter grammar to slice by observer location (#1189).
• Multi-byte hash capability is now persisted across restart with O(1) per-key lookup; operator-friendly capability badge surfaces it in the UI (#1324).
• Material Design dark mode — properly polished pass (closes #893, #1389). Primary surface set, contrast, button + chip + form-control restyle.
• Colorblind presets in the theme customizer — Wong / Deuteranopia-tuned / Plasma / Achromat ramps, applied to roles + multi-byte tiers + (in this release) the new route viewer's sequence ramp (#1378, #1408, #1414, #1407).
• Mobile gesture system — swipe rows for actions, swipe tabs to switch sections, slide-over dismiss (#1185); edge-swipe to open the nav drawer (#1184); first-visit hints so the gestures are discoverable (#1186).
• New CoreScope logo + Aldrich webfont (#1137, #1138).

🐛 Bug Fixes

Protocol / Ingest

• MQTT decode no longer panics on malformed path length — bounds check prevents slice [218:15] panic (#1214).
• MQTT half-open TCP recovery — watchdog forces paho reconnect on stall (#1336); per-attempt logging surfaces silent reconnect-loop death (#1216).
• Ingest-time vs observer time — ingestor stamps server now() for packet/observation storage; reverted earlier envelope-timestamp path that was driving counters wrong (#1233, #1372).
• Hop disambiguator quality — source-diversity confidence weighting in tier-1 resolver (#1235), hop-context + observation-count tiebreak (#1198), 6 deferred quality items (#1200).
• Geo-implausible edges rejected at neighbor-graph build time (#1230).
• Clock-skew robustness — RTC-reset outliers excluded from hash median + recent-bad count (#1288).
• Canonical resolved_path everywhere — /api/nodes/{pk}/paths (#1282) and paths-through resolver (#1353) now use the persisted resolved path instead of naive prefix matching; fixes wrong-node attribution.
• Structural pubkey attribution via new from_pubkey column (#1152).
• dbschema startup race fixed by giving optional-column migrations a canonical source (#1322).

UI / Channels / Packets

• Channels page — chat-app redesign restores prod row layout + detail view (#1376); shows latest message time, not first-seen (#1368); drops ghost "unknown" bucket for encrypted-no-key packets (#1377); stops force-enabling "show encrypted" on every init (#1410); mobile UX overhaul (#1227).
• Packets page — orders by ingest id rather than rxTime so fresh activity is actually visible (#1349); observer IATA shown on packets with new filter grammar (#1189).
• Scopes tab — fixed JSON.parse by dropping duplicate /api prefix in scope-stats fetch path (#1379).
• Live region filter no longer wipes the feed — correctly parses {observers:[...]} response shape (#1140).
• Prefix Tool Network Overview shows configured-hash-size counts, not math-only slices (#1271).
• Customizer — nodeColors stopped force-overriding ROLE_COLORS so CB presets actually propagate (#1408, #1414).
• Node detail full-page renders error state on 404 instead of silent blank (#1158); restored WCAG AA contrast on "Paths Through" links in dark mode (#1159); section reorder puts Recent Packets above Paths (#1160); dropped orphan separators from "Heard By" rows (#1161).

🎨 UI / UX Polish

• Material Design dark-mode polish (#1389, closes #893) — primary surface set, contrast pass, button + chip + form-control restyle.
• Customizer — live preview pane mirroring map / nodes / packets (#1408 deps), per-preset WCAG validator on save (#1407 chain).
• Multi-byte hash UI — capability badge with operator-friendly tooltip (#1324), config-driven counts everywhere (#1271).
• Nodes page — neighbor-aware reference-node selector (#1389 chain), geo/role/hash-size filter URL params survive reloads.
• Live page — reconnect indicator + last-event-ago badge.
• Theme — operator-customizable accent + role colors via single JSON file, shared across map / nodes / packets / live / channels.

🔧 Operator & Infra

• New config keys (see Upgrade notes for full list): packetStore.hotStartupHours, compression.gzip, compression.websocket, areas, hashRegions.
• Compression — opt-in HTTP gzip + WebSocket permessage-deflate, both default-off (#934).
• Area-based filters — define GPS polygons in config; surfaces an Area: selector across analytics / nodes / packets / map / live (#839).
• Scoped transport-route stats — hashRegions HMAC mirrors the hashChannels pattern, lets operators slice transport-route metrics by region (#915).
• /api/perf/io — exposes cancelledWriteBytesPerSec from /proc/self/io plus ingestor side (#1167).
• Ingestor/server architectural split — neighbor-graph + schema migrations are now ingestor-owned (#1286, #1289); see Upgrade notes.
• /#/perf page — parallel health fetch + sort + auto-pause refresh when tab is hidden (#1261).

🧪 Testing & dev

• Playwright E2E suite stabilized — flake fixes that unblocked master (#1330, #1310, #1317).
• 122 new assertions for the route viewer covering hop resolution priority, raw_hex byte extraction, edge weight scales (incl. boundary fix), spider-fan collision grouping + loop double-ring, channel hash → name resolution, and CB-preset CSS var integration.
• CI workflow updated to run the new route-view tests on every PR.

📊 By the numbers

• 135 PRs merged since v3.7.2
• ~101 issues closed in the same window
• 3-week cycle; ~6.4 PRs/day

Upgrade notes

No breaking changes. Drop-in replacement for v3.7.2.

New config keys (all default to safe values — existing config.json works unchanged):

• packetStore.hotStartupHours (float64, default 0 = disabled) — load only this many hours synchronously at startup; remaining retentionHours filled in background. Recommended for DBs > a few GB.
• compression.gzip (bool, default false) — HTTP gzip middleware. Skips WebSocket upgrades and clients without Accept-Encoding: gzip.
• compression.websocket (bool, default false) — WebSocket permessage-deflate.
• areas (array, optional) — GPS polygons for area-based packet attribution; surfaces an Area: dropdown on analytics/nodes/packets/map/live when configured.
• hashRegions (object, optional) — region-name → HMAC map for scoped transport-route stats; mirrors the existing hashChannels pattern.

Architectural note: the ingestor now owns the neighbor-graph table and runs schema migrations; the server is read-only. If you run ingestor and server as separate processes, ensure the ingestor starts first on a fresh DB.

Database migrations are automatic (added from_pubkey column to transmissions, scope_name column for transport-route stats; both nullable, populated lazily). No manual steps.

Known Issues

• Colorblind preset switching still requires a full page reload to apply everywhere. Most surfaces hot-recolour via cb-preset-changed events, but several (parts of the analytics charts, some legacy badge surfaces, and a few hop-detail panel elements in the new route viewer — unresolved-hop ring, spider-fan hairlines, marker text contrast) still cache colors at render time and pick up the new preset only on next mount. Operator workaround: reload the page after switching presets. Full hot-swap coverage is a follow-up.
• Mobile route view auto-fit can briefly overshoot on the first render when iOS Safari is mid-URL-bar collapse; settles within ~1.5s as the staggered refit timers fire. Cosmetic, no data loss.
• Sidebar drag-resize doesn't persist across hard reloads if localStorage is blocked (private browsing / strict cookie settings) — falls back to the 320px default silently.
• Customizer is only reachable on mobile by rotating to landscape. The settings entry point lives in a nav region that collapses on portrait widths; operator workaround is rotate-to-landscape, open the customizer, rotate back. Proper portrait surfacing is a follow-up.
• "Scan QR" when adding a PSK channel is not yet functional on either mobile or desktop. Manual hex-key entry works as a workaround; QR-scan flow is a follow-up.
• Analytics → Distance tab still launches the route map via the legacy sessionStorage flow, not the new #/map?packet=<hash>&obs=<id> deep-link. Route loads correctly but the URL isn't bookmarkable / shareable and "Back to packet" is unavailable. Other entry points (Packets page View Route, direct URL) already use the new flow. Follow-up will swap the analytics call sites.