ServiceRadar v1.5.6
This release fixes a NATS lockout that a runtime certificate rotation
caused on installs with certificate authorization enabled, and starts
removing camera stream addresses from Core: plugin results are reduced
as they arrive, existing copies are scrubbed, and agents and the
UniFi Protect and Axis plugins can now resolve a stream address at the
edge. The NVR gains a recording catalog, holds, recording policies,
per-recorder storage settings, a health view and unified timeline
playback with an access audit; the background workers that act on
policies and retention are still to come. Syslog is stored under the
sender's time at microsecond precision, and the log collector can
follow the Zen ruleset core publishes and normalize syslog itself (off
by default). netprobe degrades instead of crash-looping on a missing
capture interface, leased sweeps keep the hosts they finished before
their window closed, and edge late admission gains its bounds, late
grants and terminal-interval list, still inactive.
Upgrade Notes
- Upgrading from v1.5.3 or earlier: read the v1.5.5 and v1.5.4
upgrade notes below first. They all still apply to v1.5.6 and are
not repeated here. v1.5.4 was tagged but never published. - Runtime certificate rotation no longer locks platform components
out of NATS. Withnats.certificateAuthorization.enabled, core
admits an adopted infrastructure client only by a certificate
fingerprint it has recorded. The move tocerts.runtimeLayoutVersion
6 in v1.5.4 reissued every runtime certificate, nothing recorded the
new ones, and the broker refused core, agent-gateway, web-ng, datasvc
and the collectors. The certificate generator now writes a public
runtime-cert-rotation.json(fingerprints and certificates, no keys)
into the runtime certificate Secret. Core records each reissued
certificate as the successor of the one it replaces, in the
migrations pre-upgrade hook and again whenever the mounted record
changes. It accepts a successor only for a recorded, unrevoked
certificate at the component's current generation, with the same
issuer. An upgrade from v1.5.3 or earlier is handled this way. A
rotation that already happened on v1.5.4 or v1.5.5 has no record:
record its successors by hand with
POST /api/admin/managed-components/:id/certificates(see
docs/managed-component-certificates.md). Replaced certificates stay
authorized until you revoke them; revoke them once no pod presents
them. The flow-collector bootstrap hook now runs after the
certificate generator and migrations (hook weight 0, was -5) and
gives up after 600 s instead of holding the upgrade. The chart
refusesnats.certificateAuthorization.enabledtogether with
certs.regenerator.enabled, because the regenerator issues a new
root CA (#6732). - NATS identities of disabled features get no permissions. The
flow-collector, netflow-collector and k8s-inventory users rendered an
empty allow list when their feature was off, which NATS reads as
every subject. They now deny>for publish and subscribe, so a
client presenting one of those certificates while its feature is
disabled is refused. The chart fails to render if a static user has
an empty allow list without denying>(#6728). EventWriter no
longer lets aNats-SubjectorNats-Subject-Tokenheader replace
the subject a message was delivered on; stream selection and every
processor use the broker subject. Nothing in ServiceRadar sets those
headers, but an external publisher that used them to re-route
messages no longer can (#6727). - Camera stream addresses are being removed from Core, in stages.
A stream address is a secret: a UniFi Protect RTSPS alias works as a
bearer token, and other addresses carryuser:password@. Core now
reduces every plugin result as it enters, before anything is stored:
RTSP and RTSPS URLs keep only their origin, other URLs lose their
userinfo (including URLs inside free text), and alias keys are
removed. Addresses are no longer stored in status, state, device,
source or profile records. The two relay columns,
camera_sources.source_urland
camera_stream_profiles.source_url_override, are hidden from
readers, but they still hold the full address in this release;
the Core cutover that removes it is not in v1.5.6. Migration
20261117060000enqueues a scrub job that cleans existing rows in
primary-key batches and repeats until nothing is pending; it waits
while a Core node on an older release is connected.
StreamSecretScrubWorker.pending()reports the two relay columns
separately, asat_rest_until_stage_3. Core no longer reads a codec
hint from a stream URL query; current camera plugins send it
explicitly. Existingocsf_eventsrows that hold raw camera event
payloads are not scrubbed; new rows are reduced, and old ones expire
with event retention (#6746). - Command records no longer keep stream secrets.
platform.agent_commandsstored camera stream URLs and relay lease
tokens in clear text, and an agent's raw error could quote the URL.
Command rows are now stored redacted while the agent still receives
the real command: declared keys are replaced, any key ending in
token,secret,password,passwdorpassphraseis redacted,
every URL loses userinfo, query and fragment, and free text any agent
returns is scrubbed. Migration20261117015331enqueues a job that
cleans existing rows with the same rules; rows a Core pod still on
the old release writes after the job completes leave with the
command table's 2-day retention. AWX launch and variable-schema
inputs that contain a URL with a username or password are now
rejected (#6703). - Upgrade agents and camera plugins before the next release.
v1.5.6 agents can derive a camera stream address at stream-open time
from the plugin assignment that discovered the camera, and advertise
thecamera_source_resolutioncapability; UniFi Protect camera 0.1.5
and Axis camera 0.1.4, built on serviceradar-sdk-go v2.3.1, answer
those requests. Nothing sends resolver requests in this release, so
camera streaming behaves as before. The next release's Core cutover
depends on them: it requires agents and camera plugins from v1.5.6
or later, and agents that have not been upgraded will stop streaming
at that point (#6755, #6756). - Syslog is stored under the sender's time, at microsecond
precision. EventWriter could not read the fractional Unix seconds
the log collector sends, so every syslog row was stored under the
time it was processed. Rows are now stored under the time the sender
wrote, with the receive time inobserved_timestamp. An implausible
sender clock falls back to the receive time and keeps the sender's
time in_syslog_original_timestampwith
_syslog_timestamp_fallback=true: RFC 3164 keeps its one-hour limit
either way, and zoned formats (RFC 5424 and others) fall back when
the sender's time is more than 24 hours ahead or more than 30 days
behind. Logtimestampandobserved_timestampkeep microseconds
instead of whole seconds on the syslog JSON, collector-normalized and
OTLP paths. During the rolling upgrade, a message redelivered from an
old core pod to a new one can be stored twice with times that differ
below one second (#6740, #6743, #6752). - The log collector follows the Zen ruleset in datasvc;
collector-side normalization ships off. Withkv.enabled, the
chart renders a[rules.datasvc]section inlog-collector.toml, and
the collector fetches and watches
zen/rulesets/<stream>/<subject>.json, keeping the last valid
ruleset. For now it only logs status changes. Compose gains the same
section and grants the log collector a datasvcreaderrole limited
tozen/rulesets/; if you maintain your own datasvc configuration,
grant the same (#6729, #6733). Normalization in the collector
([rules] normalize = trueorLOG_COLLECTOR_NORMALIZE_SYSLOG=true)
is off by default; the OpenSpec change still asks for EventWriter
memory and flush time at the largest batch to be measured before it
is enabled (#6740, #6757). The log collector may now publish
logs.normalized.>under the managed-component and edge-onboarding
permissions, and core and the platform identities are denied it in
the Go NATS bootstrap and the packagednats-server.confand
nats-cloud.conf(#6740). - netprobe degrades instead of crash-looping on a missing capture
interface. Acapture_interfacesname that does not exist on the
host no longer stops netprobe: it captures on the interfaces present,
or runs attribution-only if none are, and the add-on reports
degradedwith the missing and available interfaces. Correct the
assignment'scapture_interfacesfor such hosts; a fleet-wide
assignment can reach hosts whose NICs are bridges or bonds. The
netprobe unit now allows five failed starts in five minutes
(StartLimitBurst=5,StartLimitIntervalSec=300). After that the
add-on reportscircuit_openand the agent does not reinstall the
same artifact and config, so fix the cause and change the assignment
(new version or config). The netprobe side is in add-on 0.2.76; the
degradedandcircuit_openreporting is in the v1.5.6 agent. Core
now countsdegradedadd-ons as active for rollout readiness
(#6741). - NVR recording: new records and pages, background workers still to
come. NVR stays disabled by default. This release adds the
recording catalog and gap records, recording holds, append-only
dispositions and access audit, per-camera recording policies,
per-recorder storage settings, block reconciliation state and unified
timeline playback. Saving a continuous recording policy, recorder
storage settings, or retention and capacity settings stores and
validates them; the workers that run continuous capture, push
storage settings to NVR members, reconcile recorder inventory, evict
local blocks and delete expired or reaped archive objects are not in
this release, and the settings pages say so. Timeline playback reads
blocks through the NVR local read listener or the archive reader and
returns503 recording_read_unavailablewhen neither is configured.
Placing or releasing a recording hold needs the new
devices.cameras.recordings.holdpermission, which only the admin
role holds by default (#6724, #6730, #6737, #6742, #6748). - Hosted storage plans. Hosted deployments accept a storage
entitlement from the control plane that bounds recording retention,
object budget and camera count, and archive credentials the control
plane delivers are read-only and shown as "Managed by ServiceRadar
Cloud". Until the first entitlement arrives, a hosted deployment
holds archive uploads while local recording continues. Self-hosted
installs refuse the entitlement API. Migration20261117173000adds
the entitlement and daily usage report tables, and adds
managed_by(existing rows becomeoperator) and
credential_generationcolumns, a check constraint and an index to
platform.network_credential_secrets(#6760). - Edge late admission: bounds and grants, still inactive. Migration
20261116210000adds the global-only edge pipeline setting
late_admission_max_age_seconds(code default 14 days; not a Helm
value) and the singletonplatform.edge_ledger_coverage, seeded at
now() - 30 days. It sets a 5 s lock timeout, alters only the small
settings table and inserts one row. The ledger retention worker keeps
the ledger for at least the late window plus 5 minutes (about 30
days with defaults, as before), and now prunes ended intervals from
platform.edge_authority_historyolder than that window, never open
ones or ones that endedcertificate_revokedor
revoked_cause_unknown. Core can issue late-admission grants, the
gateway has theIssueEdgeLateAdmissionGrantsRPC, and the trust
document lists terminal authority intervals; migration
20261117080000adds a partial index for them, with a 5 s lock
timeout. The gateway still does not advertise late admission, so
late records stay on the agent's disk as in v1.5.5 (#6726, #6734,
#6753). - New migrations:
20261116203000,20261116210000,
20261116232438,20261117015331,20261117021015,
20261117050200,20261117060000,20261117073000,
20261117080000,20261117173000. Helm
core.migrations.expectedVersionis20261117173000.
Whats New
Helm and NATS
- Runtime certificate rotations are recorded and adopted
infrastructure stays admitted; the flow-collector bootstrap hook runs
after migrations with a 600 s deadline (#6732). - Disabled-feature NATS identities deny everything, and the chart
refuses an empty allow list without a>deny (#6728).
Camera and NVR
- Camera relays report why a source failed to start. Agents classify
the failure (for exampleconnect_timeout,authentication_failed,
no_video_track,start_deadline_exceeded) with a short sanitized
detail in a new optionalfailure_reasonfield, core-elx validates
it, and web-ng shows it on the relay viewer, the device camera panel
and the relay list. An operator stop or an agent shutdown is not
reported as a failure, and older agents close as before (#6703). - Command records are stored redacted and existing rows are scrubbed
(#6703). - Plugin results are reduced on entry so stream addresses stay out of
Core tables except the two relay columns, and a stage-aware job
scrubs existing rows. web-ng no longer keeps stream addresses in
device page state or in the ServiceRadar Atlas stream payload
(#6746). - A capture that Core aborts before the end marker, for example on a
backwards timestamp, now fails definitely and releases the recorder
claim at once, instead of reporting an uncertain outcome and locking
the camera for the 30 s lease (#6725). - Recording catalog and gap records; retention usage counts every
archive binding that shares an endpoint and bucket and reports other
buckets separately; the Cameras tab shows cataloged copy states
(#6724). - Recording holds, append-only dispositions and held-byte figures on
the Retention and Cameras tabs (#6730). - Per-camera recording policies with a what-if preview, per-recorder
storage settings, and live Policy and Health tabs (#6737). - Historical block freeze, rotated-block adoption and reservation
resolution in Core; audited operator dispositions for quarantined
blocks and missing archive copies on the Health tab; the assigned NVR
member is picked from the configured members (#6742). - Unified timeline playback and holds on
/cameras/:camera_source_id/recordings, a timeline API under
/api/cameras/:camera_source_id/timelinewith multi-block HLS
playlists, digest-checked reads and bounded read admission, and an
append-only recording access audit (#6748). - Hosted storage entitlements, closed-day storage usage reports,
control-plane delivered archive credentials and a held-object
manifest; camera recording object storage gauges on/metrics;
date and time pickers, presets and a jump form for timeline holds,
and an empty state when a window has no playable blocks (#6760).
Agent
camera.open_relayaccepts asource_resolverand the agent derives
the stream address from the discovering plugin, keeping it only in
the reader's copy; new failure classessource_resolver_unavailable,
source_resolver_unsupportedandsource_not_found(#6755).- A leased sweep cut off by its collection window spools the hosts it
completed, then reports the slot incomplete, instead of losing them;
the scan stops a short flush reserve before the window closes
(#6731). A scan stopped by that reserve keeps waiting for spool room
until the window closes, so the batch and buffered hosts are kept
(#6736). - The agent reports netprobe
degradedand systemd add-on
circuit_openstates, and stops reinstalling an unchanged add-on
whose unit hit its start limit (#6741).
Plugins and add-ons
- UniFi Protect camera 0.1.5 and Axis camera 0.1.4 answer stream
source resolution requests without creating or enabling a stream on
the controller or camera, and can omit stream addresses from
inventory results; inventory output is unchanged unless Core asks
for that (#6756). - netprobe 0.2.76 captures on the interfaces present and reports
degradedinstead of exiting (#6741). - RDP adapter 0.1.17 with
libz-sys1.1.33 (#6754).
Edge
- Late admission bounds:
late_admission_max_age_seconds, a ledger
coverage watermark and authority-history pruning (#6726). - Core late-admission grants and the gateway
IssueEdgeLateAdmissionGrantsRPC; not called by agents yet (#6734). - The trust document lists terminal authority intervals, capped at
100,000 in the gateway's global document, which fails closed above
it; nothing consults the list yet (#6753). - The gateway records the agent's edge spool health from every
control-stream hello, including the 60-second heartbeat, not only the
first. Core'sedge-agent-spoolhealth no longer keeps reporting a
backlog that has already drained until the control stream reconnects
(#6762). - The gateway logs
edge record stream ended before lane_open_ackand
emits[:serviceradar, :agent_gateway, :edge_record, :lane_open_unacknowledged]with the agent, lane, wait and step when
an edge record stream ends before its ack. The peer certificate is
now checked when the stream opens, before the first message is read
(#6759).
Logs
- The log collector fetches and watches its Zen ruleset in datasvc
(#6729), and backs off up to 30 s when the watch keeps ending (#6733). - Collector-side syslog normalization publishes OTLP batches on
logs.normalized.syslog, and falls back to GELF onlogs.syslog
until a ruleset is active; off by default (#6740). - Syslog is stored under the sender's time with a wrong-clock fallback
for every format (#6743), at microsecond precision (#6752).
Core
- Hosted repair no longer stops on one device whose merge history is
too deep: owners come from the canonical resolver, a link whose owner
cannot be settled is kept, logged and counted (edges_unresolved,
guests_unresolved), and stale hosted links are retired again
(#6735). - EventWriter routes on the subject the broker delivered (#6727).
Dependencies
zen-engine2.1.7, which moveszipfrom 8.6.0 to 9.6.1
(RUSTSEC-2026-0545, RUSTSEC-2026-0548); three unmaintained-crate
notices with no fix are accepted in the RustSec audit with review
dates (#6754).
Docs
- A Camera Recording and NVR operator guide covering the recording
settings tabs, timeline playback, holds and permissions (#6751).
Specifications
- OpenSpec: NVR retention review findings on re-archive, loss
acceptance and object usage closed (#6744); test, benchmark and Loom
tasks for the log ingestion changes (#6749); catch-up re-runs for
interrupted sweep lease slots proposed (#6761). New changes for the
certificate rotation fix (#6732) and the netprobe capture interface
fix (#6741).
Tests and CI
- Hegel property tests for the collector normalization path (#6757).
- Generated Ash migrations are formatted with the project's
mix format; 21 committed migrations change only in line breaks, with no
version changes (#6758). - Agent-gateway tests own the gateway config they need (#6745), and
the rolling-upgrade tests connect the capable core peer inside the
wait window (#6747).