github LanternOps/breeze v0.123.0
on GitHub

6 hours ago

Breeze RMM v0.123.0: agents that get past ISP security filters, ticket types, and customer work approval. Also in this release: a remote viewer keyboard mode, EDR detections and response actions for framework vendors, workload host inventory, observed-location suggestions on the network map, Run Script on device groups, and Windows System Restore policy (server side).

⚠️ The Update Ring backfill can add rings, and it stops the upgrade on two conditions. 2026-12-26-230000-ring-app-rules-backfill.sql moves per-app block/pin rules from configuration-policy patch links onto Update Rings. To keep each policy's effective rules it may create rings named <policy> — app rules and clone rings named <ring> — <policy>. It aborts the migration if a rule it would move has a non-ASCII package id, or if a ring already carries different non-empty rules. Details under Database (#8186, #8655).

⚠️ Treat AGENT_ECH_ENABLED=false as a last resort. It is the server-wide off switch for the new agent Encrypted Client Hello. A device that can only reach the API over ECH (one behind an SNI-filtering ISP gateway) goes offline when it is set to false, and it cannot receive the setting back until it is on an unfiltered network (#8869, #8882, follow-up #8881).

⚠️ Compose now passes about 25 more variables to the API. A value already in your .env that Compose used to ignore now takes effect, including CF_ACCESS_TRUST_ENABLED, M365_ENABLED and the MCP_LLM_* group. A stale CF_ACCESS_TRUST_ENABLED=true without its team domain and AUD, or a malformed MCP_LLM_* value, refuses boot. Check .env for stale entries before upgrading (#8468, #8402).

Security — hardening

This release includes security hardening across:

  • credential headers and credential query parameters, now redacted in every Caddy log (docker/Caddyfile.prod);
  • Windows MSI downloads, verified against the signed release manifest on both BINARY_SOURCE=github and BINARY_SOURCE=local;
  • Network Proxy session revocation: closing a tunnel cancels its in-flight requests, and a ticket exchange that races Close gets no cookie;
  • automation and script targets: unknown target types, foreign group ids and partner-wide group targets are refused instead of reaching every device in the org, and device filters refuse values outside an enum instead of widening the match;
  • MCP device resolution and the trashed-ticket subject in ticket links;
  • PAM elevation chains across device org moves and merges, and recovery of ledger entries after a failed apply;
  • EDR credential changes that would move a connection to a different vendor console;
  • OAuth token-endpoint error attribution, SSO re-auth failure logging (a nonce mismatch is now logged on its own), and the audit event for redeemed installer bootstrap tokens;
  • recovery console codes, which must be re-entered after a refusal;
  • dependency updates in the helper and viewer (rustls, quinn-proto, serde_yaml).

Self-hosters are encouraged to upgrade. (#8826, #8527, #8458, #8632, #8981, #8802, #8749, #8514, #8552, #8554, #8646, #8753, #8676, #8902, #8652, #8830, #8974, #8454, #8470, #8824, #8559, #8494, #8899)

Summary

  • Agents behind ISP security filters can connect (#8869, #8882, #8949). Some ISP gateway filters, CUJO-based ones such as Comcast xFi Advanced Security and Spectrum Security Shield, match the Breeze hostname in the TLS SNI and answer with a plaintext redirect. The agent then logs tls: first record does not look like a TLS handshake and never connects. With this release's agent, a device dials plain TLS as before. Only after it sees that filter twice in 10 minutes does it retry with the hostname encrypted (Encrypted Client Hello). Needs this release's agent, and a server whose DNS publishes an ECH config (see Upgrade Notes).
  • Ticket types and related tickets (#8654, #8682, #8708, #8675). Tickets are Incident, Service Request, Problem or Change, and each type has its own Service Desk entry. A Related tickets panel links tickets to each other, for example incidents to the problem behind them. The portal shows the type, and the AI ticket tool can set it and manage links.
  • Customer work approval (#8537, #8555, #8562, #8574). Optional and off by default. Over-budget or after-hours time is held until the customer approves it in the portal, or a technician records their decision. Held time is never invoiced, drawn from a block or reported as billed.
  • Remote viewer (#8635, #8553, #8543, #8523, #8547). Keyboard mode: Type what I see (default) or Physical keys (needs this release's agent), with IME and dead-key typing. Also view-only, block local input, screenshots, Fit or 100% scaling, a connection stats overlay, and per-device preferences.
  • EDR provider framework (#8653, #8666, #8757). Detections from framework vendors appear in the incidents feed, on the Security page and on the device page, with coverage reporting so a partial sync never reads as "no threats". Response actions (isolate, release, scan) for GravityZone-managed endpoints are in the API and approval-gated AI, off per connection by default.
  • Workload host inventory (#8417, #8545, #8519, #8548, #8558). With policy turned on, agents report Docker, Podman, Hyper-V and Proxmox workloads (containerd is detect-only). A container image that falls behind its public registry counts toward patch compliance and has its own monitor kind.
  • Where a device actually is (#8433, #8589, #8659). The network map tells a device's assigned site apart from the network it is seen on, shows cross-site visitors, and suggests a site move when a device lives somewhere else. Breeze never moves a device by itself.
  • Windows System Restore policy (#4609, #4751). A new System Protection policy tab sets when Breeze requests a restore point, and patch history, scripts and software deployments show the outcome. The agent side is not in this release; until it ships, every attempt reads Awaiting the device.
  • Run Script on device groups (#8632, #8670, #8683). Run a script on saved device groups from the Run Script modal, from org-owned automations and from the AI run_script tool. Membership is resolved before approval and never truncated.
  • Trust Center, first release (#8776, #8780, #8783). Each organization gets a Trust Center tab that publishes a public, customer-authorized security-posture page built from Breeze telemetry, refreshed hourly, with documents and an embeddable badge. Publishing needs MFA. This release is the publishable page and editor; declared controls, gated documents, framework views and distribution options follow in later releases.
  • Steadier under load (#8456, #8578, #8850, #8851, #8842). Agent admission control, observe-only by default. Cross-instance cache invalidation, agent request bodies received before a database connection is held, agent sockets that stay open when the database can't answer, and Prometheus alerts for API saturation.

Added

  • Encrypted Client Hello for agent control-plane TLS. Any ECH failure falls back to plain TLS inside the same dial, and repeated failures put the host in a one-hour cooldown. enroll, bootstrap and support retry with ECH on the first filter signature, so installs work on a filtered network. A device opts out with ech: false in agent.yaml. The device's control_plane health component reports the filter (#8869, #8882, #8949).
  • Ticket types: tickets.ticket_type (default incident; planned work defaults to service_request), a ?ticketType= list filter, a type chip and selector in the queue and workbench, Service Desk nav entries per type, and the field on the Partner API and Office add-in (additive) (#8654, #8682). Ticket links with link types, cycle and duplicate rules, and a per-ticket cap (#8675, #8682). The portal shows the type as a label; customers cannot choose it (#8708).
  • Customer work approval: ticket budgets, after-hours work types, an approvals panel and Record decision dialog for staff, a portal Approvals page for billing contacts with a pending-count badge, customer emails through three new partner-editable templates, and APPROVAL_REQUIRED handling in the Outlook add-in, mobile app and AI tool (#8537, #8555, #8562, #8574).
  • Remote viewer W05/W06: keyboard mode and IME composition, Keyboard Lock in fullscreen, view-only, block local input, screenshot copy/save, Fit / 100% scaling, stats overlay, per-device preferences. On macOS, Cmd+H/M/W reach the remote, and Cmd+Q asks first while sessions are open (#8635, #8553). The agent gains a reliable input channel and wider key and pointer coverage: scancodes, extended keys, back/forward, horizontal wheel and lock-key sync (#8543, #8523, #8547).
  • EDR provider framework: an EDR providers sub-tab under Integrations → Security for connections, tenant mapping and unlinked endpoints. A third Security tab lists detections, with "Promote to incident". A read-only device section, read-only AI tools (get_edr_status, list_edr_detections), alerts for linked detections, and 180-day retention for closed detections. Customer portal toggle enableEdrVisibility (#8653, #8666). Response actions through dispatchEdrAction and approval-gated AI action tools (#8757).
  • Workload host inventory: a Workloads device tab, a workload_inventory policy tab, fleet filters hostsWorkloads and workloadRuntimes, and the AI tool query_device_workloads. Image currency is checked against public registries every 30 minutes. A new container_image_currency monitor kind and a built-in container_image_out_of_date monitor are added to each partner, not attached to any policy (#8417, #8545, #8519, #8548, #8558).
  • Topology site location: declared vs undeclared networks, an "Other / unidentified networks" roster, visitor tiles, daily presence summaries, wrong-site move suggestions, and Keep at confirmation. Moves are guarded against concurrent edits (409 device_site_changed) and audited as device.site_move (#8433, #8589, #8659).
  • System Protection policy tab (also on partner-wide "All orgs" policies), a per-device restore-point attempts ledger, outcome badges, and a System Restore tab on Windows devices (#4609, #4751, #8546, #8551, #8557, #8561, #8572).
  • Device-group targets for Run Script (target: { type: 'groups', groupIds }, 1–100 groups, at most 500 resolved devices), org-owned automations, and the AI run_script tool (groupId; at most 10 devices) (#8632, #8670, #8683).
  • Trust Center (#8776, #8780, #8783). An organization's Trust Center tab publishes a public, customer-authorized security posture page at /portal/trust/<partner>/<org>, built from Breeze telemetry, with an hourly refresh, documents and a badge snippet. Publishing needs MFA.
  • Invoice labor detail: a partner setting under Billing settings → Documents for per-entry or per-day itemization, work date, note and worked time, with a per-draft override that is frozen at issue. Refresh from time entries on draft invoices, the work date sent to accounting sync, and an AI update_time_entry action (#8627, #8650, #8669, #8684, #8709).
  • Block hours: hour_block contract lines with a live period balance and period history. A customer portal Support hours card sits behind its own toggle, which Enable all does not turn on. Threshold alerts (#8438, #8517).
  • Watchdog install retry: an agent whose watchdog could not be installed now installs it itself, with bounded, persisted retries. breeze-agent service install gains --watchdog (clear the opt-out) and --strict-watchdog (exit 78 on a watchdog failure), and prints BREEZE_WATCHDOG_INSTALL=deferred reason=<class> when it defers. The device Info tab shows a banner when the agent gives up (#8647, #8667, #8685).
  • Log-shipping health: a log-shipping line on device detail, the logShippingFailing fleet filter and quick-filter chip, a stats count, and a "No logs received since the log level was raised" check (#8628, #8644, #8662, #8681, #8707).
  • Connection health: when Breeze auto-revokes an OAuth grant after refresh-token reuse and the app never re-authorizes, the grant's owner and the partner's connected-app admins get an in-app notice within about 15 minutes. The notice clears itself once the app is re-authorized (#8871).
  • Heartbeat retain-state: agents that support it keep their current PAM, helper and remote-management state on a beat where the server could not decide that feature, instead of being sent "off" (#8645, #8660).
  • Customer portal read APIs behind new sensitive toggles, off by default and not set by Enable all: patch detail (enablePatchDetail) and vulnerability detail (enableVulnerabilityDetail). There are no portal pages for them yet (#8544, #8550).
  • Patching: custom restart-prompt title, message (with {{countdown}}) and button label per patch policy (#8440). Per-app approval rules now live on the Update Ring: the ring form gains an Application rules section, and the policy Patch tab points to the ring (#8655, #8674).
  • Discovery: pin a profile's scans to one agent; unpinned selection prefers agents on the profile's subnets (#8422).
  • Vulnerabilities: the By-software table shows the version to update to (#8481).
  • Operator recipe library: one four-eyes approval authorizes an ordered, digest-pinned set of provider effects (#8542).
  • Smaller additions: acknowledge, resolve and suppress from a device's alert history (#8452); a per-script latest-result view on the device Scripts tab (#8424); a staff-invite email template with partner overrides (#8426); responsible contacts on site and device pages (#8785); service principal scope presets with per-scope descriptions (#8524); an AI fleet device stats tool (#8525); Information-level MsiInstaller events collected on Windows, matchable by event_log rules at level info (#8431); DNS servers per adapter in IP history (#8425); native ZFS encryption detection on Linux (#8419); more RMM and remote-access signatures in Management Posture (OpenFrame, SimpleHelp, MeshCentral, TightVNC, RealVNC, UltraVNC, Splashtop) (#8513); typed error codes on agent command results (#8473); the session clipboard policy on VNC tunnel responses (#8414).
  • Mobile app: arrival suggestions to start a timer at a pinned client site, foreground only. Off unless the partner enables Location suggestions and the technician opts in. Ships with the next mobile app build (#8416).
  • Groundwork, not yet usable from the UI: ticket migration windows and the import writer for the coming ticket importers (#8486, #8782, #8884, #8779); caller verification, dark behind CALLER_VERIFICATION_ENABLED (#8560, #8576, #8657); one-way endpoint messages, hidden behind PUBLIC_ENABLE_ENDPOINT_MESSAGING (#8752, #8983); the security agents policy type, with nothing enableable yet (#8754); Network Proxy dedicated tunnel origin, off by default (#8458); the service catalog tables (#8784); the backup destination key root, with no change to existing data (#8756).

Improved

  • Database work under load: agent request bodies are received in full before the org database context opens, so a stalled upload no longer holds a pooled connection; it gets 408 (#8850). Agent WebSocket credential re-checks keep the socket open for up to 120 s when the database cannot answer, instead of closing hundreds of healthy sockets with 4001 at once (#8851).
  • Agent admission control in observe mode: under event-loop or database pressure it counts the low-priority agent telemetry and WebSocket upgrades it would refuse (breeze_admission_* metrics) (#8456).
  • Several API instances: a write on one instance now drops the matching hot-path cache entry on every instance within about a second through Redis pub/sub, instead of waiting out the 60 s / 5 min TTL (#8578).
  • Monitoring: Prometheus alert rules for database pool waiters, acquire timeouts, event-loop lag and admission shedding in monitoring/rules/breeze-rules.yml (#8842); a capacity KPI dashboard and perf-gate tooling (#8588).
  • Nightly audit-chain verification uses the org indexes; on large installs it ran for hours (#8536). Metric anomaly detection no longer times out on stale partition statistics (#8539).
  • Backup: snapshot browsing loads one directory at a time with paging (#8245), and snapshot files upload through a bounded worker pool (#8421). Bare-metal recovery readiness now uses real rebuild results (#8516).
  • PAM ownership epochs: elevation requests and actuations are anchored to the device's ownership epoch. Device org moves and merges keep the PAM chain correct (#8514, #8552, #8554, #8646, #8753).
  • LLM egress audit: raw rows older than 90 days fold into per-org daily rollups (#8664).
  • Topology: unbound stub nodes (LLDP/CDP/MAC endpoints) age out after 30 days without activity (#8465). Decommissioned devices are hidden (#8450).
  • Session Timeout goes up to 12 hours at partner and org level, with a pointer from the Users page (#8497).
  • Tier-1 indexes for discovery jobs, organization users and the automation sweep (#8476). UniFi collectors are no longer polled every 30 s when none are assigned (#8412).

Fixed

  • Remote desktop stuck on "Connecting" after a double offer between the viewer and the API (#8948).
  • Dynamic groups re-evaluate when a device's tags or custom fields change; before, they stayed stale until someone re-saved the group. The rule builder now offers custom fields (#8903, #8887).
  • Device filters: the device-group filter, tag "has any / has all" and one-item selections no longer return 500 (#8773). Tag-targeted compliance policies evaluate again, observe-only (#8895).
  • Agents on Windows: a fallback when Windows cannot rename agent.state (including pre-1607 builds) (#8859, #8886, #8717); UTF-16 PowerShell stderr in time sync errors decodes (#8829); self-written .ps1 temp files get a BOM (#8460).
  • Agents on Linux and macOS: the Linux legacy-helper cleanup no longer deletes the current helper (#8556). The macOS .pkg enrolls and updates with the agent it installed (#8781). Breeze Assist on macOS is menu-bar only, and Cmd+Q hides it instead of quitting (#8564). Assist keeps supplementary groups when spawned (#8563).
  • Agent logs with NUL bytes no longer fail the whole insert (#8825). A failed identity restart-marker write is retried (#8777, #8384).
  • SSO re-auth with Microsoft Entra accepts Entra's already-expired re-auth id_token (within 5 minutes), and failures stay on the profile page (#8974).
  • Patching: scheduled patch runs honour the assignment's role filter (#8461); the compliance view counts failed installs as failed, not pending approval (#8480).
  • DNS filtering: domains left unsynced by an earlier release reach the provider on that policy's next sync (#8447).
  • Microsoft 365: the actions manifest uses application permission IDs (#8651); Retest keeps the stored manifest version (#8750); mailbox poll failures are surfaced (#8400).
  • Devices: the search box matches the last logged-in user (#8720); a held reboot is shown, and an operator can mark a stuck command failed (#8508); at most one secondary status pill per row (#8521).
  • AI: network_discovery sends the subnets and methods the agent reads (#8801); agent runs on the fallback provider record failover provenance (#8463); action-intent delivery retries with backoff (#8459); a hand-added model can be removed (#8515).
  • Billing: catalog items can be recurring, so quotes price monthly services correctly (#8722).
  • Policies: partner users with several orgs can create org-owned peripheral policies (#8721); a value typed into a policy number field before it mounts is kept (#8892); a partner-owned policy's Monitors tab lists only attachable monitors (#8435).
  • Automations: the on-failure notification is sent once across replays (#8462); script.completed / script.failed fire after commit (#8420).
  • Smaller fixes: clearing all exit-code severity rows persists (#8626); monitor conversion preview and refresh (#8531, #8477); a failed security collection stores provider unknown (#8565); one filesystem analysis scan at a time per device, and auto scans honour the remote-tools policy (#8518, #8522); human alert resolves record manual (#8530); the audit log export honours the list filters (#8436); the org switcher refreshes after a rename or archive (#8428, #8439); settings search uses the settings registry (#8437); vault commands reach the backup helper (#8413); ticket.status_changed fires when assignment moves a ticket new→open (#8430); sync workers' final-attempt check (#8427); org moves re-point time entry sites (#8444); device placement conflicts return 409 (#8453); Network Proxy "Open In New Tab" (#8408); the watchdog-install banner survives restarts (#8686); the Breeze User-Agent on software installer downloads (#8712); pnpm dev on Linux/WSL (#8723); missing device idle translations (#8395).
  • From the pre-release sweep (#8981): the sidebar ticket-type entries re-filter the queue in place; "Verify caller" in the split view; portal approvals show the customer-facing ticket number, and the after-hours window in the viewer's time zone; Run Script keeps a partial-admission result on screen; Workloads policy-tab copy; the watchdog banner command renders as code; the block-hours period history loads; EDR sync interval bounds match the database (5–1440) instead of returning 500; the service principal create dialog scrolls.

Self-Hosting / Upgrade Notes

Upgrade command. Check how your install pins its images: grep '^BREEZE_API_IMAGE_REF=' .env.

  • Digest-pinned (value ends in @sha256:…, every guided-setup.sh install from v0.112.0): fetch the current guided-setup.sh from main, then run bash guided-setup.sh --upgrade 0.123.0. It verifies the signed image inventory, updates BREEZE_VERSION and all four image digests, then pulls and restarts.
  • Tag-form (value ends in :${BREEZE_VERSION}): set BREEZE_VERSION=0.123.0, then run docker compose pull api web portal && docker compose up -d (and pnpm install if you build from source).
  • Do not edit only BREEZE_VERSION on a digest-pinned install. It re-pulls the images you already run.
  • --upgrade does not change your docker-compose.yml. To set any of this release's new optional variables, add their lines to the api service's environment: block (or take the v0.123.0 docker-compose.yml / deploy/docker-compose.prod.yml, which already map them). A value only in .env does not reach the API.
  • See the Upgrade Guide.

No breaking changes for a stock install. No default was flipped for existing installs. Four changes are stricter than before and can surface on an install with pre-existing bad data or a custom setup, so read them before upgrading:

  • The Update Ring app-rules backfill stops the API from booting if a rule it would move has a non-ASCII package id or a ring already carries different non-empty rules (details under Database).
  • A patch policy linked to an invalid or cross-partner Update Ring now denies every patch instead of falling through (#8655).
  • Windows MSI downloads on BINARY_SOURCE=local without a staged signed manifest, or on any install without RELEASE_ARTIFACT_MANIFEST_PUBLIC_KEYS, now return 503 instead of an unverified file (#8527).
  • A device filter value outside its enum (for example osType notEquals "Windows") now fails with 400 instead of being dropped, so a stored script target or dynamic group rule with a bad value fails visibly rather than widening to every device (#8981).

Behaviour changes and feature flags.

  • Agent admission control ships observe-only (#8147, #8456). BREEZE_ADMISSION_MODE (off | observe | enforce) defaults to observe; an empty value means observe, and an invalid one falls back to it with a startup warning. Both Compose files pass it as ${BREEZE_ADMISSION_MODE:-}, so a stock deploy only measures and counts. Nothing is refused until you set enforce. Then low-priority agent telemetry gets 503 + Retry-After and agent WebSocket upgrades are paced. Heartbeats and recovery traffic are never refused. The tuning knobs are documented, commented out, in .env.example.
  • Agent Encrypted Client Hello (#8869, #8882). AGENT_ECH_ENABLED defaults to true and reaches agents as configUpdate.ech_enabled. Setting it to false is a last resort, never a casual flip: live connections stay up, but the next dial is plain, so every device that can only connect over ECH goes offline and cannot receive the restore until it is on an unfiltered network (#8881 tracks an expiring off switch). ECH engages only when your API hostname's DNS publishes an ech= HTTPS record (for example a Cloudflare-proxied hostname). A self-hosted edge without one never engages ECH; the agent stays plain and the device's control_plane health component reports the filter. Not covered: HTTP-proxied agent connections, and the install.sh / PowerShell installer downloads (#8874).
  • Heartbeat retain-state (#8616). HEARTBEAT_RETAIN_STATE_ENABLED defaults to true. false sends every agent the legacy heartbeat payload, as an older API would.
  • Endpoint messaging is hidden (#8752, #8983). The three Send message entry points (Devices bulk action, organization header, site header) sit behind the web build-time flag PUBLIC_ENABLE_ENDPOINT_MESSAGING, default off. The published web image is built with it off. The server side also needs a per-device rollout flag (endpointMessaging.enabled, default off). The endpoint_messages:send and endpoint_messages:read permissions are granted on upgrade to Org Admin, Org Technician and Partner Technician.
  • Caller verification is dark (#6356, #6357, #6358). Every route answers 404 and every web surface is hidden unless CALLER_VERIFICATION_ENABLED=true (default false, already mapped in both Compose files). Enforcement is not in this release.
  • EDR response actions (#8757). edr_connections.actions_enabled defaults to off for every connection, and only a partner admin with MFA can turn it on (API only in this release). BREEZE_EDR_ACTIONS_ENABLED (default true) is the server-wide off switch.
  • Off by default: customer work approval, workload inventory enumeration (workload_inventory.enabled, plus hyperv_enabled / proxmox_enabled), the portal toggles enableEdrVisibility, enableHourBlock, enablePatchDetail and enableVulnerabilityDetail (only enableEdrVisibility is included in Enable all), the Network Proxy dedicated tunnel origin (TUNNEL_ORIGIN_TEMPLATE unset = path mode), and mobile Location suggestions.
  • Update Ring approval rules (#8186, #8655, #8674). Per-app rules now come from the linked Update Ring only. A policy that links an invalid or cross-partner ring now fails closed: every patch candidate is denied with ring_reference_invalid, and the Patch tab offers a repair prompt. A policy with no ring and a manual approval stays approved. Saving a changed apps list, or a newly set autoApprove: true, on a policy patch link returns 400 APPROVAL_RULES_MOVED_TO_RING; reload a stale tab.
  • Device filters refuse out-of-enum values (#8981). A filter value outside a database enum now returns 400, naming the allowed set, instead of being dropped. Before, notEquals "Windows" could widen a target to every device. A stored deployment target or dynamic group with such a value now fails visibly.
  • Automation targets are validated (#8632). Create and update reject an unknown target type, empty group or device lists, and ids from another org. Partner-wide automations reject groups and devices targets. A stored broken target makes the run failed with the reason in its log, instead of falling through to every device in the org.
  • Windows MSI downloads are verified on every binary source (#8527). With BINARY_SOURCE=local, the MSI is checked against the signed manifest pair at the binaries volume root and RELEASE_ARTIFACT_MANIFEST_PUBLIC_KEYS. A volume with no staged manifest pair, a manifest older than v0.105.0, or no RELEASE_ARTIFACT_MANIFEST_PUBLIC_KEYS now gets 503 MSI not available on Windows installer downloads. BINARY_SOURCE=github without a manifest key also refuses instead of serving an unverified MSI.
  • Patch compliance counts out-of-date container images (#8558). This applies only on devices with workload inventory on: a stale image counts as pending, a current one as installed, and unchecked images never count. Severity, source and ring views exclude the term.
  • Tag-targeted compliance policies evaluate, observe-only (#8895). They write compliance rows but dispatch no remediation and publish no policy.* events, and their stale alerts close.
  • DNS filtering (#8447). Domains left unsynced by an earlier release now reach the provider on that policy's next sync; you no longer need to remove and re-add them. Duplicate-add behaviour is verified only for AdGuard Home.
  • Security & Compliance Posture report (#8776). Coverage percentages now use assessed devices as the denominator, decommissioned devices are excluded, the PDF prints · N not assessed, and a control shows grey when more than 20% of the fleet was not assessed. Expect larger not-assessed counts on fleets with offline devices.
  • Breeze Viewer (#8635). In the default Type what I see mode the viewer no longer sends physical key codes; the remote resolves keys through its own layout. Choose Physical keys for numpad and right-modifier distinctions. The choice is saved per device and becomes the default for devices without one.
  • Session Timeout (#8497). Writes above 720 minutes (12 h) are rejected, and a stored value above 720 reads as 720.
  • API contract changes: GET /backup/snapshots/:id/browse returns one directory level per request (dir, limit, cursor; entries no longer carry children) (#8245). POST /tunnels/:id/http-ticket returns { ticket, url, mode } and rejects a mode other than frame or tab. A request carrying x-breeze-tunnel-site gets 421 unless it is for a tunnel host with the dedicated origin on (#8458). Agent request bodies that stall get 408 (#8850).
  • breeze-agent enroll --site-id is deprecated and ignored; the site comes from the enrollment key. Scripts that pass it keep working and print a warning (#8492).
  • Self-hosted docker/Caddyfile.prod copies: a new global log default { format filter { … } } block redacts credential headers and query parameters in every Caddy log. Carry it over if you maintain your own copy (#8826).
  • Trust Center needs no configuration and no ingress change: pages use the existing /portal rule. If your proxy strips X-Forwarded-Proto / X-Forwarded-Host (Caddy forwards them by default), canonical URLs fall back to the internal origin (#8783).
  • Built-in monitors: each partner gains container_image_out_of_date (low severity, not attached to any policy). A deleted copy is not recreated (#8558).

Database.

  • 51 migrations (idempotent; applied automatically on boot via autoMigrate unless AUTO_MIGRATE=false). Expect [auto-migrate] Applied 51 migration(s). None modified or removed.
  • Update Ring backfill (2026-12-26-230000-ring-app-rules-backfill.sql, #8186). A data migration, not just a normalisation. It moves per-app rules from config_policy_feature_links.inline_settings->'apps' onto patch_policies.app_rules, normalised the way the runtime does.
    • A ring-less link that carries rules gets a new ring named <policy> — app rules, which approves nothing.
    • Where the policies sharing one ring disagree, the ring keeps one set, and each other set gets a clone named <ring> — <policy> with that ring's patch approvals copied. These rings appear in the Update Rings list.
    • Links to an invalid ring are skipped and counted; those policies now fail closed (ring_reference_invalid).
    • It aborts the migration, and the API does not boot, if a rule it would move has a non-ASCII package id, or if a ring already carries different non-empty rules. Check the API start log for ring_app_rules_backfill <metric>=<n> lines.
    • Policy-level auto-approve is not converted. When it relinks links, it briefly locks the affected partners and their orgs.
  • PAM org-move trigger (2026-12-27-170000, #8746). Adds a SECURITY DEFINER function and an AFTER UPDATE OF org_id ON devices trigger that restamps only un-actuated elevation requests, and removes elevation_requests from the generic device-child cascade. Fixes org moves that aborted with 42501.
  • Other backfills over existing rows:
    • 2026-12-23-150000-pam-chain-epoch-anchor.sql (#8514) sets device_epoch on existing elevation_requests and pam_actuations, then validates three foreign keys (a scan of those tables).
    • 2026-12-27-070000-invoice-line-description-edited.sql (#8606) adds invoice_lines.description_edited and, on first application, marks every time-entry line on a draft invoice as edited, so Refresh from time entries never overwrites a description that existed before the upgrade. Issued and void invoices are untouched.
    • 2026-12-27-190000-endpoint-messaging-permissions.sql (#8741) seeds endpoint_messages:send / :read and grants them to Org Admin, Org Technician and Partner Technician.
  • Concurrent index builds on discovery_jobs and organization_users (#8476), tickets, ticket_comments and ticket_external_refs (#8486, #8654), and device_commands (#8707). Each waits for long-running transactions and does not block writes. If one of the discovery_jobs, organization_users or ticket builds is interrupted, the next start refuses with an "INVALID index" message naming the index: run DROP INDEX CONCURRENTLY <name>; and restart. The device_commands build has no such check: after an interrupted start, check idx_device_commands_set_log_level_device_completed for indisvalid = false and drop it if so.
  • metric_rollups partitions get tighter autovacuum-analyze settings (ALTER TABLE … SET, SHARE UPDATE EXCLUSIVE, does not block reads or writes) (#8539). The audit-chain verify functions are replaced in place (#8536).
  • Mostly additive DDL: new tables for restore-point attempts, operator plan approvals, caller verification links, migration windows, topology presence and presence days, ticket links, LLM egress rollups, connection health incidents, EDR actions, trust pages, the service catalog and org evidence holds. New columns on devices, tickets, invoice_lines, invoices, partners, portal_branding, discovery_profiles, software_policies and backup tables. No migration adds a NOT NULL column without a constant default or runs SET NOT NULL on an existing column.
  • One-way enum additions (Postgres cannot drop an enum value): config_feature_type system_protection and security_agents, security_provider unknown, monitor_kind container_image_currency, notification_type integration, and the tickets.ticket_type enum.

Environment variables.

  • Required: none. No new required environment variables. Two new variables are validated strictly: AGENT_ECH_ENABLED and HEARTBEAT_RETAIN_STATE_ENABLED accept only true or false, and any other value refuses boot. A malformed TUNNEL_ORIGIN_TEMPLATE, or one that shares a registrable domain with an app origin, also refuses boot.
  • New, optional (mapped in the api environment of both shipped Compose files unless noted; with a custom Compose file, add them to the api service's environment: block):
    • AGENT_ECH_ENABLED (default true) (#8869).
    • HEARTBEAT_RETAIN_STATE_ENABLED (default true) (#8616).
    • BREEZE_ADMISSION_MODE (default observe), plus BREEZE_ADMISSION_LAG_MS, _DB_WAIT_MS, _EMERGENCY_MS, _BULK_MAX_IN_FLIGHT, _WS_RATE_PER_SEC, _WS_BURST, _WS_MAX_SETUPS (#8456).
    • AGENT_BODY_RECEIVE_TIMEOUT_MS (default 15000) and AGENT_BODY_RECEIVE_TOTAL_TIMEOUT_MS (default 60000) (#8850).
    • BREEZE_EDR_ACTIONS_ENABLED (default true) (#8757).
    • LLM_EGRESS_EVENTS_RETENTION_DAYS (default 90, max 3650), LLM_EGRESS_EVENTS_RETENTION_BATCH_SIZE (5000), LLM_EGRESS_EVENTS_RETENTION_MAX_BATCHES (200) (#8664).
    • TUNNEL_ORIGIN_TEMPLATE and TUNNEL_FRAME_ANCESTOR (unset = path mode) (#8458).
    • MIGRATION_WINDOW_MAX_DAYS (14), MIGRATION_WINDOW_DEFAULT_WRITES_PER_HOUR (6000), MIGRATION_WINDOW_MAX_WRITES_PER_HOUR (20000) (#8486). Mapped in the root docker-compose.yml only, not yet in deploy/docker-compose.prod.yml; add them there if you need them.
    • Web, build time: PUBLIC_ENABLE_ENDPOINT_MESSAGING (default off). It only takes effect in a web image you build yourself (#8983).
  • Newly passed to the API by Compose (#8468), each as ${VAR:-}, so unset keeps the built-in default: CF_ACCESS_TRUST_ENABLED, CF_ACCESS_TEAM_DOMAIN, CF_ACCESS_AUD, CF_ACCESS_TRUSTS_MFA, CSP_REPORT_URI, EMAIL_SUPPORT_ADDRESS, GOOGLE_WORKSPACE_ENABLED, M365_ENABLED, HP_WARRANTY_ENABLED, IP_CLASSIFY_PROVIDER, IP_CLASSIFY_API_KEY, MCP_LLM_PROVIDER, MCP_LLM_BASE_URL, MCP_LLM_MODEL, MCP_LLM_API_KEY, MCP_LLM_PRICE_INPUT_PER_M_USD, MCP_LLM_PRICE_OUTPUT_PER_M_USD, NVD_API_KEY, PARTNER_MEETING_URL, PARTNER_TRUST_MODE, SENTRY_PROFILES_SAMPLE_RATE, STRIPE_ACCOUNT_CACHE_REFRESH_ENABLED. The root docker-compose.yml also gains APPLE_APP_ATTEST_APP_ID, APPLE_APP_ATTEST_ENVIRONMENT and PARTNER_HOOKS_URL (deploy/docker-compose.prod.yml already mapped them). Both files gain METRIC_ANOMALY_EPISODE_MIN_BUCKETS, and deploy/docker-compose.prod.yml gains the four PARTNER_API_* variables from v0.122.1 (#8402). A value already in your .env that Compose used to ignore now takes effect. Check .env for stale entries, especially CF_ACCESS_* and MCP_LLM_*, before upgrading. (PARTNER_TRUST_MODE has no effect unless IS_HOSTED=true.)

Agents. Agents, Breeze Assist and Breeze Viewer update to 0.123.0 through your normal update channel. Agent-side changes: Encrypted Client Hello (#8882, #8949); heartbeat retain-state (#8645); watchdog install retry and the new service install flags (#8647, #8685); workload inventory collectors, off until policy enables them (#8417, #8545); reliable remote input and wider key coverage (#8523, #8547); caller verification and endpoint message cards, dark (#8657, #8752); Information-level MsiInstaller events (#8431); DNS servers per adapter (#8425); ZFS encryption (#8419); more Management Posture signatures (#8513); typed command error codes (#8473); and the Windows, Linux and macOS fixes above. System Protection checkpoint creation is not in this agent; it ships with a later agent release.

Known gaps in this release

  • Workload inventory: a locally built image is misclassified on Docker 29 with the containerd image store (#8993). The store gives a local build a repo digest, so the collector records one, the image-currency worker asks Docker Hub about it (401), and the Workloads tab shows "Not checked / private registry" instead of "Local build". The effect is a wrong status, an inflated "not checked" compliance count, and a lookup of the local image name against Docker Hub. Docker with the classic overlay2 image store is unaffected. Fix in the next release.
  • PAM, pre-existing, confirmed on the Windows Server 2022 lab rig: after a server-approved elevation launches the app, the user's original UAC credential prompt stays open (#8995); on an RDS session host the on-demand helper lifecycle leaves no SYSTEM helper, so elevation never completes unless helper_lifecycle_mode: always-on is set in agent.yaml (#8996). Both predate this release.
  • Heartbeat, pre-existing: a failed remote-access policy read inside the heartbeat transaction returns 500 for every agent until the read succeeds again (#8997, since #8053). It needs a database fault to trigger; it also blocks the remote-management part of the retain-state signal from being exercised.
  • Caller verification, behind CALLER_VERIFICATION_ENABLED: on a deployment with no email transport, a rejected workstation verification is retried every 5 s indefinitely (#8998). Leave the flag off, as shipped, or configure email before enabling it.
  • Not verified on real hardware before the tag: the viewer keyboard-mode and reliable-input rows (#8635, #8553, #8543, #8523, #8547), the Assist tray rows (#8563, #8564), and the installer rows (#8781 macOS pkg, #8527 MSI manifest), which need this release's signed artifacts. They are checked during rollout.
  • System Restore outcomes stay at "Awaiting the device" until an agent release that supports System Protection ships. Only Windows 11 25H2 and Server 2022 are verified so far; Windows 10 and Server 2025 are tracked in #8265.
  • ECH on the real ISP gateways (Comcast xFi, Spectrum Security Shield) can only be confirmed on a blocked PC after release. ECH does not cover HTTP-proxied connections or the installer script downloads (#8874).
  • EDR response actions have no web switch yet. A connection's actions_enabled is set through the API.

Full Changelog: v0.122.1...v0.123.0

What's Changed

New Contributors

Full Changelog: v0.122.1...v0.123.0

Check out latest releases or
releases around LanternOps/breeze v0.123.0

Don't miss a new breeze release

NewReleases is sending notifications on new releases.

Get notifications