github LanternOps/breeze v0.121.0
on GitHub

3 hours ago

Breeze RMM v0.121.0: one home for every AI model decision, a chat model picker, bring-your-own OpenAI-compatible models with tool use, automatic AI failover, AI usage chargeback to your clients, and backup destination credentials encrypted at rest.

⚠️ Roll forward only. Backup destination credentials are now encrypted at rest, and existing rows are encrypted on the first start. Rolling back to an earlier release after that start would send encrypted values to agents as storage credentials and break S3 backups. Take your usual database backup before upgrading (#7718).

⚠️ Self-hosters using MCP_LLM_PROVIDER=openai-compatible: action required. IS_HOSTED=false must now be set explicitly, the model must support tool calling to serve chat, and localhost endpoints are refused. Details under Self-Hosting / Upgrade Notes (#7771).

⚠️ Deprecated, removed in v0.122: /ai/provider and the legacy AI reviewer, Office and agent model settings fields. The AI model registry (/ai/models) replaces them. Upgrade to v0.121 before v0.122 rather than skipping it. Details under Deprecations (#7606).

Security — action required

Self-hosted operators on 0.120.0 and earlier should upgrade. A user who can configure a webhook, alert channel or image-from-URL field, or whoever runs the URL it points at, could repeatedly stop the API process, interrupting service for every tenant on the instance (GHSA-vjqq-fmr4-m576, High). A technician limited to selected organizations could read, edit or delete ticket comments in the partner's other organizations (GHSA-7xq8-mcq6-j438, Low).

Security hardening

This release includes security hardening across:

  • backup destination credential storage;
  • AI tool output;
  • ticket comment and access review permissions;
  • API key authentication;
  • patch policy and privileged-access approval handling;
  • outbound request handling;
  • Quick Support session files and folder permissions;
  • dependency updates and dependency audit checks.

Self-hosters are encouraged to upgrade. (#7716, #7723, #7729, #7751, #7782, #7718, #7687, #7706, #7591, #7614, #7662, #7721)

Summary

  • AI Providers & Models settings (#7701, #7700). Partner Settings → AI Providers & Models is now the single place for AI connections, the models your techs may use, and the default model per AI feature. Org Settings → AI → Model defaults can tighten that per client. Every AI feature now picks its model, key and price from these settings, and every model call is metered once.
  • Chat model picker (#7763, #7774). Techs pick the chat model and its options (effort, thinking) from the models you allow. Switching model mid-chat resumes the same conversation when it fits; otherwise Breeze offers a linked new chat that starts from a summary. AI agents pick their model the same way.
  • Bring your own OpenAI-compatible model (#7771, #7801). Connect vLLM, LiteLLM, Ollama, OpenRouter, a hosted gateway or another OpenAI-compatible endpoint, discover or enter its models, verify them, and use a verified model on every AI feature, tools included.
  • Automatic failover (#7775, #7789). When a model is unavailable, rate-limited, overloaded, out of credit or its key is rejected, AI calls move down an ordered backup list (up to 5). Moving between Breeze credits and your own API key happens only if you allow it. AI agent runs use a model per stage: triage, analysis, remediation.
  • AI chargeback (#7759). Bill clients for the AI usage Breeze meters: set coverage (billable, included or not billed), a markup, or a per-model price list on a billing profile. Usage closes monthly into invoice lines, and a new AI usage by client report shows it. Nobody is billed by default, and usage from before this release is never billable.
  • Model quality view (#7767). Settings → AI Usage gains a Quality view beside Spend, by model, feature or model family.
  • Quick Support (#7687). Standard (non-admin) Windows users can start Quick Support, the session keeps its files in its own folder, and the user sees an on-screen "is viewing your screen" indicator while a technician is connected.
  • Windows viewer updates install again (#7726). The in-app update of Breeze Viewer on Windows downloaded but never installed; it now installs, and a failed update says which stage failed.

Added

  • AI Providers & Models tab (#ai-provider): connections with a residency switch, models (enable, price, verify, refresh), and defaults by feature including AI agent stage roles, backup list and the cross-funding switch (#7701, #7775).
  • Org Settings → AI → Model defaults: tighten-only override; blank inherits and shows the inherited value (#7701, #7775).
  • Settings → AI Usage breakdown by model, feature, technician and org, with refusal rate and failovers, plus the Quality view (#7701, #7767, #7775).
  • Chat composer model menu and POST /ai/sessions/:id/continue for a linked continuation chat (#7763).
  • OpenAI-compatible connections with discovery (GET {base_url}/models), manual model entry and a verification harness; new models land disabled, unpriced and unverified until you price and verify them (#7771).
  • AI chargeback: an AI usage section in the billing profile drawer (Settings → Billing → Rates), ai_usage invoice lines (non-taxable by default; editable on a draft), and the AI usage by client business report with PDF (#7759).
  • AI agent runs can end blocked (model_unavailable / model_refused), notified once per agent per day; an org with no usable model skips the run instead of failing it (#7700).
  • Platform admins: a Prompt variants card on /admin/ai-models (#7767).
  • Configurable agent enrollment rate limit: AGENT_ENROLL_RATE_LIMIT (default 10 attempts) per AGENT_ENROLL_RATE_WINDOW_SECONDS (default 60), per source IP. Raise it for mass deployments behind one NAT address (#7724).
  • Windows backup captures junctions (for example a redirected Music folder) as links and recreates them on restore, instead of skipping them. Volume mount points and other reparse points are still skipped and named in the job warning (#7757).
  • Antivirus detection recognises Emsisoft, Webroot, ThreatDown and WithSecure (including F-Secure) (#7634).
  • Partner API ticket scopes (groundwork): service principals can be granted two new Partner API scopes, tickets:read and tickets:write. Neither is in any default set. The ticket endpoints arrive in a later release, so granting the scopes has no effect yet. Comments an integration posts will be credited to the integration by name, not to a user, and never trigger an automatic AI reply; tickets can record their id and link in an external PSA or ITSM (new ticket_external_refs table) (#7490).
  • New optional env vars AI_PLATFORM_INFERENCE_GEO (us or global) (#7700) and AI_GATEWAY_HEADERS_TIMEOUT_SECONDS (#7801).

Improved

  • One metering path for all AI features. Chat, topology, helper, script builder, Office, ticket drafts, the script reviewer, AI agents and enrichment are all priced from the same registry rates and recorded per call. Per-spend budget alerts cover every AI feature (#7700).
  • Model refusals in chat show the category, alternative models and a docs link instead of an empty answer (#7700).
  • Disconnecting an AI connection is a soft disconnect: the key is removed and its models disabled, and chats bound to it report it unavailable instead of silently moving elsewhere (#7700, #7771).
  • Legacy model editors retired. The Office allowed-models list and the script-reviewer model fields now point to AI Providers & Models (#7701).
  • AI chat diagnoses before it blames. Before naming a process, a vendor or the Breeze agent as the cause of load, the chat traces it to its owning process and service, works from a list of what the Breeze agent actually runs, and restates its evidence before proposing to kill a process or another destructive fix (#7764).
  • AI agent runs are no longer offered tools the agent guardrail always refuses, which saves turns (#7758).
  • Invoice payments are serialized per invoice. Pay links, portal payments, manual payments, accounting imports and voids now take a per-invoice reservation, laying the ground for autopay. Nothing new appears in the UI (#7777).
  • Metric history storage. Metric rollup rows that landed in the catch-all partition are moved into proper monthly partitions by the daily maintenance job, so their space is returned and month creation no longer scans them (#7664).
  • Patches: a per-device install supersedes an older failed job for that patch, and a patch that installed but awaits a restart shows Installed, reboot required instead of pending (#7727).
  • Add Device reuses one installer enrollment key per site instead of creating a new one on every download or link (#7761).
  • Network topology overview reads like a network map: each site network is one card, with its gateway above the LANs that route through it, and every device is a tile showing its type, name, address and whether it has an agent. Duplicate gateway/subnet boxes and endpoint <uuid> labels are gone, and graph reads are faster (#7762).
  • Custom fields are listed alphabetically everywhere (#7660).
  • Startup tasks (built-in monitor provisioning, settings-secret sealing, backup storage key history) retry with backoff after a transient failure instead of waiting for the next restart (#7725).
  • Agent log level: a raised log-shipping level now survives an agent restart, reaches the desktop helper, and the command reports the level applied (#7590).
  • Invoice PDF and email include a unit price column, matching the web preview and portal (#7638).
  • Quote tax is resolved when the quote is sent, not when it was created, so a sent quote carries the org's current rate (#7754).
  • Destructive buttons meet WCAG AA contrast in light and dark themes (#7720).
  • Connected Apps explains when MCP OAuth is not enabled on the server (MCP_OAUTH_ENABLED=true) instead of showing a bare 404 (#7722).
  • Viewer: connecting to a device whose agent runs as a service, from a viewer without WebRTC, now explains that the device needs WebRTC and what to use instead (#7755).
  • Cloud backup storage format documented: S3, B2, Azure and GCS objects keep a .gz name but are stored uncompressed; Local/NAS objects are gzip-compressed. Rename, don't gunzip, a downloaded cloud object (#7715).
  • AI tools on Quick Support devices: screenshots, screen analysis, computer control and diagnose are refused for a Quick Support session (409 SCREEN_ACCESS_UNAVAILABLE_IN_QUICK_SUPPORT); use the remote desktop session, where the user sees the indicator (#7687).

Fixed

  • Ollama and slow local model servers work as OpenAI-compatible connections: tool calls through Ollama are no longer dropped, the first-response wait is longer on self-host and configurable, and choosing a model without tool calling for chat says to choose a tool-capable model instead of showing a bare error (#7801, #7793, #7794, #7795).
  • Breeze Viewer in-app update on Windows now installs. The update package could not be unpacked, so viewers stayed on their installed version and showed "will retry on next launch" (#7726, #7681).
  • Human API keys no longer stop working when their creator signs out. A password change or reset, invite acceptance, an admin status change or an MFA factor change still invalidates them (#7591, #7489).
  • Built-in alert rules ("Patch job failures", "Reboot pending too long", the policy and config-compliance rules) can no longer be created twice. Existing duplicates are merged once at upgrade (#7750, #7650).
  • Automation webhooks: a correctly signed webhook call no longer returns 404 Automation not found (#7589, #7363).
  • Webhook settings no longer go blank once a webhook has delivered or retrying deliveries, and Test confirms the delivery was queued (#7806).
  • Quick Support can start for a standard (non-admin) Windows user (#7687, #7620).
  • Quick Support no longer writes config, state or logs into the installed agent's folders; the installed agent re-secures its config folder if another account created it (#7687, #7629, #7645).
  • Breeze Assist trusts certificates in the operating system's store, so self-hosted servers on a private CA connect (#7657, #7550).
  • Binary sync: a failed S3 upload at boot is retried, and the previous release's binary is never served under the new checksum (#7661, #7574).
  • Deleting a backup destination with job history returns a clear 409 naming what blocks it (disable the destination instead) rather than a 500 (#7636, #7622).
  • Cancelling a queued bare-metal rebuild restore closes the recovery, so the device can be rebuilt again (#7611, #7512).
  • Backup readiness score no longer drops while verifications are still running; a verification stuck over 30 minutes counts as failed (#7749, #7495).
  • Vulnerabilities: Ready in the findings list now means Remediate will actually install a patch on that device (#7752, #7499).
  • Linux encryption: a root filesystem on LVM over LUKS (the default Ubuntu/Debian encrypted install) reports as encrypted (#7579, #7478).
  • Linux recovery media: a network or TLS failure, or a wrong server URL, no longer reads as "That code did not work" or spends a code attempt (#7658, #7649).
  • Passwordless SSO accounts that already hold an MFA factor can add another passkey or TOTP (#7635, #7369).
  • Ticket mailbox (M365): when the post-consent check fails because the application access policy isn't applied yet, the mailbox shows the policy steps and Re-test instead of looping back to Reconnect (#7612, #7569).
  • Disk cleanup: Select all is capped at the 200 paths a cleanup run accepts, with a note explaining the limit, instead of failing on Delete (#7580, #7469).
  • Run again on a device's script history resolves devices in that device's organization, not the org picked in the top bar (#7581, #7479).
  • Partner billing settings are read-only, with a notice, for users without access to all organizations, instead of failing on Save (#7595, #7517).
  • Org Viewer sign-in no longer shows a hydration error or partner-only 403s (#7737, #7498).
  • Portal Backups: the "protected" count matches the table below it (#7742, #7505).
  • Device identity edits no longer show a false "changed elsewhere" banner after your own save (#7632, #7388).
  • Device mTLS certificates: devices that got their client certificate at enrollment, from admin provisioning or from quarantine approval were missing from Breeze's certificate history, so in enforce mode they could not renew or had their current certificate refused. Every issued certificate is now recorded, and the upgrade repairs existing devices once. Each superseded certificate is revoked at Cloudflare in the background over the following hour or so. A device whose certificate can't be recorded falls back to token auth (#7616, #7431, #7432).
  • Hyper-V crash-consistent backup handles VMs that are Off, Saved or Paused. Off and Saved VMs are exported as they are and stay off; a Paused VM is left Saved, with a warning on the job; a running VM is restarted even if its export fails (#7630, #7623).
  • Remote terminal: typing quickly or holding a key no longer ends the session. Input over the per-minute ceiling (6,000 messages or 8 MiB) is dropped with a warning and the session stays open (#7596, #7475).
  • Deleting a site that still has devices, including removed ones, explains why it can't be deleted and how many devices are in the way, instead of a server error. Purge removed devices (Devices → show removed → Delete permanently) or move active ones, then delete the site (#7588, #7471).
  • Organization merge no longer fails when a merged-away report run is cited as service-deliverable evidence; the evidence moves with the report run (#7613, #7443).
  • Invoices without online payment: the public invoice page and the client portal no longer show a Pay button, the invoice PDF no longer prints a "Pay online" line, and invoice emails include "View & pay" only when payment is available (#7610, #7509).
  • Xero / QuickBooks sync: when the provider rejects a customer or item as invalid (for example a name Xero won't accept), the mapping shows the provider's reason once and stops, instead of retrying five times and reporting each retry as an error (#7659, #7292).
  • Accounting workbench: confirming a mapping while the background sync is already running no longer shows an error; the row updates to Synced on its own (#7656, #7386).

Known issues

  • A chat turn that fails on a provider error before any output can end without an error shown (#7785).
  • The Models card does not refresh when background discovery or verification finishes; reload the page (#7781).
  • Breeze Assist on Windows does not run in RDP or other non-console sessions (#7542).

Self-Hosting / Upgrade Notes

Upgrade command. Check how your install pins its images: grep '^BREEZE_API_IMAGE_REF=' .env.

  • Digest-pinned (value ends in @sha256:…, every guided-setup.sh install from v0.112.0): fetch the current guided-setup.sh from main, then run bash guided-setup.sh --upgrade 0.121.0. It verifies the signed image inventory, updates BREEZE_VERSION and all four image digests, then pulls and restarts.
  • Tag-form (value ends in :${BREEZE_VERSION}): set BREEZE_VERSION=0.121.0, then run docker compose pull api web portal && docker compose up -d.
  • Do not edit only BREEZE_VERSION on a digest-pinned install. It re-pulls the images you already run.
  • --upgrade does not change your docker-compose.yml. To set any of this release's new optional variables, add their lines to the api service's environment: block (or take the v0.121.0 docker-compose.yml); a value only in .env does not reach the API.
  • See the Upgrade Guide.

Breaking changes and required actions.

  1. Roll forward only: backup destination credentials are encrypted at rest (#7718).
    • Access keys, secret keys, session tokens and the equivalent Azure, GCS and B2 credential fields in backup destinations are now stored encrypted with your application encryption key. Bucket, region, endpoint and prefix stay readable.
    • On the first start, a background sweep encrypts existing destinations and logs one line:
      [startup] Backup destination credentials sealed: <sealed>/<scanned> config(s) (<n> changed concurrently, <n> failed)
      Re-running finds nothing to do.
    • After that line, do not roll back to an earlier release. Older code would hand the encrypted values to agents as storage credentials and S3 backups would fail. Fix forward instead.
    • Key rotation: scripts/re-encrypt-secrets.ts now covers this column.
  2. MCP_LLM_PROVIDER=openai-compatible installs (#7771). The instance-wide OpenAI-compatible chat path is replaced by a connection in the new model registry.

    • Set IS_HOSTED=false explicitly. If IS_HOSTED is unset, empty or anything other than a recognised self-host value, the API refuses to start with:

      MCP_LLM_PROVIDER: MCP_LLM_PROVIDER=openai-compatible is for self-hosted Breeze (set IS_HOSTED explicitly to false). On hosted — or with IS_HOSTED unset/invalid — it is refused: it would create a connection for every partner. Add an OpenAI-compatible connection under Partner Settings → AI Providers & Models instead.

      The stock docker-compose.yml passes IS_HOSTED: ${IS_HOSTED:-false}, so a Compose install gets false unless it overrides it. Installs without Compose (systemd, Kubernetes, bare Node) must set it. Note that the stock Compose file does not pass MCP_LLM_* to the API; if you use this path you already carry your own mapping.

    • What happens at boot. Each partner gets one read-only, env-managed connection and a priced, enabled model (prices from MCP_LLM_PRICE_INPUT_PER_M_USD / MCP_LLM_PRICE_OUTPUT_PER_M_USD), verified once. Chat is pointed at it once, only where chat used the platform default. Restarts and multiple replicas don't create duplicates, and a later admin choice is kept. Partners created after boot get their connection within 10 minutes.

    • The model must support tool calling to serve chat. A model that fails verification for tool use cannot serve chat; chat then says the model can't use tools and to choose a tool-capable model under AI Providers & Models (#7801).

    • localhost is refused. An env URL on a loopback, link-local or cloud-metadata address is refused; the API still starts, bootstraps nothing, and retries at 1, 5, 15 and 60 minutes, then every 10 minutes:

      [envOpenAiBootstrap] MCP_LLM_BASE_URL refused: That host is not reachable from Breeze (loopback, link-local and metadata addresses are never allowed). — no partner was bootstrapped

      http://localhost:11434 (the usual Ollama URL) is therefore refused. Use the Docker service name or the host's LAN IP. Private-network addresses over http:// are allowed on self-host.

    • MCP_LLM_API_KEY is optional, but must be at least 8 characters when set.

    • Removing the variables keeps the connection: it stays active and read-only, and the drawer says:

      This connection was set up from the MCP_LLM_* environment variables, which are no longer set. It cannot be edited; you can disconnect it.

      Delete the MCP_LLM_BASE_URL line rather than blanking it; an empty value fails config validation and the API will not start (unchanged from earlier releases).

    • Model server sizing. Breeze's chat requests are large: allow at least 70k tokens of context (more for long chats).

    • First-response wait. The model gateway waits up to 120 s for a model server to start responding on self-host (30 s on hosted). Slow or CPU-only local servers (long prefill) may need to raise AI_GATEWAY_HEADERS_TIMEOUT_SECONDS, up to 600. The value is clamped to 5–600 s; an invalid value is ignored with one warning. When the wait runs out, the request returns a 504 that names the setting, and the API logs:
      [modelGateway] upstream sent no response headers within the <N> ms response-header deadline (…) (#7801)

    • Ollama tool calling works (verified with qwen2.5 3b and 7b on Ollama 0.35.0). LiteLLM in front of llama.cpp llama-server --jinja also works (#7801).

    • Chat on this path is now recorded in AI usage like every other AI feature.

Deprecations (removed in v0.122). The AI model registry replaces each of these. In v0.121 they still work (the retired fields are already ignored); in v0.122 /ai/provider returns 404, and a request carrying a retired field is rejected with a 400 naming the field and its replacement, with nothing in it applied (#7606).

  • /api/v1/ai/provider (GET, PATCH, DELETE, POST /key, POST /endpoint). The web app no longer calls it. Use the AI model registry API: GET /api/v1/ai/models; connect a key with POST /api/v1/ai/models/connections, rotate it with POST /api/v1/ai/models/connections/:id/key, set an endpoint with POST /api/v1/ai/models/connections/:id/endpoint, disconnect with DELETE /api/v1/ai/models/connections/:id. The single partner default model is replaced by per-feature defaults: PUT /api/v1/ai/models/assignments.
  • reviewerModel on PUT /api/v1/ai/script-policy and PUT /api/v1/partner/ai/script-policy. Use the script_reviewer feature default (AI Providers & Models → Defaults by feature).
  • allowedModels on the AI for Office organization policy (PUT /api/v1/client-ai/admin/orgs/:orgId/policy). Use the office_chat feature's permitted models, or narrow them per organization with PUT /api/v1/ai/models/orgs/:orgId/assignments.
  • model on AI agents (POST /api/v1/ai/agents, PATCH /api/v1/ai/agents/:id, POST /api/v1/ai/agents/preview), which also carried the legacy agent model allowlist. Send offeringId instead: an enabled model from GET /api/v1/ai/models; null follows the AI agents default.
  • Don't skip v0.121. The one-time move of legacy AI settings (script reviewer, AI for Office and AI agent model choices) into the registry runs in this release. An install that jumps straight to v0.122 starts each partner from the platform default and loses those choices.

Other behaviour changes.

  • AI model settings moved. Model choices previously set in the Office policy editor, the script-reviewer model fields and the partner AI provider default are migrated once, on first start, into AI Providers & Models. BREEZE_AI_SCRIPT_REVIEWER_MODEL and WORKSPACE_CONTENT_LLM_MODEL are no longer read at runtime; their values are migrated at that first start. ANTHROPIC_MODEL remains the bootstrap default.
  • Residency switch. Turning on the residency requirement under AI Providers & Models disables every AI feature whose model can't guarantee it (the confirm lists them). BYO OpenAI-compatible models are never residency-eligible.
  • AI_PLATFORM_INFERENCE_GEO (new, optional): us or global. Any other value takes platform models offline, and the API warns at boot.
  • AI chargeback is off until you configure it. Every billing profile starts as not billed. A daily job (05:28 UTC) closes the previous UTC month per org. Chargeable ledger rows are kept at least 156 days regardless of AI_INVOCATIONS_RETENTION_DAYS.
  • Self-hosted installs without a billing service never mark AI usage for credit debit, and turning billing on later cannot debit a backlog.
  • Human API keys and sign-out (#7591). A sign-out, role change, email change or org merge no longer invalidates a human API key; a password change or reset, invite acceptance, an admin status change or an MFA factor change still does. A role downgrade or membership removal still takes effect on the key immediately. Keys that stopped working after a sign-out on v0.120 (#7489) are not revived by the upgrade: mint a replacement.
  • Duplicate built-in alert rules are merged once at upgrade (#7750). The oldest copy is kept; if any copy was switched off, the merged rule is off. Open alerts from the removed copies move to the kept rule, and where two were open for the same device, the extra one is resolved as source_retired. The API logs #7650: … warnings with the counts.
  • Metric history drain (#7664). On the first daily metric-rollup maintenance run after the upgrade, rows in the catch-all metric_rollups_default partition are moved into monthly partitions. Rows past your 5-minute retention are discarded, as before. Until the move finishes (seconds to minutes, by size), those older points are missing from charts. Look for [MetricRollupMaintenance] … drained metric_rollups_default: moved=N discarded=M ….
  • Quick Support: the viewing indicator always shows during a Quick Support session, regardless of the device's remote access indicator setting.

Database.

  • 45 migrations (idempotent; applied automatically on boot via autoMigrate unless AUTO_MIGRATE=false). Expect [auto-migrate] Applied 45 migration(s). Some files are named earlier than v0.120's newest; they still apply, in order.
  • Mostly additive DDL: new tables for the AI registry cutover, AI chargeback (charges, runs, claims, per-model rates), autopay foundation tables (off by default), new columns on ai_invocations, ai_sessions, ai_budget_reservations, ai_agent_runs, billing_profiles, llm_egress_events, users and api_keys, new enum values (report type, invoice line source, three antivirus providers), metric-rollup maintenance functions, a unique index on built-in alert rules, and policy updates. For the Partner API ticket groundwork (#7490): a new ticket_external_refs table (RLS enforced), a nullable origin column on ticket_comments, and tickets:read / tickets:write added to the allowed service-principal scopes. An index on topology_node_bindings speeds up topology graph reads (#7762).
  • Three indexes are built with CREATE INDEX CONCURRENTLY outside a transaction, so writes continue: on ai_invocations, ai_budget_reservations and device_commands. A build waits for long-running transactions to finish. If the ai_budget_reservations build is interrupted, the next start refuses with an "INVALID index" message: run DROP INDEX CONCURRENTLY ai_budget_reservations_session_turn_idx; and restart the API. The other two have no such guard, so check them after the upgrade: 
    SELECT c.relname, i.indisvalid FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
 WHERE c.relname IN ('ai_invocations_chargeable_idx', 'idx_device_commands_install_patches_device_created');
    If either is false, drop it with DROP INDEX CONCURRENTLY <name>; and re-run the CREATE INDEX CONCURRENTLY from its migration (2026-11-26-100150-ai-invocations-chargeable-idx.sql or 2026-11-19-101100-device-commands-install-patches-index.sql).
  • Brief locks: CHECK constraints on ai_invocations are added without a scan and validated in separate steps while reads and writes continue. A few constraint and index steps on ai_sessions, ai_budget_reservations and llm_egress_events scan the table while holding a write lock; they took well under a second at 300,000 rows per table in our test, so on very large tables expect a few seconds of paused AI chat writes during the first boot. New columns on users and api_keys take a momentary exclusive lock, and the alert-rule unique index briefly blocks alert rule writes. Adding the ticket_comments column and re-validating its two CHECK constraints holds an exclusive lock on that table for the scan (about 50 ms at 300,000 rows in our test), and the topology_node_bindings index build blocks topology binding writes while it runs. No migration sets a lock timeout, so a long idle-in-transaction session on these tables would delay boot.
  • Device mTLS certificate repair (#7616). One migration records every device's issued client certificate in the certificate history. It holds only a read lock on devices, so agents keep checking in, but it runs inside the boot: on a large fleet it can add seconds (tens of thousands of devices) to minutes to the first start before the API serves. It always logs one line, even when there is nothing to do:
    WARNING device mTLS history reconcile: demoted <D> stale active rows, imported <I> issued certificates, skipped <S> devices sharing a certificate with another device
    Demoted certificates are revoked at Cloudflare by the revocation sweep, 200 every 5 minutes. Devices counted as skipped share one certificate with another device and are left as they were.
  • First start after the upgrade also runs two background jobs that never block /health: the backup credential encryption sweep above, and a one-time move of each partner's AI settings into the model registry ([startup] AI model registry cutover sweep: … N partner(s) cut over, 0 failed).

Environment variables.

  • Required: none. IS_HOSTED=false must be explicit only on MCP_LLM_PROVIDER=openai-compatible installs (above).

  • New, optional:

    • AI_PLATFORM_INFERENCE_GEO (us or global).
    • AI_GATEWAY_HEADERS_TIMEOUT_SECONDS (5–600; default 120 on self-host, 30 on hosted).
    • AGENT_ENROLL_RATE_LIMIT (default 10) and AGENT_ENROLL_RATE_WINDOW_SECONDS (default 60). Blank keeps the defaults; a non-integer or non-positive value stops the API from starting.

    All four are mapped in both Compose files of this release.

  • Deprecated: BREEZE_AI_SCRIPT_REVIEWER_MODEL, WORKSPACE_CONTENT_LLM_MODEL (no longer read).

  • Changed meaning: MCP_LLM_* (above).

  • deploy/docker-compose.prod.yml now passes 82 more variables to the API (#7756), the same lines the root docker-compose.yml already carries: for example the MCP OAuth, Mailgun, Twilio, SMTP, M365, Cloudflare and Redis memory monitor settings, SESSION_SECRET, JWT_SIGNING_KEYRING and SENTRY_TRACES_SAMPLE_RATE. If you run that file, a value in your .env that it used to ignore now takes effect. Nothing to do if your .env holds only what you meant to set; check it for stale entries before upgrading. Installs on the root docker-compose.yml (including guided-setup.sh) are unaffected.

Agents. Agents and Breeze Assist update to 0.121.0 through your normal update channel. Agent changes: Windows Quick Support for standard users, private session files, config-folder checks and the viewing indicator (#7687); Windows junctions in backups (#7757); crash-consistent Hyper-V backup of Off, Saved and Paused VMs (#7630); persistent log-level overrides (#7590); LUKS-under-LVM encryption status (#7579); new antivirus providers (#7634); recovery media error messages (#7658); an OpenTelemetry Go dependency update (#7706). Breeze Assist now trusts the OS certificate store (#7657).

Viewer. Windows viewers on 0.118–0.120 whose in-app update kept failing pick up 0.121.0 in-app on their next launch; no reinstall is needed (#7726). If an update still fails, the banner now names the failed stage and the log path (%LOCALAPPDATA%\com.breeze.viewer\logs\updater.log); as a fallback, download breeze-viewer-windows.msi from this release and run it once by hand.

Feature flags. No feature flag changes. Prompt variants ship in a staged state, so no AI prompt changes at upgrade. Autopay ships switched off for every partner.

What's Changed

New Contributors

Full Changelog: v0.120.0...v0.121.0

Check out latest releases or
releases around LanternOps/breeze v0.121.0

Don't miss a new breeze release

NewReleases is sending notifications on new releases.

Get notifications