PegaProx 1.4.0 is about knowing before you act: a migration preflight with one verdict per guest, a what-if simulator for failures and maintenance, restore tests that really restore, a timeline of every cluster, node and guest with a signed evidence bundle, and a hash-chained audit log. Next to that come an All Guests table across all clusters, a search language, automatic failover for PegaProx's own HA group as a beta, and two security passes, one of them a whitebox penetration test.
Please read the behaviour changes before updating. There are thirty-three, several refuse something that worked before, and the update dialog lists them as well. The full list with explanations is on the docs: Upgrading to 1.4. The most likely to bite: 100.64.0.0/10 (CGNAT, Tailscale and other tailnets) is now refused wherever private addresses are refused, including the OIDC sign-in unless "Internal IdP / Private Network" is on. See Outbound Requests and Private Addresses.
🧭 Know before you act
- Migration preflight for bulk and cross-cluster migration: a verdict per guest (ready, warning, blocked) with the reasons, ordered steps, what an abort would leave behind, a dry run and a Check step in the migrate dialogs. A block can be overridden with a confirmation that is audited.
- What-if simulator per cluster: node loss, maintenance placement, N+1/N+2 headroom, storage, bridge, VLAN, VNet and PBS failures, each statement marked as known, calculated or assumed. A risk view weighs critical guests, tenants and proven restores, ranks single failures, and gives one verdict; the maintenance dialog runs it before a node goes down.
- Restore tests that run: on a weekly schedule PegaProx restores real backups into isolated test copies (new MACs, links down or on a test bridge), checks the guest agent, ports and commands, measures the RTO, and reports per guest, cluster and tenant. Real restores need a confirmation, with a size cap and a free space check.
- Recovery assurance: coverage of critical guests, a gap list, manual confirmations and a signed evidence export that can be verified later.
- Timeline per cluster, node and guest: audit, alerts, migrations, drift, tasks, backups, node states, Site Recovery, rolling updates and syslog errors in one view, with "shortly before" marked as correlation, not cause. The window can be exported as a signed evidence bundle.
- Audit log hash-chained and checkpointed, verified by the compliance check, which now scores only the controls it could check and says why the others were not.
✨ New
- All Guests: one paged, filterable table of every guest across all clusters, with column picker, guest agent and disk I/O columns and the actions each guest allows.
- Search language: AND, OR, NOT, groups and quotes, with name:, id: (ranges), type:, cluster: and pool: next to tag:, node:, ip:, status:, mac: and notes:. Queries that found something before still find the same.
- Automatic failover for PegaProx's own HA group (#625), as a beta: a majority lease with Make leader, Force leader and a witness. Every group still starts in manual mode.
- Tenant quotas apply on every way a guest can land on a cluster: clone, restore, template deploy, cross-hypervisor, ESXi and cross-cluster landings, replicas, resizes, new disks and snapshot rollbacks.
- Power & Carbon per host (#965): idle and full-load watts per host, costed per host and interval, with the power no guest explains shown as unallocated. Power, costs and right-sizing are worked out in the background and say how old they are.
- File level restore from PBS backups into running guests (#1139).
- Site Recovery fails over and back single guests, and a cancel stops a run before its next guest.
- Rolling updates survive a restart or HA switchover, pause for quorum (QDevice votes included) and can be continued or cancelled.
- Backups: a run history per backup job, and a restore of several guests at once from the datastore browser.
- Balancer: every move is recorded with its reason and kept across restarts; the cooldown between two moves of one guest is set per cluster.
- The all-clusters overview shows the pending updates of every node and PBS server and a Ceph panel per Ceph cluster.
- Transfer network per cluster for cross-cluster migration, replication, Site Recovery and the balancer.
- Change plans with four-eyes approval, linked into config drift. An optional record: a plan runs and blocks nothing.
- Config drift reads firewall rules, HA rules, backup jobs and access, with optional policy checks.
- New event alerts for Ceph OSD latency, the RPO of PegaProx's own replications, node clock drift, guest restart loops and old restore tests; more Prometheus series and two Grafana dashboards.
- QDevice state in the tree, details and alerts (#1137); a least-privilege role recipe for Proxmox VE; node root passwords of their own per node (#1136); an OIDC username claim for AD FS (#1149).
- PBS: every sync, verify, prune and GC job per datastore with its schedule, its last run and the next one (#804); GC state from the server.
- Charts show gaps instead of a measured zero, take a time range by dragging, and read from PegaProx's own 5-minute history.
- The SMTP settings can verify the relay's certificate.
🔒 Security
Two passes: a review of everything since 1.3.0, and a whitebox penetration test by CodeAnt AI, who also give PegaProx their code review tool. Thank you.
- Task log, status and cancel requests for Proxmox VE and PBS check the task id before the call, and the API sessions refuse paths with dot segments. This flaw is also in 1.3.0 and older; updating is strongly recommended. Advisory: GHSA-c25f-58q4-978j.
- Template deploys check every redirect of the image URL; the outbound guard treats 100.64.0.0/10 as private again; ESXi servers get the address check PBS has.
- Users confined to pools or single guests no longer read a node's system journal or Ceph log, the Ceph mirroring, cluster-wide guest counts in the reports, node incidents, snapshot policies that match none of their guests, or other guests' scheduled tasks.
- A tenant's user admin reads only their own tenant's permissions; IP lockouts are cleared by a global admin only.
- SSH does not trust a host key on first use when it cannot be recorded, and writes known_hosts atomically.
- Restores over a guest that is gone keep to the tenant's VMID range; a manual restore test places its copy only where the caller may.
- The node shell takes the node address from the server only; the global search no longer shows the notes, IPs or MACs of a deleted guest on a new guest with its VMID; recovery evidence verifies only as written.
- Smaller: HA quorum values, power rates, the alert pattern guard, the WebSocket subscription, in-app update paths, TLS key writes, change plan approvers, OIDC fallback accounts and mail addresses in the support bundle.
⚠️ Behaviour changes
Thirty-three; the update dialog lists them and Upgrading to 1.4 explains each. The ones most installations notice:
- 100.64.0.0/10 is refused where private addresses are refused: OIDC sign-in (unless "Internal IdP / Private Network" is on), ACME (unless the private CA switch is on), alert and Site Recovery webhooks, SIEM, ntfy and Apprise, push subscriptions and downloads from a URL into a storage.
- SMTP: fresh installs verify the relay certificate; existing ones keep sending without the check until it is turned on.
- Power & Carbon figures drop after the update (the old formula counted idle power several times); reports before and after are not comparable.
- The weekly restore test really restores now, after a confirmation.
- A 401 or 403 from Proxmox VE, PBS, ESXi or the HA plugin reaches the browser as 502 UPSTREAM_AUTH and no longer signs the user out; API clients that matched on the upstream status check the code.
- Tenant quotas can refuse clones, restores, resizes and landings that went through before.
- Rolling updates do not start beside another run and pause as "interrupted" after a restart.
🐛 Fixes
- Syslog timestamps carry their UTC offset (#1131); PBS dialogs stack above the corporate table headers (#1150); the task bar arrow points the right way (#1125).
- The node shell opens on the clicked node and works behind a reverse proxy (#1143); an ESXi server that refuses its credentials no longer signs the user out (#1142).
- The Debian package, the Docker image and the scripts ship and keep the bundled plugins and their settings (#1134); accounts made by OIDC with a fallback name can be managed (#1141).
- ACME trusts the system CA store for a custom CA (#314) and names a certificate without CN by its first host.
- Container restore from vzdump, the live backup pane, cloud backup failure counts, scheduled tasks that stepped over a minute, scheduled rolling updates in a late pass, drift repeats and baseline notes, XCP-ng rolling updates, PBS GC figures and partial job lists.
- At 10k guests the what-if ranking, the preflight, long searches, chart ranges and the power, cost and right-sizing reports no longer hold up the server.
🧭 Known issues
- Snapshot-Iterative (zero downtime) ESXi migration is still experimental. Use Live Clone or Live Mirror for guests whose data matters.
- Automatic failover for PegaProx itself is a beta. Try it in your setup before relying on it; manual failover stays one switch away.
- Change plans are an optional record without an approval gate: nothing runs and nothing is blocked by a plan.
- The QDevice view sees the QNetd host only through the nodes; pending security update counts exist for XCP-ng only.
✅ Quality
The automated suite runs more than 13,400 tests on every change. The release was checked live on a lab instance with Proxmox VE clusters, an XCP-ng pool and an ESXi host connected, at a simulated scale of 200 nodes and 10,000 guests for the heavy paths, and an update from 1.3.0 was tested against a copy of a 1.3.0 database.
Thank you to everyone who filed, fixed, translated and sponsored along the way. 💚
Thanks in particular to @brngates98, @Clankcoll, @CKeckNorthIO, @SystemFuchs, @MartinMDS, @lordkill61, @omrzk, @si458 and @gyptazy, who filed or fixed the issues in this release.
💛 Sponsors
PegaProx is AGPL-3.0 and built in the open. Huge thanks to our Platinum sponsors who keep it moving:
💎 Platinum
Sponsor PegaProx: opencollective.com/pegaprox | pegaprox.com/#sponsor