Highlights
Fixes GitHub #132 (phantomdez): on Proxmox node reboot with the plugin configured, load climbed, VMs stalled during startup, and uninstalling the plugin + removing the storage config was the only cure.
Broker systemd ordering fix
The broker's systemd unit was Type=simple with only After=network-online.target, and nothing ordered the three heavy PVE storage callers (pvestatd, pvedaemon, pveproxy) after it. At boot those services could start before the broker's Unix socket was listening. The plugin's fallback path then re-authenticated to TrueNAS directly from each PVE worker, the initial burst tripped TrueNAS's auth.login_with_api_key rate limit (20 calls / 60 s / IP) into a 10-minute cooldown, and every subsequent pvestatd tick stayed rate-limited. pve-guests.service waited for the storages to become active, graphical.target never fired, autostart VMs never ran.
Two-part fix:
-
debian/truenas-plugin-broker.service: switched toType=notifywithNotifyAccess=main, and addedBefore=pvestatd.service pvedaemon.service pveproxy.service. Downstream services now wait until the broker's socket is actually accepting connections. -
tools/truenas-plugin-broker: added a pure-Perlsd_notify_ready()that sendsREADY=1toNOTIFY_SOCKET, called immediately after the Unix listen socket is bound. No extra fork, Perl core only.
Verification
Reproduced the failure on a test node, provisioned 5 autostart VMs on truenas-storage, rebooted. Side-by-side:
| pre-fix | beta4 | |
|---|---|---|
| boot time | 14 min 27 s (graphical.target never reached) | 3 min 58 s |
broker socket missing errors
| 57 | 0 |
TrueNAS API rate limited errors
| 57 | 0 |
| broker WS opens during boot | 1562 | 37 |
onboot=1 VMs reaching running
| 0 of 5 | 5 of 5 |
New test coverage
The regression went undetected because no existing harness exercised Proxmox node reboot with the plugin configured. tools/boot-reboot-test.sh is a new integration test that provisions N VMs with onboot=1 on a target storage, reboots the target node, waits for ssh return, and asserts:
- zero
broker socket missingentries injournalctl -b - zero
rate limitedentries injournalctl -b - broker service active
- broker unit
Type=notify - broker
Before=includes pvestatd / pvedaemon / pveproxy - all
onboot=1VMs reachedrunning
Standalone, destructive, run from an orchestrator against a disposable PVE node.
Install
Debian/Proxmox package attached. Install with:
dpkg -i truenas-proxmox-plugin_2.1.24~beta4_all.deb
Verify integrity with the attached SHA256SUMS.
Full Changelog: v2.1.24-beta3...v2.1.24-beta4