github truenas/truenas-proxmox-plugin v2.1.25-beta1

pre-release49 minutes ago

Fix

#147 — cross-node stale multipath/sd cleanup after a volume delete. When one cluster node deleted an iSCSI+multipath volume, every other node that was logged into the same target kept the local SCSI device and multipath map for that LUN. ALUA then reported every path as failed and multipathd logged tur checker reports path is down every 5 s until an operator ran multipath -f and the sd-delete sysfs write by hand on each peer. Because TrueNAS reuses LUN numbers, a later extent that picked up the same LUN could surface behind the stale sd on peer nodes.

The deleting node now drops a per-WWID marker under /etc/pve/priv/truenas-plugin/stale-wwid/<storeid>/<wwid>. Because /etc/pve is pmxcfs, the marker is replicated cluster-wide for free. Every node's next activate_storage and status call reads the hint directory and tears down any WWID that is locally present: kpartx -d on partition holders, multipath -f on the dm map, then 1 > /sys/block/<sd>/device/delete per path. Teardown is scoped by target IQN, so only sd devices behind a session to the storage's tn_target_iqn are candidates — other TrueNAS storages and non-TN iSCSI are never touched. A rate-limited floor sweep (one run per 60 s) in status cross-checks local WWIDs against TrueNAS's current iscsi.extent.query and removes any that stay absent from the array for longer than tn_reconcile_grace_s (default 120 s). The sweep fails closed on any TN query error, so an unreachable array never deletes anything. If a map is still in use by a live VM, multipath -f refuses and the teardown skips for that tick without touching the sd paths underneath.

Three new per-storage knobs, all opt-out:

  • tn_reconcile (default 1) — master switch
  • tn_reconcile_grace_s (default 120) — floor-reconcile grace window
  • tn_reconcile_hint_ttl_s (default 600) — how long the writing node keeps its own marker before unlinking it

iSCSI + multipath only. No-op on NVMe/TCP and on iSCSI deployments with tn_use_multipath=0.

New Feature

tn_status_capacity_ttl_s per-storage knob. The status() capacity cache (both the in-process cache that pvestatd consults each tick, and the cross-process on-disk stamp at /run/truenas-plugin/status-* that a fresh pvesm status picks up) used to run on hardcoded 10 s / 15 s TTLs. On large clusters this is too aggressive: with N nodes polling S storages the resulting burst of pool.dataset.get_instance calls can saturate the TrueNAS middleware (one reporter observed iscsi.extent.query latency at ~12.5 s and pool.dataset.query at ~10.4 s on 11 nodes × 3 storages). The new knob caps both TTLs together; 60-120 s is a good starting range on cluster installs. Range 1-3600, default 10 matches pre-existing behaviour.

Validation

  • 3-node PVE cluster against TrueNAS 25.10. Flipped truenas-storage to tn_use_multipath 1, allowlisted the test WWID via multipath -a, allocated a 1 GiB test volume, activated on all three nodes, verified identical /dev/mapper/<wwid> + backing sd on each, deleted on one node. Hint file appeared byte-identical on every node via pmxcfs within ~1 s. Each peer's next pvestatd tick ran the hint-driven reconcile and the map + sd were gone. Peer syslog carried [TrueNAS] reconcile: tearing down stale WWID ... (sd=sdbl) and [TrueNAS] reconcile: deleted /sys/block/sdbl (wwid ...). No regression on non-multipath storages.
  • tn_status_capacity_ttl_s=60: wiped /run/truenas-plugin/status-*, took a miss, waited 20 s, took a stamp-hit at T+28 s (legacy 15 s stamp would have missed), confirming the knob extends both TTLs together.
  • Unit tests: t/iscsi/01-stale-wwid-reconcile.t (38 tests) and t/nvme/09-status-capacity-ttl-knob.t (11 tests). prove -I t/iscsi/lib t/iscsi/*.t and prove -I t/nvme/lib t/nvme/*.t both all-pass.

Install

wget https://github.com/truenas/truenas-proxmox-plugin/releases/download/v2.1.25-beta1/truenas-proxmox-plugin_2.1.25.beta1_all.deb
sha256sum -c <(curl -sL https://github.com/truenas/truenas-proxmox-plugin/releases/download/v2.1.25-beta1/SHA256SUMS)
dpkg -i truenas-proxmox-plugin_2.1.25.beta1_all.deb
systemctl restart pvedaemon pveproxy pvestatd

GitHub renames the asset's ~ to . on upload; the published filename is ..._2.1.25.beta1_all.deb even though SHA256SUMS lists ..._2.1.25~beta1_all.deb. The bytes are identical — verify with sha256sum by hand.

Known limits

  • #147 part 2 (guest write EIO on cache=writeback + aio=io_uring iSCSI disks during stale-map noise) is not addressed here. Reporter's correlation was with the stale-map thrash + the TTL change, which this release removes, so the EIO symptom may also go away — if not, it will need its own repro with query-blockstats snapshots and multipath -ll at the EIO moment.

Full Changelog

v2.1.24...v2.1.25-beta1

Don't miss a new truenas-proxmox-plugin release

NewReleases is sending notifications on new releases.