github fabriziosalmi/proxmox-lxc-autoscale v2.0.4
v2.0.4 — correctness, and one backend fewer

latest release: v2.0.5
one hour ago

Correctness release. Nothing here is a new feature; every item is something the daemon was doing wrong, and most of it was found by running the daemon against a real Proxmox node rather than by reading the code.

The daemon no longer stops itself

The main loop caught three exception types. Anything else escaped it and ended the process, and the unit file said Restart=no, so one TypeError from a malformed pct output stopped autoscaling for that host until a human noticed. The only symptom was an absence of log lines.

The loop now survives what it does not predict, the unit restarts on failure, and it picks up the systemd confinement it never had: NoNewPrivileges, ProtectHome, PrivateTmp, ProtectKernelTunables, RestrictSUIDSGID.

Measurement no longer reports absence as idle

pvesh_stat_method returned 0.0 for a container missing from /cluster/resources. That made the method succeed, so the three fallback methods were never consulted, and 0.0 reads as idle, which argues for taking resources away.

The reserve is subtracted once

get_total_cores and get_total_memory already apply the configured reserve, and adjust_resources applied it again. A declared 10% CPU reserve was 19%; a declared 2048 MB memory reserve was 4096 MB. On a four-core host the daemon believed it had two cores to hand out.

A container can reach its own max_cores

An increment that would overshoot the ceiling was discarded rather than clamped, so a container at three cores with an increment of two never reached a ceiling of four. Observed on a live node sitting at three for cycle after cycle at 65% CPU against a 50% threshold.

Horizontal scaling no longer fills the disk

The snapshot taken before each clone was never removed, and the grace period was recorded only after a successful scale-out. A failing scale-out therefore retried every poll and left one LVM snapshot per attempt on the source container: four in ninety seconds under test, 288 a day at the default interval, with nothing pruning them.

A boost stays temporary

A boost record is the only memory of a container's original size, and two paths dropped it, which made the elevation permanent and silent. reconcile read a pct config that returned nothing as "the container no longer exists", when it equally means a timeout. evict_stale was given the containers whose metrics were read successfully rather than the containers that exist, so one failed read dropped a boost every cycle.

Secret masking actually masks

The filter was attached to the root logger. In Python a logger's filters do not apply to records propagated from child loggers, and every module uses getLogger(__name__), so almost nothing the daemon wrote was masked. There was a live path for it: an Uptime Kuma failure logs the exception, and the push token sits in the URL.

Configuration that does nothing says so

Unknown keys are reported for DEFAULT and for tiers, not only for horizontal groups. That mechanism is how min_instances was documented, written by users, and discarded without a word for months. A container listed in more than one tier is reported too, instead of resolving silently by key order in a YAML file.

Removed: the REST API backend

backend: api was accepted for a year and did nothing. The implementation existed, with tests, and no production module imported it: every operation went through pct. An operator setting it believed they had moved off root and off SSH, which the security guide recommended, and had not.

Connecting it was measured rather than assumed: 26 call sites invoke pct directly, three of them pct exec which has no REST equivalent, and the abstraction is missing the eight methods the ASG design specifies plus the per-guest node resolution it never had. That is the first three issues of the ASG milestone, not a wiring job.

backend: api is now refused at startup with a message explaining why. backend: cli is unchanged and remains the default; use_remote_proxmox still drives a remote node over SSH. Building a real one is #56.

New: scripts/live-check.py

The unit suite did not catch any of the above. It mocks run_command, and these defects all live on the other side of that mock.

This script runs the daemon against a real Proxmox node and asserts what happened to the cgroups and to the container configuration. Six checks, each corresponding to a defect that shipped. It is safe on a node with real guests: every pre-existing guest is placed in ignore_lxc, the daemon's own view is asserted to contain only the throwaway container before anything starts, and every pre-existing container config is checksummed and compared at the end.

sudo ./scripts/live-check.py --yes-i-understand-this-creates-a-container

This release was verified with it on Proxmox VE 9.1.7: six checks pass, seven pre-existing guests untouched.

Upgrading

Re-run the installer or pull main. Existing configuration is compatible, with one exception: backend: api now stops the daemon at startup instead of being ignored. If you have it set, change it to cli, which is what was running anyway.

Don't miss a new proxmox-lxc-autoscale release

NewReleases is sending notifications on new releases.