github amayer1983/docksentry v1.65.0
v1.65.0 — Event stream, CPU diagnostics, multi-host monitoring

latest releases: v1.66.0, v1.65.1
3 hours ago

Two fixes in here that had been quietly wrong, and a change of mind about which resource to blame when a container dies.

The exit code in crash alerts was always 0

v1.63.0 added the exit code to crash-restart alerts so you could tell "this crashed on its own" from "that was my own docker stop". It reported 0 every time.

docker inspect cannot answer that question. A container the restart policy has already brought back is running again by the time the monitor looks, and a running container reports ExitCode: 0 — and so did the previous sweep, since it was running then too. There was no moment at which polling could have seen the real number.

The live event stream carries it. Alerts now say exit 137 where they said exit 0.

Only the local host was being watched

Update checking has been multi-host since v1.62.0. The crash and health monitor never was — it predates hosts and kept its single local backend. On a two-host install a container died with exit 42 on the second host and nothing was reported, while the same instance logged "Managing 2 hosts" at startup.

Every managed host gets its own monitor now, and alerts carry the host: nas/nginx. Without the prefix, the same stack on two machines produces two identical messages and no way to tell which box is on fire.

The memory picture is now taken at the moment of death

The monitor polls every 60 seconds, so the snapshot naming the biggest memory consumers could be up to a minute stale — by which time the culprit may have let go, been restarted, or be the thing you killed in a panic.

There is a live event stream for this. The die event arrives 0 ms after the daemon stamps it; the snapshot now lands 0.08 seconds after the death instead of at the next sweep.

The watcher never sends anything itself. Alerting stays with the poller, which already owns debouncing, cooldowns and the event log — the watcher only records evidence the poller picks up when it builds its message. MONITOR_EVENTS=false turns it off, and if it cannot start at all, Docksentry polls exactly as it did before.

And CPU, because memory was the wrong suspect

@NotRetarded pushed back with the thing that should have made me doubt all of the above: no memory limit on the container that died, no visible memory pressure on the host, but CPU spikes on the container itself.

So I measured whether CPU starvation can kill a container. Directly, no — one pinned to 1% of a core ran happily for half a minute. But the indirect route is real, and it produces a death that cannot be told apart from an out-of-memory kill:

same container, same shutdown handler, same `docker stop -t 5`
--cpus=1.0    → exit 0, clean
--cpus=0.005  → exit 137, OOMKilled false

Starved of CPU it could not answer SIGTERM inside its grace period, so Docker escalated to SIGKILL. Exit 137, OOMKilled false — byte-identical to what the kernel's OOM killer leaves behind. A CPU-starved healthcheck also flips to unhealthy while the container keeps running, which is a second route into a restart.

An alert reporting only memory therefore points at the wrong resource in exactly the case that prompted the work. CPU now comes from the same docker stats call — no extra process — and only when something is actually holding the processor, since contention is the whole point and a CPU line on an idle host would bury the signal it exists to carry.

🔁 unifi crashed (exit 137) and was restarted by its restart policy at 16:14:47 (restart #1).
Host memory (used/total): 14.8/15.6 GB · Swap 3.9/4.0 GB
Top memory at event time: some-new-app 9.1GiB · unifi 2.2GiB
Top CPU at event time: some-new-app 198%

Also

Host memory on the status page, next to disk, from the same /proc/meminfo the alerts read — so the page and the message can never disagree about how full the machine is.

Upgrading is the usual docker pull ghcr.io/amayer1983/docksentry:latest, or nothing at all with auto-selfupdate on.

Don't miss a new docksentry release

NewReleases is sending notifications on new releases.