kOlapsis/maintenant v1.3.8

Retention cleanup / Private Certificate / Resources history graph

on GitHub

Fixes the retention cleanup, which could not keep up past ~3 containers and let the database grow without bound. Also brings resource history into the UI, adds trust for an internal CA, and fixes agent containers disappearing.

Retention could never catch up

Three resource purges ran a single DELETE ... LIMIT where every other purge already looped until drained. Deletion was capped at 1000 rows per hour whatever the number of containers, against ~360 rows/hour/container of ingestion ? past roughly three containers the purge fell behind for good. Two instances reported 2.14 GB and 1.48 GB after less than two months, 99.9 % of it raw samples. (#46)

A pass now drains each table, runs at startup instead of waiting a full hour, and comes back a minute later while a backlog remains. On an instance with a real backlog the first pass deleted 346,776 rows in 2.26 s.

Two nearby settings were broken too:

• auto_vacuum never took effect. It was set after journal_mode=WAL had already written the file header, so SQLite silently ignored it, on every installation, not just the ones that reported it. New databases come up in incremental mode; existing ones keep NONE and get a startup warning.
• The WAL was unbounded. journal_size_limit was never set and wal_autocheckpoint reached only one of the four pooled connections. Both now apply to every connection, bounding the -wal file at 64 MiB (416 MB had been observed).

Window, interval and batch size are configurable via MAINTENANT_RETENTION_SNAPSHOTS, MAINTENANT_RETENTION_INTERVAL and MAINTENANT_RETENTION_BATCH_SIZE.

Separately, the 7-day chart now reads the hourly rollup - the same buckets it already displayed, precomputed - instead of grouping a week of raw samples. That was the only thing forcing a week of raw retention, which drops to 48 hours: roughly 850k rows and 300 MB for 14 containers, against 2.8 MB for the aggregates covering 90 days.

Resource history in the UI

Container panels gain a Resources tab next to Details and Logs: CPU, memory, network and block I/O over 1h/6h/24h/7d, plus the per-container alert thresholds. The charts existed but were mounted nowhere, and could not draw in any case, they were initialised before their canvases existed.

Certificates / internal CA

New MAINTENANT_CA_CERT (and --ca-cert): a PEM bundle appended to the system roots, applied to endpoint probes and chain validation alike, and loaded before either mode starts so agents pick it up too. A host that answers but presents an untrusted certificate is reported as degraded rather than down. (#36)

Bug fixes

• Agent containers disappeared. Reconcile compared the database against the local Docker environment while listing containers without an agent filter, so anything from a remote agent was archived and never came back. Reconcile is restricted to the local runtime, containers unarchive when their agent reports, and each agent sends a full inventory every 30 seconds.
• Remote container logs returned 502.
• Broken UI behind an auth proxy. With a short session TTL (Authentik, oauth2-proxy, Cloudflare Access), the service worker served the cached shell, so the SPA booted after the proxy session was gone and rendered failed panels while every API call landed on a login page. Expiry is now detected, and re-authentication is allowed to complete.

Upgrade notes

• Automatic DB migration (26) adds block I/O to the hourly rollup. No manual steps; back up the .db first as usual.
• The first startup does real work. On a backlogged instance the initial pass deletes everything that accumulated, expect a burst of disk I/O, logged with its counters.
• Reclaiming space already used. Since auto_vacuum never applied, existing databases stay at their high-water mark: they stop growing, but the file does not shrink. Only an offline VACUUM returns the space, procedure and disk requirement in troubleshooting.
• Block I/O on the 7-day chart reads 0 for buckets aggregated before the upgrade, and fills back in over the next two days.

Pull the new image to upgrade.