fabriziosalmi/certmate v2.22.0

v2.22.0 (Audit release — silent failures, gates that did not gate)

on GitHub

v2.22.0 (Audit release — silent failures, gates that did not gate)

A 360-degree read-only audit of the codebase produced 25 findings, all of them addressed here. One was split: log rotation covered both the application log, fixed below, and the tamper-evident audit chain, which cannot be rotated by the same means and is tracked as a design issue rather than rushed (#437). Every fix below carries a regression test that was written failing first — the suite went from 1,770 to 1,974 tests.

The common thread is worth stating plainly: almost every defect in this release failed silently, or reported the opposite of what happened. A backup that omitted the CA key still reported success. A renewal that failed every night sent no alert. A save that was rejected showed a green "saved". Four CI gates were incapable of failing at all.

Data loss and disaster recovery

• The unified backup now contains the private CA, client certificates, the CRL and the audit trail (#409). It archived only certificates/ and settings.json, so everything under data/ — the private CA signing key, every client certificate it signed, the CRL and the tamper-evident audit chain — was silently absent from every backup ever taken. An operator using the private-CA feature who followed the documented restore procedure came back unable to issue, renew or revoke a single client certificate, with nothing at backup time to warn them. Both subtrees now travel under a data/ prefix; the restore side is allow-listed to exactly those two, so a tampered archive cannot use the new branch to write a settings.json around the deploy-hook validation gate. Restored private material is written 0600.
• A restored backup renews again (#410). A ZIP cannot carry symlinks, so certbot's live/<domain>/*.pem came back as flat files; certbot reports a parsefail on such a lineage and skips it, so after a restore every scheduled renewal failed — or exited 0 reporting renewed: false — forever, silently, until the certificates expired. The repair function for exactly this case existed but was only ever called when reissuing. The lineage symlinks are now rebuilt from the preserved archive/ generation, at restore time and again at the top of every renewal, so an instance restored by an older version heals itself. The repair is all-or-nothing: a partially-restored archive leaves the flat PEMs untouched rather than producing a half-linked lineage.
• A large audit log survives DR (#409 follow-up). Restored data/ entries were subject to the 10 MB per-entry cap that suits PEM files, so an append-only audit log that outgrew it was skipped with a warning while the restore still reported success. Entries now stream under a 512 MB ceiling, an overflow is an error rather than a warning, and a failed extraction removes the partial file — a truncated CA key is worse than an absent one, because it looks restored.

Authentication and authorization

• Disabling an SSO user now locks them out (#408). The OIDC login path never consulted the row's enabled flag, which the local-password path has always checked. An administrator disabling a user — an action that also revokes their live sessions — achieved nothing against anyone who authenticates through the identity provider: the next SSO login minted a fresh 8-hour session with their old role. Both the subject-match and email-link branches now refuse a disabled user.
• An identity-provider demotion now demotes (#408). The role was derived from the claims only when a user was first provisioned, so removing someone from an administrator group in Keycloak or Okta left them an administrator in CertMate indefinitely. The role is re-derived on every login; set sync_role_on_login: false in the OIDC block for deployments where the provider only authenticates and roles are managed locally.
• Secrets are masked completely in API responses (#432). MaskedString revealed the first four and last four characters of every secret it touched — roughly twenty fields, including DNS API tokens and the Google service-account key — and GET /api/settings, where those fields live, is readable by the viewer role. The web settings route already masked the same values as a flat sentinel, so the API was simply the weaker of two answers about identical data. Both now use the same sentinel.
• API-key expiry is a date, not a string (#432). It was stored verbatim and compared as text, so "31/12/2026" sorted after every ISO timestamp and never expired — the opposite of what the operator wrote — and a JSON number raised an uncaught TypeError inside the authentication loop, rejecting every bearer token on the instance. Expiry is now validated and normalised on write and compared as a datetime on read, failing closed: a key whose expiry cannot be parsed is treated as expired. The key listing shares the same helper, so the "expired" badge can no longer disagree with whether the key actually works.
• The API rate limiter can no longer be reset at will (#420). The bucket was a hash of the caller-supplied bearer token, computed before the token was validated, so varying the Authorization header produced a fresh bucket on every request — the limits never fired, which also left the deliberately unauthenticated OCSP and CRL endpoints unprotected, and flooded the limiter until its eviction discarded legitimate per-IP buckets. The per-key bucket remains (clients behind one NAT must not share a limit), under a coarse per-IP ceiling of 600 requests/minute that is checked first. Retention was also an hour against a 60-second decision window; it now follows the window.
• certmate.service runs one worker (#411). It shipped --workers 4 while sessions, session revocation and the login rate limiter are per-process in-memory state — the Dockerfile already used one worker. Revoking a compromised administrator's session took effect on roughly one request in four.

Silent failures

• A failed unattended renewal now notifies (#417). The scheduler audited and logged the failure but never published certificate_failed, the event the notifier turns into an email or a Slack message. A certificate whose DNS credentials had been rotated could fail every night for a month while the operator, who had configured both channels, heard nothing. Failures while checking a domain publish too.
• Deleting a certificate now deletes it from the storage backend (#419). The API removed the local directory and the settings entry but never called the storage backend, so with Vault, AWS Secrets Manager, Azure Key Vault or Infisical configured, the full PEM bundle — private key included — stayed in the external store indefinitely while the UI showed the certificate gone. A backend outage no longer fails the local deletion but is logged at ERROR naming the domain and the backend.
• The external-copy warning is set on renewal, and cleared on recovery (#423). A storage failure during renewal was only logged, so a Vault token expiring mid-life meant nightly renewals kept succeeding locally while the disaster-recovery copy silently went stale. In the other direction, one failed store at creation left the warning in metadata forever, long after the backend recovered — and an operator who learns to ignore a stale warning will ignore the real one.
• Saving notification settings reports the truth (#415). The handler read the response body without checking the status, so a 403 or a 503 rendered as a green "Notification settings saved". An operator could configure SMTP, see success, close the tab, and never receive an alert.
• A failed load of the SSO settings no longer offers a form that wipes them (#424). On a failed GET the component left its configuration at the blank defaults and rendered the full editable form with an enabled Save. An administrator opening the tab during a transient 500 saw "SSO not configured", changed something, saved — and wrote the blank defaults over a live configuration, disabling SSO for the organisation. The form is now hidden behind an explicit failure state with a retry.

Certificate correctness

• Reissuing preserves per-certificate configuration (#421). create_certificate wrote metadata.json wholesale, so Edit and Reissue dropped the deployment probe settings the PATCH endpoint stores there — a mail server configured for smtp-starttls:587 reverted to https-tls:443 and the dashboard reported it "not deployed". The merge is driven by an explicit set of keys a reissue owns, which matters in both directions: a reissue that drops a domain alias still clears it.
• Client-certificate renewal keeps the validity and refuses CSR identities (#422). Renewal ignored days_valid, so a 730-day certificate silently halved on every renewal, and it always generated a new server-side keypair — which for a certificate issued from a client-supplied CSR produced something the client cannot use, while stamping the original superseded and non-renewable. CertMate never held that private key and cannot renew such an identity: renewal now refuses with an error saying what is needed, before touching the original, and the nightly sweep skips those certificates rather than writing an identical failure every tick.

Reliability

• The log stream no longer pins a worker thread at 100% CPU (#418). It tailed the file with while True: readline() — no sleep, no exit condition — so the generator spun forever and a client disconnect was never detected. Eight page loads wedged the single-worker container. It now sleeps between empty reads, emits a keepalive comment (which is what surfaces a dead client), and gives up after 30 idle minutes.
• File logging, rotated by construction (#431). CertMate never wrote an application log file: the parameter existed and nothing passed one, so logs/certmate.log — the path the log stream reads — was created by nothing. Set CERTMATE_LOG_FILE to write one; it is always a rotating handler (10 MB, five backups, both tunable), and there is no code path that produces an unbounded log. stdout remains the default, which is what docker logs and every shipper expect. An unwritable path warns and continues: a certificate manager that refuses to boot because it cannot write a log has failed at the wrong thing.

CI and release gates

Four gates reported success without checking anything. This matters more than any single bug in the list above, because it is why a green board could not be trusted.

• The container CVE scan can fail, and scans the right image (#412). It ran with no exit code — so it passed regardless of findings — against the floating :latest tag, which on a push to main is the previous release's image, and it was skipped entirely on pull requests. It now rebuilds the image under test from the buildx cache, runs on pull requests, and fails on a CRITICAL with a fix available. The long tail of unfixable base-image findings stays reported rather than blocking, because a permanently red gate is one nobody reads.
• release.sh publish refuses a red main (#413). It gated on version, tag, notes and emoji but never asked whether the code builds, so a dependency bump landing between prepare and publish was enough to tag and push a broken tree — and the tag push is what publishes :latest. It now reads the run status for the exact commit being released, treating a cancelled run as a failure, and ignores the self-hosted workflows that would otherwise hang a release whenever that host is offline.
• The UI suite can no longer pass having asserted nothing (#414). Any Chromium launch failure became a skip, and CI installed the browser without system dependencies, so a missing library turned all 17 tests into skips and the job into a green check. The browser launch is now verified explicitly, "no browser" is a failure rather than a skip, and the job fails if every collected test skipped.
• Lint and coverage have teeth (#428). flake8 ran with --exit-zero in CI, the Makefile and the release script, which made the .flake8 config advisory; the failing set now also covers the bug classes that were already clean. Coverage had no floor at all — it could fall to zero with CI still green — and now has one, as a ratchet.

Documentation

• The multi-account DNS API is documented at the address it actually answers on (#416). /api/settings/dns-providers/<provider>/accounts appears 35 times across the README and five translations and returns 404; the real routes are /api/dns/<provider>/accounts. PUT .../default-account never existed in any version — the default travels as set_as_default in the account payload, which is what the examples now show.
• HOST and FLASK_DEBUG are no longer documented as working (#429). Neither is read anywhere. HOST is the security-relevant one: an operator setting HOST=127.0.0.1 believed they had bound to loopback while the service listened on every interface. The documentation now says how to actually get that.
• Corrected against the code (#430): the CSV batch cap is 100, not "100 to 30,000"; ARM v7 images are not published; SECURITY.md declared 2.19.x supported; the /health example in the Docker Hub README was invented; and docs/testing.md documented a test runner, a pre-commit configuration and a coverage requirement that do not exist.

Hygiene

• The published image no longer ships the test suite (#426). .dockerignore patterns without ** match the build-context root only, so tests/, docs/, scratch/ and the agent worktrees under .claude/ were copied into the production image.
• Command-palette entries work when you are already on the page (#425). Eleven of them navigate to /settings#<tab>; a fragment-only change does not reload, and both pages read the hash exactly once, so choosing a tab you were already on changed the URL and did nothing.
• Copying a new API key works over plain HTTP (#427). navigator.clipboard is undefined in a non-secure context, which is how CertMate is commonly run on a LAN, so the copy button silently did nothing — on a token that is shown once and unrecoverable afterwards. All three copy paths now share one implementation with a fallback.