fabriziosalmi/certmate v2.32.0

v2.32.0 (Stop handles nobody called, a lock that excluded nobody, and a queue with no bottom)

on GitHub

v2.32.0 (Stop handles nobody called, a lock that excluded nobody, and a queue with no bottom)

A minor release: eleven changes, most of them from the same twenty-category
audit of v2.30.0 that produced v2.31.0, and all of them one shape — a mechanism
that was published, looked like it worked in review, and did nothing.

The renewal sweep can take on a certificate that says how it was issued

A certificate present on disk but absent from settings could not be renewed.
The sweep named it and moved on, because nothing knew which DNS provider,
account or CA had issued it — so a certificate CertMate had created itself, and
whose settings entry had been lost to a failed restore or a hand-edited
settings.json, expired while the dashboard listed it.

It now reads that certificate's own metadata.json. When the metadata names a
provider, an account and a CA, and that provider is configured on this
instance right now, the sweep renews it and writes the settings entry back
after the renewal succeeds — so the next sweep finds it the ordinary way. When
the metadata cannot answer, or the provider it names is gone, nothing changes:
the certificate is still named as one the sweep cannot renew, which is the
honest answer rather than a guess at credentials.

The sweep summary gained reregistered, so a run says how many certificates it
adopted rather than leaving it to be inferred.

Async issuance refuses work it is not going to reach

POST /api/certificates/create with async: true accepted everything. The pool
has two workers, each running a certbot subprocess that takes tens of seconds to
minutes, and the queue behind them had no limit — so a client in a retry loop,
or a script walking a domain list, could park hundreds of jobs there and get a
202 Accepted and a job id for each one, for certificates that would not be
attempted for hours.

CERTMATE_ISSUANCE_QUEUE_LIMIT (default 20, clamped 1-500) bounds how much
unfinished issuance may exist at once — queued and running together. Past it:

HTTP 429
{
  "code": "ISSUANCE_QUEUE_FULL",
  "error": "Too much issuance is already in progress",
  "hint": "20 job(s) queued or running against a limit of 20. Retry once some finish, or raise CERTMATE_ISSUANCE_QUEUE_LIMIT / CERTMATE_ISSUANCE_WORKERS."
}

Finished jobs do not count towards it, and the depth check and the registration
happen under one lock, so two requests arriving together cannot both take the
last slot.

X-CertMate-API-Version moves from 2.0 to 2.1. Minor, not major: 429 was
already reachable on every /api/ path through the rate limiter, so a client
that handles 429 needs no change. What is new is a condition, not a type.

Two gauges came with it, because a 429 with nothing to graph leaves an operator
guessing whether to raise the limit or find what is stuck:
certmate_issuance_queue_depth and certmate_issuance_queue_limit, plus
certmate_event_dispatch_backlog for the other queue in the process.

The four state directories are configurable

certificates/, data/, backups/ and logs/ were derived from the module's
own location and nothing could move them. That is a good default — it is what
makes the image and the systemd unit work with no configuration — but it was the
only answer available, so a deployment wanting certificates on one volume and
backups on another had to bind-mount over the install tree.

CERTMATE_CERT_DIR, CERTMATE_DATA_DIR, CERTMATE_BACKUP_DIR and
CERTMATE_LOGS_DIR now move them one at a time. The defaults are unchanged,
and the boot-time writability check that turns a bad mount into a clean startup
error still runs on whatever was resolved.

create_app(test_config=...) also reads them. It had accepted a configuration
that could not affect any of this and reported no error.

Shutting down actually stops things

IssuanceExecutor.shutdown() had no caller anywhere in the tree, and the
slow-request watchdog's stop event was handed to the container and never set.
Both read in review like a safety net. What was wired covered half the
components and one of the two deployment shapes: python app.py stopped the
scheduler and drained the event bus, and under gunicorn — which is how the image
runs — an atexit hook drained the bus and nothing else.

So a container stopped during an async issuance left the pool's queued jobs
recorded nowhere. The job registry is in memory and dies with the process: the
certificate an operator asked for never appeared, and no line anywhere said so.

There is now one ordered shutdown, registered on atexit and called from the
Ctrl-C path: scheduler, then issuance pool, then watchdog, then event bus —
producers before consumers, because draining the bus first drains a queue the
scheduler is still filling. Nothing is joined; what could not be finished is
named:

Issuance executor stopped with 2 job(s) unfinished: create a.example.com (running),
create b.example.com (queued). These were not completed and are not retried on
startup; re-run them if the certificate is still needed.

A webhook that will never work is not retried three times

The webhook retry left its loop on success and on nothing else. A receiver
answering 404 — a Slack app removed, a path typo, a token revoked into a 401 —
was sent three times with 1s + 2s of sleep between the attempts, on every
notification, for as long as it stayed configured. Under a renewal sweep that is
once per certificate, on the sweep's own thread.

Two kinds of failure are permanent now: the receiver answered with a 4xx that
describes the request rather than the moment (408, 425 and 429 excluded, and 5xx
is what backoff is for), or CertMate refused to send at all — no URL, a
non-http scheme, an SSRF-guarded target, a channel missing its token, a broken
payload template. DNS, connection refused, TLS and timeouts still get the full
backoff.

An HTTP rejection also carries its status now instead of arriving as prose, so
the delivery log reads attempts: 1, status: 404 rather than
attempts: 3, status: null — the difference between reading it as a flaky
receiver and reading it as a webhook to fix.

The activity page read 827 KiB to show 100 lines

The audit log tail walks backwards in 8 KiB blocks, and its stop condition
counted blocks against the number of entries asked for. An audit line is
around 170 bytes, so a hundred entries live in two blocks and the loop read a
hundred and one. Measured on a 3.2 MiB log of 20,000 entries:

Writing the test for that found a second defect in the same function: the result
was sliced by lines rather than entries, so a log with noise between the entries
returned 33 rows for a request of 100, with nothing to say why.

A credentials file that cannot be written says so

For twelve DNS providers the credentials builder ended except (KeyError, Exception): return None, with no log line. None legitimately means "this
provider authenticates through the environment", so an OSError — a read-only
config directory, a full disk, a permission the container lost — became "no
credentials file needed". certbot then ran without --dns-<provider>-credentials
and complained about missing plugin credentials, the operator went looking at
their DNS account, and the error that actually happened existed in no log, no
metric and no response. Only the template lookup is caught now.

A lock that excluded nobody, and a status nothing ever assigned

safe_file_write took an exclusive flock on the temporary file the same call
had just created — a name no other process can open — under the comment "Use
file locking for safety", and released it before the rename that publishes the
content. It excluded nothing, and it stopped readers of that function from
asking how concurrent writes are handled. The atomicity was always the rename,
and both docstrings now say so, including what the lock did not do.

certmate_certificates_by_status{status="renewal_failed"} was exported as 0 on
every scrape of every deployment, because nothing assigned it and nothing could:
a certificate whose renewal failed is still valid, expiring soon or expired.
An alert written on it could never fire, which reads as "no renewal has ever
failed here". It is removed; failures are counted by
certmate_certificate_renewals_total{status="failure"}, which the sweep
increments and which the shipped CertMateRenewalsFailing rule alerts on.

Two gates that now catch something

The complexity ratchet was a single --max-complexity=115. That is the tree's
real worst, and a test kept it from drifting above what the tree contains — but
one number for 1,700 functions has to be set by the worst of them, so everything
below the worst was ungated and a new function at 100 passed. Each function has
its own ceiling now: a general limit of 40, and the thirteen already above it
pinned at what they measure. It failed the first change made after it, which is
what a gate is for.

The list of endpoints the SDK uses was nine literals checked against a running
container. The SDK calls nineteen, and the ten it did not know about include
certificates/{}/deploy, settings/dns-providers and dns/{}/accounts — so
the drift check written to catch a renamed endpoint had stopped covering half
the client. It is read out of the SDK's own source now and compared against the
application's route table, verbs included, in the unit tier.

v2.31.0 (One error shape, five metrics nobody was writing, and a key that was parsed on every read)

A minor release: nine changes, all of them from a twenty-category audit of
v2.30.0 and the issue queue behind it. One is a breaking change for API
clients, which is why it is first.

Breaking: code in an error body is always a string

Measured against a running instance, three unauthenticated 404s produced three
different bodies. code was a string symbol on the application's own errors
(CERTIFICATE_NOT_FOUND, DOMAIN_OUT_OF_SCOPE), the HTTP status integer on
failures raised by the framework, and absent from the most common 404 of all —
including on two endpoints reporting the identical condition, one of which set
a code and the other nothing. A client could not branch on the field without
checking its type first, which is the same as not having the field.

It is one type now, everywhere:

{"error": "Certificate not found for domain: example.com", "code": "CERTIFICATE_NOT_FOUND"}

Failures from the HTTP layer carry message and status as well, and status
is where the number went. If you were reading code as an integer, read
status instead — that is the whole migration. X-CertMate-API-Version moved
from 1.0 to 2.0 to say so, because the rule written beside that constant
makes a retyped response field a major bump.

docs/api.md now lists the 39 error codes the API emits. None of them had been
documented, so branching on one meant reading the source.

Two smaller things came with it: a 404 from a flask-restx resource no longer
appends "you have requested this URI ... but did you mean" with matching routes
from the route table, and seven certificate-not-found responses that carried no
code now carry the one their sibling download endpoint already used.

A certificate read stopped re-validating its private key

private_key_state answers whether a usable private key sits beside a
certificate, and it answered by loading the key — which is where OpenSSL
validates it. Measured on one machine: 51.6 ms for RSA-2048, 275.5 ms for
RSA-4096, 0.020 ms for EC P-256. End to end, reading one certificate's
information took 58.7 ms with a real RSA key beside it and 0.19 ms without one.
The default key shape is RSA-2048, so on the common installation almost all of
that was re-answering a question whose two input files had not moved.

It runs once per domain in the dashboard listing, in every Prometheus
collection — serially, on the request thread — and in every renewal sweep. The
answer is now remembered per (sha256(privkey.pem), sha256(cert.pem)): hashing
both costs tens of microseconds, and unlike an mtime or a TTL it is exact, so a
renewal, a re-key or a torn publish all change a digest and the comparison runs
again. In the project's own benchmark, 120 RSA domains went from 5,631 ms to
9 ms on a repeat listing.

That benchmark could not see any of this before: its fixture generated EC keys,
the one shape where loading a key is free. It measures both now.

Five metrics were exported and never written

certmate_certificate_requests_total, the creation duration histogram,
certmate_acme_rate_limit_hits_total and the cache hit/miss counters were
declared, scraped and permanently empty — six recorder methods had no caller
anywhere in the application. A missing metric is a gap somebody notices while
building a dashboard; a dead one is a flat line and an alert that never fires,
which reads as nothing going wrong.

Certificate creation now records its outcome and duration on all three of
its exits, where renewal already did. A refusal that is a CA rate limit is
counted apart from an ACME error, because the two mean opposite things about
what to do next: an error is worth retrying, a rate limit is what retrying
causes. The same distinction gives the API an ACME_RATE_LIMITED code whose
message says to wait. Cache hits and misses are counted at the lookup.

certmate_dns_provider_api_calls_total was removed rather than left
exported: CertMate does not talk to a DNS provider API in this process — certbot
does, in its own subprocess, and the DNS-alias hook is a separate short-lived
process whose counters never reach this registry.

The four alerts monitoring/prometheus-alerts.yml had omitted with a note
saying the metrics could never fire are back with them.

A shutdown says what it could not deliver

The event bus had no stop, drain, close or join at all. Its workers are daemon
threads on purpose — joining them would let a 300-second deploy hook hold
shutdown open until the runtime killed it anyway — and the cost of that choice
was invisible: a container stopped during or just after a renewal sweep
discarded whatever was queued. That is typically the deploy hook for a
certificate that had just been renewed, so the service went on serving the old
certificate while the dashboard showed a success, and nothing recorded it.

Queued work now gets a bounded deadline to start, and whatever is left is
logged with its event and its domain, so the line found after a restart names
the certificates to redeploy by hand. CERTMATE_EVENT_DRAIN_SECONDS sets the
deadline (default 5, clamped 0-60).

Two answers that outlived the thing that invalidated them

A storage migration that could not enumerate the source returned the empty
result it had accumulated, which the API rendered as HTTP 200, success: true,
"Migration completed: 0/0 certificates migrated", with an audit record stamped
success. An operator moving off a backend before decommissioning it was told it
had worked. Enumerated nothing and could not enumerate are different answers and
no longer share a return value. A source that genuinely holds no certificates
still succeeds; a per-certificate failure is still reported per certificate.

The certbot readiness probe ran once per process and froze its answer for
the life of that process. A transient failure at startup made the instance
permanently unready, which under an orchestrator is a restart loop rolling the
same dice; and a certbot that broke afterwards never turned /health/ready red.
The answer now has a TTL (CERTMATE_CERTBOT_PROBE_TTL, default 300, clamped
30-3600), a lock so a burst of scrapes runs certbot once between them, and both
health endpoints ask for a current answer instead of reading the startup
snapshot.

An expiry that could not be read is not an expired certificate

Reported three times by the same person (#88, #92, #786). null <= 0 is true in
JavaScript, so a certificate whose validity CertMate could not parse rendered
with a red Expired badge; v2.2.4 guarded the four sites the dashboard had then.
The command palette, written after that fix, was not guarded — days > 0 ? ... : 'Expired' is false for a missing value, so the same wrong answer arrived by the
other door. Unknown is now its own state, and a test fails the build on either
shape anywhere in the frontend.

#786 stays open: that report is about the certificate list, whose guards hold,
so the cause there is not yet known and closing it would repeat what happened
to #92.

The image ships what it runs

The runtime stage copies an allowlist rather than the build context, and the
allowlist named directories — so COPY scripts/ put all fourteen files in the
image, of which the running process reads two. release.sh with the entire
release procedure, the lockfile regenerator, five CI gates, the theme codemod
and the walkthrough recorder no longer ship. The image test now asserts the
contents of that directory in both directions.

Also

• The per-module coverage floors watch modules/core as well as the HTTP
  layer. storage_backends.py could have lost all 877 of its covered
  statements and the project figure would have landed at 76.3%, above the 75%
  floor, with a green build.
• The README's Quick Start was telling operators to configure Route53, Azure,
  Google Cloud DNS and PowerDNS through .env. None of those variables is read
  anywhere; only CLOUDFLARE_TOKEN bootstraps an account from the environment.
  The guard that caught the same fiction in .env.template now asks the README.
• CERTMATE_BACKUP_PASSPHRASE is offered in .env.example and in the compose
  file. It decides whether an automatic backup can restore the instance at all,
  and it appeared in neither of the two files an operator edits.
• The compose file's bind mounts carry the preparation docs/docker.md
  documents.
• CONTRIBUTING.md's pre-push commands are character-for-character the ones CI
  runs; two rows had drifted.

New environment variables

• CERTMATE_EVENT_DRAIN_SECONDS (default 5, clamped 0-60) - how long a
  shutdown waits for queued event dispatches to start before naming what is
  left.
• CERTMATE_CERTBOT_PROBE_TTL (default 300, clamped 30-3600) - how long the
  certbot readiness answer stays good for.