v2.29.0 (Two things you asked for, and five defects the tests all agreed were fine)
A minor release: two new capabilities and two new API surfaces —
GET /api/deploy/pending and a csr field on certificate creation.
Deploy hooks can wait for a maintenance window
Requested by a user deploying to RDS: "we don't know exactly when [the renewal
runs]. This would mean that the deployment-script runs at a random time."
The diagnosis was right. A certificate renews when it is due — the sweep
fires at 02:00 with an hour of jitter, and only for whatever is inside the
threshold that night — and the deploy hook then ran immediately. For a hook that
restarts a database or reloads a load balancer, that is an outage at an hour
nobody chose.
A window separates the two. The certificate still renews whenever it is due;
the deploy is held until the window next opens.
Absent means immediately, so nothing changes for any hook that does not use it.
The same field works on typed deploy targets, which is usually where it is
wanted: a Kubernetes secret rollout is exactly that kind of deploy.
The queue is on disk, so a restart between the renewal and the window does not
lose the deploy. Two renewals before the window opens produce one deploy,
not two. Deploy Now ignores the window — pressing it is choosing that
moment. A held deploy is invisible everywhere else (the certificate renewed, the
history is empty, nothing failed), so GET /api/deploy/pending lists them with
the window they are waiting for and its next opening.
A window may cross midnight and belongs to the day it starts on, so a
Friday-night 22:00-04:00 window is open at 02:00 on Saturday without you
having to tick Saturday too. Daylight saving is handled by the zone: on the
spring-forward day a 02:00-04:00 Rome window opens at the local 03:00,
because 02:00-02:59 does not happen.
Certificates whose private key never reaches CertMate
Requested by a user with appliances that generate their key on the device and
cannot export it: "These devices typically expose the CSR, but never the
private key."
Submit the CSR — through the API or the create form — and CertMate obtains the
certificate and manages the chain, without ever seeing a key.
curl -X POST https://certmate.local/api/certificates/create \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d "$(jq -n --arg csr "$(cat device.csr)" \
'{domain: "api.example.com", dns_provider: "cloudflare", csr: $csr}')"These report private_key_state: "external" and do not ask to be renewed
for the missing key. That distinction is load-bearing: v2.27.1 made a keyless
certificate report needs_renewal, because that is what a restored share-safe
backup looks like. Without a separate state a CSR-only certificate would have
been reissued on every nightly sweep.
Renewal re-submits the stored CSR. certbot renew can never renew these —
certbot says so itself when it issues one — so CertMate re-runs issuance with
the CSR it kept. Rotating the key on the device means submitting a new CSR.
Typed deploy targets cannot serve one: they all publish the key alongside the
certificate, and they now say exactly that instead of failing with a generic
error on every renewal. Shell hooks work normally, and CERTMATE_KEY_PATH is
left unset rather than pointed at a file that does not exist.
Full documentation: docs/csr-only-certificates.md.
Saving deploy hooks deleted every deploy target
Found while building the windows. Settings -> Deploy has no editor for the typed
deploy targets added in v2.24.0: the page never reads them, so it never sends
them back, and the save assigned the whole deploy_hooks block. Opening that
screen and pressing Save — without touching anything — silently removed
every configured Kubernetes secret target. No error, no audit entry naming the
loss, and the next renewal simply stopped publishing to the cluster.
A key the payload does not mention is now left as it was; an explicit value,
including an empty list, still replaces.
Two advertised requirements files could not be installed on amd64
requirements-storage-all.txt and requirements-infisical-storage.txt carried
infisical-python>=2.3.6. That version ships an aarch64 wheel, macOS wheels and
Windows wheels — and no manylinux x86_64 wheel and no source distribution. It
installs on the arm64 image and cannot be installed on the amd64 image at
all. Both files are advertised as EXTRA_REQUIREMENTS build options.
It was invisible because every dependency check ran on one architecture while
the images are published for two. CI now resolves each optional requirements
file for both published architectures and fails when they disagree.
The extras install layer could move a pin the base layer holds
Each EXTRA_REQUIREMENTS file was a separate pip resolution that knew nothing
about what the base layer pinned. Measured: a second-layer
pip install "cryptography<46" silently took 46.0.7 down to 45.0.7, below the
floor pyopenssl==26.0.0 requires — an image where certbot --version no
longer answers. It had already happened once here, with an extras file dragging
certbot off its pin in a build that reported success. Extras now install under
a constraints file, so pip refuses at build time and names the conflict.
SECURITY.md gains a Supply-chain posture section recording what is
guaranteed, what is not (no hash verification), and why hash pinning is not
attempted incrementally.
settings.json declares its shape, and a file from the future is refused
There was already a product version on disk and a downgrade warning. That
field moves on every release and cannot answer "is this a shape I understand",
and it logged and continued — leaving an older process free to write the file
back without the fields it never knew about.
settings_schema_version moves only when the shape does, and a file declaring a
newer one stops the process rather than degrading. CERTMATE_ALLOW_SCHEMA_DOWNGRADE=1
proceeds anyway for an operator who has read the release notes.
Under the hood
create_certificate went from 590 lines and cyclomatic complexity 115 to 19,
in seven reviewed steps, each pinned by a golden-master test of the exact
certbot command. Two defects fell out of it: a propagation-delay formula written
four times, one of which had drifted, and a broad exception handler that turned
an introduced NameError into a wrong value with nothing logged.
Domain-to-path validation lived in six places with three different return
conventions, and the end-of-string anchor had been corrected in one and left as
$ in three — so a name with a trailing newline was accepted at some entry
points and refused at others. One home now, and every entry point is tested
against it.
A note on how the CSR work was verified
The real-CA end-to-end test found five defects that all 46 unit tests had
agreed were fine — including that certbot refuses to overwrite its own output in
--csr mode, which made the first renewal of every such certificate fail
before it ever contacted the CA. Issuance would have looked perfect, then
silence until expiry. Each now has a test that bites.
v2.28.0 (Two silent failures, and two things you can now do without curl)
A minor release: DELETE /api/inventory/<fingerprint> is a new endpoint, which
is what makes it minor rather than a patch. Four of the six changes came from
user reports.
Adopting a certificate ran no deploy hook, and said nothing
Adoption is a real issuance. POST /api/inventory/<fingerprint>/adopt reads the
observed certificate, then issues a new one over ACME and brings the domain
under management — it does not import the certificate you already had.
Every other issuance path announces itself on the event bus afterwards. The
adopt handler did not, and three subscribers hang off that bus, each of which
silently ignores what it does not recognise: notifications, deploy hooks,
and the deployment-status cache.
The missing notification is the visible half. The dangerous half is that no
deploy hook ran, so the load balancer kept serving the OLD certificate — and the
cached "deployed & matching" verdict was not evicted, so the dashboard agreed
with it for up to cache_ttl. Nothing told the operator.
Reported by @ratiugtun, who diagnosed it correctly down to the call site and
asked which of two designs we wanted. Adoption now publishes
certificate_created, which every subscriber already honours, and carries an
adopted marker so the alert reads Certificate Adopted rather than
"Created". A dedicated certificate_adopted event would have had to be taught
to five places, and missing any one would have fixed the notification while
leaving the deploy hooks dead — the shape of the original bug.
The inventory could not forget anything
Reported by @SpeeDFireCZE: add a domain to the discovery configuration, scan,
remove the domain again, and the discovered certificates stay in the list with
no way to remove them.
That was not a filtering bug. The inventory had no delete at all — no
manager method, no route. Removing a domain from the discovery configuration
correctly stops future scans finding it, but every row already recorded stayed
forever, and the only way to clear one was deleting inventory.db.
There is now a Forget button on every inventory row and a
DELETE /api/inventory/<fingerprint> endpoint. The certificate itself is never
touched; CertMate only forgets that it observed it, and the endpoint observations
go with the record.
It forgets an observation rather than suppressing one: a domain still in the
discovery configuration is recorded again on the next sweep. Both the dialog and
the API response say so, so nobody deletes the same row after every scan.
SESSION_TIMEOUT_HOURS did not reach the browser
The session lifetime existed as three numbers: a constant nothing read, the
environment variable, and a cookie max_age hardcoded to eight hours in two
places. Only the environment variable was configurable, and it governed only the
server-side record.
So SESSION_TIMEOUT_HOURS=24 still logged you out after eight. The shorter
direction is quieter and it is the one the documentation recommends —
SESSION_TIMEOUT_HOURS=1 for high-security deployments left a cookie alive
seven hours after the session it named was already dead.
Both cookie mint sites now take the lifetime from the AuthManager, so the record
and the cookie cannot drift. The default is unchanged at 8 hours;
SESSION_TIMEOUT_HOURS is now in .env.example.
SAN entry is a chip editor
Asked for twice, independently, by @Peter-A-Tech. The SAN field was one text
input holding a comma-separated string — awkward at three names, unusable at the
~20 real certificates carry, because you could not see what you typed and
changing one in the middle meant retyping the lot.
Names are now chips: Enter or a comma adds one, each has its own remove button,
Backspace on an empty field removes the last. Pasting a list works — commas,
semicolons, spaces or line breaks, de-duplicated and normalised — which is the
case the request was actually about. A widget that took one name at a time would
have been worse than the field it replaces at exactly that size.
Nothing changed underneath: the API already accepted san_domains as a list.
A wildcard's probe host is settable from the UI
Also reported by @SpeeDFireCZE. A wildcard certificate reports "cannot be
verified" — correctly, because a wildcard does not cover its own apex, so
probing example.com for *.example.com compares against the wrong name. The
fix was a deployment_host, and the only way to set it was the API.
Settings → Probe now has a Host field, next to the Port and Protocol it
already had. For a wildcard it suggests a covered name (www.<apex>), shows the
host in the configured list, and clearing it removes the host so a wrong one is
correctable in place.
The README no longer promises what the chart refuses to render
CertMate is single-instance by design: the renewal scheduler runs inside the web
process, so a second replica is a second scheduler issuing against the same
store. The Helm chart already enforced this — it fails at template time when
replicaCount != 1.
The README led with "easy scaling" and "high availability deployments" while the
constraint sat in a docker-compose comment about 73% of the way down. Worse,
docs/architecture.md carried a High Availability section giving operational
steps for multi-instance deployments — shared storage, a load balancer with
sticky sessions — replicated verbatim into all four translations. Following it
produces duplicate ACME orders, not availability.
All five now describe active/standby, which is what works, and say the part that
makes single-instance acceptable out loud: renewal begins 30 days before expiry,
so a failover has weeks of slack.
v2.27.1 (Six fixes, four of them things that were quietly wrong)
A patch release. No new endpoints and no new capability — but four of these
were failures an operator could hit without ever seeing an error, and two came
from user reports.
A certificate with no private key was reported as healthy
get_certificate_info decided a certificate existed by looking at cert.pem
alone. A domain directory holding a valid certificate and no key came back as
exists: true, days_left: 74, needs_renewal: false — and the renewal sweep
left it for another six weeks, for an instance that cannot complete a TLS
handshake for that name.
That is exactly the state restoring a share-safe backup produces, because those
deliberately carry no key material. An operator verifying a recovery the obvious
way — the API lists my certificates with sane expiries — was told the node was
fine.
Certificates now report private_key_present, private_key_state and usable,
and a missing or mismatched key forces renewal instead of waiting for an expiry
that is irrelevant. A key that does not match the certificate counts as
unusable too: cert.pem from one issuance beside privkey.pem from another cannot
handshake either.
One deliberate exception, because getting it wrong would have been worse than
the bug: the storage-backend listing path fetches the certificate without the
key on purpose, to avoid pulling private keys out of a secrets backend for a
dashboard. It reports unknown, not missing, and does not force renewal.
Configuring a second DNS provider deleted the first
Reported by a user. Saving one provider replaced the whole dns_providers
subtree, so configuring Route53 removed the Cloudflare credential. The provider
stayed selectable on the certificate form, so the failure surfaced later as an
issuance that could not find credentials. Providers are now merged. Removal is
unaffected — it has its own endpoint.
Single sign-on could fail permanently after an IdP change
Reported by a user connecting Authentik: every login failed with "Algorithm of
'RS256' is not allowed", and every algorithm they tried failed in turn.
The accepted signing algorithms come from the identity provider's discovery
document, which was fetched once and cached for the life of the process — and
the cached client was keyed on the issuer URL alone. So changing the signing
algorithm at the IdP, or rotating the client secret in CertMate, had no effect
until the container restarted.
The cache now follows the whole configuration, and a failure on the algorithm
drops the cached document so the next attempt refetches. The log names both the
algorithm and what the IdP advertises, and the login page says which IdP setting
to check.
Adding an API_BEARER_TOKEN to an existing install locked the operator out
Enforcement asked whether the operator had supplied a token; authentication
checked the token that was stored. Those agree on a fresh install. They diverge
for anyone who ran once without a token — one was generated — and then added or
rotated API_BEARER_TOKEN: the first-run screen asked for the token they had
just configured and answered 401, and the way out was a reset script.
The supplied token is now authoritative and is reconciled at startup.
This is a behaviour change: an operator who rotated the token through the UI
while leaving a stale API_BEARER_TOKEN set will find the variable winning.
The backup list says which archives carry private keys
Every backup made before v2.26.0 contains the private key of every certificate,
while its manifest says secrets_masked: true. That has been a published
advisory since v2.26.0 — something an operator has to go and read, while the
archives sit on their disk described by a manifest that says the wrong thing.
The list now reads the archive itself and marks those rows. An archive that
cannot be inspected reports unknown, never no keys: conflating those would
be the same false reassurance the old manifests gave.
Rebuilt on a current base image
66 fewer container CVEs (216 to 150 on the built image, nothing new introduced).
The three remaining criticals are perl-base and Debian has no fix for them
yet.
Upgrading
Nothing to do, with one thing worth knowing: if you set API_BEARER_TOKEN and
have since rotated the token in the UI, the environment variable now wins.