mattrobinsonsre/terrapod v1.7.4

on GitHub

A patch release for Terrapod — an open-source platform replacement for Terraform Enterprise. It fixes two Helm chart defects introduced by the token signing key added in v1.7.3, one of which can cause intermittent 401 failures on agent runs.

Upgrade first, then restart the API once

If you are on v1.7.3 and have seen agent runs fail with 401 on artifact uploads or binary-cache fetches, upgrade and then run:

kubectl -n <namespace> rollout restart deploy/<release>-api

The fix stops the signing key changing again, but it cannot retire an API pod that is already holding a superseded key. One restart clears it. Deployments that never saw 401s need no action beyond the upgrade.

Bug Fixes

• A signing key change now rolls the API pods together. v1.7.3 moved the token signing key from a value derived from the database URL to a generated Kubernetes Secret, injected via secretKeyRef. Environment variables from a secretKeyRef are captured when a pod starts and never refresh, and the key is cached per process — so if the Secret changed while the Deployment was running, older pods kept the old key while newer ones used the new one. Runner tokens minted by one pod were then rejected by the other, and because requests are load-balanced, roughly half of each run's API calls returned 401 while the rest succeeded. The API pod template now carries a checksum/ annotation over the key, so a change rolls every pod together and a rotation is atomic instead of leaving the fleet split.

• GitOps renderers no longer generate a new key on every render. The chart generated the key with randAlphaNum, relying on a lookup of the existing Secret to reuse it on subsequent runs. That works under helm install and helm upgrade. It does not work under helm template, which is what Argo CD and Flux run: lookup returns nothing there and .Release.IsInstall is always true, so every render produced a fresh key and rotated it underneath the running deployment. Generation is now conditional on lookup actually working; when it does not, the chart emits nothing and helm.sh/resource-policy: keep leaves the existing Secret untouched.

• Listener-only releases no longer render the token signing Secret. The key is read by the API alone — a listener requests runner tokens over HTTP and never signs anything — but a release with api.enabled: false still created one. That left a generated key nothing reads, and in a namespace shared with the API release, two releases declaring overlapping objects.

• docs/rate-limiting.md now shows a verification curl against a path this release line actually serves.

Notes for operators

• No key is rotated by upgrading. An API release still renders the Secret and reuses the stored value; the upgrade is a no-op for the key itself.
• A stray Secret from a listener-only release is not deleted automatically. helm.sh/resource-policy: keep means Helm leaves it in place. Remove it by hand once you have confirmed the API release owns the one it reads.
• Setting the key explicitly remains the most predictable option. api.tokenSigningKey.existingSecret points at a Secret you manage; the chart then generates nothing and the value only changes when you change it.

Status

Stable — 1.7 is the current supported line, alongside 1.6 for security fixes.

Full Changelog: v1.7.3...v1.7.4