mattrobinsonsre/terrapod v1.10.1

on GitHub

Terrapod is a free, open-source platform replacement for Terraform Enterprise. This
is a security patch for the per-workspace cloud identity feature that shipped in
v1.10.0.

Everything in this release needs configuration that does not exist before
v1.10.0. A deployment on v1.9.x or earlier can upgrade and will experience none
of it; the OIDC issuer, its signing keys and the cloud-identity mint are all
absent from those lines. The affected population is whoever enabled
per-workspace cloud identity since v1.10.0. Two exceptions are noted under
Breaking Changes.

Breaking Changes

A patch normally carries none — see Versioning and
support.
These two are a deliberate, documented exception, justified by the scope
above: the form each replaces has existed in a shipped release for three days, so
the migration cost is near zero now and grows from here. Both fail in the safe
direction — the cloud refuses the exchange rather than widening it.

• The OIDC subject now identifies the workspace by id — sub is
  workspace:<name>@<id>[:phase:<phase>] rather than workspace:<name>. A
  workspace name is neither stable nor exclusive: any authenticated user holding
  workspace-create permission may take a free name, an admin may rename, and a
  delete frees the name at once. On AWS and Azure sub is the only thing a trust
  policy can bind, so a role trusting a name was assumable by whichever workspace
  held that name when a run executed. Update the trust policy to the id-bearing
  form; api.config.auth.oidc_issuer.subject_format: name restores the old
  subject for the length of a migration and is removed in 2.0.
• An unusable issuer configuration now fails startup rather than coming up
  degraded — an operator-supplied signing key that is not RSA, is under 2048
  bits, or does not use exponent 65537; and a resolved issuer URL that is not an
  absolute https URL without query or fragment. Note webhookIngress.tls=false
  produces an http issuer, which the chart rendered without complaint and no
  cloud will fetch.

Security

• Signing-key rotation now revokes every key it should. Rotation retired only
  the key currently signing, and during a propagation window that is the
  previous, already-retired key — so a second rotation retired nothing and left
  the incoming key published with no retirement date for any grace window to
  measure. Rotating twice in quick succession is what an operator does during a
  compromise. Rotation also takes an advisory lock, so two replicas cannot each
  leave their own key unretired.
• A replica that cannot reach the database stops serving revoked keys. The
  grace window was applied only when a reload succeeded, so an unreachable
  replica published a withdrawn key for as long as the outage lasted. It is now
  filtered on every read, bounding the exposure by the grace window instead.
• The runner-token endpoint checks the caller's certificate. It compared only
  the listener id in the path, which the caller chooses, so a valid listener
  certificate from any agent pool could obtain a runner token — and with it
  another pool's workspace cloud identity — for a run it did not own.
• A listener can only report on runs in its own agent pool. The job-status
  and live-log endpoints verified the caller's certificate against the listener
  id in the path — which proves only that the caller is the listener it claims
  to be — and then took the run from the path unchecked. So a certificate from
  any pool could report a Job's outcome for any run in the deployment, and the
  run reconciler reads that report back to decide whether a plan or an apply
  completed. The boundary is the pool rather than the owning listener, because
  listeners within a pool are interchangeable by design and any of them may
  answer a request the reconciler broadcast to the pool.
• A listener can no longer confirm a run. The status endpoint accepted any
  transition the state machine permitted, including planned to confirmed,
  which skips the apply capability check, the human confirmation, the staleness
  and plan-expiry guards and the mergeability gate. A listener may now report only
  the outcomes its Job can have produced.
• A runner token and a cloud-identity mint are live only while their phase is
  executing. Both previously refused only runs in a terminal state, and a
  plan-only run never reaches one — it finishes planned and stays there. Every
  speculative pull-request plan, module-impact plan and drift run is plan-only,
  which is to say the runs executing unmerged code were the ones the check could
  never fire on.
• The runner strips its own cloud identity tokens from the run log. With
  TF_LOG=DEBUG the AWS provider logs the token in full, inside the request body
  it sends to STS; its own redaction cannot match the token's format. The runner
  now removes those exact values from the engine's output. This covers the tokens
  Terrapod minted — it does not make TF_LOG safe for credentials of your own
  that a provider chooses to print.
• A run can no longer pin another workspace's configuration version. Not
  exploitable on any released version, because the archive is addressed by the
  run's own workspace, but it was one refactor from executing another workspace's
  code under this workspace's cloud identity.
• The OIDC issuer's signing keys are replicated between HA nodes, including
  the private half, so a promoted node signs with keys the clouds already trust
  rather than generating its own.

Bug Fixes

• A runner token's requested lifetime is validated — a non-integer returned a
  500 and a negative value minted an already-expired token. And
  runners.maxTokenTTLSeconds: 0 now means the default cap rather than no cap at
  all.

• A BYO signing-key change can be staged. api.oidcSigningKey.nextExistingSecret
  names a second key that is published in the JWKS and never used to sign, so an
  operator holding their own key can publish the incoming one, wait for the clouds
  to refetch, and then promote it — the same overlap a generated-key rotation has
  always had. Previously a BYO deployment published exactly one key, so federated
  runs failed from the cutover until each cloud refetched on its own schedule.

• Upgrading one HA node before the other no longer stalls replication. The
  signing keys added to replication in this release are a class an older peer
  does not know, and the follower asked it for them on every cycle. The 404 came
  back before the cycle committed, so it rolled back every change the cycle had
  applied and the cursor never advanced — stopping replication for every class,
  visible only as climbing lag. Since the documented procedure is to upgrade the
  follower first, that made this patch unsafe to take in the order the docs give.
  An unknown class is now skipped and retried, as the receiving side already did.

• Promoting a staged BYO signing key no longer restarts the API pods. During
  the documented promotion step both settings name the same key for one helm upgrade, which the chart describes as correct and the startup check refused —
  so following the procedure took every API pod down mid-rotation. It now warns
  and publishes the key once.

• A trailing slash on the issuer URL is accepted again. Terrapod strips it, so
  the value was harmless, but the check ran at import and refused to start. Note
  the warning it now emits instead: issuer matching is exact, so a cloud
  configured with the slash will reject every token, because the minted iss
  has had it stripped.

• The live log of a run using cloud identity no longer stalls. The redaction
  above holds back enough of the stream that a secret cannot straddle two reads,
  which for an identity token is most of a kilobyte — and a slow apply emitting
  one short progress line every ten seconds filled that in minutes, not seconds.
  The log of a long apply appeared frozen in multi-minute blocks. It now releases
  each complete line as it arrives.

Status

Beta — this is a security patch to the v1.10 line; v1.10.1 supersedes v1.10.0 for
any deployment using per-workspace cloud identity.

Full Changelog: v1.10.0...v1.10.1