mattrobinsonsre/terrapod v1.10.9

on GitHub

Terrapod is a free, open-source, self-hosted platform replacement for Terraform Enterprise. This is a fix release on the 1.10 line, correcting runner admission on a fixed-size agent pool — a defect reported by @the3venthoriz0n — plus the diagnostic runbook for the symptom it produces.

Bug Fixes

• A burst of queued runs no longer admits more runners than listener.maxConcurrent — the cap was enforced once per admission pass rather than per claim. An SSE run_available event starts a drain, and each event was dispatched as its own task, so several passes could run concurrently, each reading occupancy before any of the others had launched anything. Every one of them then saw the same free capacity and claimed against it. On a fixed-size pool receiving a burst — a monorepo push that queues many workspaces at once, or a scheduled drift sweep — that launched more runner Jobs than the pool was configured to allow, which is visible as runner pods pending against a namespace quota, or as nodes scaling out past what the pool was sized for. Admission passes are now serialised and occupancy is re-read before each claim. Note that listener.maxConcurrent is, and always was, pool-wide rather than per-listener-pod; docs/deployment.md and docs/runners.md said otherwise in five places and are corrected. With several listener replicas sharing a runner namespace the bound is maxConcurrent + replicas - 1, because a Job created by one replica is visible to the next list but not instantaneously. Nothing to configure. (#2399)
• The cap can no longer stall a pool that has free capacity — the same fix, narrowed. Counting every non-terminal Job as occupying a slot meant a Job that would never produce a pod held one indefinitely: a Job blocked by a namespace ResourceQuota reports no pods, no failures and no conditions, so it looked identical to one that was simply slow; a Job stuck Terminating under foreground deletion stayed listed with no terminal condition; and a Job retrying under backoffLimit had the same shape between attempts. Any of those three silently consumed a slot for the lifetime of the listener pod, and a pool at its cap then admitted nothing at all. A Job now occupies a slot while it has a pod, or for a grace window after creation, and a Job being deleted releases its slot immediately. The occupancy list also carries a request timeout and the drain a deadline, so a hung API call releases the admission lock rather than holding it for the pod's life. (#2399)
• The queue-backlog runbook is usable again — its diagnostic told an operator to read .attributes["active-runs"] and .attributes.capacity from the listener list. Neither is serialised, so the documented command returned nulls, in the runbook for the exact symptom the admission fix above produces. The step now reads fields the API returns and separates the three ways to read occupancy, because they disagree by design: the terrapod_listener_active_runs gauge (which counts the runner namespace, carries no pool label so it must be aggregated with max rather than sum, is refreshed once per heartbeat and holds its last value when a read fails — so it reads stale and low during apiserver trouble, and returns nothing at all unless the chart's scrape is enabled), and a direct kubectl count of runner Jobs, which is normally higher than the admission figure because admission ignores a Job being deleted, one finished but not yet reaped, and one that has had no pod for more than a minute. A count above listener.maxConcurrent is therefore the expected reading rather than a broken cap; the previous text invited an operator to conclude otherwise. (#2406)
• The runbook now covers a backlog cause that has no other symptom — when a listener cannot list Jobs it refuses to claim rather than risk over-admitting, and while that lasts it heartbeats normally, answers /ready with 200, and holds its last occupancy figure. The queue simply stops draining. There is now a diagnosis step with the two log lines to grep for and what they point at. (#2406)

Other

• A gate now validates every API attribute the documentation tells a reader to read against the snapshot of what the serialisers actually emit, so a runbook step cannot again instruct an operator to read a field that does not exist. (#2406)
• The Playwright container image used by this line's end-to-end CI is pulled through the project's mirror rather than from Docker Hub, which removes an anonymous-pull rate limit from the release gate. No effect on a deployment. (#2361)

Status

Stable — a fix release for the previous minor. No schema, API or configuration changes; upgrading requires no action. Operators running a fixed-size agent pool with listener.maxConcurrent set should take this release.

Full Changelog: v1.10.8...v1.10.9