cloudposse/atmos v1.221.0-rc.5

on GitHub

• route Terraform --all, --components, and --query through the scheduler-backed Terraform adapter
• build Terraform dependency graphs from dependencies.components first, with settings.depends_on fallback
• preserve query-path auth manager setup, store resolver bridging, YAML function processing, and per-component CI hook capture
• includes #2348 identity/auth fixes in this stack so local --identity terraform testing works
• include the credential-store concurrency-safety prerequisite discovered by concurrency validation
• keep effective scheduler concurrency fixed at 1 for this PR

Stacking

This PR is stacked on PR 2 and targets codex/dag-scheduler-core.

PR 4 is #2468 and is stacked on this branch to introduce plan-only --max-concurrency wiring.

Supersedes the earlier fork-headed draft #2462 now that the stack branches exist in cloudposse/atmos.

Draft note

This branch is back to the intended PR 3 review shape: Terraform --all, --components, and --query share the graph-backed scheduler path, but execution remains sequential.

The temporary ATMOS_EXPERIMENTAL_DAG_MAX_CONCURRENCY validation hook has been removed. User-visible plan concurrency now belongs to PR 4.

This branch retains the narrow credential-store concurrency-safety prerequisite discovered during validation:

• credential-store initialization no longer mutates global Viper env bindings per component and preserves ATMOS_KEYRING_TYPE precedence

Validation

• go test ./pkg/scheduler ./pkg/scheduler/adapters ./internal/exec -run TestExecuteTerraformQuery|TestExecuteTerraformQueryNoMatches|TestBuildTerraformDependencyGraph|TestExecuteTerraformAllUsesGraphBackedSequentialOrder|TestExecuteTerraformComponentsUsesGraphBackedSequentialOrder|TestExecuteTerraformQueryUsesGraphBackedSequentialOrder|TestExecuteTerraformKeepsIndependentComponentsSequential|TestBuildTerraformGraph
• go test ./pkg/auth/credentials
• go test -race ./pkg/auth/credentials -run TestNewCredentialStoreWithConfig_ConcurrentInitialization
• go test ./pkg/auth ./internal/exec -run TestCreateAndAuthenticateManagerWithAtmosConfig|TestSetupTerraformAuth|TestProcessComponentConfig_PropagatesAuthManager|TestProcessComponentConfig_AuthManagerGuardBranches
• built build/atmos and live-tested against a downstream stack with terraform plan --all and an explicit identity

Validation findings carried forward

• The first concurrency-4 validation run exposed an auth race: per-component credential-store initialization called global viper.BindEnv, causing fatal error: concurrent map writes. This PR fixes that narrowly in pkg/auth/credentials.
• Higher-concurrency validation also showed local Terraform working-directory contention when multiple logical aliases share one physical Terraform component directory. PR 4 keeps path-based locking while introducing plan concurrency.

Follow-up discussion

The longer-term way to unlock true parallelism for aliases sharing one physical Terraform folder would be per-node isolated workdirs plus isolated TF_DATA_DIR and generated files. That needs repo-owner discussion because it changes the operator debugging model: Atmos would need to decide whether and how to retain those per-node copies for inspection, how atmos terraform shell maps to them, and how cleanup/debug artifacts are managed.

Summary by CodeRabbit

• New Features

  • Graph-backed Terraform scheduler with deterministic dependency order, reversed destroy order, per-resource serialization, concurrency control, per-component output capture/hooks, and signal-aware cancellation.
  • New Terraform run options: --failure-mode, --max-concurrency, log-order, hide (including no-changes), and execution-summary file.
  • Line-prefixing writer for prefixed log output.

• Bug Fixes

  • Credential keyring type now respects ATMOS_KEYRING_TYPE and is safe for concurrent init.
  • Workdir sync/hash skips Terraform/OpenTofu runtime dirs.
  • More tolerant Git repo opening for worktrees.

• Tests

  • Large expansion of tests covering scheduler behavior, CLI options, concurrency, logging, auth, and new utilities.

• Add a ref:<name> version spec to --use-version (and version.use in atmos.yaml / ATMOS_USE_VERSION) that installs Atmos from the latest commit of a branch or tag, e.g. atmos --use-version=ref:main version.
• Accepts branch names, tag names, and slash-qualified refs for disambiguation: ref:main, ref:release/v1.199, ref:v1.199.0, ref:heads/main, ref:tags/v1.199.0.
• Resolves the ref to its full commit SHA via the GitHub API, then reuses the existing sha: install/cache path unchanged; ref versions always re-execute and fail hard on resolution errors.
• Docs (version/use.mdx), a minor blog post, and a roadmap milestone.

why

• Previously --use-version only accepted PR numbers (pr:1234), commit SHAs (sha:ceb7526), and releases — a branch name like main was rejected, even though branch/tag pushes already publish the same build-artifacts-* from the Tests workflow.
• ref: lets you pin a moving target once (ref:main) instead of chasing a new sha: after every merge, making it trivial to test unreleased fixes on a branch.
• The ref is re-resolved on every run so a mutable branch always tracks the latest build, while the SHA-keyed cache avoids reinstalling when the ref hasn't moved. Resolving to the full SHA also sidesteps GitHub's head_sha filter, which only matches full (not short) SHAs.

references

• Docs: Version Pinning
• Changelog: website/blog/2026-06-04-use-version-ref.mdx

Summary by CodeRabbit

• New Features

  • Support for git branches/tags via --use-version=ref: (resolves refs to commit SHAs and uses existing artifact download/cache).

• Behavior Changes

  • CI artifact selection now prefers the newest workflow run that contains the platform artifact (may pick in-progress or failed runs if they include the artifact).
  • Re-exec/version switching treats ref: like immutable versions (resolve → install/cache).

• Bug Fixes

  • Clearer, user-friendly error when a ref does not exist (with actionable hints).

• Documentation

  • Added CLI docs, blog post, and roadmap entry describing ref: usage and caching.

• Implement shell completion for semantic-typed flags and arguments (component/stack types)
• Add interactive prompting for missing required semantic-typed values
• Support custom component types in shell completions

What Changed

• New custom component type provider system (pkg/component/custom)
• Shell completion for semantic-typed arguments and flags in custom commands
• Interactive prompting for missing required semantic-typed values
• Extended command schema to support semantic types and components
• Comprehensive test coverage for completion and prompting functionality

Why This Matters

This feature enables custom commands to provide superior developer experience through:

• Tab completion for component and stack arguments/flags
• Interactive prompts for required semantic-typed values
• Support for custom component types beyond built-in types

References

• closes #1787
• closes #444
• closes #438

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Custom component types with registry support, CLI integration, and template access to resolved component data.
  • Enhanced CLI semantic completion and interactive prompting for selecting component and stack values.
  • Aggregated component listing across stacks for discovery and completion.

• Documentation

  • New guides, examples, and blog post demonstrating custom component types and workflows.
  • Schema updates to validate custom component manifests.

• Tests

  • Broad test coverage for completion, providers, processing, and stack handling.

• Add a new gist at gists/aws-ami-packer-github-actions/ demonstrating an end-to-end AWS AMI pipeline with Atmos + Packer + GitHub Actions:
  • Build a hardened Amazon Linux 2023 AMI with Packer, orchestrated by Atmos.
  • Validate it on a live test instance, optionally scan it, and gate promotion behind a manual approval.
  • Tag the approved image ScanStatus=approved and share it across AWS accounts.
• Drive the whole build from stack configuration (no hardcoded HCL) and operate the result through a tree of atmos ami custom commands (get-ami-id, tag, share, launch/terminate test instances, …).
• Include reference IAM/OIDC policies and an org SCP that enforces "launch only approved AMIs".
• Wire the gist into the docs-site file browser (tags + related-docs links) and announce it with a blog post.

why

• "How do I use Atmos + Packer to build AMIs, and automate the build → approve → share process?" is a frequent community question. This gist is a vendor-neutral, copy-and-adapt reference recipe that combines several Atmos features into one production-shaped workflow.
• Like all gists, it's shared as-is (not part of the CI-tested examples), so users adapt it to their environment and Atmos version.

references

• Gist: gists/aws-ami-packer-github-actions/
• Blog post: website/blog/2026-06-01-gist-aws-ami-packer-github-actions.mdx

Summary by CodeRabbit

• New Features

  • Added a complete gist showing an end-to-end AMI build/validate/approve/share pipeline using Atmos + Packer + GitHub Actions, with reusable setup and tool-install steps, approval gate, optional vulnerability scan, and cross-account sharing.

• Documentation

  • Added detailed README, customization checklist, policy templates, and a blog post documenting setup, governance (OIDC, IAM, SCP), local execution, and cleanup guidance.

• Add five new !git.* YAML functions that expose Git repository metadata from the origin remote: !git.repository (the <owner>/<repo> slug, e.g. cloudposse/atmos), !git.owner, !git.name, !git.host, and !git.url.
• Add the atmos.Resolve template function, which evaluates any Atmos YAML-function string (!git.*, !exec, !store, !terraform.output, …) at template-render time so its result can be composed with other strings and template variables in a single value.
• The new YAML functions are parsed generically (GitHub/GitLab/Bitbucket/Azure DevOps), support a fallback value, and work in both stack/component processing and atmos.yaml config preprocessing.
• Includes unit tests, per-function docs, two changelog posts, a roadmap update, and a follow-up PRD.

why

• Users needed the repository slug (and its parts) for tagging resources and building backend paths, previously only achievable by shelling out via !exec echo ${GITHUB_REPOSITORY:-$(git remote get-url origin | sed …)}.
• A bare YAML tag owns the entire scalar and Atmos renders Go templates before YAML functions, so composing a function result with extra text (e.g. prefixing workspace_key_prefix with the repo slug) was impossible without !exec; atmos.Resolve makes that composition native:

  workspace_key_prefix: '{{ atmos.Resolve .settings.context.repo }}/{{ or .metadata.name .metadata.component }}'

references

• Extends the existing Git YAML function family from the Git YAML Functions changelog.
• Docs: /functions/yaml/git.repository, /functions/template/atmos.Resolve.
• Follow-up: docs/prd/lazy-yaml-function-template-values.md (lazy-Stringer auto-deref so {{ .settings.context.repo }} evaluates without atmos.Resolve).

Summary by CodeRabbit

• New Features

  • Added Git repository metadata YAML functions (!git.repository, !git.owner, !git.name, !git.host, !git.url).
  • Added atmos.Resolve template function to evaluate YAML functions during template rendering for inline composition.

• Documentation

  • Added PRD, docs pages, blog posts, and roadmap entries describing the new YAML functions and atmos.Resolve.

• Tests

  • Added tests covering Git YAML tag resolution and the new template Resolve behavior.

• Chores

  • Updated link-checker configuration to exclude slow/intermittent targets.

• Fix the github/sts auth integration ignoring a successfully minted Atmos Pro STS token because mint() decoded the response with a flat struct instead of the canonical API envelope.
• Add a shared, reusable primitive — dtos.Envelope[T] + pro.DecodeEnvelope[T] — and route mint() through it so every Atmos Pro response unwraps the nested data payload through one sanctioned path.
• Fix the bug-masking test fixture (the simulated broker now emits the real envelope shape) and add a regression test asserting mint() persists 1 token, not 0, plus decoder unit tests including a canary that a flat payload decodes to empty data.

why

• Every Atmos Pro API route returns { "success": true, "status": 200, "data": { "tokens": [...], "excluded": [...] } }, but mint() decoded straight into the flat stsResponse (top-level tokens), so it always read 0 tokens — the CLI logged GitHub STS: no tokens granted, never wrote the git insteadOf config, and cross-repo import: calls fell back to the ambient GITHUB_TOKEN and failed with remote: Repository not found, even though the server had minted a valid token (HTTP 200, so no error surfaced).
• The existing e2e test passed only because its simulated broker returned the unwrapped {tokens,excluded} shape the real server never sends; matching the fixture to the real envelope and adding the regression/canary tests prevents this whole class of "decoded a Pro response without the envelope" bug from recurring.

references

• mint() was the only Pro call bypassing the shared AtmosApiResponse envelope that ExchangeOIDCToken / LockStack already use.

Summary by CodeRabbit

• Bug Fixes

  • Clearer STS error messages and correct unwrapping of canonical API envelopes.
  • Prevent ambient tokens from being baked into Git URLs by honoring insteadOf rewrites (including file-mode).
  • Avoid invalid git checkout/fetch for empty refs by fetching default branch and skipping bad checkouts.
  • Warn when component source is misplaced under metadata and accept simple-form source strings.

• New Features

  • Provision credential brokers before Git source detection so token rewrites apply.

• Tests

  • Expanded tests covering envelope decoding, STS handling, broker provisioning, git insteadOf, and default-ref behavior.

• Documentation

  • Added fix notes on STS envelope/token-shadowing and updated PRD guidance for source.