cloudposse/atmos v1.196.0-rc.4

on GitHub

what

• Add Google Cloud Storage (GCS) backend support to !terraform.state Atmos YAML function
• Implement performance optimizations (client caching, retry logic, extended timeouts)
• Create unified Google Cloud authentication system for consistency across GCP services
• Update documentation with GCS backend usage examples and authentication methods

why

The !terraform.state YAML function allows reading the outputs (remote state) of components in Atmos stack manifests directly from the configured Terraform/OpenTofu backends.

Previously, the !terraform.state YAML function only supported:

• local (Terraform and OpenTofu)
• s3 (Terraform and OpenTofu)

This PR adds support for:

• gcs (Google Cloud Storage - Terraform and OpenTofu)

With GCS backend support, users can now leverage the high-performance !terraform.state function instead of the slower !terraform.output or !store functions when using Google Cloud Storage for Terraform state storage.

Implementation Details

GCS Backend Features

• Full Authentication Support: JSON credentials, service account file paths, and Google Application Default Credentials (ADC)
• Service Account Impersonation: Support for impersonate_service_account configuration
• Performance Optimizations:
  • Client caching to avoid recreating GCS clients for repeated operations
  • Retry logic with exponential backoff (up to 3 attempts) for transient failures
  • Extended timeouts (30 seconds) to match S3 backend performance
• Robust Error Handling: Graceful handling of missing state files and detailed error context
• Resource Management: Proper cleanup and explicit resource management

Usage

The GCS backend works seamlessly with existing !terraform.state syntax:

# Get the `output` of the `component` in the current stack
subnet_id: !terraform.state vpc private_subnet_id

# Get the `output` of the `component` in the provided `stack` 
vpc_id: !terraform.state vpc dev-us-east-1 vpc_id

# Get complex outputs using YQ expressions
first_subnet: !terraform.state vpc .private_subnet_ids[0]

GCS Backend Configuration

The GCS backend supports all standard Terraform GCS backend configurations:

# atmos.yaml
components:
  terraform:
    backend_type: gcs
    backend:
      gcs:
        bucket: "my-terraform-state-bucket"
        prefix: "terraform/state"
        
        # Authentication options (choose one):
        
        # Option 1: JSON credentials content
        credentials: |
          {
            "type": "service_account",
            "project_id": "my-project",
            ...
          }
          
        # Option 2: Service account file path  
        credentials: "/path/to/service-account.json"
        
        # Option 3: Use Application Default Credentials (ADC)
        # (no credentials field needed - uses environment/metadata)
        
        # Optional: Service account impersonation
        impersonate_service_account: "terraform@my-project.iam.gserviceaccount.com"

Performance Benefits

Compared to !terraform.output, the !terraform.state function with GCS backend:

• ✅ No Terraform execution - Reads state directly from GCS
• ✅ No provider initialization - Skips all module and provider setup
• ✅ No varfile generation - Bypasses Terraform configuration preparation
• ✅ Cached clients - Reuses GCS clients for multiple operations
• ✅ Parallel execution - Multiple state reads can happen concurrently

Testing

• Comprehensive Test Suite: 100% test coverage for all new functionality
• Mock Implementations: Complete interface-based testing for GCS operations
• Authentication Testing: Validates all credential types and authentication flows
• Error Scenario Coverage: Tests for missing files, network failures, and invalid configurations
• Caching Validation: Ensures client caching works correctly across operations
• Retry Logic Testing: Validates exponential backoff and failure recovery

Backward Compatibility

• ✅ No breaking changes to existing configurations
• ✅ Existing backends (local, s3) remain unchanged
• ✅ Same function syntax - no new parameters or options required
• ✅ Graceful fallbacks - continues to work with !terraform.output and !store functions

Files Changed

Core Implementation

• internal/terraform_backend/terraform_backend_gcs.go - GCS backend implementation
• internal/terraform_backend/terraform_backend_gcs_test.go - Comprehensive test suite
• internal/terraform_backend/terraform_backend_registry.go - Register GCS backend
• internal/terraform_backend/terraform_backend_utils.go - Updated error messages

Unified Authentication System

• internal/gcp/auth.go - New unified Google Cloud authentication (created)
• internal/gcp/auth_test.go - Authentication tests (created)
• pkg/store/google_secret_manager_store.go - Updated to use unified auth
• internal/gcp_utils/gcp_utils.go - Removed (replaced by unified auth)

Configuration & Documentation

• internal/exec/terraform_generate_backend.go - GCS backend validation
• website/docs/core-concepts/stacks/yaml-functions/terraform.state.mdx - Updated documentation
• errors/errors.go - Added GCS-specific error types
• go.mod - Added GCS storage dependency

Migration Guide

For users currently using !terraform.output or !store with GCS-stored state:

Before (slower)

# Using !terraform.output (requires Terraform execution)
vpc_id: !terraform.output vpc dev-us-east-1 vpc_id

# Using !store (requires separate state management)  
vpc_id: !store google-secret-manager dev/vpc/vpc_id

After (faster)

# Using !terraform.state (direct GCS state access)
vpc_id: !terraform.state vpc dev-us-east-1 vpc_id

Simply update your backend configuration to use gcs and replace function calls - no other changes needed!

Summary by CodeRabbit

• New Features

  • GCS-backed Terraform state support and unified Google Cloud authentication integration.

• Bug Fixes

  • Stricter backend config validation with clearer error responses and updated supported-backends messaging.

• Tests

  • Comprehensive unit tests added for GCS backend behavior and GCP authentication handling.

This PR addresses multiple authentication issues when using Atmos in containerized environments with mounted credential files:

1. Auth Pre-Hook Error Propagation - Terraform execution now properly aborts when authentication fails (e.g., Ctrl+C during SSO)
2. AWS Credential Loading Strategy - New LoadAtmosManagedAWSConfig() function provides proper isolation while preserving Atmos-managed profile selection
3. Noop Keyring Validation - Container auth now properly isolated from external environment variables
4. Whoami with Noop Keyring - atmos auth whoami now works in containerized environments
5. Test Coverage - Added test to verify auth errors properly abort execution

Changes

1. Auth Pre-Hook Error Propagation (internal/exec/terraform.go:236)

• Problem: Errors from auth pre-hook were logged but not returned, causing terraform execution to continue even when authentication failed (e.g., user presses Ctrl+C during SSO)
• Fix: Added return err after logging auth pre-hook errors
• Impact: Terraform commands now properly abort on auth failures

2. AWS Credential Loading Strategy (pkg/auth/cloud/aws/env.go)

• Problem: SDK's default config loading allowed IMDS access and was affected by external AWS_PROFILE, causing conflicts in containers
• Solution: Created LoadAtmosManagedAWSConfig() function that:
  • Clears credential env vars (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
  • Preserves profile/path vars (AWS_PROFILE, AWS_SHARED_CREDENTIALS_FILE, AWS_CONFIG_FILE)
  • Allows SDK to load from Atmos-managed credential files
• Impact: Proper isolation while still using Atmos-managed profiles

3. Noop Keyring Credential Validation (pkg/auth/credentials/keyring_noop.go)

• Problem: Used unrestricted config.LoadDefaultConfig() which allowed IMDS access and was affected by external AWS_PROFILE
• Fix: Changed to use LoadAtmosManagedAWSConfig()
• Impact: Container auth now properly isolated from external env vars

4. Whoami with Noop Keyring (pkg/auth/manager.go)

• Problem: Whoami() expected credentials from keyring, but noop keyring returns ErrCredentialsNotFound by design
• Fix: Added check for ErrCredentialsNotFound and fallback to buildWhoamiInfoFromEnvironment()
• Impact: atmos auth whoami now works in containerized environments

5. Test Coverage (internal/exec/terraform_test.go)

• Added TestExecuteTerraform_AuthPreHookErrorPropagation to verify auth errors properly abort execution
• Test validates that terraform doesn't continue on auth failure
• Updated test fixture to include required name_pattern configuration

Technical Details

The key insight is that Atmos sets AWS_PROFILE=identity-name (in pkg/auth/cloud/aws/setup.go:59) but the previous isolation approach cleared ALL AWS env vars including AWS_PROFILE. This caused the SDK to look for a non-existent [default] section.

The new LoadAtmosManagedAWSConfig preserves AWS_PROFILE while still preventing external credential conflicts.

Test Plan

• go build . - Build succeeds
• go test ./internal/exec -run TestExecuteTerraform - All terraform tests pass
• TestExecuteTerraform_AuthPreHookErrorPropagation - New test passes
• Verified test fails when fix is removed (terraform continues execution)
• Verified test passes when fix is restored (terraform aborts on auth error)

References

Fixes authentication issues in containerized environments with mounted credentials.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added --login and cached-credentials-first flows across auth commands; whoami now shows validation and expiry.
  • Atmos-managed credentials moved to XDG-compliant locations; improved shell enter/exit messages.
  • Geodesic helper script for building/testing in containerized environments.

• Bug Fixes

  • Terraform pre-hook errors now abort execution.
  • Improved propagation of user-abort during authentication.

• Documentation

  • XDG migration guides and Geodesic/CLI docs updated.

• Tests

  • Broad expansion of auth, AWS credential, auth-context and output-propagation tests.