runs-on/runs-on v2.9.0

on GitHub

This is a large release, with many internal and external changes. Please review the first section below carefully.

Note 2025-10-17: please use v2.9.1 instead, since it includes important fixes.

Potentially breaking changes

• Update default linux image to ubuntu24. Set image=ubuntu22-full-x64 or image=ubuntu22-full-arm64 if you want to keep using Ubuntu22.
• When a job is canceled due to a spot interruption, all the failed jobs from that failed workflow will get retried, instead of only the first job interrupted. Fixes #192.
• Prometheus metrics endpoint removed, along with the ServerPassword stack parameter. RunsOn now ships with OTEL integration. Fixes #322. The /metrics endpoint anyway had a long-standing issue where some prometheus scrapers were unable to reach the AppRunner endpoint due to how the Envoy proxy from AWS handles requests.
• disk=large and disk=default labels are deprecated. If present, they will be automatically translated into the new volume label, but once you have adopted v2.9.0, you should update your workflows for future upgrades.
• RunnerLargeDiskDeviceName and RunnerDefaultDiskDeviceName are removed (now always use the AMI root volume device name).
• Add stack parameter RunnerConfigAutoExtendsFrom to always force a specific value for repository configuration _extends directive (even if no local config file exists). Fixes #366. Note that it defaults to .github-private, meaning that if you leave that default, RunsOn will always attempt to load the config file from that repo as a base configuration. Set it to . (only extend from current repo _extends directive) to keep the previous behavior. This has been a long requested feature (and source of confusion) and is what new users expect, so this is why the breaking setting is enabled by default.
• Custom tag precedence is now: stack custom runner tags < custom runner tags < repository custom runner tags. This allows to set default tags at the stack level, which can be overriden by runner-level tags, but in the end repo-level tags always taking precedence (if set) to make sure repo admins can control the final tag value when needed.

Deprecations (please send feedback!)

• disk label support is going to be removed in the next minor version (replaced by volume label, which is much more flexible).
• RunnerLargeVolumeThroughput and RunnerLargeDiskSize are deprecated stack parameters.

Also, now that external networking is supported, next version will just set sane defaults for some VPC features when using embedded networking. As such, those parameters will be removed:

• VpcFlowLogFormat: [DEPRECATED, use external networking if you need to fine-tune this].
• VpcFlowLogS3BucketArn: [DEPRECATED, use external networking if you need to fine-tune this]
• VpcFlowLogRetentionInDays: [DEPRECATED, use external networking if you need to fine-tune this]
• VpcCidrSubnetBits: [DEPRECATED, use external networking if you need to fine-tune this]

Then, DefaultAdmins will be removed, since it's just better for admin-level people to use SSM to log into the runners if needed:

• DefaultAdmins: [DEPRECATED, prefer to use SSM for admin access].

Finally, I don't think ECInstanceDetailedMonitoring is useful since default cloudwatch metrics are useless anyway, and you're better off using the new runs-on/action metrics:

• ECInstanceDetailedMonitoring: [DEPRECATED. See https://runs-on.com/monitoring/job-metrics/#performance-metrics for better performance metrics].

Now that native Slack webhook integration is supported, I believe we can also remove the AlertTopicSubscriptionHttpsEndpoint parameter, which was originally introduced for that use case (but required an adapter in between). Please reach out if you think we should keep it.

Warm pools (BETA)

RunsOn can now operate pools of stopped or hot instances, which means pick-up times will be improved.

See https://github.com/runs-on/runs-on/blob/main/adrs/20250727-warm-pools.md for all details.

Volume overrides

Runners can now override the default volume settings directly within labels or through the configuration file.

Examples:

# full
runs-on: runner=2cpu-linux-x64/volume=gp3:80g:125mbps:3000iops
# partial
runs-on: runner=2cpu-linux-x64/volume=80g:250mbps

GitHub webhook redelivery on failures

RunsOn now ships with a background job to check (every 5min) for failed webhook deliveries from the github side. If it finds some (matching the current stack labels), it will attempt to redeliver them once. This is especially useful under very high load, as AppRunner can sometimes rate-limit incoming webhooks when GitHub sends a burst of webhooks all at once.

You'll get alerted (over SNS, Slack, etc.) if failed webhooks have been redelivered. Cloudwatch dashboard also has a widget showing recent runs and redeliveries (if any).

Runner details

• Add original labels
• Add pool details (if any)

Click to view example

Slack integration

Can now define a slack webhook URL (AlertTopicSlackWebhookUrl stack parameter), so that alerts also get sent there.

OTEL integration

Can now pass OTEL endpoint and headers (OtelExporterOtlpEndpoint, OtelExporterOtlpHeader). Only HTTP transport enabled for now. Metrics will be shipped there. Example dashboard below using Signoz:

# Scheduled status (has instance_type and instance_lifecycle)
runs_on_jobs_total{status="scheduled",repo_full_name="acme/api",workflow_name="CI",instance_type="t3.medium",instance_lifecycle="spot",pool_name="default",org="acme",installation_id=12345,stack_name="runs-on-prod",region="us-east-1"} 42

# Completed status (no instance_type or instance_lifecycle)
runs_on_jobs_total{status="completed",repo_full_name="acme/api",workflow_name="CI",pool_name="default",conclusion="success",org="acme",installation_id=12345,stack_name="runs-on-prod",region="us-east-1"} 42

runs_on_pool_instances_total{pool_name="default",state="running",installation_id=12345,org="acme"} 5
runs_on_pool_instances_total{pool_name="default",state="stopped",installation_id=12345,org="acme"} 10

runs_on_rate_limiter_tokens{limiter="github_api"} 4500.5

runs_on_rate_limiter_burst{limiter="github_api"} 5000

runs_on_spot_circuit_breaker_active{} 0

{
  "metric_type": "jobs_summary",
  "queued": 1234,
  "scheduled": 1200,
  "in_progress": 34,
  "completed": 1150,
  "interrupted": 16
}

{
  "metric_type": "job_event",
  "status": "completed",
  "conclusion": "success",
  "repo_full_name": "acme/api",
  "workflow_name": "CI",
  "instance_type": "t3.medium",
  "instance_lifecycle": "spot",
  "pool_name": "default",
  "interrupted": true,
  "internal_queue_duration_seconds": 12.5,
  "overall_queue_duration_seconds": 45.2,
  "job_duration_seconds": 180.3
}

{
  "metric_type": "pool_instances",
  "installation_id": 12345,
  "org": "acme",
  "pool_name": "default",
  "running": 5,
  "stopped": 10,
  "pending": 2
}

{
  "metric_type": "rate_limiter",
  "limiter": "github_api",
  "tokens": 4500.5,
  "burst": 5000
}

{
  "metric_type": "spot_circuit_breaker",
  "active": false,
  "interruption_count": 42
}

{
  "metric_type": "spot_interruption",
  "interruption_time": "2025-10-10T14:30:00Z",
  "trip_count": 3,
  "recovery_minutes": 15,
  "circuit_breaker_active": false,
  "active_until": "2025-10-10T14:45:00Z",
  "instance_id": "i-1234567890abcdef0",
  "job_id": "987654321",
  "job_name": "build",
  "job_url": "https://github.com/owner/repo/actions/runs/123456789/job/987654321",
  "repo_full_name": "owner/repo"
}

Pre/Post custom job hooks

You can now launch custom scripts within the "Set up runner" and "Complete runner" sections of a workflow. If /runs-on/pre.custom.sh or /runs-on/post.custom.sh scripts are found, the RunsOn agent will execute them in their respective job section. They are executed after the RunsOn-specific scripts, and RunsOn will fail the step if those custom scripts fail. See https://docs.github.com/en/actions/how-tos/manage-runners/self-hosted-runners/run-scripts for more details.

Misc

• Improve failure message when invalid runner spec (missing family). Fixes #343.
• Fix permission issue with Docker and ECR login in preinstall scripts on instances with local disks. Fixes #362.
• Auto-resize Windows disks. Fixes #369.
• Properly disable all ipv4 public addresses whenever launching in private subnets. Previously this was only done when Private=only stack parameter was set, leading to increased costs when running mixed networking mode (public + private runners allowed) stacks.
• When instance received a spot interruption warning, let AWS perform the termination so that we don't get billed if runtime was <1h. Fixes #365.
• Surface job error after all schedule attempts have been exhausted. Fixes #357.
• Fix SSH setup issues on AlmaLinux images. #330.

Upgrade

• Upgrade Guide
• CloudFormation Versioned template URL: https://runs-on.s3.eu-west-1.amazonaws.com/cloudformation/template-v2.9.0.yaml