agent-substrate/substrate v1.0.0-rc.2 on GitHub

v1.0.0-rc.2 is the second release candidate for Substrate 1.0. It brings release-1.0 up to main as of November 3 and hardens the sandboxes before GA: actors run as the image's USER, capabilities.add is limited to a short allowlist, /tmp and /dev/shm are tmpfs, and mounts are nodev on both runtimes. It also fixes a worker outage from microVM memory overshoot, a 15 s stall on every atenet-router start, an actor and worker assignment that could drift apart, and the device plugin wedging after an atelet restart.

Upgrading from rc.1 is possible but breaks gVisor snapshots and needs an image cache reset. See Upgrading from v1.0.0-rc.1 at the end. Upgrading from v0.6.0 is not supported; reinstall.

What's Changed

⚠️ Breaking Changes

Settings to change before upgrading:

Snapshot location path segments are validated (#2784 by @aramase). Each segment of --snapshot-location (SNAPSHOT_LOCATION) must match ^[A-Za-z0-9_-]+$ and cannot equal atespaces in any case. Fix the path before upgrading or ate-api-server refuses to start.
--cordon-control-plane is now --cordon-system-infra-components (#3210 by @haiyanmeng). The env var is ATE_INSTALL_CORDON_SYSTEM_INFRA_COMPONENTS, the config key cordonSystemInfraComponents, and the shared pool label and taint ate.dev/workloadType=ate-system-infra. The old flag and config key fail; the old env var is ignored silently and the install stops cordoning. Relabel and retaint the pool, then redeploy with the new flag; the PR has the gcloud commands.
ActorTemplates may only add 12 capabilities (#3580 by @destijl, #3487 by @BenTheElder). securityContext.capabilities.add accepts only AUDIT_WRITE, CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL, NET_BIND_SERVICE, SETFCAP, SETGID, SETPCAP, SETUID and SYS_CHROOT; MKNOD, SYS_ADMIN, SYS_RAWIO and the rest are rejected at admission. Unknown names in add or drop are rejected too. Check your templates for typos and for capabilities outside the list. Unprivileged ping keeps working without NET_RAW.
Custom credential-exchange sidecars lose --actor-jwt-jwks-file (#3447 by @aramase). The exchange now fetches keys through OIDC discovery. Drop the flag from custom sidecar configs, and set --actor-jwt-issuer-ca-file only when the issuer uses a private CA other than the service DNS one.
The public MintActorCertificate RPC is removed (#3427 by @EItanya). Calls return UNIMPLEMENTED. Remove references to its client method and message types before upgrading. Actor JWTs and the worker certificates atunnel uses are unchanged.
Worker pods need the new worker image before the new ate-controller (#2428 by @tjorourke). Workers get runtime-specific capability sets and the pod template changed. Move every WorkerPool to the new worker image first; a microVM pool left on the old image stops working once ate-controller rolls it. The PR has the upgrade and rollback steps.
Pools hold a little less (#3104 by @BenTheElder). A worker's capacity is its memory limit minus 64 MiB, and each microVM actor books its limit plus 128 MiB + 8 MiB/GiB. A pool sized to exactly fit its actors needs limits.memory raised by about that much. MicroVM guests now see their whole memory limit.

Behavior changes after upgrading:

Containers run as the image's USER and WORKDIR, and the image cache layout changed (#2392 by @BenTheElder). Images with a USER stop running as root and capabilities.add no longer helps a non-root user. After an upgrade or rollback, atelet refuses the old image cache: delete /var/lib/ate/image-cache on each node once no actor runs there, or reboot the node. A custom atelet securityContext needs CHOWN, DAC_OVERRIDE, FOWNER and FSETID.
/tmp and /dev/shm are tmpfs on both runtimes (#3533 by @nybidari). ROOTFS snapshots no longer keep /tmp, and image-provided files there are hidden. MicroVM /tmp counts against actor memory, capped at half the guest's memory; gVisor /dev/shm is 64 MiB. Mount a durableDir at either path for persistence or more space. /run now comes from the image rootfs.
gVisor MEMORY snapshots from before this release don't restore (#3533, #3580, #2392). runsc restore rejects a checkpoint whose mounts or user differ from the new spec, so suspended gVisor actors crash on resume and templates created before the upgrade crash on first resume. RevertActor doesn't help. Recreate the templates and replace the actors. MicroVM actors are unaffected.
MicroVM assets move to kata 4.3.1, Envoy to v1.39.5 (#3181 by @dims). A microVM MEMORY snapshot taken on kata 4.1.0 restores at ROOTFS fidelity on its next resume: a fresh guest with its rootfs writes and volumes, no memory state. Recreate templates whose golden you want warm again.
A microVM actor that overshoots its booking is killed, not its worker (#3429 by @BenTheElder). Each actor's cgroup gets memory.high at what it books; one stuck there is killed and reported crashed with "killed for exceeding its memory limit". Other actors keep running. Without PSI the worker sets no limit and warns at startup.
MicroVM guests get containerd's default masked and read-only paths (#3249 by @hdp617). A uid 0 entrypoint can no longer write guest sysctls such as vm.max_map_count, matching gVisor. Goldens and checkpoints taken before this release keep a writable /proc/sys until re-taken.
WorkerPool NetworkPolicies are named substrate-<pool> (#3130 by @yufan-su). Pools with similar long names no longer share a policy. On an in-place upgrade the old substrate-<truncated>-<hash> policy stays until the pool is deleted; once the new one exists, delete the old ones with the command in the PR.
An actor and its worker assignment are written in one transaction (#3223 by @sfunkenhauser). Earlier releases could leave them out of sync after an interrupted write, and this release no longer repairs that. A worker that can't be deleted or an actor stuck in RESUMING is the symptom.
Atespace names must not be reused (#3450 by @SHRUTI6991). A request authorized against a deleted atespace can apply to a new one with the same name, and nothing enforces uniqueness. Pick a new name instead.

Bug Fixes

atenet-router no longer stalls 15s on every start by @chuangw6 in #3346
An actor and worker assignment that drift apart no longer wedge the worker; see Breaking Changes by @sfunkenhauser in #3223
atelet: register the device plugin on a new socket each time, fixing the wedge after a restart by @BenTheElder in #3486 and #3323
Resuming a microVM actor on the same node no longer reads its rootfs upper back from disk by @BenTheElder in #3485
Leave page cache out of the memory guard's overshoot check by @BenTheElder in #3476
A gVisor actor going over its memory limit no longer takes down every actor on its worker by @BenTheElder in #3106
Suspending actors no longer OOM-kills tight worker pods by @BenTheElder in #3353
ate-setup: reconcile ate-api-server ConfigMaps and roll out on config changes by @moficodes in #3370
An actor's new egress connections no longer fail with 403 an hour after it starts by @chw120 in #3144
Resume restores an uploaded snapshot when its node is gone by @igooch in #3094
atelet: clear node-local caches after a node reboot by @igooch in #3376
Actors can reach the network while they boot, before their wakeup probe passes by @QuentinBisson in #2117
atenet: classify a 403 route outcome as denied, not resume_error by @teemow in #2534
Export atenet-egress traces and metrics with service.name=atenet-egress by @bochengchu in #3417
atelet: detach Checkpoint from caller cancellation once underway by @chw120 in #2602
atelet, ateworker: finish a checkpoint retry after atelet restarts by @chw120 in #3218
ateworker: reuse a completed checkpoint only on the same boot by @chw120 in #3213
atelet: keep the worker's Unavailable when a checkpoint fails by @aditya-shantanu in #3123
microvm: forward sysctls and lift the guest's privileged-port limit to match gVisor by @dims in #1904
Fix statusz port in k8s credential provider by @yufan-su in #3067

Features: Networking and Egress

Add kubectl ate proxy actor command by @juli4n in #3191
Allow the egress gateway to use more than one credential provider by @thompsonmax in #3409
ate-api-server's --egress-gateway-spiffe-id is repeatable, and ate-setup sets it from ATE_EGRESS_GATEWAY_EXTRA_SPIFFE_IDS by @thompsonmax in #3360
atenet/router: fail fast on FailedPrecondition instead of parking by @ygao-g in #3472
actornet: pin the sysctls a sandbox namespace inherits from the node by @BenTheElder in #3310

Features: Security and Identity

Verify actor JWTs in the credential exchange through OIDC discovery by @aramase in #3447
Generate the actor identity CA with ECDSA P-256 so external relying parties accept it by @destijl in #3251
manifests: Run control plane containers as non-root by @mikedickey in #3028
atecontroller: Limit the Secrets read to the egress MITM CA pool by @mikedickey in #3036
manifests: Give podcertificate-controller its own ServiceAccount by @mikedickey in #3059
ateapi: deny streaming RPCs that have no authorization rule by @SHRUTI6991 in #3262
Enable runsc core tagging when supported by the kernel by @fvoznika in #2887
Reject snapshotConfig.storageLocation without its preview gate by @Sneha-at in #3357

Features: Workers and Actors

Update gVisor to release 20261102.0 by @nybidari in #3574
microvm: skip software bounce buffer (SWIOTLB) by default by @jiayimeow in #3456
require workers to run in their own cgroup namespace by @BenTheElder in #3110
Move public APIs under api/<api-name>/v1 by @juli4n in #2996
Rename SnapshotStorage.status to state by @laoj2 in #3215
UpdateActor fails with Aborted while another operation on the actor is in progress by @zoez7 in #3131
UpdateActor rejects Actors marked for deletion by @laoj2 in #3341
CreateActor fails if the golden tag is marked for deletion by @laoj2 in #3382
Update a deleting tag returns FAILED_PRECONDITION by @laoj2 in #3340
The template reconciler skips templates marked for deletion by @laoj2 in #3342
ateapi: carry the golden actor's crash message into the template status by @dims in #2644
ateapi accepts snapshot report from atelet for actors in CRASHED state by @zoez7 in #3148
Prevent storing actors/tags with malformed durable snapshot URIs by @laoj2 in #3082
Reject WorkerPool names longer than 63 characters by @haiyanmeng in #3136
atelet: verify microvm SandboxConfig assets before writing sandbox-runtimes.json by @shrutiyam-glitch in #3143
atelet: emit a Warning Event on worker pods when syncing sandbox runtimes fails by @shrutiyam-glitch in #3146

Features: Install and Operations

ci: automate release artifact build and publish by @stshrive in #2112
Releases publish the envoy-dataplane image by @MushuEE in #3252
Releases publish the Kubernetes Secrets credential provider image by @MushuEE in #3220
ate-setup: retry each ko release image with backoff by @annapendleton in #3446
ate-setup: port microvm-deps installation from shell to Go by @noahlwest in #2092
GKE provisioning moves to substrate-gke: tools/setup-gcp and hack/teardown.sh are removed by @MushuEE in #3505
The GCP Secret Manager credential provider moved to agent-substrate-plugins/credprovider-gcp-secret-manager by @BenTheElder in #3365
Delete hack/install-ate.sh and hack/install-ate-kind.sh by @HavenXia in #3354
On size10 clusters, the bundled Postgres accepts 500 connections instead of 300 by @haiyanmeng in #3141
Base images drop the distrusted eMudhra roots and the kind Jaeger moves to v2.22.0 by @dims in #3196
Demos: fix the counter, sandbox, jupyter, multi-template, parking and claude-code-multiplex walkthroughs by @moficodes, @HavenXia, @zoez7 and @bochengchu in #3500, #3498, #3509, #3510, #3391, #3560 and #3553

Documentation

Generate API reference documentation and publish it on the Substrate website by @juli4n in #2775
Add Hugo Docsite by @moficodes in #1974
Document how people, pods and CI jobs should authenticate to ate-api by @thompsonmax in #3119
Document the actor JWT contract, how relying parties verify it, federation and key rotation by @thompsonmax in #2605 and #2911
Document using more than one egress credential provider by @thompsonmax in #3410
docs: fix outdated statements in the egress guides by @yufan-su in #3442
doc: add details on how a DNS denial from egress policy looks inside the actor by @yufan-su in #3363
docs: document sandbox class differences and fix stale passages by @nybidari in #3468
docs: align control-plane docs with the current API and CLI by @haiyanmeng in #3404
docs: note that pools selected by ActorTemplate.worker_selector must share a CPU architecture by @shrutiyam-glitch in #3458
docs: clone the repo before quickstarting by @rikatz in #3507
docs: update and add security diagrams by @destijl in #3415
docs: Update the threat model and associated reporting skill by @destijl in #3355
docs: say "no escape" from the sandbox on the ate.dev home page and security overview by @destijl in #3440
docs: remove inaccurate early-stage architecture documentation by @destijl in #3517
docs: add extensibility guide for gateways by @destijl in #3317
Add documentation for SLOs and alerting by @adrianludwin in #2476
docs: average ate.workerpool.workers across replicas before summing by @cloudonly in #3193
docs: normalize actor resume latency and activation rate by @moficodes in #3474
site: publish per-page Markdown and fix llms.txt and llms-full.txt by @moficodes in #3435
correct the minimum kernel for the image cache: Linux 6.7, or 6.6.23 by @BenTheElder in #3128
docs: use kvm-arm.vgic_v3_group1_trap=1 for Lima microVM setup on arm64 by @BenTheElder in #3149
atelet: say snapshot cache hits fall back to copies across mounts by @roycaihw in #3064
Document ReportActorCrashed in the WorkerService authz table by @SHRUTI6991 in #3324
docs: fix the README note on enabling beta APIs by @HavenXia in #3266
README: add an animated SVG by @HavenXia in #3423

Other Changes

Remove outdated roadmap and update security vulnerability reporting guidelines by @destijl in #3122
docs: require DCO sign-off instead of the Google CLA by @BenTheElder in #3237
atelet: return Canceled when Run or Restore fails to reset actor dirs by @zoez7 in #3063
ateworker: reuse completed checkpoint when workload is already stopped by @chw120 in #3015
ateapi: squash PostgreSQL migrations and lock them once merged by @iplay88keys in #2991
objectstore-plugin: move the credential exchange's actor checks into actorauthz by @aramase in #3138

New Contributors

@mikedickey made their first contribution in #3036
@teemow made their first contribution in #2534
@adrianludwin made their first contribution in #2476

Full Changelog: v0.6.0...v1.0.0-rc.2

Images

Container images for this release are published as ghcr.io/agent-substrate/substrate/<image>:v1.0.0-rc.2 for linux/amd64 and linux/arm64. The images a release publishes are listed in Release images.

Install

From a checkout of v1.0.0-rc.2, with Go, install the release images into the current kubectl context. On GKE, the default --sandbox-classes=gvisor,microvm downloads microVM guest assets from GitHub and uploads them with gcloud to gs://<bucket>-assets (created with atelet read access by tools/setup-gcp create bucket and create iam in https://github.com/ai-on-gke/substrate-gke). Add --kind for a kind cluster, or --sandbox-classes=gvisor to skip microVM staging.

git clone --depth 1 --branch v1.0.0-rc.2 https://github.com/agent-substrate/substrate.git
cd substrate
go run ./cmd/ate-setup deploy ate-system \
  --image-repo ghcr.io/agent-substrate/substrate --image-tag v1.0.0-rc.2 \
  --storage-bucket-name=<bucket> \
  --credential-provider='{"name":"k8s.io"}' \
  --ingress-allowed-spiffe-ids=<SPIFFE IDs of your ingress clients>

See Installing pre-built images for the flags.

kubectl-ate

kubectl-ate is attached for Linux and macOS on amd64 and arm64. Download the binary for your platform and its .sha256 file, check it, and put it on your PATH as kubectl-ate:

shasum -a 256 -c kubectl-ate-linux-amd64.sha256
install -m 0755 kubectl-ate-linux-amd64 ~/.local/bin/kubectl-ate

Upgrading from v1.0.0-rc.1

The version skew policy doesn't cover upgrades between release candidates, and several changes since rc.1 break compatibility. You can still upgrade an rc.1 install to rc.2 with the upgrade runbook if you accept the effects below. Otherwise, reinstall.

gVisor actors keep only their volumes through this upgrade (see below), so before you start, let their work finish or save the state you need in a volume. Suspending every microVM actor before you start lets you clear the image cache at step 2 right away and makes stale assignments less likely, at the cost of downtime for the whole upgrade.

Image cache (#2392): rc.2 changes the on-disk layout of atelet's image cache, and at step 2 the new atelet exits until the old cache is gone. Delete /var/lib/ate/image-cache on each node once no actors run there, or reboot the node. The cache only holds pulled images, so nothing is lost.
Worker capabilities (#2428): the worker pod template changed. At step 3, move every WorkerPool to the new worker image, because a microVM pool left on the rc.1 image stops working at step 4b. Step 4b then rolls every pool once more.
gVisor snapshots (#3533, #3580, #2392): rc.2 changes the mounts and user that gVisor actors run with, so MEMORY snapshots (the default) and golden snapshots taken on rc.1 don't restore. Create new templates with the same volumes, and move each suspended gVisor actor to its new template with UpdateActor before you resume it. The first resume then restores only its volumes, so its memory and rootfs changes are lost. An actor that already crashed needs RevertActor first. MicroVM actors are unaffected.
Stale assignments (#3223): rc.1 wrote an actor and its worker assignment separately, so an interrupted write could leave them out of sync, and rc.2 no longer repairs that. If it happened, you'll see a worker that can't be deleted or an actor stuck in RESUMING.

Rolling back to rc.1 needs the image cache step again. Check the breaking changes above for settings to change before upgrading, such as the snapshot location (#2784), capabilities in ActorTemplates (#3487, #3580) and custom sidecar flags (#3447).