agent-substrate/substrate v0.6.0 on GitHub

What's Changed

⚠️ Breaking Changes

require actor cpu and memory limits by @BenTheElder in #2285
rename ateom -> ateworker by @BenTheElder in #2254
gvisor SandboxConfig no longer accepts a bare runsc asset by @ericcurtin in #2798
ateapi: Remove the PauseActor RPC by @zoez7 in #2813
ateworker no longer exports to localhost:4317 without a relay or endpoint, and takes its own exporter settings by @QuentinBisson in #2189
Remove any_state and allow deleting actors in any state by @laoj2 in #2727
[bug] fix microVM actors with one character container names by @BenTheElder in #2801
Add a default cluster snapshot storage location with snapshotConfig.storageLocation override in preview mode by @Sneha-at in #2544
Renumber proto fields consecutively, with ResourceMetadata as field 1 by @laoj2 in #2888
resource limits and volume capacity are integers now: cpuMillicores, memoryBytes, capacityBytes by @BenTheElder in #2769
imagecache: Verify layer content against its diffID before it enters the pool by @igooch in #2279
atelet, nodepath: separate per-actor snapshot directories under /var/lib/ate/snapshots by @roycaihw in #2746
imagecache: Refuse a layout version 1 pool and log diffID mismatches by @igooch in #2979
On GKE, micro-VM assets move to their own bucket and nodes lose access to the snapshot bucket by @ILLUM1N0X in #3037
On GCS, snapshot transfers run as a dedicated service account and their tokens no longer run short near the end of each hour by @ILLUM1N0X in #3093

Bug Fixes

Reduce memory utilization in Envoy egress gateway by deduplicating policy object by @yanavlasov in #2651
atenet-egress: refuse an actor's own CONNECT with 403 by @bochengchu in #2721
fix(ateom-gvisor): isolate and evict checkpoint page cache to prevent actor cgroup OOM by @chw120 in #1917
atelet: make durable-dir volumes writable by non-root containers by @dims in #1906
skip device nodes when unpacking image layers by @BenTheElder in #2739
setup-gcp: raise node ARP table and inotify sysctls for high sandbox density by @chw120 in #2444
egress: return a retryable 503 instead of a 504 when a credential provider is slow by @BenTheElder in #2787
ateinterceptors: elide List responses from the RPC log by @dims in #2494
atelet: only ate-api-server may call AteomHerder by @destijl in #2516
Fake data plane installs no longer fail at startup on the --preview flag by @haiyanmeng in #2884
setup-gcp no longer disables Filestore on an existing cluster by @ericcurtin in #2485
Large GCS uploads no longer leave scratch objects behind by @ericcurtin in #2178
ateapi: crash the actor when atelet's returned snapshot state conflicts by @igooch in #2794
Snapshot uploads reserve only the memory their upload path uses by @roycaihw in #2848
imagecache: Pin a pull's layers so eviction cannot retire them mid-pull by @igooch in #1509
Grant fake-workersync the controller authz role so it can manage Workers by @haiyanmeng in #2885
atelet: Bound background snapshot uploads with a timeout by @msau42 in #2831
ateworker-microvm: give the guest CRNG reseed 60s on restore by @roycaihw in #2743
atecontroller: deregister workers when their WorkerPool is deleted by @ericcurtin in #2093
TestSyncer_Lifecycle no longer flakes on a worker that is still UNAVAILABLE by @ericcurtin in #2824
ateapi: check mounted volumes are attachable and test assigned_node detach on crash by @lekhoavu2909 in #2611
Revert and delete no longer hang when an actor's node is deleted by @eren23 in #2486
ateworker: clear partial checkpoint files and preserve microVM base on merge by @chw120 in #2623
atelet: Wait for superseded uploads to stop by @msau42 in #2833
Retry GCS rate-limit errors when deleting and copying snapshots by @sairajp-rewind in #2143
atelet: Answer a retried Checkpoint from the snapshot it committed by @msau42 in #2849
Components no longer warn about a partial telemetry resource at startup by @baizhenyu in #2998
resources: mask debug_redact fields in validation error text by @git286 in #3071
ateworker-microvm, atelet: classify pre-serving restore failures as Unavailable by @roycaihw in #2744
atelet: Clean up whole actor directory on detach by @sairajp-rewind in #2871
ateapi,atelet: reclaim actor node state when a worker's pod is gone by @sairajp-rewind in #2797
objectstorage: fail a ranged read cut short by cancellation by @cheftako in #3049
setup-gcp: say which node pools need replacing after enabling the beta APIs by @mastersingh24 in #1852

Features: Networking and Egress

Router authn/z part 2: ate-setup: add --ingress-auth-mode by @ahmedtd in #2388
Router authn/z part 2.5: podcertcontroller: Don't load pod in podidentity signer by @ahmedtd in #2389
atenet-egress: log every failed request, sample successes at 1% by @ygao-g in #2693
write /etc/hosts for actors so localhost resolves by @BenTheElder in #2738
benchmarking/atunnel: atunnel CPU and memory benchmark by @chuangw6 in #2761
E2E checks the actor JWTs the egress gateway injects by @thompsonmax in #2256
router authn/z part 3: Convert all e2e tests so that they pass for static-mtls ingress auth mode by @ahmedtd in #2390
e2e: fix the ingressauth flake: retry a refused handshake that ends without the router's alert by @dims in #2961
atenet-egress: raise the envoy config for a million tunnels case by @yufan-su in #2732
Use crypto/rand for upstream DNS transaction IDs by @bochengchu in #3020
Configure preview golden egress policies on actor templates by @EItanya in #2159
atenet-egress: log access log paths without the query string by @git286 in #3077
atenet: Router checks the routed worker's SPIFFE ID on atunnel's cert by @LiorLieberman in #2678
Fix the DNS to return NXDOMAIN by @bowei in #2774
atenet-egress: reload actor-identity CA bundle in atenet-egress when it rotates by @yufan-su in #3081
atenet-router: raise the tunnel circuit breaker to 1m by @yufan-su in #3087

Features: Security and Identity

ate-api-server's objectstore plugin serves per-actor snapshot credentials on GCS by @ILLUM1N0X in #2762
hardening: fix submounts of read-only image volumes being writable by @BenTheElder in #2944
Upgrade Go to 1.27.3 by @BenTheElder in #2984
ate-api-server accepts actor JWT issuers with a path by @thompsonmax in #2781
router authn/z part 4: Remove deprecated-insecure mode by @ahmedtd in #2977

Features: Observability

fix flaky lifecycle check when collector series expire by @BenTheElder in #2786
ateapi: trace PostgreSQL statements in the store by @git286 in #1462

Features: Workers and Actors

Add a "preview gate" concept, use it for external volumes by @thockin in #1994
ate-api-server: Create one unified configuration file by @ahmedtd in #2021
Align volume and env message names across ateapi, atelet, and ateom pbs by @laoj2 in #2666
write resolv.conf into the gVisor rootfs like micro-VM by @BenTheElder in #2765
ateapi: Downgrade runtime-incompatible durable resumes to ROOTFS by @shrutiyam-glitch in #2880
Spread single-request GCS transfers over the connection pool by @Oneimu in #2719
report actors whose sandbox dies on its own as crashed by @BenTheElder in #2695
Resuming a suspended actor detaches it when its node has no room by @igooch in #2881
objectstorage: write snapshot parts as sized zstd frames and pool upload buffers by @Nishanth29 in #2835
controlapi: Allow template repoint with local snapshot and add kubectl ate update actor by @zoez7 in #2814
atelet: serve shared snapshot restores from the node-local file cache by @dberkov in #2796
Run gVisor sandboxes without a pause container by @milantracy in #2773
setup-gcp no longer gives GKE nodes read access to every bucket in the project by @ILLUM1N0X in #3042
Detect worker restarts by an epoch the worker process registers by @sfunkenhauser in #2988

Features: Install and Operations

Summarize test failures in the CI job summary and annotations by @BenTheElder in #2791
benchmarking: add ate_env boomer user class and workload by @sairajp-rewind in #2735
glutton: add RunScript, a script tree run inside the actor as one request by @maxsmythe in #2162
Benchmarking: measure ate-api-server cold resume capacity on the fake data plane by @haiyanmeng in #2686

Dependencies

build(deps): bump github.com/klauspost/compress from 1.18.6 to 1.18.7 in /hack/tools/ko by @dependabot[bot] in #2980

Documentation

Add the version skew policy by @HavenXia in #2714
docs: describe the sandbox asset cache, and assert its size instrument in the metrics suite by @dberkov in #2792
Update security policy, threat model, and security guide for GA by @destijl in #2723
authz: cover every WorkerService RPC in the e2e suite and document authz in the architecture by @SHRUTI6991 in #2868
Drop the unused actorIdentityJWTProvider setting and fix the actor identity docs by @thompsonmax in #2597
Document injecting actor JWTs with egress policies by @thompsonmax in #2596
revise AGENTS.md, CONTRIBUTING.md, and PR template based on recurring maintainer feedback by @BenTheElder in #2812
Update Version Skew doc / Upgrade doc for SandboxConfig design by @HavenXia in #2973
docs: ate.workerpool.workers is reported by every ateapi replica by @git286 in #3056
agents: update rules for inline bash by @stshrive in #3078

Other Changes

HostingNode and MintAteomActorJwt trust a Worker's node only while that Worker holds the actor by @aramase in #2668
ExternalVolume no longer has a status field by @ericcurtin in #2456
deploy podcertificate-controller creates missing CA pools by @HavenXia in #2680
e2e: replace external image busybox with e2e test server in network policy e2e by @yufan-su in #2670
atelet: Add snapshot state tracker and return the latest local or durable snapshot by @msau42 in #2677
atelet: Add background workers to the snapshot tracker by @msau42 in #2696
Require snapshot generation to be passed to atelet by @msau42 in #2690
ate-setup deploy apiserver works on kind clusters by @HavenXia in #2682
ateworker internal packages cleanup by @BenTheElder in #2700
Update suspend and pause tests to use SuspendActor and PauseActor APIs by @zoez7 in #2688
Overhaul flag and subcommand errors in kubectl-ate by @thockin in #2547
hack: ATE_KIND_RESERVED_CPU reserves kubelet and system CPU on kind nodes by @dims in #2495
oidcjwt: take signing keys from a KeySource, and expose Substrate claims by @ILLUM1N0X in #2671
ate-api-server applies atelet's ReportActorSnapshotState reports by @igooch in #2622
hack/verify: run the benchmarking Python suites by @ygao-g in #2709
Remove the pause operation label by @zoez7 in #2701
Update unknown request rules in api-style-guide by @HavenXia in #2711
ate-setup: use the envoy-dataplane image in atenet-router by @panchoruy in #2655
ate-api-server can apply the snapshot state Checkpoint returns, and logs stale snapshot reports by @igooch in #2698
atunnel refactor by @bowei in #2720
atelet API: add DetachActor, which uploads an actor's latest local snapshot and frees the node by @igooch in #2725
objectstore-plugin: derive actor storage grants and mint downscoped GCS tokens by @ILLUM1N0X in #2672
docs: introduce zero-trust security as a co-equal pillar in README by @destijl in #2652
Mint worker-for-actor certificates only for the Worker that the actor is assigned to by @aramase in #2702
benchmarking: plumb the actor CPU limit through deploy.sh and the orchestrator by @maxsmythe in #2717
benchmarking/sweperf: run each trajectory on a fresh actor by @pmandewalkar in #2323
Limit memory used by concurrent snapshot uploads by @EItanya in #2708
benchmarking: generate sweperf ActorTemplates from an image catalog by @pmandewalkar in #2379
Suspend and pause preserve concurrent actor crashes and report failure by @cloudonly in #2707
imagecache: read mountinfo lines of any length; fix two tests for 64 KiB pages and root by @dims in #2624
benchmarking: record Kubernetes version, worker capacity and run metadata in trial_summary by @Nishanth29 in #2232
Merge the pause workflow into the suspend workflow by @zoez7 in #2694
Use a Go vanity import in the "main" substrate repo: go.ate.dev/substrate by @thockin in #2181
atelet, ateworker: probe CPU features and watch sandbox-runtimes.json for dynamic worker registration by @shrutiyam-glitch in #2660
ate-setup: enforce ateapi authorization by default and manage access with kubectl ate by @SHRUTI6991 in #2591
docs: say when to mark proto fields debug_redact and what it covers by @git286 in #2267
atelet: stage sandbox assets per actor from the node-local file cache by @dberkov in #2699
Send and require snapshot object storage URI on all checkpoints by @msau42 in #2752
Add ate-idp-server to serve the discovery document and key set for actor JWTs by @thompsonmax in #2099
kube-api-linter: fix API type docs and the sandboxClass default by @ericcurtin in #2461
resources.ToAPIError returns OK on nil errors by @laoj2 in #2767
atelet: run on nodes GKE creates for a ComputeClass by @roycaihw in #2679
atelet: Keep the local snapshot even after uploading, and cleanup in the background after a timer by @msau42 in #2733
atelet: move default port to an IANA-unassigned port by @linde in #2665
atelet: pass SandboxRuntime on Run and Restore by @shrutiyam-glitch in #2755
benchmarking(boomer): disentangle dynConfig by @maxsmythe in #2160
e2e: remove redundant egress e2e which uses external sites by @yufan-su in #2764
Deploy ate-idp-server with every install by @thompsonmax in #2599
Background snapshot uploads name the actor they act for, ahead of per-actor snapshot credentials by @ILLUM1N0X in #2778
test(volume): verify node payload and mock driver publishContext round-trip. by @lekhoavu2909 in #2094
ateworker-microvm: enable virtio-balloon with free page reporting by default by @ajaysundark in #2376
ate-api-server applies the snapshot state atelet returns when a suspend finalizes by @igooch in #2766
Suspending a paused actor can mint the per-actor credential for its snapshot upload by @ILLUM1N0X in #2776
Per-actor snapshot credentials work on installs that enforce authorization by @ILLUM1N0X in #2757
Use collapse sections to improve counter demo readme by @cheftako in #2783
atelet: resolve an actor's sandbox assets from the runtimes its workers advertise by @dberkov in #2731
SuspendActor accepts suspend_options to choose a durability mode (API only) by @zoez7 in #2779
ateapi/authz: drop can_pause by @zoez7 in #2782
atelet: resolve an actor's sandbox runtime from the request's sandbox_runtime by @dberkov in #2788
benchmarking: add memory working set to server telemetry by @Nishanth29 in #2386
ateworker: introduce declarative validation for the Ateworker RPCs by @shrutiyam-glitch in #2581
benchmarking: add lifecycle_mode=none and agentsession_snapshot_steps by @chw120 in #2771
Update default runsc flags configuration by @fvoznika in #2361
ateapi: remove SandboxConfig from ActorTemplate by @shrutiyam-glitch in #2780
atecontroller serves ate.workerpool.* on :8080 while also pushing by @ericcurtin in #2634
benchmarking: add OOM kills to server telemetry by @Nishanth29 in #2391
benchmarking/sweperf: log a task record per task by @Nishanth29 in #2537
atelet: prewarm each worker's default sandbox runtime as the syncer plans it by @dberkov in #2741
atenet: dial both IPv4 and IPv6 loopbacks explicitly by @ahmedtd in #2740
ateapi: Record the async upload of a local snapshot by @msau42 in #2795
Replace clients of PauseActor to call SuspendActor(ASYNCHRONOUS) instead by @zoez7 in #2790
Atelet uploads local snapshots asynchronously by @msau42 in #2789
atelet: Give async uploads and detaches their own workers by @msau42 in #2802
Support splitting container rootfs and SNAPSHOT_FIDELITY_ROOTFS on gVisor by @nybidari in #2673
SuspendActor returns a suspended actor as it is, and atelet uploads a paused actor's snapshot in the background by @zoez7 in #2804
atelet: run the syncer's timing tests on a synctest bubble by @dberkov in #2793
atelet: Reject stale snapshot generations by @msau42 in #2803
SuspendActor defaults to suspend asynchronously by @zoez7 in #2805
atelet: Report background uploads and detaches to ate-api-server by @msau42 in #2810
atelet: emit metrics for async snapshot upload and detach by @msau42 in #2815
Run golden snapshots in the template's atespace by @cloudonly in #2543
atelet: Require snapshot.object on Restore by @msau42 in #2834
CreateTag uploads a paused actor's snapshot instead of failing by @zoez7 in #2817
atelet: evict the sandbox asset cache to a budget, each worker linking its default runtime by @dberkov in #2756
Preserve actors when stale snapshot requests are rejected by @cloudonly in #2819
demos/counter: blank line between each summary tag and the fence after it by @dberkov in #2854
atelet: implement DetachActor by @igooch in #2852
csi: reformat external volumes objects by @hajiler in #2653
A node's per-actor snapshot credential stops working once the actor leaves that node by @ILLUM1N0X in #2893
ate-setup: deploy ate-idp-server with ate-api-server by @HavenXia in #2891
atelet: Serialize lifecycle RPCs per actor by @msau42 in #2890
atelet: one eviction loop for every node-local file cache, with one metric family by @dberkov in #2889
ateapi, atelet: remove sandbox_assets from Run and Restore RPCs by @shrutiyam-glitch in #2879
atunnel/egress: retry an Unavailable certificate mint by @dims in #2906
ateworker-microvm: skip redundant chv HTTP shutdown in teardownActor by @jiayimeow in #2904
benchmarking: pass a whole-number actorCPU to deploy.sh as a string by @maxsmythe in #2912
benchmarking(agentsession): phase in session starts and retry wakes the fleet had no room for by @maxsmythe in #2915
objectstorage: size small snapshot upload buffers to actual data by @chw120 in #2945
atelet: Wait for a terminated actor's upload or detach to stop by @msau42 in #2943
ateapi: record a sync suspend's local snapshot copy and resume from it safely by @cloudonly in #2883
Remove all logic from ./hack/install-ate.sh by @ahmedtd in #2718
ate-setup: front the NFS CSI controller with an Envoy mTLS proxy by @hajiler in #2937
docs/observability: fix stale and wrong information by @git286 in #2886
Move internal/ateworker/tunnel to internal/atunnel by @bowei in #2981
Fix an image cache eviction race that could drop an image record after a failed pull by @BenTheElder in #2989
Remove workerpool autoscaling demo by @laoj2 in #2916
Snapshot transfers on GCS run on a credential scoped to the actor by @ILLUM1N0X in #2924
ateapi: always enforce authorization by @SHRUTI6991 in #2942
Do not crash DELETING actors by @laoj2 in #3004
fix(benchmarking): use ate-setup subcommands in install-ate.sh invocations by @sairajp-rewind in #3007
ateapi: Remove sandbox_class from Worker spec by @shrutiyam-glitch in #2922
atenet-egress: remove the unused by-address dial path by @LiorLieberman in #2892
atunnel uses less memory for idle egress connections by @chuangw6 in #2908
actornet: disable IP forwarding in sandbox network namespaces by @destijl in #2969
atelet, ateapi: report worker runtimes per sandbox class by @shrutiyam-glitch in #3022
Make egress benchmark work with ingress authz by @yanavlasov in #3043
atelet: Move local checkpoint directory with a single rename by @msau42 in #3048
atenet-router: CONNECT tunnels to different ports of one actor no longer cross by @LiorLieberman in #3003
ResumeActor: merge code paths for resuming from "paused" and suspended states by @zoez7 in #2967
benchmarking: set ATE_INGRESS_ALLOWED_SPIFFE_IDS in orchestrator.py by @sairajp-rewind in #2976
docs: replace obsolete PauseActor and paused actor references by @zoez7 in #2926
Add delete_time to ResourceMetadata by @laoj2 in #3057
docs: two-step changes explicitly need minor releases by @HavenXia in #3073
Crash a paused actor on resume when its node is gone by @igooch in #3010
Enable H/2 to origins in Envoy egress by @yanavlasov in #3058
A resume that moves an actor finishes its snapshot upload even if the request times out by @igooch in #2940
netns.GetFromName rejects names that resolve outside /run/netns by @yufan-su in #3019
atunnel benchmark measures 100,000 idle egress connections by @chuangw6 in #3072
atelet: fix a data race in the JWT minter test fake (flaky test fix) by @dims in #3054
benchmarking/sweperf: honor --lifecycle-mode for suspend by @pmandewalkar in #3068
atelet: invalidate node-local snapshots across node reboots by @msau42 in #3050
tarutil: drop fsync of rootfs-upper and durable-dir tars (#1954) by @pmandewalkar in #3026
benchmarking: rename --lifecycle-mode values to sync/async and default to async by @chw120 in #3097

New Contributors

@ajaysundark made their first contribution in #2376
@mastersingh24 made their first contribution in #1852

Full Changelog: v0.5.0...v0.6.0

agent-substrate/substrate v0.6.0 on GitHub

What's Changed

⚠️ Breaking Changes

Bug Fixes

Features: Networking and Egress

Features: Security and Identity

Features: Observability

Features: Workers and Actors

Features: Install and Operations

Dependencies

Documentation

Other Changes

New Contributors

Committers in this release

agent-substrate/substrate v0.6.0
on GitHub