Second release candidate on the 1.2.0 line, carrying everything merged since release-1.2.0-rc.1 — the headline is the Credential Vault (OSEP-0022) landing end to end for fast sandboxes: an egress-held vault with durable per-subject persistence and pause/resume transfer, the user-reachable server-routed vault API, and a pool-pinned MITM CA. Network policy follows the same egress-held pattern, and NetworkRule gains port-scoped rules in the classic profile — fast sandboxes reject them fail-closed until the fastlet nft path enforces them. Images publish now; SDK/CLI packages ship at the 1.2.0 stable bump. Please test and report issues.
Highlights
- Credential Vault (OSEP-0022) lands end to end for fast sandboxes: the egress sidecar holds the vault with durable per-subject files, sealed snapshot transfer for pause/resume, and stale-subject takeover on slot reuse (#2249 #2252 #2255 #2250); the server exposes the user-reachable
POST|GET|PATCH|DELETE /v1/sandboxes/{id}/credential-vaultAPI over the FastPath handler relay with credential values never appearing on any public read (#2263), and the MITM CA is pinned per pool so an egress restart no longer rotates the trust anchor under live sandboxes (#2251) - Network policy mirrors the vault: the recorded fast-sandbox policy becomes egress-held state with relay CRUD, durable persistence, and pause/resume snapshot transfer (#2309), and
NetworkRulegains port-scoped rules end to end in the classic profile — spec, sidecar nftables, server validation, and all five SDKs, while fast sandboxes rejectportsfail-closed for now — plus an operator-side TCP port floor that no user policy can open (#2212) - Lifecycle and diagnostics polish:
lifecycle.renewalPolicyreplaces the opaque renew extensions key (OSEP-0009) (#2296); the diagnostics context API reaches server, all five SDKs, and the CLI (#2292); execd fixes make start failures distinguishable, preserve session output byte-for-byte, and stop silent disk growth (#2303 #2318 #2295)
Server
- Implement the server side of the Credential Vault (OSEP-0022, fast-sandbox profile): the user-reachable
POST|GET|PATCH|DELETE /v1/sandboxes/{id}/credential-vaultAPI pushes complete vault revisions over the FastPathInvokeHandlerActionhandler relay and treats the relay response as the synchronous egress-validated ack — the CRD never carries vault material, and every public read is sanitized server-side; PATCH composes in memory (delete → replace → add, each name resolving at most once) withexpectedRevisionas the cross-replica backstop, non-fsb backends proxy the sidecar's/credential-vaultsurface, and the fsbcredentialProxycreate-time rejection is lifted (#2263) - Bake the pinned MITM CA into server-created SandboxTemplates (OSEP-0022): new
[kubernetes] template_mitm_ca_secret/template_mitm_ca_secret_keyconfig (default off) emitsspec.egressMitmCertificate— the certificate entry only, never the private key — on every server-created template; the Secret must exist in each template namespace (ops-runbook distribution), and a missed copy fails the build observably (#2288) - Add
lifecycle.renewalPolicy(OSEP-0009): the renew-on-access opt-in moves from the opaqueextensions["access.renew.extend.seconds"]create key to a structuredrenewalPolicy.extendSeconds(300–86400), mirroring theidlePolicypattern; the legacy key is deprecated (supported throughout 1.x, removed in 2.0), setting both is accepted only when the parsed values match, the value normalizes into the existing Docker label / Kubernetes annotation so pre-upgrade sandboxes keep renewing, and fast sandbox rejects it with 400 rather than silently accepting a no-op (#2296) - Implement the diagnostics context endpoint (OSEP contract from rc.1):
GET /v1/sandboxes/{sandboxId}/diagnostics/contextreturns the sandbox ID, lifecycle state, backend type, and backend-defined identifiers for Docker, Kubernetes, and Fast Sandbox without connecting to execd — Kubernetes context stays available before a Pod is allocated, and extension backends that don't implement it return 501 (#2292) - Forward the still-encoded path through the sandbox proxy: Starlette bound the
{full_path:path}route from the percent-decoded path and the proxy pasted that value back into the upstream URL, so a decoded?started a query,#a fragment,%2Fbecame a separator,%25and non-UTF-8 escapes were rewritten, and%2E%2Esegments were resolved — the proxy now takes the client's still-encoded tail from the raw path (validating it decodes to the routed path, with a re-encode fallback) and reads the query from the raw query string; backends receive request targets byte-for-byte, matching the ingress gateway, and the%2E%2Eprefix-escape on Docker host-mapped endpoints is closed (#2228) - Report BatchSandbox expiry while the pod is still Running: a past
spec.expireTimewas still reported asRunning(Ready pod with IP), so wall-clock billing kept growing after the configured timeout; expiry now checks before the ready states and returnsTerminated/SANDBOX_EXPIREDwithlast_transition_atset to theexpireTimeitself (#2238) - Reject gVisor snapshots that would silently drop writes:
docker commitunder gVisor's default--overlay2=root:selfcan reportREADYwhile restoring none of the sandbox's writes (only the opaque/.gvisor.filestore.*entry is visible); snapshot preflight now resolves the container's runtime and refuses gVisor unless the effective args include--overlay2=none, and create fails closed when the filesystem diff is filestore-only (#2241) - Apply
[docker]security profiles andsecure_runtimeto the execd cache container: it ignoredapparmor_profile/seccomp_profile/docker_runtime, so on a host whose default seccomp profile blocks the execd image, everyPOST /sandboxesfailed 503 with no configuration escape (#2254) - Make concurrent Docker metadata patches safe: the store's read-modify-write was unlocked and every write shared one temporary name, so two concurrent
PATCH /metadatarequests for a sandbox raced into a 500 (FileNotFoundErroron rename) or silently dropped the other's key; a store-wide lock plus per-write temp names serialize patches (last write wins per key), a failed write no longer leaves its temp file behind, and the endpoint contract is now documented per runtime — Kubernetes patches write back the whole label set it read, fsb patches send only their own keys fenced by UID and generation (#2267 #2281) - Resolve proxy endpoints off the event loop: HTTP/WebSocket proxy handlers called the synchronous
get_endpointdirectly — a Docker socket call per proxied request on Docker, an apiserver read on Kubernetes cache misses — blocking the loop for every other request and stream; a slow Docker daemon (e.g. during an image pull) was enough to freeze the server; the lookup now runs viaasyncio.to_threadwith tenant context preserved (#2268) - Apply the renew cooldown when the Redis consumer is enabled: proxy renews skipped
min_interval_secondsentirely (every proxied request ran aget_sandbox+ expiration renew) and queue intents were only throttled per ingress replica, so N replicas renewed up to N times per interval; the cooldown now applies to every renew source and starts only after a successful renew (#2276 #2279)
SDKs
Code changes below ride the 1.2.0 stable release — rc ships no SDK packages.
- Add the Credential Vault client contract to all five SDKs (OSEP-0022): template-backed sandboxes route vault calls through the server on the same template-origin signal as network policy (
OPEN-SANDBOX-ORIGIN: templateor the localcreateFromTemplateflag), while sidecar-direct adapters stay untouched; Python'sSandbox.credential_vaultnow returns the vault service for both profiles, and Go gainsSandbox-level vault methods branching on the same signal (#2263) - Add
lifecycle.renewalPolicyto all five SDKs (OSEP-0009): a typed create field with client-side fail-fast validation (extendSeconds300–86400) wired through the create converters, replacing the opaque extensions key (#2296) - Implement diagnostics across the SDK surface (with #2292):
contextis exposed through both the manager and sandbox facades in all five languages, and JavaScript, Go, and C# gain public logs/events APIs matching the existing Python/JVM surface (custom factories keep working — the new diagnostics service is optional) - Support port-scoped egress rules in all five SDKs (#2212):
NetworkRule.ports(TCP 1–65535, at most 256 per rule,targetnow optional) with client-side validation mirroring the server (range, duplicates, cap, domain+ports rejected, netmask-CIDR+ports rejected); JavaScript encodes the target-or-ports invariant as a union type, Python normalizes a blank wire target on read-back
Python
- Stop propagating a cancelled endpoint fetch to waiters: the shared-fetch endpoint cache set the fetching caller's
CancelledErroron the shared future, so concurrent waiters — e.g.TaskGroupsiblings — raisedCancelledErrorthough nothing cancelled them; waiters now re-resolve (cached endpoint, newer in-flight fetch, or their own), while the cancelled caller still raises as before (#2277) - Forward
credential_proxythrough sandbox pools:PoolCreationSpec.credential_proxywas dropped by both sync and async creation paths, so warmup and direct-create sandboxes silently lost the requested credential-proxy configuration; custom factories stay compatible (the keyword is omitted when unset) (#2283) - Regenerate the lifecycle client to drop stale hardcoded defaults (
timeout_seconds: 2,wake_on_access: true→ UNSET) left behind by the spec change in #2212; wire behavior is unchanged — the server applies the same defaults (#2280)
Go
- Authenticate proxied data-plane requests with the API key: proxy mode sent the tenant API key — a credential that can create, pause, and delete every sandbox of the tenant — into the untrusted sandbox as
X-EXECD-ACCESS-TOKEN(and as egress auth) while omitting theOPEN-SANDBOX-API-KEYheader the server middleware reads, so multi-tenant proxy requests failed 401 and surfaced as readiness timeouts; the tenant key now rides the server header and is never smuggled into the sandbox or egress sidecar, endpoint-supplied and caller-supplied proxy keys win regardless of casing, and direct mode stays key-free (#2274)
Controller
- Fix task-executor treating an empty exit file as exit code 0: the shim creates the exit file before writing it, and
Inspectignored the parse error — a shim killed in between (or a failed write, e.g. full disk) reported a failed or interrupted task asSucceeded; an invalid or empty file now falls through to the pid check (Runningwhile alive,ProcessCrashedonce gone) (#2270)
Execd
- Preserve bash session output line endings:
run_in_sessionstdout events dropped\n/\rbytes, so blank lines were unrecoverable and concatenating a run's events did not reconstruct its output; event text now carries its original line ending (blank lines surface as events containing only the ending), aligning the session stream with the foreground/commandcontract — consumers should concatenate event text per run instead of treating each event as a newline-free line (#2318) - Report
exit_codeon command errors: a command that ran and exited non-zero and a command whose process never started (e.g.fork/exec: argument list too long) reported the same error event, so clients reading the exit status fromevaluecould report a start failure as success; the new optionalerror.exit_codeis present only when the process ran and finished non-zero (-1on signal termination) (#2303) - Serve
bindsmount points through the isolated session files API: the merged-view router only registered overlay mounts, sosession.filesfailed withErrPathOutsideOverlayson the same bind destinations thatsession.runcould read; each bind now registers a routed view (dest → source, read-only binds reject writes, bind views shadow overlays on prefix ties) (#2299) - Truncate command output files before unlink: forked children (daemons) legitimately outlive commands while holding the output fd, so an unlinked file kept consuming disk as invisible
(deleted)growth until the last holder exited; every existing deletion point now truncates to zero first, and background output is only touched at janitor deletion time so the retention window stays readable (#2295) - Omit invalid upper-usage metrics on scan errors: a failed filesystem scan emitted a misleading
execd.isolation.upper.usage_bytesof 0 while sessions and data still existed; invalid readings are now omitted (a valid empty upper still reports 0), and quota admission fails closed whenupper_max_bytes > 0and a scan fails instead of silently bypassing accounting (#2257) - Avoid redundant JDK MITM CA imports: sandboxes with several installed JDKs often share one trust store, but each JDK re-ran the keytool import (~6.2 s for four JDKs); initialization is now deduplicated by resolved trust-store path with certificate-content comparison, cutting the stage to ~0.6 s while still replacing rotated certificates (#2310)
- Default PTY sessions to
TERM=xterm-256colorwhen the resolved user environment has no usableTERM, so interactive programs get colors; an explicitTERMalways wins (#2313)
Networking
egress
The OSEP-0022 vault items concern the fast-sandbox profile's credential-vault surface; the OSEP-0023 follow-ups remain scoped to the default-off experimental revision runtime.
- Add the egress-held vault for fast sandboxes (OSEP-0022), landing in stages: the internal
/vault/{sandboxUid}relay surface with create-only POST (409 on duplicate, atomic across server replicas), full-revision PUT withexpectedRevisionand anexpectedIncarnationprecondition so a stale writer can never land on a reset's successor, value-including GET, and DELETE that keeps the policy (#2249); durable per-subject persistence — one file per subject under a Pod-scoped volume (optional tmpfs), the same commit protocol as policy files so acked means durable and applied, incarnation-fenced reload, and purge on unbind (#2252); and the superseded fastlet-proxy vault route removed with its pending-push cache (#2247) - Carry the vault across pause/resume (OSEP-0022 "Vault Lifecycle"):
VAULT_EXPORTcopies the vault file into the snapshot bundle strictly before runtime teardown and seals the subject (all vault ops answer 409 while paused; enforcement continues), andVAULT_IMPORTis the only authorized incarnation transfer — the bundle re-stamps with the restored fence beforedata-plane-ready, validated against the current policy before any durable effect; a genuine export failure leaves the subject unsealed and fails the pause checkpoint, so teardown never runs without a bundle and the egress-local vault survives for the retry (#2255); avaultTransfercapability signal lets an egress without vault persistence skip the transfer entirely — the checkpoint proceeds without the vault (pre-transfer semantics) instead of failing (#2290) - Define stale-subject takeover on slot reuse: the fast-sandbox dispatch key is the slot IP and slots are reused, so a
SET_BINDINGcan claim a key still held by a dead-but-registered subject — egress now treats that as authoritative, tearing down the stale subject's nft rules, DNS refcount, MITM redirect, and vault slice under the registry lock before registering the successor deny-first; traffic is never dispatched to stale rules, late unloads stay fence-guarded no-ops, and every failure path leaves each subject either fully registered or fully evicted (#2250) - Pin the fast-sandbox MITM CA (OSEP-0022 "MITM Trust Anchor"): the pool-pinned, versioned CA is seeded from a read-only secret mount into the mitmdump confdir (idempotent, fail-closed on an unloadable pair), replacing generate-on-start — an egress restart no longer rotates the CA and silently invalidates every live sandbox's TLS trust; a missing or mismatched CA refuses only the MITM addon (policy enforcement unaffected) with
mitmCaFingerprint/mitmCaDownsurfaced on/healthzand the actions status, so a template↔pool Secret mispairing is visible at a glance (#2251) - Align the fast-sandbox network policy with the vault pattern: the recorded policy becomes egress-held state — a
/policy/{sandboxUid}relay surface with revision preconditions (unknown subject 404, incarnation mismatch 412, sealed 409), durable per-subject files using the same commit protocol,POLICY_EXPORT/POLICY_IMPORTsnapshot transfer mirroring the vault, and, after an egress restart, the committed file superseding the replayed create-time binding input (#2309) - Support port-scoped egress rules end to end (classic profile):
NetworkRule.ports(TCP 1–65535, ≤256 per rule) makestargetoptional — port-only rules, or ports combined with an IP/CIDR; domain+portsis rejected since domain rules evaluate at the DNS proxy layer; deny-before-allow precedence holds across scoping, and a PATCH merge-key bug that let a second port-only rule evict the first is fixed; an operator-shippedreachable.ports.capwhitelist is enforced as an always-deny ahead of every user rule so evendefaultAction: allowcannot open a non-whitelisted port; the fast-sandbox route rejectsportswith 400 until the fastlet nft path can enforce them (fail-closed, documented) (#2212) - Add static TLS passthrough selectors to the experimental live-revision runtime:
OPENSANDBOX_EGRESS_MITMPROXY_PASSTHROUGH_HOSTSis a bounded exact/wildcard host array applied at ClientHello with 443-overlap rejection; existing nonempty nativeignore_hosts/allow_hostsnow reject startup in live mode — migrate static TLS bypass entries to the array (legacy mode unchanged, no public interceptionMode added) (#2229) - Emit TLS admission counters: one best-effort event per final ClientHello admission feeds
egress.mitm.tls.connections_totalwith closedmode/decision/reasonlabels — it measures admission decisions, not handshakes or requests, and is explicitly not security evidence (#2265) - Add the installation-only subject revision IPC foundation (OSEP-0023 Phase 4): an authenticated v2 Unix-socket channel routes target-bound revision transactions to independent, admission-disabled installation receivers with frozen request bytes and lost-reply reconciliation; no production caller yet (#2284)
- Ship
egress-trace, a read-only admission viewer inside the egress image: live policy state, nft sets and DNS leases with TTLs, real outbound connections annotated with their admission verdict, and aprobe <domain>subcommand separating policy denial from resolver failure; no sidecar behavior change, and decrypted L7 contents are deliberately never shown (#2230)
Misc
- Console: land the third of five planned console PRs — admin and platform ops, with
/api/admin/sandboxesaggregation, per-tenant detail proxying, a pools API (documented as not tenant-isolated), and admin global sandbox list / pools / overview KPIs in the web UI (#2131); preserve empty successful upstream responses in the BFF proxy instead of failing JSON decode on 200/201/205 (#2271) - CLI: add
osb terminal <sandbox-id>, attaching the local terminal to a real PTY inside the sandbox over the execd PTY WebSocket — raw mode withSIGWINCH-driven resize, replay on reconnect, remote exit code propagation, and support for both direct and server-proxy connection modes (interactive TTY, POSIX) (#2313); report out-of-range duration inputs as Click validation errors instead of uncaughtOverflowError/ValueError(#2272) - OSEPs: OSEP-0022 revised — the vault rides the action binding as an AES-256-GCM-sealed revision on one control channel, with the per-pool pinned CA as the MITM trust anchor (#2174); the planned OpenSandbox runtime driver ask is withdrawn (the firecracker driver ships as-is; deny-first timing replaces the pre-create healthz probe) (#2259); MITM CA namespace distribution becomes an ops-runbook deployment prerequisite instead of a resident sync controller (#2282)
- Examples: add an all-in-one Docker image for the DeerFlow example bundling the OpenSandbox server (host Docker runtime) and the DeerFlow backend, for one-command workshop starts without a local Python toolchain (#2262)
- Docs: reflow hard-wrapped markdown prose into single-line paragraphs across 148 files (~7.4k collapsed fragments, rendering unchanged) and codify the prose style rule in
AGENTS.md(#2285) - Tests/CI: cache Go modules for the five Kubernetes nightly e2e jobs (
setup-gosilently skipped its cache without a rootgo.sum, re-downloading the full module graph nightly) (#2306); close the credential-vault happy path in the fast-sandbox smoke — relay CRUD, injection through the real mitm for every auth type, rotation, restart durability, and purge (#2256); dump every fastlet Pod container's logs (including egress) on pause/resume verify failure (#2290) - release: bump platform version to
1.2.0-rc.2across charts and image references
Upgrade & Compatibility
- Breaking: bash-session stdout event text now includes its original line endings, and blank lines appear as events — consumers must concatenate event text per run to recover exact output instead of treating each event as a newline-free line (#2318)
- Behavior change: a BatchSandbox past its
spec.expireTimenow reportsTerminated/SANDBOX_EXPIREDinstead ofRunning, even while the pod is still Ready — callers that treatRunningas alive should check expiry (#2238) - Behavior change: Docker-backed snapshots of gVisor containers without
--overlay2=noneare rejected withSNAPSHOT::UNSUPPORTED_RUNTIME(409) instead of publishing a silently incomplete image (#2241) - Behavior change (experimental egress live mode only): nonempty native
ignore_hosts/allow_hostsnow reject startup — migrate static TLS bypass entries toOPENSANDBOX_EGRESS_MITMPROXY_PASSTHROUGH_HOSTS(#2229) - Deprecation: the
extensions["access.renew.extend.seconds"]create key is deprecated in favor oflifecycle.renewalPolicy.extendSeconds— supported throughout 1.x, removed in 2.0; setting both is accepted only when values match (#2296) - Behavior change: execd upper-directory quota admission fails closed when
upper_max_bytes > 0and a usage scan fails (previously the quota check was silently bypassed) (#2257) - Images:
opensandbox/<component>:release-1.2.0-rc.2(all three registries) - Packages: unchanged — SDK/CLI artifacts publish at the
1.2.0stable release - Charts: render from this tag (
helm template ./manifests/charts/opensandbox) - Kubernetes: supported range v1.21 – v1.34 (charts declare
kubeVersion: ">=1.21.1-0") - Skew: server ↔ CLI/SDK same line supported; ±1 minor warns
👥 Contributors
Thanks to these contributors ❤️
- @Aling33
- @BootstrapperSBL
- @JasonBuildAI
- @Jun516
- @Pangjiping
- @acodercat
- @alex-feel
- @comqx
- @hardness1020
- @hpliStartAgain
- @ninan-nn
- @pingkui
Artifacts
- Images:
opensandbox/<component>:release-1.2.0-rc.2(Docker Hub / GHCR / ACR)
Installation
# Platform (Kubernetes) — render the chart at this tag and apply
git clone https://github.com/opensandbox-group/OpenSandbox
cd OpenSandbox
git checkout release-1.2.0-rc.2
helm dependency build manifests/charts/opensandbox # package file:// sub-charts (not committed)
# the default render needs required values (server env / api key) and the
# ingress gateway — see docs/deployment before applying
helm template ./manifests/charts/opensandbox | kubectl apply -f -
# or point your GitOps platform (Argo / Flux) at the repo path + tagVerify what you installed against the BOM: see Release Verification.