SkyPilot v0.14.0: Dynamic Job Groups, Multi-User Slurm, AWS Neuron, Read-Only Workspaces, and More
SkyPilot v0.14.0 introduces dynamic job groups, letting a running job group launch more managed jobs that are tracked and cancelled with it, and makes Slurm a multi-user backend: jobs are submitted as the authenticated user, with per-user QoS/account settings and stop/start for container clusters. This release also adds AWS Trainium/Inferentia (Neuron) on Kubernetes, read-only visibility for private workspaces, and a conda-free default image.
Get it now with:
uv pip install "skypilot[all]==0.14.0"Or, upgrade your team SkyPilot API server:
NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.14.0
helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
--set apiService.image=berkeleyskypilot/skypilot:$VERSION \
--version $VERSION --devel --reuse-valuesDeprecations & Breaking Changes:
- Conda is no longer installed by default (#10132).
provision.install_condanow defaults tofalse, and conda has been removed from the default Kubernetes images, which instead ship system Python plus a default user venv (~/sky-user-env). Setprovision.install_conda: trueto opt back in; the opt-in install now uses a current Miniforge with conda-forge as the only channel. - Job Groups enforce in-group networking by default (#10297, #10330). Groups whose tasks cannot be co-located now fail at submission instead of being silently spread across clusters, and networking setup failures fail the job. Set
inter_connection: falseon the group to allow cross-cluster placement. - Tighter API authorization: request listing and tracking endpoints are scoped to the calling user (#10274, #10278),
sky debug-dumpis admin-only (#10277), download paths are authorized against the user's own roots (#10683), principals with an unrecognized role are denied (#10501), and non-admins can no longer grant the admin role via service accounts (#9932, #10482). rbac.restrict_config_to_adminsnow defaults totrue(#10042): users with theuserrole can no longer read the API server config. Set it tofalseto restore the previous behavior.- Kubernetes: use
disk_sizeinstead of theephemeral_storagefield (#9273). - Async request submission no longer prints the API stream link (#10081).
Highlights
[New] Dynamic Job Groups
A task in a job group can now launch further managed jobs with sky jobs launch or sky.jobs.launch, and they attach to the group automatically as dynamic tasks (#10698, #10701, #10702, #10725, #10734, #10736, #10732). Dynamic tasks are listed under the group in sky jobs queue and the dashboard, numbered after the group's own tasks, addressed like any other task (sky jobs logs 122 3, sky jobs cancel 122 --task 3), and cancelled together with the group. A common use is an eval watcher that launches one evaluation job per checkpoint while the trainer keeps training:
---
name: train-and-eval
execution: parallel
primary_tasks: [trainer, eval-watcher]
---
name: trainer
resources:
accelerators: H100:8
run: python train.py --checkpoint-dir /checkpoints
---
name: eval-watcher
resources:
cpus: 2
run: |
for ckpt in $(python watch_checkpoints.py /checkpoints); do
sky jobs launch -y -d -n "eval-$ckpt" eval.yaml --env CKPT=$ckpt
done$ sky jobs queue
ID TASK NAME ... STATUS
122 - train-and-eval ... RUNNING
↳ 0 trainer [P] ... RUNNING
↳ 1 eval-watcher [P] ... RUNNING
↳ 2 eval-step-1 ... SUCCEEDED
↳ 3 eval-step-2 ... RUNNING
↳ 4 eval-step-3 ... RUNNINGJob groups also gain an inter_connection: true | false knob (#10297): networking is a real guarantee when enabled, and false skips the networking machinery entirely so tasks can be placed across clusters or clouds.
[New] Multi-User Slurm
Slurm clusters shared through one API server can now run each user's work under their own Unix identity (#10294, #10477, #10685, #10686). With slurm.submit_as_user, provisioning, srun, rsync, cancellation, and SSH all run as the mapped user, so Slurm accounting, fairshare, and file permissions apply per person. Service accounts default to their creator's identity. A new slurm.quota block maps to sbatch --qos / --account and resolves per workspace (#10680, #10631).
slurm:
submit_as_user: true
username_map:
jane.doe@example.com: jdoe
inference-prod: inference-svc
quota:
queue: normal # -> #SBATCH --qos=normal
account: pre-training # -> #SBATCH --account=pre-trainingSlurm container (Pyxis) clusters also support sky stop / sky start, snapshotting each rank's filesystem to shared storage and restoring it into a new allocation (#10544), plus autostop (#10645) and host path volume mounts (#10438).
[New] AWS Trainium & Inferentia on Kubernetes
SkyPilot now schedules AWS Neuron accelerators (Trainium and Inferentia) on Kubernetes (#10169). It recognizes the aws.amazon.com/neuron resource, detects accelerator type and count from node labels (including Karpenter), adds the Neuron taint toleration, and lists Neuron devices in sky show-gpus.
sky launch --gpus Trainium2:16 --infra k8sOther new hardware in this release: GCP RTX PRO 6000 Blackwell (G4) (#9634), InfiniBand on Nebius B200/B300 (#10369), OCI GB200/GB300 NCCL profiles for network_tier: best (#10204), and SR-IOV RDMA delivery on Kubernetes via kubernetes.rdma (#10554).
[New] Read-Only Visibility for Private Workspaces
A private workspace can now let non-members see it and its clusters and managed jobs (including logs) without being able to modify anything (#10228). Members and admins keep full access. The same change hardens per-resource write isolation so mutating an existing cluster is always gated by the cluster's own workspace.
workspaces:
private-ws:
private: true
read_access: all # default: allowed_users
allowed_users:
- alice@example.comSet workspace_config.read_access: all to make this the org-wide default.
Why Is My Job Not Running?
Managed jobs now explain themselves. sky jobs queue -v shows the latest launch-progress event for a waiting job, such as a Slurm pending reason or a Kubernetes image pull (#10692), and, with API server metrics enabled, SkyPilot exports where a job's start-up time went as metrics broken down by stage (controller queue, provisioning, node start-up, setup) (#10638). Event logs also record who requested a cluster stop/termination or job cancellation (#10621, #10619, #10661).
$ sky jobs queue -v
ID NAME STATUS ... DETAILS
56266 train STARTING ... Launching (pending: Resources; partition: dev)What's new: compared to v0.13.0
API Server
- Run multiple isolated local API servers on one machine via
SKYPILOT_API_SERVER_LOCAL_PORT/sky api start --portandSKY_RUNTIME_DIR(#10213);SKYPILOT_DISABLE_LOCAL_API_SERVERblocks local server autostart (#10093). - The service account token API accepts a role for new tokens (#10115); throttled
last_used_atwrites (#10391). - Grant private-workspace access to users added to
allowed_usersbefore their first login (#10079); surface the resolvedactive_workspaceto admin policies (#10241). - Refuse uploads that cannot fit on the server's disk (#10714); process client task YAMLs in memory (#10435); move chunked-upload work off the event loop (#10556).
- Request lifecycle fixes: mark a request FAILED when enqueue fails (#10617), fail
/api/getfor requests interrupted by a restart (#10257), and reclaim orphaned request logs (#10675, #10657). - Opt-in executor pool sizing that accounts for server and controller memory (
SKYPILOT_MEMORY_AWARE_WORKER_SIZING) (#10550); fixed retention-daemon cadence (#10538) and hourly requests GC (#10555). - Security: never regenerate the JWT signing secret on a DB read failure (#10566); redact credentials from config debug logs (#10511) and basic-auth passwords in printed URLs (#10286); parameterize request-status queries (#10273).
- Helm chart:
storage.existingClaimto attach a pre-provisioned PVC (#10085, #10086),apiService.hostto set the bind interface (#10050), configurable clients-dir subPath (#10101). - Launches blocked on a cluster lock held by another operation release their executor worker while they wait, and the client shows what they are waiting on (#10210, #10291, #10013, #10534).
- Auth runs on its own executor with bounded database lookups (#10313, #10373, #10185); idle-in-transaction time is bounded on PostgreSQL (#10687).
- Configurable state-DB connection pool size (#10791).
- Optional
SO_REUSEPORTfor spreading connections across API server workers (#10016).
Observability & Metrics
- New provisioning, pending-time, and availability metrics with matching Grafana panels and 30-day availability tracking (#10625).
- Managed-job recovery gauges (#10186),
/api/getlatency by request name (#10349), client-side DB instrumentation (#10525), SQLite size and local disk metrics (#10363, #10674), SSH round-trip timing (#10656), and event-loop stall attribution (#10523). /metricsruns on its own event loop, stays responsive when a collector hangs, and caches the multiprocess merge (#10503, #10377, #10573, #10575).- Slurm GPU metrics federated via the login node, with
/federatescoping for a shared Prometheus (#10572, #10603).
Kubernetes
- AWS Neuron (Trainium/Inferentia) support (#10169) and SR-IOV RDMA via
kubernetes.rdma(#10554), with apod_configopt-out from OCI RoCE host networking (#10546). - EFA improvements: co-locate multi-node EFA jobs in one AZ (#10054) and derive EFA count on scale-from-zero clusters (#10053).
- Kueue: don't count admission wait against
provision_timeoutand show when a launch is waiting for queue admission (#10252), and pools now work under Kueue quota (#10718). - Clearer failures: fail on persistent volume mount failures instead of hanging (#10218), detect PVCs that will not bind (#10440), bound pod wait loops and explain missing pods (#10460, #10272), surface which node a terminated cluster lost (#10233), and recommend
set_pod_resource_limitsafter an OOM (#10551). sky local up --num-nodesfor multi-node local clusters (#10138); package-manager-agnostic bootstrap and graceful Docker CLI skip on non-Debian images (#10077, #10382).- Report node GPU total from capacity with allocatable alongside (#10395); display actual pod requests when they diverge from the spec (#10214); check ephemeral storage when fitting instances (#10131).
- Let pods with DRA claims terminate gracefully (#10751); request gzip-compressed API responses (#10153); fixed a logging fork deadlock on Python < 3.13 (#10104) and multi-node rank crashes on multi-NIC / hostNetwork nodes (#9924).
Volumes
- Scoped auto-mounts:
kubernetes.auto_mountsentries takescope: personal | workspace | global(#10411), and the volume listing is filtered to the caller's workspaces (#10531). - Refresh a volume's size from the cloud and show pending resizes (#10549, #10553); show why a volume is not ready (#10486); fail early when an auto-mounted volume is not ready (#10402).
- Volume page filtering (#10316); don't fail launches when tagging AWS volumes is denied (#10627); validate volumes at creation (#10513).
Managed Jobs & Serve
- Dynamic job groups and the
inter_connectionknob (see Highlights). - Server-side infra filtering for
sky jobs queue(#10613) and smaller payloads for field-filtered queries (#10176). - Controller resilience: emergency recovery for unexpected controller errors (#9868, #10416), retry transient DB and cluster-status errors (#10655, #10040), fail deterministic task-prep errors fast (#10326), and fail jobs whose pool no longer exists (#10203).
- Exit code and log pointer in RECOVERING events (#10256);
--num-jobswithout a pool (#10552); fixed partial state writes on Job Group sync failure and parallelized group teardown (#10415). - Serve: fixed
least_loadreplica decrement and rotation (#10445).
Core & Backend
- Launch failover stops trying further zones when a provisioner reports a request it can never serve, such as starting a legacy Nebius preemptible VM (#10697).
- Region-free cost estimates priced with the joint regional minimum (#10199).
- Extract external links (e.g. W&B) from job logs server-side (#9934); concise health-probe failures in unhealthy-cluster events (#10201).
sky debug-dump: fast-fail unreachable kube contexts, parallel per-cluster collection, and bounded controller steps so a stuck controller cannot hang the dump (#10041, #10161, #10068).- More robust runtime installs: fall back when an apt repository or distro mirror is unusable (#10673, #10512).
- Fixed Ray job submit failures from a dashboard-agent port conflict (#9856); don't abort graceful teardown when command runners are unavailable (#10604); guard against a missing
~/.bashrcand missing Git (#10245, #10230).
Infrastructure & Cloud Integrations
Slurm
- Submit as authenticated users,
username_map, andquota.queue/quota.account(see Highlights). - Container cluster stop/start and autostop (#10544, #10645, #10652) and host path volume mounts (#10438).
- GPU availability from node
AllocTRES, CPU/memory allocation and usage in the Slurm node-info API, and batched inventory queries over one SSH connection (#10285, #10293, #10425); best-effort multi-cluster aggregation (#10284, #10289). - Run skylet under an allocation-owned keeper step (#10516); respect custom task CPU demands (#10447); fail fast when a task exits nonzero (#10428).
- Fixed leaked allocations on terminate (#10296), multi-node run-done barrier failures (#10321), step-scoped
SLURM_*env leaking into user scripts (#10309), and containerized runs broken by an exportedwhichfunction (#10716).
AWS
- Skip default security group pre-creation when the user specifies security groups (#10178).
GCP
Azure
- Boot from private Shared Image Gallery images (#9843).
Nebius
- Personalized (per-tenant) pricing (#9107, #10222), InfiniBand on B200/B300 (#10369), and optional boot disk encryption (#10172).
- Catalog fetcher switched to billing v1 (#9829, #10400); wait for instance termination before deleting the security group (#10220).
OCI / RunPod
- OCI: GB200/GB300 NCCL profiles for
network_tier: best(#10204). - RunPod: corrected region to data center mapping (#10072); label account SSH keys with the SkyPilot user (#9975).
User Experience
Dashboard
- Reworked Infra page: unified table, per-GPU-type summary, and responsive layout (#10067), with Slurm clusters broken down by partition, CPU-only and unreachable clusters listed, and progressive per-cluster loading (#10317, #10318, #10329, #10629); node table pagination (#10480).
- Clusters page defaults to "My Clusters" with owner and activity toggles (#9884, #10089).
- Filters are named in the URL for clusters, managed jobs, users, volumes, and infra, so filtered views can be bookmarked and shared (#10455, #10459, #10505).
- Managed jobs: PENDING reason in the status tooltip (#10202), recovery count, Started, and Partition on the detail page (#10004, #10647), and summed job-group durations (#10678).
- URL-template and scoped external links (#10090, #10194); fenced code blocks rendered in error messages (#10088); owning user shown for pools (#10653).
- New SkyPilot branding: logo and favicon (#10181, #10215, #10237).
CLI
sky api loginworks whenSKYPILOT_API_SERVER_ENDPOINTis set and follows HTTP to HTTPS redirects (#10404, #10624).sky jobs logs/sky jobs canceladdress job-group tasks by index (#10734); fixedsky api logstraceback on 4xx (#10341).
Dependencies
- Lifted the
typing_extensionscap and droppedpydantic<2.12(#10649). - Unpinned the Nebius SDK now that 0.4.3 is loop-aware (#10320).
Documentation & Examples
- Dynamic job group docs (#10732), Hugging Face Hub storage guide (#10117), refreshed Sandboxes (#10109) and Sky Batch pages (#10453).
- New examples: Hugging Face storage batch transcription (#10558); slime agentic coding-agent RL on Job Groups with Sandboxes (#10200), and resilient Ray training (#10602) and GB300 NVL72 Ray recovery (#10644) with Dynamic Node Set recovery. Sandboxes and Dynamic Node Set recovery are offered in SkyPilot Platform.
- Updated logos and brand assets across docs and README (#10238, #10179); restyled Ask AI widget with dark mode (#10180).
Testing & CI/CD
- Dashboard tests now run in CI (#10510); golden snapshot tests for the Kubernetes Ray template (#10226).
- Buildkite: global per-test serialization via
concurrency_group(#10150), no auto-retry for exclusive steps (#10385), and cancellation mirrored from GitHub (#10760); nightly tagging and Slack notifications (#10192, #10719). - Moved tests off retired AWS p3/V100 instances (#10098, #10107, #10113); numerous test reliability and flakiness fixes.
Contributors
Thank you to all contributors who made this release possible!
@DanielZhangQD, @aylei, @zpoint, @kevinmingtarja, @ishankaul1, @SeungjinYang, @lloyd-brown, @rohansonecha, @cg505, @stephenkevn, @Michaelvll, @MadhavJivrajani, @trevanth24, @hopechong, @ryxli, @Anai-Guo, @liuxu623, @romilbhardwaj, @ciprianf, @hentt30, @ahluwalij, @alex000kim, @complynx, @concretevitamin, @davanstrien, @daw3rd, @edmsimplicio, @himanshu1573, @intagliated, @JGSweets, @Jobarion, @Khas-Erdene-Tsogtsaikhan, @klenovda, @kyuds, @nicosuter, @pauldonzier, @pavneeta, @samuelv8, @spiffaz, @typesafeainate, @yash170106065
New Contributors
- @ciprianf made their first contribution in #10072
- @nicosuter made their first contribution in #10050
- @spiffaz made their first contribution in #10156
- @edmsimplicio made their first contribution in #10230
- @yash170106065 made their first contribution in #9924
- @pavneeta made their first contribution in #10260
- @liuxu623 made their first contribution in #10284
- @complynx made their first contribution in #10345
- @Anai-Guo made their first contribution in #10356
- @ryxli made their first contribution in #10436
- @intagliated made their first contribution in #9932
- @klenovda made their first contribution in #9829
- @himanshu1573 made their first contribution in #10369
- @typesafeainate made their first contribution in #10172
- @davanstrien made their first contribution in #10558
- @daw3rd made their first contribution in #10716
Full Changelog
For a complete list of changes, see the commit history.