github skypilot-org/skypilot v0.14.0
Release v0.14.0

latest releases: v1.0.0.dev20261120, v1.0.0.dev20261119, v1.0.0.dev20261118...
one month ago

SkyPilot v0.14.0: Dynamic Job Groups, Multi-User Slurm, AWS Neuron, Read-Only Workspaces, and More

SkyPilot v0.14.0 introduces dynamic job groups, letting a running job group launch more managed jobs that are tracked and cancelled with it, and makes Slurm a multi-user backend: jobs are submitted as the authenticated user, with per-user QoS/account settings and stop/start for container clusters. This release also adds AWS Trainium/Inferentia (Neuron) on Kubernetes, read-only visibility for private workspaces, and a conda-free default image.

Get it now with:

uv pip install "skypilot[all]==0.14.0"

Or, upgrade your team SkyPilot API server:

NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.14.0

helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
  --set apiService.image=berkeleyskypilot/skypilot:$VERSION \
  --version $VERSION --devel --reuse-values

Deprecations & Breaking Changes:

  • Conda is no longer installed by default (#10132). provision.install_conda now defaults to false, and conda has been removed from the default Kubernetes images, which instead ship system Python plus a default user venv (~/sky-user-env). Set provision.install_conda: true to opt back in; the opt-in install now uses a current Miniforge with conda-forge as the only channel.
  • Job Groups enforce in-group networking by default (#10297, #10330). Groups whose tasks cannot be co-located now fail at submission instead of being silently spread across clusters, and networking setup failures fail the job. Set inter_connection: false on the group to allow cross-cluster placement.
  • Tighter API authorization: request listing and tracking endpoints are scoped to the calling user (#10274, #10278), sky debug-dump is admin-only (#10277), download paths are authorized against the user's own roots (#10683), principals with an unrecognized role are denied (#10501), and non-admins can no longer grant the admin role via service accounts (#9932, #10482).
  • rbac.restrict_config_to_admins now defaults to true (#10042): users with the user role can no longer read the API server config. Set it to false to restore the previous behavior.
  • Kubernetes: use disk_size instead of the ephemeral_storage field (#9273).
  • Async request submission no longer prints the API stream link (#10081).

Highlights

[New] Dynamic Job Groups

A task in a job group can now launch further managed jobs with sky jobs launch or sky.jobs.launch, and they attach to the group automatically as dynamic tasks (#10698, #10701, #10702, #10725, #10734, #10736, #10732). Dynamic tasks are listed under the group in sky jobs queue and the dashboard, numbered after the group's own tasks, addressed like any other task (sky jobs logs 122 3, sky jobs cancel 122 --task 3), and cancelled together with the group. A common use is an eval watcher that launches one evaluation job per checkpoint while the trainer keeps training:

---
name: train-and-eval
execution: parallel
primary_tasks: [trainer, eval-watcher]
---
name: trainer
resources:
  accelerators: H100:8
run: python train.py --checkpoint-dir /checkpoints
---
name: eval-watcher
resources:
  cpus: 2
run: |
  for ckpt in $(python watch_checkpoints.py /checkpoints); do
    sky jobs launch -y -d -n "eval-$ckpt" eval.yaml --env CKPT=$ckpt
  done
$ sky jobs queue
ID    TASK  NAME              ...  STATUS
122   -     train-and-eval    ...  RUNNING
 ↳    0     trainer [P]       ...  RUNNING
 ↳    1     eval-watcher [P]  ...  RUNNING
 ↳    2     eval-step-1       ...  SUCCEEDED
 ↳    3     eval-step-2       ...  RUNNING
 ↳    4     eval-step-3       ...  RUNNING

Job groups also gain an inter_connection: true | false knob (#10297): networking is a real guarantee when enabled, and false skips the networking machinery entirely so tasks can be placed across clusters or clouds.

[New] Multi-User Slurm

Slurm clusters shared through one API server can now run each user's work under their own Unix identity (#10294, #10477, #10685, #10686). With slurm.submit_as_user, provisioning, srun, rsync, cancellation, and SSH all run as the mapped user, so Slurm accounting, fairshare, and file permissions apply per person. Service accounts default to their creator's identity. A new slurm.quota block maps to sbatch --qos / --account and resolves per workspace (#10680, #10631).

slurm:
  submit_as_user: true
  username_map:
    jane.doe@example.com: jdoe
    inference-prod: inference-svc
  quota:
    queue: normal           # -> #SBATCH --qos=normal
    account: pre-training   # -> #SBATCH --account=pre-training

Slurm container (Pyxis) clusters also support sky stop / sky start, snapshotting each rank's filesystem to shared storage and restoring it into a new allocation (#10544), plus autostop (#10645) and host path volume mounts (#10438).

[New] AWS Trainium & Inferentia on Kubernetes

SkyPilot now schedules AWS Neuron accelerators (Trainium and Inferentia) on Kubernetes (#10169). It recognizes the aws.amazon.com/neuron resource, detects accelerator type and count from node labels (including Karpenter), adds the Neuron taint toleration, and lists Neuron devices in sky show-gpus.

sky launch --gpus Trainium2:16 --infra k8s

Other new hardware in this release: GCP RTX PRO 6000 Blackwell (G4) (#9634), InfiniBand on Nebius B200/B300 (#10369), OCI GB200/GB300 NCCL profiles for network_tier: best (#10204), and SR-IOV RDMA delivery on Kubernetes via kubernetes.rdma (#10554).

[New] Read-Only Visibility for Private Workspaces

A private workspace can now let non-members see it and its clusters and managed jobs (including logs) without being able to modify anything (#10228). Members and admins keep full access. The same change hardens per-resource write isolation so mutating an existing cluster is always gated by the cluster's own workspace.

workspaces:
  private-ws:
    private: true
    read_access: all        # default: allowed_users
    allowed_users:
      - alice@example.com

Set workspace_config.read_access: all to make this the org-wide default.

Why Is My Job Not Running?

Managed jobs now explain themselves. sky jobs queue -v shows the latest launch-progress event for a waiting job, such as a Slurm pending reason or a Kubernetes image pull (#10692), and, with API server metrics enabled, SkyPilot exports where a job's start-up time went as metrics broken down by stage (controller queue, provisioning, node start-up, setup) (#10638). Event logs also record who requested a cluster stop/termination or job cancellation (#10621, #10619, #10661).

$ sky jobs queue -v
ID     NAME   STATUS    ...  DETAILS
56266  train  STARTING  ...  Launching (pending: Resources; partition: dev)

What's new: compared to v0.13.0

API Server

  • Run multiple isolated local API servers on one machine via SKYPILOT_API_SERVER_LOCAL_PORT / sky api start --port and SKY_RUNTIME_DIR (#10213); SKYPILOT_DISABLE_LOCAL_API_SERVER blocks local server autostart (#10093).
  • The service account token API accepts a role for new tokens (#10115); throttled last_used_at writes (#10391).
  • Grant private-workspace access to users added to allowed_users before their first login (#10079); surface the resolved active_workspace to admin policies (#10241).
  • Refuse uploads that cannot fit on the server's disk (#10714); process client task YAMLs in memory (#10435); move chunked-upload work off the event loop (#10556).
  • Request lifecycle fixes: mark a request FAILED when enqueue fails (#10617), fail /api/get for requests interrupted by a restart (#10257), and reclaim orphaned request logs (#10675, #10657).
  • Opt-in executor pool sizing that accounts for server and controller memory (SKYPILOT_MEMORY_AWARE_WORKER_SIZING) (#10550); fixed retention-daemon cadence (#10538) and hourly requests GC (#10555).
  • Security: never regenerate the JWT signing secret on a DB read failure (#10566); redact credentials from config debug logs (#10511) and basic-auth passwords in printed URLs (#10286); parameterize request-status queries (#10273).
  • Helm chart: storage.existingClaim to attach a pre-provisioned PVC (#10085, #10086), apiService.host to set the bind interface (#10050), configurable clients-dir subPath (#10101).
  • Launches blocked on a cluster lock held by another operation release their executor worker while they wait, and the client shows what they are waiting on (#10210, #10291, #10013, #10534).
  • Auth runs on its own executor with bounded database lookups (#10313, #10373, #10185); idle-in-transaction time is bounded on PostgreSQL (#10687).
  • Configurable state-DB connection pool size (#10791).
  • Optional SO_REUSEPORT for spreading connections across API server workers (#10016).

Observability & Metrics

  • New provisioning, pending-time, and availability metrics with matching Grafana panels and 30-day availability tracking (#10625).
  • Managed-job recovery gauges (#10186), /api/get latency by request name (#10349), client-side DB instrumentation (#10525), SQLite size and local disk metrics (#10363, #10674), SSH round-trip timing (#10656), and event-loop stall attribution (#10523).
  • /metrics runs on its own event loop, stays responsive when a collector hangs, and caches the multiprocess merge (#10503, #10377, #10573, #10575).
  • Slurm GPU metrics federated via the login node, with /federate scoping for a shared Prometheus (#10572, #10603).

Kubernetes

  • AWS Neuron (Trainium/Inferentia) support (#10169) and SR-IOV RDMA via kubernetes.rdma (#10554), with a pod_config opt-out from OCI RoCE host networking (#10546).
  • EFA improvements: co-locate multi-node EFA jobs in one AZ (#10054) and derive EFA count on scale-from-zero clusters (#10053).
  • Kueue: don't count admission wait against provision_timeout and show when a launch is waiting for queue admission (#10252), and pools now work under Kueue quota (#10718).
  • Clearer failures: fail on persistent volume mount failures instead of hanging (#10218), detect PVCs that will not bind (#10440), bound pod wait loops and explain missing pods (#10460, #10272), surface which node a terminated cluster lost (#10233), and recommend set_pod_resource_limits after an OOM (#10551).
  • sky local up --num-nodes for multi-node local clusters (#10138); package-manager-agnostic bootstrap and graceful Docker CLI skip on non-Debian images (#10077, #10382).
  • Report node GPU total from capacity with allocatable alongside (#10395); display actual pod requests when they diverge from the spec (#10214); check ephemeral storage when fitting instances (#10131).
  • Let pods with DRA claims terminate gracefully (#10751); request gzip-compressed API responses (#10153); fixed a logging fork deadlock on Python < 3.13 (#10104) and multi-node rank crashes on multi-NIC / hostNetwork nodes (#9924).

Volumes

  • Scoped auto-mounts: kubernetes.auto_mounts entries take scope: personal | workspace | global (#10411), and the volume listing is filtered to the caller's workspaces (#10531).
  • Refresh a volume's size from the cloud and show pending resizes (#10549, #10553); show why a volume is not ready (#10486); fail early when an auto-mounted volume is not ready (#10402).
  • Volume page filtering (#10316); don't fail launches when tagging AWS volumes is denied (#10627); validate volumes at creation (#10513).

Managed Jobs & Serve

  • Dynamic job groups and the inter_connection knob (see Highlights).
  • Server-side infra filtering for sky jobs queue (#10613) and smaller payloads for field-filtered queries (#10176).
  • Controller resilience: emergency recovery for unexpected controller errors (#9868, #10416), retry transient DB and cluster-status errors (#10655, #10040), fail deterministic task-prep errors fast (#10326), and fail jobs whose pool no longer exists (#10203).
  • Exit code and log pointer in RECOVERING events (#10256); --num-jobs without a pool (#10552); fixed partial state writes on Job Group sync failure and parallelized group teardown (#10415).
  • Serve: fixed least_load replica decrement and rotation (#10445).

Core & Backend

  • Launch failover stops trying further zones when a provisioner reports a request it can never serve, such as starting a legacy Nebius preemptible VM (#10697).
  • Region-free cost estimates priced with the joint regional minimum (#10199).
  • Extract external links (e.g. W&B) from job logs server-side (#9934); concise health-probe failures in unhealthy-cluster events (#10201).
  • sky debug-dump: fast-fail unreachable kube contexts, parallel per-cluster collection, and bounded controller steps so a stuck controller cannot hang the dump (#10041, #10161, #10068).
  • More robust runtime installs: fall back when an apt repository or distro mirror is unusable (#10673, #10512).
  • Fixed Ray job submit failures from a dashboard-agent port conflict (#9856); don't abort graceful teardown when command runners are unavailable (#10604); guard against a missing ~/.bashrc and missing Git (#10245, #10230).

Infrastructure & Cloud Integrations

Slurm

  • Submit as authenticated users, username_map, and quota.queue / quota.account (see Highlights).
  • Container cluster stop/start and autostop (#10544, #10645, #10652) and host path volume mounts (#10438).
  • GPU availability from node AllocTRES, CPU/memory allocation and usage in the Slurm node-info API, and batched inventory queries over one SSH connection (#10285, #10293, #10425); best-effort multi-cluster aggregation (#10284, #10289).
  • Run skylet under an allocation-owned keeper step (#10516); respect custom task CPU demands (#10447); fail fast when a task exits nonzero (#10428).
  • Fixed leaked allocations on terminate (#10296), multi-node run-done barrier failures (#10321), step-scoped SLURM_* env leaking into user scripts (#10309), and containerized runs broken by an exported which function (#10716).

AWS

  • Skip default security group pre-creation when the user specifies security groups (#10178).

GCP

  • RTX PRO 6000 Blackwell (G4) support (#9634); federated credentials for GCS COPY (#10212).

Azure

  • Boot from private Shared Image Gallery images (#9843).

Nebius

  • Personalized (per-tenant) pricing (#9107, #10222), InfiniBand on B200/B300 (#10369), and optional boot disk encryption (#10172).
  • Catalog fetcher switched to billing v1 (#9829, #10400); wait for instance termination before deleting the security group (#10220).

OCI / RunPod

  • OCI: GB200/GB300 NCCL profiles for network_tier: best (#10204).
  • RunPod: corrected region to data center mapping (#10072); label account SSH keys with the SkyPilot user (#9975).

User Experience

Dashboard

  • Reworked Infra page: unified table, per-GPU-type summary, and responsive layout (#10067), with Slurm clusters broken down by partition, CPU-only and unreachable clusters listed, and progressive per-cluster loading (#10317, #10318, #10329, #10629); node table pagination (#10480).
  • Clusters page defaults to "My Clusters" with owner and activity toggles (#9884, #10089).
  • Filters are named in the URL for clusters, managed jobs, users, volumes, and infra, so filtered views can be bookmarked and shared (#10455, #10459, #10505).
  • Managed jobs: PENDING reason in the status tooltip (#10202), recovery count, Started, and Partition on the detail page (#10004, #10647), and summed job-group durations (#10678).
  • URL-template and scoped external links (#10090, #10194); fenced code blocks rendered in error messages (#10088); owning user shown for pools (#10653).
  • New SkyPilot branding: logo and favicon (#10181, #10215, #10237).

CLI

  • sky api login works when SKYPILOT_API_SERVER_ENDPOINT is set and follows HTTP to HTTPS redirects (#10404, #10624).
  • sky jobs logs / sky jobs cancel address job-group tasks by index (#10734); fixed sky api logs traceback on 4xx (#10341).

Dependencies

  • Lifted the typing_extensions cap and dropped pydantic<2.12 (#10649).
  • Unpinned the Nebius SDK now that 0.4.3 is loop-aware (#10320).

Documentation & Examples

  • Dynamic job group docs (#10732), Hugging Face Hub storage guide (#10117), refreshed Sandboxes (#10109) and Sky Batch pages (#10453).
  • New examples: Hugging Face storage batch transcription (#10558); slime agentic coding-agent RL on Job Groups with Sandboxes (#10200), and resilient Ray training (#10602) and GB300 NVL72 Ray recovery (#10644) with Dynamic Node Set recovery. Sandboxes and Dynamic Node Set recovery are offered in SkyPilot Platform.
  • Updated logos and brand assets across docs and README (#10238, #10179); restyled Ask AI widget with dark mode (#10180).

Testing & CI/CD

  • Dashboard tests now run in CI (#10510); golden snapshot tests for the Kubernetes Ray template (#10226).
  • Buildkite: global per-test serialization via concurrency_group (#10150), no auto-retry for exclusive steps (#10385), and cancellation mirrored from GitHub (#10760); nightly tagging and Slack notifications (#10192, #10719).
  • Moved tests off retired AWS p3/V100 instances (#10098, #10107, #10113); numerous test reliability and flakiness fixes.

Contributors

Thank you to all contributors who made this release possible!

@DanielZhangQD, @aylei, @zpoint, @kevinmingtarja, @ishankaul1, @SeungjinYang, @lloyd-brown, @rohansonecha, @cg505, @stephenkevn, @Michaelvll, @MadhavJivrajani, @trevanth24, @hopechong, @ryxli, @Anai-Guo, @liuxu623, @romilbhardwaj, @ciprianf, @hentt30, @ahluwalij, @alex000kim, @complynx, @concretevitamin, @davanstrien, @daw3rd, @edmsimplicio, @himanshu1573, @intagliated, @JGSweets, @Jobarion, @Khas-Erdene-Tsogtsaikhan, @klenovda, @kyuds, @nicosuter, @pauldonzier, @pavneeta, @samuelv8, @spiffaz, @typesafeainate, @yash170106065

New Contributors

Full Changelog

For a complete list of changes, see the commit history.

Don't miss a new skypilot release

NewReleases is sending notifications on new releases.