github skypilot-org/skypilot v0.11.0
SkyPilot v0.11.0

latest releases: v1.0.0.dev20261118, v1.0.0.dev20261117, v0.14.3...
11 months ago

SkyPilot v0.11.0: Multi-Cloud Pools, Fast Managed Jobs, Enterprise-Readiness at Large Scale, Programmability

SkyPilot v0.11.0 delivers major new features: Pools and Managed Jobs Consolidation Mode; significant improvement enterprise-readiness at large scale: supports hundreds of AI engineers with a single API server instance, avoid OOM, >10x performance improvement on many requests, additional observability, and more; and UX improvements: Templates, Python SDK, CI/CD, Git Support, and more.

Get it now with:

uv pip install "skypilot>=0.11.0"

Or, upgrade your team SkyPilot API server:

NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.11.0

helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
  --set apiService.image=berkeleyskypilot/skypilot:$VERSION \
  --version $VERSION --devel --reuse-values

Highlights

[Beta] SkyPilot Pools: Batch inference across clouds & k8s

SkyPilot supports spawning a pool that launches a set of workers across many clouds and Kubernetes clusters (docs, #6260, #6426, #6459, #6552, #6591, #6665, #6675, #7332, #7963, #8008, #7876, #8047, #7930,#8039, #7846, #7855, #7919, #7920). Jobs can be scheduled on this pool and distributed to workers as they become available.

Key benefits include:

  • Fully utilize your GPU capacity across clouds & k8s
  • Unified queue for jobs on all infra
  • Keep workers warm, scale elastically
  • Step aside for other higher priority jobs; reschedule when GPUs become available

Learn more in our blog post.

batch_inference_architecture

[GA] Managed Jobs Consolidation mode

Consolidation Mode is general available (#7122, #7127, #7396, #7459, #7498, #7560, #7601, #7619, #7717, #7720, #7847, #8082, #8021, #8106). This enables:

  • 6x faster job submission
  • consistent credentials across the API server and jobs controller.
  • Persistent managed jobs state on postgres
# config.yaml
jobs:
  controller:
    consolidation_mode: true  # Currently defaults to False.

Frame 5

Efficient and Robust Managed Jobs

We have significantly optimized the Managed Jobs controller, allowing it to handle 2000+ parallel jobs on a single 8-CPU controller—an 18x improvement in job capacity with the same controller size (#7051, #7371, #7379, #7408, #7432, #7473, #7487, #7488, #7494, #7519, #7585, #7595, #7945, #7966, #7979,#8036, #8095).

image 1

Enterprise-Ready SkyPilot at Large-scale

  • Support hundreds of AI engineers with a single SkyPilot API server instance
  • Significant reduction in memory consumption and avoid OOM for API server

image_1

  • CLI/SDK/Dashboard speed up with a large amount of clusters, jobs

image_2

  • Comprehensive API server metrics for operation

image_3

  • SSO support with Microsoft Entra ID

image_4

Kubernetes

UX, Robustness, Performance Improvement

  • Robust SSH for SkyPilot cluster on Kubernetes
  • Robust multi-node(pod) provisioning with retry and recovery (#7820, #7854, #7852)
  • Improve resource cleanup after termination
  • Intelligent GPU name detection
  • Improved volume support: label support, name validation, SDK support.

Volume Support for Existing PVC and Ephemeral Volumes (#7915, #7971,#8179)

  • Exising PVC: Reference pre-existing Kubernetes PersistentVolumeClaims as a SkyPilot volume (#7915).

    # volume.yaml
    name: existing-pvc-name
    type: k8s-pvc
    infra: k8s/context1
    use_existing: true
    config:
      namespace: namespace
  • Ephemeral Volumes: automatically create volumes when a cluster is launched and deleted when the cluster is torn down, making them ideal for temporary storage across multiple nodes, such as caches and intermediate results.

    # task.sky.yaml
    file_mounts:
      /mnt/cache:
        size: 100Gi

Cloud Support

CoreWeave Integration Announcement

CoreWeave now officially support SkyPilot (#6386, #6519, #6895, #7756, #7838). This integration provides: Infiniband Support, Object Storage, Autoscaling.

See the announcement on Coreweave blog.

image_5

AMD GPU Support Announcement

SkyPilot now fully supports AMD GPUs on Kubernetes clusters (#6378, #6944). This includes: GPU Detection and Scheduling, Dashboard Metrics, ROCm Support.

See the announcement on AMD Rocm blog.

image_6

More Clouds Support

  • Together AI Instant cluster
  • Seeweb support

User Experience

SkyPilot Templates

SkyPilot now ships predefined YAML templates for launching clusters with popular frameworks and patterns. Templates are automatically available on all new SkyPilot clusters. (#7935, #7965)

You can now launch a multi-node Ray cluster by adding a single line to your YAML’s run block:

run: |
  # One-line setup for a distributed Ray cluster
  ~/.sky/templates/ray/start_cluster

  # Submit your job
  python train.py

Programmability: SkyPilot Python SDK

SkyPilot Python SDK is significantly improved with:

  • Type hints

image_7

  • Log streaming
logs = sky.tail_logs(cluster_name, job_id, follow=True, preload_content=False)
for line in logs:
    if line is not None:
        if 'needle in the haystack' in line:
            print("found it!")
            break
logs.close()
  • Admin policy: build admin policy with helper functions
resource_config = user_request.task.get_resource_config()
resource_config['use_spot'] = True
user_request.task.set_resources(resource_config)

Integrating with CI/CD

With the improved SDK, we can integrate SkyPilot with GitHub Actions and other orchestrator to spin off your AI workloads automatically. (#7932)

image_8

Native Git Support

You can now use your private git repositories directly as your SkyPilot workdir (#6294, #6257, #6268). SkyPilot handles the cloning and syncing automatically. You can also use --git-url and --git-ref options with sky serve up (#8012).

# task.sky.yaml
workdir:
  url: <https://github.com/my-org/my-repo.git>
  ref: 1234ab # commit hash or branch name

You can also find the commit hash of your workdir in the Dashboard:

image_1 1

Autostop/Autodown Based on SSH Sessions

In addition to running jobs on clusters, you can now configure autostop/autodown to wait for active SSH sessions (#6361, #6485).

# task.sky.yaml
resources:
  autostop:
    wait_for: jobs_and_ssh

Examples: Distributed LLM Pretraining/Finetuning/Serving

We released high-performance distributed training examples for large models with checkpointing support (#6525, #6551, #6242, #6273, #6443).

  • Kimi-K2 multi-node serving (#7988)
  • Torchtitan framework support
  • Llama 4 Maverick 400B: Training on more than 16 H200 GPUs.
  • OpenAI GPT-OSS: Model pretraining and finetuning.
  • Agentic training: VeRL example for reinforcement learning.

image_2 1


What's new: compared to v0.10.5

API Server: Robustness & Security Improvement

  • TLS support in Helm: Added TLS (HTTPS) support to the Kubernetes ingress configuration for SkyPilot's Helm charts (#6864, #8046).
  • Support for air-gapped environment for SkyPilot API server deployment: use private container registries in the Helm chart (#7955, #7973,#8076).
  • More robust SSO authentication for SSH connections: Server-side SSO and service account tokens are now properly enforced for WebSocket connections used by the SSH proxy system (#7863, #7861).
  • Memory leak fix: Fixed resource cleanup in status refresh daemon when errors occur (#8100).
  • Performance improvement with PostgreSQL Database connection pooling (#7829, #8005).
  • Fast rolling update by avoiding PVC issue with grafana deployment (#7904)
  • Better message during API server upgrade (#7921)
  • API server metrics: CPU/memory (#7923), limit the Prometheus scraper to the current API server (#7922), executors (#7980)
  • Avoid API server hanging due to event loop blocking (#8097, #8101)
  • Fixed user info handlers (#8234)
  • Backward compatibility: Fixed server/client version compatibility issues (#7873).
  • Deprecation: rbac.rules → rbac.namespaceRules; ingress.nodePortEnabled → ingress-ngnix.controller.service.type=NodePort (see details) helm chart

Kubernetes Improvements

  • Make Pod status visible during cluster launching: Better experience with detailed pod event messages when pods are stuck pending (e.g., volume mount failures, image pull issues) (#7959, #8062).
  • Post-provision commands support: Execute custom shell commands on Kubernetes pods during instance initialization with post_provision_runcmd (#7943).
  • Rsync race condition fix: Fixed intermittent "rsync not found" errors during pod setup (#7844, #8023).
  • GPU labeler fix for race condition on Kubernetes clusters with multiple API server and dependency (#7779, #8126, #8111).
  • Fix allowing all contexts: ****Fixed feature support detection to evaluate all contexts in allowed_contexts configuration (#7878).

Core & Backend

  • Fixed --retry-until-up: Fixed the -retry-until-up flag to properly retry failed launches across all cloud zones (#8079).
  • Fixed premature log termination for cluster provision log sky logs --provision during cluster launches (#7888).
  • Add AWS AMI metadata caching to reduce AWS API calls and avoid credential issues (#7961, #7998, #8000, #8013, #7997).
  • Robustify SkyPilot runtime setup (#7985, #7901)
  • Speed up the storage check (#8044)
  • Fixed Fluent-bit GPG key download for cloud logging setup (#7898).
  • Fixed regression in service/pool cleanup process (#8047).
  • Fixed backward compatibility for jobs queue API with older server versions (#8015).
  • Faster SSH tunnel opening and cleanup fix for grpc connection: Optimized SSH tunnel opening under high concurrent load from 8+ seconds to ~3.7 seconds (#7900, #7887, #8105).

Infrastructure & Cloud Integrations

  • Support GCP B200 spot instances (#8125).
  • AWS Trainium & Inferentia support improvement: dynamically detects Trainium/Inferentia instances via the AWS API, maps them correctly for sky show-gpus, fix the default Neuron-based AMI (#7896, #7958).
  • Fixed AWS g6f, gr6f instance fetching (#7949)
  • AWS config file support: Added support for AWS_CONFIG_FILE environment variable for custom credential paths (#8050).
  • Azure: Updated Azure provisioning to use Standard SKU public IP addresses after Azure retired Basic SKU and fix dependency, and fix credential checking (#8007, #7909, #7996, #7924).
  • Samsung Cloud Platform (SCP) Multi-node Support (#7288).
  • Seeweb: Added Docker image support for Seeweb cloud provider (#7493).
  • RunPod: Added CPU-only instance support, fix dependency, and default image (#7460, #8018, #7906).
  • Nebius disk cleanup: Fixed resource leak when VM creation fails due to quota errors (#8004).

SSH Node Pools

  • Per-Context Configuration: Define different pod_config and custom_metadata specifications for multiple SSH node pools through context_configs (#7660, #7913).
  • Provision Timeout: New provision_timeout field for controlling SSH node provisioning wait times (#7660).
  • Fix provisioning deadlock when using custom images (#7883).
  • Fixed regression where SSH node pool contexts were incorrectly hardcoded (#8087).

User Experience

Admin Control Enhancements

  • Additional information in admin policy: user information (#7800), request type (#7840. #8042).
  • New examples for adming policy: rate limiter policy (#7902, #7903), static GPU quota (#7917)
  • Allow workspace to enable specific capacity of a cloud (#7978)
  • Secrets as SecretStr: Task secrets are now stored using Pydantic's SecretStr to prevent accidental exposure in logs (#8040).

SDK Improvements

  • Resource Configuration API: New Task.get_resources_config() returns resources as a dictionary matching YAML format, and Task.set_resources() now accepts dictionaries—making programmatic resource manipulation as intuitive as editing YAML (#7857).
  • GPU Count in Setup: New SKYPILOT_SETUP_NUM_GPUS_PER_NODE environment variable available during setup phase for configuring software based on GPU count (#7092).

Dashboard Improvements

  • Infra page: keep infra page even when error occurs and more info for workspace (#8085, #7845, #7918, #8099)
  • Dashboard now preserves cached data during transient server errors instead of clearing tables (#7831).
  • Hover over resources column to view complete resource specifications (#7880).
  • Added navigation between pool detail pages and jobs dashboard (#7856).
  • Hide already-selected filter values from dropdown menus (#7884).
  • GPU metrics panel: Added XID error monitoring panel to Grafana dashboard (#8041).
  • GPU metrics fix: Fixed GPU metrics display on cluster detail page (#8025, #8045, #8059).
  • Fixed region display format for jobs to match clusters (#7885).
  • Fixed cluster / jobs filtering by users (#7929)

CLI Improvements

  • Fixed sky show-gpus --all: Fixed JSON serialization error when listing all accelerators (#7865).
  • Fixed sky cost-report: Fixed JSON serialization error with NumPy float values (#7974).
  • Allow purging a cluster when CloudUserIdentityError (#7934)
  • Fix duplicated log expansion for sky serve logs (#8002)
  • More efficient log streaming (#7951)
  • Include cluster name for the local log dir (#7942)
  • Clearer feedback when updating pools and services (#8010).
  • Deprecation: Removed sky status --kubernetes for the team SkyPilot API server (#7989), removed sky local up --ips in favor of sky ssh up (#8065)

Documentation & Examples

  • Added PyTorch multi-node distributed training documentation (#7848).
  • Added marimo notebook example (#8031).
  • Improved GPT-OSS fine-tuning examples with checkpointing and WandB integration (#7696).
  • Updated SDK examples for vector database and local GPT (#8003).
  • Reorganized Pools documentation with better discoverability (#7897).
  • Improved admin policy documentation with quickstart guide (#7866).
  • Added server-side debug logging documentation (#8001).

Testing & CI/CD

  • Added shared GKE API server with PostgreSQL testing to nightly builds (#7830).
  • Added RunPod and Lambda cloud provider tests to nightly builds (#8051).
  • Added SSO/Okta authentication test coverage (#7877).
  • Improved test reliability and memory usage (#7834, #7841, #7864).
  • Fixed release workflow for GitHub release notes generation (#7079).

Contributors

Thank you to all contributors who made this release possible!

@adocherty, @alex000kim, @aylei, @brianstrauch, @cblmemo, @cg505, @coopslarhette, @cwhitak3r, @DanielZhangQD, @hyoxt121, @kevinmingtarja, @kyuds, @lloyd-brown, @massaindustries, @Michaelvll, @rohansonecha, @romilbhardwaj, @SalikovAlex, @seongsukwon-moreh, @SeungjinYang, @zpoint

New Contributors:

Special thanks to the community for bug reports, feature requests, and pull requests that helped improve SkyPilot!

Full Changelog

For a complete list of changes, see the commit history.

Don't miss a new skypilot release

NewReleases is sending notifications on new releases.