SkyPilot v0.11.0: Multi-Cloud Pools, Fast Managed Jobs, Enterprise-Readiness at Large Scale, Programmability
SkyPilot v0.11.0 delivers major new features: Pools and Managed Jobs Consolidation Mode; significant improvement enterprise-readiness at large scale: supports hundreds of AI engineers with a single API server instance, avoid OOM, >10x performance improvement on many requests, additional observability, and more; and UX improvements: Templates, Python SDK, CI/CD, Git Support, and more.
Get it now with:
uv pip install "skypilot>=0.11.0"Or, upgrade your team SkyPilot API server:
NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.11.0
helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
--set apiService.image=berkeleyskypilot/skypilot:$VERSION \
--version $VERSION --devel --reuse-valuesHighlights
[Beta] SkyPilot Pools: Batch inference across clouds & k8s
SkyPilot supports spawning a pool that launches a set of workers across many clouds and Kubernetes clusters (docs, #6260, #6426, #6459, #6552, #6591, #6665, #6675, #7332, #7963, #8008, #7876, #8047, #7930,#8039, #7846, #7855, #7919, #7920). Jobs can be scheduled on this pool and distributed to workers as they become available.
Key benefits include:
- Fully utilize your GPU capacity across clouds & k8s
- Unified queue for jobs on all infra
- Keep workers warm, scale elastically
- Step aside for other higher priority jobs; reschedule when GPUs become available
Learn more in our blog post.
[GA] Managed Jobs Consolidation mode
Consolidation Mode is general available (#7122, #7127, #7396, #7459, #7498, #7560, #7601, #7619, #7717, #7720, #7847, #8082, #8021, #8106). This enables:
- 6x faster job submission
- consistent credentials across the API server and jobs controller.
- Persistent managed jobs state on postgres
# config.yaml
jobs:
controller:
consolidation_mode: true # Currently defaults to False.
Efficient and Robust Managed Jobs
We have significantly optimized the Managed Jobs controller, allowing it to handle 2000+ parallel jobs on a single 8-CPU controller—an 18x improvement in job capacity with the same controller size (#7051, #7371, #7379, #7408, #7432, #7473, #7487, #7488, #7494, #7519, #7585, #7595, #7945, #7966, #7979,#8036, #8095).
Enterprise-Ready SkyPilot at Large-scale
- Support hundreds of AI engineers with a single SkyPilot API server instance
- Significant reduction in memory consumption and avoid OOM for API server
- CLI/SDK/Dashboard speed up with a large amount of clusters, jobs
- Comprehensive API server metrics for operation
- SSO support with Microsoft Entra ID
Kubernetes
UX, Robustness, Performance Improvement
- Robust SSH for SkyPilot cluster on Kubernetes
- Robust multi-node(pod) provisioning with retry and recovery (#7820, #7854, #7852)
- Improve resource cleanup after termination
- Intelligent GPU name detection
- Improved volume support: label support, name validation, SDK support.
Volume Support for Existing PVC and Ephemeral Volumes (#7915, #7971,#8179)
-
Exising PVC: Reference pre-existing Kubernetes PersistentVolumeClaims as a SkyPilot volume (#7915).
# volume.yaml name: existing-pvc-name type: k8s-pvc infra: k8s/context1 use_existing: true config: namespace: namespace
-
Ephemeral Volumes: automatically create volumes when a cluster is launched and deleted when the cluster is torn down, making them ideal for temporary storage across multiple nodes, such as caches and intermediate results.
# task.sky.yaml file_mounts: /mnt/cache: size: 100Gi
Cloud Support
CoreWeave Integration Announcement
CoreWeave now officially support SkyPilot (#6386, #6519, #6895, #7756, #7838). This integration provides: Infiniband Support, Object Storage, Autoscaling.
See the announcement on Coreweave blog.
AMD GPU Support Announcement
SkyPilot now fully supports AMD GPUs on Kubernetes clusters (#6378, #6944). This includes: GPU Detection and Scheduling, Dashboard Metrics, ROCm Support.
See the announcement on AMD Rocm blog.
More Clouds Support
- Together AI Instant cluster
- Seeweb support
User Experience
SkyPilot Templates
SkyPilot now ships predefined YAML templates for launching clusters with popular frameworks and patterns. Templates are automatically available on all new SkyPilot clusters. (#7935, #7965)
You can now launch a multi-node Ray cluster by adding a single line to your YAML’s run block:
run: |
# One-line setup for a distributed Ray cluster
~/.sky/templates/ray/start_cluster
# Submit your job
python train.pyProgrammability: SkyPilot Python SDK
SkyPilot Python SDK is significantly improved with:
- Type hints
- Log streaming
logs = sky.tail_logs(cluster_name, job_id, follow=True, preload_content=False)
for line in logs:
if line is not None:
if 'needle in the haystack' in line:
print("found it!")
break
logs.close()- Admin policy: build admin policy with helper functions
resource_config = user_request.task.get_resource_config()
resource_config['use_spot'] = True
user_request.task.set_resources(resource_config)Integrating with CI/CD
With the improved SDK, we can integrate SkyPilot with GitHub Actions and other orchestrator to spin off your AI workloads automatically. (#7932)
Native Git Support
You can now use your private git repositories directly as your SkyPilot workdir (#6294, #6257, #6268). SkyPilot handles the cloning and syncing automatically. You can also use --git-url and --git-ref options with sky serve up (#8012).
# task.sky.yaml
workdir:
url: <https://github.com/my-org/my-repo.git>
ref: 1234ab # commit hash or branch nameYou can also find the commit hash of your workdir in the Dashboard:
Autostop/Autodown Based on SSH Sessions
In addition to running jobs on clusters, you can now configure autostop/autodown to wait for active SSH sessions (#6361, #6485).
# task.sky.yaml
resources:
autostop:
wait_for: jobs_and_sshExamples: Distributed LLM Pretraining/Finetuning/Serving
We released high-performance distributed training examples for large models with checkpointing support (#6525, #6551, #6242, #6273, #6443).
- Kimi-K2 multi-node serving (#7988)
- Torchtitan framework support
- Llama 4 Maverick 400B: Training on more than 16 H200 GPUs.
- OpenAI GPT-OSS: Model pretraining and finetuning.
- Agentic training: VeRL example for reinforcement learning.
What's new: compared to v0.10.5
API Server: Robustness & Security Improvement
- TLS support in Helm: Added TLS (HTTPS) support to the Kubernetes ingress configuration for SkyPilot's Helm charts (#6864, #8046).
- Support for air-gapped environment for SkyPilot API server deployment: use private container registries in the Helm chart (#7955, #7973,#8076).
- More robust SSO authentication for SSH connections: Server-side SSO and service account tokens are now properly enforced for WebSocket connections used by the SSH proxy system (#7863, #7861).
- Memory leak fix: Fixed resource cleanup in status refresh daemon when errors occur (#8100).
- Performance improvement with PostgreSQL Database connection pooling (#7829, #8005).
- Fast rolling update by avoiding PVC issue with grafana deployment (#7904)
- Better message during API server upgrade (#7921)
- API server metrics: CPU/memory (#7923), limit the Prometheus scraper to the current API server (#7922), executors (#7980)
- Avoid API server hanging due to event loop blocking (#8097, #8101)
- Fixed user info handlers (#8234)
- Backward compatibility: Fixed server/client version compatibility issues (#7873).
- Deprecation:
rbac.rules→rbac.namespaceRules;ingress.nodePortEnabled→ingress-ngnix.controller.service.type=NodePort(see details) helm chart
Kubernetes Improvements
- Make Pod status visible during cluster launching: Better experience with detailed pod event messages when pods are stuck pending (e.g., volume mount failures, image pull issues) (#7959, #8062).
- Post-provision commands support: Execute custom shell commands on Kubernetes pods during instance initialization with
post_provision_runcmd(#7943). - Rsync race condition fix: Fixed intermittent "rsync not found" errors during pod setup (#7844, #8023).
- GPU labeler fix for race condition on Kubernetes clusters with multiple API server and dependency (#7779, #8126, #8111).
- Fix allowing all contexts: ****Fixed feature support detection to evaluate all contexts in
allowed_contextsconfiguration (#7878).
Core & Backend
- Fixed
--retry-until-up: Fixed the-retry-until-upflag to properly retry failed launches across all cloud zones (#8079). - Fixed premature log termination for cluster provision log
sky logs --provisionduring cluster launches (#7888). - Add AWS AMI metadata caching to reduce AWS API calls and avoid credential issues (#7961, #7998, #8000, #8013, #7997).
- Robustify SkyPilot runtime setup (#7985, #7901)
- Speed up the storage check (#8044)
- Fixed Fluent-bit GPG key download for cloud logging setup (#7898).
- Fixed regression in service/pool cleanup process (#8047).
- Fixed backward compatibility for jobs queue API with older server versions (#8015).
- Faster SSH tunnel opening and cleanup fix for grpc connection: Optimized SSH tunnel opening under high concurrent load from 8+ seconds to ~3.7 seconds (#7900, #7887, #8105).
Infrastructure & Cloud Integrations
- Support GCP B200 spot instances (#8125).
- AWS Trainium & Inferentia support improvement: dynamically detects Trainium/Inferentia instances via the AWS API, maps them correctly for
sky show-gpus, fix the default Neuron-based AMI (#7896, #7958). - Fixed AWS g6f, gr6f instance fetching (#7949)
- AWS config file support: Added support for
AWS_CONFIG_FILEenvironment variable for custom credential paths (#8050). - Azure: Updated Azure provisioning to use Standard SKU public IP addresses after Azure retired Basic SKU and fix dependency, and fix credential checking (#8007, #7909, #7996, #7924).
- Samsung Cloud Platform (SCP) Multi-node Support (#7288).
- Seeweb: Added Docker image support for Seeweb cloud provider (#7493).
- RunPod: Added CPU-only instance support, fix dependency, and default image (#7460, #8018, #7906).
- Nebius disk cleanup: Fixed resource leak when VM creation fails due to quota errors (#8004).
SSH Node Pools
- Per-Context Configuration: Define different
pod_configandcustom_metadataspecifications for multiple SSH node pools throughcontext_configs(#7660, #7913). - Provision Timeout: New
provision_timeoutfield for controlling SSH node provisioning wait times (#7660). - Fix provisioning deadlock when using custom images (#7883).
- Fixed regression where SSH node pool contexts were incorrectly hardcoded (#8087).
User Experience
Admin Control Enhancements
- Additional information in admin policy: user information (#7800), request type (#7840. #8042).
- New examples for adming policy: rate limiter policy (#7902, #7903), static GPU quota (#7917)
- Allow workspace to enable specific capacity of a cloud (#7978)
- Secrets as SecretStr: Task secrets are now stored using Pydantic's
SecretStrto prevent accidental exposure in logs (#8040).
SDK Improvements
- Resource Configuration API: New
Task.get_resources_config()returns resources as a dictionary matching YAML format, andTask.set_resources()now accepts dictionaries—making programmatic resource manipulation as intuitive as editing YAML (#7857). - GPU Count in Setup: New
SKYPILOT_SETUP_NUM_GPUS_PER_NODEenvironment variable available during setup phase for configuring software based on GPU count (#7092).
Dashboard Improvements
- Infra page: keep infra page even when error occurs and more info for workspace (#8085, #7845, #7918, #8099)
- Dashboard now preserves cached data during transient server errors instead of clearing tables (#7831).
- Hover over resources column to view complete resource specifications (#7880).
- Added navigation between pool detail pages and jobs dashboard (#7856).
- Hide already-selected filter values from dropdown menus (#7884).
- GPU metrics panel: Added XID error monitoring panel to Grafana dashboard (#8041).
- GPU metrics fix: Fixed GPU metrics display on cluster detail page (#8025, #8045, #8059).
- Fixed region display format for jobs to match clusters (#7885).
- Fixed cluster / jobs filtering by users (#7929)
CLI Improvements
- Fixed sky show-gpus --all: Fixed JSON serialization error when listing all accelerators (#7865).
- Fixed sky cost-report: Fixed JSON serialization error with NumPy float values (#7974).
- Allow purging a cluster when CloudUserIdentityError (#7934)
- Fix duplicated log expansion for
sky serve logs(#8002) - More efficient log streaming (#7951)
- Include cluster name for the local log dir (#7942)
- Clearer feedback when updating pools and services (#8010).
- Deprecation: Removed
sky status --kubernetesfor the team SkyPilot API server (#7989), removedsky local up --ipsin favor ofsky ssh up(#8065)
Documentation & Examples
- Added PyTorch multi-node distributed training documentation (#7848).
- Added marimo notebook example (#8031).
- Improved GPT-OSS fine-tuning examples with checkpointing and WandB integration (#7696).
- Updated SDK examples for vector database and local GPT (#8003).
- Reorganized Pools documentation with better discoverability (#7897).
- Improved admin policy documentation with quickstart guide (#7866).
- Added server-side debug logging documentation (#8001).
Testing & CI/CD
- Added shared GKE API server with PostgreSQL testing to nightly builds (#7830).
- Added RunPod and Lambda cloud provider tests to nightly builds (#8051).
- Added SSO/Okta authentication test coverage (#7877).
- Improved test reliability and memory usage (#7834, #7841, #7864).
- Fixed release workflow for GitHub release notes generation (#7079).
Contributors
Thank you to all contributors who made this release possible!
@adocherty, @alex000kim, @aylei, @brianstrauch, @cblmemo, @cg505, @coopslarhette, @cwhitak3r, @DanielZhangQD, @hyoxt121, @kevinmingtarja, @kyuds, @lloyd-brown, @massaindustries, @Michaelvll, @rohansonecha, @romilbhardwaj, @SalikovAlex, @seongsukwon-moreh, @SeungjinYang, @zpoint
New Contributors:
- @aadyotb made their first contribution in #7898
- @BenedTj made their first contribution in #7884
- @chris-clem made their first contribution in #8125
- @coloryourlife made their first contribution in #8103
- @girishramnani made their first contribution in #8137
- @jilongliao made their first contribution in #6864
- @jusstinn made their first contribution in #8012
- @koaning made their first contribution in #8031
- @MalumaDev made their first contribution in #7715
- @mmcclean-aws made their first contribution in #7896
- @NavarrePratt made their first contribution in #8019
- @soovittt made their first contribution in #8086
Special thanks to the community for bug reports, feature requests, and pull requests that helped improve SkyPilot!
Full Changelog
For a complete list of changes, see the commit history.












