github skypilot-org/skypilot v0.14.3
Release v0.14.3

46 minutes ago

SkyPilot v0.14.3

Get it now with:

uv pip install "skypilot>=0.14.3"

What's Changed

  • [Docs] Job groups: rework 'Dynamically attaching jobs to a job group' for readability by @ishankaul1 in #10764
  • [Dashboard] Add plugin slots for the Infra cell and field by @SeungjinYang in #10797
  • [Docs] README: refresh the News section by @hopechong in #10795
  • [Jobs] include_tree on the managed jobs queue: one fetch for a job and its dynamic tasks by @ishankaul1 in #10767
  • [Kubernetes] Main containers being created after init containers finish are not a stall by @MadhavJivrajani in #10801
  • [Server] Cap the storage the API server keeps file mounts on by @aylei in #10793
  • [Core] Say when a queued launch was admitted by @DanielZhangQD in #10802
  • [Metrics] Count WebSocket handshake attempts by @cg505 in #10688
  • [Test] Stop shrinking test_api_server_start_stop below its declared size by @DanielZhangQD in #10805
  • [Server] Spawn ssh-proxy subprocesses without forking the API server by @aylei in #10804
  • [CI] Pin googleapis-common-protos<1.75.1 to match the protobuf 5.29.6 dev pin by @lloyd-brown in #10808
  • [Test] Match managed-job GCP instances on the name prefix that survives truncation by @lloyd-brown in #10809
  • [Dashboard] Prevent empty-state flashes on Jobs refresh and scope switches by @rohansonecha in #10810
  • [Slurm] Support executable sudo allowlists for user impersonation by @kevinmingtarja in #10811
  • [Server] Keep the metrics event loop free while the federation routes work by @DanielZhangQD in #10812
  • [Dashboard] Gray dash and tooltip for a job with no name by @ishankaul1 in #10816
  • [Slurm] Support reduced sudo permissions without container snapshots by @kevinmingtarja in #10820
  • [Jobs] Keep a task's submission and start times across a parked restart by @aylei in #10788
  • [Jobs] Finalize managed jobs atomically instead of in separate writes by @aylei in #10386
  • [Dashboard] Pass the job group's aggregate status to the detail status badge slot by @aylei in #10829
  • [Tests] Reset the SkyPilotContext ContextVar between unit tests by @aylei in #10830
  • [Jobs] Count the launch errors the controller swallows by @DanielZhangQD in #10831
  • [Jobs] Say why a managed job failed to submit, not just that it did by @DanielZhangQD in #10834
  • [Slurm] Keep the full text of scontrol node attributes that contain spaces by @rohansonecha in #10839
  • [Metrics] Federate LiteLLM proxy metrics by @rohansonecha in #10841
  • [Jobs] Don't construct an exception per unselected column when building job rows by @zpoint in #10832
  • [Slurm] Stop teardown from issuing redundant sudo rm on the login node by @aylei in #10844
  • [Jobs] Resume a pipeline at the pending task after a controller restart by @cg505 in #10078
  • [Docs] Link docs and README logos to skypilot.ai by @hopechong in #10847
  • [Core] Status refresh: do not mark a cluster UP when its handle has no cached IPs by @kevinmingtarja in #10140
  • [Core] Status refresh: keep the ray health check for stopped-then-restarted clusters by @zpoint in #10850
  • [Slurm] Preserve the latest cluster job's exit code on teardown by @kevinmingtarja in #10851
  • [Jobs] Report managed job tasks that stalled before they ever ran by @DanielZhangQD in #10824
  • [Kubernetes] Stop workload pods holding the provisioner's permissions by @DanielZhangQD in #10845
  • [Test] Stop the read-only workspace tests leaking IS_SKYPILOT_SERVER into their worker by @DanielZhangQD in #10854
  • [Azure] Support ACR authentication via the VM's managed identity by @ahluwalij in #10148
  • [Docs] Add analytics for a better user experience by @hopechong in #10857
  • [Test] Check TailLogs tests for leaked threads by identity, not count by @SeungjinYang in #10860
  • [Server] Accept padded JWT segments from external auth proxies (PyJWT 2.14) and warn when the header cannot be decoded by @cg505 in #10858
  • [Core] Make per-user SSH key bootstrap converge across processes sharing a DB by @cg505 in #10785
  • [k8s] Fail fast when a pod disappears while waiting for it to be scheduled by @cg505 in #10800
  • [Test] Make the local-disk path-resolution tests deterministic under xdist by @cg505 in #10864
  • [Docs] Point Try SkyPilot Platform button to /demo by @hopechong in #10865
  • [Test] Skip the fusermount-server fd leak test on remote servers by @zpoint in #10868
  • [Nebius] Filter grpc.aio poller errors from the asyncio logger by @cg505 in #10862
  • Support for Nebius Spot Pricing by @Jobarion in #10853
  • [Jobs] Support multiple runtimes and runtime-owned recovery by @kevinmingtarja in #10836
  • [Helm] Compute overview: include Slurm GPU allocation; show 0% when idle by @zpoint in #10867
  • [CLI] Add --output id|json to sky jobs launch by @aylei in #10879
  • [Jobs] Keep a per-task history of the scheduler allocations runtimes report by @aylei in #10873
  • [Jobs] Managed job dependencies by @aylei in #10876
  • [Examples] Add Halo training recipe by @advpropsys in #10835
  • [Deps] Make Vast opt-in: drop it from the [all] extra by @Michaelvll in #10895
  • [Dashboard] Pin shepherd.js to 13.0.3 by @Michaelvll in #10892
  • [Examples] Use the official InvokeAI image for the Stable Diffusion example by @Michaelvll in #10898
  • [Slurm] Expand Slurm hostlists in-tree by @Michaelvll in #10890
  • [Jobs] Support runtime log readers by managed job ID by @kevinmingtarja in #10884
  • [Slurm] Protect the requeue sbatch option by @rohansonecha in #10904
  • [Dashboard] Allow customizing list-page filter bars by @stephenkevn in #10875
  • [Examples] Drop the copied xformers patch from vicuna-llama-2; own fusermount usage text by @Michaelvll in #10894
  • [Test] Make the dynamic job group smoke tests pass on a shared basic-auth API server by @ishankaul1 in #10859
  • [Dashboard] Plugin slots on the workspace Allowed Users list by @lloyd-brown in #10902
  • [Core] Handle raw PermissionError in child process walk by @amitgelber in #10856
  • [CI Fix] Use a user venv in apiserver-start-stop setup on bare cloud VMs by @lloyd-brown in #10912
  • [Core] Create the default user Python env on VMs without conda by @lloyd-brown in #10919
  • [Core] Keep reading skylet tunnel output so port-forward never blocks by @kevinmingtarja in #10922
  • [CI Fix] Make test_cli_output border check robust; skip test_batch_diffusion on Nebius by @lloyd-brown in #10920
  • [CI Fix] Wait for pool shutdown in pool teardown smoke tests by @lloyd-brown in #10918
  • [Core] Keep skylet tunnel bookkeeping per API server host by @kevinmingtarja in #10923
  • [Nebius] Delete the cluster security group after autodown by @lloyd-brown in #10917
  • [Nebius] Wait for workers to terminate on worker-only teardown by @lloyd-brown in #10921
  • [API] Schedule the post-save sky check as its own request instead of blocking config save by @ishankaul1 in #10821
  • [Docs] Fix favicon rendering with black corners by @hopechong in #10951
  • [CI] Nightly: run the Kubernetes jobs-consolidation lane with gRPC enabled by @kevinmingtarja in #10933
  • [API] Validate client user ids used as filesystem paths by @DanielZhangQD in #10897
  • [Helm] Spread API server pods per ReplicaSet with matchLabelKeys by @aylei in #10938
  • [Jobs] Treat a just-ended launch as activity in the unattended stall scan by @aylei in #10941
  • [API] Fail a request whose result cannot be stored by @aylei in #10946
  • [Test] Fix flaky test_pool_autoscaling_scale_up_to_max_then_down_to_zero by @DanielZhangQD in #10957
  • [Server] Allow pinning executor pool and jobs controller counts by @aylei in #10916
  • [API] Export the API server container's memory and processes by type; reap gauges of reused pids by @aylei in #10943
  • [API] Contain task file-mount sources to the backend's user roots by @DanielZhangQD in #10899
  • [Tests] Allow retry-log streaming to attach before task failure by @kevinmingtarja in #10936
  • [API] Skip stored results when listing request status by @aylei in #10939
  • [Examples] Pin huggingface-hub<1.0 in the HuggingFace GLUE examples by @zpoint in #10971
  • [Jobs] Leave job file contents and task YAML out of queue reads that do not show them by @aylei in #10944
  • [Core] Restore sky.provision log propagation after each provision by @DanielZhangQD in #10968
  • [AWS] Share one botocore loader across sessions by @aylei in #10931
  • [Core] End a followed job log tail when its kubectl exec session is gone by @aylei in #10958
  • [Kubernetes] OCI network_tier=best: pick NCCL env per node shape by @DanielZhangQD in #10967
  • [DB] Select psycopg2 explicitly for sync Postgres engines by @cg505 in #10975
  • [API] Stamp the reason and verified account on rejected credentials by @DanielZhangQD in #10970
  • [CLI] Keep sky status -o json stdout parseable by @exdsgift in #10763
  • [Jobs] Reset each managed job at most once per HA recovery pass, via compare-and-swap by @cg505 in #10327
  • [API] Validate the client-supplied user id before it becomes a cluster owner by @Khas-Erdene-Tsogtsaikhan in #10437
  • [Test] Shrink test_job_group_rl_architecture to 0.5 CPU / 2 GB per pod by @zpoint in #10907
  • [Kubernetes] Assign hostNetwork Ray ports on the server and retire the ray-ports ConfigMap by @DanielZhangQD in #10852
  • [Test] Give the host-network get_cluster_info test a real provider_config by @DanielZhangQD in #10981
  • [Core] Build the cost report a chunk of history at a time by @aylei in #10949
  • [Core] Merge Python-computed Kubernetes pod fields after template render by @kevinmingtarja in #10973
  • [Jobs] Add jitter to the managed job status-check gap by @cg505 in #10913
  • [Test] Add Nebius GPU spot e2e smoke test by @cg505 in #10863
  • [Docs] Drop the "under active development" banner from the Slurm docs by @Michaelvll in #10955
  • [Jobs] Skip local file-mount blob when recovery sources are cloud URLs by @hentt30 in #10977
  • [Jobs] Scope shared bucket uploads by workspace by @hentt30 in #10959
  • [API] Fall back to the API server when an SSH redirect target fails by @DanielZhangQD in #10871
  • [Core] Add OTLP as an external logging store by @zpoint in #10908
  • [Kubernetes] Fix relaunching a hostNetwork cluster whose Ray stopped by @DanielZhangQD in #10986
  • [Jobs] Let managed jobs in non-default workspaces launch on a jobs controller VM by @zpoint in #10979
  • [Jobs] Make the consolidation-mode recovery sweep targeted and rate-limited by @aylei in #10557
  • [Kubernetes] Use the nvidia-rtx-pro-6000 GKE label for RTX PRO 6000 by @MohammadHijjawi97 in #10926
  • [Core] Redact credentials from debug dumps by @aylei in #10985
  • [Core] Return freed memory to the OS under jemalloc in release_memory by @aylei in #10942
  • [Core] Bound the log tail reader on logs with few newlines by @aylei in #10945
  • [RunPod] Migrate to REST API v2 (v1 retires Nov 15, 2026; GraphQL early 2027) by @ciprianf-flexion in #10846
  • [Test] Smoke-test hostNetwork relaunch after Ray stops, on the queue that can run it by @DanielZhangQD in #10987
  • [Core] Tidy comments and docstrings by @Michaelvll in #10989
  • [Jobs] Check network connectivity once per controller process per status gap by @cg505 in #10982
  • [Test] Trim the ray patches payload timestamp test docstring by @DanielZhangQD in #11000
  • [Core] Report provisioning errors on an existing cluster instead of asserting INIT by @DanielZhangQD in #11002
  • [Test] Pin the GKE test cluster's pod range outside 10.0.0.0/9 by @zpoint in #11007
  • [Kubernetes] Add remote_identity: NONE for workload pods with no API identity by @DanielZhangQD in #11001
  • [Jobs] Release a job group's launch slot once it starts running by @DanielZhangQD in #11026
  • [DB] Use the SQLite lock timeout for the sync engine, bounded on the auth path by @ciprianf-flexion in #11006
  • [Jobs] Reset the status-check window only while the job is alive by @DanielZhangQD in #11025
  • [API] Stop honoring the controller-name check bypass from requests by @DanielZhangQD in #11027
  • [API Server] Run the API server middlewares as pure ASGI by @aylei in #11029
  • [Tests] Pin both launches in test_managed_jobs_api_access to Kubernetes by @zpoint in #11031
  • [Core] Compute the Kubernetes node selector and spot toleration in Python by @kevinmingtarja in #10990
  • [Dashboard] Use k / TB suffixes for large CPU and memory counts by @claude[bot] in #11044
  • [Docker] Bump base image to Python 3.13 by @kevinmingtarja in #10105
  • [Core] Scope cluster history and cluster events to the caller's readable workspaces by @trevanth24 in #11024
  • [Jobs] Add job.wait_for_scheduling_timeout to cancel managed jobs that never start by @trevanth24 in #11004
  • [Jobs] Stagger job controller starts by @aylei in #10790
  • [API] Refuse controller-only launch fields from non-loopback callers by @DanielZhangQD in #11051
  • [Nebius] Reuse SDK clients and close retired instances by @SalikovAlex in #11036
  • [Core] Bound provisioning commands so a wedged transport cannot hang forever by @zpoint in #10781
  • [Jobs] Retry emergency bookkeeping through a database outage; stop when the job record is gone by @cg505 in #10978
  • [Dashboard] Plugin access check for opening a workspace's page by @lloyd-brown in #11045
  • [Recipes] Support recipes managed by an external source by @lloyd-brown in #11043
  • [Core] Retry the cluster health probe on all clouds by @nordmtr in #11016
  • [Docs] Move slime example from Agents to Training by @ishankaul1 in #11042
  • [Dashboard] Fit more job details above the fold on the job page by @stephenkevn in #10991
  • [Dashboard] Lay out the Infra, Clusters and Jobs headers for narrow screens by @stephenkevn in #11060
  • [Core] Ignore client overrides of workspace_config by @trevanth24 in #11048
  • [Users] Scope the user list and service-account tokens to private workspaces by @trevanth24 in #11023
  • [Core] Gate cluster log and job reads on the cluster's workspace by @trevanth24 in #11047
  • [Test] Isolate test_cluster_read_permission from leaked config env by @DanielZhangQD in #11070
  • [Core] Ignore client overrides of admin_policy and logs by @DanielZhangQD in #11069
  • [Jobs] Print a finished job's logs when the followed tail never saw it RUNNING by @zpoint in #11056
  • [Serve] Re-resolve the upload blob on every replica launch in consolidation mode by @cg505 in #11066
  • [SSH Node Pools] Deploy hosts that need a sudo password, without exposing it by @rohansonecha in #11067
  • [Docs] Put SSO in the auth doc title so it shows in the sidebar by @aylei in #11072
  • [Jobs] Do not read the runtime observation cursor from the DB on every poll by @cg505 in #10983
  • [DB] Treat asyncpg shutdown, resource and read-only errors as transient by @cg505 in #11057
  • [Server] Add managed job submission plugin hook by @kevinmingtarja in #10796
  • [Slurm] Pick the skylet keeper's node from squeue, like the provisioner by @ryxli in #11063
  • [Test] Set logs.store on the API server in the log store smoke tests by @zpoint in #11081
  • [Dashboard] Show the entrypoint and task YAML of cluster jobs by @zpoint in #11050
  • [SSH Node Pools] Generate a random k3s join token per pool by @DanielZhangQD in #11079
  • [API Server] Cut auth CPU for service-account Bearer requests by @aylei in #11074
  • [Tests] Skip log store smoke tests under --dependency by @zpoint in #11083
  • [Core] Stop rebuilding the cached wheel on every launch after a source mtime bump by @kevinmingtarja in #11082
  • [Core] Only include SSH credentials in status for clusters the caller owns by @kevinmingtarja in #11085
  • [Test] Raise the jobs-queue SLA in the scale test and fix false negatives in its output checks by @DanielZhangQD in #11091
  • [Test] Read the jobs controller name from the status table in test_multi_tenant_managed_jobs by @zpoint in #11092
  • [Test] Wait for the scale test's RDS instance to be deleted before removing its groups by @DanielZhangQD in #11096
  • [Jobs] Emergency recovery keeps a running job on its cluster by @cg505 in #10963
  • [API] Back off a log stream's status check while the request is parked by @cg505 in #11084
  • [Catalog] Read AcceleratorName as a string so a catalog with no GPU rows can be filtered by @ishankaul1 in #11101
  • [Verda] Keep the existing catalog when no GPU instance type is available by @ishankaul1 in #11102
  • [Jobs] Propagate scheduler subprocess exit code in consolidation mode by @cg505 in #9646
  • [Kubernetes] Map H100-MEGA to the nvidia-h100-mega-80gb GKE label by @MadhavJivrajani in #11104
  • [API] Drop unused FastAPI prefix argument for FastAPI 0.145+ by @vnavkal in #11116
  • [Core] Rename workspaces while they have active resources by @DanielZhangQD in #11055
  • [Test] Restore the config env var after admin policy tests by @DanielZhangQD in #11119
  • [Test] Pin fastapi<0.145 in the backward compatibility base env by @DanielZhangQD in #11120
  • [Core] Stop shipping another server's or checkout's cached SkyPilot wheel by @DanielZhangQD in #11121
  • [Dashboard] Make the viewer role selectable for users and service accounts by @kevinmingtarja in #11090
  • fix(slurm): treat CONFIGURING allocations as INIT in query_instances by @ryxli in #11115
  • fix(slurm): accept Slurm's full sbatch time grammar and check it at load by @ryxli in #11134
  • fix(slurm): remove a stale global-scope container before every launch by @ryxli in #11135
  • [Azure] Update A10 GRID driver for vGPU 18 by @ahluwalij in #10469
  • [Jobs] Launch a pool job again when its controller restarts before a worker is picked by @aylei in #11138
  • [Logs] Placement-aware logging agents and following running managed jobs from the log store by @aylei in #11076
  • [Core] Upgrade the bundled Ray to 2.56.1 by @DanielZhangQD in #10585
  • [Test] Give the ray-port provisioner test a region by @DanielZhangQD in #11146
  • [Images] Bake VM images from a given SkyPilot commit; switch AWS/GCP to -ray256 images by @DanielZhangQD in #11126
  • [Test] Treat the local-disk test roots as charged so they do not depend on the runner's /tmp mount by @ishankaul1 in #11151
  • [Jobs] Retry a finished job's storage teardown before failing it by @MadhavJivrajani in #11150
  • [Core] Let plugins register per-workspace config properties by @lloyd-brown in #11132
  • [Kubernetes] Share aws eks get-token results across kubectl processes on the API server by @aylei in #11155
  • [API Server] Resolve the active workspace for /validate by @trevanth24 in #11100
  • [Kubernetes] Let a task's remote_identity: NONE beat per-context config by @DanielZhangQD in #11137
  • [Kubernetes] Tell workers the head's Ray port when its GCS is not ready yet by @DanielZhangQD in #11161
  • [k8s] Check a cluster's owner by its recorded context and cluster UID by @aylei in #11127
  • [Jobs] Preserve any_of semantics when adding Kubernetes annotations by @dyld-w in #10490
  • [Core] Let sky api cancel <id> cancel another user's request for admins by @zpoint in #11093
  • [Dashboard] Let a plugin table column sort by a value it computes by @zpoint in #11144
  • [Jobs] Check workspace read access in /jobs/events by @DanielZhangQD in #11163
  • [CI] Fail docker build instead of overwriting an existing image tag by @zpoint in #11164
  • [Docker] Upgrade pip to clear critical certifi advisory in Trivy scan by @zpoint in #11167
  • [API] Drop client-supplied DB pool and timeline env vars before the request env overlay by @DanielZhangQD in #11166
  • [Core] Don't fail reads on a read-only workspace when its enabled-clouds cache is cold by @DanielZhangQD in #11162
  • [API] Ignore a client-supplied ssh_proxy_command override by @DanielZhangQD in #11168
  • [Jobs] Drain and bound the API server step-down on consolidation lock loss by @aylei in #11169
  • [Examples] Add flying-roadster-rl: RL post-training an LLM pilot by @Michaelvll in #11171
  • [Dashboard] Show a not-found page for unknown URLs by @stephenkevn in #11173
  • [Jobs] Fix completed-task logs while a job group is active by @kevinmingtarja in #10826
  • [Plugins] Add a hook that observes the effective request after admin policy by @zpoint in #11124
  • [k8s] Cap the connections each Kubernetes API client keeps open by @kevinmingtarja in #11136
  • [Test] Fix order-dependent flake in test_check_server_healthy_or_start_rechecks_status by @zpoint in #11175
  • [Serve] Keep the autoscaler version and replica ids across controller recovery by @leshanbog in #10960
  • [Kubernetes] Harden fusermount-server by @aylei in #11170
  • [API Server] Find clusters in newly created workspaces when proxying SSH by @kevinmingtarja in #11178
  • [Jobs] Show a pending managed job's submission time in the queue by @kevinmingtarja in #11177
  • [Core] Limit infra errors and probes to the caller's allowed contexts and clusters by @DanielZhangQD in #11195
  • [Jobs] Name the submission log by job id span so batch launches fit the file name limit by @DanielZhangQD in #11194
  • [Core] Scope infra-read endpoints to the caller's workspaces by @DanielZhangQD in #11160
  • [Test] Check only SkyPilot's own routes in the infra guard test by @DanielZhangQD in #11198
  • [Slurm] Drop login shell banners from command output and allow non-GPU gres by @eugeneteoh in #11054
  • [Jobs] Add a per-user limit on claimed jobs that are not running by @lloyd-brown in #11176
  • [EFA] Skip warm nodes running a different accelerator when sizing EFA by @trevanth24 in #10492
  • fix(api): bound SQLite request-retention cleanup by @vilmos-uagi in #11089
  • [Azure] Bake Ray 2.56.1 into the default images (-ray256 gallery definitions) by @DanielZhangQD in #11196
  • [Dashboard] Keep Duration next to Submitted in job details by @stephenkevn in #11205
  • [Jobs] Count jobs against the controller job cap in cost units by @cg505 in #11208
  • Fix AWS VPC failover across regions by @amitgelber in #11040
  • [Serve] Preserve installed TLS credentials across updates by @amitgelber in #11041
  • [Serve] Resume persisted replica teardown during recovery by @amitgelber in #10717
  • [Storage] Bound controller staging buckets and preserve upload errors by @amitgelber in #10901
  • [API Server] Add guide and tool to migrate an API server from SQLite to PostgreSQL by @amitgelber in #10988
  • [Buildkite] Stop the pipeline generator reading --serve as --serve-consolidation by @zpoint in #11221
  • [Jobs] Don't hold back recovered jobs with a live task under the per-user launch limit by @cg505 in #11219
  • [Core] Merge the default user blocklist into a custom rbac.roles.user by @DanielZhangQD in #11222
  • [AWS] Enable ap-south-2 and refetch stale zone mappings by @zpoint in #11207
  • [Test] Dashboard e2e tests against a seeded API server, with twists by @zpoint in #11174
  • [Dashboard] Let non-admin users create, rotate and delete their own service accounts by @lloyd-brown in #11235
  • [Jobs] Make copying task logs into the controller log optional by @samuelv8 in #11212
  • [Test] Dashboard e2e: fix flakes on busy runners (warm-up, server sizing, teardown, waits) by @zpoint in #11229
  • [Core] Retry a stale or briefly missing request config file by @cg505 in #11224
  • [Jobs] Restore a job's config file only when it is missing by @cg505 in #11234
  • [Docs] Update SkyPilot Platform links and move Sandboxes under Jobs by @romilbhardwaj in #11236
  • [CI] Pause the monthly auto-trigger for the release pipeline by @zpoint in #11240
  • [Nebius] Readable, fail-fast sky check; progress-based launch timeout; SA setup docs by @Michaelvll in #11250
  • [Kubernetes] Parse the kubeconfig again only when it changes by @DanielZhangQD in #11232
  • [Core] Skip config YAML dumps that no handler would log by @DanielZhangQD in #11231
  • [Kubernetes] Keep Kubernetes clients across a sync handler's config refresh while the kubeconfig is unchanged by @DanielZhangQD in #11237
  • [Kubernetes] Keep recording a deleted pod's events past same-second repeats by @DanielZhangQD in #11241
  • [Test] Purge failed pools in pool smoke test teardown by @aylei in #11255
  • [Core] Record why a status refresh removed a cluster, and show it in a Details column by @DanielZhangQD in #11239
  • [Core] Show what ended a cluster as its last event, not a later status refresh by @DanielZhangQD in #11253
  • [Kubernetes] Retry a launch when its Kubernetes context is unreachable by @aylei in #11254
  • [Kubernetes] Name the node and its state in abnormal-cluster events by @DanielZhangQD in #11252
  • [CLI] Support --all for sky serve status by @lgqfhwy in #11256
  • [Docs] Remove seed announcement banner and refresh README news by @hopechong in #11259
  • [Dashboard] Show Infra with zone for terminated clusters by @bufocoder[bot] in #11260
  • [Kubernetes] Retry pod creation while an admission webhook is unreachable, then report it by @kevinmingtarja in #11080
  • [Test] Fail get_available_gpus when the infra is reported not enabled by @kevinmingtarja in #11263
  • [Core] Remove ExtensionContext.register_rbac_rule, which is never enforced by @DanielZhangQD in #11265
  • [Core] Make advisory and lease leader election exclude each other by @aylei in #10956
  • [Core] Fix YAML loader fallback with LazyImport by @cg505 in #11262
  • [Core] Floor ibm-vpc to wheel releases and sort the [all] extra by @zpoint in #11275
  • [API] Let service accounts act on behalf of users by @aylei in #11268
  • [Test] Deflake test_a_second_principal_does_not_reload_immediately by @MadhavJivrajani in #11278
  • [Tests] Skip the webhook-unavailable smoke test against a remote API server by @rohansonecha in #11279
  • [Server][Jobs] Do not start a local API server from inside the API server by @cg505 in #10571
  • [CI] Notify downstream only after the nightly Docker image is pushed by @zpoint in #11276
  • [Test] Dashboard e2e: bound wait_for_response; retry reading the config mid-rewrite by @zpoint in #11267
  • [Storage] Support .skyignore negation (!) patterns by @Hawon-Oh in #9833
  • [Core] Install the logging agent without sudo or apt, and keep the launch when it fails by @aylei in #11284
  • [Kubernetes] Fail a launch whose pod spec the API server rejects as invalid by @cg505 in #11283
  • [Dashboard] Keep unreachable and loading contexts from slowing the infra table by @rohansonecha in #11287
  • [Dashboard] Show GKE TPU accelerators by generation by @rohansonecha in #11288
  • [Jobs] Treat pool checkout timeouts as transient bookkeeping errors by @amitgelber in #11039
  • [Core] Isolate PostgreSQL advisory locks from the state pool by @amitgelber in #11030
  • [API Server] Let plugins run selected requests in the API server process by @aylei in #11291
  • [Kubernetes] Support KAI Scheduler by @ryxli in #11053
  • [Core] Remove the per-process advisory lock cap by @cg505 in #11307
  • [Kubernetes] Set TPU_NAME on GKE TPU pods by @DanielZhangQD in #11299
  • [Core] Keep node_names and other list fields in refreshed cluster records by @DanielZhangQD in #11300
  • [API Server] Skip the users upsert for authenticated requests; re-read service-account tokens in the background by @aylei in #11292
  • [Kubernetes] Parse the kubeconfig again only when it changes by @DanielZhangQD in #11286
  • [Core] Reload the RBAC policy only when it changed by @DanielZhangQD in #11295
  • [Kubernetes] Read realtime GPU availability from a node snapshot provider by @aylei in #11315
  • [Kubernetes] Fail fast on contexts known to be unreachable by @aylei in #11316
  • [API Server] Let plugins choose how a scheduled request executes by @aylei in #11318
  • [Kubernetes] Move the Kubernetes API error classifiers into one module by @cg505 in #11304
  • [Docs] Show the README logo that matches the GitHub theme by @stephenkevn in #11309
  • [Kubernetes] Fix yapf formatting in kubernetes/utils.py by @cg505 in #11325
  • [Jobs] Re-attach to a live launch request after a controller restart (continue the launch beside the initial launch) by @cg505 in #11280
  • [API Server] Keep the shared log handler when /validate applies the client's env by @cg505 in #11327
  • [Core] Reload the server config only when its source changed by @DanielZhangQD in #11297
  • [Core] Write multi-row RBAC policy changes in one transaction by @DanielZhangQD in #11317
  • [Core] Check workspace membership without the 1 h cache by @DanielZhangQD in #11319
  • [Test] Stop test_backend_utils from leaking SKYPILOT_CONFIG by @DanielZhangQD in #11334
  • [Server] Hide SSH key paths from non-admin SSH Node Pool reads by @DanielZhangQD in #11333
  • [Kubernetes] Keep the requested chip count for GKE TPU label names by @rohansonecha in #11331
  • [Core] Size parallel threads by the container's CPU limit by @samuelv8 in #11301
  • [Kubernetes] Do not treat GKE TPU requests as GCP TPU VMs by @rohansonecha in #11338
  • [API Server] Release executor memory only after RSS grows by a threshold by @aylei in #11335
  • [Test] Fix flaky test_follow_tail_exits_when_stdin_closes by @zpoint in #11294
  • [Jobs] Find the cluster without a workspace filter before downloading logs by @lloyd-brown in #11339
  • [Kubernetes] Multi-host TPU slices on GKE by @rohansonecha in #11337
  • [Docs] Add fault-tolerant PyTorch training example with torchft by @kevinmingtarja in #11344
  • [Jobs] Check workspace read access for managed job logs by @DanielZhangQD in #11345
  • [Kubernetes] Let sky check re-check only some contexts by @aylei in #11354
  • [API Server] Build /api/status listings off the event loop by @aylei in #11348
  • [Kubernetes] Return an empty list from realtime GPU availability when no GPUs are found by @aylei in #11358
  • [API] Keep worker threads warm instead of starting a thread per task by @aylei in #11351
  • Release 0.14.3 by @lloyd-brown in #11399

New Contributors

Full Changelog: v0.14.2...v0.14.3

Don't miss a new skypilot release

NewReleases is sending notifications on new releases.