skypilot-org/skypilot v0.14.0rc1

Release v0.14.0rc1

on GitHub

SkyPilot v0.14.0rc1

Get it now with:

uv pip install "skypilot>=0.14.0rc1"

What's Changed

• [Kubernetes] Display actual pod resource requests when they diverge from the resource spec by @lloyd-brown in #10214
• [Jobs] Reset transient job-status-check window after recovery by @spiffaz in #10156
• [Dashboard] Restyle the YAML editor's search panel by @stephenkevn in #10208
• [Dashboard] Fix top-bar logo vertical alignment by @stephenkevn in #10207
• [Jobs] Fail jobs terminally when their pool no longer exists by @lloyd-brown in #10203
• [Kubernetes] Fail provisioning on persistent volume mount failures instead of hanging by @lloyd-brown in #10218
• [Test] okta smoke: don't gate login success on the in-app dashboard redirect by @zpoint in #10188
• [Dashboard] Make the settings config editor card plugin-replaceable by @ishankaul1 in #10158
• [Dashboard] Use dark circular favicon for better tab visibility by @hopechong in #10215
• [Dashboard] Rework infra page: unified table, per-GPU-type summary, responsive by @stephenkevn in #10067
• [Test] Cover more documented env vars in managed jobs env smoke test by @kevinmingtarja in #10224
• [Kubernetes] Don't install conda by default and remove it from K8s images by @DanielZhangQD in #10132
• [Core] Handle missing Git during commit metadata collection by @edmsimplicio in #10230
• [RunPod] Label account SSH keys with the SkyPilot user by @ciprianf in #9975
• [Core] debug-dump: bound controller SSH work + add best-effort overall deadline by @claude[bot] in #10161
• [Nebius] Keep personal pricing fetch failures out of CLI output by @lloyd-brown in #10222
• [Test] Add golden snapshot tests for kubernetes-ray.yml.j2 rendering by @kevinmingtarja in #10226
• [Test] Poll for STOPPED after sky stop in test_resize by @lloyd-brown in #10217
• [Test] Pin zone in skyserve auto_restart.yaml so replica termination finds the instance by @lloyd-brown in #10216
• [Kubernetes] Surface which node a terminated cluster lost by @SeungjinYang in #10233
• [Docs] Trim redundant optimization section from Overview; link SkyPilot Platform by @concretevitamin in #10235
• [Docs] Update logo to official brand assets by @stephenkevn in #10238
• [Test] Add smoke test for k8s accelerator node-affinity scheduling constraint by @kevinmingtarja in #10225
• [Test] Add managed jobs smoke test for pod_config targeting the ray-node container by @kevinmingtarja in #10223
• [Dashboard] Use new brand favicon by @stephenkevn in #10237
• [Metrics] Stop cluster-label stamping from stalling the metrics server by @rohansonecha in #10240
• [Test] Fix apiserver-start-stop smoke test setup on bare cloud VMs by @zpoint in #10242
• [Test] Drop conda-version guard from test_custom_default_conda_env by @DanielZhangQD in #10244
• [Core] Guard source ~/.bashrc against a missing ~/.bashrc by @DanielZhangQD in #10245
• [Jobs] Retry transient cluster-status errors in controller monitor loop by @kevinmingtarja in #10040
• [Core] Restore the config override in replace_skypilot_config when the body raises by @kevinmingtarja in #10236
• [LLM] slime example: pin image to known-good nightly tag by @ishankaul1 in #10249
• [Test] Pin failed-mount escalation smoke test to a single Kubernetes context by @MadhavJivrajani in #10248
• [Core] PostgresLock: run advisory-lock statements in autocommit mode by @MadhavJivrajani in #10251
• [Server] Park launch as WAITING on cluster lock contention instead of blocking a worker by @aylei in #10210
• [Core] Surface resolved active_workspace to admin policies by @DanielZhangQD in #10241
• [API] Support running multiple isolated local API servers on one machine by @lloyd-brown in #10213
• [Core] Compute Kubernetes accelerator node affinity in Python by @kevinmingtarja in #10232
• [Nebius] Wait for instance termination before deleting the cluster security group by @lloyd-brown in #10220
• [Jobs] Collapse repeated job-status polls in the controller log by @SeungjinYang in #10255
• [k8s] Do not count queue-gated (Kueue admission) time against provision_timeout by @MadhavJivrajani in #10252
• [Server] Fail /api/get definitively for requests interrupted by a server restart by @aylei in #10257
• [Launcher] Fix multi-node rank crash on multi-NIC / hostNetwork K8s n… by @yash170106065 in #9924
• Update Together AI references in installation guide by @pavneeta in #10260
• [Jobs] Include exit code and log pointer in RECOVERING event for user-job failure restarts by @lloyd-brown in #10256
• [Dashboard] Infra table: unify data cell color across columns by @stephenkevn in #10267
• Update docstr for resolved_config by @DanielZhangQD in #10270
• [GCP] Support federated credentials for GCS COPY by @samuelv8 in #10212
• [Nebius] Pin nebius SDK below 0.4 by @zpoint in #10280
• [Server] Validate and parameterize request-status queries by @DanielZhangQD in #10273
• [Kubernetes] Bound pod wait-loop polls with a request timeout by @aylei in #10272
• [LLM] slime example: offload optimizer state to host RAM by default by @ishankaul1 in #10269
• [Slurm] Fix node info aggregation for multi-cluster setups by @liuxu623 in #10284
• [Core] Clamp open-files (nofile) limit raise to the attainable hard limit by @claude[bot] in #10263
• [Server] Park jobs-controller launches on cluster lock contention too by @lloyd-brown in #10291
• [API] Export metrics for the on-demand request thread executor by @kevinmingtarja in #10287
• [Slurm] Enrich node info with CPU/memory allocation and sampled usage by @rohansonecha in #10293
• [Jobs] Release log-tail threads when the client disconnects by @kevinmingtarja in #10290
• [Dashboard] Persist rows-per-page for the cluster jobs and pools tables by @zpoint in #10279
• [Test] Fix test_minimal job numbering on Slurm by @kevinmingtarja in #10298
• [Server] Return the executor slot when the monitor thread fails to start by @DanielZhangQD in #10300
• [Server] Make debug dump creation admin-only by @DanielZhangQD in #10277
• [Test] Pin stale-pod-cleanup kubectl to the cluster's landed context by @zpoint in #10302
• [k8s] Keep the spinner updating while pods await queue admission by @zpoint in #10305
• [Analytics] Add analytics to gain insights to usage by @hopechong in #10266
• [Slurm] Fix leaked allocation when terminating a cluster with a running task by @kevinmingtarja in #10296
• [Slurm] Unset step-scoped SLURM_* env before running user scripts by @kevinmingtarja in #10309
• [Slurm] Submit jobs as authenticated users by @kevinmingtarja in #10294
• [Slurm] Add SlurmClient.get_all_jobs_info() for per-node job attribution by @rohansonecha in #10306
• [Dashboard] Break Slurm clusters down by partition on the Infra page by @kevinmingtarja in #10317
• [Dashboard] Show CPU-only Slurm clusters on the Infra page by @aylei in #10318
• [Slurm] Use node AllocTRES for GPU availability accounting by @liuxu623 in #10285
• [API] Give auth its own thread executor so streams cannot starve it by @kevinmingtarja in #10313
• [Nebius] Unpin nebius SDK now that 0.4.3 is loop-aware by @zpoint in #10320
• [Slurm] Stop the multi-node run-done barrier from failing completed jobs by @kevinmingtarja in #10321
• [Test] Stop the auth-pool test from spawning 128 threads by @kevinmingtarja in #10322
• [Core] Optionally route the state DB through a transaction pooler by @aylei in #10301
• [Dashboard] Add filtering to the volumes page by @stephenkevn in #10316
• [Slurm] Make multi-cluster node info aggregation actually best-effort by @kevinmingtarja in #10289
• [Slurm] Keep unreachable Slurm clusters listed on the Infra page by @kevinmingtarja in #10329
• [Jobs] Fail deterministic task-prep errors fast instead of retrying them by @DanielZhangQD in #10326
• [Server] Scope request-listing body to the request owner by @DanielZhangQD in #10278
• Fix/k8s bootstrap package manager agnostic by @trevanth24 in #10077
• [Client] Fix sky api logs traceback on 4xx from /api/stream by @DanielZhangQD in #10341
• [Jobs] Fix managed job crash on empty imagePullSecrets in task config by @DanielZhangQD in #10325
• [Core] Read-only visibility of private workspaces for non-members by @DanielZhangQD in #10228
• [Test] Classify /slurm_cluster_names as read in workspace-access guard by @DanielZhangQD in #10343
• [Core] Add plugin hook for task-overrideable config keys by @kevinmingtarja in #10339
• [Test] Stop smoke-test teardowns from skipping cleanup on failure by @DanielZhangQD in #10342
• [Jobs] Add inter_connection knob for JobGroups: require or skip in-group networking by @ishankaul1 in #10297
• [Jobs] Review fixes for inter_connection: networking hardening, terminal errors, admin-policy header preservation by @ishankaul1 in #10330
• [Server] Scope request-tracking endpoints to the calling user by @DanielZhangQD in #10274
• [API] Measure /api/get latency broken out by request name by @aylei in #10349
• [Nebius] Add SkyPilot user-agent prefix by @complynx in #10345
• [Core] Add lease-based leader election alternative to advisory locks by @aylei in #10347
• [Test] Remove the shared GKE API server nightly lane by @zpoint in #10358
• [Examples] Serve CIFAR-10 from a SkyPilot-maintained GCS mirror by @cg505 in #10261
• [Test] Pin only object stores whose cloud is enabled on the API server by @zpoint in #10361
• [Jobs] Don't re-create user buckets in post-job storage cleanup by @aylei in #10353
• [Test] Pin enable_docker off in the non-Debian image smoke test by @lloyd-brown in #10366
• [Helm] History dashboard: bound GPU utilization to 0-100 (fix >100%) by @rohansonecha in #10368
• fix(schemas): accept fractional GPU counts in ManagedJobRecord by @Anai-Guo in #10356
• [Core] Lease leader election: heartbeat on a dedicated unpooled connection by @aylei in #10372
• [API] Bound auth middleware DB lookups with a deadline by @aylei in #10373
• [Test] Make sky status -o json output tests hermetic by @ishankaul1 in #10106
• [Core] Add equality for the Cloud class; fix JobGroup common-infra selection by @ishankaul1 in #10381
• [CI] Don't auto-retry exclusive smoke steps by @aylei in #10385
• [Server] Keep /metrics responsive when a collector's data source hangs by @aylei in #10377
• [Jobs] Add a managed-job addressed external log read-back hook by @aylei in #10376
• [Test] Pin provision_timeout in test_launching_with_pending_pods by @SeungjinYang in #10389
• [k8s] Gracefully skip Docker CLI install on non-Debian images when enable_docker: ALL by @lloyd-brown in #10382
• [API] Throttle per-request service-account last_used_at writes by @MadhavJivrajani in #10391
• [k8s] Report node GPU total from capacity, with allocatable alongside by @rohansonecha in #10395
• [Test] Resolve managed jobs by job id in smoke tests, not by reused names by @rohansonecha in #10396
• [API] Redact basic-auth password when printing the API server URL by @Khas-Erdene-Tsogtsaikhan in #10286
• [Test] Fix cloud-cmd exclusion in test_services_on_kubernetes by @zpoint in #10401
• [Core] Remove queue name keys from config instead of setting them to None by @aylei in #10134
• [Jobs] Add index on job_info.schedule_state for scheduler queries by @aylei in #10259
• [Jobs] Pass task_name to read_managed_job_logs and document id scope by @aylei in #10388
• [Kubernetes] Add scope (personal/workspace/global) to auto_mounts by @lloyd-brown in #10411
• [Server] Export SQLite database on-disk size metrics by @aylei in #10363
• [Kubernetes] Size the inline-command limit against the exec request URL by @DanielZhangQD in #10403
• [Kubernetes] Fail early when an auto-mounted volume is not ready by @DanielZhangQD in #10402
• [Test] Fix container-logs smoke tests: bare kubectl misses non-default namespace by @rohansonecha in #10420
• [Slurm] Batch inventory queries over one SSH connection by @kevinmingtarja in #10425
• [Slurm] Fail fast when a task exits nonzero by @kevinmingtarja in #10428
• [Docs] Update README news section by @hopechong in #10429
• [Test] container-logs: don't require cluster-scoped pod list by @zpoint in #10433
• [Kubernetes] Let a new volume settle before recording it as not ready by @zpoint in #10443
• fix(slurm): surface invalid GPU partition mappings by @ryxli in #10436
• [Jobs] Fix partial state writes on Job Group Phase-2 sync failure; parallelize group teardown by @ishankaul1 in #10415
• [Jobs] Emergency recovery: skip per-task surgery for Job Groups by @ishankaul1 in #10416
• [Test] Give optimizer dryrun launches unique cluster names by @ishankaul1 in #10448
• Revert "[Kubernetes] Let a new volume settle before recording it as not ready" by @DanielZhangQD in #10451
• [Logs][Test] Bound apt stalls in the fluent-bit agent install, and raise log-store smoke-test timeouts by @zpoint in #10362
• [Dashboard] Drop unknown filter properties from the URL instead of crashing by @zpoint in #10441
• [Core] Let volume_refresh refresh a named subset of volumes by @DanielZhangQD in #10452
• feat(slurm): support host path volume mounts by @ryxli in #10438
• fix(slurm): respect custom task CPU demands by @ryxli in #10447
• [Jobs] Keep the local log copy when external delivery is unconfirmed by @aylei in #10410
• [Docs] Revamp Sky Batch page visuals by @Michaelvll in #10453
• [Test] Make volume smoke tests behave against a shared remote API server by @rohansonecha in #10461
• [Core] Force sslmode=disable in the DB pooler hostport rewrite by @cg505 in #10463
• [API] Process client task YAMLs in memory by @kevinmingtarja in #10435
• [Test] Wait for a created volume to be mountable before launching against it by @DanielZhangQD in #10467
• [Test] Make cancel-pytorch smoke test work on H100 by @aylei in #10418
• [API] Sweep the clients dir off the event loop, and GC legacy task YAMLs by @aylei in #10473
• [Kubernetes] Bound the wait for pods to run, and say why a pod went missing by @SeungjinYang in #10460
• fix(k8s): ignore stale Kueue ErrWorkloadCompose in the pending-reason scan (#10379) by @Anai-Guo in #10380
• [Slurm] Impersonate submit users via sudo -u instead of sudo su by @claude[bot] in #10477
• [Dashboard] Paginate the node table on the infra context detail page by @rohansonecha in #10480
• users: block non-admin from granting admin role via service account u… by @intagliated in #9932
• [API] Validate the role a service account may be granted by @DanielZhangQD in #10482
• [Kubernetes] Detect a volume that will not bind while waiting for pods by @DanielZhangQD in #10440
• [Jobs] Bound the job-status retry window by retries as well as elapsed time by @aylei in #10499
• [Kubernetes] Let the k8s images be rebuilt on a different base by @DanielZhangQD in #10500
• [Test] Pin the auto-mount launches to the context their volumes live on by @DanielZhangQD in #10470
• [Dashboard] Show why a volume is not ready by @DanielZhangQD in #10486
• [Dashboard] Name the clusters filter params in the URL by @zpoint in #10455
• [Server] Serve /metrics on its own event loop by @aylei in #10503
• [API] Reload the policy under the lock before mutating it by @DanielZhangQD in #10488
• [Dashboard] Name the managed-jobs filter params in the URL by @zpoint in #10459
• [Dashboard] Name the filter params on users, volumes, recipes and infra by @zpoint in #10505
• [Dashboard] Run the dashboard tests in CI by @zpoint in #10510
• [Logs] Fall back to the canonical archive when the distro mirror is down by @zpoint in #10512
• [CLI] Allow sky api login when SKYPILOT_API_SERVER_ENDPOINT is set by @zpoint in #10404
• [LLM] slime example: pin swebench/swesmith deps; declare inter_connection in Job Group YAMLs by @ishankaul1 in #10498
• [Core] Validate the volume where it is created, not only in /volumes/validate by @DanielZhangQD in #10513
• [API] Deny a principal whose role the policy system does not recognize by @DanielZhangQD in #10501
• [CI] Notify the downstream nightly on no-op cycles too by @zpoint in #10529
• [Server] Attribute event loop stalls to the code that caused them by @aylei in #10523
• [API] Tell the user why a parked request is still waiting by @DanielZhangQD in #10534
• [Test] Fix the inline-env queue assertion matching a log line instead of the queue row by @DanielZhangQD in #10539
• fix(slurm): run skylet under an allocation-owned keeper step by @ryxli in #10516
• [Volumes] Filter the volume listing to the caller's accessible workspaces by @trevanth24 in #10531
• [Core] Redact credentials from SkyPilot config debug logs by @zpoint in #10511
• [Metrics] Client-side database instrumentation for the API server by @aylei in #10525
• [Kubernetes] Let pod_config opt out of OCI RoCE host networking by @DanielZhangQD in #10546
• [Volume] Refresh a volume's size from the cloud, not only at creation by @DanielZhangQD in #10549
• [Server] Account server and job controller process memory when sizing the executor pools by @aylei in #10550
• [Test] Give the Kubernetes volume smoke tests run-unique resource names by @DanielZhangQD in #10565
• [Server] Never regenerate the JWT signing secret on a DB read failure by @DanielZhangQD in #10566
• [Docs] Document --num-jobs without a pool by @Michaelvll in #10552
• [Server] Release the GIL during the hourly sky_logs prune (use os.stat over DirEntry.stat) by @kevinmingtarja in #10568
• [Metrics][Dashboard] Slurm: federate GPU metrics via the login node, and up-node/power-saved infra handling by @rohansonecha in #10572
• [Volume] Show a resize that has not landed yet by @DanielZhangQD in #10553
• [API server] Run the requests GC daemon once an hour by @aylei in #10555
• [Volumes] Make a volume's readiness cheap to ask about by @DanielZhangQD in #10564
• [Server] Cache the multiprocess metrics merge across scrapes by @aylei in #10573
• [Server] Give the retention daemons a fixed cadence and a startup jitter by @zpoint in #10538
• [Volumes] Read only the names when scoping the listing refresh by @DanielZhangQD in #10578
• [Dashboard] Let a jobs-table row link to its own detail page by @rohansonecha in #10580
• [k8s] Recommend set_pod_resource_limits when an OOM'd pod had no memory limit by @zpoint in #10551
• [API] Stop rewriting the whole policy table on every casbin write by @DanielZhangQD in #10577
• [Docs] Document -b/-s on the k8s image build script by @DanielZhangQD in #10583
• [Server] Move the chunked-upload rename and part scan off the event loop by @aylei in #10556
• Switch Nebius catalog fetcher to billing v1 by @klenovda in #9829
• [Dashboard] Complete external-jobs rendering in the managed-jobs table by @rohansonecha in #10591
• [Kubernetes] Resolve a volume's default context through allowed_contexts by @SeungjinYang in #10592
• [Test] Don't lose a smoke test's real failure to an unbound proc by @SeungjinYang in #10593
• [Test] task_labels_kubernetes: scope the label check to the cluster's own pod by @SeungjinYang in #10597
• [Core] Add register_slurm_property for plugin schema extension by @rohansonecha in #10600
• [Catalog] Skip Nebius presets with unresolved billing SKUs by @Michaelvll in #10400
• [Metrics] Slurm: scope and route /federate for a shared Prometheus (filter, via) by @Michaelvll in #10603
• [Kubernetes] Add kubernetes.rdma.mode for SR-IOV RDMA delivery by @DanielZhangQD in #10554
• [Core] Don't abort graceful teardown when command runners are unavailable by @claude[bot] in #10604
• [Slurm] Run credential check as the SSH user; keep infra partition toggle collapsible by @rohansonecha in #10610
• [Dashboard] Rename "Requested Resources" column to "Resources" on managed jobs page by @Michaelvll in #10609
• [Nebius] Support InfiniBand on Blackwell (B200/B300) platforms by @himanshu1573 in #10369
• [Nebius] Add optional boot disk encryption by @typesafeainate in #10172
• [k8s] Apply Ray patches from the pod spec, not from the installed skypilot by @DanielZhangQD in #10582
• [Server] Rebuild the state DB engine once the connection budget is known by @aylei in #10611
• [API] Run async continue-condition waits on a shared event loop by @aylei in #10584
• [Core] Record who requested a cluster stop/termination in its event log by @SeungjinYang in #10621
• [Docs] Add resilient Ray training example by @kevinmingtarja in #10602
• [Jobs] Record who requested a job cancellation in the event log by @SeungjinYang in #10619
• [Dashboard] Add jobs.detail.gpu-metrics plugin slot by @rohansonecha in #10620
• [Client] sky api login: follow an HTTP->HTTPS redirect by @SeungjinYang in #10624
• [Kubernetes] Reuse ephemeral volume pod injection by @kevinmingtarja in #10622
• [Dashboard] Give a log stream's last line the same treatment as the rest, and bound the partial-line carry by @DanielZhangQD in #10615
• [Agent] Add Slurm migration reference to the SkyPilot skill by @Michaelvll in #10541
• [Jobs] Scope managed-job log reads to the caller's workspaces by @DanielZhangQD in #10614
• [Server] Mark request FAILED when enqueue fails instead of leaving it PENDING forever by @aylei in #10617
• [Dashboard] Slurm Infra: per-cluster progressive loading, node-derived GPU columns, bounded inventory SSH by @rohansonecha in #10629
• [Dashboard] Plugin-registered table filter properties by @ishankaul1 in #10618
• [Slurm] Apply workspace config to sbatch options by @kevinmingtarja in #10631
• [Docs] Shrink the header search button on mobile by @stephenkevn in #10632
• Revert "[Server] Rebuild the state DB engine once the connection budget is known" by @SeungjinYang in #10633
• [Examples] Add Hugging Face storage batch transcription example by @davanstrien in #10558
• [Slurm] Support stopping/starting container clusters using fs snapshots by @kevinmingtarja in #10544
• Revert "[Jobs] Scope managed-job log reads to the caller's workspaces (#10614)" by @zpoint in #10637
• Run config reload off the background event loop in cleanup_sky_logs by @kevinmingtarja in #10569
• [API] Add provisioning/pending/start-time metrics and availability dashboard panels by @lloyd-brown in #10625
• [Dashboard] Build the app tree once, after plugin registration settles by @SeungjinYang in #10643
• [Docs] Add GB300 NVL72 Ray recovery example by @kevinmingtarja in #10644
• [Dashboard] Let plugin routes declare the nav link they belong under by @ishankaul1 in #10646
• [Dashboard] Show Started and Partition on the managed job detail page by @ishankaul1 in #10647
• [Slurm] Support autostop for container clusters by @kevinmingtarja in #10645
• [Server] Add a second plugin install pass for innermost middleware by @DanielZhangQD in #10648
• Lift the typing_extensions cap by bumping selenium; drop pydantic<2.12 by @zpoint in #10649
• [k8s] Distinguish the context a cluster is submitted to from the one it runs on by @SeungjinYang in #10651
• [Slurm] Fix container-cluster stop/start on NFS-backed clusters by @kevinmingtarja in #10652
• [AWS] Don't fail the launch when volumes can't be tagged by @zpoint in #10627
• [Core] Don't serialize an absent config path as the JSON string 'null' by @Anai-Guo in #10532
• [Slurm] Retry transiently missing jobs during status refresh by @liuxu623 in #10458
• [Server] Delete a log tail's request log once its stream ends by @aylei in #10657
• [Jobs] Filter managed jobs by infra, server-side by @zpoint in #10613
• [Jobs] Run sync state DB reads off the controller event loop by @aylei in #10654
• Stop a payload-shaped task log line from truncating the stream by @DanielZhangQD in #10670
• [SERVE] Load balancing least_load policy replica decrement bug and rotation issue by @JGSweets in #10445
• [Server] Keep the hook-logs request log that its client reads back by @zpoint in #10672
• Don't let one unusable apt repository block a runtime install by @zpoint in #10673
• [Server] Reclaim request logs that outlived their request row by @aylei in #10675
• [Jobs] Surface who requested a cancellation in the job's details by @lloyd-brown in #10661
• [Dashboard] Sum all task durations in job group duration display by @SeungjinYang in #10678
• [Dashboard] Show the owning user for pools by @claude[bot] in #10653
• [Slurm] Add quota.queue / quota.account config for sbatch --qos / --account by @kevinmingtarja in #10680
• [Slurm] Configure Unix submit usernames for users and service accounts by @kevinmingtarja in #10685
• [Slurm] Default service accounts to their creator’s Unix identity by @kevinmingtarja in #10686
• [Server] Publish the API server's own local disk footprint on /metrics by @aylei in #10674
• [API] Authorize download paths against authenticated user roots by @kevinmingtarja in #10683
• Distinguish an unsupportable request from unavailable resources by @SeungjinYang in #10697
• [Slurm] Add a scope-resolved getter for slurm.quota sub-fields by @SeungjinYang in #10700
• [Metrics] Track blob lookup sizes and cache hits by @kevinmingtarja in #10705
• [Server] Bound the auth users upsert with server-side timeouts aligned to the auth deadline by @cg505 in #10684
• [Server] Fix double close of the kubectl port-forward stdout fd under uvloop by @cg505 in #10689
• [Core] Bound idle-in-transaction time on every Postgres engine transaction by @cg505 in #10687
• [Jobs] Explain why a managed job has not started, and let a runner say more by @DanielZhangQD in #10692
• [Test] Guard the blob-upload route scan against FastAPI router wrappers by @DanielZhangQD in #10711
• [API] Count middleware rejections at the edge (no client-visible change) by @cg505 in #10691
• [Server] Count auth-path DB timeouts by @DanielZhangQD in #10710
• [Dashboard] Render plugin-contributed rows for contexts not enabled for compute by @aylei in #10579
• [CI] Notify Slack when the nightly run is cancelled, not only when it fails by @ishankaul1 in #10719
• [CI] Let the nightly notifier depend on every other job by @zpoint in #10720
• [Test] Retry on host port conflicts when starting the remote-server container by @zpoint in #10724
• [Server] Add opt-in SO_REUSEPORT to balance connections across API server workers by @aylei in #10016
• [Pools] Make pools work under Kueue quota by @lloyd-brown in #10718
• [Kubernetes] Let pods with DRA claims terminate gracefully by @SeungjinYang in #10751
• Fix SLURM containerized run poisoned by exported which bash function by @daw3rd in #10716
• [k8s] Prefer the release's base suite when an apt install cannot fetch its packages by @MadhavJivrajani in #10755
• [Jobs] State changes/migration to support dynamic job group by @ishankaul1 in #10698
• [Jobs] Attach a managed job to the job it is launched from (--job-group) by @ishankaul1 in #10701
• [Jobs] Cancel the jobs launched from a job with it; the root sweeps them when it finishes by @ishankaul1 in #10702
• [Jobs] Show jobs launched from a job group under it in sky jobs queue and the dashboard by @ishankaul1 in #10725
• [Jobs] Number dynamic tasks like tasks; address one as by @ishankaul1 in #10734
• [Dashboard] Dynamic tasks read as tasks of their group by @ishankaul1 in #10736
• [Tests] Smoke tests for jobs launched from inside a job group by @ishankaul1 in #10726
• [Docs] Launching jobs from inside a job group by @ishankaul1 in #10732
• [Jobs] Let an unknown task fall through to the log reader instead of raising by @ishankaul1 in #10758
• [Provision] Prefer the release's base suite when a container's apt install cannot fetch its packages by @zpoint in #10759
• [Server] Stop transient processes from growing the multiproc metrics dir by @aylei in #10575
• [Server] Refuse an upload that cannot fit on the server's local disk by @aylei in #10714
• [CI] Cancel the Buildkite build when its GitHub run is cancelled by @zpoint in #10760
• [Dashboard] Dynamic task UX nits: 'of this job' tooltip, drop (N dynamic) from group badges, Dynamic pill in the jobs list by @ishankaul1 in #10761
• Add fleet GPU capacity and infra counts by @Michaelvll in #10749
• [Metrics] Measure the SSH round trip past the API server by @zpoint in #10656
• [Provision] Report the execution target in the 'Instances launched' event by @SeungjinYang in #10783
• [Test] Move pinned miniconda smoke-test image off EOL Debian bullseye by @cg505 in #10784
• [Jobs] Retry transient DB errors in a job's final cleanup and bookkeeping by @aylei in #10655
• [Jobs] Restore a job's config file atomically by @aylei in #10782
• [Test] Let the pipeline generator's -k use pytest's own selection by @zpoint in #10787
• [Provision] Revert the EOL-bullseye apt workarounds now that the pinned image is on bookworm by @zpoint in #10786
• [Core] Time async and generator functions in time_me by @aylei in #10626
• [Server] Let a deployment state its state-DB connection pool size by @aylei in #10791
• [Core] Report where a managed job's start-up time goes by @DanielZhangQD in #10638
• Release 0.13.2rc1 by @Michaelvll in #10817

New Contributors

• @spiffaz made their first contribution in #10156
• @edmsimplicio made their first contribution in #10230
• @yash170106065 made their first contribution in #9924
• @pavneeta made their first contribution in #10260
• @liuxu623 made their first contribution in #10284
• @complynx made their first contribution in #10345
• @Anai-Guo made their first contribution in #10356
• @Khas-Erdene-Tsogtsaikhan made their first contribution in #10286
• @ryxli made their first contribution in #10436
• @intagliated made their first contribution in #9932
• @klenovda made their first contribution in #9829
• @himanshu1573 made their first contribution in #10369
• @typesafeainate made their first contribution in #10172
• @davanstrien made their first contribution in #10558
• @daw3rd made their first contribution in #10716

Full Changelog: v0.13.1rc1...v0.14.0rc1