kubernetes-sigs/kueue v0.18.4

on GitHub

Changes since v0.18.3:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

• Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.17.0, v0.18.0.
• Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.18.1, v0.18.2, v0.18.3.

Changes by Kind

Feature

• Helm: Added enableVisibilityAuthReaderRoleBinding Helm value (default: true) to make the visibility server's auth-reader RoleBinding in kube-system optional. Set to false when deploying under a GitOps project that cannot manage resources in kube-system, and create the RoleBinding out-of-band instead. (#13048, @amy)

Bug or Regression

• AFS: Fixed a Denial of Service (DoS) vulnerability where deleting a LocalQueue could cause the Kueue scheduler to hang during AdmissionFairSharing calculations. (#13214, @Vaishnav88sk)
• AFS: Fixed a race in Admission Fair Sharing penalty updates where concurrent workload operations could lose penalty changes, causing LocalQueues to receive incorrect priority. (#13285, @MaysaMacedo)
• DRA: Fixed a bug where byte-valued Partitionable Devices (counter-based) resources were displayed as raw byte integers in Workload and ClusterQueue status.
  These resources are formatted using human-readable BinarySI units, such as Mi and Gi. (#13038, @amarkdotdev)
• DRA: introduce a safeguard for invalid parameter combinations to prevent nil dereference crashes (#12979, @mykysha)
• ElasticJobsViaWorkloadSlices: Fixed a bug that allowed a replacement Workload slice to reference a Workload from another namespace when both used the same ClusterQueue, potentially causing the unrelated Workload to be treated and finished as the replaced slice. Workload slice replacements are now restricted to Workloads in the same namespace. (#13081, @mykysha)
• ElasticJobsViaWorkloadSlices: Fixed a bug that could cause elastic Jobs to
  stall after Pods succeeded or failed, because terminal Pods continued to count
  against the active Workload slice's admitted PodSet count and prevented
  replacement Pods from being ungated. (#13204, @garg02)
• ElasticJobsViaWorkloadSlices: Fixed a bug where an elastic job could permanently fail to start (FailedToStart) due to stale Kueue-owned annotations on the pod template, e.g. after its workload was deleted, or after eviction of a previously scaled-up job. (#13131, @mcochner)
• ElasticJobsViaWorkloadSlices: Fixed a bug where reclaimable Pod accounting after scaling down an elastic Job could reserve quota for Pods that were no longer running. Reserved quota now tracks the remaining running Pods for indexed and non-indexed Jobs. (#13263, @Shreesha001)
• ElasticJobsViaWorkloadSlices: Fixed a bug where scaling down an elastic Job could leave a stale reclaimablePods count, causing Kueue to account for less quota than the Job's remaining Pods were using. (#13060, @Shreesha001)
• KueueViz: Fixed a security issue in kueueviz where WebSocket connections continued streaming cluster data after a bearer token expired or was revoked.
  Connections are now closed within 30 seconds of token invalidation. (#13130, @Vaishnav88sk)
• KueueViz: Navigating to an invalid cohort now displays a graceful error message instead of crashing the UI. (#13246, @Vaishnav88sk)
• KueueViz: Prevent workload detail pages from crashing when Kubernetes Events have missing or invalid timestamps. (#13207, @YQ-Wang)
• MultiKueue: Fixed a bug where a remote Workload finishing with reason OutOfSync was mirrored as a terminal finish, leaving the manager Job stranded. Kueue now resets the MultiKueue AdmissionCheck to Retry, retries the Workload, and emits a warning event identifying the worker cluster. (#13086, @Smuger)
• MultiKueue: Fixed a bug where a transient watch reconnect to a worker cluster could evict a running admitted workload. Kueue now measures the worker-lost grace from when the worker cluster's connection first dropped, rather than from the admission check's transition time, and retries immediately only when the reserving worker is reachable but its remote workload is gone. (#12999, @kevin85421)
• MultiKueue: Fixed a bug where admitted workloads could remain stuck instead of being evicted and retried after workerLostTimeout when reconnecting to a worker cluster failed after its connection configuration changed. (#13188, @kevin85421)
• MultiKueue: Fixes an observability bug where Pods scheduled in a worker cluster could still appear unscheduled
  in the manager cluster (as PodScheduled=False would be preserved). The PodScheduled condition is now
  synchronized from the worker cluster, while preserving SchedulingGated for unschedulable Pods to avoid spurious
  scale-ups. (#13197, @fg91)
• Observability: Fix verbose DRS logs failing to report DRS values due to JSON parsing error when handling fair sharing weight set to 0. (#13157, @kshalot)
• RayJob, RayCluster, RayService, JobSet, MPIJob, and Kubeflow Trainer jobs: Fixed a bug where changing a running job's pod set count, for example adding a worker group to a running RayCluster, could crash the Kueue controller during reconciliation. (#13104, @ivnovakov)
• TAS & Scheduling: Fixed a bug where Workloads owned by a single Pod could be reassigned after eviction or during TAS node hot swap, even though the existing Pod could not consume the new assignment. The fix applies when the SkipReassignmentForPodOwnedWorkloads feature gate is enabled. The gate is Beta and enabled by default in 0.19+, and Alpha and disabled by default in the 0.17 and 0.18 release branches. (#12980, @yakticus)
• TAS: Added a fix for premature node replacement when a node remains NotReady while the workload's Pods are still running, which could cause the topology assignment to diverge from the actual Pod placement and corrupt per-node capacity accounting. The termination-driven behavior applies when TASReplaceNodeDueToNotReadyOverFixedTime is disabled. The gate is deprecated and disabled by default in 0.19+, and Beta and enabled by default in the 0.17 and 0.18 release branches. (#13096, @yakticus)
• TAS: Fix a performance bug where repeatedly checking the enablement of the TASRespectNodeAffinityPreferred feature gate inside a hot sorting loop could significantly increase the scheduling time (14% by the attached benchmark). (#13145, @j-skiba)
• TAS: Fixed a bug where a PodSet slice size that did not evenly divide its count could make the topology ungater panic repeatedly, so the workload's Pods stayed stuck gated. The ungater no longer panics and ungates the Pods that fit the topology assignment. (#13268, @ivnovakov)
• TAS: Fixed a bug where a PodSet with subGroupIndexLabel set but a missing or zero subGroupCount could crash the tas-ungater controller. Kueue now falls back to greedy domain assignment for these pods instead of panicking. (#13065, @reruno)
• TAS: Fixed a performance bug that caused remaining capacity to be repeatedly recalculated and resource maps to be unnecessarily copied during workload evaluation, particularly when evaluating multiple preemption candidate sets in
  large clusters. The fix is guarded by the Beta TASCachingRemainingResources feature gate, which is enabled by default. (#13235, @j-skiba)
• TAS: domain selection is now deterministic when multiple domains tie on score; ties are broken by the domains' levelValues ordering. (#13031, @mvanhorn)
• TAS: fixed excessive scheduling latency for workloads requiring preemption caused by repeatedly evaluating node selectors, tolerations, and affinity for each preemption simulation. The optimization is controlled by the beta TASCacheNodeMatchResults feature gate, enabled by default. (#13206, @j-skiba)
• VisibilityOnDemand: Fixed a bug where a large or negative limit query parameter on the pending-workloads endpoints could crash the Kueue controller manager via memory exhaustion or a panic. The limit is now capped at 100000. (#13053, @reruno)

Other (Cleanup or Flake)

• Observability: Introduced logging of node replacements by NodeHotSwap. (#13215, @dkaluza)
• Observability: Introduced logging of unhealthy nodes on workload updates. (#13273, @dkaluza)
• TAS: Improved scheduling evaluation performance and reduced memory allocations for Topology-Aware Scheduling (TAS). (#13327, @j-skiba)
• TAS: improved workload evaluation performance by optimizing domain-ordering tie-breaks for sibling node domains with equal available capacity. (#13331, @j-skiba)