kubernetes-sigs/kueue v0.20.0-rc.0 on GitHub

Changes since v0.19.0:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.18.0, v0.19.0.
DRA & ResourceTransformation: Fixed a bug where DRA device-class mapping or a resource transformation under the reserved resource name pods was silently discarded or left the Workload permanently pending.
Remove or rename those entries before upgrading, or the kueue-controller-manager will fail to start. Renaming a mapping name or an outputs key also requires updating the matching ClusterQueue nominalQuota entries in the same change. (#13989, @thc1006)
LeaderWorkerSet: Fixed a quota bypass where raising spec.leaderWorkerTemplate.size on an already-admitted, Kueue-managed LeaderWorkerSet ran more pods per group than the reserved quota covered. spec.leaderWorkerTemplate.size is now immutable while the LeaderWorkerSet is managed by Kueue, behind the new LWSImmutableGroupSize feature gate (Beta, enabled by default). spec.replicas stays mutable.

If you change spec.leaderWorkerTemplate.size on a Kueue-managed LeaderWorkerSet, recreate it at the new size instead, or disable the LWSImmutableGroupSize feature gate to keep the previous behavior, which also restores the quota bypass. (#13279, @ivnovakov)

TAS: Enforce stricter slice-size validation for Workloads. When podSetSliceRequiredTopology is specified, podSetSliceSize must also be specified and must be greater than 0. Non-positive slice sizes in topology constraints are also rejected.

If you create Workload objects directly (or via custom controllers), update manifests before upgrade so that:

podSetSliceRequiredTopology is never set without podSetSliceSize
podSetSliceSize is always greater than 0
podSetSliceSize is not set when podSetSliceRequiredTopology is absent
every podsetSliceRequiredTopologyConstraints entry has size greater than 0

If you need a phased rollout, temporarily disable TASValidateWorkloadSliceSize, clean up invalid Workloads, then re-enable it. (#12728, @mszadkow)

TAS: Fix a bug where TASRecomputeAssignmentWithinSchedulingCycle can be enabled even if TopologyAwareScheduling is disabled.

If you disable TopologyAwareScheduling, also set TASRecomputeAssignmentWithinSchedulingCycle=false before upgrading. (#14237, @tenzen-y)

TrainJob: support Trainer v2.3.

If you use Trainer v2.2, you have to update it to v2.3 since Trainer v2.2 and v2.3 don't have API compatibility. (#14504, @SatishChoudhary642)

V1beta2: set unserved (served: false) for v1beta1 types and resources.

To ensure your environment is consistent, we recommend running the following migration script before installing Kueue v0.20 and verifying cluster stability: https://raw.githubusercontent.com/kubernetes-sigs/kueue/main/hack/migrate-to-v1beta2.sh.
The script triggers a "no-op" update for all existing Kueue objects, forcing the API server to pass them through conversion webhooks and save them in the v1beta2 version. (#14558, @mbobrovskyi)

Changes by Kind

API Change

Add scheduling.quotaReleaseStrategy to the Kueue Configuration API.
Administrators can set this field to OnTermination (default, releases quota
when pods begin terminating) or OnTerminalBestEffort (holds quota until pods
fully terminate, required for TopologyAwareScheduling). (#13224, @vic-comm)
Allow Workload PodSet minCount to be 0, matching PodSet count. Previously minCount had to be at least 1, so count=0 with minCount=0 was rejected. (#14720, @vladikkuzn)
DynamicQuotaOrchestration: Added the DynamicQuotaOrchestration API (#14557, @pajakd)
DynamicQuotaOrchestration: Added the status.effectiveQuotas field to ClusterQueue and Cohort in v1beta2 to support dynamic quota allocation. (#14389, @j-skiba)
ResourceFlavor spec.tolerations can now be updated when spec.topologyName is set. (#13622, @tomsen02)
Revert scheduling.quotaReleaseStrategy from Kueue Configuration API. (#15371, @vic-comm)
TAS: Allowed mutating Topology.spec.levels when kubernetes.io/hostname is the lowest level both before and after the change. Workloads left pending under the previous levels are automatically retried, and the usage of already-admitted workloads is preserved across the change. (#13910, @tomsen02)
TAS: Allowed updating spec.nodeLabels of a ResourceFlavor with topologyName set. Admitted workloads keep their quota reservation and topology assignment, pending workloads are requeued against the new node set, and usage of workloads admitted on overlapping flavors is counted exactly once. (#13738, @tomsen02)
WaitForPodsReady: Add an unscheduledTimeout field to the Kueue Configuration API (#15372, @tenzen-y)

Feature

Add kueue_execution_time_seconds and kueue_local_queue_execution_time_seconds histogram metrics tracking total workload execution time from admission to completion. (#13420, @sohankunkerkar)
Added a Warning event on the object carrying the kueue.x-k8s.io/priority-class label when Kueue looks up the WorkloadPriorityClass it names and finds it missing. (#13775, @thc1006)
Added the kueue_pending_scheduling_hashes metric to report the number of unique pending scheduling equivalence hashes per ClusterQueue. (#12520, @RooobinYe)
Custom Metric Labels (Alpha): Workload SourceKind support for the kueue_pending_workloads metric. (#13469, @Singularity23x0)
DRA: Workloads with AdminAccess device requests are now admitted with zero quota charge instead of being rejected. AdminAccess provides shared read-only access to already-allocated devices. (#13642, @sohankunkerkar)
DynamicQuotaOrchestration: Added the DynamicQuotaOrchestrator controller behind the DynamicQuotaOrchestration feature gate to discover and aggregate capacity across referenced CapacityProviders into effective capacity status. (#15199, @j-skiba)
DynamicQuotaOrchestration: Implemented proportional capacity distribution and soft validation for DynamicQuotaOrchestrator. (#15283, @pajakd)
DynamicQuotaOrchestration: Registered reconciler watches and added integration tests for DynamicQuotaOrchestrator capacity distribution. (#15286, @pajakd)
DynamicQuotaOrchestration: Supported status.effectiveQuotas in ClusterQueue scheduling and controllers under the DynamicQuotaOrchestration feature gate, falling back to spec.resourceGroups when unset or disabled. (#15040, @j-skiba)
DynamicQuotaOrchestration: Supported status.effectiveQuotas in Cohort scheduling and controllers under the DynamicQuotaOrchestration feature gate, falling back to spec.resourceGroups when unset or disabled. (#15107, @j-skiba)
Graduate RejectUpdatesToCQWithInvalidOnFlavors to Beta stage enabled by default. (#14776, @tenzen-y)
Helm: Added support for setting affinity on the kueue-controller-manager Deployment via controllerManager.affinity. (#14955, @HsiuChuanHsu)
Helm: the controller-manager Deployment now supports optional controllerManager.strategy, controllerManager.hostNetwork, and controllerManager.dnsPolicy values. (#13817, @dinhxuanvu)
Kueue supports KubeRay v1.7 History Server options in MultiKueue and accounts for collector sidecar resources in Ray workload quotas. (#14670, @YQ-Wang)
KueurViz: support WebSocket heartbeat (#13970, @amirialy)
MultiKueue: Added a new metric multikueue_cluster_status that reports the active status of each worker cluster, per manager ClusterQueue referencing it, labeled by cluster_queue, cluster, active, and replica_role. (#13798, @Mostafahassen1)
MultiKueue: Added support for reusing clientConnection configuration (QPS and Burst) for worker clusters via the MultiKueueReuseClientConnectionConfigForWorkers Alpha feature gate. (#15280, @alien1403)
MultiKueue: Forwarded in-place serveConfigV2 (Ray Serve application config) updates on a RayService from the manager to the worker cluster, so editing the Serve config on the manager now takes effect on the worker promptly. Changes to rayClusterConfig/upgradeStrategy (zero-downtime upgrade) are not yet propagated. (#14036, @kevin85421)
MultiKueue: The MultiKueueCluster Active condition message now surfaces the reconnect backoff progress (failed connection attempts and next retry time) while the cluster is disconnected, so connectivity and reconnect status are visible via kubectl without reading controller logs. (#13726, @kevin85421)
Observability: Added dynamic_quota_orchestrator label to kueue_cluster_queue_info and kueue_cohort_info metrics, reporting the managing orchestrator when dynamic effective quota is active. (#15201, @j-skiba)
Observability: Added kueue_preemption_target_recomputations_total metric to track the result of overlapping preemption target recomputations within a scheduling cycle. (#14494, @SatishChoudhary642)
Observability: Added kueue_workload_recovery_wait_time_seconds and kueue_local_queue_workload_recovery_wait_time_seconds metrics to measure the time taken for workloads to recover when WaitForPodsReady recovery is enabled. (#14766, @alien1403)
Promote MultiKueueKubeConfigPathValidation to beta (#13706, @kannon92)
Security: Promote TLSOptions to stable. (#13373, @kannon92)
Support Workload custom labels in LocalQueue admitted workload metrics when CustomMetricLabels feature gate is enabled. (#13635, @Vaishnav88sk)
TAS: Added support for PodSet slicing alongside PodSet grouping, allowing workloads such as LeaderWorkerSet to co-locate grouped leader PodSets with sliced worker PodSets. This behavior is gated by the TASGroupedPodSetSlicing feature gate. (#15111, @pajakd)
TAS: Added the TASNodeFeasibilityForAllLevels feature gate, Beta and enabled by default. Topologies that do not declare kubernetes.io/hostname as their lowest level now check capacity and node feasibility per node instead of per aggregated domain, so node taints, node selectors and node affinity exclude individual nodes inside a domain. Workloads that fit a domain's total capacity but no single node are no longer admitted. Requires TopologyAwareScheduling. (#14191, @sohankunkerkar)
TAS: Reduced CPU time and memory allocations for snapshot creation by reusing cached topology trees when scheduling-relevant Node data is unchanged. Controlled by the TASCacheTopologyTree feature gate, which is Beta and enabled by default. (#13819, @akshay-pm)
TAS: enable hostPort conflict detection to node feasibility checks, skipping nodes with occupied host ports when SchedulerLibraryIntegration is enabled. (#13555, @sohankunkerkar)
The WorkloadPriorityClassDefaulting feature gate is graduated to Beta and enabled by default. (#13719, @sanskar-singh-2403)
Workload: Enabled the UnadmittedWorkloadsObservability feature gate by default, reporting granular reasons (e.g., WaitingForQuota, ExceedsMaxQuota, WaitingForPodsReady, Misconfigured, or Suspended) in the QuotaReserved status condition and metrics for unadmitted workloads. (#13063, @j-skiba)
WorkloadAwareScheduler: Added the kueue.x-k8s.io/workload annotation to Pods created by Kueue-managed jobs when the SchedulerLibraryIntegration feature gate is enabled; previously only TopologyAwareScheduling added it. (#14551, @Singularity23x0)
WorkloadAwareScheduler: Delegated TAS node readiness and spec.unschedulable checks to the scheduler-library instead of applying them when building the TAS node cache. Controlled by the SchedulerLibraryIntegration feature gate, which is Alpha and disabled by default. (#14203, @alien1403)
kueue_local_queue_pending_workloads now emits one series per workload label combination when CustomMetricLabels is enabled with a SourceKindWorkload custom label (#14167, @carmal891)

Documentation

Website: Added a Light / Dark / System theme switcher to the documentation site. (#13492, @Shreesha001)

Bug or Regression

AFS: Fixed a bug that could modify cached Workload data while calculating LocalQueue fair-sharing usage, potentially producing inconsistent scheduling snapshots. (#13180, @aburan28)
AFS: Fixed a bug where a LocalQueue with fairSharing.weight: 0 could be prioritized for admission instead of deprioritized when AdmissionFairSharing is enabled. (#13481, @sumanthd032)
AFS: Fixed a bug where transient LocalQueue lookup errors in the heap comparator could flip the scheduling order between fair-sharing and priority-based, causing inconsistent workload admission when Admission Fair Sharing was enabled. (#13570, @sanskar-singh-2403)
AFS: Fixed entry-penalty accounting leaks that could inflate LocalQueue fair-sharing usage when a Workload was re-admitted or exited before settlement. (#14153, @apullo777)
AFS: Fixed pending Workload snapshot ordering when a referenced LocalQueue is missing. (#13427, @YQ-Wang)
AdmissionChecks: Fix a bug where the Workload has an Admitted=True condition regardless of AdmissionCheck Rejection state. (#14309, @TapanManu)
AdmissionFairSharing: Fixed a bug where resource usage smaller than one milli-unit was truncated to zero before it could accumulate, so with a long usageHalfLifeTime the consumedResources for CPU and extended resources such as GPUs stayed at 0 permanently and were ignored by fair sharing. (#13621, @Shreesha001)
AdmissionFairSharing: Fixed a bug where workloads admitted via AdmissionChecks could keep their entry penalty permanently, inflating LocalQueue fair-sharing usage and deprioritizing later workloads. (#12786, @apullo777)
AdmissionFairSharing: Fixed preemption ordering for Workloads from same-named LocalQueues in different namespaces so that LocalQueue usage is considered. (#14547, @tomsen02)
AdmissionFairSharing: Fixed stale fair-sharing usage caused by entry penalties being applied to non-usage-based ClusterQueues or reapplied during second scheduling passes. (#13844, @apullo777)
AdmissionFairSharing: Fixed transient LocalQueue lookup errors causing a pending-Workload snapshot to mix
fair-sharing comparisons with standard queue ordering, resulting in a non-transitive comparator and inconsistent
admission order. When a lookup fails, the entire snapshot now falls back to standard queue ordering. (#13433, @YQ-Wang)
CLI: Fixed a bug where kueuectl create clusterqueue rejected valid decimal quantities such as cpu=1.5 or memory=1.5Gi in --nominal-quota, --borrowing-limit and --lending-limit with invalid resources specification. (#15299, @henry3260)
CLI: Fixed a bug where kueuectl list workload --clusterqueue never matched pending Workloads, so combining it with --status pending always returned no results. (#15336, @henry3260)
CLI: Fixed a bug where kueuectl list workload could print the JOB TYPE and JOB NAME columns in a random order for Workloads with multiple owner references, such as pod groups. (#15357, @henry3260)
CLI: Fixed a bug where kueuectl list with -o yaml or -o json produced invalid output when the result spanned more than one page (KUEUECTL_LIST_REQUEST_LIMIT, 100 by default). All pages are now merged into a single document. (#15348, @henry3260)
CLI: Fixed kueuectl list pods incorrectly treating any output format containing "wide" as -o wide, causing JSONPath and similar expressions to be silently ignored. (#15156, @DevaanshPathak)
CLI: Fixed shell completion for the --clusterqueue and --localqueue flags, which returned no suggestions once a positional argument was typed, for example in kueuectl create localqueue NAME -c <TAB>. kueuectl delete workload completion now also includes inactive Workloads. (#15375, @henry3260)
ClusterQueue: Fixed a bug where a terminating ClusterQueue (one with a deletion timestamp still retained by the resource-in-use finalizer because workloads are reserving quota) stopped updating status.pendingWorkloads, status.admittedWorkloads, and status.reservingWorkloads and never set its Active condition to Terminating, leaving stale status. Kueue now keeps the status of a terminating ClusterQueue accurate. (#13680, @kaushik229)
ConcurrentAdmission: Fix preemption ordering by waiting for more-preferred variants to be evaluated for admission before opening the preemption gate for a less-preferred variant. (#14272, @yuluo-yx)
Corrected invalid PodSet info errors to report the expected and actual PodSet counts in the correct order. (#13671, @cryo-zd)
DRA: Fixed DeviceClass validation errors reporting a duplicated request field path with an incorrect request index in counter-based and capacity-based quota paths. (#13826, @cryo-zd)
DRA: Fixed a bug where a negative extended-resource request quantity (reachable only when the WorkloadValidateResourcesAreNonNegative validation is disabled, or on a Workload created before that validation existed) could be merged as a negative DRA quota charge, silently offsetting a legitimate charge on the same logical resource. Negative extended-resource requests are now dropped the same way zero-valued ones already are. (#14367, @pujitha24)
DRA: Fixed a bug where deactivating a pending Workload caused Kueue's internal resource adjustments (RuntimeClass overhead, LimitRange defaults, limits-derived requests) to be written back into the user's Workload spec. (#15095, @tomsen02)
DRA: Fixed a bug where extended resource quota could be charged against a DeviceClass the scheduler would not allocate from when multiple DeviceClasses share the same extendedResourceName. (#14044, @thc1006)
DRA: Fixed a startup crash when KueueDRAIntegrationPartitionableDevices or KueueDRAIntegrationConsumableCapacity feature gates are enabled but the ResourceSlice API (resource.k8s.io/v1) is not available on the cluster. (#13629, @MaysaMacedo)
DRA: Fixed config validation silently accepting capacity names with more than one slash, which produced a mapping that never matched any device. (#14951, @NasitSony)
DRA: Fixed quota undercount when two extended resource names sharing a deviceClassMappings key were requested by different containers in the same PodSet. (#14200, @pujitha24)
ElasticJobsViaWorkloadSlices & ProvisioningRequest: Fixed scale-from-zero admission for elastic jobs. Kueue now
omits zero-count PodSets, which are invalid in a ProvisioningRequest. If there are no other PodSets requiring ProvisioningRequests the AdmissionCheck is marked Ready. (#13975, @neilb-dotcom)
ElasticJobsViaWorkloadSlices: Fixed elastic jobs (e.g. autoscaling RayClusters via ElasticJobsViaWorkloadSlices) leaving scaled-up pods stuck SchedulingGated after the origin workload slice was deleted. (#14139, @dinhxuanvu)
ElasticJobsViaWorkloadSlices: Fixed the bug that changes to the kueue.x-k8s.io/priority-class label were not
reflected on the live Workload slices. (#13780, @thc1006)
FairSharing: Collapsed the per-candidate FairSharing preemption log into one entry per ClusterQueue and serialize its DominantResourceShare values, reducing scheduler log volume at verbosity 4. (#14348, @venuchitta)
FairSharing: Fix a bug where Kueue could miss valid preemption targets after selecting workloads from the preemptor's own ClusterQueue and lowering its DRS. The fix is guarded by the Alpha FairSharingReevaluatePreemptionCandidates feature gate, which is disabled by default. Enabling the gate may increase exposure to the known fair-sharing preemption-loop issue tracked in #14543. (#14128, @lightZebra)
FairSharing: skip the FairSharing preemption tournament when the preemptor's dominant resource share is +Inf, since no candidate can be preempted, avoiding wasted per-candidate evaluation and its V(4) log volume. (#14490, @venuchitta)
Fixed a bug where a ClusterQueue with flavorFungibility.preference: PreemptionOverBorrowing could leave workloads pending indefinitely. A flavor that required preemption but had no preemption candidates could outrank a later flavor that fits, purely because its quota was sourceable at a shallower borrowing level in the cohort tree. (#13616, @YQ-Wang)
Fixed a bug where a Workload could be re-nominated to the same ResourceFlavor indefinitely and never reach the remaining flavors of its ResourceGroup. The flavor scan progress recorded for a Workload was discarded whenever the ClusterQueue's allocatable resource generation advanced, whenever the Workload was skipped due to in-cycle contention, or whenever the Workload was updated, all of which happen continuously on a busy Cohort. This most visibly affected Topology-Aware Scheduling, where a Workload whose topology cannot be placed on the flavor selected by quota needs to fall through to the next flavor. Controlled by the new PreserveFlavorScanProgress feature gate, enabled by default. (#13956, @varunsyal)
Fixed a bug where a prebuilt or externally created Workload could be treated as equivalent to its Job even when the Job's pod template declared pod-level resources or resourceClaims that the Workload's PodSet omitted, letting the Workload reserve less quota than its Pods actually request. (#14436, @pujitha24)
Fixed a bug where a transient ProvisioningRequest or PodTemplate creation error could remain in Workload status and later be reported as the cause of an unrelated deactivation. (#13785, @apullo777)
Fixed a bug where deleting a child object whose owner was already deleted (e.g. mixed foreground/background propagation during namespace teardown) could leave the child stuck in Terminating, because Kueue webhooks denied the garbage collector's finalizer-removal request with "workload owner not found". The tolerance applies only to objects already being deleted, and is gated by the new SkipAncestorCheckForDeletedWorkloads feature gate (Beta, enabled by default). (#13857, @tomsen02)
Fixed a bug where elastic-job worker pods could remain SchedulingGated for up to ~90s after a scale rollover when the ungater requeued a slice that had already finished. (#14261, @dinhxuanvu)
Fixed a bug where, with TASFailedNodeReplacementFailFast disabled, replacement pods for a workload whose node became unhealthy were ungated onto that same unhealthy node and immediately terminated, exhausting the pod recreation budget instead of waiting for a replacement domain. (#14099, @varunsyal)
Fixed a controller panic triggered by Namespace updates after a ClusterQueue failed to initialize because its Cohort had a cycle. (#13571, @YQ-Wang)
Fixed a quantity larger than int64 on a resource other than cpu being converted to a number of another magnitude, or of another sign, when Kueue computes a Workload's requests. A large enough resource transformation product could arrive negative and then be floored to zero, so the Workload was admitted against no quota at all. (#14042, @thc1006)
Fixed elastic job pods being ungated against a workload slice that was already being evicted, which allowed more pods to start than the slice still holding the reservation granted. (#13914, @thc1006)
Fixed missing UpdatedWorkload event when the AdmissionGatedBy annotation is propagated from a StatefulSet to its Workload. (#13931, @Shreesha001)
Fixed overly broad ClusterRole permissions by scoping webhook configuration and CRD access to only Kueue's own resources using resourceNames (#13098, @prash2512)
Fixed resource totals wrapping to a negative number when two contributions to the same resource sum past the int64 range. Both Requests implementations now saturate in Add and Sub, as they already did in Mul, so an unrepresentable total is no longer read as an empty request. (#14100, @thc1006)
HA: Fix a data race between concurrent reconciles in non-leading replicas, where the leader-aware decorator used one shared object as the destination for every lookup. (#13978, @thc1006)
Helm: Add kueueViz.{backend,frontend}.ingress.tlsEnabled to explicitly enable or disable TLS independently of tlsSecretName, allowing TLS without a chart-managed Secret. When unset, the existing tlsSecretName-based behavior is preserved. (#13784, @meln5674)
Helm: Fix a bug where user-defined metricsService labels are not propagated to the rendered manifests. (#14999, @HsiuChuanHsu)
Importer: Fixed a bug where the Pod importer picked a single ResourceFlavor for the whole Pod, so Pods whose resources map to different flavors could be imported with a wrong flavor assignment. Flavors are now resolved per requested resource. (#12924, @mszadkow)
Job: Fixed a bug where failed indexes of Indexed Jobs using "backoffLimitPerIndex" continued to hold quota after being recorded in "status.failedIndexes". (#13486, @garg02)
JobFramework: Fixed ancestor resolution to verify that each controller ownerReference's UID matches the referenced object. Previously an object whose ownerReference named a Kueue-managed ancestor with a stale or mismatched UID was treated as managed by that ancestor and was skipped by Kueue (not suspended/gated and no Workload created). (#13802, @vladikkuzn)
KueueCtl: Fixed a bug where kueuectl delete workload deleted a recreated owner with a different UID. (#14831, @DevaanshPathak)
KueueCtl: Fixed the kueuectl list clusterqueue comand to respect KUEUECTL_LIST_REQUEST_LIMIT
and paginate API requests instead of issuing an unbounded LIST request. (#14864, @ErikJiang)
KueueViz: Fixed a bug that displayed thousands of duplicate error notifications when a Workload was preempted. Users receive a single notification for each preemption. (#13175, @Vaishnav88sk)
KueueViz: Fixed crashes that occurred when the UI displayed error details containing values that could not be serialized. (#13984, @Dasmat13)
Kueueviz: Fixed the bug the WebSocket 1005 error would be shown on the dashboard after selecting a namespace. (#14788, @mykysha)
LeaderWorkerSet & StatefulSet: Fixed reconciliation errors in one independent branch cancelling the other branches.
LeaderWorkerSet Workload creation, update, and deletion branches now continue independently, as do StatefulSet
Pod finalization and Workload reconciliation. (#13921, @thc1006)
LeaderWorkerSet: Fixed a race when an existing Workload’s queue name and the LWS’s
kueue.x-k8s.io/admission-gated-by annotation changed during the same reconciliation. Kueue now persists both
changes atomically, preventing the Workload from entering the queue without its admission gate if the second
update is delayed or fails. (#14143, @tenzen-y)
MPIJob: Fixed TAS defaulting for runLauncherAsWorker jobs with missing or additional replica-spec entries, preventing a webhook panic and preserving rank-based topology placement. (#14421, @thc1006)
MPIJob: Hardened orderedReplicaTypes against a nil ReplicaSpec value in mpiReplicaSpecs, avoiding a nil pointer dereference if such an object is ever constructed. Kubernetes API server schema pruning already prevents this from being reached through normal cluster usage. (#13714, @pujitha24)
ManagedJobsNamespaceSelector: Fixed a bug that added the queue-name label and suspended Jobs in excluded namespaces. Jobs in excluded namespaces are left unchanged.` (#13375, @PannagaRao)
MultiKueue & LeaderWorkerSet: Fixed a bug that prevented workloads from using PrebuiltWorkloads whose names exceeded the 63-character label limit when "WorkloadIdentifierAnnotations" was disabled. Kueue falls back to annotations for these Workloads. (#13636, @Dasmat13)
MultiKueue: Fixed a bug where a Job is dispatched again due to propagated spec.ttlSecondsAfterFinished even after Job completion. Enable the Alpha MultiKueueBatchJobClearingTTLSecondsAfterFinishedOnWorkerCluster feature gate to enable fixing. (#14734, @kevin85421)
MultiKueue: Fixed a bug where a remote workload finishing with reason
OwnerNotFound was mirrored back verbatim, permanently finishing the manager
Workload and leaving the manager Pod's scheduling gates stuck. Such finishes
are now treated as a sync failure and reset for re-dispatch, matching
existing OutOfSync handling. (#14997, @NasitSony)
MultiKueue: Fixed a bug where a stale status.nominatedClusterNames could cause Server-Side Apply field manager conflicts with external dispatchers. Kueue now clears the field through a MutatingAdmissionPolicy when a Workload is admitted or evicted. (#13749, @vic-comm)
MultiKueue: Fixed a bug where scaling an elastic job managed through workload slices could delete the running remote objects of the replaced slice mid-handover, disrupting the job's pods. The replaced slice is now finished with reason WorkloadSliceReplaced, matching the scheduler, so its remote objects are kept during the handover. (#13489, @kevin85421)
MultiKueue: Fixed a bug where the WorkloadPriorityClass controller incorrectly updated the priority of MultiKueue remote workloads when a WorkloadPriorityClass value changed. Remote workloads are now skipped during priority synchronization. (#14963, @weizhoublue)
MultiKueue: Fixed an issue where remote-cluster watcher goroutines could continue running after a worker cluster was removed, disconnected, or reconfigured. (#13705, @andrewseif)
MultiKueue: Fixed stale observedGeneration on the AdmissionCheckActive condition after updating to a MultiKueueConfig that preserves the cluster health result. (#14838, @cryo-zd)
MultiKueue: Fixed watch establishment to prevent timeouts from blocking indefinitely on delayed watch responses. (#14020, @Dasmat13)
MultiKueue: The example create-multikueue-kubeconfig.sh now grants update and patch on ray.io/rayclusters to the MultiKueue worker ServiceAccount. Without this, elastic RayCluster worker-group replica changes made on the management cluster (via the ElasticJobsViaWorkloadSlices feature gate) were rejected on the worker cluster with a 403 Forbidden and never propagated. (#13649, @kevin85421)
MultiKueue: The example worker-cluster RBAC generated by create-multikueue-kubeconfig.sh now grants update on workloads, which is required to propagate scale-down of elastic workloads (ElasticJobsViaWorkloadSlices) to the worker cluster. Without it, scaling an elastic workload down failed with a Forbidden error and the Workload reconcile looped. (#13692, @kevin85421)
MultiKueue: Truncate quota automation condition messages so unsupported manager/worker resource configurations can be reported successfully. (#14985, @cryo-zd)
MultiKueue: an elastic RayCluster (ElasticJobsViaWorkloadSlices) managed by MultiKueue is now rejected at admission if enableInTreeAutoscaling is set, as MultiKueue does not support Ray autoscaling yet. Previously such a RayCluster was accepted but deleted right after admission due to inconsistent autoscaler-sidecar accounting between the manager and the worker. (#13244, @kevin85421)
MultiKueue: share one rate limiter across per-worker MultiKueue REST clients. (#15310, @weizhoublue)
Observability: Aligned controller logger names so that event-handler and reconcile logs of the same controller share one name. Core controllers use <kind>-reconciler (for example clusterqueue-reconciler), and the multikueue, provisioning and tas controllers use <subcomponent>-<kind>-reconciler (for example multikueue-workload-reconciler, tas-node-reconciler). Log filters that match on the previous logger names need to be updated. (#15259, @henry3260)
Observability: Fixed kueue_pod_scheduling_gate_removal_seconds observing negative durations when the controller clock trails the apiserver clock. The negative observations made the histogram's _sum decrease, which broke rate() over that series. (#14474, @Antrikshgwal)
Observability: Fixed a bug where Kueue metrics could silently report incorrect quota and usage values for very large resource quantities due to integer overflow, potentially misleading dashboards and alerts. Metrics now preserve large values correctly and report unlimited quotas as "+Inf". (#13585, @benkermani)
Observability: Fixed a bug where a LocalQueue could continue reporting stale admitted/reserving workload counts and resource usage after its referenced ClusterQueue was deleted. (#13708, @andrewseif)
Observability: Fixed a bug where the kueue_cluster_queue_resource_pending metric could be permanently inflated when a LocalQueue resync pushed a workload that was already tracked as inadmissible in the ClusterQueue. (#13660, @RooobinYe)
Observability: Fixed a bug where the kueue_pod_scheduling_gate_removal_seconds metric was missing the replica_role label (leader, follower, or standalone) carried by the other Kueue metrics. (#14488, @gangadhar-res)
Observability: Fixed a bug where the kueue_ready_wait_time_seconds, kueue_admitted_until_ready_wait_time, kueue_local_queue_ready_wait_time_seconds and kueue_local_queue_admitted_until_ready_wait_time_seconds metrics were emitted after failure recovery, skewing the metric towards longer wait times. (#14626, @kshalot)
Observability: Fixed a panic that could crash the manager when CustomMetricLabels is enabled with a ClusterQueue-sourced label and a Pod's Kueue scheduling gate is removed. (#14419, @thc1006)
Observability: Scheduling hash re-computations are now logged at V5 via the contextual logger. (#15055, @apullo777)
Pending Workloads rejected by a LimitRange are requeued when the LimitRange's max, min, or maxLimitRequestRatio change, or the LimitRange is deleted. (#14967, @tomsen02)
Pod Integration: Fixed a bug where a Pod could bypass ClusterQueue quota by setting kueue.x-k8s.io/pod-group-name to another Workload's name. Kueue now only adopts Workloads created by the pod-group framework (stamped with kueue.x-k8s.io/is-group-workload). (#13014, @vladikkuzn)
Pod: Fixed a bug where a serving pod group's evicted pod could be left stuck in Terminating forever, since its kueue.x-k8s.io/managed finalizer was only removed for a Workload deletion, not for other evictions (e.g. a recoveryTimeout eviction). This could cause a legitimate replacement pod to be deleted as excess instead, or permanently block a same-name (StatefulSet-owned) replacement from ever being created. Kueue now removes the finalizer as soon as an evicted pod has actually terminated. (#14632, @mszadkow)
PodGroup integration: Fixed a bug where a Pod could bypass ClusterQueue quota by setting kueue.x-k8s.io/pod-group-name to another Workload's name. Kueue now only adopts Workloads created by the pod-group framework (stamped with kueue.x-k8s.io/is-group-workload), and no longer finalizes a foreign Workload that merely shares the pod group name, which previously marked it Finished and released its quota while its pods were still running. (#13960, @vladikkuzn)
ProvisioningRequest: Fixed a bug where the Active condition's observedGeneration on a ProvisioningRequest AdmissionCheck was not updated when a configuration change kept the check healthy, leaving observedGeneration permanently behind metadata.generation. (#14931, @weizhoublue)
RayCluster: Fixed an unclear validation error for Kueue-managed RayClusters that enable in-tree autoscaling without being configured as elastic jobs. The error explains that "ElasticJobsViaWorkloadSlices" and the "kueue.x-k8s.io/elastic-job: "true"" annotation are required. (#13996, @kevin85421)
RayJob, RayCluster, RayService, and SparkApplication: Fixed a bug where removing the kueue.x-k8s.io/queue-name label from an unsuspended job was accepted, so the job stopped being managed by Kueue while its pods kept running and its resources were no longer counted against quota. Removing the label is now rejected, both from an unsuspended job and from a suspended job in a namespace with a default LocalQueue. Controlled by the ValidateRayAndSparkJobUpdates feature gate, which is Beta and enabled by default. (#13293, @ivnovakov)
RayJob: Fixed a bug where Workloads created for KubeRay RayJobs that ended in ValidationFailed could remain admitted and continue holding quota indefinitely. (#13415, @amirialy)
RayService: Fixed a bug where elastic (autoscaling) RayService pods could stay stuck in SchedulingGated on kueue.x-k8s.io/elastic-job after the origin workload slice was deleted, leaving the RayCluster below its desired replica count. (#14703, @kevin85421)
ResourceTransformations × DRA: Fixed negative generated totals so they no longer reduce retained Pod requests or DRA logical-resource charges. Negative outputs can still offset other generated outputs, and contributions to the same resource are now summed deterministically. Also fixed multiplyBy to scale generated outputs only; with Retain, the original input quantity remains unchanged. (#13986, @thc1006)
Reverted the incomplete fix that refused adopting non-pod-group Workloads by pod group name, along with its PodIntegrationValidateGroupOwner feature gate. The guard covered only Workload adoption and left the finalization path able to remove finalizers from and delete a foreign Workload. The fix will re-land with the finalization path covered and integration tests. (#13945, @vladikkuzn)
Scheduler: The scheduler now persists NoMatchingFlavor for workloads that request both covered and uncovered resources. (#15304, @PannagaRao)
Scheduling: Fix a bug in BestEffortFIFO where a workload with failed preemption could remain sticky at the queue head. (#14863, @vic-comm)
Scheduling: Fix preemption thrashing/loops caused by desynchronized eviction completion times by prioritizing preemptor workloads at the head of the scheduling queue. This is guarded by the PrioritizePreemptorWorkloads Alpha feature gate, disabled by default. (#13797, @Nilsachy)
Scheduling: Fix workloads becoming stranded after scheduling snapshot failures, and stale pending accounting when a LocalQueue moves to another ClusterQueue. (#13885, @apullo777)
Scheduling: Fixed a bug where Workloads differing only in PodSet names formed separate equivalence classes, so BestEffortFIFO queues re-evaluated each one individually and admission slowed on busy clusters. Controlled by the new SchedulingEquivalenceHashingIgnorePodSetName gate, enabled by default. (#14784, @venuchitta)
Scheduling: Fixed a bug where a Workload deactivated with a derived DeactivatedDueTo<Cause> reason (such as DeactivatedDueToRequeuingLimitExceeded) could remain stuck after reactivation because its WorkloadRequeued condition was not transitioned. Such Workloads are now reactivated correctly. (#14874, @adibmbrk)
Scheduling: Fixed a bug where editing the nodeTaints of a non-TAS ResourceFlavor did not retry workloads that had been made inadmissible by the taint, leaving them pending until an unrelated event triggered a retry. (#13670, @tomsen02)
Scheduling: Fixed a bug where editing the tolerations or nodeLabels of a non-TAS ResourceFlavor did not retry workloads that had been left inadmissible by the previous spec, leaving them pending until an unrelated event triggered a retry. (#13734, @tomsen02)
Scheduling: Fixed a bug where negative container resource requests or limits
could create artificial ClusterQueue quota credit, allowing Workloads to bypass
configured quota limits. Kueue now floors negative values to zero during quota
accounting and rejects them during Workload validation by default. The
validation is controlled by the Beta
WorkloadValidateResourcesAreNonNegative feature gate. (#12838, @vladikkuzn)
Scheduling: Fixed a bug where requeueing a Workload recomputed its scheduling equivalence hash even when neither the Workload nor its effective resource requests had changed, adding avoidable CPU and allocation overhead on the scheduler's requeue path. (#14958, @apullo777)
Scheduling: Fixed a bug which would charge the quota based on the LimitRange (if specified) for workloads
with only limits specified. That could create a mismatch between the charged quota and the resources actually
used by the running Pods. (#15023, @tomsen02)
Scheduling: Kueue now recomputes an assignment calculated during nomination if its preemption targets overlap
with targets selected for workloads processed earlier in the same scheduling cycle. This fixes a starvation scenario
in which a large “hero” workload, on a busy cluster, could repeatedly conflict with earlier workloads on preemption
targets and remain unscheduled; see #13320 for details.
The behavior is controlled by the Beta RecomputeAssignmentUponPreemptionTargetsOverlap feature gate. (#13863, @pajakd)
SparkApplication: Fixed a bug where errors adding volumes or volume mounts to driver and executor Pods were silently ignored. Configuration errors are reported during reconciliation instead. (#13548, @onkar717)
SparkApplication: Fixed a bug where workloads using dynamic allocation could remain unready after executor scale-down. Workloads are now considered ready when the configured minimum number of executors is running. (#14505, @zhengchenyu)
StatefulSet: Fixed a bug where scaling a StatefulSet to zero and then back up to a different number of replicas left the Workload with the original pod count, causing it to be admitted and charged quota for the wrong number of pods. (#15279, @sohankunkerkar)
TAS NodeHotSwap: Fix a bug where workload node replacements are potentially stuck (#15365, @tenzen-y)
TAS: Fix a bug where RecomputeAssignmentUponPreemptionTargetsOverlap doesn't work correctly when TASHandleOverlappingFlavors is enabled. (#14366, @tenzen-y)
TAS: Fixed a bug that could prevent admission of otherwise feasible LeaderWorkerSet workloads when the selected leader domain reduced the capacity available to workers. For example, a 1-CPU leader and four 2-CPU workers can now be placed across 2-, 4-, and 3-CPU hosts in one rack by assigning the leader to the 3-CPU host, leaving capacity for all four workers. (#13383, @YQ-Wang)
TAS: Fixed a bug where a completed PodSet in a multi-PodSet workload kept its topology domain occupied, preventing admission of pending workloads that had sufficient quota. This behavior is gated by the ReclaimablePods feature gate. (#15219, @kshalot)
TAS: Fixed a bug where a grouped PodSet (e.g. an LWS leader) with no requests for the TAS-managed resource was rejected with "no TAS flavor assigned". (#12562, @mszadkow)
TAS: Fixed a bug where a node whose hostname matched the value of the topology's top level was silently excluded from placement. The node stayed Ready with free capacity but never received pods, because its domain was recorded as its own parent and never registered as a topology root. (#14010, @akshay-pm)
TAS: Fixed a bug where cross-flavor TAS usage was matched against topology domains a ResourceFlavor does not hold, adding redundant per-node work and V(3) log lines to every scheduling cycle. (#14174, @venuchitta)
TAS: Fixed a bug where in-place pod resize or node migration of a non-TAS pod on a TAS-relevant node never updated the scheduler's usage cache, causing TAS workloads to see stale capacity until the pod terminated. (#13764, @sohankunkerkar)
TAS: Fixed a bug where inadmissible TAS workloads were not automatically requeued when non-TAS pods terminated, potentially leaving workloads stuck pending despite available capacity. (#8709, @sohankunkerkar)
TAS: Fixed a bug where node replacement treated sibling topology domains with a common string prefix as the same domain. (#14292, @tomsen02)
TAS: Fixed a bug where replacing an unhealthy node could assign a workload to a node already claimed by another workload in the same scheduling cycle, leaving its pod permanently Unschedulable until the PodsReady timeout evicted it. (#14572, @varunsyal)
TAS: Fixed a bug where resource accounting was incorrect after a ResourceFlavor was deleted and recreated, including ClusterQueues with multiple TAS flavors, allowing workloads to be admitted against topology capacity already used by other admitted workloads. (#13606, @tomsen02)
TAS: Fixed a bug where scaling up an elastic workload (ElasticJobsViaWorkloadSlicesWithTAS) with a leader/workers pod set group could overwrite the running leader pod's TopologyAssignment with a newly computed placement, causing the leader pod to be restarted and lose state. Kueue now preserves the leader's existing assignment and only places the newly added workers. (#13574, @RooobinYe)
TAS: Fixed a bug where the scheduler panicked and crash-looped when logging the snapshot at verbosity >= 6, if a topology domain had capacity but no admitted TAS workloads. (#13560, @venuchitta)
TAS: Fixed a bug where updating spec.nodeTaints on a ResourceFlavor with spec.topologyName set did not retry inadmissible workloads, leaving them Pending until an unrelated event triggered a requeue. (#13647, @tomsen02)
TAS: Fixed a bug where workloads were rejected when a node capacity-to-request ratio exceeded the int32 range and the VectorizedResourceRequests feature gate was disabled. (#13528, @tomsen02)
TAS: Fixed an issue in queue management where workloads requiring a second pass of scheduling could be pre-queued multiple times concurrently, causing duplicate backoff timer callbacks. (#13745, @j-skiba)
TAS: Fixed an issue where workloads taking a second pass to complete a delayed topology assignment or replace a failed node could lose their existing quota reservation when waitForPodsReady.blockAdmission was enabled. Replacing a failed node could also clear the admission of a running workload. (#13685, @apullo777)
TAS: Fixed inconsistent use of the vectorized SliceRequests implementation introduced in #2953 to
optimize TAS hot paths. The remaining direct uses of MapRequests in non-hot paths are now replaced
with the Requests abstraction, with the implementation selected by factory functions based on the
VectorizedResourceRequests feature gate. The previous MapRequests implementation remains available
when the feature gate is disabled. (#13338, @j-skiba)
TAS: Fixed regression where admission failure events for Topology-Aware Scheduling (TAS) missed reporting the limiting resource when a node's remaining capacity was zero. (#13395, @j-skiba)
TAS: Prebuilt Workloads whose topology request differs from their Job or single Pod are now rejected as out of sync instead of being adopted. For TAS Indexed Jobs, the prebuilt Workload's podIndexLabel must also match the Job-derived value. Pod groups are unaffected. (#14585, @tomsen02)
TAS: Reduced excessively large assumptions-violation logs to a short summary. Operators can view the individual leaf domain IDs at verbosity 6 when detailed diagnostics are needed. (#14242, @venuchitta)
TAS: Workload PodSetTopologyRequest PodSetSliceSize is now validated to be at least 1. Creating or updating workloads with PodSetSliceSize set to 0 is rejected by API validation. (#13634, @mszadkow)
The Workload validating webhook panicked when the QuotaReserved condition was set while status.admission was absent, so the API server refused the request with an internal error rather than one naming the field. It is now refused as a validation error. An update that leaves a Workload in the state it was already in is allowed through, so an object that entered etcd without passing through admission, from a restore or a migration, can still be updated and removed. (#14014, @thc1006)
TrainJob: Fix a bug where TrainJobs are stuck by using merge patches, instead of Updates, when admitting
or stopping TrainJobs, thus preserving the fields not represented in Kueue's vendored Trainer API. (#14836, @robert-bell)
TrainJob: Fixed a bug where the TrainJob mutating webhook added a duplicate Kueue-owned runtimePatch entry on every update, causing spec.runtimePatches to accumulate stale entries. (#15213, @kannon92)
TrainJob: Fixed admission dropping runtime-defined tolerations when ResourceFlavor tolerations were applied, which could leave pods Pending on tainted nodes. (#15162, @izturn)
VisibilityOnDemand: Fixed a bug where the PositionInLocalQueue on the ClusterQueue pendingworkloads was being inflated when two LocalQueues in different namespaces share the same name (for example, the auto-created default LocalQueue). (#14431, @pujitha24)
VisibilityOnDemand: Fixed a panic in the pending-workloads endpoints when Prebuild Workloads (BYOW) w/o priority are created (#14411, @thc1006)
Workload: Fixed a bug where Workloads with invalid labels or annotations in PodSet template metadata could be admitted and fail later when creating Pods. Kueue now rejects them during admission, controlled by the WorkloadValidationForPodSetMetadata feature gate (Beta, enabled by default). (#13679, @Dasmat13)
WorkloadPriorityClass: Fixed a bug where a Workload that had reserved quota was repeatedly written with
a priorityClassRef the API server rejects, when its owner's WorkloadPriorityClass label was removed. (#14565, @gangadhar-res)
WorkloadPriorityClass: Fixed the WorkloadPriorityClass controller to update workloads referencing a changed class through a bounded, cancellable worker pool instead of a serial, uninterruptible loop, and to report a single update error instead of one per failed workload. (#14737, @pujitha24)
Workloads: Fixed a bug that caused Workloads to be finalized as orphaned while their owner Job was being deleted. Kueue now waits until the owner Job no longer exists before finalizing the Workload. (#12849, @mbobrovskyi)
Workloads: Fixed a bug that could crash the Kueue controller when a namespace-scoped user created a Workload without a controller owner. Kueue now ignores the unrelated Workload. (#15346, @mbobrovskyi)
Workloads: Fixed a bug where a workload requeued after a backoff period was accounted using its raw spec rather than its effective resources, dropping requests derived from limits, LimitRange defaults, and RuntimeClass overhead. Kueue now correctly accounts the adjusted resources on requeue, preventing ClusterQueue overcommitment. (#15096, @tomsen02)

Other (Cleanup or Flake)

DRA: Narrowed ResourceSlice change notifications to only ClusterQueues covering DRA-mapped resources, reducing unnecessary inadmissible workload requeue churn. (#13639, @sohankunkerkar)
Helm: Aligned the default integration framework ordering with the Kustomize
controller configuration. This does not change the set of enabled integrations
or their runtime behavior. (#13314, @YQ-Wang)
KueueViz: Removed debug console.log and console.error statements from the frontend. WebSocket connection events, flavor data updates, and message-parse errors no longer appear in the browser developer console. Errors are still visible in the KueueViz UI through normal React state handling. (#13983, @Dasmat13)
MultiKueue: The MultiKueueBatchJobClearingTTLSecondsAfterFinishedOnWorkerCluster feature gate is now Alpha and disabled by default. Enable it explicitly to omit ttlSecondsAfterFinished from worker Jobs. (#15126, @kevin85421)
Observability: Fixed logs at verbosity 6 and below that did not conform to the JSON Lines format, allowing log collectors to parse them consistently. (#13637, @Dasmat13)
RayJob, RayCluster, RayService: Kueue no longer writes the internal, write-only kueue.x-k8s.io/raycluster-podset-replica-sizes annotation on Ray workloads. It was superseded by the kueue.x-k8s.io/raycluster-generation annotation and was not read by any controller code. (#13605, @kevin85421)
Scheduling: Unified the remaining resource-request construction paths to use factory methods that select either the MapRequests or SliceRequests implementation based on the VectorizedResourceRequests feature gate. (#13668, @Vaishnav88sk)
TAS: The Workload API now rejects negative subGroupCount values (#13108, @reruno)
The locked SkipFinalizersForPodsSuspendedByParent feature gate and the legacy parent-suspended Pod finalizer cleanup have been removed. (#13316, @YQ-Wang)
Upgraded the bundled KubeRay (ray-operator) dependency to v1.7.0. (#14669, @kevin85421)

Uncategorized

Helm: Add enableAlphaAPIs flag (default false) to conditionally render alpha CRDs (CapacityProvider, DynamicQuotaOrchestrator). (#15117, @pajakd)
LeaderWorkerSet: Fixed a bug where pods recreated from a LeaderWorkerSet whose pod templates were never stamped with the kueue.x-k8s.io/queue-name label stayed permanently scheduling-gated, eventually deactivating the Workload. (#14138, @anguszzzz)