github kubernetes-sigs/kueue v0.20.7

latest release: v0.19.14
58 minutes ago

Changes since v0.20.6:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

  • Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.19.0, v0.20.0.
  • Patch releases: Review the patch release notes leading up to this version, but only within this minor release line; see: v0.20.1, v0.20.2, v0.20.3, v0.20.4, v0.20.5, v0.20.6.

Changes by Kind

Bug or Regression

  • AdmissionChecks: Fixed retry backoff delaying eviction and unnecessarily retaining quota and topology reservations. (#18785, @garg02)
  • CLI: Fixed a bug where kueuectl create resourceflavor --tolerations key=:NoSchedule created an Exists toleration that also matched taints with nonempty values. It now creates an Equal toleration with an empty value. To tolerate any value, omit the = (for example key:NoSchedule). (#18943, @git-jxj)
  • ClusterQueue: Fixed a bug where deleting a ClusterQueue could stay blocked by its resource-in-use finalizer when its LocalQueue was deleted before a Workload that the scheduler was admitting. Kueue now rechecks the ClusterQueue when such a Workload is deleted. (#18975, @ivnovakov)
  • ConcurrentAdmission: Fixed a bug where adding a flavor to a ClusterQueue created a new Variant Workload for each finished Parent Workload in it, which the scheduler could then admit. Kueue no longer creates Variants for a finished or deleting Parent Workload. (#18961, @ivnovakov)
  • ConcurrentAdmission: Fixed the ClusterQueue status.pendingWorkloads and the kueue_pending_workloads metric counting a pending Workload once per Variant. A Workload is now counted once.
    ConcurrentAdmission: Fixed the LocalQueue status.pendingWorkloads and the kueue_local_queue_pending_workloads metric counting a pending Workload once per Variant. The LocalQueue status also no longer counts the Parent Workload after a Kueue restart or an eviction. A Workload is now counted once. (#19075, @Narwhal-fish)
  • DRA: Fixed a bug where a Workload whose quota charges were recomputed while the scheduler was processing it, for example after a DeviceClass listed in deviceClassMappings was created or a LimitRange default changed, could be requeued with its old charges and reserve quota with them. (#18781, @digvijay-y)
  • DRA: Fixed a bug where a deviceClassMappings entry with two counter or capacity sources for the same driver and name charged a device's quota twice. Kueue now rejects such entries at configuration validation, and checks every counter source when two mappings name the same DeviceClass. (#18923, @sohankunkerkar)
  • DRA: Fixed a bug where a workload whose ResourceClaimTemplate names a missing DeviceClass was admitted and held quota while its Pods could never schedule. (#18936, @sohankunkerkar)
  • FairSharing: Fixed a bug where a ClusterQueue could not reclaim its nominal quota by preemption from Workloads that kept running on a flavor removed from another ClusterQueue's quota. (#18913, @YevheniiOrlovEngineering)
  • FlavorFungibility: Fixed a bug where a Workload that needed to borrow one resource and preempt for another in the same flavor was never admitted with preference: PreemptionOverBorrowing, because no lower-priority Workloads were preempted. With whenCanPreempt: MayStopSearch, such a flavor also no longer stops the search when a later flavor fits without borrowing. (#18892, @YevheniiOrlovEngineering)
  • Importer: Fixed a bug where the Pod importer always used the first ResourceFlavor of a resource group, so a Pod could be imported with a flavor whose node labels or taints it does not match. The importer now uses the first flavor that fits the Pod's node selector, required node affinity and tolerations, and rejects the Pod when none does. (#19000, @burrows99)
  • KueueViz: Fixed Cohorts in a parent cycle missing from the Cohorts tree view. (#18917, @onkar717)
  • LeaderWorkerSet: Fixed a bug where Kueue logged an already exists error and requeued the LeaderWorkerSet when it tried to create a Workload it had already created. (#18891, @henry3260)
  • MultiKueue: Fixed a bug where multikueue_workloads_dispatched_total, multikueue_workloads_admitted_total and multikueue_workloads_evicted_total kept reporting a deleted ClusterQueue. (#19050, @henry3260)
  • MultiKueue: Fixed a bug where deleting a manager Job with --cascade=orphan left the remote Job and Workload running on the worker cluster. (#19067, @JanKaczmarski)
  • MultiKueue: Fixed a bug where removing a worker cluster from a MultiKueueConfig re-dispatched the Workloads admitted on that cluster to another cluster without deleting their remote objects, so the same job could run twice. Workloads already admitted on the removed cluster now keep running there until they finish. (#18984, @henry3260)
  • MultiKueue: Fixed a bug where requeueing all of a worker's Workloads after the worker reconnected could delay dispatching new Workloads. Kueue now requeues those Workloads at low priority. (#18997, @kevin85421)
  • MultiKueue: Fixed a bug where restarting the manager could delay dispatching new Workloads until all unfinished Workloads were reconciled. Kueue now queues those Workloads at low priority when the controller starts. (#19094, @kevin85421)
  • MultiKueue: Fixed a bug where the Incremental dispatcher retried deleted Workloads with backoff until Kueue was restarted, logging an error on every retry. (#19051, @Zhuoxi2000)
  • Observability: Fixed kueue_cohort_subtree_admitted_active_workloads keeping stale custom-label series after a ClusterQueue was relabeled with CustomMetricLabels enabled. The count now moves to the new label series immediately. (#19059, @vshalpnjabi)
  • Observability: Fixed a bug that caused LocalQueue pending workload metrics to report zero when the associated ClusterQueue did not exist or was deleted. Now Kueue reports these pending workloads as inadmissible and refreshes the metrics when a ClusterQueue is deleted. (#19017, @akshat-lakhera)
  • Observability: Fixed a bug where kueue_cluster_queue_resource_reservation, kueue_cluster_queue_resource_usage and kueue_cohort_subtree_resource_reservations kept stale values for a ClusterQueue being deleted while its admitted workloads finished. (#19125, @ivnovakov)
  • Observability: Fixed a bug where kueue_local_queue_admitted_active_workloads and kueue_local_queue_reserving_active_workloads could be missing, and the admitted gauge could later become negative, after a single LocalQueue update changed spec.stopPolicy together with a custom metric label value or a label matched by metrics.localQueueMetrics.localQueueSelector. (#19089, @henry3260)
  • Observability: Fixed a bug where kueue_local_queue_unadmitted_workloads could be reported for a LocalQueue after it was deleted or stopped, and where a Workload outside the queue, such as an inactive one, stayed counted in its old LocalQueue after it moved to a LocalQueue that did not exist. (#18789, @thc1006)
  • Observability: Fixed a bug where deleting a pending Workload during a scheduling cycle could log a spurious "Could not update Workload status" error and record a Warning event for the deleted Workload. (#18791, @bodyhedia44)
  • Pod: Fixed a bug where a stale single-pod reconcile request for a Pod that joined a pod group after creation, such as a StatefulSet or LeaderWorkerSet Pod, was retried with backoff until the Pod was deleted, repeatedly logging got a single pod reconcile request for a pod group. (#19057, @henry3260)
  • ProvisioningRequest: Fix a bug where a Pod or pod group with ephemeral containers (for example added with kubectl debug) remained pending when the ClusterQueue had a ProvisioningRequest admission check, because the PodTemplate created for the ProvisioningRequest was rejected. (#19054, @vladikkuzn)
  • ProvisioningRequest: Fix a bug where a Workload stayed Pending on a ProvisioningRequest admission check when a PodSet had a container without an image, such as the KubeRay autoscaler or the RayJob SidecarMode submitter. The ProvisioningRequest PodTemplate now fills missing images with the placeholder image kueue.x-k8s.io/placeholder-image:placeholder, and already stuck Workloads recover without being recreated. (#18839, @vladikkuzn)
  • Scheduling: Fixed a bug where an inadmissible Workload could stay pending until an unrelated event if Kueue temporarily failed to read its Namespace, for example due to an informer-cache miss. Kueue now retries these Workloads automatically. (#19060, @Om-Beast)
  • Scheduling: Fixed a bug where changing the boost on an admitted Workload with a Pod PriorityClass did not trigger reevaluation of pending Workloads. Kueue now reevaluates pending Workloads when the boost changes. This behavior is gated by PriorityBoost. (#19015, @weizhoublue)
  • Scheduling: Fixed a panic in the scheduler when WorkloadRequestUseMergePatch was enabled and a Workload released its quota while Kueue was evicting it, for example for preemption. (#19048, @thc1006)
  • TAS × ElasticJobsViaWorkloadSlices: Fixed a bug where an elastic Job kept unused topology capacity reserved after scaling down, preventing waiting Workloads from using it. Kueue now releases that capacity once the removed Pods terminate. (#18928, @garg02)
  • TAS: Fixed a bug where a ResourceFlavor nodeLabels entry with an empty value (for example key: "") also matched nodes that do not have the label. Nodes now match only if they have the label with exactly that value. (#19101, @kaushik229)
  • TAS: Fixed a bug where an AppWrapper without explicit TAS annotations, admitted to a TAS-only ClusterQueue, was suspended and started again repeatedly. With MultiKueue, the Workload on the worker cluster was finished as OutOfSync and the AppWrapper did not keep running. (#18999, @toruiwasa)
  • TAS: Fixed a bug where replacing a failed node failed, or evicted the Workload, when another node in its topology assignment was cordoned or NotReady while its Pods kept running. This affected clusters with the TASReplaceMultipleFailedNodes feature gate disabled (the default). (#18788, @nsega)
  • TAS: remove every node that a re-placed PodSet group no longer uses from status.unhealthyNodes when the TASReplaceMultipleFailedNodes feature gate is enabled, instead of removing only the first one. (#19052, @saylee21)
  • WorkloadPriorityClass: Fixed a bug where changing a WorkloadPriorityClass value and quickly changing it back could leave Workloads that reference it at the intermediate priority. Kueue now also sets the class value back when the priority of such a Workload is changed directly.
    WorkloadPriorityClass: Added the WorkloadPriorityClassWatchesWorkloads feature gate, Beta and enabled by default. Disable it to keep a priority changed directly on a Workload until its WorkloadPriorityClass is reconciled again. (#18766, @thc1006)
  • Workloads: Fixed a bug that caused Job reconciliation to report an error when the Workload already existed but was not yet visible in the informer cache. Reconciliation now returns success and relies on the Workload creation event to requeue the Job. (#19055, @robfischer1)
  • Workloads: Fixed a bug that caused Kueue to admit a Workload when the summed limits of its Pods exceeded a type: Pod LimitRange max. Now Kueue marks the Workload inadmissible with the LimitRange message, and the Workload does not reserve quota. (#19061, @dundysm)
  • Workloads: Fixed a bug that caused generated Workload and ProvisioningRequest names to contain .- after truncation. The API server rejected these names. Now Kueue removes trailing dots from the truncated name before it appends the hash suffix. (#18959, @burrows99)

Don't miss a new kueue release

NewReleases is sending notifications on new releases.