kubernetes-sigs/kueue v0.13.0-rc.0

on GitHub

Changes since v0.12.0:

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Helm:

  • Fixed KueueViz installation when enableKueueViz=true is used with default values for the image specifying parameters.
  • Split the image specifying parameters into separate repository and tag, both for KueueViz backend and frontend.

  If you are using Helm charts and installing KueueViz using custom images,
  then you need to specify them by kueueViz.backend.image.repository, kueueViz.backend.image.tag,
  kueueViz.fontend.image.repository and kueueViz.frontend.image.tag parameters. (#5400, @mbobrovskyi)

• TAS: Drop support for MostFreeCapacity mode

The TASProfileMostFreeCapacity feature gate is no longer available.
If you specify that, you must remove it from the .featureGates in your Kueue Config or kueue-controller-manager command-line flag, --feature-gates. (#5536, @lchrzaszcz)

• The API Priority and Fairness configuration for the visibility endpoint is installed by default.

If your cluster is using k8s 1.28 or older, you will need to either update your version of k8s (to 1.29+) or remove the FlowSchema and PriorityLevelConfiguration from the installation manifests of Kueue. (#5043, @mbobrovskyi)

Changes by Kind

Deprecation

• Promote Cohort CRD version to v1beta1

  The Cohort CRD v1alpha is no longer supported.
  The .spec.parent in Cohort v1alpha1 was replaced with .spec.parentName in Cohort v1beta1. (#5595, @tenzen-y)

Feature

• AdmissionFairSharing: preemption candidates are now ordered within ClusterQueue with respect to LQ's usage.
  The ordering of candidates coming from other ClusterQueues is unchanged. (#5632, @PBundyra)
• Allow setting the controller-manager's Pod PriorityClassName from the Helm chart (#5631, @kaisoz)
• Helm: support for specifying nodeSelector and tolerations for all Kueue components (#5820, @zmalik)
• Introduce two-level TAS scheduling (#5353, @lchrzaszcz)
• TAS: Increase the maximal number of Topology Levels (.spec.levels) from 8 to 16. (#5635, @sohankunkerkar)
• [TAS] Add FailFast mode on Node's failure (#5861, @PBundyra)

Bug or Regression

• Add Cohort Go client library (#5597, @tenzen-y)

• Fix a bug that would allow a user to bypass localQueueDefaulting. (#5451, @dgrove-oss)

• Fix a bug where the GroupKindConcurrency in Kueue Config is not propagated to the controllers (#5818, @tenzen-y)

• Fix the bug that Kueue, upon startup, would incorrectly admit and then immediately deactivate
  already deactivated Workloads.

  This bug also prevented the ObjectRetentionPolicies feature from deleting Workloads
  that were deactivated by Kueue before the feature was enabled. (#5625, @mbobrovskyi)

• Fix the bug that the webhook certificate setting under controllerManager.webhook.certDir was ignored by the internal cert manager, effectively always defaulting to /tmp/k8s-webhook-server/serving-certs. (#5432, @ichekrygin)

• Fixed bug that doesn't allow Kueue to admit Workload after queue-name label set. (#5047, @mbobrovskyi)

• Helm: Fix a templating bug when configuring managedJobsNamespaceSelector. (#5393, @mtparet)

• MultiKueue: Fix a bug that batch/v1 Job final state is not synced from Workload cluster to Management cluster when disabling the MultiKueueBatchJobWithManagedBy feature gate. (#5615, @ichekrygin)

• MultiKueue: Fix the bug that Job deleted on the manager cluster didn't trigger deletion of pods on the worker cluster. (#5484, @ichekrygin)

• TAS: Fix a bug for the incompatible NodeFailureController name with Prometheus (#5819, @tenzen-y)

• TAS: Fix a bug that Kueue unintentionally gives up a workload scheduling in LeastFreeCapacity if there is at least one unmatched domain. (#5803, @PBundyra)

• TAS: Fix a bug that LeastFreeCapacity Algorithm does not respect level ordering (#5464, @tenzen-y)

• TAS: Fix a bug that the tas-node-failure-controller unexpectedly is started under the HA mode even though the replica is not the leader. (#5848, @tenzen-y)

• TAS: Fix bug which prevented admitting any workloads if the first resource flavor is reservation, and the fallback is using ProvisioningRequest. (#5426, @mimowo)

• TAS: Fix the bug when Kueue crashes if the preemption target, due to quota, is using a node which is already deleted. (#5833, @mimowo)

• TAS: fix the bug which would trigger unnecessary second pass scheduling for nodeToReplace
  in the following scenarios:

  1. Finished workload
  2. Evicted workload
  3. node to replace is not present in the workload's TopologyAssignment domains (#5585, @mimowo)

• TAS: fix the scenario when deleted workload still lives in the cache. (#5587, @mimowo)

• Use simulation of preemption for more accurate flavor assignment.
  In particular, in certain scenarios when preemption while borrowing is enabled,
  the previous heuristic would wrongly state that preemption was possible. (#5529, @pajakd)

• Use simulation of preemption for more accurate flavor assignment.
  In particular, the previous heuristic would wrongly state that preemption
  in a flavor was possible even if no preemption candidates could be found.

  Additionally, in scenarios when preemption while borrowing is enabled,
  the flavor in which reclaim is possible is preferred over flavor where
  priority-based preemption is required. This is consistent with prioritizing
  flavors when preemption without borrowing is used. (#5698, @gabesaba)

Other (Cleanup or Flake)

• KueueViz: reduce the image size from 1.14 GB to 267MB, resulting in faster pull and shorter startup time. (#5860, @mbobrovskyi)
• RBAC permissions for the Cohort API to update & read by admins are now created out of the box. (#5431, @vladikkuzn)
• Support for Kubernetes 1.33 (#5123, @mbobrovskyi)