github kubernetes-sigs/kueue v0.21.0-rc.0

pre-release2 hours ago

Changes since v0.20.0:

Actions Required Before Upgrading

(No, really, you MUST read this before you upgrade)

  • Minor releases: Review the .0 release notes for each new minor version you cross; see: v0.19.0, v0.20.0.

  • CLI: Fixed a bug where kueuectl create clusterqueue added a flavor set only in --borrowing-limit or --lending-limit with a zero nominal quota, so a misspelled flavor name could leave the ClusterQueue inactive. Such flavors are now rejected.

    If you create a borrow-only flavor with only --borrowing-limit, also pass a zero --nominal-quota for it, for example --nominal-quota "alpha:cpu=0" --borrowing-limit "alpha:cpu=4". (#16651, @Eason09053360)

  • ConcurrentAdmission: ClusterQueue manifests omitting spec.concurrentAdmissionPolicy.migration.mode now default to TryPreferredFlavors.

In apis/kueue/v1beta2, the Go field ConcurrentAdmissionMigration.Mode changed from ConcurrentAdmissionMigrationMode to a pointer (*ConcurrentAdmissionMigrationMode). Downstream Go code constructing or manipulating ConcurrentAdmissionMigration must pass a pointer, and callers reading the field should handle nil pointers. (#17012, @Dasmat13)

  • Helm: Added managerConfig.structuredOverrides to customize manager settings without copying the full configuration. With mode omitted, supplied controllerManagerConfigYaml takes precedence, including explicitly empty strings; otherwise, overrides merge with chart defaults.

Optional managerConfig.mode: structured selects structured overrides and ignores YAML. mode: yaml selects supplied YAML or chart defaults and ignores structured overrides. Wrapper charts can leave mode unset to preserve consumers' YAML customizations.

Migration is optional. To migrate with --reuse-values, copy intentional YAML customizations into structuredOverrides and explicitly supply mode: structured in your upgrade values. Keep mode in your values file; reuse retains it when omitted. To switch back, explicitly set mode: yaml and supply your YAML configuration. Without mode, migrate by removing the YAML key and using --reset-values with all desired customizations.

ACTION REQUIRED only in these cases:

  • Your values contain unknown keys directly under managerConfig: remove or correct them. Only mode, structuredOverrides, and controllerManagerConfigYaml are allowed.
  • Your values use invalid inputs: mode must be yaml or structured when supplied; structuredOverrides must be a map and controllerManagerConfigYaml a string. Type validation applies even to unused inputs.
  • Your tools read the default YAML directly from values.yaml: update them to read the rendered manager ConfigMap. Defaults now live in templates/manager/_config.tpl.

Existing users with valid YAML settings do not need to change their configuration or migrate.

Mode does not change Helm's removal behavior. Reuse flags retain omitted saved overrides and can discard null removal instructions. For removals, use --reset-values with all customizations to retain. See the chart README section "Removing overrides during upgrades". (#16312, @prash2512)

  • LeaderWorkerSet: Kueue now rejects spec.groupIdentity: Hash when a LeaderWorkerSet is created under Kueue management or an update brings it under management, for example by adding the kueue.x-k8s.io/queue-name label. Kueue does not support this mode: the leader Pods ran without admission while the worker Pods stayed gated.

If you create LeaderWorkerSets with spec.groupIdentity: Hash that Kueue manages, including through a default LocalQueue or manageJobsWithoutQueueName, use spec.groupIdentity: Ordinal instead, or disable the LeaderWorkerSetRejectGroupIdentityHash feature gate to keep accepting them. If Kueue already manages such a LeaderWorkerSet, delete it and create it again with spec.groupIdentity: Ordinal; until then, its leader Pods keep running without admission. (#18250, @henry3260)

  • Logging: Scheduler cycle progress, scheduling attempts, and workload requeue messages require verbosity 3 or higher.

ACTION REQUIRED
If your observability dashboard (e.g., Grafana Dashboard) relies on those scheduling logs, you have to bump the logging level to v3 or higher. (#17227, @kevin85421)

  • MultiKueue: Kueue now reads ClusterProfile objects as multicluster.x-k8s.io/v1alpha2 instead of the deprecated v1alpha1. Kueue no longer reads the deprecated status.credentialProviders field, and it now applies tls-server-name, insecure-skip-tls-verify and disable-compression from status.accessProviders[].cluster.

If you use the MultiKueueClusterProfile feature gate, upgrade theclusterprofiles.multicluster.x-k8s.io CRD to cluster-inventory-api v0.2.0 or later before upgrading Kueue, and make sure your cluster manager sets status.accessProviders instead of status.credentialProviders. (#17331, @mbobrovskyi)

  • Observability: Reduced log noise from the job reconciler and job webhooks used by all job integrations:
  • Most per-job log records now use verbosity 3 or higher, so they are no longer shown at the default verbosity 2.
  • Most errors are now logged once instead of twice, and reconciliation errors identify the failed operation.
  • Some log keys were renamed, for example oldQueueName and newQueueName to prevLocalQueue and localQueue.

If your dashboards, alerts or log queries depend on Kueue log messages, keys or verbosity levels,
review them before upgrading. For the complete list of logging changes, see #17747.
To keep the job reconciler's per-job messages, most of which moved from level 2 to 3, set the log level to 3 or higher. (#17747, @mimowo)

  • Observability: Updated logging for TAS controllers. Routine per-object reconcile and event logs moved from level 2 to 3, eviction and pod-ungate logs now run after writes succeed, duplicate error logs were removed, and some log keys were renamed (flavor to resourceFlavor, ResourceFlavors to resourceFlavors, podset/podSetName to podSet, domain to domainID, oldReady/oldLevels to previous*).

If your dashboards, alerts, or log queries depend on TAS log messages, keys, or levels, review them before upgrading. Set the log level to 3 or higher to keep per-object reconcile and event logs. (#19044, @JanKaczmarski)

  • Observability: Updated logging for manager startup, DRA, elastic jobs, core controllers (AdmissionCheck, ClusterQueue, ResourceFlavor), and MultiKueue (AdmissionCheck, IncrementalDispatcher). Routine per-object logs moved from level 2 to 3, startup logs moved to level 1, duplicate error logs were removed, and some log keys were renamed.

If your dashboards, alerts, or log queries depend on Kueue log messages, keys, or levels, review them before upgrading (see #19034). Set the log level to 3 or higher to keep per-object reconcile and event logs. (#19034, @JanKaczmarski)

  • Observability: Updated logging for scheduler, flavor-assignment, and preemption. Scheduler per-Workload messages now use verbosity level 3 instead of level 2.

If your dashboards, alerts or log queries depend on Kueue log messages, keys or verbosity levels,
review them before upgrading. For the complete list of logging changes, see #17746.
To keep the scheduler's per-Workload messages, which moved from level 2 to 3, set the log level to 3 or higher. (#17746, @mimowo)

  • Pods: Fixed a bug where a Pod could bypass ClusterQueue quota by setting the kueue.x-k8s.io/role-hash annotation to another role's value and running an oversized Pod under that role's reservation. Kueue now verifies, before removing the scheduling gate, that each Pod of a group fits the effective spec (LimitRange and RuntimeClass adjustments applied) of the PodSet its role hash names; a Pod that does not fit is kept gated and a Warning event is emitted. The gated Pod does not fail the Workload or disturb running peers, but the group cannot become PodsReady until the Pod is replaced by one that fits; with waitForPodsReady, the Workload is evicted and requeued (up to the requeue limit). The check is controlled by the Beta PodIntegrationVerifyRoleRequests feature gate, enabled by default in 0.21. If you set the kueue.x-k8s.io/role-hash annotation yourself and replacement Pods of a group can request more than the Pods the Workload was built from, set PodIntegrationVerifyRoleRequests=false, or recreate the group (delete its Workload and re-create the Pods) so a new Workload is built from the larger Pods. (#13382, @vladikkuzn)
  • RayCluster: Added the Beta RayClusterHeadPodReady feature gate, enabled by default. PodsReady checks now use head readiness so unready workers do not block readiness of a usable head. Disable the gate to restore readiness checks based on the cluster state.

This feature requires KubeRay v1.3.0 or later. If using an older KubeRay version, set RayClusterHeadPodReady=false. (#17591, @kevin85421)

  • RayService: Fixed suspend and resume handling to use the top-level spec.suspend field when the Beta KubeRayServiceUsingTopLevelSuspend feature gate is enabled (default).

If you use the RayService integration, upgrade KubeRay to v1.7.0 or newer before upgrading Kueue. To temporarily retain compatibility with older KubeRay versions, disable KubeRayServiceUsingTopLevelSuspend. (#16870, @kevin85421)

Changes by Kind

Deprecation

  • MultiKueue: Added a feature-gated capacity provider that sums quota from connected worker ClusterQueues for Dynamic Quota Orchestration. Use MultiKueueCapacityProviderConfig; MultiKueueConfig.spec.quotaManagement is deprecated and ignored. Removed the MultiKueueManagerQuotaAutomation feature gate, its ClusterQueue condition, and the multikueue_cluster_status metric.

    If you use MultiKueueManagerQuotaAutomation, remove it from your feature gates before upgrading. To keep the manager quota in sync, enable DynamicQuotaOrchestration and DQOMultiKueueCapacityProvider, then create a MultiKueueCapacityProviderConfig, a CapacityProvider, and a DynamicQuotaOrchestrator for the manager ClusterQueue. (#18382, @JanKaczmarski)

API Change

  • ConfigurablePreemptions: Added preemptorPriorityClassSelector and priorityClassSelector (sourceRef and nameExpressions) to PreemptionConfig, allowing administrators to filter preempting and candidate workloads by WorkloadPriorityClass or Pod PriorityClass. (#16684, @danylott)
  • DQO: when the referenced CapacityProviders orchestrate more than 128 distinct flavors in total, the DQO now sets EffectiveCapacityComputed=False with reason AggregationFailed instead of failing to update its status. (#16767, @ekam-walia)
  • DRA: Add resources.excludeDeviceClassNames to the Kueue Configuration, behind the alpha KueueDRAExcludeDeviceClasses feature gate. Kueue does not charge quota for DRA requests on the listed DeviceClasses, including extended resources they back, and does not reject them for allocationMode: All or a missing mapping. (#18632, @7xuanlu)
  • ProvisioningRequestConfig: Added validation that rejects negative values for retryStrategy.backoffLimitCount, retryStrategy.backoffBaseSeconds and retryStrategy.backoffMaxSeconds. Previously a negative value was accepted and led to invalid requeue delays. (#17545, @nikhil3495)
  • WorkloadLevelWaitForPodsReady: Make TimeoutSeconds an optional field in the workload WaitForPodsReady. (#18446, @MaysaMacedo)

Feature

  • Added a Warning event with reason PriorityClassNotFound when Kueue cannot resolve the scheduling.k8s.io PriorityClass a job names, making it visible why a suspended job stays unqueued with no Workload created. (#15175, @Viswalahiri)
  • Added the kueue_feature_enabled metric, which reports whether each Kueue feature gate is enabled (1) or disabled (0), labeled by gate name and stage. (#15001, @Viswalahiri)
  • CLI: Added cohort (short name co) and topology (short name topo) to the kueuectl pass-through commands get, describe, edit, patch, and delete. (#17708, @Dasmat13)
  • CLI: kueuectl list pods --for pod/NAME now warns when the Pod group is identified by an annotation and the command must list Pods in the group's namespace and filter them client-side. With --all-namespaces, the command now lists only that namespace, so Pods of same-named groups in other namespaces are no longer included. (#18174, @fat-catTW)
  • DRA: Allow one DeviceClass to have separate quota for each consumable capacity dimension. (#17642, @sohankunkerkar)
  • DRA: Graduated consumable-capacity quota to Beta. DeviceClass capacity dimensions can use separate quota resources, and Kueue reports when request rounding increases a charge. (#17636, @sohankunkerkar)
  • DRA: Kueue now emits a Warning event with the error when a Workload's DRA resources cannot be resolved, for example when a DeviceClass is missing from deviceClassMappings. (#16857, @Dasmat13)
  • Deployment: Allow adding or changing the kueue.x-k8s.io/queue-name label on Deployments with ready replicas. The change starts a rolling update, and new Pods wait for admission in the new LocalQueue. (#17575, @Dasmat13)
  • Deployment: Allowed adding the kueue.x-k8s.io/priority-class label to a running Deployment. Removing the label is still rejected. (#18287, @kaushik229)
  • DynamicQuotaOrchestration: Added a built-in kueue.x-k8s.io/local-capacity CapacityProvider, behind the alpha DQOLocalCapacityProvider feature gate, so that quota follows the allocatable resources of the healthy Nodes of each ResourceFlavor. (#16080, @sumukha-radhakrishna)
  • ElasticJobsViaWorkloadSlices: Fixed a bug where elastic Jobs using partial scale-up could lack a minimum-count baseline for later scale-ups. Kueue now sets the initial MinCount to the pod set’s Count. This behavior is gated by elastic partial scale-up. (#16921, @mszadkow)
  • ElasticJobsViaWorkloadSlices: Fixed a bug where elastic Pods ungated for a ProvisioningRequest could carry the consume-provisioning-request annotation without provisioning-class-name when they were created before the job template was stamped. Kueue now applies both annotations, preventing such Pods from being pinned by the cluster autoscaler to nodes that are never created. (#16514, @neilb-dotcom)
  • FairSharing: Promote the FairSharingReevaluatePreemptionCandidates feature gate to Beta and enable it by default in 0.21. (#17285, @JanKaczmarski)
  • Helm and manifests: Added opt-in NetworkPolicies that restrict ingress to the Kueue controller-manager and KueueViz to the ports they serve. Enable them with networkPolicy.enabled=true in Helm, or by applying the new network-policies.yaml release manifest. (#14142, @Anjali-Chauhan1)
  • KEP-975: partial preemption supports Fair Sharing in Alpha (#18851, @zhengchenyu)
  • KueueViz: The workloads dashboard now shows the Pods of pod group Workloads, including StatefulSet and LeaderWorkerSet Workloads. (#16849, @henry3260)
  • KueueViz: add --listen / KUEUEVIZ_LISTEN and --log-level / KUEUEVIZ_LOG_LEVEL backend configuration; keep --port / KUEUEVIZ_PORT for compatibility. (#16522, @asivanadi0)
  • LeaderWorkerSet: Allowed adding tolerations to the Pod templates of a LeaderWorkerSet managed by Kueue, for example to tolerate a new node taint, without recreating it. Kueue copies the added tolerations to the Workloads of the LeaderWorkerSet, including admitted ones, and in MultiKueue to the worker cluster. Removing or changing existing tolerations is still rejected. Controlled by the AllowAddingPodTemplateTolerations feature gate (Beta, enabled by default), which also allows adding tolerations to the PodSets of any Workload that holds a quota reservation. (#17676, @mbobrovskyi)
  • LeaderWorkerSet: Allowed adding, removing and changing the containers and init containers in the Pod templates of a LeaderWorkerSet managed by Kueue without recreating it, when the effective resource requests of the Pods stay the same and no host port is added. In MultiKueue, Kueue now also copies container changes, such as a new image, to the worker cluster. Controlled by the AllowMutatingContainersWhenEffectiveRequestsDontChange feature gate (Beta, enabled by default). (#17903, @mbobrovskyi)
  • Migrate-to-v1beta2.sh: add --resources=LIST to migrate only the listed resource kinds, for example
    --resources=cohorts,localqueues,clusterqueues. Default: all kinds, as before. (#17159, @anguszzzz)
  • Migrate-to-v1beta2.sh: skip objects already annotated with the target storage version (--reannotate to patch them
    anyway), patch in batches (--batch-size, default 100), keep going past per-object errors, and check each CRD's
    storage version first. The script now needs get on customresourcedefinitions. (#16984, @anguszzzz)
  • MultiKueue: Add the alpha feature gate MultiKueueJobSuspendedWorkaroundDisabled (disabled by default). Enable it on manager clusters running Kubernetes 1.36.2 or newer to stop Kueue from setting the JobSuspended condition on suspended local batch/Jobs, a workaround needed only for Kubernetes 1.36.0 and 1.36.1. (#18615, @YevheniiOrlovEngineering)
  • MultiKueue: Added support for RayService in-tree autoscaling on worker clusters using elastic workload slices. Requires the MultiKueueRayInTreeAutoscaling feature gate. (#16259, @kevin85421)
  • MultiKueue: Reduced the delay in dispatching Workloads after a worker cluster reconnects by no longer resyncing finished Workloads that are not being deleted. (#18277, @kevin85421)
  • MultiKueue: Workloads using CEL selectors with count-based deviceClassMappings can reserve quota on the manager without matching local ResourceSlices. Device availability is checked on the worker clusters. (#17916, @ekam-walia)
  • Observability: Added Workload custom metric labels to kueue_cohort_subtree_admitted_workloads_total, allowing the admitted Workloads in a Cohort subtree to be broken down by Workload labels or annotations when the CustomMetricLabels feature gate is enabled. (#16730, @Narwhal-fish)
  • Observability: Added kueue_local_queue_workload_creation_latency_seconds, the Workload creation latency per LocalQueue. Like the other LocalQueue metrics, it follows the LocalQueueMetrics feature gate and the metrics.localQueueMetrics settings. When the CustomMetricLabels feature gate is enabled, the LocalQueue custom labels move from kueue_workload_creation_latency_seconds to the new metric, so dashboards that group the old metric by those labels need the new one. (#18295, @thc1006)
  • Observability: Added the alpha MetricsAccessControl feature gate and metrics.accessControlMode configuration. The default Delegated mode preserves authenticated HTTPS. With the gate enabled, None allows unauthenticated HTTPS metrics scraping on an explicit loopback IP and port for trusted sidecars. (#16035, @ankushbbbr)
  • Observability: Graduate the MetricForWorkloadCreationLatency feature gate to GA. Kueue now reports workload creation latency without requiring the MetricForWorkloadCreationLatency feature gate. (#16493, @mbobrovskyi)
  • Observability: When the CustomMetricLabels feature gate is enabled, admission metrics include configured Workload label and annotation dimensions. (#14577, @Sidd-Prabhu)
  • Pod: Add DeclareWorkloadFailureOnUnretriablePodGroups FG (enabled by default) for marking unretriable PodGroups as Failed. (#18195, @tenzen-y)
  • Scheduling: Improved classical preemption performance by reusing candidate ordering and generating candidates on demand. This behavior is gated by the ClassicalPreemptionCandidateOrderCache feature gate. (#16340, @zhengchenyu)
  • SparkApplication: Added opt-in dynamic executor scaling through ElasticJobsViaWorkloadSlices. This behavior requires the ElasticJobsViaWorkloadSlices and ElasticJobsViaWorkloadSlicesForSparkApplication feature gates, plus the kueue.x-k8s.io/elastic-job: "true" annotation. (#17127, @reruno)
  • SparkApplication: Added partial scale-up for elastic SparkApplications with dynamic allocation. This requires the ElasticJobsViaWorkloadSlicesWithPartialReplicaScaleUp feature gate and the kueue.x-k8s.io/elastic-job-scale-up-strategy: "partial" annotation. Kueue admits at least spec.dynamicAllocation.minExecutors executors, including on the first admission. (#18064, @reruno)
  • TAS: Added a feature-gated scheduler-library integration for determining topology assignments and preemption targets. This behavior is gated by SchedulerLibraryDeepIntegration (#16347, @alien1403)
  • TAS: Graduated TASValidateWorkloadSliceSize to GA. Webhook validation of WorkloadSlice topology fields (podSetSliceSize, podsetSliceRequiredTopologyConstraints) is now always active. (#17167, @mbobrovskyi)
  • TAS: Graduated TASProfileMixed and TASHandleOverlappingFlavors feature gates to GA. (#18193, @JanKaczmarski)
  • TAS: Support workload preemption when SchedulerLibraryDeepIntegration is enabled. (#17681, @kshalot)
  • Validate preferred topology requests when WAS deep integration is enabled (#17419, @yuluo-yx)
  • VisibilityOnDemand: The visibility APIService no longer skips TLS verification. With internal certificate management, Kueue issues a serving certificate for the visibility server during startup and injects its CA into the APIService; cert-manager installations are unchanged. The first upgrade takes longer while the certificate is issued. To downgrade to an earlier version, first run kubectl delete apiservice v1beta2.visibility.kueue.x-k8s.io; the older manifests or chart recreate it. (#16357, @Anjali-Chauhan1)
  • WorkloadLevelWaitForPodsReady: Added the spec.waitForPodsReady field to Workload. Kueue now stores the timeouts from the kueue.x-k8s.io/wait-for-pods-ready annotation in this field instead of in a Workload annotation, and migrates existing Workloads automatically. (#17526, @MaysaMacedo)
  • Workloads: Add the kueue.x-k8s.io/workload annotation to PodTemplates unconditionally. (#15412, @amirialy)

Bug or Regression

  • AdmissionChecks: Fixed retry backoff delaying eviction and unnecessarily retaining quota and topology reservations. (#18116, @garg02)

  • CLI: Fixed a bug that caused kueuectl create clusterqueue to reject quota specifications with hyphenated resource names. Kueue now accepts these resource names. (#16981, @Dasmat13)

  • CLI: Fixed a bug where kubectl-kueue release binaries were built with the .0 Go patch release instead of the latest patch of the Go minor version declared in go.mod. (#18475, @vladikkuzn)

  • CLI: Fixed a bug where kueuectl create clusterqueue failed with "must match the name in coveredResources" when flavors listed the same resources in a different order, for example --nominal-quota "alpha:cpu=1;memory=1,beta:memory=2;cpu=2". (#16531, @Eason09053360)

  • CLI: Fixed a bug where kueuectl create clusterqueue rejected valid quota values, such as cpu=1e-3 or the resource name foo--bar.example.com/gpu, and accepted flavor or resource names that the API server rejects. Errors now name the invalid spec and flag. (#16718, @Eason09053360)

  • CLI: Fixed a bug where kueuectl create resourceflavor --tolerations key=:NoSchedule created an Exists toleration that also matched taints with nonempty values. It now creates an Equal toleration with an empty value. To tolerate any value, omit the = (for example key:NoSchedule). (#17056, @git-jxj)

  • CLI: Fixed a bug where kueuectl delete workload deleted the owning batch/v1 Job but left its Pods running and the Workload in place. Owners are now deleted with background propagation, matching kubectl delete. (#16510, @Eason09053360)

  • CLI: Fixed a bug where kueuectl list workload failed with "resource name may not be empty" when a pending Workload had no spec.queueName. Such Workloads are now listed with empty LocalQueue and ClusterQueue columns. (#16667, @henry3260)

  • CLI: Fixed a bug where kueuectl list commands accepted negative KUEUECTL_LIST_REQUEST_LIMIT values. kueuectl now rejects them with an error; 0 is still accepted and disables pagination. (#17881, @fat-catTW)

  • CLI: Fixed a bug where kueuectl version printed only the client version, without the controller manager image, when Kueue was installed with Helm under a release name other than kueue. (#16609, @Eason09053360)

  • CLI: Fixed a bug where kueuectl list workload --for could list workloads more than once when a paginated job-UID search ended with an empty page. (#16850, @Huangpinhan)

  • CLI: Fixed the usage line of kueuectl delete workload to show that it accepts multiple names, that names and --all are mutually exclusive, and that --all-namespaces works only with --all. (#17875, @mbobrovskyi)

  • CLI: Skip remote ClusterQueue lookup in create localqueue when using --dry-run=client. (#17055, @Dasmat13)

  • ClusterQueue: Fixed a bug where deleting a ClusterQueue could stay blocked by its resource-in-use finalizer when its LocalQueue was deleted before a Workload that the scheduler was admitting. Kueue now rechecks the ClusterQueue when such a Workload is deleted. (#18662, @ivnovakov)

  • Cohort: a ClusterQueue status write no longer reconciles every ClusterQueue in its Cohort tree. ClusterQueues are notified when a Cohort, or a ClusterQueue under it, is created, deleted or changes its spec or quotas. (#17957, @aleksandr-inherent)

  • ConcurrentAdmission: Fixed a bug where Workloads were not cleaned up after concurrentAdmissionPolicy was removed from a ClusterQueue, leaving stale parent/variant Workloads and permanently held quota. (#14954, @NasitSony)

  • ConcurrentAdmission: Fixed a bug where a parent Workload could remain admitted on a flavor removed from its ClusterQueue when a surviving variant was already admitted. The parent is now marked for eviction so it can be re-admitted on a remaining flavor. (#17144, @yuluo-yx)

  • ConcurrentAdmission: Fixed a bug where adding a flavor to a ClusterQueue created a new Variant Workload for each finished Parent Workload in it, which the scheduler could then admit. Kueue no longer creates Variants for a finished or deleting Parent Workload. (#18086, @ivnovakov)

  • ConcurrentAdmission: Fixed a bug where generating Workload variants could change the parent’s metadata or overwrite a sibling variant’s flavor annotation. Each variant now has independent labels and annotations. (#17946, @vshalpnjabi)

  • ConcurrentAdmission: Fixed a bug where pod set updates from admission checks on the admitted Variant workload, such as the ProvisioningRequest annotations, were not applied to the job, leaving its pods unschedulable. (#17176, @akshat-lakhera)

  • ConcurrentAdmission: Fixed a bug where, during migration to a more favorable ResourceFlavor, Kueue evicted the already evicted Variant Workload again on every scheduling cycle. This could keep the Parent Workload from being admitted, and it emitted repeated EvictedDueToFlavorMigration events and kueue_evicted_workloads_total increments. (#18222, @ivnovakov)

  • ConcurrentAdmission: Fixed the ClusterQueue status.pendingWorkloads and the kueue_pending_workloads metric counting a pending Workload once per Variant. A Workload is now counted once. (#18748, @Narwhal-fish)

  • ConcurrentAdmission: Fixed the LocalQueue status.pendingWorkloads and the kueue_local_queue_pending_workloads metric counting a pending Workload once per Variant. The LocalQueue status also no longer counts the Parent Workload after a Kueue restart or an eviction. A Workload is now counted once. (#18806, @Narwhal-fish)

  • ConfigAPI: Fixed a bug where a clientConnection.burst of 0, or a clientConnection.qps of 0, was accepted, so the manager's API requests failed at once, or stopped going through after the first burst requests. With client-side rate limiting enabled, Kueue now rejects a qps of 0 and a non-positive burst at startup; a negative qps still disables rate limiting. (#14442, @thc1006)

  • Configuration: Fixed the admissionFairSharing.usageSamplingInterval and admissionFairSharing.resourceWeights validation errors to report the value at the field they name. (#14445, @thc1006)

  • DQO: Fixed a bug where CapacityProviders that together reported more than 64 distinct resources for a flavor could leave stale effective capacity marked as computed. Kueue now sets EffectiveCapacityComputed=False with reason AggregationFailed and clears status.effectiveCapacity. (#17698, @7xuanlu)

  • DQO: Fixed a bug where a long Distributed condition message (over 32 KiB) caused the API server to reject the DynamicQuotaOrchestrator status update. Kueue now truncates the message. (#18578, @vshalpnjabi)

  • DRA: Fixed a bug where CEL device availability checks counted devices from stale or incomplete ResourceSlice pools, so a Workload could reserve quota for devices the DRA allocator cannot use. (#17649, @junnhwan)

  • DRA: Fixed a bug where Workloads whose device requests come from LimitRange defaults stayed pending after a matching DeviceClass was created. (#17803, @weizhoublue)

  • DRA: Fixed a bug where a Workload updated while the scheduler was processing it could be requeued with quota charges computed for its previous spec, causing over- or under-admission. Kueue now hands such a Workload back to the workload controller to recompute its charges. (#14562, @Anjali-Chauhan1)

  • DRA: Fixed a bug where a Workload whose quota charges were recomputed while the scheduler was processing it, for example after a DeviceClass listed in deviceClassMappings was created or a LimitRange default changed, could be requeued with its old charges and reserve quota with them. (#17388, @digvijay-y)

  • DRA: Fixed a bug where a deviceClassMappings entry with two counter or capacity sources for the same driver and name charged a device's quota twice. Kueue now rejects such entries at configuration validation, and checks every counter source when two mappings name the same DeviceClass. (#18921, @sohankunkerkar)

  • DRA: Fixed a bug where a workload requesting a DeviceClass by its implicit extended resource name deviceclass.resource.kubernetes.io/<class> was treated as an ordinary extended resource, so its quota was not charged to the deviceClassMappings logical resource and the workload could stay pending. (#17141, @henry3260)

  • DRA: Fixed a bug where a workload whose ResourceClaimTemplate names a missing DeviceClass was admitted and held quota while its Pods could never schedule. (#18809, @sohankunkerkar)

  • Deployment, StatefulSet & LeaderWorkerSet: Fixed a bug where, once a WorkloadPriorityClass named default existed, updates to an object with ready replicas and no kueue.x-k8s.io/priority-class label were rejected, even when Kueue did not manage the object. Kueue no longer adds the default WorkloadPriorityClass to such objects on update. (#16474, @henry3260)

  • Deployment, StatefulSet, and LeaderWorkerSet: Fixed a bug where updates to running objects without a queue-name label were rejected after a default LocalQueue was created. Kueue now applies the default only when the queue-name label can be set. (#17865, @tomsen02)

  • DynamicQuotaOrchestrator: Fixed a bug where stale CapacityProvider status could drive effective-capacity computation after a provider specification update. DQO now waits for current-generation synchronization. (#17523, @Om-Beast)

  • DynamicQuotaOrchestrator: Report Distributed=Unknown with reason DistributionIncomplete when quota redistribution fails instead of retaining a stale success condition. (#18552, @Om-Beast)

  • Elastic workloads: Fixed Pods remaining Pending after scale-up when ElasticJobsViaWorkloadSlicesForProvisioningRequests is enabled. ProvisioningRequests belonging to a replaced workload slice are now retained while their capacity bookings remain valid. (#17119, @neilb-dotcom)

  • ElasticJobsViaWorkloadSlices: Added rank-based Pod ungating, prioritizing lower completion indexes for indexed Jobs when capacity is limited. This requires TopologyAwareScheduling to be enabled, but also applies to workloads using non-TAS flavors. (#16704, @mszadkow)

  • ElasticJobsViaWorkloadSlices: Fixed a bug that prevented Kueue from releasing quota when an admitted elastic job using partial scale-up scaled below its previous minimum replica count. (#16742, @weizhoublue)

  • ElasticJobsViaWorkloadSlices: Fixed a bug where WaitForPodsReady could treat an elastic batch Job as not ready while a replacement Workload slice was pending admission. Kueue now evaluates readiness against the granted replica count in the latest admitted slice. (#16754, @mszadkow)

  • ElasticJobsViaWorkloadSlices: Fixed a bug where a ClusterQueue charged both a workload slice and its replacement during an elastic scale-up, so its usage could briefly exceed the quota and block or preempt other workloads. (#18397, @Pradeep39)

  • ElasticJobsViaWorkloadSlices: Fixed a bug where an admission retry could cause a zero-count PodSet to lose a suitable flavor and block later scale-up. Kueue now selects a suitable flavor across retries. (#16466, @prash2512)

  • ElasticJobsViaWorkloadSlices: Fixed a bug where an admitted elastic Workload could be evicted with PodsReadyTimeout while its scale-up Workload slice waited for quota, even after its Pods became ready. Kueue now updates PodsReady on the admitted Workload slice. (#17643, @kevin85421)

  • ElasticJobsViaWorkloadSlices: Fixed a bug where an elastic Workload replacement slice that grows one PodSet and shrinks another could remain pending when the growing PodSet appeared before the shrinking PodSet in spec.podSets. (#16662, @abhayjoshi201)

  • ElasticJobsViaWorkloadSlices: Fixed a bug where resizing an elastic job could let the scheduler admit workloads beyond a cohort's quota in the same scheduling cycle. With ElasticJobsViaWorkloadSlicesWithTAS enabled, the replaced slice's topology usage no longer defers other workloads to the next scheduling cycle. (#16125, @apullo777)

  • ElasticJobsViaWorkloadSlices: Fixed a bug where, with SchedulingEquivalenceHashing enabled, a replacement workload slice that fits could be deferred because another replacement slice with the same requests did not fit. (#17161, @Narwhal-fish)

  • FailureRecovery: Fixed a bug where the KueueFailureRecovery condition and Warning event reported an incorrect grace period for Pods with a non-zero DeletionGracePeriodSeconds. Kueue now reports the correct combined grace period. (#18032, @kshalot)

  • FairSharing: Fix a bug where workloads waiting for their second pass of scheduling (for example a delayed topology assignment) were skipped and requeued with backoff when their ClusterQueue also nominated a pending workload in the same cycle. (#18626, @YevheniiOrlovEngineering)

  • FairSharing: Fixed kueue_cohort_weighted_share to report NaN instead of +Inf when a zero-weight Cohort is borrowing quota, matching its help text and kueue_cluster_queue_weighted_share. (#16581, @abhayjoshi201)

  • FairSharing: Fixed a bug where a ClusterQueue could not reclaim its nominal quota by preemption from Workloads that kept running on a flavor removed from another ClusterQueue's quota. (#18827, @YevheniiOrlovEngineering)

  • FairSharing: Fixed a bug where a ClusterQueue or Cohort whose weighted share exceeded the int64 range could report a negative status.fairSharing.weightedShare. Such a share is now reported as the int64 maximum. (#16487, @thc1006)

  • FairSharing: Fixed a bug where a ClusterQueue or Cohort with weight 0 that borrowed only resources its Cohort had nothing lendable for got a weighted share of 0 instead of the maximum, and was ranked as if it were within its quota. (#16488, @thc1006)

  • FairSharing: Fixed a bug where a borrowing ClusterQueue or Cohort with a very large weight, such as 1e308, could report a status.fairSharing.weightedShare of 0 and be ranked as if it were within its quota. Kueue now keeps such a share above zero: the status reports 1, and the kueue_cluster_queue_weighted_share and kueue_cohort_weighted_share metrics report the smallest positive float64 value. (#16528, @thc1006)

  • FairSharing: Fixed a bug where a workload could be blocked from reclaiming its parent Cohort's nominal quota when that Cohort's DominantResourceShare was inflated by borrowing on an unrelated flavor. The nominal-quota preemption claim is now evaluated at the level where the preemptor's and target's subtrees meet, so descendants of a Cohort get preferential access to the Cohort's own nominal quota, and a ClusterQueue within its own nominal quota no longer preempts across a borrowing parent Cohort. (#13307, @amitaekbote)

  • FairSharing: Fixed a bug where restoring same-ClusterQueue workloads could make cross-ClusterQueue preemption targets violate fair-sharing rules and cause preemption loops. Kueue now validates targets against the final simulated post-admission state. (#16343, @JanKaczmarski)

  • FairSharing: Fixed the preemption bug where workloads within nominal quota were preempted if the ClusterQueue borrowed on an uncontested flavor or within an internal sub-cohort. (#16738, @akshat-lakhera)

  • Fix: Elastic scheduler reuses oversized predecessor grants for shrunken PodSet minimums (#16371, @weizhoublue)

  • Fixed a bug where any RuntimeClass create, update, or delete requeued every pending Workload in the cluster. Kueue now reconciles only pending Workloads that use that RuntimeClass. (#17386, @dlanov)

  • FlavorFungibility: Fixed a bug where a Workload that needed to borrow one resource and preempt for another in the same flavor was never admitted with preference: PreemptionOverBorrowing, because no lower-priority Workloads were preempted. With whenCanPreempt: MayStopSearch, such a flavor also no longer stops the search when a later flavor fits without borrowing. (#18823, @YevheniiOrlovEngineering)

  • Helm: Fix subsequent upgrades after migrating manager configuration with null inputs. (#18139, @prash2512)

  • Helm: Invalid KueueViz authentication modes now fail chart rendering instead of silently disabling authentication. (#17908, @weizhoublue)

  • Importer: Fixed a bug where the Pod importer always used the first ResourceFlavor of a resource group, so a Pod could be imported with a flavor whose node labels or taints it does not match. The importer now uses the first flavor that fits the Pod's node selector, required node affinity and tolerations, and rejects the Pod when none does. (#18977, @burrows99)

  • Job integrations: Fixed a bug where a job created right after its namespace could stay suspended without a Workload until the next informer resync. (#16982, @Narwhal-fish)

  • Job integrations: Fixed a bug where a kueue.x-k8s.io/max-exec-time-seconds label above 2147483647 was accepted by the webhook and then silently ignored, so the job ran with no execution time limit. Such values are now rejected when the job is created. Existing objects are not affected. (#18218, @nikhil3495)

  • Job integrations: Fixed a race where changing a suspended job's queue and workload priority together could expose the new priority in the old queue before the queue change was applied. (#18329, @guming-learning)

  • Job: Fixed a bug where lowering parallelism of a suspended Job below its kueue.x-k8s.io/job-min-parallelism value was accepted, leaving the Job suspended indefinitely because its Workload failed validation. The webhook now rejects updates that lower the Pod count to or below the minimum and reports the accepted range for the annotation correctly. (#16250, @henry3260)

  • JobFramework: Fixed Workloads inheriting Kueue's internal labels and annotations from the Job or Pod when they were listed in labelKeysToCopy or as custom metric sources. (#14227, @thc1006)

  • JobSet: Fixed a bug where an evicted Workload could keep its quota reserved, and was not requeued, if a JobSet restart had recreated child Jobs that had already succeeded. (#17147, @MaysaMacedo)

  • Jobs: Fixed a bug where a Job could adopt an unrelated, already finished prebuilt Workload and run without quota management. Kueue now stops the Job instead. (#16994, @bolubo)

  • KubeRay: Reject elastic RayCluster, RayJob and RayService with required or preferred topology at creation when
    the ElasticJobsViaWorkloadSlicesWithTAS feature gate is enabled. Previously, such jobs were accepted, but their
    Workloads stayed pending. (#18568, @reruno)

  • KueueViz: Fix "undefined" in taint and toleration labels on the ResourceFlavor page. (#18014, @onkar717)

  • KueueViz: Fix YAML dialogs switching resources across namespaces (#17800, @weizhoublue)

  • KueueViz: Fix the LocalQueue detail page missing the Status cell in the workloads table and returning null instead of an empty array for queues with no workloads. (#16392, @Dasmat13)

  • KueueViz: Fix the workload Status column showing the first condition type instead of the workload status. (#17104, @onkar717)

  • KueueViz: Fix the workload detail page not showing the ClusterQueue name and preemption state. (#16279, @onkar717)

  • KueueViz: Fix the workload details page failing when the Workload's LocalQueue was deleted. (#17105, @onkar717)

  • KueueViz: Fixed "Pod Sets Count" on workload pages showing the first pod set's pod count instead of the number of pod sets. (#17538, @onkar717)

  • KueueViz: Fixed ClusterQueue name disclosure in Workload details when users lack LocalQueue read permission. (#17906, @weizhoublue)

  • KueueViz: Fixed Cohorts in a parent cycle missing from the Cohorts tree view. (#17779, @onkar717)

  • KueueViz: Fixed a bug where a backend component failure could prevent graceful HTTP server shutdown. KueueViz now shuts down the server and stops the components before reporting the failure. (#17230, @fat-catTW)

  • KueueViz: Fixed a bug where an unknown KUEUEVIZ_AUTH_MODE value could start the backend without authentication and authorization middleware. KueueViz now exits with an error unless the value is Disabled or TokenReview. (#17143, @fat-catTW)

  • KueueViz: Fixed a bug where distinct user group or extra attribute sets could incorrectly share a cached authorization result. (#17249, @fat-catTW)

  • KueueViz: Fixed a bug where the backend HTTP server could hang indefinitely during graceful shutdown if active HTTP requests did not complete. The server now enforces a 30-second shutdown timeout. (#17135, @fat-catTW)

  • KueueViz: Fixed a bug where the cohorts list exposed ClusterQueue names to users without permission to list ClusterQueues. (#17246, @henry3260)

  • KueueViz: Fixed a bug where the workloads dashboard listed unrelated Pods for Workloads without a kueue.x-k8s.io/job-uid label, such as pod groups. Such Workloads now show an empty Pods list. (#16696, @henry3260)

  • KueueViz: Fixed a bug where the workloads dashboard, cohort, ClusterQueue, and ResourceFlavor websockets could push live updates when a related resource changed, even if the caller lacked cluster-wide watch permission for that resource. Callers without that permission still receive the initial snapshot filtered by their list permissions. (#17207, @HarshithaMS005)

  • KueueViz: Fixed expanding a workload on the dashboard also expanding workloads with the same name in other namespaces. (#17493, @onkar717)

  • KueueViz: Fixed indefinite loading on the Local Queues, Cluster Queues, Cohorts, and Resource Flavors pages when no resources are found. (#16739, @weizhoublue)

  • KueueViz: Fixed links to missing ClusterQueues and Cohorts, such as "N/A" on the dashboard linking to /cluster-queue/undefined. (#17235, @onkar717)

  • KueueViz: Fixed the ResourceFlavor detail page always showing "None" for node labels and node taints. (#17272, @onkar717)

  • KueueViz: Fixed the dashboard tooltip of a preempted workload not linking to the preemptor workload. (#17273, @onkar717)

  • KueueViz: Fixed the workload owner reference always showing "N/A". (#17536, @onkar717)

  • KueueViz: The ClusterQueue page now shows the namespace of each LocalQueue. (#17780, @onkar717)

  • KueueViz: The namespaces websocket now only provides LocalQueue-backed live updates to users with cluster-wide watch access to LocalQueues. Users without that permission still receive the initial namespace snapshot filtered by their list permissions. (#17088, @fat-catTW)

  • LeaderWorkerSet: Fixed a bug where Kueue logged an already exists error and requeued the LeaderWorkerSet when it tried to create a Workload it had already created. (#18783, @henry3260)

  • LeaderWorkerSet: Fixed a bug where changing the kueue.x-k8s.io/priority-class label could give a LeaderWorkerSet's Workloads different priorities. Kueue now resolves the WorkloadPriorityClass once and also updates Workloads that already use the class but have an old priority value. (#13904, @thc1006)

  • LeaderWorkerSet: Fixed a bug where changing the kueue.x-k8s.io/queue-name label while no replica was Ready, but a Workload already held a quota reservation, made the LeaderWorkerSet reconciler fail on every retry and left the Workload in the old queue. Kueue now rejects the change in that case. (#17255, @Andrushika)

  • LocalQueue: Fix an issue where stopping a LocalQueue wiped its cumulative counters and histograms, and deleting a stopped LocalQueue failed to remove its metric series. (#17968, @vic-comm)

  • Manifests: Fixed a bug that prevented the alpha-enabled manifest from installing alpha CRDs and starting the controller manager. The manifest now installs the alpha CRDs and configures compatible feature gates. (#18184, @Fazlul105)

  • MultiKueue & LeaderWorkerSet: Fixed a bug where finishing, preempting, or scaling down one group of a LeaderWorkerSet deleted the LeaderWorkerSet on the worker cluster, stopping and recreating the other groups still running there. Kueue now deletes the remote LeaderWorkerSet only after its last group is removed from the worker. (#16612, @henry3260)

  • MultiKueue: Enabling MultiKueueRayInTreeAutoscaling now requires SkipChildJobSuspension to be enabled. Kueue reports a startup validation error if this dependency is disabled or omitted. (#16470, @kevin85421)

  • MultiKueue: Fix a bug where a Pod that had ephemeral containers added (for example, by kubectl debug) before dispatch was never created on the worker cluster. (#17895, @vladikkuzn)

  • MultiKueue: Fix a bug where a gated manager Pod kept the status copied from the worker Pod after its workload was evicted on the worker and requeued. (#17420, @marina-gonzalez-alvarez)

  • MultiKueue: Fixed a bug where Kueue ignored a ClusterProfile CRD installed after kueue-controller-manager started, so MultiKueueCluster objects that use clusterProfileRef (alpha MultiKueueClusterProfile feature gate) stayed inactive until a restart. (#18404, @yindia)

  • MultiKueue: Fixed a bug where multikueue_workloads_dispatched_total, multikueue_workloads_admitted_total and multikueue_workloads_evicted_total kept reporting a deleted ClusterQueue. (#17423, @henry3260)

  • MultiKueue: Fixed a bug where serveConfigV2 updates to a RayService with in-tree autoscaling were not propagated to the worker cluster when MultiKueueRayInTreeAutoscaling was enabled. (#17464, @kevin85421)

  • MultiKueue: Fixed a bug where a Workload could stay pending forever when no worker cluster was connected at its first nomination, for example while the only worker was reconnecting. Kueue now retries the Workload until a worker cluster is connected. (#18320, @kevin85421)

  • MultiKueue: Fixed a bug where a replaced workload slice could keep holding quota on a worker cluster when the scheduler failed to finish it after admitting the replacement slice. (#16473, @kevin85421)

  • MultiKueue: Fixed a bug where an autoscaling elastic RayCluster that was preempted on the manager during a worker-side scale-up could keep its worker pods running indefinitely. The pods were no longer counted in the worker ClusterQueue usage, while the manager kept holding quota. (#17719, @kevin85421)

  • MultiKueue: Fixed a bug where deleting a manager Job with --cascade=orphan left the remote Job and Workload running on the worker cluster. (#18445, @JanKaczmarski)

  • MultiKueue: Fixed a bug where removing a worker cluster from a MultiKueueConfig re-dispatched the Workloads admitted on that cluster to another cluster without deleting their remote objects, so the same job could run twice. Workloads already admitted on the removed cluster now keep running there until they finish. (#16509, @henry3260)

  • MultiKueue: Fixed a bug where requeueing all of a worker's Workloads after the worker reconnected could delay dispatching new Workloads. Kueue now requeues those Workloads at low priority. (#18407, @kevin85421)

  • MultiKueue: Fixed a bug where restarting the manager could delay dispatching new Workloads until all unfinished Workloads were reconciled. Kueue now queues those Workloads at low priority when the controller starts. (#18949, @kevin85421)

  • MultiKueue: Fixed a bug where the Incremental dispatcher retried deleted Workloads with backoff until Kueue was restarted, logging an error on every retry. (#17458, @Zhuoxi2000)

  • MultiKueue: Fixed a bug where the MultiKueue admission check on the manager stayed Pending with an empty message when no worker could reserve quota. The message now lists the QuotaReserved reason and message reported by each worker cluster, such as DRAResourcesUnresolved. (#18102, @henry3260)

  • MultiKueue: Fixed a bug where the Workload and Job on a worker cluster removed from a MultiKueueConfig kept running after their Workload was evicted and dispatched to another cluster, so the same job could run twice. This fix does not cover a kueue-controller-manager restart before the cleanup completes. (#16516, @henry3260)

  • MultiKueue: Fixed a bug where the The workload was admitted on "<worker>" event was emitted on every reconcile of an admitted Workload. The event is now emitted once, when the MultiKueue admission check becomes Ready. (#17934, @unaimeds)

  • MultiKueue: Fixed a bug where the remote copy of an autoscaling elastic RayCluster could be pointed back at a replaced workload slice after a worker-side scale-up. If the job was then preempted on the manager, its worker pods could keep running while the manager kept holding quota. (#17774, @kevin85421)

  • MultiKueue: Fixed a regression where MultiKueueOrchestratedPreemption, enabled by default since v0.20, added two status writes for every Workload replicated to a worker cluster, reducing MultiKueue admission throughput. status.preemptionGates no longer lists a Closed state for gates that have not been opened; such gates remain closed. (#17157, @YQ-Wang)

  • MultiKueue: Fixed batch/Job re-dispatch eviction loop caused by status.failed counter rejection on Kubernetes 1.35+. (#17059, @mishradwaterlaw)

  • MultiKueue: Fixed manager startup failure when the MultiKueueClusterProfile feature gate is enabled but the ClusterProfile CRD is not installed. (#17228, @yindia)

  • MultiKueue: Immediately resynchronize workloads when a worker watch reconnects instead of waiting for the next polling interval. (#16175, @Kunal241207)

  • MultiKueue: The manager no longer reads remote Workloads from worker clusters when it reconciles an elastic workload slice that was replaced by a newer slice. Many such slices could delay dispatching new Workloads after a manager restart or a worker reconnect. (#18232, @kevin85421)

  • Observability: Changing a custom metric label value on a ClusterQueue, LocalQueue, or Cohort now keeps the object's counter and histogram series under the old value only when every custom label of that kind sets trackedValues. Previously ClusterQueue and LocalQueue always reset them, and Cohort always kept them. (#17260, @Andrushika)

  • Observability: Fix stale pending-workload metrics when LocalQueues are created, resumed, stopped, or deleted, including Workloads observed before their LocalQueue. (#15576, @tomsen02)

  • Observability: Fixed kueue_cohort_subtree_admitted_active_workloads keeping stale custom-label series after a ClusterQueue was relabeled with CustomMetricLabels enabled. The count now moves to the new label series immediately. (#18486, @vshalpnjabi)

  • Observability: Fixed kueue_local_queue_admitted_active_workloads doubling after a ClusterQueue custom label change, and going negative after a LocalQueue label change. (#17336, @Andrushika)

  • Observability: Fixed a bug that caused LocalQueue pending workload metrics to report zero when the associated ClusterQueue did not exist or was deleted. Now Kueue reports these pending workloads as inadmissible and refreshes the metrics when a ClusterQueue is deleted. (#19001, @akshat-lakhera)

  • Observability: Fixed a bug that caused kueue_cohort_subtree_admitted_active_workloads to over-count when CustomMetricLabels was enabled. Each series now counts workloads only from ClusterQueues with matching custom label values. (#18078, @vshalpnjabi)

  • Observability: Fixed a bug where kueue_cluster_queue_resource_reservation, kueue_cluster_queue_resource_usage and kueue_cohort_subtree_resource_reservations kept stale values for a ClusterQueue being deleted while its admitted workloads finished. (#18631, @ivnovakov)

  • Observability: Fixed a bug where kueue_cluster_queue_weighted_share and kueue_replaced_workload_slices_total kept reporting a deleted ClusterQueue, and kueue_cluster_queue_weighted_share kept a stale series with the old cohort label after a ClusterQueue moved to another Cohort. (#16653, @henry3260)

  • Observability: Fixed a bug where kueue_local_queue_admitted_active_workloads and kueue_local_queue_reserving_active_workloads could be missing, and the admitted gauge could later become negative, after a single LocalQueue update changed spec.stopPolicy together with a custom metric label value or a label matched by metrics.localQueueMetrics.localQueueSelector. (#19026, @henry3260)

  • Observability: Fixed a bug where kueue_local_queue_unadmitted_workloads could be reported for a LocalQueue after it was deleted or stopped, and where a Workload outside the queue, such as an inactive one, stayed counted in its old LocalQueue after it moved to a LocalQueue that did not exist. (#18608, @thc1006)

  • Observability: Fixed a bug where a Workload status change while its LocalQueue referenced a deleted ClusterQueue could leave stale unadmitted-workload metric series. Kueue now reports the retained metrics with the recreated ClusterQueue’s labels. (#18221, @akshat-lakhera)

  • Observability: Fixed a bug where custom-label updates to a ClusterQueue or LocalQueue could leave stale metric gauge series in Prometheus. (#17046, @akshat-lakhera)

  • Observability: Fixed a bug where deleting a ClusterQueue could remove the kueue_cohort_subtree_quota and kueue_cohort_subtree_resource_reservations series of its Cohort while other ClusterQueues or child Cohorts remained in it. (#16762, @henry3260)

  • Observability: Fixed a bug where deleting a pending Workload during a scheduling cycle could log a spurious "Could not update Workload status" error and record a Warning event for the deleted Workload. (#18750, @bodyhedia44)

  • Observability: Fixed a bug where the kueue_replaced_workload_slices_total metric was not exposed on the metrics endpoint. (#16743, @Narwhal-fish)

  • Observability: Fixed a bug where the metrics of a deleted LocalQueue, such as kueue_local_queue_evicted_workloads_total, could reappear after the deletion and stay, if Kueue recorded them for a Workload while the LocalQueue was being deleted. (#18325, @thc1006)

  • Observability: Fixed a bug where, with the CustomMetricLabels feature gate enabled and a custom label taken from Workloads, a Workload deleted or relabeled right after admission could keep its stored label values in memory, or leave kueue_admitted_active_workloads and kueue_local_queue_admitted_active_workloads at -1 for its old label values and +1 for the new ones. (#17856, @thc1006)

  • Observability: Fixed stale kueue_pending_workloads series that remained after a ClusterQueue was stopped or resumed when CustomMetricLabels was enabled with a Workload-sourced custom label, which caused pending workloads to be double-counted. (#16670, @kevin85421)

  • Observability: Improved Workload controller logging:

    • Log Workload inadmissibility and recovery from DRA validation errors at verbosity 3, after the corresponding status update succeeds. DRA validation failures no longer produce error logs on every reconcile.
    • Move routine per-Workload messages from verbosity 2 to 3 or higher to reduce log noise.
    • Rename the queue and prevQueue log keys to localQueue and prevLocalQueue, and include the namespace in LocalQueue references.
    • Identify the failed operation in reconciliation errors. If updating the Workload status after a DRA failure also fails, report both errors.

    DRA: Fixed a case where a Workload kept Requeued=True after DRA validation failed again. The controller now sets Requeued=False even if the same validation error is already recorded in the Workload status. (#17093, @mimowo)

  • Observability: Refresh pending-workload metrics when ClusterQueues stop or resume, without relying on subsequent Workload updates. (#16833, @tomsen02)

  • Pod Integration: Fixed a bug where a Pod group that finished because of the kueue.x-k8s.io/retriable-in-group: "false" annotation was marked Finished with reason Succeeded even when some of its Pods failed. Such Workloads now finish with reason Failed unless all Pods succeeded. (#16156, @henry3260)

  • Pod integration: Fixed a bug where a pod group's new Workload could be built from a pod still terminating on a node, pinning it there so it could not be admitted. The fix is on by default through the Beta FinalizeTerminatingPodGroups feature gate. (#17454, @anguszzzz)

  • Pod, AppWrapper & Kubeflow Jobs: Fixed a bug where changing the kueue.x-k8s.io/queue-name label of a suspended job to an invalid LocalQueue name was accepted, so the Workload could not move to the new queue and the job reconciler failed on every retry. Kueue now rejects the update. (#16221, @henry3260)

  • Pod: Avoid counting a terminating Pod's deletion grace period twice when deciding whether its PodGroup still holds quota. (#16076, @gola)

  • Pod: Fixed a bug where a stale single-pod reconcile request for a Pod that joined a pod group after creation, such as a StatefulSet or LeaderWorkerSet Pod, was retried with backoff until the Pod was deleted, repeatedly logging got a single pod reconcile request for a pod group. (#18266, @henry3260)

  • Pod: Fixed a bug where, with SkipReassignmentForPodOwnedWorkloads enabled, a Workload owned by a single Pod that was preempted or evicted due to node failures while its Pod was still scheduling-gated (for example while an AdmissionCheck was pending) stayed parked with Requeued=False forever and its Pod was never deleted. Such Workloads are now requeued, and Workloads already left in that state by an earlier version are requeued on upgrade. (#17053, @nsega)

  • Preemption: Fixed a bug where the capacity reservation made when a preempting Workload finds no candidates could go negative for a ClusterQueue already using more than its nominal quota plus borrowing limit, which could keep another Workload from preempting that ClusterQueue's workloads in the same scheduling cycle. (#14410, @thc1006)

  • PriorityBoost: Fixed a bug where changing the kueue.x-k8s.io/priority-boost annotation on an admitted Workload without a WorkloadPriorityClass didn't requeue pending Workloads. (#17991, @PannagaRao)

  • ProvisioningRequest: Fix a bug where a Pod or pod group with ephemeral containers (for example added with kubectl debug) remained pending when the ClusterQueue had a ProvisioningRequest admission check, because the PodTemplate created for the ProvisioningRequest was rejected. (#17855, @vladikkuzn)

  • ProvisioningRequest: Fix a bug where a Workload stayed Pending on a ProvisioningRequest admission check when a PodSet had a container without an image, such as the KubeRay autoscaler or the RayJob SidecarMode submitter. The ProvisioningRequest PodTemplate now fills missing images with the placeholder image kueue.x-k8s.io/placeholder-image:placeholder, and already stuck Workloads recover without being recreated. (#17859, @vladikkuzn)

  • ProvisioningRequest: Fix a bug where workloads whose container or initContainer volumeMounts reference volumes added at runtime (StatefulSet and LeaderWorkerSet volumeClaimTemplates, JobSet volumeClaimPolicies) remained pending when the ClusterQueue had a ProvisioningRequest admission check, because the PodTemplate created for the ProvisioningRequest was rejected. (#17786, @vladikkuzn)

  • ProvisioningRequest: Fixed a bug where podSetMergePolicy merged PodSets admitted to different ResourceFlavors, so the ProvisioningRequest requested all their capacity on the first PodSet's flavor. Kueue now merges only PodSets assigned to the same flavors. (#16834, @henry3260)

  • ProvisioningRequest: Fixed a bug where one admission check waiting for its ProvisioningRequest could discard the status updates of the Workload's other provisioning checks, while still emitting UpdatedAdmissionCheck events for transitions that were never saved. (#14414, @thc1006)

  • Ray integrations: Fixed a bug where workloads with worker groups using replicas and numOfHosts were not admitted because SubGroupCount and topology index labels were missing from the topology request. The KubeRayEvictOnInconsistentTopologyRequest feature gate is disabled by default, allowing existing workloads to continue running with old topology requests. (#15677, @spencer-p)

  • Ray integrations: Use zero instead of one as the worker replica fallback when replicas is absent, matching the KubeRay CRD default. (#16897, @kevin85421)

  • RayCluster: Fixed a bug where worker PodSet counts could fall below KubeRay’s minimum replica count, leaving workers scheduling-gated when capacity became available. Kueue now respects worker replica bounds when calculating PodSet counts. (#16884, @kevin85421)

  • RayJob: Fixed a bug that allowed elastic RayJobs with required or preferred submitter topology when ElasticJobsViaWorkloadSlicesWithTAS was enabled. Kueue now rejects these RayJobs at creation. (#18613, @weizhoublue)

  • RayService: Fixed webhook validation errors referencing spec.rayClusterSpec instead of the actual API field spec.rayClusterConfig. (#17060, @cryo-zd)

  • Scheduler: Fixed a bug where a PodSet total past the largest Quantity was stored in resourceUsage as MaxInt64. The cache then under-counted that usage, and an even Pod count made the admission webhook reject the Workload on every retry. Kueue now records a total that fits in a Quantity, and sets QuotaReserved=False with reason UnrecordableResourceUsage when the total cannot be stored. (#17195, @HarshithaMS005)

  • Scheduling: Fixed a bug that could delay workload admission after leader failover when the new leader lacked Cohort quota. Kueue now keeps Cohorts in the scheduler cache on every replica. (#17346, @dundysm)

  • Scheduling: Fixed a bug that could prevent ClusterQueues in the same tree from requeueing workloads when quota became available after a Cohort cycle was resolved. (#16438, @henry3260)

  • Scheduling: Fixed a bug that prevented HoldAndDrain from evicting admitted Workloads when their LocalQueue was held or deleted. Kueue now evicts those Workloads when the ClusterQueue enters HoldAndDrain. (#16728, @henry3260)

  • Scheduling: Fixed a bug where Kueue could set Pod node selectors nondeterministically when assigned ResourceFlavors specified conflicting values for the same node label. Kueue now keeps the value from the first flavor in sorted resource-name order. (#16170, @bolubo)

  • Scheduling: Fixed a bug where Workloads blocked by a closed preemption gate could retain a stale QuotaReserved condition and status.resourceRequests. Such Workloads now report the PreemptionGated reason on the QuotaReserved condition instead of AdmissionGated. (#16627, @henry3260)

  • Scheduling: Fixed a bug where a CPU amount past the Quantity limit appeared in quota messages without the milli suffix, so it read as cores. (#16682, @HarshithaMS005)

  • Scheduling: Fixed a bug where a Workload that needed to borrow quota could not preempt Workloads in its own ClusterQueue, even when withinClusterQueue allowed preemption. Kueue now preempts those Workloads. (#16060, @zhengchenyu)

  • Scheduling: Fixed a bug where a Workload whose total request exceeded the int64 range could be admitted against a quota it did not fit. (#16479, @HarshithaMS005)

  • Scheduling: Fixed a bug where a cold-started scheduler could admit workloads onto occupied nodes before existing ClusterQueues and their admitted Workloads were loaded into its cache. Kueue now waits for cache-filling handlers to process their initial data before scheduling. (#16630, @dundysm)

  • Scheduling: Fixed a bug where an inadmissible Workload could stay pending until an unrelated event if Kueue temporarily failed to read its Namespace, for example due to an informer-cache miss. Kueue now retries these Workloads automatically. (#18592, @Om-Beast)

  • Scheduling: Fixed a bug where changing the boost on an admitted Workload with a Pod PriorityClass did not trigger reevaluation of pending Workloads. Kueue now reevaluates pending Workloads when the boost changes. This behavior is gated by PriorityBoost. (#18127, @weizhoublue)

  • Scheduling: Fixed a bug where every Workload event built the Workload's effective resources and scheduling hash for the DRA check even when no DeviceClass backs an extended resource, and compared whole PodSpecs to detect resource defaults, adding avoidable CPU overhead to the Workload controller. (#17424, @sohankunkerkar)

  • Scheduling: Fixed a bug where fair-sharing preemption could repeatedly evict and re-admit Workloads when a preemptor was within nominal quota for one requested resource but still borrowing another. (#17034, @fat-catTW)

  • Scheduling: Fixed a bug where partial admission of a PodSet whose total request overflowed int64 could admit more Pods than the quota allows, because the reduced request was derived from the saturated total instead of the per-Pod request. (#14520, @thc1006)

  • Scheduling: Fixed a bug where the scheduler cache could drop an admitted Workload when its admission write was committed but the client observed an error, such as a timeout. This left the ClusterQueue's usage under-counted until the Workload's next update. (#17041, @kaushik229)

  • Scheduling: Fixed a panic in the scheduler when WorkloadRequestUseMergePatch was enabled and a Workload released its quota while Kueue was evicting it, for example for preemption. (#18338, @thc1006)

  • SparkApplication: Fixed PodsReady evicting running applications whose executors the Spark Operator no longer tracks in status.executorState, and counting completed executors as ready. Kueue now counts the executor Pods that are running and ready. (#18547, @Pradeep39)

  • SparkApplication: Fixed a bug where a SparkApplication without spec.executor.instances could not be suspended when its Workload was evicted or deactivated, because Kueue wrote back an invalid instances: 0. (#15610, @henry3260)

  • StatefulSet: Fixed a bug where changing a StatefulSet’s queue while its Workload held quota could cause repeated Workload update failures when the StatefulSet had no ready replicas. Kueue now rejects the queue change until the Workload releases its quota. (#16320, @weizhoublue)

  • StatefulSet: Fixed a bug where labels listed in integrations.labelKeysToCopy, and the labels and annotations used as CustomMetricLabels sources, were not copied from a StatefulSet into its Workload. (#16187, @nsega)

  • StatefulSet: Fixed a bug where scaling a StatefulSet to zero released its Workload's quota while its Pods were still running, which could let other Workloads be admitted onto resources still in use. (#16044, @gola)

  • TAS × ElasticJobsViaWorkloadSlices: Fixed a bug where an elastic Job kept unused topology capacity reserved after scaling down, preventing waiting Workloads from using it. Kueue now releases that capacity once the removed Pods terminate. (#17374, @garg02)

  • TAS: Bound Pods whose Workload no longer has an active reservation continue to count against node capacity until the Pods terminate or are deleted. (#16339, @gola)

  • TAS: Fix a workload in its second pass (node replacement with TASFailedNodeReplacementFailFast disabled, or delayed topology assignment after a ProvisioningRequest) not finding preemption targets when quota is tight, because its quota was counted twice. (#18598, @kevinmingtarja)

  • TAS: Fix feasible elastic Workload slice replacements being rejected during preemption because the predecessor's topology usage was counted twice. (#17460, @gola)

  • TAS: Fixed a Pod group never being admitted when the kueue.x-k8s.io/pod-group-pod-index-label annotation named a label that does not hold the Pod's index. Such a Pod is now refused at creation; the new TASRejectInvalidPodIndexLabel feature gate disables the refusal. (#15867, @rjgoyln)

  • TAS: Fixed a bug that could prevent grouped PyTorchJob and JobSet pods submitted through AppWrapper from receiving index-based topology placement when no pod index offset was set. Kueue now applies the default offset. (#17801, @weizhoublue)

  • TAS: Fixed a bug where Jobs that request a topology, and whose Workloads were admitted while the TopologyAwareScheduling feature gate was disabled, were stopped and requeued after the gate was enabled. (#18503, @kshalot)

  • TAS: Fixed a bug where Pods in a PodSet group whose indexes start from 0, such as JobSet or Ray workers, were ungated without rank ordering, which could split a kueue.x-k8s.io/podset-slice-required-topology slice across topology domains. Kueue now sets kueue.x-k8s.io/pod-index-offset: "0" on grouped JobSet and Ray Pod templates when it starts the job, which replaces the PodSet group offset. This applies to workloads admitted after the upgrade. (#17289, @henry3260)

  • TAS: Fixed a bug where Pods of Kubeflow Jobs (v1), such as PyTorchJob, and of MPIJobs without runLauncherAsWorker in a kueue.x-k8s.io/podset-group-name PodSet group were ungated without rank ordering, and the topology ungater logged an error on every reconcile. Kueue now sets kueue.x-k8s.io/pod-index-offset: "0" on their grouped Pod templates when it starts the job. This applies to workloads admitted after the upgrade. (#17689, @henry3260)

  • TAS: Fixed a bug where Pods of a TrainJob in a kueue.x-k8s.io/podset-group-name PodSet group were ungated without rank ordering, and the topology ungater logged an error on every reconcile. Kueue now sets kueue.x-k8s.io/pod-index-offset: "0" on these Pods when it starts the TrainJob. This applies to workloads admitted after the upgrade. (#17862, @henry3260)

  • TAS: Fixed a bug where a ResourceFlavor nodeLabels entry with an empty value (for example key: "") also matched nodes that do not have the label. Nodes now match only if they have the label with exactly that value. (#18500, @kaushik229)

  • TAS: Fixed a bug where a Workload could be admitted onto a node whose Pod slots had run out earlier in the same scheduling cycle, or onto a node or topology domain with room for only some of its PodSets, leaving Pods Pending. In a ClusterQueue with a pods quota, a PodSet whose Pods request no resources can now be admitted. (#14443, @thc1006)

  • TAS: Fixed a bug where a Workload using delayed topology assignment or failed node replacement could lose its quota reservation, and for a failed node replacement also its admission. (#16462, @apullo777)

  • TAS: Fixed a bug where a Workload using delayed topology assignment or failed node replacement could miss a scheduling retry until it was updated again. (#16468, @apullo777)

  • TAS: Fixed a bug where a Workload with multiple PodSets reported an empty (Pending) QuotaReserved reason when one PodSet needed preemption for quota and another failed topology placement even on an empty cluster. The topology failure is now reported as TopologyPlacementFailed. (#17398, @chengjoey)

  • TAS: Fixed a bug where a failed-node replacement workload was evicted immediately even when in-flight preemptions were about to free a suitable topology placement; it is now deferred until the preemptions complete, falling back to the eviction when no placement becomes available. (#15263, @bolubo)

  • TAS: Fixed a bug where a second scheduling pass (failed-node replacement or delayed topology assignment) computed from an outdated Workload could overwrite a newer admission or drop status updated in the meantime, such as an Evicted condition. The scheduler now rejects the stale write, keeps the Workload's reservation, emits an OutdatedScheduleCycle event and retries the second pass. (#15761, @anguszzzz)

  • TAS: Fixed a bug where a workload with non-adjacent PodSet group members (for example, a JobSet) could start with another PodSet's labels and node selectors, leaving some Pods scheduling-gated. (#17052, @apullo777)

  • TAS: Fixed a bug where an AppWrapper without explicit TAS annotations, admitted to a TAS-only ClusterQueue, was suspended and started again repeatedly. With MultiKueue, the Workload on the worker cluster was finished as OutOfSync and the AppWrapper did not keep running. (#18600, @toruiwasa)

  • TAS: Fixed a bug where an MPIJob with runLauncherAsWorker that set kueue.x-k8s.io/pod-index-offset alongside kueue.x-k8s.io/podset-group-name was admitted but never started, while holding quota. Kueue now accepts "1", which matches the offset implied by the PodSet group, and rejects other values; remove the annotation from existing MPIJobs that use another value. (#17640, @henry3260)

  • TAS: Fixed a bug where an elastic Workload with a TAS PodSet group could reassign its running leader to a different domain when the workers scaled up from zero, with the ElasticJobsViaWorkloadSlicesWithTAS feature gate enabled. (#16776, @Narwhal-fish)

  • TAS: Fixed a bug where preferred node affinity could be ignored, placing Pods on a less preferred domain, when TASCacheNodeMatchResults reused cached node matches. This affects users who enabled TASRespectNodeAffinityPreferred. (#18215, @adibmbrk)

  • TAS: Fixed a bug where replacing a failed node could move a Pod of a PodSet group, such as a LeaderWorkerSet leader, to a different topology domain than the rest of its group. Kueue now keeps the group in its required domain, or evicts the Workload when the domain has no room for the replacement. (#17515, @Narwhal-fish)

  • TAS: Fixed a bug where replacing a failed node failed, or evicted the Workload, when another node in its topology assignment was cordoned or NotReady while its Pods kept running. This affected clusters with the TASReplaceMultipleFailedNodes feature gate disabled (the default). (#18123, @nsega)

  • TAS: Fixed a bug where replacing a failed node never completed for a Workload with some Pods already succeeded, leaving the recreated Pod gated. Kueue now keeps the admitted Pod count and resource usage when it updates the topology assignment. (#17375, @Narwhal-fish)

  • TAS: Fixed a bug where stuck pods could break rank-based pod ungating after node replacement or workload re-admission. (#16788, @pajakd)

  • TAS: Fixed a bug where the Active condition message of a ClusterQueue listed TAS flavors without a Topology in a random order, which could cause repeated status updates of the ClusterQueue. The flavors are now listed in name order. (#17392, @spicysharkco)

  • TAS: Fixed a bug where the pending message of a Workload with several PodSets that could not be placed could name a different PodSet in each scheduling cycle. Kueue now reports the first failed PodSet in alphabetical order. (#17579, @henry3260)

  • TAS: Fixed a bug where waiting Workloads could remain inadmissible after a deactivated Workload's Pods finished terminating when SchedulerLibraryIntegration was enabled. Kueue now requeues them for admission. (#17581, @kshalot)

  • TAS: Fixed a bug where workloads placed below the root topology level (for example, required or preferred at rack or node level, or unconstrained) broke ties between equal-capacity domains alphabetically instead of preferring smaller or already partially used parent blocks, fragmenting larger blocks needed by subsequent workloads. Controlled by the TASAncestorCapacityTieBreak feature gate (Beta, enabled by default). (#18516, @HristoMohamed)

  • TAS: Fixed a bug where, with TASRespectNodeAffinityPreferred enabled, a node that could not run the Pod still added its preferred-affinity score to its topology domain, so TAS could choose a less preferred node. (#18454, @sohankunkerkar)

  • TAS: Fixed a bug where, with ElasticJobsViaWorkloadSlicesWithTAS enabled, Pods left after an elastic Job scale-down could be ungated into topology domains assigned to Pods at lower ranks. Kueue also no longer logs the failed to read rank information from pods error on every reconcile of a TAS Indexed Job with completions above parallelism. (#16771, @ivnovakov)

  • TAS: Fixed a bug with FlavorFungibilityPreserveScanProgress enabled where workloads using TryNextFlavor could repeatedly retry a topology-blocked ResourceFlavor or skip viable borrowing flavors across scheduling cycles when all flavors required borrowing or when a later flavor had nominal quota. (#16766, @j-skiba)

  • TAS: Fixed a scheduler panic when an elastic Workload scaled the workers of a PodSet group from 0 to the same count as the leader, with the ElasticJobsViaWorkloadSlicesWithTAS feature gate enabled. (#16740, @Narwhal-fish)

  • TAS: Fixed workloads with non-adjacent PodSet group members getting stuck during flavor-scan resume or delayed/failed-node topology reassignment. (#16658, @apullo777)

  • TAS: Improved required-topology placement failure messages to identify the required level and explain that all Pods must fit in a single topology domain. (#17048, @HasonoCell)

  • TAS: With KueueDRADeviceFeasibility enabled, fixed a bug where PodSets of one Workload could be placed on a node whose device plugin publishes fewer devices of a DRA-backed extended resource than they need together, leaving Pods Pending. The same miscount could keep an elastic scale up pending. (#16625, @thc1006)

  • TAS: remove every node that a re-placed PodSet group no longer uses from status.unhealthyNodes when the TASReplaceMultipleFailedNodes feature gate is enabled, instead of removing only the first one. (#18118, @saylee21)

  • TrainJob: Fixed a bug where a TrainJob was never admitted when a Workload with the same name as the TrainJob existed in the same namespace. (#17563, @henry3260)

  • WaitForPodsReady: Fixed a bug where a very large timeoutSeconds or recoveryTimeoutSeconds in the kueue.x-k8s.io/wait-for-pods-ready annotation could overflow into a small timeout and pass validation. Such values are now rejected for exceeding waitForPodsReady.maxTimeoutOnWorkload. (#16971, @nikhil3495)

  • Workload: Fixed a bug where a Workload that already stored a negative resource request or limit, or a pods entry in its limits or pod-level resources, could not be evicted, deactivated or deleted and stayed in Terminating. Kueue now accepts updates that keep such a value unchanged and still rejects new or changed ones. (#15235, @roshpr)

  • Workload: Fixed the validation message shown when changing spec.maximumExecutionTimeSeconds on an admitted Workload to say "immutable while the workload is admitted" instead of "while workload quota reserved". (#14449, @thc1006)

  • Workload: Reject negative Pod overhead when WorkloadValidateResourcesAreNonNegative is enabled. (#17090, @nareshpatel1015)

  • WorkloadPriorityClass: Added the WorkloadPriorityClassWatchesWorkloads feature gate, Beta and enabled by default. Disable it to keep a priority changed directly on a Workload until its WorkloadPriorityClass is reconciled again. (#18758, @thc1006)

  • WorkloadPriorityClass: Fixed a bug where changing a WorkloadPriorityClass value and quickly changing it back could leave Workloads that reference it at the intermediate priority. Kueue now also sets the class value back when the priority of such a Workload is changed directly. (#16459, @thc1006)

  • Workloads now account for the higher of their declared Pod overhead and the RuntimeClass overhead. A RuntimeClass mismatch between a suspended Job and its prebuilt Workload is treated as out of sync; already admitted Jobs that may still have running Pods are not stopped by this new comparison during an upgrade. (#17588, @sheldon-lang)

  • Workloads: Fixed a bug that caused Job reconciliation to report an error when the Workload already existed but was not yet visible in the informer cache. Reconciliation now returns success and relies on the Workload creation event to requeue the Job. (#18010, @robfischer1)

  • Workloads: Fixed a bug that caused Kueue to admit a Workload when the summed limits of its Pods exceeded a type: Pod LimitRange max. Now Kueue marks the Workload inadmissible with the LimitRange message, and the Workload does not reserve quota. (#18723, @dundysm)

  • Workloads: Fixed a bug that caused generated Workload and ProvisioningRequest names to contain .- after truncation. The API server rejected these names. Now Kueue removes trailing dots from the truncated name before it appends the hash suffix. (#18803, @burrows99)

Other (Cleanup or Flake)

  • API: Kueue no longer provides the kueue.x-k8s.io/v1beta1 APIs. (#16485, @mbobrovskyi)
  • AgentSkills: Do not require the acknowledgment that the skills are experimental. (#16860, @mimowo)
  • CLI: Improved the invalid resource group error from kueuectl create clusterqueue to name the conflicting flavors and the resources each one covers. (#16632, @Eason09053360)
  • ConfigAPI: Kueue logs configured internal labels in integrations.labelKeysToCopy at startup. Kueue does not copy these labels to a Workload. (#16779, @thc1006)
  • DRA: Kueue now excludes invalid ResourceSlice pools from device matching and quota calculations, matching the DRA allocator’s pool validation. (#18133, @yindia)
  • Graduated RemoveFinalizersWithStrictPatch to GA. (#18176, @JanKaczmarski)
  • Graduated ValidateRayAndSparkJobUpdates feature gate to GA. (#18038, @JanKaczmarski)
  • Graduated AssignQueueLabelsForPods to GA. (#18180, @JanKaczmarski)
  • Observability: Improved the performance of LocalQueue pending-workload metrics by avoiding a scan of all pending workloads in the ClusterQueue on each queue change. (#17414, @sohankunkerkar)
  • Observability: Improved the performance of LocalQueue pending-workload metrics with Workload custom metric labels by avoiding a scan of all pending workloads in the ClusterQueue on each queue change. (#17448, @sohankunkerkar)
  • Scheduling: Reduced the CPU cost of Workload events by computing the scheduling equivalence hash only for Workloads in the queue. (#17461, @sohankunkerkar)
  • StatefulSet: introduced the DisableStatefulSetWorkloadNameFallback feature gate (Beta, enabled by default) to disable fallback lookup for legacy workload names without owner UID. (#17937, @Dasmat13)

Don't miss a new kueue release

NewReleases is sending notifications on new releases.