volcano-sh/volcano v1.16.0

on GitHub

Summary

Volcano v1.16.0 introduces NamespaceQueue and the fairshare plugin to support different tenancy models. Tenants can manage queue hierarchies and quotas within their own namespaces, using administrator-authorized resource pools. For tenants isolated by namespace and sharing a queue, fairshare adds Slurm-inspired historical usage decay to improve long-term fairness.

This release also broadens the HyperNode ecosystem. Other schedulers can reuse topology discovery and supernode management through the independently deployable HyperNode controller and its API. PodGroup topology anti-affinity separates workload instances across topology domains. Scheduler performance improvements include the Unschedulable Job Cache, shared node traversal for scoring plugins, and lower overhead in cache updates and queue lookups. Additional changes include load estimation for bulk Pod dispatch, Kubernetes 1.36 support, DRA scoring, queue-level quotas per nodegroup, accelerator enhancements, and stability fixes.

What's New

Key Features Overview

• Scheduler Performance and Scalability: Reduces backlog retries with an unschedulable Job cache and improves node scoring, cache updates, and queue lookups.
• Namespace-scoped Queues: Lets tenants configure queue hierarchies and quotas within their namespaces, using administrator-authorized resource pools.
• Per-Namespace Fair Share Scheduling: Applies Slurm-inspired usage decay to improve long-term fairness among tenants isolated by namespace and sharing a queue.
• PodGroup Topology Anti-Affinity: Separates workload instances across HyperNode domains with required or preferred placement rules.
• Broader HyperNode Integration with Other Schedulers: Makes topology discovery and supernode management available through a standalone controller and the HyperNode API.

Key Feature Details

Scheduler Performance and Scalability

Background and Motivation:

v1.16.0 introduces the Unschedulable Job Cache to improve scheduling under large pending backlogs, alongside improvements to node scoring, cache updates, and queue lookups.

Unschedulable Job Cache

Pending Jobs may remain blocked by queue quotas, insufficient resources, or placement constraints across many scheduling sessions. Repeating the same checks in every session consumes time that could be used to schedule other Jobs.

The UnschedulableJobCache records why a Job could not be scheduled and uses plugin-provided event hints to decide when it should be evaluated again. Later sessions skip the checks covered by the cached result until a relevant change occurs. This reduces redundant work and improves scheduling performance in clusters with persistent unschedulable backlogs.

Key Capabilities:

• Selective skipping: Uses recorded rejection causes to skip enqueue checks, whole-Job evaluation, or checks for specific tasks while respecting gang requirements.
• Event-driven wakeups: Retries Jobs when relevant conditions change. For example, a Node label update can wake a Job blocked by node affinity, resource release can wake a resource-blocked Job, and a Queue update can wake a Job blocked by quota. Storage and DRA events are also supported through plugin hints.
• Safe fallback: Caches a Job only when every rejecting plugin provides a safe event hint. Other Jobs continue through normal per-session evaluation.
• Gang and fairness preservation: Keeps cached Jobs visible to pending demand, queue accounting, and gang requirements. All normal scheduling checks run again before placement after a wakeup.
• Bounded retries: A configurable watchdog retries cached Jobs after a maximum skip duration, defaulting to five minutes. The cache is in memory and is cleared on scheduler restart.

Configuration:

The feature is disabled by default. Enable it with the scheduler feature gate:

helm upgrade --install volcano volcano-sh/volcano \
  --namespace volcano-system --create-namespace --version 1.16.0 \
  --set custom.scheduler_feature_gates="UnschedulableJobCache=true"

Initial coverage includes the upstream predicates integrated by Volcano, native Resource Fit, proportion, and non-hierarchical capacity. Hierarchical capacity rejections and rejections from plugins without hint providers continue to be evaluated each session.

Optional debug metrics report cache skips, event wakeups, and watchdog expirations. They are disabled by default because they carry Job identity labels. Enable them when needed with custom.scheduler_unschedulable_job_cache_debug_metrics=true and scheduler metrics enabled.

Related:

• Issue: #5551
• PRs and contributor: #5848, #5644, @JesseStutler
• Design and performance results: Unschedulable Job Cache
• User guide: Enable the Unschedulable Job Cache

Multiple scoring plugins can now score nodes in a shared traversal, reducing scoring overhead when several plugins are active. Node updates and queue lookups also avoid unnecessary object copying and traversal. (#5928, #5556, #5506)

Volcano will continue to improve performance and scalability. Future work will explore scheduling efficiency, cache and event processing, and status updates, guided by workload benchmarks and production feedback.

Namespace-scoped Queues

Background and Motivation:

Platforms use different tenancy models. Some assign resource pools to tenants through cluster-scoped Queues. Others use namespaces as the management boundary for teams or projects and want tenants to configure queue quotas and hierarchies within that boundary. NamespaceQueue supports this namespace-based model, aligning queue management with tenant ownership.

NamespaceQueue extends hierarchical queue management to namespaces while preserving administrator control over resource-pool authorization. Tenants can create parent/child queues and configure resource attributes within their namespaces. The resource integrates with the scheduler's common queue model, allowing cluster Queues and NamespaceQueues to participate in the same resource hierarchy.

Key Capabilities:

• A namespaced NamespaceQueue CRD with capability, deserved, guarantee, priority, reclaimability, and dequeue strategy settings.
• Namespace-local parent/child hierarchies, with cluster/<name> references for cluster Queue parents.
• Administrator authorization through cluster Queue spec.allowedNamespaces. Omitted or empty lists deny NamespaceQueue attachment, and ["*"] allows all namespaces.
• Explicit workload references through spec.queue: namespace/<name>, resolved within the workload's namespace. Bare queue names continue to reference cluster Queues.
• Controller-maintained authorization and readiness conditions, with final scheduler readiness checks before admission.

Configuration:

The feature is disabled by default. The Helm switch enables NamespaceQueue=true for the controller manager, scheduler, and admission webhook and selects the hierarchical capacity configuration when using the chart's default scheduler configuration. Custom scheduler configurations must enable capacity with enableHierarchy: true. Initial NamespaceQueue support is limited to the capacity plugin. Do not enable proportion alongside it. NamespaceQueue hierarchies allow five levels below a cluster Queue by default. Set --max-namespacequeue-depth on both the controller manager and admission webhook to change that limit.

helm upgrade --install volcano volcano-sh/volcano \
  --namespace volcano-system --create-namespace --version 1.16.0 \
  --set custom.namespace_queue_enable=true

Create the namespace and an authorized parent Queue first, grant tenants namespace-local RBAC permissions for namespacequeues, then create the NamespaceQueue. Wait for Ready=True and confirm Authorized=True before submitting workloads:

apiVersion: v1
kind: Namespace
metadata:
  name: team-a
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: research
spec:
  capability:
    cpu: "100"
    memory: "200Gi"
  deserved:
    cpu: "100"
    memory: "200Gi"
  allowedNamespaces:
  - team-a
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: NamespaceQueue
metadata:
  name: training
  namespace: team-a
spec:
  parent: cluster/research
  capability:
    cpu: "40"
    memory: "80Gi"
  deserved:
    cpu: "40"
    memory: "80Gi"
  reclaimable: true

Reference it in a Job or PodGroup in team-a:

spec:
  queue: namespace/training

Related:

• Issue: #5251
• PRs and contributors: #5590, #5629, @gitGurugu
• Design and configuration: Namespace-scoped Queues
• User guide: NamespaceQueue User Guide

Per-Namespace Fair Share Scheduling

Background and Motivation:

Volcano organizes resource sharing and quota governance around cluster-scoped Queues, which fits deployments that assign a dedicated queue to each tenant. Other platforms isolate teams or projects by namespace, with multiple tenants submitting into one Queue. These deployments need fairness within the shared queue as well as fairness across queues.

Current-allocation snapshots do not capture resource consumption over time. A namespace submitting a continuous stream of GPU jobs can repeatedly regain scheduling priority as its jobs finish, leaving later arrivals with little opportunity to run.

Inspired by Slurm fair-share scheduling, the new fairshare plugin tracks cumulative resource-seconds per namespace within a queue and applies exponential half-life decay. Lower historical consumption receives earlier consideration, while older usage gradually loses influence. This complements DRF's current-resource fairness and requires no administrator-assigned namespace weights.

Key Capabilities:

• Historical usage ordering within a queue, with a default four-hour half-life.
• Configurable target queues and tracked resource keys, including per-queue resource overrides.
• Optional enqueue gating and ConfigMap persistence across scheduler restarts.
• Namespace fair-share and decayed-usage metrics.

Configuration:

Add fairshare to the scheduler configuration before drf. Keep priority first if PriorityClass should take precedence. Plugin list order determines which comparison wins. This example shows a complete basic configuration using the capacity plugin:

actions: "enqueue, allocate, backfill"
tiers:
- plugins:
  - name: priority
  - name: gang
  - name: conformance
  - name: fairshare
    arguments:
      fairshare.targetQueues: "gpu-queue"
      fairshare.resourceKey: "nvidia.com/gpu"
      fairshare.halfLifeMinutes: "240"
      fairshare.enableEnqueueGate: "false"
      fairshare.persistState: "false"
- plugins:
  - name: drf
  - name: capacity
  - name: predicates
  - name: nodeorder

Fairshare influences pending-job ordering and optional enqueue admission. It does not preempt or reclaim already-running jobs. Persistence is disabled by default. Enabling it uses the configurable fairshare.stateNamespace, fairshare.stateConfigMap, and fairshare.flushIntervalSeconds settings.

Related:

• Issues: #5160, #4165
• PR and contributor: #5161, @dmildh-absci
• Design: Fair Share Scheduling
• User guide: How to Use Fairshare

PodGroup Topology Anti-Affinity

Background and Motivation:

Network topology scheduling can keep a workload together within one HyperNode domain, but this does not by itself separate different workload instances. Multiple inference replicas may all land in one supernode, exposing the service to a shared failure domain.

PodGroup anti-affinity expresses isolation at the workload-group level. It selects peer PodGroups by labels and namespace selectors and applies the separation rule at a named or numbered HyperNode tier. The scheduler intersects this policy with existing network topology constraints before evaluating placement.

Key Capabilities:

• Required rules for mandatory separation and preferred rules for best-effort separation.
• Selection of peer PodGroups in the current namespace, all namespaces, or namespaces matching labels.
• Composition with networkTopology so each group can stay together while different groups use separate domains.
• Explicit unconstrained, constrained, and rejected HyperNode gradient results for composing topology plugins.

Configuration:

Add these entries to the existing scheduler plugin list and retain the other configured actions/plugins:

- name: group-topology-affinity
  arguments:
    weight: 10
- name: network-topology-aware
  arguments:
    weight: 10

The HyperNode tree must contain the specified tier. Apply the following policy to each PodGroup requiring mutual isolation, changing its name for each instance:

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: inference-a
  namespace: default
  labels:
    app: model-a
spec:
  minMember: 2
  queue: default
  networkTopology:
    mode: hard
    highestTierName: supernode
  topologyAffinity:
    podGroupAntiAffinity:
      required:
      - podGroupSelector:
          matchLabels:
            app: model-a
        topologyTierName: supernode

Rules select PodGroup labels, not Pod labels. Each term must specify exactly one of topologyTierName and topologyTier. Preferred terms additionally require a weight from 1 to 100. Required terms omit weight. Member Pods must use schedulerName: volcano and their PodGroup's scheduling.k8s.io/group-name annotation.

The plugin must be enabled explicitly. Enforcement applies to allocate. BestEffort Pods placed through backfill do not receive this isolation guarantee. SubGroup affinity/anti-affinity, cross-PodGroup affinity, relocation of running Pods, and eviction solely to clear an anti-affinity conflict are outside the implemented scope.

Related:

• Issue: #5347
• PR and contributor: #5349, @wangyang0616
• Design: PodGroup Topology Affinity
• User guide: PodGroup Anti-Affinity

Broader HyperNode Integration with Other Schedulers

Background and Motivation:

Platforms using other schedulers also need to discover cluster topology and manage supernodes and their hierarchy. v1.16.0 makes the HyperNode controller independently deployable, opening its topology discovery and management capabilities to more scheduling systems.

Platforms can deploy just the HyperNode controller, reuse discovery methods including UFM and node labels, and manage supernodes through HyperNode resources. Other schedulers can consume that topology through the HyperNode API to implement topology-aware placement. Standalone and integrated deployments share the discovery and reconciliation implementation, so improvements can serve both modes. HyperNode remains part of Volcano's source and release lifecycle.

Key Capabilities:

• A dedicated vc-hypernode-controller-manager binary, image, Deployment, RBAC, and Helm configuration.
• controller-manager, standalone, and disabled deployment modes, with integrated mode retained as the default.
• Shared discovery and reconciliation behavior across integrated and standalone modes.
• Independent resource, replica, placement, logging, health, and metrics settings.

Configuration:

For a topology-only installation:

helm upgrade --install volcano volcano-sh/volcano \
  --namespace volcano-system --create-namespace --version 1.16.0 \
  --set custom.hypernode_controller_mode=standalone \
  --set custom.controller_enable=false \
  --set custom.scheduler_enable=false \
  --set custom.admission_enable=false

Discovery uses the existing custom.controller_config_override configuration. When migrating an existing integrated installation, first set custom.hypernode_controller_mode=disabled and wait for the controller-manager rollout before enabling standalone, so only one process reconciles HyperNodes. To retain other Volcano components, change only the HyperNode mode.

Related:

• Issue: #5133
• PRs and contributor: #5137, #5136, @wangyang0616
• Design: Standalone HyperNode Controller
• User guide: Install the Standalone HyperNode Controller

Other Notable Changes

Features and Enhancements

Load-Aware Scheduling for Bulk Pod Dispatch

Background and Motivation:

Monitoring data arrives after workload placement. During a large scheduling burst, a node can still look lightly loaded even after receiving many Pods, causing subsequent placements to concentrate on that node before Prometheus or another metrics backend catches up.

The usage plugin now maintains a session-level shadow load cache for assigned Pods whose usage is not yet visible in metrics. Scoring uses measured load plus estimated CPU and memory consumption. Each estimate is recorded as a per-Pod snapshot, so allocation rollback removes the same estimate rather than recomputing it against a changed node state.

Key Capabilities:

• Session warm-up for assigned Pending Pods and recently started Running Pods within the metrics delay window.
• Incremental estimated-load updates on allocation and deallocation.
• Configurable request, burst, and load-risk estimation, plus fixed BestEffort estimates.
• Safe fallback when monitoring data is missing or stale.

Configuration:

Add or update this plugin entry in the existing scheduler configuration. Keep the metrics source configured. The metrics.interval setting defines the monitoring delay window and defaults to 30 seconds.

- name: usage
  enablePredicate: false
  arguments:
    usage.weight: 5
    cpu.weight: 1
    memory.weight: 1
    estimator:
      request_ratio: 0.7
      burst_ratio: 0
      risk_threshold: 0.6
      risk_factor: 1.2
      be_cpu: 250m
      be_mem: 200Mi

For Guaranteed and Burstable Pods, the estimate is (request * request_ratio + (limit - request) * burst_ratio) * applied_risk_factor, clamped to the effective limit. Missing or undersized limits fall back to the request. Setting enablePredicate: false retains scoring without rejecting new Pods solely because a usage threshold has been reached.

Related:

• Issue: #5313
• PR and contributor: #5314, @3th4novo
• Design and configuration guide: Usage-Based Scheduling

Additional features and enhancements:

• Node scoring integration: Reuses the session NodeMap, batches nodeorder PreScore/Score work, and executes active score plugins in one parallel node traversal before plugin-owned normalization and aggregation. (#5497, #5928, @JesseStutler, @songmingming0827, Issue #5498)

• Kubernetes 1.36 support: Updates Kubernetes dependencies to 1.36.1, the Go toolchain requirement to 1.26, generated APIs, RBAC, installation manifests, and CI. Review the compatibility matrix for the target cluster. (#5287, @wangyang0616, Issue #5279)

• DRA scoring: Applies upstream DynamicResources Score and plugin-owned NormalizeScore to prioritized device requests, including Score-only plugins that do not implement PreScore. DRA scoring follows the existing predicates DRA enablement. (#5728, @miantalha45)

• Queue quotas per nodegroup: Caps how much a queue can allocate from each nodegroup while allowing placement in other permitted groups. This supports logical overflow from preferred fixed-capacity pools to fallback pools. (#5461, @ruanwenjun, Issue #4954)

• HAMi Ascend soft partitioning: Adds hami-vnpu-core integration for Ascend memory virtualization and compute time slicing. Enable the external device configuration's hamiVnpuCore setting and select huawei.com/vnpu-mode: hami-core on Pods. Deployments must use a matching device plugin/runtime. The final implementation no longer injects a PostStart hook. (#5462, #5722, @DSFans2014) See How to Use vNPU for the complete setup.

• Accelerator health and device selection: Checks Ascend health using allocatable devices and adds vGPU UUID-based allowlist/denylist constraints. (#5418, #5726, @DSFans2014, @goyalankit)

• Colocation resource accounting: Adds pod-level resource settings and accounts for restartable init containers/native sidecars and Pod overhead in effective requests used by the Volcano agent. (#5656, #5822, @miantalha45, @Tau721, Issue #5788)

• Queue and eviction observability: Adds inqueue resource metrics, queue workload-state metrics, and volcano_eviction_transactions_total{action}. The eviction counter records committed Statements containing evictions, once per transaction. It is neither a victim-Pod count nor confirmation of asynchronous deletion success. (#5243, #5826, #5754, @Aman-Cool, @OneSizeFitsQuorum, @ruanwenjun, Issue #5242, Issue #5753, Issue #5825)

• Operational tuning and diagnostics: Makes scheduler job/task updater concurrency configurable and logs loaded scheduler configuration for troubleshooting. (#5873, #5128, @ruanwenjun, @Tau721, Issue #5127, Issue #5872)

• CLI usability and release artifacts: Adds namespace columns and stable sorting to vcctl pod list, and release automation for Linux/macOS amd64/arm64 CLI archives, checksums, and SLSA provenance. (#5648, #5665, #5384, @aeron-gh, @hajnalmt, @xuezhaojun, Issue #5376, Issue #5647, Issue #5664)

API Changes

• NamespaceQueue and parent authorization: Adds scheduling.volcano.sh/v1beta1 NamespaceQueue, generated clients/informers, status conditions, cluster Queue spec.allowedNamespaces, and workload queue references using namespace/<name>. Enable with Helm custom.namespace_queue_enable. Custom deployments must enable NamespaceQueue in all three participating components and use hierarchical capacity. Cross-namespace parent references are unsupported. (#5590, @gitGurugu)
• PodGroup topology policy: Adds spec.topologyAffinity.podGroupAntiAffinity with required/preferred terms, PodGroup and namespace selectors, and HyperNode tier selection. The key feature section shows the supported API. (#5349, @wangyang0616)
• NUMA allocation reporting: Adds Numatopology.spec.podAllocations with Pod UID, name, namespace, and per-container resource allocations. CPU allocations use cpuset strings. The scheduler consumes these reports to distinguish released resources from placements still awaiting kubelet allocation. Use a resource-exporter version that reports the new field. (#5467, #5504, @Tau721, Issue #5466, Issue #5500)
• Topology plugin interface change: Changes HyperNodeGradientForJobFn and HyperNodeGradientForSubJobFn to return HyperNodeGradientResult. Unconstrained: true explicitly permits the subtree. Otherwise empty gradients reject it. Adds optional HyperNodeCandidateFn nomination validation. Custom topology plugins must adapt and rebuild. (#5349, @wangyang0616)
• Cache-aware scheduler plugins: Adds Session.AddHintProvider and QueueingHint-style event registration for unschedulable-job retries, plus structured Status.InsufficientResources and PrePredicateError rejection data. Rejections from plugins without safe hints fall back to normal scheduling. (#5848, @JesseStutler)
• Nodegroup queue limits: Adds the Queue annotation volcano.sh/nodegroup-resource-limits, containing a JSON mapping from nodegroup names to resource quantities. Limits are per queue and are not inherited by child queues. (#5461, @ruanwenjun, Issue #4954)

metadata:
  annotations:
    volcano.sh/nodegroup-resource-limits: |
      {"fixed": {"cpu": "100", "memory": "200Gi"}, "overflow": {"cpu": "50"}}

• Fairshare and usage arguments: Adds the fairshare.* arguments and nested usage plugin arguments.estimator settings documented in the feature sections. The estimator keys are request_ratio, burst_ratio, risk_threshold, risk_factor, be_cpu, and be_mem. (#5161, #5314, @3th4novo, @dmildh-absci, Issue #5160, Issue #5313)
• SVC host environment injection: Adds Job plugin option svc: ["--inject-hosts-env=false"] to suppress both VC_<TASK>_HOSTS and VC_<TASK>_NUM environment variables for large jobs. Host files remain available under /etc/volcano/. (#5472, @t2wang)
• vGPU UUID constraints: Adds Pod annotations volcano.sh/vgpu-use-gpuuuid and volcano.sh/vgpu-nouse-gpuuuid for allowlist/denylist device selection. UUIDs are exact matches after whitespace trimming. Denylist entries take precedence. (#5726, @goyalankit)
• Scheduler worker and cache flags: Adds --job-updater-worker-num and --task-updater-worker-num (both default to 16 and must be positive), --unschedulable-job-cache-max-skip-duration (default 5m), and --unschedulable-job-cache-debug-metrics (default false). Helm exposes custom.scheduler_job_updater_worker_num, custom.scheduler_task_updater_worker_num, and custom.scheduler_unschedulable_job_cache_debug_metrics. (#5873, #5848, @JesseStutler, @ruanwenjun, Issue #5872)
• Kubernetes integration and CSI watches: Uses the upstream VolumeBinding implementation in place of Volcano's fork and updates Kubernetes integration APIs. --csi-storage now defaults to true and controls scheduler-cache CSI informer registration. Setting it false reduces those watches without disabling upstream VolumeBinding checks. The legacy Volcano CSIStorage gate is retained for compatibility. (#5287, @wangyang0616, Issue #5279)
• HyperNode Helm settings: Adds custom.hypernode_controller_mode, the standalone controller image settings, and per-component replica, resource, placement, security, and observability values. The default deployment mode remains controller-manager. (#5137, @wangyang0616)
• Helm lifecycle controls: Adds custom.admission_init_ttl (default 3600 seconds) for completed admission-init Job cleanup and retention annotations for the existing Volcano CRDs. Review uninstall behavior and CRD ownership during upgrades. (#5475, #5731, @3th4novo, @vanshika2720, Issue #5727)

Bug Fixes

• PreFilter integration: Registers upstream PreFilter hooks for CSI limits, node affinity, and volume-zone checks and honors their skip sets in stable filters, avoiding unnecessary per-node calls. (#5505, #5542, @JesseStutler, @vanshika2720, Issue #5536)
• Queue admission reservations: Moves inqueue mutation to the confirmed JobEnqueued callback, avoiding phantom reservations when a later plugin rejects admission. Fixes milli-unit accounting for scalar inqueue resources. (#5339, #5391, @Aman-Cool, @WHOIM1205, Issue #5387)
• Hierarchical scheduling-gate capacity: Includes gate-reserved resources from descendant queues in ancestor quota checks, preventing sibling queues from removing gates beyond parent capacity. (#6047, @devzizu)
• Completed and terminating PodGroups: Preserves Completed state after finished Pods are deleted so completed workloads do not reserve quota again, and prevents terminating workloads from falling back into Inqueue. (#5996, #5796, @halcyon-r, @yuanhuaiwang, Issue #4195, Issue #5794)
• Queue lifecycle and metrics: Returns Closed queues to Closing on resync when PodGroups remain and clears stale allocated-resource metrics for jobless capacity queues. (#5955, #5050, @Aliexe-code, @magic-peach)
• Fairshare enqueue demand: Uses PodGroup MinResources for Volcano Jobs that do not yet have Pods, preventing enqueue gates from seeing zero demand. (#6007, @CapThunder19, Issue #6006)
• DRA count quota correctness: Uses saturating count arithmetic and closes a remaining overflow path so large device counts cannot wrap into negative quota usage. (#5621, #5625, @thc1006, Issue #5620)
• NUMA allocation consistency: Uses Pod-level allocation reports and explicit unassigned-Pod tracking to distinguish released CPUs from new placements awaiting allocation, preventing false NUMA policy failures. (#5504, @Tau721, Issue #5500)
• PVC informer race: Caches and resyncs tasks when their PVC is not yet visible, preventing Pods such as Spark executors with newly created PVCs from remaining permanently outside scheduler consideration. (#5596, @gucardona)
• Snapshot and node state: Restores image-locality ImageStates, refreshes affinity lists during snapshot updates, and restores node state after unsuccessful updates. (#5343, #5638, #5525, @avinxshKD, @kitianFresh, @SquareCatFirst, Issue #5524, Issue #5637)
• Scheduler panic protection: Adds missing nil checks for PodGroup-related cache paths and skips backfill tasks when scoring returns no usable node. (#5569, #5901, @moezdil, @Tau721, Issue #5568)
• Allocation continuation: Honors NeedContinueAllocating after PrePredicate failures so eligible remaining tasks can still be evaluated. (#5528, @A69SHUBHAM, Issue #5527)
• Sharding default thresholds: Retains the allocation-rate policy default range when its arguments are omitted. (#5743, @aeron-gh, Issue #5710)
• Infinite-resource diagnostics: Formats effectively unlimited resource values consistently in scheduler resource strings. (#5276, @r0hansaxena, Issue #5148)
• HyperNode topology rebuilds and discovery: Corrects UpdateHyperNode rebuild logic, avoids duplicate HyperNodes when discovery sees a stale informer cache, and handles missing or non-string UFM endpoints without panicking. (#5076, #5894, #6057, @harshrajdebug, @hzxuzhonghu, @wangyang0616, Issue #5887, Issue #6049)
• Topology nominations: Honors nominated HyperNodes without hard topology mode and persists/prioritizes SubJob nominations across scheduling cycles. (#5440, #5476, @vzhou-p)
• PodGroup identity and deletion: Guards annotation patches by Pod UID when Pods are recreated and handles tombstone deletion events safely. (#5344, #5735, @avinxshKD, @YiD11, Issue #5331, Issue #5734)
• Job restart and completion handling: Avoids deleting the PodGroup for targeted Pod restarts, bounds Job condition growth, treats Succeeded dependencies as ready, and settles JobFlow as Failed when its Job is Terminated. (#5342, #5422, #5439, #5675, @aeron-gh, @Aman-Cool, @avinxshKD, Issue #5385, Issue #5421, Issue #5424, Issue #5672)
• Partition minimum availability: Falls back to replica count when partition minPartitions is omitted. (#5402, @togettoyou, Issue #5401)
• CronJob reconciliation: Returns status-update failures for retry and refetches conflicting Jobs in the CronJob's own namespace. (#5956, #5957, @omlahore)
• vGPU allocation metadata: Synchronizes TaskInfo allocation annotations across binding and rollback, preserves committed annotations, and clears speculative/device annotations during release. (#5588, #5416, @106umao, @archlitchi, Issue #5335, Issue #5532)
• GPU memory, MIG, and device safety: Prevents idle-memory underflow, validates annotation usage values, deep-copies MIG usage, and guards missing devices in node information and batch scoring. (#5470, #5404, #5550, #5414, #5217, @DSFans2014, @netliomax25-code, @PrasannaMishra001, Issue #5216)
• HAMi device-plugin coordination: Fixes large-cluster vGPU scheduling coordination. (#5393, @linuxfhy, Issue #5361)
• Ascend vNPU correctness: Corrects allocation resource accounting, supports the vnpus.configs device-config wrapper, fixes spread scoring and node-lock keys, and supports normal priority preemption for HAMi Ascend. (#5363, #5399, #5558, #5573, #5843, @DSFans2014, @miantalha45, @shivansh-gohem, Issue #5360, Issue #5372, Issue #5828)
• Reclaim device feasibility: Rechecks device feasibility after tentative evictions before committing a reclaim decision. (#5855, @miantalha45, Issue #5852)
• Agent Scheduler queue and snapshot handling: Completes or requeues popped Pods on early exits and omits not-ready placeholder nodes from incremental snapshots. (#5560, #5927, @togettoyou, @xiezhayang, Issue #5924)
• Agent Scheduler deployment settings: Rejects non-positive worker counts and uses custom.agent_scheduler_replicas in the Helm Deployment. (#5668, #5799, @bhuvan-somisetty, @miantalha45, Issue #5666, Issue #5798)
• CPU quota handling: Configures GOMAXPROCS using container CPU quotas across Volcano components. (#5651, @avinxshKD, Issue #5649)
• Network QoS CNI discovery: Matches CNI plugins by type rather than display name. (#5516, @avinxshKD, Issue #5515)
• CLI correctness: Deduplicates Pod lists by namespace and name, guards nil event data, returns failures from JobFlow/JobTemplate file operations, and rejects invalid describe output formats. (#5636, #5653, #5971, #6012, @aeron-gh, @iamtanishqjain, @Psykii22, Issue #5652, Issue #5970, Issue #6011)
• Validation and diagnostic messages: Separates PodGroup/CronJob validation errors, reports all invalid subgroup and hierarchical-weight entries, corrects CLI/Network QoS descriptions and duplicated words, fixes agent CPU-recovery validation wording, and logs the correct nodeorder weight. (#5487, #5610, #5959, #5962, #5538, #5593, #5632, #5594, #5510, @A69SHUBHAM, @aeron-gh, @anxkhn, @magic-peach, Issue #5443, Issue #5489, Issue #5509)
• Helm installation and uninstall: Prevents Helm 4 admission-init cleanup races that leave installations pending and preserves the existing Volcano CRDs and their custom resources on uninstall after applying the updated retention annotations. (#5475, #5731, @3th4novo, @vanshika2720, Issue #5727)
• DRA capacity scaling: Replaces device-count and minimum-available additive loops with constant-time Quantity multiplication, avoiding scheduler stalls for very large counts and preserving exact capacity totals. (Issue #5627, commit c7fb3fe)

Security Fixes Included

GHSA-j38h-7pfq-cxmw: DRA capacity accounting denial of scheduling service

Tenant-controlled device or minimum-task counts could trigger excessive iteration in DRA capacity aggregation. Calculations performed under the scheduler cache lock could exhaust CPU time and block scheduling for workloads across namespaces. The ResourceClaim-triggered path requires DRA support, explicit enablement of DRAConsumableCapacity (disabled by default), a capacity-bearing ExactCount request, and permission to create the associated ResourceClaim and Pod.

The fix replaces count-proportional loops with constant-time quantity multiplication, validates invalid counts, and uses overflow-safe device-count arithmetic. It was first released in v1.15.2 and is included in v1.16.0.

Affected versions: v1.15.0 and v1.15.1 (>= 1.15.0, < 1.15.2).

Fixed versions: v1.15.2 on the release-1.15 branch. Volcano v1.16.0 includes the fixes. Users on affected versions should upgrade.

CVSS rating: Medium, 6.5 (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H).

Reported and fixed by @thc1006, with advisory coordination by @JesseStutler. See issue #5627, the capacity-scaling fix, and the v1.15.2 security release note.

Other security and supply-chain improvements:

• Dependency security updates: Updates Prometheus, golang.org/x/crypto, x/net, x/text, and several indirect dependencies, including compression, OpenTelemetry, etcd, gRPC, and module-verification libraries, to address upstream security advisories. These updates are included alongside the DRA count/loop hardening listed under Bug Fixes. (#5444, #5605, #5757, #5883, #5931, #5943, @copilot-swe-agent, @dependabot, @gucardona, @omlahore)
• Supply-chain controls: Adds release SBOM generation and Security Insights metadata, CLI release provenance, read-only default workflow permissions, explicit permissions for reusable workflows, pinned GitHub Actions and Docker base images, and provenance-workflow corrections. (#5367, #5384, #5369, #5445, #5469, #5481, #5567, #5787, @copilot-swe-agent, @FAUST-BENCHOU, @JesseStutler, @kubeboiii, @shivansh-gohem, @xuezhaojun, Issue #5376, Issue #5465)

Dependencies

• Kubernetes and Go: Builds against Kubernetes 1.36.1 and requires Go 1.26 for source builds. Generated clients and scheduler integration follow the Kubernetes 1.36 APIs. (#5287, @wangyang0616, Issue #5279)
• Networking and security libraries: Updates golang.org/x/crypto to v0.56.0, x/net to v0.58.0, x/text to v0.41.0, and Prometheus to v0.311.3 in the reviewed source snapshot. (#5444, #5605, #5757, #5883, #5943, @copilot-swe-agent, @dependabot, @gucardona, @omlahore)
• Runtime and observability libraries: Updates CNI, cgroups, filesystem notifications, Elasticsearch, Prometheus common, and the OTLP trace exporter, together with indirect security fixes. (#5695, #5717, #5756, #5694, #5714, #6000, #5931, #5696, @dependabot, @gucardona)
• Test libraries: Updates e2e-framework, Ginkgo, Gomega, and gomonkey. (#5693, #5758, #5715, #5718, @dependabot)

Testing, CI, and Maintenance

• Cache and admission maintenance: Removes the unnecessary NodeInfo clone on node updates, indexes child Queue lookups in admission, and reduces string-construction overhead. (#5556, #5506, #5495, @DSFans2014, @vanshika2720, Issue #5555)
• Release and dependency automation: Updates checkout, artifact downloads, pull-request creation, release uploads, FOSSA and CodeQL automation. Groups CodeQL updates and restores preview chart releases. (#5697, #5700, #5701, #5719, #5760, #5761, #5704, #5786, @copilot-swe-agent, @dependabot, @JesseStutler)
• CI reliability: Limits workflow approval to fork PRs, enables validation on branch pushes, skips secret-dependent FOSSA checks for fork PRs, fixes FOSSA scanning and Spark image builds, and stabilizes Agent Scheduler and cleanup tests. (#5702, #5912, #5383, #5375, #5985, #5380, #5396, @agrawalcodes, @copilot-swe-agent, @FAUST-BENCHOU, @JesseStutler, @ranxi2001, Issue #5374, Issue #5397, Issue #5976)
• Regression coverage: Adds queue open-state tests and fixes hierarchical queue test synchronization and Job-state test diagnostics. Feature PRs also include coverage for NamespaceQueue, topology anti-affinity, cache wakeups, and standalone HyperNode ownership transitions. (#5611, #5412, #5595, @anxkhn, @copilot-swe-agent, @kryz73)
• Scheduler and agent cleanup: Removes redundant preemptor requeue logic, migrates deprecated string sets to generic sets, and normalizes repository line endings. (#5818, #5909, #5419, @autsu, @DarkestSky, @hajnalmt, Issue #5776)
• Documentation and community: Uses common community sources for adopter information, refreshes ecosystem links (including Grove), updates talks/use cases, clarifies getting started and Slack instructions, repairs links and wording, and makes Go setup instructions version-independent. (#5354, #5415, #6020, #5914, #5341, #5617, #5438, #5390, #5599, #5604, #5660, #5934, #6053, #5420, @choosen23, @copilot-swe-agent, @DSFans2014, @JesseStutler, @kube-gopher, @safiya2610, @shettythrishal328-dev, @vivek41-glitch, @yash-pokhriyal, Issue #5598)
• Feature documentation and proposals: Documents HyperNode tier-name usage and vNPU monitoring. Adds a queue-scoped overcommit design proposal. The proposal does not enable that capability in this release. (#5484, #5738, #5746, @archlitchi, @miantalha45, @vanshika2720, Issue #5483)
• Community ownership: Updates approver and reviewer records. (#5464, #5736, #5755, @devzizu, @hajnalmt)
• Bug-report evidence: Documents required version, configuration, reproduction, and diagnostic evidence for bug reports. (#5706, @JesseStutler)

Upgrade Instructions

To upgrade after Volcano v1.16.0 has been published:

helm repo add volcano-sh https://volcano-sh.github.io/helm-charts
helm repo update
helm upgrade volcano volcano-sh/volcano \
  --namespace volcano-system --version 1.16.0

# Alternatively, for a manifest-based installation:
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.16.0/installer/volcano-development.yaml

• Update CRDs and RBAC together with the control-plane components, including NamespaceQueue and the new PodGroup/Numatopology fields. Follow the chart or installation method's CRD upgrade procedure and verify that the API server has the updated schemas before applying new fields.
• Review the Kubernetes compatibility matrix. Kubernetes 1.23 deployments are unsupported. Source builds require Go 1.26, and custom scheduler integrations must follow the updated Kubernetes APIs.
• Rebuild custom topology plugins for the new HyperNode gradient return types. Review custom scoring adapters against the upstream plugin-owned normalization path.
• NamespaceQueue and UnschedulableJobCache are disabled by default. Enable them explicitly after reviewing their scope. NamespaceQueue requires hierarchical capacity. This cache version does not cache hierarchical capacity rejections.
• Review custom scheduler configuration when enabling fairshare, load estimation, nodegroup limits, or topology anti-affinity. Fairshare must precede DRF for historical-usage ordering to take effect. Topology anti-affinity requires the new plugin and is not enforced for BestEffort Pods placed by backfill.
• When enabling new NUMA allocation reporting, update resource-exporter to a version supporting spec.podAllocations. Accelerator integrations likewise require compatible external device plugins and runtimes.
• When switching HyperNode ownership, roll out disabled mode before standalone. Embedded controller-manager mode remains the default.
• Helm admission-init Jobs now use TTL cleanup rather than immediate successful-hook deletion. The existing CRDs receive retention annotations so Helm uninstall preserves their resources. Inspect ownership and cleanup requirements when reinstalling. The new chart-managed NamespaceQueue CRD does not carry that retention annotation in this version.
• Gang-aware gangPreempt and gangReclaim remain explicit actions introduced in v1.15.0. This release does not merge them into the legacy preempt and reclaim actions.

Contributors

Thank you to all contributors who made this release possible:

@106umao, @3th4novo, @A69SHUBHAM, @aeron-gh, @agrawalcodes, @Aliexe-code, @Aman-Cool, @anxkhn, @archlitchi, @autsu, @avinxshKD, @bhuvan-somisetty, @CapThunder19, @choosen23, @copilot-swe-agent, @DarkestSky, @dependabot, @devzizu, @dmildh-absci, @DSFans2014, @FAUST-BENCHOU, @gitGurugu, @goyalankit, @gucardona, @hajnalmt, @halcyon-r, @harshrajdebug, @hzxuzhonghu, @iamtanishqjain, @JesseStutler, @kitianFresh, @kryz73, @kube-gopher, @kubeboiii, @linuxfhy, @magic-peach, @miantalha45, @moezdil, @netliomax25-code, @omlahore, @OneSizeFitsQuorum, @PrasannaMishra001, @Psykii22, @r0hansaxena, @ranxi2001, @ruanwenjun, @safiya2610, @shettythrishal328-dev, @shivansh-gohem, @songmingming0827, @SquareCatFirst, @t2wang, @Tau721, @thc1006, @togettoyou, @vanshika2720, @vivek41-glitch, @vzhou-p, @wangyang0616, @WHOIM1205, @xiezhayang, @xuezhaojun, @yash-pokhriyal, @YiD11, @yuanhuaiwang

Full Changelog

v1.15.0...v1.16.0