github volcano-sh/volcano v1.8.0
on GitHub

latest releases: v1.10.0, v1.10.0-alpha.0, v1.9.0...
15 months ago

What's New

Add JobFlow to support lightweight workflow orchestration

The workflow orchestration engine is widely used in high-performance computing, AI biomedicine, image processing, beauty, game AGI, scientific computing and other scenarios, helping users simplify the management of multiple parallel tasks and dependencies, and greatly improving the overall computing efficiency.

JobFlow is a lightweight task flow orchestration engine that focuses on Volcano job orchestration. It provides Volcano with job probes, job completion dependencies, job failure rate tolerance, and other diverse job dependency types, and supports complex process control primitives. The specific capabilities are as follows:

  • Support large-scale job management and complex task flow orchestration.
  • Support real-time query of the running status and task progress of all associated jobs.
  • Support automatic operation of jobs and scheduled start to release labor costs.
  • Various action strategies can be set for different tasks, and corresponding actions can be triggered when the task meets certain conditions, such as timeout retry, node failure drift.

Refer to the links for more details. (JobFlow doc, @hwdef, @lowang-bh, @zhoumingcheng)

Support vGPU scheduling and isolation

Since the outbreak of ChatGPT, there have been more and more research and development of AI large models, and different types of AI large models have been launched one after another. In production environment, users have pain points such as low resource utilization and inflexible GPU resource allocation. They have to purchase a large amount of redundant heterogeneous computing power to meet business needs, and heterogeneous computing power itself is expensive. It has brought a great burden to the development of the enterprise.

Starting from version 1.8, Volcano provides an abstract general framework for sharing devices (GPU, NPU, FPGA...), developers can customize multiple types of shared devices based on this framework. Currently Volcano has supported GPU device multiplexing, resource isolation based on this framework, details are as follows:

  • GPU sharing: Each task can apply to use part of the resources of a GPU card, and the GPU card can be shared among multiple tasks.
  • Device memory control: GPU can be allocated according to device memory (for example: 3000M) or allocated in proportion (for example: 50%) to realize GPU virtualization resource isolation capability.

Refer to the links for more details.

Support the preemption capability for GPU and user-defined resources

Currently, Volcano supports CPU, Memory and other basic resource preemption. GPU resources and user self-managed resources such as NPU, network resources have not been supported yet.

In version 1.8, the predication is refactored to provide more detailed response such as Unschedulable and UnschedulableAndUnresolvable for different scenarios.
The GPU preemption function has been released based on the optimized framework, and the user developed scheduling plugins based on Volcano can be adapted and upgraded according to business scenarios.

Refer to the link for more details. (#2916, @wangyang0616)

Support ElasticSearch monitoring systems in node load-aware scheduling and rescheduling

The status of the kubernetes cluster changes in real time with the creation and termination of tasks. In some scenarios such as adding or deleting nodes, changing the affinity of Pods and Nodes, and dynamically changing the lifecycle of jobs, etc. The following problems will occur. Resource utilization is unbalanced, node performance bottlenecks are offline, etc. At this time, load aware scheduling and rescheduling can help user solve the above problems.

Prior to Volcano version 1.8, the load awareness scheduling and rescheduling only supports Prometheus. Starting from version 1.8, Volcano optimizes the monitoring index acquisition framework and adds support for ElasticSearch monitoring system.

Refer to the links for more details.

Optimize Volcano's ability to schedule microservices

Add Kubernetes default scheduler plugin enable and disable switch

Volcano is a unified integrated scheduling system that not only supports computing jobs such as AI and BigData, but also supports microservice workloads. It is compatible with scheduling plugins such as PodTopologySpread, VolumeZone, VolumeLimits, NodeAffinity, and PodAffinity of the Kubernetes default scheduler, and Kubernetes default scheduling plugins capabilities Enabled by default in Volcano.

Since Volcano 1.8, the Kubernetes default scheduling plugins can be freely selected to be turned on and off through the configuration file, and all of them are turned on by default. If you choose to turn off some plugins, such as: turn off the PodTopologySpread and VolumeZone plugins, you can set the corresponding values ​​in the predicate plugin is false.
Refer to the links for more details. (#2748, @jiangkaihua)

Enhance scheduler to keep compatibility with ClusterAutoscaler

In the Kubernetes platform, Volcano is not only used as a scheduler for batch computing services, but also used as a scheduler for general services. Node horizontal scaling is one of the core functions of Kubernetes, which plays an important role in coping with the surge of user traffic and saving operating costs. Volcano optimizes job scheduling and other related logic, and enhances the compatibility and interaction with ClusterAutoscaler, mainly in the following two aspects:

  • The pod that enters the pipeline state in the scheduling phase triggers capacity expansion in time.
  • Candidate nodes are graded in gradients to reduce the impact of cluster terminating pods on scheduling load, and prevent pods from entering invalid pipeline states, resulting in cluster expansion by mistake.

Refer to the links for more details. (#2782, #3000, @wangyang0616)

Provide tolerance for exception of device plugin

When device plugin crashs or fails to report resouces for some reason and the total resource amount of the node is less than the allocated resource amount, Volcano considers that the node data is inconsistent, make the node as OutOfSync and isolates the node, and stops scheduling any new workload to the node. The isolocation machinism brought some impact to the cluster for example device plugin has no chance to be scheduled to the OutOfSync node. In Volcano v1.8, the machinism is enhanced to tolerate the exception of device plugin, the non-GPU workload like device plugin is still allowed to be scheduled to OutOfSync node.

Refer to the link for more details. (#2999, @Monokaix)

Add helm charts for Volcano

As Volcano is used in production environments and cloud environments with more and more users, simple and standard installation actions are crucial. Since version 1.8, Volcano has optimized charts package publishing and archiving actions, standardized the installation and use process, and completed the migration of historical versions v1.6 and v1.7 to the new helm warehouse.
Refer to the link for more details. (Volcano helm-charts, @wangyang0616)

Other Notable Changes

Bug Fixes

Check out latest releases or
releases around volcano-sh/volcano v1.8.0

Don't miss a new volcano release

NewReleases is sending notifications on new releases.

Get notifications