github Project-HAMi/HAMi v2.11.0
on GitHub

3 hours ago

What's Changed

Key Features

General:

No longer mount HAMI-core to whole GPU allocation, metrics will be exported by NVML directly
Flexible-MIG
Support Cluster Autoscaler
Real-usage monitoring for Flexible-mig workloads

Remote CUDA:

Allows running vllm on remote cuda

Schedule:

PodGroup Support
New schedule policy: mutex
Schedule policy combinations
GPU/CPU Numa Align

Devices:

Full AMD Support
Ascend 950 Support
libvnpu performance optimization

✨ Features Details

  • ci: allow manual dispatch of the Trivy image scan by @moezdil in #2768
  • fix(monitor): replace MustNewConstMetric with safe helper and track InitContainers by @devGPP23 in #2716
  • feat(scheduler): account native sidecar container GPU resources by @maishivamhoo123 in #2723
  • Add nil node checks to PatchNodeAnnotations and RemoveNodeAnnotation by @KunwarSidhu47 in #2792
  • feat: add groundwork for NUMA-aware device refit by @saiyam1814 in #2729
  • feat: add scheduler NUMA refit for device allocations by @saiyam1814 in #2731
  • ci: run CodeQL on every pull request by @moezdil in #2816
  • test(util): add unit tests for GetNode() function by @Kevindua26 in #2798
  • [kind/bug] Restore model-aware scheduling for Cambricon MLU devices by @asadjan4611 in #2780
  • [Feature] Add Prometheus metrics for scheduler leader and cache sync state by @KunwarSidhu47 in #2758
  • Feat/scheduler poddisruptionbudget by @im-Toqeer-506 in #2773
  • feat: add host PID broker server and device plugin integration by @iemAnshuman in #2417
  • test(device): add unit tests for ReplaceUsage and GetResourceQuota by @KunwarSidhu47 in #2825
  • feat(e2e): add MIG E2E targets and fix GPU preflight fail-fast by @haitwang-cloud in #2742
  • [Feature] Expose Hardware Health and Error Metrics in vGPUmonitor (Fixes #2717) by @Gurkaran18 in #2732
  • feat(device plugin): add auto detecting nvidiaDriverRoot by @FouoF in #2848
  • test(scheduler): verify NUMA refit PATCH timeout releases allocLock by @asadjan4611 in #2869
  • webhook: validate device-scoring-weights at admission by @asadjan4611 in #2827
  • feat(nvidia): score MIG placements and add backtracking fallback. by @maishivamhoo123 in #2855
  • test(scheduler): add Go benchmarks for the Filter scoring path by @im-Toqeer-506 in #2907
  • feat(dashboard): compare allocated and host GPU memory by @Utkarsh56016 in #2922
  • Add template to vNPU-mode by @archlitchi in #2970
  • feat(helm): add OpenShift &SELinux helm chart support by @spencercjh in #2208
  • Cleanup: Replace %v with %w in fmt.Errorf() for better error handling by @Kevindua26 in #2998
  • test(e2e): add scheduler policy E2E suite by @haitwang-cloud in #2793
  • feat(chart): expose node-lock-retry-timeout and extender httpTimeout by @im-Toqeer-506 in #2944
  • perf(scheduler): stop deep-copying whole Pods into the usage snapshot by @togettoyou in #2978
  • feat(nvidia): allow a pod to prefer a MIG profile over 3g via annotation. by @maishivamhoo123 in #3014
  • Feat: pod level and container level overwrite env by @peachest in #2966
  • enhance(device-plugin): pair NVML init and shutdown on the MIG manager session by @FouoF in #2989
  • feat(scheduler): add remote-gpu device backend for lupine-served GPUs by @moezdil in #3004
  • feat(ascend): honor plugin-advertised hami-core percentage budget by @curry30000 in #2952
  • index device UUIDs once per scrape by @im-Toqeer-506 in #3064
  • feat(scheduler): validate device resource names and defaults at startup by @im-Toqeer-506 in #3010
  • Add Missing Node Label to Memory Controller Metric in vGPUmonitor (Fixes #2771) by @Gurkaran18 in #2776
  • feat(mthreads): configurable per-card memory units (mthreads S5000) + whole-card/slice mixing guard by @wjluo in #2988
  • test: add make bench target and run Go benchmarks in CI by @im-Toqeer-506 in #3067
  • fix: disable MIG when using hami-core by @FouoF in #3043
  • Fix/scheduler metrics audit by @im-Toqeer-506 in #3088
  • fix(scheduler): guard allocation annotations on update, not just create by @moezdil in #3074
  • feat(device-plugin): run GPUs for lupine instead of advertising them locally by @moezdil in #3013
  • fix(device-plugin): auto driverroot make mount unchange by @FouoF in #3092
  • Add CDI support for dynamic MIG devices by @asadjan4611 in #3070
  • support all four NeuronCores on Inferentia1 devices by @asadjan4611 in #3082
  • fixing the reject unknown fields in device-config.yml by @asadjan4611 in #2941
  • fix(scheduler): fail closed on invalid HAMi device requests instead of binding GPU-less pods by @Eshiv-Pandey in #2994
  • perf(scheduler): build the usage snapshot only for candidate nodes by @im-Toqeer-506 in #3024
  • add profiling support with dedicated HTTP bind address by @im-Toqeer-506 in #3117
  • refactor(vGPUMonitor): cleanup the temporary files of HAMi-Core in Device Plugin by @Shouren in #3049
  • feat!: move all vendor configs to devices.VENDOR by @spencercjh in #3124
  • feat: report memory and core resources to node capacity by @shkatara in #2874
  • feat: only use values in values.yaml as defaults for device-config by @spencercjh in #3152
  • fix(remotegpu): select Lupine server using all pod container requests by @sanket-jadhav-cse in #3166
  • feat(amd): support gpumem-percentage memory requests by @moezdil in #3179
  • feat(ascend): schedule ENPU workloads with policy isolation by @maverick-woo in #3110
  • refactor(device): share the allocation check with every vendor plugin by @moezdil in #3125
  • feat(amd): support amd.com/numa-bind for multi-GPU requests by @moezdil in #3186
  • fix(vGPUmonitor): surface collection failures and add health probes by @moezdil in #3162
  • feat(amd): check the namespace ResourceQuota in Fit by @moezdil in #3180
  • feat: add helm-docs workflow for chart doc by @spencercjh in #3189
  • feat(device-plugin): match nodeconfig by label selector by @usr-bin-ksh in #3173
  • fix(remotegpu): run library-copy init first by @sanket-jadhav-cse in #3201
  • Optimize the Dynamic MIG instance lifecycle by @asadjan4611 in #3128
  • feat(helm): expose Ascend hami-core mode slot capacity by @spencercjh in #3203
  • feat(amd): set a configurable runtimeClassName on AMD GPU Pods by @moezdil in #3194
  • fix(remotegpu): exclude NotReady Lupine servers by @asadjan4611 in #3184
  • feat(amd): honor the topology-aware policy by XGMI links and NUMA by @moezdil in #3191
  • feat(vGPUMonitor): whole-GPU hook bypass and direct NVML monitoring by @vesta-mlp in #3112
  • fix(device-plugin): mount GPU limits at /overrideEnv for login sessions. by @maishivamhoo123 in #3220
  • fix(device-plugin): stop mounting the host root for auto driver root by @adity1raut in #3113
  • Replace int32 to int64 to get rid of range check by @archlitchi in #3233
  • feat: add limitDelivery switch and stable usage.cache name by @maishivamhoo123 in #3226
  • feat(amd): default the GPU count, memory and cores, and honor device cordon by @moezdil in #3192
  • docs(chart): document devices.cambricon values and account a count above 1 as whole MLUs by @B-White84 in #3218
  • enhancement remote mode by @FouoF in #3247
  • chore: update build images, HAMi-core, CI tooling and refresh the root docs by @moezdil in #3237
  • feat: introduce deviceSplitCount for both hami-core and enpu mode by @spencercjh in #3267
  • feat(support): add a redacted HAMi diagnostic bundle collector by @im-Toqeer-506 in #3149
  • feat(remotegpu): select Lupine server pools by label by @asadjan4611 in #3195
  • Add 'nvidia-smi' to remote client, optmize the documents by @archlitchi in #3278
  • feat: add MIG monitoring support-v2.11-Support pod with dynamic-mig by @tsingst in #3270
  • feat(chart): add securityContext for the lupine server by @moezdil in #3272
  • Update default core value for hygon by @archlitchi in #3285
  • enhance CI with benchmark comparisons and reporting by @im-Toqeer-506 in #3146
  • fix(remotegpu): requeue pods waiting for a lupine fleet change by @moezdil in #3274
  • fix(monitor): resolve dynamic MIG identities and watch profiling fields by @FouoF in #3304
  • feat: introduce global.imageSource to switch between upstream and CN mirrors by @spencercjh in #3306
  • feat: enhance fields validator and add complete coverage helm-unittest by @spencercjh in #3324
  • support session image pull secrets in configuration and templates by @im-Toqeer-506 in #3329
  • feat(scheduler): honor preferred nodeAffinity in node score by @falconlee236 in #3298
  • feat(vGPUmonitor): add an unknown GPU process count metric by @heuristicwave in #3305
  • Fix vgpu-monitor can't discover mig-instance properly by @archlitchi in #3349
  • Configure Ascend950DT 9582 whole-card scheduling by @panpan0000 in #3316
  • fix(amd): select quota-feasible GPU combinations in topology scheduling by @Eshiv-Pandey in #3361

🐛 Bug Fixes

  • fix(nodelock): serialize expired-lock recovery by @AyushSrivastava1818 in #2733
  • fix(plugin): don't cache node annotation as up-to-date when the patch fails by @adity1raut in #2545
  • fix(device): reject malformed and negative device allocation annotations by @AyushSrivastava1818 in #2765
  • fix(nvidia): collapse duplicate MIG candidates in topology scheduling by @Eshiv-Pandey in #2751
  • fix(skill): align vGPU metrics summary with actual codebase definitions by @haitwang-cloud in #2761
  • fix(scheduler): include init container requests in device locking by @AyushSrivastava1818 in #2755
  • fix(dashboard): align node GPU memory ratio scale by @blackdragoon26 in #2840
  • fix(vGPUmonitor): gracefully skip GPU utilization metrics when unsupported by @SK8-infi in #2808
  • fix: NUMA refit follow-ups from post-merge review by @saiyam1814 in #2817
  • fix(scheduler): keep the configured policy for unrecognized policy an… by @IIITManjeet in #2769
  • fix: reject invalid device count (0/negative) across biren, cambricon… by @iron-prog in #2740
  • fix(scheduler): account terminating pods replayed by informer sync by @moezdil in #2812
  • fix(nvidia): honor enableGetPreferredAllocation in GetDevicePluginOptions by @AyushSrivastava1818 in #2845
  • fix(scheduler): keep candidate nodes for non HAMi pods on the simulation path by @moezdil in #2838
  • fix(metax): let multi container pods spread over exclusive GPUs by @moezdil in #2837
  • fix(plugin): pair per device limits by UUID when aligning kubelet IDs by @moezdil in #2814
  • fix(awsneuron): cap whole-device core accounting to addressable cores by @moezdil in #2809
  • fix(enflame): keep DRS profile core costs within the device budget by @moezdil in #2810
  • fix(kunlun): make topology selection independent of device slice order by @moezdil in #2811
  • fix(amd): keep allocation annotations decodable for empty registered types by @moezdil in #2815
  • fix(device): reject GPU core percentage requests out of range [0,100] by @AyushSrivastava1818 in #2784
  • fix(device-plugin): remove panic(0) traps from getAPIDevices and dead util code by @devGPP23 in #2775
  • fix(monitor): reuse the node-scoped pod informer for metrics by @moezdil in #2896
  • fix(monitor): report metrics server listen failures by @moezdil in #2897
  • fix(chart): target the scheduler metrics port by name by @moezdil in #2895
  • fix(ascend): gate 910C module pair allocation on SuperPod by @moezdil in #2853
  • fix(metrics): drop the constant container index label by @moezdil in #2898
  • fix(scheduler): keep bare reasons in FilteringFailed events by @im-Toqeer-506 in #2881
  • fix(device-plugin): propagate IMEX channels into the NVIDIA plugin by @Eshiv-Pandey in #2892
  • fix(scheduler): use phase-aware accounting during NUMA refit by @asadjan4611 in #2868
  • fix(ascend): guard against empty ResourceMemoryName injection in admission by @KunwarSidhu47 in #2889
  • fix(scheduler): validate NUMA refit quota with container positions by @asadjan4611 in #2836
  • fix(dashboard): use native node labels for host panels by @blackdragoon26 in #2852
  • fix: remove duplicate effectivePodDeviceUsage function by @FouoF in #2955
  • fix(util): return the canonical GPU scheduler policy for padded annotations by @moezdil in #2854
  • fix(kunlun): compare elements instead of indices in calcscore sort by @Woolgathererer in #2924
  • fix(scheduler): keep multi-slot device usage across annotation round trip by @Eshiv-Pandey in #2877
  • fix(monitor): count task priorities in a map, not a slice index by @moezdil in #2931
  • fix(chart): render service labels at the right indent level by @moezdil in #2932
  • fix(scheduler): clean annotationless pod deletes from cache by @0z5a in #2968
  • Fix Ascend webhook reject pod with 'vnpu-core' resource by @archlitchi in #2975
  • fix(nvidia): guard Allocate against empty DevicesIds requests by @Woolgathererer in #2956
  • fix(chart): honor kube-scheduler image pull secrets by @spencercjh in #2976
  • fix(chart): keep scheduler livenessProbe inside the kubeScheduler container guard by @magic-peach in #2977
  • fix(ascend): do not overwrite ASCEND_VISIBLE_DEVICES for Ascend containers by @peachest in #2953
  • fix(hack): write the coverage mode line and data to the same file by @im-Toqeer-506 in #2935
  • fix(scheduler): prune printedLog when a node is deleted by @im-Toqeer-506 in #2939
  • fix(scheduler): require caller authentication on /refit by @AyushSrivastava1818 in #2882
  • fix(chart): keep workload labels on scraped HAMi metrics by @moezdil in #2934
  • fix(nvidia): make managed admission env mutation idempotent by @asadjan4611 in #2936
  • fix(scheduler): serve the extender verbs on loopback, not the cluster port by @moezdil in #3039
  • fix(scheduler): verify pod identity before the extender acts on a request by @moezdil in #3040
  • fix(scheduler): deny pods created with scheduler-owned allocation annotations by @moezdil in #3042
  • fix(ci): make the release version check compare real values by @moezdil in #2962
  • fix(awsneuron): allocate by device index, not slice position by @moezdil in #2960
  • fix(iluvatar): read the scaled core limit as a per card value by @moezdil in #2961
  • fix(ci): run the compile, chart and e2e jobs on merge by @moezdil in #2963
  • fixing the awsneuron: derive core geometry per node by @asadjan4611 in #3029
  • fix(scheduler): score devices against the request on model-typed backends by @adity1raut in #2870
  • fix(monitor): drop duplicate NVIDIA prefix from device_type label by @rapsealk in #2991
  • fix(device-plugin): reject an allocation larger than the container requests by @moezdil in #3075
  • fix(device): enforce minimum limit across multiple ResourceQuota objects by @ManthanNimodiya in #3094
  • fix: disable cuda dnf repo to recover CI by @spencercjh in #3153
  • fix(remotegpu): keep multi-container pod allocations on one Lupine server by @sanket-jadhav-cse in #3164
  • fix(chart): drop unused pods and nodes verbs from the device-plugin ClusterRole by @blackdragoon26 in #3157
  • fix(scheduler): report undecodable live Pod allocations by @asadjan4611 in #2959
  • fix(amd): round core requests up to whole WGPs by @moezdil in #3174
  • fix(vGPUmonitor): make the MIG lock wait respect context cancellation by @moezdil in #3159
  • fix(scheduler): gpumem score before gpumem-percentage by @diegodevxd in #3106
  • fix(nodelock): keep an expired lock while its holder waits to be allocated by @moezdil in #3127
  • fix: defer Ascend memory trimming until node filtering by @asadjan4611 in #3107
  • fix(util): serve only the pod the node lock names by @moezdil in #3126
  • fix(hygon): prevent dcucores integer overflow by @AyushSrivastava1818 in #3012
  • fix(nvidia): reject fractional gpu memory percentages by @AyushSrivastava1818 in #2789
  • fix(chart): use lists for device plugin extraEnvs defaults by @spencercjh in #3205
  • fix(amd): judge node health by allocatable GPUs by @moezdil in #3182
  • fix(amd): read a registration without a type as the AMD type by @moezdil in #3212
  • fix(scheduler): keep HAMI_NODELOCK_EXPIRE unless --node-lock-timeout is set by @togettoyou in #3223
  • fix(scheduler): restore steady-state usage on pod replay by @phantom5125 in #3231
  • fix(deps): update Go dependencies and bump golang.org/x/net to v0.60.0 by @moezdil in #3236
  • Fix healthcheck always return true for some device vendors by @archlitchi in #3255
  • fix(chart): require Kubernetes 1.23, guard the PSP and pin images by digest by @moezdil in #3238
  • fix(device-plugin): honor disable-core-limit flag by @B-White84 in #3252
  • fix(device-plugin): mark all replicas/MIG slices unhealthy on XID by @Samurai007AK in #3265
  • fix(device-plugin): check MIG allocations against the registered profiles by @moezdil in #3264
  • fix(vGPUmonitor): skip terminated pods in per-container metrics by @loiht2 in #3294
  • fix(scheduler): ignore negative percentage by @diegodevxd in #3313
  • fix(scheduler): store quota memory limits scaled by memoryFactor by @loiht2 in #3339
  • fix(scheduler): hold allocLock when removing a pod's allocation by @saiyam1814 in #3350
  • fix(chart): add actual NVML names to MIG allowlist by @usr-bin-ksh in #3352

📚 Documentation

🔨 Other Changes

  • chore: add new linters to golangci-lint config by @dongjiang1989 in #2794
  • chore: upgrade to go 1.27 by @spencercjh in #2851
  • expose scheduler kube-qps, kube-burst and kube-timeout as values by @im-Toqeer-506 in #2905
  • build(deps): bump github/codeql-action from 4.37.8 to 4.37.9 by @Shouren in #2927
  • ✏️ rename DCU(HYGON) to HCU across resources, charts, and docs by @kinglong08 in #2969
  • chore(dashboard): remove legacy Grafana dashboard by @togettoyou in #3031
  • test(util): cover the delete-and-recreate race PatchPodAnnotations' UID pin closes by @moezdil in #3061
  • chore: remove the docs now hosted on the website by @moezdil in #3050
  • chore: add moezdil to approvers by @moezdil in #3145
  • chore(chart): add admission.k8s.io/v1 to webhook admissionReviewVersions by @blackdragoon26 in #3158
  • test(iluvatar): add coverage for AddResourceUsage by @karthikeyan-lgk in #3151
  • chore: remove the HCU doc transferred to the website by @spencercjh in #3198
  • chore: remove the docs transferred to the website by @moezdil in #3200
  • test(e2e): add an AMD GPU end-to-end suite by @moezdil in #3193
  • Remove requesting annotations by @archlitchi in #3227
  • chore: build the images with golang 1.27.2 by @moezdil in #3219
  • chore(deps): group golang.org/x/* packages by @Shouren in #3280
  • deprecation(vGPUmonitor): deprecate hami_container_device_memory_bytes by @loiht2 in #3273
  • chore(dependabot): group some packages under k8s.io by @Shouren in #3292
  • deprecation(scheduler): deprecate the limit label of hami_resource_quota_used metric by @loiht2 in #3299
  • build: bump Go to 1.27.3 and pin the dcgm base image by @moezdil in #3300
  • build: bump DCGM to 4.7.0-2 and go-dcgm to v1.4702.0 by @moezdil in #3303
  • build(deps): bump golang.org/x/net from 0.61.0 to 0.62.0 by @moezdil in #3331
  • test: add e2e tests for limits-file delivery feature by @maishivamhoo123 in #3355
  • Update HAMi-core and image tag by @archlitchi in #3362

New Contributors

Full Changelog: v2.10.0...v2.11.0

Check out latest releases or
releases around Project-HAMi/HAMi v2.11.0

Don't miss a new HAMi release

NewReleases is sending notifications on new releases.

Get notifications