github Project-HAMi/HAMi v2.10.0
on GitHub

2 hours ago

General

DRA

Scheduling Features

Device Support

HAMi-core

What's Changed

✨ New Features

🐛 Bug Fixes

  • fix device typos by @DSFans2014 in #1608
  • fix: precedence bug in schedulerName check by @hoteye in #1627
  • fix: add nil checks to prevent panics in leaderelection by @haitwang-cloud in #1603
  • fix: panic on nil resourceReqs in scheduler calcScore by @yxxhero in #1626
  • Fix the issue Iluvatar device scheduling policy binpack and spread are reversed by @qiangwei1983 in #1631
  • Fix contact link and add slack channel to README.md by @archlitchi in #1635
  • fix: resolve cardinality explosion in Device_memory_desc_of_container by @maishivamhoo123 in #1628
  • fix: handle GetMemoryInfo ERROR_NOT_SUPPORTED for unified memory GPUs by @jsl9208 in #1637
  • fix typos by @DSFans2014 in #1657
  • Fix: optimize nodelock scalability with exponential backoff and listers by @maishivamhoo123 in #1663
  • fix apply kubescheduler config version by @CoderTH in #1666
  • fix: failing readiness probe when replica > 1 by @Shouren in #1677
  • fix: ci deepth by @Atroxgod in #1690
  • fix(scheduler): correct slot usage prediction and add device type fil… by @maishivamhoo123 in #1700
  • fix vastai fit loop iteration direction by @DSFans2014 in #1715
  • fix: retain terminating pod in cache to prevent premature eviction by @maishivamhoo123 in #1719
  • fix(chart): derive ld.so.preload from devicePlugin.libPath to fix non-default path deployments by @ilia-medvedev in #1714
  • fix: support device allocation for multi-container with init containers by @haitwang-cloud in #1650
  • fix: Reponse with correct content-type by @Shouren in #1604
  • fix: suppress scheduler cleanup noise for unrelated vendors by @Yonsun-w in #1749
  • fix(scheduler): add missing return in ondelpod by @CFH2436 in #1759
  • fix: global image tag always covers per-component image tag by @FouoF in #1774
  • fix(device-plugin): align kubelet allocation with scheduler annotations (#1741) by @xrwang8 in #1743
  • fix(chart): sanitize managedNodeSelector keys for environment variables by @almazkhalikov in #1783
  • fix: filter device does not work by @DSFans2014 in #1817
  • Fix: Handle Kernel 6.17 handshake edge cases in NVIDIA health checks by @maishivamhoo123 in #1810
  • fix: git errors when building with latest hami-core by @Shouren in #1782
  • fix: mig does not work by @DSFans2014 in #1819
  • fix(scheduler): guard against zero-value division in ComputeScore (#1… by @lin121291 in #1820
  • fix(ascend): check unmarshal error before iterating node devices by @mesutoezdil in #1831
  • fix(device): parse handshake annotation timestamp in local timezone by @mesutoezdil in #1816
  • fix: recover scheduling on nodes with stale Deleted_ handshake by @saiyam1814 in #1843
  • fix A100-80G template by @DSFans2014 in #1847
  • fix: allocation failed when using mig in CDI mode by @DSFans2014 in #1826
  • fix: propagate parse errors in DecodeNodeDevices by @mesutoezdil in #1855
  • fix: return error from DecodePodDevices on decode failure by @mesutoezdil in #1857
  • fix: propagate parse errors in DecodeContainerDevices by @mesutoezdil in #1856
  • Fix e2e for HAMi by @archlitchi in #1884
  • Fix time-out logic incorrectly triggered by @archlitchi in #1879
  • fix(e2e): default helm version from VERSION file instead of v2.4.1 by @archlitchi in #1886
  • fix(quota): clamp Used to zero in RmUsage to prevent negative tracking by @mesutoezdil in #1880
  • fix(e2e): picks kubeconfig in order: --kubeconfig flag, KUBE_CONF, ~/.kube/config. by @FouoF in #1902
  • fix(vGPUmonitor): prevent index out of bounds panic for priority above 1 by @mesutoezdil in #1881
  • fix(quota): roll back usage when Filter evicts a stale pod entry by @mesutoezdil in #1905
  • fix: rollback quota usage when filter annotation patch fails by @bolin-dai in #1898
  • fix(chart): add DRA resource RBAC for kube-scheduler on K8s 1.34+ by @spencercjh in #1917
  • fix: replace --build-arg TARGET_ARCH with --platform in docker build by @haitwang-cloud in #1921
  • fix(scheduler): clean up podManager on Bind failures to prevent resource leak by @peachest in #1927
  • fix(charts): change scheduler anti-affinity to preferred for rolling … by @lvkaili in #1934
  • fix(examples): consolidate nvidia example and update changelog by @haitwang-cloud in #1938
  • Fix: Skip nvidia-mig-parted when MIG is disabled by @maishivamhoo123 in #1930
  • fix_leaderNotify Block by @archlitchi in #1957
  • fix(device): CheckUUID ignores noUseKey when useKey is also set by @mesutoezdil in #1965
  • fix(device): prevent panic in CheckHealth when Requesting has no timestamp by @mesutoezdil in #1964
  • fix(quota): clamp Used to zero in RmUsage to prevent negative tracking by @mesutoezdil in #1966
  • fix: return snapshot in InspectAllNodesUsage to avoid data race by @DSFans2014 in #1967
  • fix(chart): declare metrics port on vGPU monitor container by @ntheanh201 in #1990
  • fix(nvidia): validate gpumem-percentage range to prevent silent unschedulability by @saiyam1814 in #1997
  • fix(webhook): count the resource to fit quota in webhook when reqnum > 1 by @DSFans2014 in #2001
  • fix(device-plugin): surface NVML error when MIG device placement cannot be determined by @saiyam1814 in #1999
  • fix(vgpu-init): correct source and destination path handling by @haitwang-cloud in #2018
  • fix(nvidia): refresh cache on register annotation changes by @LynPtl in #2022
  • fix(ascend): reject -core on hard-split vNPU and fix "nor"->"not" typo by @Wangmin362 in #2026
  • fix(ascend): stop whole-card/memory-less hami-core requests from bypassing the node mode gate by @Wangmin362 in #2029
  • fix(scheduler): accept a vastai-only or biren-only device configuration by @Wangmin362 in #2050
  • fix(device-plugin): avoid nil pointer panic in CheckHealth when checkHealth returns nil by @Wangmin362 in #2043
  • fix(scheduler): treat an empty use/nouse gpuuuid annotation as no constraint by @Wangmin362 in #2045
  • fix(cambricon): avoid scheduler panic when a leading container requests no MLU by @Wangmin362 in #2053
  • fix: skip owner check for legacy lock format by @DSFans2014 in #2054
  • fix: return deep copies from ListPodsInfo to prevent data race by @mesutoezdil in #2055
  • fix: correct typos in comments and docs by @mesutoezdil in #2057
  • fix(scheduler): avoid the racy read of the node cache in register() by @Wangmin362 in #2068
  • fix(CI): invalid generated chart SemVer by @FouoF in #2073
  • fix(scheduler): do not exit on a transient error updating the leader label by @Wangmin362 in #2075
  • fix(scheduler): scope per-type device list in fitInDevices by @imantaba in #2105
  • fix webhook to deny privileged containers by @Jay2006sawant in #2139
  • fix: bound int32 conversion in enflame device parsing by @mesutoezdil in #2145
  • fix(metrics): resolve high-cardinality leak in memory_allocated_bytes by @Aryanbhargava18 in #2142
  • fix(mthreads): guard against zero device count in GenerateResourceRequests by @imantaba in #2134
  • fix(enflame): bound profile values before int32 conversion in Fit by @mesutoezdil in #2190
  • fix(ascend): update ascend config according to latest driver by @FouoF in #2223
  • fix(metrics): prevent invalid memory ratios for unknown total memory by @blackdragoon26 in #2204
  • fix(scheduler): handle ResourceQuota tombstones by @blackdragoon26 in #2259
  • fix(monitor): fix use-after-unmap crash in loadCache + raise cudevshr.go coverage to 90% by @adity1raut in #2194
  • fix(device): guard nil maps in mthreads MutateAdmission by @imantaba in #2254
  • fix(scheduler): preserve concurrent node locks by @ShiroKSH in #2197
  • fix(device): correct AllocatedCardsInsufficientRequest count during partial allocation by @princexpoddar in #2238
  • fix(vGPUmonitor): scan all devices in CheckBlocking by @imantaba in #2159
  • fix(nvidia): handle zero-score topology combinations by @Eshiv-Pandey in #2160
  • fix(nodelock): let a pod re-acquire its own still-valid node lock by @adity1raut in #2255
  • fix(device): use HasPrefix and TrimPrefix in DelQuota by @SK8-infi in #2272
  • fix: release node lock on allocate response failure by @cotishq in #2293
  • fix(ascend): skip unhealthy devices in Fit() by @Eshiv-Pandey in #2241
  • fix(device-plugin): release node lock when bind-phase patch fails by @blackdragoon26 in #2307
  • fix(vGPUmonitor): bound v0 spec metric sums to active proc slots by @imantaba in #2282
  • fix(mthreads): correct invalid memory slice value in error message by @Eshiv-Pandey in #2325
  • fix(device): use len(devices) for GenReason in kunlun and awsneuron by @blackdragoon26 in #2292
  • fix(scheduler): return deep copy in nodeManager.GetNode to prevent concurrent map read/write by @AyushSrivastava1818 in #2333
  • fix(nvidia): treat gpumem-percentage of 0 as unset by @Wangmin362 in #2156
  • fix(scheduler): return early when predicate body is nil by @hemantrajputt in #2299
  • fix(scheduler): enforce ResourceQuota for every device backend by @Lakshya77089 in #2347
  • fix: add missing metric descriptors to Describe() in vGPUmonitor by @devGPP23 in #2240
  • fix(scheduler): apply ResourceQuota updates without dropping the limit by @Lakshya77089 in #2386
  • fix(monitor): skip dirs without underscore in Update instead of panicking by @Nakshatra480 in #2371
  • fix(vGPUmonitor): skip uninitialized short UUIDs instead of exit by @SK8-infi in #2364
  • fix(metrics): replace dangerous MustNewConstMetric with safe helper in scheduler by @devGPP23 in #2326
  • fix(monitor): clamp SetDeviceSmLimit and SetDeviceMemoryLimit to maxDevices by @Nakshatra480 in #2362
  • fix(device): don't panic on asymmetric GPU P2P link data in calculateGPUPairScore by @Brijesh-Thakkar in #2306
  • fix(scheduler): prevent nil body panic bug in Predicate and Bind handlers by @devGPP23 in #2384
  • fix(vGPUmonitor): skip exited process slots in v1 spec metric aggregations by @Nakshatra480 in #2314
  • fix (ascend) - validate module pair count and prevent under allocation for ascend 910C by @princexpoddar in #2369
  • fix(monitor): clamp DeviceNum to maxDevices to prevent scrape-path panic by @Nakshatra480 in #2400
  • fix(quota): honor an explicit zero limit in ResourceQuota by @veyron-kairo in #2313
  • fix(scheduler): set LimitSet in the remaining ResourceQuota test fixtures by @adity1raut in #2432
  • fix(device-plugin): return error on failed Send in ListAndWatch by @AyushSrivastava1818 in #2353
  • fix(device-plugin): skip devices that do not support event based healthchecking by @Wangmin362 in #2393
  • fix(device,plugin): stop MIG usage corruption and fd leak by @adity1raut in #2245
  • fix(amd): empty nouse-gputype annotation should not exclude every device by @adity1raut in #2395
  • fix(device): skip empty comma-separated members in shared CheckType by @adity1raut in #2435
  • fix(device-plugin): resolve stale annotation in multi-container Allocate by @peachest in #2380
  • fix: skip unhealthy devices in Fit() for all non-nvidia backends by @Eshiv-Pandey in #2260
  • fix(metrics): normalize AMD core allocation ratio to percentage by @yxxhero in #2527
  • Fix client test ignoring TEST_WITH_REAL_CLUSTER by @KunwarSidhu47 in #2521
  • fix(ascend): guard nil Requests in MutateAdmission by @keshav9926 in #2416
  • fix(util): handle nil pod gracefully in utility functions to prevent … by @Rickydama3 in #2499
  • fix(monitor): ignore unrelated files when loading cache by @hyy321 in #2529
  • fix(cambricon): enforce ResourceQuota for percentage/whole-card memory requests by @adity1raut in #2536
  • fix(monitor): use IsValidUUID to skip uninitialised device UUIDs in scrape path by @Nakshatra480 in #2465
  • fix(scheduler): copy node info into nodeManager instead of storing the caller's by @Lakshya77089 in #2504
  • fix(monitor): add v0.MinSize and fix file-size discriminator for v0 cache by @Nakshatra480 in #2503
  • fix(device): hand out copies from PodManager instead of stored pointers by @Lakshya77089 in #2472
  • fix(nvidia): clean up node health bookkeeping on deletion by @AyushSrivastava1818 in #2590
  • fix(cambricon): prevent node lock leaks by reading from apiserver in ReleaseNodeLock by @jianzhangbjz in #2329
  • fix(hygon,metax): guard int32 narrowing in GenerateResourceRequests by @Eshiv-Pandey in #2388
  • fix(awsneuron): handle init-container annotation indexes by @Eshiv-Pandey in #2173
  • fix: Handle HTTP response write errors in scheduler routes by @shinigami-777 in #2571
  • Fix bug about auto-memory trim when using hami-core mode for vNPU by @archlitchi in #2696
  • fix(scheduler): remove stale device vendors on zero-device update by @AyushSrivastava1818 in #2550
  • fix(metrics): add device_type label to node memory ratio metrics (#2370) by @princexpoddar in #2554
  • fix(vGPUmonitor): add missing node label to host GPU metrics by @SK8-infi in #2580
  • fix(ascend): guard int32 narrowing in GenerateResourceRequests by @Eshiv-Pandey in #2601
  • fix(scheduler): detect stale pod device allocations during usage reconciliation by @AyushSrivastava1818 in #2618
  • fix(scheduler): rollback partial device locks by @AyushSrivastava1818 in #2626
  • fix test crash in client test by @KunwarSidhu47 in #2683
  • fix(scheduler): allow multiple MIG instances on one GPU by @saiyam1814 in #2724
  • fix(awsneuron): prevent odd NeuronCore request truncation by @Eshiv-Pandey in #2674
  • fix(scheduler): don't cache a nil NodeUsage when node is missing from snapshot by @adity1raut in #2436
  • fix(scheduler): reconcile health on device discovery error by @AyushSrivastava1818 in #2568
  • fix(kunlun): honour use-gpuuuid and nouse-gpuuuid annotations by @Lakshya77089 in #2577
  • fix(device): reject int32-overflowing memory requests in iluvatar and mthreads by @veyron-kairo in #2285
  • fix(scheduler): reject filter requests with no Pod instead of panicking by @adity1raut in #2689
  • fix: use atomic load/store for shared-region limit fields by @om7057 in #2179
  • fix(device-plugin): make MigInstanceManager own its NVML init/shutdown by @AdityaShome in #2610
  • fix(scheduler): preserve per-container slot count when collapsing pod device usage by @Eshiv-Pandey in #2623

📚 Documentation

  • docs: remove stale tasklist.md, migrate content to separate issues by @ManishSharma1609 in #1801
  • docs: add CNCF copyright disclaimer to README files by @mesutoezdil in #1830
  • docs: complete tasklist.md migration - add hardware entries, resource… by @maishivamhoo123 in #1829
  • docs: add v2.9.0 documentation audit report by @mesutoezdil in #1761
  • docs: rewrite English README with improved structure and clarity by @rootsongjc in #1849
  • docs: remove legacy docs/ content superseded by project-hami/website by @mesutoezdil in #1913
  • docs: add ecosystem integrations section to README by @mesutoezdil in #1970
  • docs: update supported versions to v2.9.x by @mesutoezdil in #1976
  • docs: fix typo and annotation keys in protocol.md by @mesutoezdil in #1978
  • docs: mark Biren as completed in roadmap by @DSFans2014 in #1979
  • docs: fix typo and mark AMD and MIG as supported in roadmap by @mesutoezdil in #1975
  • docs: fix typos in design.md by @mesutoezdil in #1974
  • docs: fix Spread score description in scheduler-policy.md by @mesutoezdil in #1977
  • docs: add issue and PR lifecycle policy to CONTRIBUTING.md by @mesutoezdil in #1988
  • docs: add AMD Instinct vGPU design draft (#1707) by @kenji-mido in #1985
  • docs(CONTRIBUTING): add guideline to separate cosmetics from function… by @fishman in #1995
  • docs: fix typo and wrong repo URL in release-process.md by @mesutoezdil in #2008
  • docs: add missing changelog entries for v2.4.0 through v2.9.0 by @mesutoezdil in #2007
  • docs: add contribution gates to AI assistance policy by @mesutoezdil in #2019
  • docs: add vulnerability scope note to security policy by @mesutoezdil in #2063
  • docs(MAINTAINERS): add community managers to offload administrative tasks by @fishman in #2069
  • docs: clarify claiming roadmap items before starting work by @mesutoezdil in #2079
  • docs: design for init-container GPU resource accounting by @maishivamhoo123 in #2064
  • docs: add SECURITY-INSIGHTS.yml security metadata file by @mesutoezdil in #2093
  • docs: drop stale glibc upper bound from prerequisites by @mesutoezdil in #2102
  • docs: declare secret scanning tooling in SECURITY-INSIGHTS.yml by @mesutoezdil in #2113
  • docs: fix annotation key and encoding format in protocol.md by @chidwipak in #2185
  • docs: correct stale golangci-lint version references in CLAUDE.md by @asadjan4611 in #2402
  • docs: add AWS Neuron device to roadmap support table by @KunwarSidhu47 in #2422
  • docs: document ResourceQuota interaction for init container accounting. by @maishivamhoo123 in #2535
  • docs: add Kunlunxin XPU to supported devices list (Fixes #2608) by @KunwarSidhu47 in #2609
  • docs: fix typo in Helm chart README (#2585) by @Kanika0306 in #2587
  • docs: design for sidecar container GPU resource accounting by @maishivamhoo123 in #2584
  • docs(chart): document all default Kunlunxin custom resources by @asadjan4611 in #2706
  • docs: make the bug-report template accelerator-neutral by @asadjan4611 in #2668

🔨 Other Changes

  • add device type label in metrics by @xiyichan in #1612
  • security: add io.LimitReader to scheduler routes to prevent DoS #554 by @maishivamhoo123 in #1620
  • add fouof to approvers by @FouoF in #1642
  • Update Maintainers information by @archlitchi in #1646
  • [Snyk] Security upgrade tensorflow/tensorflow from 2.20.0rc0-gpu to 2.21.0rc0-gpu by @wawa0210 in #1652
  • chore: remove deprecated scheduler policy configmap by @haitwang-cloud in #1651
  • [Snyk] Security upgrade tensorflow/tensorflow from 2.21.0rc0-gpu to 2.21.0rc1-gpu by @wawa0210 in #1681
  • vGPUmonitor: skip devices with invalid UTF-8 UUID during container init by @charford in #1703
  • Helm - Render nvidia.overwriteEnv from values with a default of false. by @jcustenborder in #1706
  • Update dashboard.md fix nvidia.com/gpucores comment by @Nov11 in #1712
  • update nvidia_dp and nvidia_container_runtime module by @archlitchi in #1731
  • chore: resolve staticcheck and modernize linter warnings by @maishivamhoo123 in #1728
  • chore: using LeaderElectionConfiguration for kubernetes v1.23 and newer version by @Shouren in #1737
  • Add general Technical review to documents by @archlitchi in #1752
  • test: improve coverage for pkg/version/version.go by @Yonsun-w in #1748
  • Configure ASCEND_VISIBLE_DEVICES env for container and RuntimeClassName for pods by @peachest in #1738
  • test: improve nvidia device.go UT coverage to 98.5% by @kenwoodjw in #1757
  • security: upgrade golang for security issue by @Shouren in #1772
  • Upgrade Go to 1.26.2 by @luohua13 in #1791
  • update benchmarks by @maverick-woo in #1790
  • add DSFans2014 to approvers by @DSFans2014 in #1805
  • Disable host network for device plugin by @luohua13 in #1789
  • bump HAMi-DRA version to v0.2.0 by @FouoF in #1845
  • Preparations to release v2.9.0 by @archlitchi in #1850
  • Sync the Chinese and Japanese README with the English version by @rootsongjc in #1852
  • Update example images and captions in README files by @rootsongjc in #1873
  • chore: update golangci-lint version and fix lint violations by @dongjiang1989 in #1892
  • test: add unit tests for nvinternal info, mig, and watch packages by @pmady in #1893
  • build(deps): bump github.com/ccoveille/go-safecast/v2 by @ccoVeille in #1914
  • chore(e2e): allow concurrent running in one runner by @FouoF in #1918
  • test(util): add coverage for GetGPUSchedulerPolicyByPod by @mesutoezdil in #1920
  • test(util): add coverage for SchedulerPolicyName.String by @mesutoezdil in #1919
  • test(nodelock): add coverage for setupNodeLockTimeout by @mesutoezdil in #1928
  • test(policy): add coverage for NodeScore.SnapshotDevice by @mesutoezdil in #1926
  • chore(e2e): no longer delete namespace when clean up by @FouoF in #1939
  • examples: add vLLM cross-vGPU tensor parallel deployment by @haitwang-cloud in #1947
  • chore(scheduler): remove unused default resource flags by @zzzkl in #1953
  • build(docker): align HAMi with HAMi-core compile image by @spencercjh in #1958
  • build(docker): fix stale cuda 13 runtime references by @spencercjh in #1961
  • cleanup: tidy go.sum to match go.mod by @usernameisnull in #1973
  • cleanup: remove dead code by @usernameisnull in #1972
  • community: add mesutoezdil as reviewer by @mesutoezdil in #1983
  • Export nvidia device plugin methods for ascend device plugin by @peachest in #1989
  • deprecation: remove labels marked as to be deprecated by @FouoF in #2021
  • Update project status to Incubating in documentation by @rootsongjc in #2032
  • remove dra from charts by @archlitchi in #2038
  • Remove 'deleted' annotations when registering by @archlitchi in #2052
  • chore: remove dead code by @mesutoezdil in #2060
  • Remove postStart hook for hami-core-vNPU tasks by @archlitchi in #2062
  • remove unused CtrIDs from PodInfo by @DSFans2014 in #2071
  • device-plugin: opt-in NUMA topology for vGPU replicas by @saiyam1814 in #2065
  • chore: remove unreviewable binary artifact imgs/hami-arch.pptx by @mesutoezdil in #2097
  • chore: remove dead script, boilerplate file, and unused ParseConfig stubs by @mesutoezdil in #2098
  • chore: remove unreferenced images from imgs/ by @mesutoezdil in #2099
  • build: bump Go toolchain to 1.26.5 by @mesutoezdil in #2092
  • test: add native fuzz test for DecodeContainerDevices by @mesutoezdil in #2146
  • test: add tests for predicate and bind routes, including error handling by @adity1raut in #2136
  • chore: remove unused variables by @DSFans2014 in #2189
  • test(rm): raise nvml_devices.go coverage up to 80% by @adity1raut in #2177
  • test(kunlun): raise vdevice.go coverage up to 95% by @adity1raut in #2176
  • chore(metrics): remove leftover client_golang tutorial scaffolding by @veyron-kairo in #2263
  • test: add fuzz and round-trip tests for node/pod device annotation pa… by @asadjan4611 in #2376
  • test(device-plugin): Improve test coverage for device-plugin/nvidiadevice/nvinternal/cdi by @shinigami-777 in #2351
  • test(vGPUmonitor): expect short UUID to be skipped, not error by @imantaba in #2407
  • Missing Test for Node Discovery Handshake on Malformed/Empty Annotations by @aniket866 in #2368
  • cleanup(metax): decouple MetaxSDevices.ScoreNode from the scheduler policy string by @asadjan4611 in #2413
  • test(plugin): fix TestGetNumaNode failure caused by existing PCI bus ID by @shinigami-777 in #2546
  • test(device): fuzz container codec round trip by @blackdragoon26 in #2583
  • chore: bump Go builder to 1.26.6 by @mesutoezdil in #2710
  • ci: move release workflow write permissions to job level by @mesutoezdil in #2711
  • chore: bump golang.org/x/mod to v0.40.0 by @mesutoezdil in #2747

New Contributors

Full Changelog: v2.8.0...v2.10.0

Check out latest releases or
releases around Project-HAMi/HAMi v2.10.0

Don't miss a new HAMi release

NewReleases is sending notifications on new releases.

Get notifications