ultralytics/ultralytics v8.4.86

v8.4.86 - Fix select_device re-request after CUDA_VISIBLE_DEVICES remap applied (#25018)

on GitHub

🌟 Summary

Ultralytics v8.4.86 fixes a CUDA device selection regression that could interrupt training when using nonzero GPU indices, restoring reliable multi-GPU workflows. 🚀

📊 Key Changes

• Fixed CUDA device re-selection after remapping 🛠️
  Resolved an issue where training with a command like device=3 could fail later during validation or final evaluation after CUDA_VISIBLE_DEVICES remapped that GPU to cuda:0.

• Restored expected behavior for remapped GPUs 🔁
  If CUDA_VISIBLE_DEVICES already matches the requested device, Ultralytics now correctly recognizes that CUDA has already applied the remap and returns the proper visible device index.

• Added regression test coverage ✅
  A new test covers the specific case where a nonzero physical GPU, such as GPU 3, is remapped to the single visible CUDA device cuda:0 after CUDA initialization.

• Version bump 📦
  Updated the package version from 8.4.85 to 8.4.86.

🎯 Purpose & Impact

• Prevents unexpected training crashes 💥➡️✅
  Users training on a specific nonzero GPU, for example device=3, should no longer see invalid CUDA device errors during later training stages such as trainer.final_eval().

• Improves production reliability 🏭
  This is especially important for servers, clusters, and shared GPU environments where jobs are often assigned to GPUs using CUDA_VISIBLE_DEVICES.

• Keeps recent validation improvements intact 🔒
  The fix preserves the stricter device validation introduced previously while correcting the special case where CUDA remapping has already taken effect.

• Better support for multi-GPU systems 🖥️
  Developers and teams using machines with multiple GPUs can expect more predictable behavior when selecting specific GPU indices.

• No user action required beyond upgrading ⬆️
  Install the latest release with:

  pip install -U ultralytics

What's Changed

• Fix select_device re-request after CUDA_VISIBLE_DEVICES remap applied by @glenn-jocher in #25018

Full Changelog: v8.4.85...v8.4.86