π Summary
More stable training by skipping non-finite loss batches, smoother RT-DETR device/precision moves, and better ONNX profiling for multi-input/dynamic-shape models. π
π Key Changes
-
Training stability
- Skip non-finite loss batches (NaN/Inf) during forward pass with a clear warning: βNon-finite forward pass, skipping batch...β π‘οΈ (See PR: Skip non-finite training forward passes)
- Clean initialization of running training loss (tloss) for more consistent logging π
- Version bump to 8.3.210 π§©
-
RT-DETR device/precision consistency
- Added a custom
_apply
toRTDETRDetectionModel
so anchors and masks follow.to()
,.cuda()
,.cpu()
,.half()
, etc. π (See PR: Add _apply to RTDETR)
- Added a custom
-
ONNX benchmarking improvements
- Profiling now supports multi-input models and dynamic shapes with sensible defaults and clear validation errors β
- Builds input dictionaries for all inputs and preserves correct dtypes during warmup and timed runs β±οΈ (See PR: Support profiling for multi-input ONNX models)
π― Purpose & Impact
-
More reliable training out of the box
- Prevents crashes and bad updates from invalid lossesβespecially helpful with mixed precision (FP16/BF16) and aggressive hyperparameters βοΈ
- You may notice occasional warnings and slightly fewer processed batches per epoch; overall convergence should be steadier π
-
Fewer device/dtype gotchas for RT-DETR
- Seamless moves across CPU/GPU and precision changes without manual fixes, reducing edge-case bugs in training, inference, export, and deployment π§
-
Robust and accurate ONNX profiling
- Benchmark complex ONNX models (multiple inputs, dynamic shapes) without tedious setup, enabling clearer performance comparisons and faster iteration π
Quick update tip:
- Upgrade to the latest version:
pip install -U ultralytics
β
What's Changed
- Add
_apply
method to RTDETR for betterdevice
consistency by @Y-T-G in #22341 - Support profiling for ONNX models with multiple inputs by @lmycross in #22262
ultralytics 8.3.210
Skip non-finite training forward passes by @glenn-jocher in #22353
Full Changelog: v8.3.209...v8.3.210