ultralytics/ultralytics v8.4.118

v8.4.118 - Add standalone LLM model interface (#25761)

on GitHub

🌟 Summary

Ultralytics v8.4.118 introduces a standalone OpenAI-compatible LLM interface alongside YOLO, while improving OBB training, dataset handling, model training reliability, and documentation workflows. 🚀

📊 Key Changes

• 🤖 New standalone LLM model interface by @glenn-jocher

  • Add from ultralytics import LLM for text and image-based language-model requests.
  • Supports OpenAI Responses and Chat Completions APIs, including synchronous and asynchronous calls.
  • Accepts images from local paths, URLs, data URLs, NumPy arrays, and PIL images.
  • Supports reusable prompts, request overrides, conversation state, API keys, and OpenAI-compatible service endpoints.
  • Uses the optional openai dependency and remains independent of Ultralytics Platform and workflow-runtime components.

• 📐 Improved oriented bounding box training

  • Mosaic, CutMix, and RandomPerspective now preserve OBB orientation when objects are clipped by image boundaries.
  • Prevents clipped objects from receiving incorrect rotation angles during training.

• ⚡ Faster CopyPaste augmentation

  • Batches instance concatenation instead of repeatedly copying growing arrays.
  • Reduces unnecessary processing overhead, especially when many objects are copied.

• 🧠 More reliable YOLOE behavior

  • Validates visual prompts before modifying model state.
  • Accepts flat prompts for supported batched image sources.
  • Rejects invalid string class labels and mismatched vocabularies earlier with clearer errors.
  • Preserves gradient settings when converting YOLOE convolution layers to linear layers.

• 🏋️ Training and inference stability fixes

  • Correctly resets dataloader workers when resuming after Mosaic augmentation is closed.
  • Allows repeated train() and tune() calls on the same model object.
  • Prevents duplicate World model callbacks across multi-dataset training.
  • Fixes classification prediction for models without predefined transforms.
  • Ensures classification validation loaders do not discard samples when compiling.
  • Fixes SAM and related predictor models being created with incompatible inference-only tensors.

• 🗂️ Dataset and prediction improvements

  • Classification auto-splitting now recognizes all supported image formats, including JPEG, BMP, WebP, TIFF, AVIF, HEIC, and uppercase extensions.
  • Missing classification images now raise a clear FileNotFoundError instead of failing later with an unrelated directory error.
  • Preserves original filenames when loading images after EXIF correction.
  • Keeps bounding-box fallbacks for malformed grounding segmentation labels.

• 📚 Documentation and deployment updates

  • Standardizes strict documentation validation on Zensical and updates contributor instructions.
  • Documentation redeployment now detects Python docstring and all configuration-file changes.
  • Restores model benchmark chart placeholders, including for YOLO26, while moving production site features to the centralized publisher.
  • Documents replacing the model behind an existing deployment without changing its endpoint URL, API key, or deployment identity.
  • Updates Albumentations examples for current 2.x constructor names and refreshes the Rust inference dependency to 0.0.34.
  • Adds API reference documentation for the new LLM interface.

🎯 Purpose & Impact

• 🚀 Broader AI capabilities: Developers can now use Ultralytics as a unified entry point for YOLO vision models and OpenAI-compatible language models, including multimodal image understanding.
• 🔌 Flexible integration: The new LLM class works with OpenAI and compatible providers without requiring Platform or workflow features.
• 🎯 Better OBB accuracy: Rotated-object datasets should receive more consistent training targets when augmentation crops objects at image edges.
• ⚡ Improved performance: CopyPaste augmentation can run more efficiently, particularly on images containing many instances.
• 🛡️ More predictable training: Resume, compile, repeated training, World models, SAM, and classification workflows are less likely to produce silent errors or invalid results.
• 🧰 Easier maintenance: Broader dataset format support, clearer validation errors, and more accurate documentation reduce setup and debugging time for users.

What's Changed

• Preserve OBB orientation through clipped augmentations by @Nikhi00718 in #25723
• Redeploy docs when Python or cfg sources change by @glenn-jocher in #25733
• Standardize Docs validation on Zensical by @glenn-jocher in #25736
• Simplify strict Docs validation and restore model charts by @glenn-jocher in #25738
• Document deployment model replacement by @glenn-jocher in #25745
• Remove the YOLO11 podcast audio embed by @raimbekovm in #25765
• Align grounding segmentation fallback by @raimbekovm in #25743
• Batch the concatenation loop in CopyPaste.apply_instances to avoid O(n^2) growth by @JESUSROYETH in #25732
• Update the Albumentations examples to the 2.x constructor arguments by @raimbekovm in #25759
• Standardize export data defaults by @raimbekovm in #25742
• Cache external test assets before pytest by @glenn-jocher in #25737
• Accept all image formats in classification auto-split and report missing images instead of crashing by @doublecurry in #25749
• Validate yoloe visual prompts by @raimbekovm in #25744
• Reject a short YOLOE vocabulary before the head is re-parameterized by @raimbekovm in #25747
• Build loaded models outside inference mode by @raimbekovm in #25748
• Register the World pretrain hook once per model by @raimbekovm in #25758
• Build the name=model save directory from the model stem by @raimbekovm in #25763
• Reset dataloader workers when resuming after mosaic closure by @JESUSROYETH in #25762
• Restore the model override after training by @raimbekovm in #25760
• Fix: autocast_list loses image filename after exif_transpose by @Q-qqq in #25741
• Fix classification predict crash when the model has no transforms by @ahmet-f-gumustas in #25735
• Build SAM predictor models outside inference mode by @raimbekovm in #25756
• fix: only apply drop_last to the classify train loader under compile, not val by @JESUSROYETH in #25734
• Revert "Build loaded models outside inference mode" by @glenn-jocher in #25768
• Name the ImageNet pseudo-label teacher checkpoints and correct stored dtype by @Bovey0809 in #25754
• Bump ultralytics-inference docs version to 0.0.34 by @onuralpszr in #25778
• Add standalone LLM model interface by @glenn-jocher in #25761

New Contributors

• @doublecurry made their first contribution in #25749

Full Changelog: v8.4.117...v8.4.118