github mudler/LocalAI v3.5.0
on GitHub

one day ago




🚀 LocalAI 3.5.0

Welcome to LocalAI 3.5.0! This release focuses on expanding backend support, improving usability, refining the overall experience, and keeping reducing footprint of LocalAI, to make it a truly portable, privacy-focused AI stack. We’ve added several new backends, enhanced the WebUI with new features, made significant performance improvements under the hood, and simplified LocalAI management with a new Launcher app (Alpha) available for Linux and MacOS.

TL;DR – What’s New in LocalAI 3.5.0 🎉

  • 🖼️ Expanded Backend Support: Welcome to MLX! mlx, mlx-audio, mlx-vlm are now all available in LocalAI. We also added support to WAN for video generation, and a CPU and MPS version of the diffusers backend! Now you can generate and edit images from MacOS or if you don't have any GPU (albeit slow).
  • ✨ WebUI Enhancements: Download model configurations, a manual model refresh button, streamlined error streaming during SSE events, and a stop button for running backends. Models now can also be imported and edited via the WebUI.
  • 🚀 Performance & Architecture: Whisper backend has been rewritten in Purego with integrated Voice Activity Detection (VAD) for improved efficiency and stability. Stablediffusion also benefits from the Purego conversion.
  • 🛠️ Simplified Management: New LocalAI Launcher App (Alpha) for easy installation, startup, updates, and access to the WebUI.
  • ✅ Bug Fixes & Stability: Resolutions to AMD RX 9060XT ROCm errors, libomp linking issues, model loading problems on macOS, CUDA device detection improvements, and more.
  • Enhanced support for MacOS: whisper, diffusers, llama.cpp, MLX (VLM, Audio, LLM), stable-diffusion.cpp will now work on MacOS!

What’s New in Detail

🚀 New Backends and Model Support

We've significantly expanded the range of models you can run with LocalAI!

  • mlx-audio: Bring text to life with Kokoro’s voice models on MacOS with the power of MLX!. Install with the mlx-audio backend. Example configuration: 
    backend: mlx-audio
name: kokoro-mlx
parameters:
  model: prince-canuma/Kokoro-82M
  voice: "af_heart"
  known_usecases:
    - tts
  • mlx-vlm: Experiment with the latest VLM models. While we don't have any models in the gallery, it's really easy to configure, see #6119 for more details. 
    name: mlx-gemma
backend: mlx-vlm
parameters:
  model: "mlx-community/gemma-3n-E2B-it-4bit"
template:
  use_tokenizer_template: true
known_usecases:
- chat
  • WAN: Generate videos with Wan2.1 or Wan 2.2 models using the diffusers backend, supporting both I2V and T2V. Example configuration: 
    name: wan21
f16: true
backend: diffusers
known_usecases:
  - video
parameters:
  model: Wan-AI/Wan2.1-T2V-1.3B-Diffusers
diffusers:
  cuda: true
  pipeline_type: WanPipeline
  step: 40
options:
    - guidance_scale:5.0
    - num_frames:81
    - torch_dtype:bf16
  • Diffusers CPU and MacOS Support: Run diffusers models directly on your CPU without a GPU or with a Mac! This opens up LocalAI to a wider range of hardware configurations.

✨ WebUI Improvements

We've added several new features to make using LocalAI even easier:

  • Download Model Config: A "Get Config" button in the model gallery lets you download a model’s configuration file without installing the full model. This is perfect for custom setups and easier integration.
  • Manual Model Refresh: A new button allows you to manually refresh the on-disk YAML configuration, ensuring the WebUI always has the latest model information.
  • Streamlined Error Handling: Errors during SSE streaming events are now displayed directly to the user, providing better visibility and debugging information.
  • Backend Stop Button: Quickly stop running backends directly from the WebUI.
Screenshot From 2025-08-15 22-25-52
  • Model import and edit: Now models can be edited and imported directly from the WebUI.
Screenshot 2025-08-14 at 22-28-59 LocalAI - Import Model Screenshot 2025-08-14 at 22-28-47 LocalAI - Edit Model gpt-oss-20b
  • Installed Backend List: Now displays installed backends in the WebUI for easier access and management.

🚀 Performance & Architecture Improvements

  • Purego Whisper Backend: The Whisper backend has been rewritten in Purego for increased performance and stability. This also includes integrated Voice Activity Detection (VAD) for detecting speech.
  • Purego Stablediffusion: Similar to Whisper, Stablediffusion has been converted to Purego, improving its overall architecture and enabling better compatibility.

🛠️ Simplified Management – Introducing the LocalAI Launcher (Alpha)

We're excited to introduce the first version of the LocalAI Launcher! This application simplifies:

  • Installation
  • Startup/Shutdown
  • Updates
  • Access to the WebUI and Application Folder
Screenshot From 2025-08-26 11-46-35 Screenshot From 2025-08-26 11-46-50 Screenshot From 2025-08-26 11-46-21

Please note: The launcher is in Alpha and may have bugs. The macOS build requires workarounds to run due to binaries not yet signed, and specific steps for running it are needed: https://discussions.apple.com/thread/253714860?answerId=257037956022#257037956022.

✅ Bug Fixes & Stability Improvements

  • AMD RX 9060XT ROCm Error: Fixed an issue causing errors with AMD RX 9060XT GPUs when using ROCm. This error, "ROCm error: invalid device function", occurred because of device function incompatibility. The fix involves updating the ROCm image and ensuring the correct GPU targets are specified during compilation. Recommended kernel versions and verification steps for GPU detection are available [here](link to troubleshooting doc if created).
  • libomp Linking: Resolved a missing libomp.so issue on macOS Docker containers.
  • macOS Model Loading: Addressed a problem where models could not be loaded on macOS. This was resolved by bundling necessary libutf8 libraries.
  • CUDA Device Detection: Improved detection of available GPU resources.
  • Flash Attention: Set auto for flash_attention in llama.cpp, allowing the system to optimize performance.

Additional Improvements

  • System Backend: Added a new "system" backend path (LOCALAI_BACKENDS_SYSTEM_PATH or via command-line arguments) defaulting to /usr/share/localai/backends. This allows specifying a read-only directory for backends, useful for package management and system-wide installations.
  • P2P Model Sync: Implemented automatic synchronization of installed models between LocalAI instances within a federation. Currently limited to models installed through the gallery, and configuration changes are not synced. Future improvements will address these limitations.
  • Diffusers Image Source Handling: Enhanced image source selection in the diffusers backend, prioritizing ref_images over src for more robust loading behavior.
  • Darwin CI Builds: Added support for building some Go-based backends (Stablediffusion and Whisper) on Darwin (macOS) in the CI pipeline.

🚨 Important Notes

  • Launcher (Alpha): The LocalAI Launcher is in its early stages of development. Please report any issues you encounter. The MacOS build requires additional steps due to code signing.
  • Model Configuration Updates: Changes to model configuration files are not currently synchronized when using P2P model sync.

The Complete Local Stack for Privacy-First AI

LocalAI Logo

LocalAI

The free, Open Source OpenAI alternative. Acts as a drop-in replacement REST API compatible with OpenAI specifications for local AI inferencing. No GPU required.

Link: https://github.com/mudler/LocalAI

LocalAGI Logo

LocalAGI

A powerful Local AI agent management platform. Serves as a drop-in replacement for OpenAI's Responses API, supercharged with advanced agentic capabilities and a no-code UI.

Link: https://github.com/mudler/LocalAGI

LocalRecall Logo

LocalRecall

A RESTful API and knowledge base management system providing persistent memory and storage capabilities for AI agents. Designed to work alongside LocalAI and LocalAGI.

Link: https://github.com/mudler/LocalRecall

Thank you! ❤️

A massive THANK YOU to our incredible community and our sponsors! LocalAI has over 35.000 stars, and LocalAGI has already rocketed past 1100+ stars!

As a reminder, LocalAI is real FOSS (Free and Open Source Software) and its sibling projects are community-driven and not backed by VCs or a company. We rely on contributors donating their spare time and our sponsors to provide us the hardware! If you love open-source, privacy-first AI, please consider starring the repository, contributing code, reporting bugs, or spreading the word!

Full changelog 👇

👉 Click to expand 👈

What's Changed

Bug fixes 🐛

  • fix(llama.cpp): support gfx1200 by @mudler in #6045
  • fix: chmod grpc processes only if needed by @mudler in #6051
  • fix: do not show invalid backends by @mudler in #6058
  • fix(hipblas-llama.cpp): create symlink to libomp by @mudler in #6140
  • fix: select portable environment if detected by @mudler in #6158
  • fix: register backends to model-loader during installation by @mudler in #6159

Exciting New Features 🎉

  • chore(build): Convert stablediffusion-ggml backend to Purego by @richiejp in #5989
  • fix(backend-detection): default to CPU if there is less than 4GB of GPU available by @mudler in #6057
  • fix(llama-cpp/darwin): make sure to bundle libutf8 libs by @mudler in #6060
  • feat(backends): add system backend, refactor by @mudler in #6059
  • feat(webui): add import/edit model page by @mudler in #6050
  • chore(rocm): bump rocm image, add gfx1200 support by @mudler in #6065
  • feat(ui): General improvements by @mudler in #6072
  • Tune the "dark gray" font color in the webui to make it more readable by @lxnay in #6078
  • feat(diffusers): add torchvision to support qwen-image-edit by @mudler in #6103
  • feat(p2p): automatically sync installed models between instances by @mudler in #6108
  • feat(img2img): Add support to Qwen Image Edit by @mudler in #6113
  • feat(diffusers): add MPS version by @mudler in #6121
  • feat: bundle python inside backends by @mudler in #6123
  • Add mlx-vlm by @mudler in #6119
  • feat(llama.cpp): Add gfx1201 support by @mudler in #6125
  • feat(backends): add CPU variant for diffusers backend by @mudler in #6128
  • chore(diffusers): support both src and reference_images in diffusers by @mudler in #6135
  • feat(launcher): add LocalAI launcher app by @mudler in #6127
  • feat(mlx-audio): Add mlx-audio backend by @mudler in #6138
  • feat: Add a model refresh button to manually refresh on-disk yaml by @mcowger in #6150
  • feat(diffusers): add support for wan2.2 by @mudler in #6153
  • feat(whisper-cpp): Convert to Purego and add VAD by @richiejp in #6087
  • chore: stream errors while streaming SSE by @mudler in #6160
  • feat(gallery): Add 'Get Config' button for models by @mcowger in #6154
  • feat(flash_attention): set auto for flash_attention in llama.cpp by @mudler in #6168
  • chore(ci): Build some Go based backends on Darwin by @richiejp in #6164

🧠 Models

  • chore(model gallery): add baichuan-inc_baichuan-m2-32 by @mudler in #6042
  • chore(model gallery): add gemma-3-270m-it-qat by @mudler in #6063
  • chore(model-gallery): ⬆️ update checksum by @localai-bot in #6071
  • chore(model gallery): add thedrummer_gemma-3-r1-27b-v1 by @mudler in #6074
  • chore(model gallery): add thedrummer_gemma-3-r1-12b-v1 by @mudler in #6075
  • chore(model gallery): add thedrummer_gemma-3-r1-4b-v1 by @mudler in #6076
  • chore(model gallery): add wingless_imp_8b-i1 by @mudler in #6077
  • chore(model gallery): add impish_longtail_12b by @mudler in #6082
  • chore(model gallery): add lfm2-vl-450m by @mudler in #6090
  • chore(model gallery): add lfm2-vl-1.6b by @mudler in #6091
  • chore(model gallery): add lfm2-1.2b by @mudler in #6092
  • chore(model gallery): add opengvlab_internvl3_5-30b-a3b by @mudler in #6143
  • chore(model gallery): add opengvlab_internvl3_5-14b by @mudler in #6144
  • chore(model gallery): add opengvlab_internvl3_5-8b by @mudler in #6145
  • chore(model gallery): add opengvlab_internvl3_5-4b by @mudler in #6146
  • chore(model gallery): add opengvlab_internvl3_5-2b by @mudler in #6147

📖 Documentation and examples

  • chore(docs): update list of supported backends by @mudler in #6134

👒 Dependencies

  • chore(deps): bump grpcio from 1.71.0 to 1.74.0 in /backend/python/vllm by @dependabot[bot] in #6036
  • chore(deps): bump grpcio from 1.71.0 to 1.74.0 in /backend/python/diffusers by @dependabot[bot] in #6037
  • chore(deps): bump sentence-transformers from 5.0.0 to 5.1.0 in /backend/python/transformers by @dependabot[bot] in #6028
  • chore(deps): bump grpcio from 1.71.0 to 1.74.0 in /backend/python/common/template by @dependabot[bot] in #6035
  • chore(deps): bump grpcio from 1.71.0 to 1.74.0 in /backend/python/bark by @dependabot[bot] in #6033
  • chore(deps): bump grpcio from 1.71.0 to 1.74.0 in /backend/python/rerankers by @dependabot[bot] in #6022
  • chore(deps): bump grpcio from 1.71.0 to 1.74.0 in /backend/python/exllama2 by @dependabot[bot] in #6019
  • chore(deps): bump oneccl-bind-pt from 2.3.100+xpu to 2.8.0+xpu in /backend/python/rerankers by @dependabot[bot] in #6021
  • chore(deps): bump intel-extension-for-pytorch from 2.3.110+xpu to 2.8.10+xpu in /backend/python/bark by @dependabot[bot] in #6043
  • chore(deps): bump intel-extension-for-pytorch from 2.3.110+xpu to 2.8.10+xpu in /backend/python/common/template by @dependabot[bot] in #6034
  • chore(deps): bump actions/checkout from 4 to 5 by @dependabot[bot] in #6014
  • chore(deps): bump actions/download-artifact from 4 to 5 by @dependabot[bot] in #6015
  • chore(deps): bump torch from 2.3.1+cxx11.abi to 2.8.0 in /backend/python/common/template by @dependabot[bot] in #6025
  • chore(deps): bump oneccl-bind-pt from 2.3.100+xpu to 2.8.0+xpu in /backend/python/common/template by @dependabot[bot] in #6016
  • chore(deps): bump grpcio from 1.71.0 to 1.74.0 in /backend/python/transformers by @dependabot[bot] in #6013
  • Revert "chore(build): Convert stablediffusion-ggml backend to Purego (#5989)" by @mudler in #6064
  • chore(build): Use Purego with stablediffusion backend by @richiejp in #6067
  • chore(deps): bump protobuf from 6.31.0 to 6.32.0 in /backend/python/transformers by @dependabot[bot] in #6100
  • chore(deps): bump actions/checkout from 4 to 5 by @dependabot[bot] in #6101
  • chore(deps): bump securego/gosec from 2.22.7 to 2.22.8 by @dependabot[bot] in #6098
  • chore(deps): bump grpcio from 1.71.0 to 1.74.0 in /backend/python/coqui by @dependabot[bot] in #6097
  • chore(deps): bump intel-extension-for-pytorch from 2.3.110+xpu to 2.8.10+xpu in /backend/python/coqui by @dependabot[bot] in #6095
  • chore(deps): bump torch from 2.3.1+cxx11.abi to 2.8.0 in /backend/python/coqui by @dependabot[bot] in #6099
  • chore(deps): bump transformers from 4.48.3 to 4.55.2 in /backend/python/coqui by @dependabot[bot] in #6096
  • chore(deps): bump the go_modules group with 4 updates by @dependabot[bot] in #6161

Other Changes

  • docs: ⬆️ update docs version mudler/LocalAI by @localai-bot in #6046
  • chore: ⬆️ Update ggml-org/whisper.cpp to 5527454cdb3e15d7e2b8a6e2afcb58cb61651fd2 by @localai-bot in #6047
  • chore: ⬆️ Update ggml-org/llama.cpp to f4586ee5986d6f965becb37876d6f3666478a961 by @localai-bot in #6048
  • chore: ⬆️ Update ggml-org/whisper.cpp to 16c2924cb2c4b5c9f79220aa7708eb5b346b029b by @localai-bot in #6055
  • chore: ⬆️ Update ggml-org/llama.cpp to 29c8fbe4e05fd23c44950d0958299e25fbeabc5c by @localai-bot in #6054
  • chore(deps): bump llama.cpp to 'df36bce667bf14f8e538645547754386f9516326 by @mudler in #6062
  • chore: ⬆️ Update ggml-org/whisper.cpp to 040510a132f0a9b51d4692b57a6abfd8c9660696 by @localai-bot in #6069
  • chore: ⬆️ Update ggml-org/llama.cpp to 5e6229a8409ac786e62cb133d09f1679a9aec13e by @localai-bot in #6070
  • chore: ⬆️ Update ggml-org/llama.cpp to 1fe00296f587dfca0957e006d146f5875b61e43d by @localai-bot in #6079
  • chore: ⬆️ Update ggml-org/llama.cpp to 21c17b5befc5f6be5992bc87fc1ba99d388561df by @localai-bot in #6084
  • chore(ci): Build modified backends on PR by @richiejp in #6086
  • chore: ⬆️ Update ggml-org/llama.cpp to 6d7f1117e3e3285d0c5c11b5ebb0439e27920082 by @localai-bot in #6088
  • chore: ⬆️ Update ggml-org/whisper.cpp to fc45bb86251f774ef817e89878bb4c2636c8a58f by @localai-bot in #6089
  • Revert "chore(deps): bump intel-extension-for-pytorch from 2.3.110+xpu to 2.8.10+xpu in /backend/python/coqui" by @mudler in #6102
  • Revert "chore(deps): bump torch from 2.3.1+cxx11.abi to 2.8.0 in /backend/python/coqui" by @mudler in #6104
  • Revert "chore(deps): bump transformers from 4.48.3 to 4.55.2 in /backend/python/coqui" by @mudler in #6105
  • feat(swagger): update swagger by @localai-bot in #6111
  • chore: ⬆️ Update ggml-org/llama.cpp to fb22dd07a639e81c7415e30b146f545f1a2f2caf by @localai-bot in #6112
  • chore: ⬆️ Update ggml-org/llama.cpp to 7a6e91ad26160dd6dfb33d29ac441617422f28e7 by @localai-bot in #6116
  • chore: ⬆️ Update ggml-org/llama.cpp to cd36b5e5c7fed2a3ac671dd542d579ca40b48b54 by @localai-bot in #6118
  • feat(mlx): add mlx backend by @mudler in #6049
  • chore(deps): bump llama.cpp to '45363632cbd593537d541e81b600242e0b3d47fc' by @mudler in #6122
  • chore: ⬆️ Update ggml-org/llama.cpp to 710dfc465a68f7443b87d9f792cffba00ed739fe by @localai-bot in #6126
  • fix(rfdetr): use cpu torch for cpu builds by @mudler in #6129
  • chore(Dockerfile): drop python from images by @mudler in #6130
  • Revert "fix(rfdetr): use cpu torch for cpu builds" by @mudler in #6131
  • chore: ⬆️ Update ggml-org/whisper.cpp to 7745fcf32846006128f16de429cfe1677c963b30 by @localai-bot in #6136
  • chore: ⬆️ Update ggml-org/llama.cpp to 043fb27d3808766d8ea8195bbd12359727264402 by @localai-bot in #6137
  • chore: ⬆️ Update ggml-org/llama.cpp to c4e9239064a564de7b94ee2b401ae907235a8fca by @localai-bot in #6139
  • chore(refactor): cli -> cmd, update docs by @mudler in #6148
  • chore: ⬆️ Update ggml-org/llama.cpp to 8b696861364360770e9f61a3422d32941a477824 by @localai-bot in #6151
  • chore: ⬆️ Update ggml-org/llama.cpp to fbef0fad7a7c765939f6c9e322fa05cd52cf0c15 by @localai-bot in #6155
  • chore: ⬆️ Update ggml-org/llama.cpp to c97dc093912ad014f6d22743ede0d4d7fd82365a by @localai-bot in #6163
  • feat(swagger): update swagger by @localai-bot in #6162
  • chore: ⬆️ Update ggml-org/llama.cpp to 3d16b29c3bb1ec816ac0e782f20d169097063919 by @localai-bot in #6165
  • chore: ⬆️ Update ggml-org/llama.cpp to e92d53b29e393fc4c0f9f1f7c3fe651be8d36faa by @localai-bot in #6169
  • chore(backends): add stablediffusion-ggml and whisper for metal by @mudler in #6173
  • chore: ⬆️ Update leejet/stable-diffusion.cpp to 4c6475f9176bf99271ccf5a2817b30a490b83db0 by @localai-bot in #6171
  • fix(ci): Set default Darwin backend lang to python by @richiejp in #6175
  • chore: ⬆️ Update ggml-org/llama.cpp to d4d8dbe383e8b9600cbe8b42016e3a4529b51219 by @localai-bot in #6172

New Contributors

Full Changelog: v3.4.0...v3.5.0

Check out latest releases or
releases around mudler/LocalAI v3.5.0

Don't miss a new LocalAI release

NewReleases is sending notifications on new releases.

Get notifications