The biggest release yet — six new AI features, all running on-device.
New
- AI dubbing with a fully-local engine — a new NLLB-200 translate-service batch-translates transcripts on-device, with Sarvam cloud fallback for Indian language pairs and LLM fallback as a last resort. A full dubbing pipeline (Whisper → translate → XTTS → ffmpeg) is also available at
/api/dub/*. - Video background removal — per-frame matting via rembg, inserted non-destructively as an alpha WebM layer above the source clip.
- Auto B-roll from your own footage — each transcript segment is matched against the existing CLIP embedding index and the best clip is auto-inserted.
- Multilingual captions — NLLB-first translation with multi-language quick-add chips and per-track SRT/VTT export.
- Edit by speaker — scope remove, tighten-gaps, and isolate operations to a single speaker using diarization.
- Automatic multicam sync — client-side audio cross-correlation aligns camera angles without timecode; preview offsets with confidence scores, apply in one undoable transaction.
Also in this release
- Local CLIP-powered visual search, thumbnail A/B testing, AI copilot, YouTube viral score, TurboQuant quantized models, video templates
- New model support: Sarvam AI, Smallest AI, Kimi K2, Google Gemma; GPU-accelerated docker-compose profile
- Hardened upload endpoints, stabilized VPS deployment, repaired build/CI pipeline, version-control initialization race fix
Full changelog: https://github.com/Ekaanth/OpenCut-AI/blob/main/apps/web/content/changelog/0.4.0.md