Fish Audio Drama 3 — Pre-Release

An open-source speech generation model that turns text into directed performances.

Pre-trained on several hundred billion tokens across ~80 languages and dialects, Drama 3 combines a 35B-A3B hybrid linear-attention MoE backbone (initialized from Qwen3.5) with a new codebook-MTP audio decoder and RL from a single human-preference reward model. It clones voices from multiple references, follows natural-language direction, revises a take from a note in the next turn, voices multi-character scenes, edits speech by infilling, and synthesizes long-form and streaming audio, all in one model.

Technical Report · Model · Docs · Playground

Model

Requires an NVIDIA GPU with at least 48 GB (FP8). S2-Pro remains fully supported with the same command line, API server and WebUI; the model type is read from the checkpoint's config.json.

Highlights

• Hybrid MoE backbone: Gated DeltaNet linear attention with one gated full-attention layer in four; 256 experts, top-8 plus a shared expert, ~3B active per token
• Codebook-MTP decoder: 13 layers (4 shared + 9 per-codebook) replace S2's Fast AR; each layer attends causally over the whole sequence, including the reference audio
• Directed performance: point tags like [laughs], scoped tags like <whisper>...</whisper>, style descriptions and free-form director's notes
• Multi-turn revision: when a take misses, the next turn gives a note such as [happier versus previous] and the model redoes it, keeping the original reference and the latest take in context; on Step-Audio-Edit, three rounds raise emotion accuracy from 55.5% to 58.2% and style accuracy from 76.5% to 81.8%
• Editing by infilling: insert, delete or replace a word or a sentence inside existing speech; only the edit window is regenerated from the target text, while the audio before and after it is kept. Paralinguistic events are added the same way, e.g. insert [coughs] and infill there. Lower WER than Cartesia Sonic-3's infill API on Ming-Freeform-Audio-Edit in both English and Chinese
• Scenes & long-form: multi-character scenes with <|speaker:i|> (up to 16 speakers), long-form synthesis and streaming text input
• RL Alignment: GRPO with a single learned human-preference reward model, no rule-based rewards and no KL penalty
• Fast inference: block-scaled FP8 experts and CUDA graphs; 271 frames/s per stream on one H200 with SGLang-Omni, against 177 for S2

What's Changed

Model & Inference

• New fish_omni model family for Drama 3: FP8 experts, CUDA-graph decoding, multi-GPU loading, float16 fallback for GPUs older than SM80
• Long-text chunking for Drama 3 (~300 UTF-8 bytes, at most three speakers per chunk), with all chunks validated before generation
• Both models use the official Fish Audio SDK sampling defaults (temperature 0.7, top-p 0.7, top-k 30)
• Fixed: S2-Pro crash with small or negative max_new_tokens; faster attention over the active KV-cache prefix

Docker & Deployment

• Server images now bundle the WebUI at http://localhost:8080/ui
• DEVICE and optional DECODER_CHECKPOINT_PATH in Docker / Compose; longer healthcheck start period for large checkpoints
• AMD ROCm image for S2-Pro (RDNA3/RDNA4); CUDA 12.9 for Blackwell (sm_120)

API & Server

• Awesome WebUI replaces the Gradio WebUI; it loads each model's own defaults
• GET /v1/tts/defaults, per-reference speaker field (0–15), invalid input returns 400 instead of 500
• Fixed: non-streaming pcm / opus output, streaming WAV header, uvicorn workers > 1, reference ID path traversal

Finetune

• Fine-tuning code removed from this repo (training script, LoRA, dataset and VQ extraction tools)

Docs & Infra

• README & MkDocs rewritten for Drama 3 across 7 languages (Spanish added)
• License updated to Fish Audio Community License; S2-Pro weights remain under the Fish Audio Research License
• Removed legacy code (Gradio WebUI, i18n, quantize tool)