github ggml-org/llama.cpp b12590

pre-releaseone hour ago
Details

mtmd: drama3 voice cloning (#32586)

  • mtmd: drama3 voice cloning

The Fish S2 codec encoder (fish_s2_spkenc) now also serves drama3:

  • conversion: drama3 writes the codec encoder and the audio encoder KV, shared with fish_s2
  • fish_s2_spkenc: outputs the codes of the clip (gen-side reference) and, in a drama3_gen mmproj,
    builds the drama3 backbone rows (E_lm[semantic] + E_cb[1][code 1]) / sqrt(2)
  • drama3_gen tower: teacher forcing when the caller gives the codes of the next frame
  • drama3 pipeline: reference-bracket-v1 prompt with one reference (vLLM-Omni fish_omni), the
    reference frames go through the backbone as codec rows and through the tower with their codes
  • mtmd_gen_state_get_ref_codes(): read-only access to the reference codes of a gen state

Claude-Session: https://claude.ai/code/session_01PpX3DmaxNSmJbxGaxnUBrX

  • fix correctness

  • clean up

  • rm redundant code

  • fish_s2: rm unused import


Co-authored-by: Xuan Son Nguyen son@huggingface.co

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Don't miss a new llama.cpp release

NewReleases is sending notifications on new releases.