ROCm/FastFlowLM v0.9.14

🚀 FastFlowLM v0.9.14: New ASR Capabilities & API Integration

on GitHub

✨ What’s New

🎙️ 1. New Model: whisper-large-v3-turbo (by OpenAI)

The first Automatic Speech Recognition (ASR) model on FLM:

• Runs fully offline on AMD Ryzen™ AI NPU
• Multilingual audio recognition
• Supports MP3, WAV, OGG and M4A formats
• Lightweight footprint — only 900MB memory

Try it out:

1. Start flm in CLI mode with ASR enabled:

# CLI
flm run gemma3:4b --asr 1 # Load the ASR model (whisper-v3:turbo) in the background, with concurrent LLM loading (gemma3:4b).

1. Type (replace filename.mp3 with your audio file path):

/input "path\to\audio_sample.mp3" summarize it

🌐 2. OpenAI-Compatible ASR API: v1/audio/transcriptions API

FastFlowLM now supports the OpenAI v1/audio/transcriptions endpoint — making it easy to integrate ASR into any OpenAI-compatible client or UI.

How to use:

1. Start your FLM server with ASR enabled:

# serve
flm serve gemma3:4b --asr 1 # # Load the ASR model (whisper-v3:turbo) in the background, with concurrent LLM loading (gemma3:4b).

1. Send audio to:
   POST /v1/audio/transcriptions
   via any OpenAI client or Open WebUI.

Examples: OpenAI client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:52625/v1",  # FastFlowLM's local API endpoint
    api_key="flm",  # Dummy key (FastFlowLM doesn’t require authentication)
)

with open("audio.mp3", "rb") as f:
    resp = client.audio.transcriptions.create(
        model="whisper-v3",
        file=f,
    )
    print(resp.text)

Example: Open WebUI
1. Follow Open WebUI setup guide.
2. In the bottom-left corner, click User icon, then select Settings.
3. In the bottom panel, open Admin Settings.
4. In the left sidebar, navigate to Audio.
5. Set Speech-to-Text Engine to OpenAI.
6. Enter:
-- API Base URL: http://host.docker.internal:52625/v1
-- API KEY: flm (any value works)
-- STT Model: whisper-large-v3-turbo (type in the model name; can be different)
7. Save the setting.
8. You're ready to upload audio files! (Choose an LLM to load and use concurrently)

🌟 Summary

FastFlowLM v0.9.14 introduces offline ASR with whisper-large-v3-turbo and support for OpenAI’s v1/audio/transcriptions API, making speech-to-text integration seamless across clients and WebUI. Just start the server with --asr 1 and you're ready to transcribe.