fixie-ai/ultravox

Health v2 · maintenance only

Adoption not part of the score

4555 stars · 386 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Ultravox is a fast multimodal LLM that understands human speech directly without a separate ASR stage, by projecting audio into the LLM's embedding space. It extends open-weight models like Llama 3, Mistral, and Gemma for real-time voice interactions and streaming text output.

Use cases

• build real-time voice AI agents
• transcribe and understand speech without ASR
• run a speech-native LLM for low-latency voice conversations
• fine-tune a multimodal audio LLM on open-weight models
• stream text from live audio input
• build voice-to-voice chatbots

When to choose

• you need low-latency voice AI without the ASR-to-LLM pipeline overhead
• you want to preserve paralinguistic cues like tone and timing in speech understanding
• you want to self-host or fine-tune a speech-capable LLM on your own models
• you're building real-time conversational voice agents

When to avoid

• you only need traditional batch speech-to-text transcription
• you have no GPU resources for inference
• you need a fully managed hosted solution without running your own infrastructure
• your use case is text-only LLM work

Facets

library · maturity active

llm-inference speech-recognition machine-learning llm-training large-language-models speech-processing artificial-intelligence python cloud multimodal-llm voice-ai speech-native audio-understanding real-time-voice asr-free llama mistral voice-agents ai-agents gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="fixie-ai/ultravox")