AssemblyAI Speech-to-Text and Voice AI

AssemblyAI provides speech-to-text APIs, audio intelligence models, and an LLM Gateway for applying language models to transcripts. This skill corrects common mistakes that training data gets wrong — deprecated APIs, discontinued SDKs, and non-obvious auth patterns.

Authentication

All endpoints use the same header:

Authorization: YOUR_API_KEY

NOT Authorization: Bearer ... — just the raw API key, no Bearer prefix. This is the #1 mistake.

Base URLs

Streaming EU region: As of March 2026, the EU region moved from AWS eu-west-1 (Ireland) to AWS eu-north-1 (Stockholm). The customer-facing endpoint host (streaming.eu.assemblyai.com) is unchanged.

SDKs

Only Python, JS/TS, and Ruby SDKs are maintained. For Java, Go, or C#, use the REST API directly.

Speech-to-Text Models

universal-3-5-pro is the model to use across the board. It is GA for both realtime/streaming and pre-recorded/async transcription. Use it everywhere by default; drop to universal-2 only for cost or for languages outside Universal-3.5 Pro's 18. universal-3-pro (async) and u3-rt-pro (streaming) have both been superseded by universal-3-5-pro and removed from their model lists/enums.

Default-model transition (async): accounts created on/after July 7, 2026 already default to ["universal-3-5-pro", "universal-2"] when speech_models is omitted (and cannot request universal-3-pro/universal). All remaining accounts switch on September 2, 2026 — from that date, an omitted/empty/singular speech_model, or ["universal"], routes to universal-3-5-pro; speech_models: ["universal-3-pro"] returns an error; and streaming u3-rt-pro connections are silently redirected to universal-3-5-pro. Pin speech_models explicitly if you need deterministic behavior across the transition.

Pre-Recorded

On POST /v2/transcript, speech_models is a priority list with fallback — the documented default is ["universal-3-5-pro", "universal-2"] (already live for accounts created on/after July 7, 2026; all accounts from September 2, 2026 — see the transition note above). Universal-3.5 Pro handles its 18 languages and automatically falls back to Universal-2 for the rest. The async speech_models enum now accepts only universal-3-5-pro and universal-2 (universal-3-pro has been removed, superseded by universal-3-5-pro). The singular speech_model field rejects current model names with a 400 (legacy names best/universal still route to the default; nano is no longer documented). Universal-3.5 Pro supports contextual prompt (a plain-language description of the audio) and keyterms_prompt (up to 1,000 terms; native in-model biasing — Universal-2's keyterms are phonetic post-processing, ≤200 terms, English only, terms shorter than 5 or longer than 50 characters ignored). Check the speech_model_used response field to see which model actually ran.

Streaming

For realtime/streaming STT, use speech_model: "universal-3-5-pro" by default. The raw API parameter is optional and defaults to universal-3-5-pro; set it explicitly when pinning behavior or using an SDK that requires the field. The streaming spec speech_model enum now lists only universal-3-5-pro, universal-streaming-english, and universal-streaming-multilingual. The mode connection param (universal-3-5-pro only) trades off accuracy vs latency: min_latency, balanced (default), max_accuracy.

Medical Mode (Add-On)

domain: "medical-v1" enables Medical Mode — an add-on that improves accuracy for medical terminology (medications, procedures, conditions, dosages). Works with both pre-recorded and streaming models.

• Pre-recorded: Universal-3.5 Pro (domain: "medical-v1" in request body), Universal-2
• Streaming: universal-3-5-pro, universal-streaming-english, universal-streaming-multilingual
• Supported languages: English, Spanish, German, French (4 languages only)
• Billed as a separate add-on. If used with an unsupported language, the API ignores domain and returns a warning — transcript still completes and you are NOT charged for Medical Mode.

Prompting (Universal-3.5 Pro)

prompt and keyterms_prompt are complementary — use either, or both together. Neither changes the output format, and both work the same way for streaming and async (POST /v2/transcript). Transcription behavior (verbatim, punctuation, formatting) is built in and managed by AssemblyAI.

• prompt (string; length cap varies by API — streaming ≤1750 chars, async up to ~1,500 words, Sync STT ≤4096 chars): a plain-language description of the audio — its domain, scenario, or full details. It carries context, not instructions — formatting/behavioral commands (punctuation rules, "transcribe verbatim", "don't…") are not supported and are ignored. The model stays grounded in the audio: irrelevant or only-partially-applicable context won't make it insert words that weren't spoken, so you can safely send the same description on every session/segment.
• keyterms_prompt (string[]): an explicit list of names/brands/domain terms to boost. Streaming: up to 100 terms, ≤50 chars each. Async: up to 1,000 terms.

Contextual prompt — three levels of specificity (use the least specific that covers your case):

Best practices:

• Start with no prompt and no keyterms_prompt — the model is optimized out of the box. Add context only for domain vocabulary it gets wrong, starting at the broadest level.
• Write plain, complete sentences that describe the recording; keep it to one short block, not a keyword list (that's what keyterms_prompt is for).
• Keyterms: use exact spelling/capitalization; avoid common words (over-inclusion causes overcorrection/hallucination).
• Specify language via language_code (preferred) or by naming it in the prompt (e.g. "Spanish customer support call…").
• Streaming: update both mid-session via UpdateConfiguration; a new keyterms array replaces the prior set, [] clears it.

Sync STT API (short-form audio, ≤120s)

A separate synchronous endpoint for short clips — send audio in one HTTP request, get the transcript back in the response. No polling, no transcript ID, no upload step. Fully launched (GA) July 14, 2026 at $0.45/hr. Ideal for voice-message transcription, short call recordings, or voice-agent pipelines that do their own turn detection and submit completed utterances.

• Endpoint: POST https://sync.assemblyai.com/v1/transcribe (routes are versioned as of July 2026 — the unprefixed /transcribe still works for legacy clients, but new code should use /v1/). Global host routes to nearest region; use sync.us.assemblyai.com / sync.eu.assemblyai.com for data residency
• Required header: X-AAI-Model: universal-3-5-pro (the only sync model in the SDKs; u3-sync-pro is a legacy accepted value)
• Auth: Authorization: YOUR_API_KEY (Bearer prefix optional here, unlike the async REST API; or pass ?token=YOUR_API_KEY)
• Body: multipart/form-data with an audio part (Content-Type: audio/wav or audio/pcm) and an optional config JSON part
• config fields: prompt (≤4096 chars), keyterms_prompt (string[] — renamed from word_boost in July 2026; the aliases word_boost and keyterms are still accepted), conversation_context (string or string[] — prior conversation turns oldest-first, for continuity across a multi-turn conversation; oldest dropped when over the context budget), language_codes (ISO 639-1 list; singular language_code also accepted, default en — steers the default prompt toward the named language(s); ignored when a custom prompt is set), timestamps (bool, default false — opt-in per-word start/end in ms via forced alignment; omitted from the response if the aligner is unavailable, never estimated), and for audio/pcm also sample_rate + channels (required for raw PCM; WAV reads them from its header). Unknown config fields are rejected with a 400 — don't pass async-API params here
• Audio limits: 80ms–120s, ≤40MB, 16-bit only, mono/stereo (stereo down-mixed), sample rates 8000/16000/22050/24000/32000/44100/48000 Hz
• Response: { text, words[{text, start?, end?, confidence}], confidence, audio_duration_ms, session_id, request_time_ms } — word start/end (integer milliseconds) appear only when timestamps: true; only the clip-level durations carry the _ms suffix
• Pre-warming: GET https://sync.assemblyai.com/v1/warm establishes the connection ahead of the first request (returns 200 {"warm":"toasty"}). Python SDK: SyncTranscriber.warm() plus aai.settings.keepalive_expiry to hold the connection; Node SDK has SyncTranscriber too
• 30s per-request deadline (504 inference_timeout). For audio >120s use the async REST API; for live mic audio use Streaming.

curl -X POST https://sync.assemblyai.com/v1/transcribe \
  -H 'Authorization: YOUR_API_KEY' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F '[email protected];type=audio/wav'

LeMUR is Deprecated

LeMUR is deprecated (sunset March 31, 2026 — already sunset). Use the LLM Gateway instead. The LLM Gateway is an OpenAI-compatible API. Key difference: you pass transcript text directly in messages (no transcript_ids). Transcribe first, then include transcript.text in your prompt.

See references/llm-gateway.md for models, tool calling, structured outputs, and examples.

Key Gotchas

Pre-Recorded / Async

Streaming

Voice Agent API

LLM Gateway

Common Mistakes

Reference Files

Read the relevant reference file based on what the user needs:

API Spec Source of Truth

https://github.com/AssemblyAI/assemblyai-api-spec