output_modalities: ["text"] in your session.update. Then the spoken half of the call is entirely yours to build. If everything you’ve done so far has been speech-to-speech, that half is new: there’s no audio-out event to wait for, and the provider you pick has its own failure modes that show up as call defects rather than API errors. This page covers the ones that actually bite.
Full code on GitHub
A Python agent that speaks each reply through ElevenLabs as it streams, with every section of this page in code.
Stream the delta, don’t wait for the whole reply
Waiting forresponse.output_text.done adds the entire generation to your time-to-first-audio. response.output_text.delta starts arriving as soon as the model has anything to say, well before the turn finishes, and most streaming TTS providers accept text incrementally for exactly this reason: feed a context small amounts of text at a time and it starts synthesizing before it has the whole thing.
the shape, not a specific SDK
One context per response, and cancel and flush are two different steps
Open one TTS streaming context perresponse_id, not one for the whole call. A response can be cancelled mid-stream, on barge-in, and a context that’s still mid-turn when that happens needs to be told to stop rather than quietly finishing a reply the caller already talked over.
That cancellation is two separate steps, and treating them as one leaves audio playing that the caller has already interrupted:
- Cancel the TTS context for that response. Stop feeding it text and stop it synthesizing more audio for a reply that’s no longer going to be said.
- Flush what’s already queued in the speaker. Audio synthesized before the cancellation can still be sitting in a playback buffer, and closing the TTS context doesn’t reach into that buffer and clear it. Do that as its own step, or the caller keeps hearing the tail end of a reply the model has already abandoned.
input_audio_buffer.speech_started while a reply is still playing. It reports a confirmed turn, so by then Dialog-RSN-1 has confirmed the caller is talking. Most replies finish generating well before they finish playing, so this is usually the only signal you get. If the reply was still generating, response.done with status: "cancelled" follows too, unprompted. Treat that as the same signal, not a second one.
Guard unspeakable text before it reaches the synthesizer
Empty text and punctuation-only text are both things Dialog-RSN-1 can legitimately produce, and most TTS vendors 400 on them. A reply that’s only an emotion tag with nothing after it, or a turn where the model has nothing further to add, can arrive as an empty or near-empty string once you’ve stripped what you’re going to strip anyway. If your harness treats a TTS failure as fatal to the call, that 400 takes the call down over text that was never meant to be spoken. Check what’s left after stripping tags and whitespace, and skip the synthesis call entirely when there’s nothing left to say, rather than sending it and handling the error after the fact:the check, not a specific SDK
Strip the emotion tag before it reaches TTS
If you’re using the emotion-tag pattern, the tag has to come off before the text reaches your synthesizer, not after. Send[reassuring] Don't worry, I can get that sorted for you. straight to TTS and most providers will say the bracket and the word “reassuring” out loud, right before the reply itself.
Parse the prefix once, at the point text leaves your harness for TTS, and use it for what it’s for, a style or voice parameter, rather than letting it fall through as spoken content:
the split, not a specific SDK
Report playback back
Without a playback report, the server guesses what the caller heard. It assumes your TTS speaks at a typical rate, about 15 characters a second, starting from the reply’s first text. It treats the reply as audible for that long, and on a barge-in it cuts history back to the words it estimates were heard. Your synthesizer’s real pace, buffering and network delay all make that guess drift from what actually happened. Sendpoly.output_audio.started, .delta, and .stopped as your TTS starts, progresses through, and finishes or is cut off from a reply, and the guess is replaced by what really played. A backchannel partway through a long reply gets confirmed as an interruption instead of acted on, for as long as the reply is actually playing. And history is rewritten to spoken_text on .stopped, so the model’s next turn knows exactly where the caller cut in. Once a session has sent one report, the server stops estimating for it, so send them for every reply. See the overview for how playback reports change barge-in behavior, and the reference for the event shapes.
Measure time-to-first-audio, not time-to-first-token
The firstresponse.output_text.delta is not what the caller feels; the first sample your TTS actually plays is. Everything upstream of that, the model’s latency, your own text buffering, the round trip to your synthesizer, sits inside a gap the caller experiences as silence. Time-to-first-token looks fast on a dashboard and still leaves a caller wondering if the call dropped.
This is covered in more depth, with the metric definitions and how to separate model latency from TTS latency, in Evaluations.
Next steps
Response style cookbook
The emotion-tag recipe this page assumes, and the other spoken-output patterns.
Reference: playback reports
The exact shape of
poly.output_audio.started / .delta / .stopped.Overview
How barge-in and playback reports fit together.
Evaluations
Time-to-first-token versus caller-perceived latency, and how to measure the gap between them.