Build a browser voice agent with Cartesia

Build a voice agent you can talk to in the browser, where one model listens, decides when you’ve finished, and writes the reply. It takes about ten minutes to run and about 450 lines of Python to read.

Full code on GitHub

A FastAPI server, a browser page with a live transcript and event log, and a terminal probe that streams a sample caller through it.

What you’ll build

A Python server that holds one Dialog-RSN-1 connection per browser tab and forwards the caller’s audio to it.
Streaming speech: Cartesia speaks each reply from its first few words, while Dialog-RSN-1 is still writing it.
Barge-in: talk over the agent and it stops as soon as Dialog-RSN-1 confirms it’s you, and the model learns which words you heard.
A browser page with a microphone picker, voice and turn-taking settings, a live transcript with per-turn latency, and a raw event log.

Why this shape

A typical browser voice agent chains three services on its server:

A speech-to-text service transcribes the caller and sends an end-of-turn event when it thinks they’ve finished.
The server appends the transcript to its own messages list and calls an LLM.
It sends the reply to a TTS service and forwards the audio to the page.

That’s three hops before the caller hears anything, and a conversation history you manage yourself. Dialog-RSN-1 folds the first two steps into one. It takes the caller’s audio directly, decides when the turn is over, and streams the reply as text. It keeps the conversation history in the session. Your server does two things: pass audio in, and pass text to a speech engine.

	Chained pipeline	This demo
End of turn	An end-of-turn event from speech-to-text, tuned with a confidence threshold and a timeout	Dialog-RSN-1 `speech_stopped`, confirmed by the model, tuned in plain language
Reply	An LLM, called with the transcript	Written by the same model that heard the audio
History	A `messages` list on your server	Held in the Dialog-RSN-1 session
Speech	A separate TTS service	Cartesia, streaming from the first words
Interruptions	Your server stops playback and trims its own history	Barge-in, with playback reports back to the model

Prerequisites

Python 3.12 and uv.
A Dialog-RSN-1 API key. Sign up or log in at studio.poly.ai and create one in the API Keys tab. The quickstart walks through it.
A Cartesia API key. The free tier works.
Chrome, Edge or Safari, and headphones if you’ll use the microphone.

Run it

git clone https://github.com/PolyAI-LDN/dialog-rsn-1-voice-agent
cd dialog-rsn-1-voice-agent
cp .env.example .env    # add POLY_API_KEY and CARTESIA_API_KEY
uv sync
uv run app.py

Open http://127.0.0.1:3000. Click Start conversation and talk. Or click Play the sample caller: a recorded caller asks about a trip to Lisbon, then talks over the answer, so you see a barge-in without a microphone. To check the whole pipeline from a terminal, leave the server running and stream the same sample through it. The probe counts from audio_end_ms, where Dialog-RSN-1 marks the end of speech, about half a second after the caller’s last word:

uv run probe.py

(caller speaking)
  turn confirmed 133 ms after audio_end_ms
  first text  378 ms after audio_end_ms
  first audio 563 ms after audio_end_ms
Caller: Hi, I'm going to Lisbon for a long weekend next month. What should I make sure I see?
Agent:  In Lisbon, you should definitely see the Belém Tower and the Jerónimos Monastery. The Alfama neighborhood is also great for exploring, and LX Factory is a fun spot for shopping and food. Do any of those catch your eye?
(caller speaking)
(interrupted after 5676 ms, 6564 ms dropped)
  heard: 'In Lisbon, you should definitely see the Belém Tower and the Jerónimos Monastery. The Alfama neighborhood'
  turn confirmed 155 ms after audio_end_ms
  first text  356 ms after audio_end_ms
  first audio 516 ms after audio_end_ms
Caller: Sorry, quick question - is it easy to get around on foot?
Agent:  Yes, the city center is very walkable, though it can be quite hilly. Do you need any other tips for getting around?
  heard: 'Yes, the city center is very walkable, though it can be quite hilly. Do you need any other tips for getting around?'

Step 1: Open one socket and configure the session

Each browser tab gets its own Call, which opens a Dialog-RSN-1 WebSocket with the key in a header. The browser never sees the key.

self.rsn = await websockets.connect(
    config.DIALOG_REALTIME_URL, max_size=None,
    additional_headers={"X-API-KEY": config.POLY_API_KEY})

config.DIALOG_REALTIME_URL defaults to wss://api.studio.poly.ai/v1/realtime. Set DIALOG_REALTIME_URL in .env to point somewhere else. Then it sends one session.update:

def session_update(instructions: str) -> dict:
    return {"type": "session.update", "session": {
        "type": "realtime",
        "instructions": instructions,
        "output_modalities": ["text"],
        "audio": {"input": {
            "format": {"type": "audio/pcm", "rate": config.SAMPLE_RATE},
            # Dialog-RSN-1 decides when the caller has finished, then answers on its own.
            "turn_detection": {"type": "server_vad", "create_response": True},
        }},
    }}

Three fields do the work:

output_modalities: ["text"]. This agent speaks through its own Cartesia stream, so it asks Dialog-RSN-1 for text. Sessions start with ["audio"], where the server speaks the reply itself, so set this or the caller hears two voices.
rate: 24000. The page records at 24 kHz, and Cartesia returns 24 kHz, so the page never resamples.
create_response: true. When Dialog-RSN-1 confirms the caller has finished, it starts the reply itself. You don’t send response.create.

The page waits for session.updated before it sends any audio, so Dialog-RSN-1 reads every frame at the right rate.

Step 2: Forward the caller’s audio

The page sends the microphone as binary PCM16, 100 ms per frame. The server wraps each frame in input_audio_buffer.append and passes it on:

if msg.get("bytes"):
    # The caller's audio, 100 ms at a time, forwarded as it arrives.
    await self.rsn.send(json.dumps({"type": "input_audio_buffer.append",
                                    "audio": base64.b64encode(msg["bytes"]).decode()}))

There’s no buffering, transcript handling or turn logic on the server. Compare a chained pipeline, where the server watches for the end-of-turn event, reads the transcript, appends it to messages and starts an LLM call. Here, the next thing your server hears about is the reply.

Step 3: Stream the reply into speech

Dialog-RSN-1 sends the reply as response.output_text.delta events. Each delta goes straight to Cartesia:

if kind == "response.created":
    # One Cartesia context per response, so a cancelled one can't leak into the next.
    self.tts.start(event["response"]["id"])

elif kind == "response.output_text.delta":
    # Stream each delta straight on. Don't wait for the whole reply.
    await self.tts.push(event["delta"])

elif kind == "response.output_text.done":
    await self.tts.finish()

cartesia_tts.py keeps one Cartesia WebSocket open for the whole call. Each response gets its own Cartesia context, named after the response_id. Text goes in with continue: true, so Cartesia keeps the rhythm of one sentence across many small pieces. Two details matter:

Send whole words. A delta can end mid-word. push() holds the text after the last space until the word is complete.
Lower max_buffer_delay_ms. Cartesia waits up to this long for more text before it speaks. Its default, 3 seconds, favours prosody. The demo sets 250 ms. Dialog-RSN-1 streams text faster than that in our runs, so the limit costs little.

Audio comes back as base64 PCM chunks, and the server sends each one to the page as a binary frame. The page’s player worklet plays them as they arrive.

Step 4: Stop talking when the caller does

When the caller starts speaking, Dialog-RSN-1 sends input_audio_buffer.speech_started. It has already checked that the sound is the caller and not noise, so the server acts on it at once:

elif kind == "input_audio_buffer.speech_started":
    # The caller is talking. Dialog-RSN-1 has already confirmed it's really them,
    # so stop the agent's voice now.
    await self.interrupt()

Stopping takes two steps, because audio lives in two places:

async def interrupt(self) -> None:
    """Barge-in, in two steps: stop synthesizing, then stop playing."""
    rid = self.tts.context
    if rid is None or self.reports.get(rid, {}).get("stopped"):
        return   # nothing to stop: that reply already played to the end
    await self.tts.cancel(rid)
    if rid == self.audio_for:
        # The browser empties its queue and answers `flushed` with what it played.
        await self.to_browser({"type": "agent.flush", "response_id": rid})

Cancelling the Cartesia context stops new audio. Flushing the page’s queue stops audio that’s already been synthesized but not yet played. Skip the second step and the agent keeps talking for seconds after you’ve cut in.

Step 5: Tell Dialog-RSN-1 what the caller heard

Only the page knows what reached the speakers. Its player worklet counts the samples it has played and reports every 250 ms. The server turns those counts into Dialog-RSN-1 playback reports:

spoken = self.tts.spoken_text(rid, played_ms)
if stopped:
    sent["stopped"] = True
    await self.to_rsn({"type": "poly.output_audio.stopped", "response_id": rid,
                       "audio_ms": played_ms, "dropped_ms": dropped_ms, "spoken_text": spoken})
elif len(spoken) > sent["text"]:
    await self.to_rsn({"type": "poly.output_audio.delta", "response_id": rid,
                       "audio_ms": played_ms, "delta": spoken[sent["text"]:]})
    sent["text"] = len(spoken)

spoken_text comes from Cartesia’s word timestamps: every word whose audio had started by played_ms. These reports do two things:

They make the barge-in check follow the real speaker. Dialog-RSN-1 checks that an interruption comes from the caller for as long as it thinks the reply is audible. Without reports, it guesses that window from a typical speaking rate of 15 characters a second, counted from the reply’s first words. With them, the window lasts exactly as long as your speaker plays, so a cough or an “mm-hm” three seconds into a slow reply still gets the check.
They record where the caller cut in. Dialog-RSN-1 cuts the reply in its history back to spoken_text, so the model knows what the caller missed. In the sample, the agent had written its whole reply, but the caller heard only up to “The Alfama neighborhood”. The page strikes through the words the caller never heard.

They’re optional. Leave them out and Dialog-RSN-1 estimates both from the speaking rate: how long each reply plays, and on a barge-in, how much of it the caller heard.

Tune turn-taking in plain language

A chained pipeline tunes the end of the turn with numbers, such as a confidence threshold and a timeout. Dialog-RSN-1 has no timing knobs. The model reads each pause in context and gives a verdict: still going, or finished. The realtime schema’s silence_duration_ms and threshold are accepted and ignored, and idle_timeout_ms is rejected with unsupported_parameter. You can tell it about your callers instead. The page’s Turn-taking menu picks one extra line for the instructions:

TURN_TAKING = {
    "balanced": {
        "label": "Balanced",
        "prompt": "",
    },
    "patient": {
        "label": "Patient: callers read out numbers and think aloud",
        "prompt": "Callers often pause while they read out account numbers, dates or "
                  "addresses, or while they think. Treat those pauses as part of the turn "
                  "and wait until they have clearly finished.",
    },
    "brisk": {
        "label": "Brisk: short answers, quick back-and-forth",
        "prompt": "Callers here give short answers like yes, no, or a single word. As soon "
                  "as a short answer is complete, treat the turn as finished.",
    },
}

What the latency looks like

The page and the probe time every turn from audio_end_ms on speech_stopped, the point in the caller’s audio where Dialog-RSN-1 places the end of speech. Over twenty-two turns against us-1 production, on three days:

Measure	After `audio_end_ms`
Turn confirmed (`speech_stopped` arrives)	80 to 240 ms
First reply text	240 to 870 ms
First reply audio	410 ms to 1.4 s

The audio_end_ms mark sits about 500 ms after the caller’s last word, so the caller waits about half a second longer than the table shows. First audio reaches them about 0.9 to 1.9 s after they stop talking. It’s first text plus Cartesia’s own time to first audio, which ran about 130 to 430 ms.

Troubleshooting

Symptom	Likely cause	Fix
”Dialog-RSN-1 refused the connection (HTTP 401)“	The key is missing, or it’s a personal access token	Create a key in the API Keys tab at studio.poly.ai and set it as `POLY_API_KEY` in `.env`
`unsupported_parameter` after `session.update`	A field this service doesn’t support, such as `idle_timeout_ms`, crept in	Remove it. None of the update was applied, so fix it and resend
The agent keeps interrupting itself	The microphone hears the speakers	Wear headphones. Echo cancellation helps but isn’t perfect
The agent talks on for seconds after you interrupt	The page’s audio queue wasn’t flushed	Flush on `speech_started` as well as cancelling the TTS context (Step 4)
Replies sound choppy	Words split across Cartesia messages	Send whole words only (Step 3)
First audio is slow, first text is fast	Cartesia is waiting for more text	Lower `max_buffer_delay_ms`

Where to go next

The streaming text-to-speech guide covers the same streaming and barge-in pattern with ElevenLabs, in a terminal.
The prompt guide covers turn-taking instructions in depth.
The API reference lists every event this demo uses, including the poly.output_audio.* playback reports.

Overview

Quickstart

Prompt guide

Prompt cookbook

Evaluations

Reference

Guides

Build a browser voice agent with Cartesia

Full code on GitHub

What you’ll build

Why this shape

Prerequisites

Run it

Step 1: Open one socket and configure the session

Step 2: Forward the caller’s audio

Step 3: Stream the reply into speech

Step 4: Stop talking when the caller does

Step 5: Tell Dialog-RSN-1 what the caller heard

Tune turn-taking in plain language

What the latency looks like

Troubleshooting

Where to go next

Full code on GitHub

​What you’ll build

​Why this shape

​Prerequisites

​Run it

​Step 1: Open one socket and configure the session

​Step 2: Forward the caller’s audio

​Step 3: Stream the reply into speech

​Step 4: Stop talking when the caller does

​Step 5: Tell Dialog-RSN-1 what the caller heard

​Tune turn-taking in plain language

​What the latency looks like

​Troubleshooting

​Where to go next

What you’ll build

Why this shape

Prerequisites

Run it

Step 1: Open one socket and configure the session

Step 2: Forward the caller’s audio

Step 3: Stream the reply into speech

Step 4: Stop talking when the caller does

Step 5: Tell Dialog-RSN-1 what the caller heard

Tune turn-taking in plain language

What the latency looks like

Troubleshooting

Where to go next