Build a LiveKit voice agent

Build a voice agent on LiveKit Agents where one model hears the caller, decides when they’ve finished, and writes the reply. LiveKit carries the audio and a text-to-speech plugin speaks the reply. In about ten minutes you’ll be talking to it, in your terminal or in a LiveKit room.

Full code on GitHub

A hotel front desk agent with two tools, ElevenLabs speech, and a scripted caller that runs a whole call, barge-in included.

What you’ll build

A front desk agent for a fictional hotel with two tools: one looks up a booking, the other changes its checkout time.
An AgentSession that uses Dialog-RSN-1 as the model and ElevenLabs as the voice. Dialog-RSN-1 replaces the speech-to-text plugin, the turn detector and the LLM.
Barge-in. Talk over the agent and it stops, and the model’s history keeps only the words the caller heard.
A scripted caller that joins the room and runs a whole call, barge-in included, so you can test without a microphone.

What changes from a standard LiveKit pipeline

A standard LiveKit pipeline chains speech-to-text, an LLM and text-to-speech, with a turn detector deciding when the caller has finished. Dialog-RSN-1 takes the caller’s audio and handles every job except the voice:

Job	Standard LiveKit pipeline	With Dialog-RSN-1
Speech-to-text	An STT plugin, such as Deepgram or AssemblyAI	None. Dialog-RSN-1 hears the audio and sends the transcript
End of turn	VAD plus LiveKit’s turn detector model	Dialog-RSN-1 decides. You tune it through the instructions
Reply	An LLM, called with the transcript	Dialog-RSN-1 writes it from the audio it heard
Voice	A TTS plugin	Any TTS plugin LiveKit supports

Dialog-RSN-1 plugs into LiveKit through its own plugin, livekit-plugins-polyai. Dialog-RSN-1 speaks the OpenAI Realtime protocol, but LiveKit’s OpenAI plugin isn’t built for it. Why not the OpenAI plugin explains the difference.

Before you begin

A Dialog-RSN-1 API key. Sign up or log in at studio.poly.ai and create one in the API Keys tab. The quickstart walks through it.
An ElevenLabs API key. The free tier works.
Python 3.10 or later, and uv.
The LiveKit CLI: brew install livekit-cli.
For Step 6: a LiveKit Cloud project, or a local server (brew install livekit).

Step 1: Get the code

git clone https://github.com/PolyAI-LDN/dialog-rsn-1-livekit
cd dialog-rsn-1-livekit
uv sync

The repo has three files you’ll work with:

File	What it is
`agent.py`	The agent: instructions, two tools, and the `AgentSession`
`hotel.py`	The instructions and in-memory demo bookings
`caller.py`	A scripted caller for Step 6

Step 2: Configure your environment

cp .env.example .env

Open .env and set two keys:

POLY_API_KEY=YOUR_DIALOG_RSN_1_KEY
ELEVEN_API_KEY=YOUR_ELEVENLABS_KEY

The ElevenLabs plugin reads ELEVEN_API_KEY, not ELEVENLABS_API_KEY. Leave the LIVEKIT_* values alone until Step 6.

Step 3: Point the session at Dialog-RSN-1

agent.py builds the whole session in a few lines:

session = AgentSession(
    # Dialog-RSN-1 hears the caller, decides when they've finished, and streams the reply
    # as text. It replaces the STT, the LLM and the turn detector.
    llm=polyai.realtime.RealtimeModel(),
    # The plugin asks Dialog-RSN-1 for text, so a TTS voices each reply as it streams.
    tts=elevenlabs.TTS(voice_id="EXAVITQu4vr4xnSDxMaL", model="eleven_flash_v2_5"),
)

await session.start(agent=FrontDesk(), room=ctx.room)
# The plugin drops per-reply instructions, so the greeting comes from INSTRUCTIONS.
await session.generate_reply()

Compared with the LiveKit starter template, the session leaves out three things:

No stt. Dialog-RSN-1 takes the room’s audio as it arrives. You still see the caller’s transcript in LiveKit, because Dialog-RSN-1 sends it with each reply. It comes at the end of a completed reply, so a turn whose reply is cancelled before it finishes generating has no transcript.
No turn_handling. You have no turn detector to configure. Dialog-RSN-1 judges each pause in the context of the conversation. To make it more or less patient, say so in the instructions. The prompt guide shows how.
No instructions on generate_reply(). The plugin drops per-reply instructions with a warning, so the greeting lives in the agent’s main instructions.

polyai.realtime.RealtimeModel() reads POLY_API_KEY and connects to wss://api.studio.poly.ai/v1/realtime. To use a different endpoint, set DIALOG_REALTIME_URL. It asks Dialog-RSN-1 for text, and the TTS plugin speaks each reply as the text streams in. The two tools are ordinary LiveKit function tools:

@function_tool
async def look_up_reservation(
    self, context: RunContext, confirmation_code: str = "", last_name: str = ""
) -> str:
    """Find a reservation by confirmation code or by the guest's last name.

    Args:
        confirmation_code: Confirmation code, for example GM40912.
        last_name: The guest's last name.
    """
    return json.dumps(look_up_reservation(confirmation_code, last_name))

Each tool returns its result as a JSON string, which gives the model structured data to read. Dialog-RSN-1 accepts function tools only and refuses any other kind, so the plugin sends function tools and skips the rest with a warning.

Step 4: Run the agent

lk agent console agent.py

The console runs the agent on your machine and connects it to your microphone and speakers. You don’t need a LiveKit server for this step. Wear headphones so the agent doesn’t hear itself. To type instead of talk, add --text. Text mode skips TTS and runs only Dialog-RSN-1 and the tools:

lk agent console agent.py --text

Step 5: Test the conversation

The agent greets you with “Hello, Grand Meridian Hotel. How can I help you today?” Then try four things:

Say “Can you look up booking GM40912?” The agent calls look_up_reservation and reads back the booking.
Say “Could I check out at two instead?” The agent tells you whether there’s a fee and asks you to confirm before it calls change_checkout_time.
Ask something with a long answer, then talk over the reply. The agent stops and answers your new question.
Pause mid-sentence, for example while you read out the booking code. The agent should wait for you to finish.

If the agent doesn’t start, see Troubleshooting.

Step 6: Run it in a LiveKit room

To use LiveKit Cloud, put your project’s URL, API key and secret in .env. To stay on your machine, start a local server and keep the defaults from .env.example:

livekit-server --dev

Start the agent. It joins every new room on the server:

uv run agent.py dev

On LiveKit Cloud, you can talk to the agent from your project’s Agent Console. On a local server, or without a microphone, run the scripted caller in a second terminal:

uv run caller.py

The caller joins a new room and plays five recorded lines, each one after the agent finishes its reply. The third line, “Sorry, stop, stop”, plays 1.5 seconds into a reply instead. The first three exchanges look like this:

[  7.4s] agent : Hello, Grand Meridian Hotel. How can I help you today?
[ 14.8s] caller: Hi, can you look up booking G M four zero nine one two please?
[ 22.0s] agent : I've found your booking, Amara. You're currently scheduled to check out at 11:00 on September 30th. What would you like to change?
[ 31.6s] caller: Can you tell me everything about my booking and describe the hotel for me in detail?
[ 34.3s] agent : I can tell you about your booking - you have a Deluxe King room, checking out
[ 39.4s] caller: Sorry, stop, stop. Actually, what time is checkout?
[ 44.0s] agent : Your current checkout time is 11:00 AM on September 30th. Would you like to change that?

The agent stops at “checking out”, where the caller cut in, and Dialog-RSN-1’s history stops there too. When the caller interrupts, LiveKit deletes the full reply from the server and adds back only the words the caller heard. The model builds its next reply on that shorter version. When the call ends, the caller saves the agent’s side to call.wav so you can listen back.

Add Dialog-RSN-1 to an existing LiveKit agent

Install the plugin from GitHub. It isn’t on PyPI yet:

uv add "livekit-plugins-polyai @ git+https://github.com/PolyAI-LDN/livekit-plugins-polyai"

Import it: from livekit.plugins import polyai.
Add POLY_API_KEY to your environment.
In your AgentSession, set llm=polyai.realtime.RealtimeModel(). Remove stt and turn_handling, and keep your tts.
Move any instructions you pass to generate_reply() into the agent’s main instructions.

Why not the OpenAI plugin

You can point LiveKit’s openai.realtime.RealtimeModel at Dialog-RSN-1 by changing base_url and the key, but it’s built for OpenAI’s models. livekit-plugins-polyai is built for Dialog-RSN-1:

LiveKit’s OpenAI plugin	`livekit-plugins-polyai`
Asks for audio by default, so Dialog-RSN-1 speaks the replies in its own voice and LiveKit skips your TTS	Asks for text, so your TTS plugin speaks every reply
Sends OpenAI settings that Dialog-RSN-1 ignores, such as a voice, semantic VAD tuning and a transcription model	Sends only the fields Dialog-RSN-1 uses
Lets you pass OpenAI options that Dialog-RSN-1 rejects, such as `interrupt_response=False`. A rejected `session.update` applies none of its fields	Has no options Dialog-RSN-1 rejects

The plugin implements LiveKit’s realtime model interface directly instead of patching another plugin, so a LiveKit upgrade is less likely to break it. Found a bug, or want a feature? Open an issue or send a pull request.

Choose a different voice

To change the ElevenLabs voice, pass a different voice_id. The repo uses Sarah, which works on the free tier. Some library voices, such as Rachel, don’t. To switch provider, swap the plugin. For Cartesia, install its plugin:

uv add "livekit-agents[cartesia]~=1.8.3"

Add CARTESIA_API_KEY to .env, then change the tts line:

from livekit.plugins import cartesia

tts=cartesia.TTS(),

Nothing else in agent.py changes. LiveKit’s TTS models page lists every provider it supports.

Troubleshooting

Symptom	Likely cause	Fix
`Dialog-RSN-1 needs an API key`	The Dialog-RSN-1 key is missing	Copy `.env.example` to `.env` and set `POLY_API_KEY`
`Dialog-RSN-1 refused the connection (401)`	The Dialog-RSN-1 key is wrong	Create a key in the API Keys tab at studio.poly.ai and set it as `POLY_API_KEY` in `.env`
The agent speaks in a voice you didn’t choose	The session uses LiveKit’s `openai.realtime.RealtimeModel`, which asks Dialog-RSN-1 for audio by default	Use `polyai.realtime.RealtimeModel()`
`ElevenLabs API key is required`	The key is under the wrong name	Rename it to `ELEVEN_API_KEY`
`ws_url is required, or set LIVEKIT_URL`	`agent.py dev` has no LiveKit server to join	Set the `LIVEKIT_*` values in `.env`, or use `lk agent console`
`caller.py` prints “no agent spoke within 30 seconds”	The agent isn’t running, or it’s on a different server	Start `uv run agent.py dev` with the same `LIVEKIT_URL` as the caller
The agent keeps interrupting itself	The microphone picks up the agent’s voice	Wear headphones

What doesn’t carry over

Turn-handling options. LiveKit’s turn detector, endpointing delays and preemptive generation don’t apply, because Dialog-RSN-1 decides when the turn ends.
Per-reply instructions. generate_reply(instructions=...) has no effect, because the plugin doesn’t send them. Put the behavior in the agent’s instructions.
STT features. With no STT plugin, options such as keyterm prompting have nothing to configure. Dialog-RSN-1 transcribes the caller as part of each turn.
Playback reports. The plugin doesn’t send Dialog-RSN-1’s poly.output_audio.* events yet. The streaming text-to-speech guide explains what they add.

Where to go next

The OpenAI Realtime migration guide lists every field Dialog-RSN-1 accepts and rejects.
The prompt guide covers turn-taking instructions in depth.
The API reference lists every client and server event.
The livekit-plugins-polyai README lists every option and how each LiveKit feature maps to Dialog-RSN-1.
LiveKit’s Agents docs cover deployment, telephony and the rest of the framework.

Overview

Quickstart

Prompt guide

Prompt cookbook

Evaluations

Reference

Guides

Build a LiveKit voice agent

Full code on GitHub

What you’ll build

What changes from a standard LiveKit pipeline

Before you begin

Step 1: Get the code

Step 2: Configure your environment

Step 3: Point the session at Dialog-RSN-1

Step 4: Run the agent

Step 5: Test the conversation

Step 6: Run it in a LiveKit room

Add Dialog-RSN-1 to an existing LiveKit agent

Why not the OpenAI plugin

Choose a different voice

Troubleshooting

What doesn’t carry over

Where to go next

Full code on GitHub

​What you’ll build

​What changes from a standard LiveKit pipeline

​Before you begin

​Step 1: Get the code

​Step 2: Configure your environment

​Step 3: Point the session at Dialog-RSN-1

​Step 4: Run the agent

​Step 5: Test the conversation

​Step 6: Run it in a LiveKit room

​Add Dialog-RSN-1 to an existing LiveKit agent

​Why not the OpenAI plugin

​Choose a different voice

​Troubleshooting

​What doesn’t carry over

​Where to go next

What you’ll build

What changes from a standard LiveKit pipeline

Before you begin

Step 1: Get the code

Step 2: Configure your environment

Step 3: Point the session at Dialog-RSN-1

Step 4: Run the agent

Step 5: Test the conversation

Step 6: Run it in a LiveKit room

Add Dialog-RSN-1 to an existing LiveKit agent

Why not the OpenAI plugin

Choose a different voice

Troubleshooting

What doesn’t carry over

Where to go next