Dialog-RSN-1 quickstart for coding agents

Set your coding agent up once, then build a voice agent in three prompts. At each step you’ll know what the agent should build, how to check it works, and what to say when it goes wrong. What you’ll build:

A coding agent that knows the Dialog-RSN-1 protocol and can test its own work.
A script that sends one message and saves the spoken reply to reply.wav.
A live voice conversation: your microphone in, the model’s voice out, and you can interrupt it.
A tool the model calls mid-conversation.

Before you start:

A coding agent that can run shell commands: Claude Code, Codex, Cursor or Gemini CLI.
An empty folder to build in.
Python 3.10 or later with uv. The skill’s connection check runs on it, even if you build in JavaScript.
For Codex, Cursor or Gemini CLI: Node 22.20 or later, which npx skills needs.

The first prompt says “in Python”. Change it to “in JavaScript” if you prefer Node. If you’d rather write the code yourself, follow the quickstart instead.

Set up your agent

Do this once. Every step after it relies on it.

Install the Dialog-RSN-1 skill

Dialog-RSN-1 speaks the OpenAI Realtime protocol. Coding agents know that protocol well, so they assume all of it carries over, and some of it doesn’t. The Dialog-RSN-1 skill gives your agent the differences, the audio patterns, and a script that checks your key against the live API.

Claude Code
Codex, Cursor, Gemini CLI

In Claude Code, run:

/plugin marketplace add polyai/dialog-rsn-1-skill
/plugin install dialog-rsn-1@polyai

The plugin also connects the PolyAI docs as an MCP server, so Claude Code can search them while it works.This installs it for your user account. To share it with everyone on a project, run the same two commands in a terminal with --scope project:

claude plugin marketplace add polyai/dialog-rsn-1-skill --scope project
claude plugin install dialog-rsn-1@polyai --scope project

In your project folder, run:

npx skills add polyai/dialog-rsn-1-skill

This copies the skill into .agents/skills/dialog-rsn-1, where these agents look for skills, and records it in skills-lock.json. If it asks which agents to install for, pick yours. Codex, Cursor and Gemini CLI may be listed together as “universal”. Add -g to install it for all your projects.

Get an API key

Your code reads the key from the POLY_API_KEY environment variable. Don’t paste the key into the chat: the agent never needs to see it.

From your agent: paste the agentic setup prompt into it. It installs the PolyAI CLI, signs you in with GitHub, and saves POLY_API_KEY to your shell profile. Open a new terminal and restart the agent afterwards.
By hand: create a key at studio.poly.ai in the API Keys tab, then put it in a .env file in your folder as POLY_API_KEY=.... The quickstart has the steps.

Check it works

Ask your agent:

Prompt

Run the dialog-rsn-1 skill's connection check.

It runs a spoken turn and a tool turn against the live API. To run it yourself after npx skills add, use uv run .agents/skills/dialog-rsn-1/scripts/check.py. You should see something like this, though the wording and audio length vary:

output

key    POLY_API_KEY from environment
host   wss://api.studio.poly.ai/v1/realtime (default)
ok     connected, session.created
ok     spoken turn: 'Hello! How can I help you today?', 1.9 s of 24 kHz audio
ok     tool call: check_availability({"day":"Friday"})
ok     reply after tool: 'On Friday, I have openings at 9:30 and 2:00.'
PASS

If it fails, the last line says why. A 401 means a wrong key, or a key from a different Studio than the host: see Troubleshooting.

Step 1: hear your first reply

Prompt

In Python, write first_call.py: connect to Dialog-RSN-1, PolyAI's realtime voice
model (my API key is in the POLY_API_KEY environment variable), tell it to be a
friendly concierge for the Harbour Hotel (checkout is at 11, breakfast is 7 to 10 in
the Quay restaurant, the spa opens at 9), send the text "What can you help me with?",
print the reply, and save the spoken reply to reply.wav. Run it to check it works.

The facts in the prompt matter. Dialog-RSN-1 says it doesn’t know rather than make something up, so a concierge with no facts answers most questions with “I don’t have information about that”. What it should build. One WebSocket to wss://api.studio.poly.ai/v1/realtime, with the key in a header. After session.created, a session.update with the instructions. Then the turn, which is always two events: conversation.item.create adds your message, and response.create asks for a reply. The reply streams back as response.output_audio.delta (the speech, 24 kHz PCM16) and response.output_audio_transcript.delta (the words), until response.done. The server speaks the reply itself, so there’s no text-to-speech to set up. The quickstart walks through the same script line by line. Check it. Play reply.wav (afplay reply.wav on macOS, aplay reply.wav on Linux, or open the file). You should hear the concierge offer help with checkout, breakfast or the spa. If it goes wrong:

What you see	Tell your agent
No `reply.wav`, or the reply is printed but there’s no audio	”Leave output_modalities on the default so the server speaks the reply, and save response.output_audio.delta”
`reply.wav` plays too fast or too slow	”The reply audio is 24 kHz mono PCM16. Write the WAV header to match”
The script hangs or exits with nothing printed	”Handle error events and check response.status on response.done”

Step 2: talk to it

Prompt

Now turn it into a live voice conversation in voice_agent.py: stream my microphone
in, play the reply as it arrives, and stop playback when I interrupt. I'll talk to it
myself afterwards, so give it a way to test without a microphone and test it that way.

What it should build. Microphone audio streamed in about 20 ms frames with input_audio_buffer.append, with the sample rate set in audio.input.format. The server decides when you’ve finished speaking, so your code sends no commit and runs no voice detection. Reply audio plays as it arrives. When input_audio_buffer.speech_started arrives while the reply is playing, playback stops at once and queued audio is dropped. The server cancels the rest of the reply. For the test without a microphone, the skill has the agent generate a clip of caller speech and stream it in as if you’d said it. Check it. Run voice_agent.py with headphones on. Ask the concierge something, then interrupt it mid-sentence. It should stop talking and answer what you just said. The mic and speaker need an audio library: in Python, sounddevice (which uses PortAudio); in Node, ffmpeg and ffplay. On macOS, allow your terminal to use the microphone when asked. If it goes wrong:

What you see	Tell your agent
It hears you but never answers	”Set audio.input.format.rate to the mic’s real sample rate, and keep streaming audio during silence”
It keeps talking when you interrupt	”Stop and flush playback as soon as input_audio_buffer.speech_started arrives”
It interrupts itself	Use headphones, or ask: “Turn on echo cancellation for the mic”
It only answers after you press a key (the code sets `turn_detection` to `null`, which is push-to-talk)	“Use the default server_vad turn detection, and don’t commit the audio buffer yourself”
`unsupported_parameter` naming `idle_timeout_ms` or `interrupt_response: false`	”Remove that field. The server handles turn-taking and interruptions”

Step 3: give it a tool

Prompt

Add a check_availability(day) tool to voice_agent.py that returns made-up appointment
slots for the hotel spa, and have the concierge offer them to me. Test it.

What it should build. A function tool in session.tools. When the model calls it, response.function_call_arguments.done carries the name, the arguments and a call_id, and that response ends. The code runs the function, sends the result back as a function_call_output item, then sends response.create again, so the model can speak the answer. Check it. Ask “Is the spa free on Friday?” You should hear the slots read back. If it goes wrong:

What you see	Tell your agent
The concierge goes quiet after the tool runs	”Send response.create after the function_call_output”
`unsupported_parameter` naming `tool_choice` or `tools[0].type`	”Use function tools only, with tool_choice auto”

Build your own

You now have a voice agent that listens, talks, takes interruptions and calls tools. Some prompts to take it further. Each has a page of prompts in the same shape as this one, and a guide that walks through the code:

Prompt	With your agent	Guide
”Build a browser version with a small Node relay that holds the key”	Browser	Browser with JavaScript
”Speak the replies with my own ElevenLabs voice instead of the server voice”	Your own TTS	Streaming to your TTS
”Port this OpenAI Realtime app to Dialog-RSN-1”	OpenAI Realtime migration	OpenAI Realtime migration
”Make this a LiveKit agent”	LiveKit	LiveKit

Voice agents read differently from chat. Ask for short replies, no lists or markdown, and numbers written the way they’re said. Then test by talking, not by reading the transcript. The prompt guide covers what instructions can and can’t do.

Without the skill

Point your agent at the docs instead. Every page is available as Markdown: add .md to its URL. The index of every page is docs.poly.ai/llms.txt. Start the agent with:

Prompt

Read https://docs.poly.ai/api-reference/dialog-rsn-1/reference.md before writing any
code. Dialog-RSN-1 uses the OpenAI Realtime protocol, but follow that page wherever it
differs from what you know.

The docs are also an MCP server at https://docs.poly.ai/mcp, which lets your agent search them:

Claude Code
Codex
Gemini CLI
Cursor

claude mcp add --transport http polyai-docs https://docs.poly.ai/mcp

codex mcp add polyai-docs --url https://docs.poly.ai/mcp

gemini mcp add --transport http polyai-docs https://docs.poly.ai/mcp

Add this to .cursor/mcp.json:

{ "mcpServers": { "polyai-docs": { "url": "https://docs.poly.ai/mcp" } } }

Troubleshooting

Symptom	Likely cause	Tell your agent
`401` on connect	A wrong key, or a key from a different Studio than the host. A studio.poly.ai key only works on `api.studio.poly.ai`	If your key is from `studio.us.poly.ai`: “Set DIALOG_REALTIME_URL to wss://api.us.poly.ai/v1/realtime”
`401` from your code, but the connection check passes	The key and the host come from different places, such as a key in your shell profile and a `DIALOG_REALTIME_URL` in `.env`	”Read POLY_API_KEY and DIALOG_REALTIME_URL from the same place, and print where each one came from”
`402` on connect	No balance on the account	Nothing: add funds at studio.poly.ai
`429` with `session_limit_reached`	Too many sessions open on your account at once, often a test run that’s still connected or another program on your account. The cap is 2 until a card is on file: see limits	Nothing: close the other session, or wait for it to end. There’s no `Retry-After`
`POLY_API_KEY is not set`	The agent started before the key was saved	Open a new terminal, restart the agent, and ask again
Empty reply in a text UI	Replies are spoken by default	”Set output_modalities to text and read response.output_text.delta”
`audio_synthesis_failed`	The voice id isn’t a real Cartesia voice	”Leave audio.output.voice unset, or use a real Cartesia voice id”
`conversation_already_has_active_response`	A new `response.create` before the last one finished	”Wait for response.done before the next response.create”
`unsupported_parameter` or `unknown_parameter`	A field this API refuses	”Remove the field named in error.param”

Every error code is in the reference.

Where to go next

Dialog-RSN-1 skill

The skill’s rules, references and connection check.

Reference

Every event, the session object, function calling and error codes.

Guides

Browser, your own TTS, speech to text, LiveKit and more.

Limits and pricing

Concurrency, session length and the connected-minute rate.

Overview

Quickstart

Prompt guide

Prompt cookbook

Evaluations

Reference

Coding agents

Guides

Dialog-RSN-1 quickstart for coding agents

Set up your agent

Install the Dialog-RSN-1 skill

Get an API key

Check it works

Step 1: hear your first reply

Step 2: talk to it

Step 3: give it a tool

Build your own

Without the skill

Troubleshooting

Where to go next

Dialog-RSN-1 skill

Reference

Guides

Limits and pricing

​Set up your agent

​Install the Dialog-RSN-1 skill

​Get an API key

​Check it works

​Step 1: hear your first reply

​Step 2: talk to it

​Step 3: give it a tool

​Build your own

​Without the skill

​Troubleshooting

​Where to go next

Dialog-RSN-1 skill

Reference

Guides

Limits and pricing

Set up your agent

Install the Dialog-RSN-1 skill

Get an API key

Check it works

Step 1: hear your first reply

Step 2: talk to it

Step 3: give it a tool

Build your own

Without the skill

Troubleshooting

Where to go next