Dialog-RSN-1 quickstart

Prerequisites: an API key and a WebSocket client in any language.

Getting an API key

You create the key in Agent Studio at studio.poly.ai:

Go to studio.poly.ai and sign up, or log in if you already have an account.
Open the API Keys tab in your workspace’s top bar, then select API key.
Give the key a name. Leave Agents on All agents, and leave at least one Resource Access permission on. The form needs one, and Dialog-RSN-1 accepts the key whichever you pick.
Select Create, then copy the key. It’s only shown once.

It’s a single bearer token that works on your first connection. Store it in an environment variable, such as POLY_API_KEY, or in a secret manager rather than in source control.

A connection is a conversation

You open one WebSocket and keep it open. One connection carries the whole conversation: you authenticate once, and every turn travels over the same socket as JSON text frames. To open it, connect and send your API key in the X-API-KEY header:

connect

URL:     wss://api.studio.poly.ai/v1/realtime
Header:  X-API-KEY: YOUR_API_KEY

Authorization: Bearer YOUR_API_KEY works too, so an existing realtime client, including the official openai SDK, keeps its auth and only needs a new URL and key.

Keep your key server-side. Treat it like any API credential: call it from a server or trusted backend, not from code that ships to a browser or a device you don’t control. A browser’s native WebSocket can’t set headers at all, so connecting from a page means putting a relay in front, as the browser guide covers.

The moment the connection is accepted, before you’ve sent anything, the server sends you a session.created event. You’re already in a session, running default settings you’ll want to change: 24 kHz PCM16 audio in, server turn detection, and spoken replies.

Telling it how to behave

Connecting gives you a working session before you configure anything, so configuration is a separate step rather than a set of required connection parameters. Send session.update with your instructions, tools, and anything else; the server replies with session.updated, echoing back the full configuration in effect.

you send: session.update

{
  "type": "session.update",
  "session": {
    "instructions": "You are a helpful, concise assistant."
  }
}

A new session speaks its replies. Dialog-RSN-1 writes the answer and the server voices it, so you get the reply as audio plus a transcript of what it says, with no text-to-speech of your own. To speak replies with your own voice instead, see bring your own text-to-speech below. Session updates are cheap; send another one later if your needs change mid-conversation.

The shape of a turn

Every turn is the same two-part move. Adding something to the conversation and asking for a response to it are separate events:

you send: conversation.item.create

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "content": [{ "type": "input_text", "text": "What's the capital of France?" }]
  }
}

you send: response.create

{ "type": "response.create" }

That separation pays off twice later. You can seed a conversation with prior history without firing a reply at each item. Once you’re using tools, you add the tool’s result, then ask for the follow-up in a second step, because a tool result on its own doesn’t end a turn. For now, the two travel together: add the message, then ask for a response to it.

Watching the answer arrive

The reply comes back as a short burst of events rather than one block of JSON. Three of them matter at first:

response.output_audio_transcript.delta carries the words of the reply. Append them to see what it says.
response.output_audio.delta carries the speech: base64 PCM16 audio, 24 kHz mono. Play it, or collect it.
response.done tells you the turn is over and how it ended.

Ignore the rest for now.

server sends: the response

response.created
response.output_item.added
response.content_part.added
response.output_audio_transcript.delta   delta: "The capital "
response.output_audio_transcript.delta   delta: "of France is Paris."
response.output_audio.delta              delta: "<base64 PCM16>"
response.output_audio.delta              delta: "<base64 PCM16>"
response.output_audio.done
response.output_audio_transcript.done
response.content_part.done
response.output_item.done
response.done                            response.status: "completed"

dialog-rsn-1: “The capital of France is Paris.”

The audio streams at speaking pace, so response.done arrives once the reply has been spoken. The events between response.created and response.done track which output item the server is writing and where its content part begins and ends. A fuller integration will use them. For a first pass, the deltas and the final status are enough to hold a conversation. That’s the loop: add an item, ask for a response, collect the deltas until response.done. Repeat it for a longer conversation. Each new message builds on the conversation so far, so you have no separate history to maintain.

All of it, together

The same sequence as one script, running on the standard websockets package. It prints the reply as it streams and saves the speech to reply.wav. You need Python 3.10 or later, pip install 'websockets>=14', and POLY_API_KEY set in your shell. The version floor matters: connect() took extra_headers before websockets 14 and additional_headers from 14 on, so the script below raises TypeError on an older install rather than failing to authenticate.

python

# pip install 'websockets>=14'
import asyncio, base64, json, os, wave, websockets

URL = "wss://api.studio.poly.ai/v1/realtime"
API_KEY = os.environ["POLY_API_KEY"]

async def main():
    headers = {"X-API-KEY": API_KEY}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        await ws.recv()  # session.created

        await ws.send(json.dumps({
            "type": "session.update",
            "session": {"instructions": "Be concise."},
        }))
        event = json.loads(await ws.recv())
        if event["type"] != "session.updated":
            raise RuntimeError(event["error"]["message"])

        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {
                "type": "message", "role": "user",
                "content": [{"type": "input_text", "text": "What's the capital of France?"}],
            },
        }))
        await ws.send(json.dumps({"type": "response.create"}))

        audio = bytearray()
        async for raw in ws:
            event = json.loads(raw)
            if event["type"] == "response.output_audio_transcript.delta":
                print(event["delta"], end="", flush=True)
            elif event["type"] == "response.output_audio.delta":
                audio += base64.b64decode(event["delta"])
            elif event["type"] == "error":
                # The connection stays open after an error, so a turn can fail without the
                # socket closing. Without this branch the script exits with nothing said.
                raise RuntimeError(event["error"]["message"])
            elif event["type"] == "response.done":
                status = event["response"]["status"]
                if status != "completed":
                    raise RuntimeError(f"response {status}")
                print()
                break

        with wave.open("reply.wav", "wb") as wav:
            wav.setnchannels(1)
            wav.setsampwidth(2)  # PCM16
            wav.setframerate(24000)
            wav.writeframes(audio)
        print(f"Saved {len(audio) / 48000:.1f} s of audio to reply.wav")

asyncio.run(main())

output

The capital of France is Paris.
Saved 2.5 s of audio to reply.wav

Play reply.wav in any audio player, or with afplay reply.wav on macOS. The error branch is not optional padding. Almost every failure on this API answers with an error event and leaves the connection open, so a turn that fails arrives as an event rather than a dropped socket. The few that close the socket, such as an oversized frame or an idle connection, are listed under close codes. Skip the branch and a failed generation prints nothing and exits zero. The same goes for session.update: a refused update answers with error instead of session.updated and applies nothing, which is why the script checks the reply.

Bring your own text-to-speech

The server’s voice is the quickest way to a spoken reply. When you want your own voice, your own TTS vendor, or control over playback, ask for text instead. Set output_modalities to ["text"]:

you send: session.update

{
  "type": "session.update",
  "session": {
    "instructions": "You are a helpful, concise assistant.",
    "output_modalities": ["text"]
  }
}

The reply then streams as response.output_text.delta, with no audio events, and you speak it however you like. In the script above, add "output_modalities": ["text"] to the session and read response.output_text.delta in place of response.output_audio_transcript.delta. The streaming text-to-speech guide takes it from there. Ask for one or the other: ["text", "audio"] is refused.

Beyond a single turn

The same add-then-respond shape extends in two directions:

Function calling: declare tools in session.tools; a call arrives as response.function_call_arguments.done, and you answer it with a function_call_output item, followed by another response.create, the same two-part pattern from earlier, with a tool result in place of a user message.
Voice in: append PCM16 audio via input_audio_buffer.append instead of creating a text item; Dialog-RSN-1 transcribes speech as part of the same response, streaming the transcript alongside the reply.

For the full event list, field reference, and error codes, see the Dialog-RSN-1 reference.

Overview

What realtime streaming is, and how turn-taking, generation, and prompting fit together.

Dialog-RSN-1 reference

Every event, the session object, function calling, and error codes.

Guides

Build a voice agent from scratch, or on top of a framework you already use.

Error codes

The platform-wide error reference.

Overview

Quickstart

Prompt guide

Prompt cookbook

Evaluations

Reference

Guides

Getting an API key

A connection is a conversation

Telling it how to behave

The shape of a turn

Watching the answer arrive

All of it, together

Bring your own text-to-speech

Beyond a single turn

Overview

Dialog-RSN-1 reference

Guides

Error codes

​Getting an API key

​A connection is a conversation

​Telling it how to behave

​The shape of a turn

​Watching the answer arrive

​All of it, together

​Bring your own text-to-speech

​Beyond a single turn

Overview

Dialog-RSN-1 reference

Guides

Error codes

Getting an API key

A connection is a conversation

Telling it how to behave

The shape of a turn

Watching the answer arrive

All of it, together

Bring your own text-to-speech

Beyond a single turn