Transcribe speech with the OpenAI Realtime SDK

Stream your microphone to Dialog-RSN-1 through the official openai SDK, and print each thing you say as a line of text. Then change the instructions and the same program becomes a support agent that hears the caller and answers in text. The SDK needs a new base URL and key, nothing else.

Full code on GitHub

A terminal transcriber in Python, and a voice-in, text-out support agent in Python and Node.

What you’ll build

A transcriber. Speak, and each finished sentence prints with a timestamp, optionally appended to a file.
A support agent. Speak, and Sam from a fictional outdoor store answers in text about three orders held in the prompt.
The same support agent in Node, on the npm openai package.

How transcription works on Dialog-RSN-1

Dialog-RSN-1 has no separate speech-to-text model. It transcribes the caller as part of writing its reply, so every transcript belongs to a turn. That shapes the whole program:

You need a reply to get a transcript. Set create_response: true so each turn starts a reply on its own. With create_response: false, no transcript arrives until you send response.create.
The transcriber asks for a one-word reply. Its instructions say to answer “ok”, which keeps each turn short.
The transcript arrives when the turn ends. You get no partial text while the caller is still talking. The whole line lands in a few events just after input_audio_buffer.speech_stopped.
Only a completed reply sends .completed. If the caller cuts in while the reply is still generating, or the reply fails, the turn gets at most some .delta events and no .completed. The transcript is the last thing the model writes, so a turn interrupted early may get no transcript at all.
The reply starts before the transcript. The first response.output_text.delta comes within about 100 ms of speech_stopped, and the transcript follows. The support agent holds the reply back so the terminal reads in order.
audio.input.transcription changes nothing. The service accepts the field and echoes it in session.updated, but transcripts arrive the same with or without it.

Prerequisites

A Dialog-RSN-1 API key. Sign up or log in at studio.poly.ai and create one in the API Keys tab. The quickstart walks through it.
For Python: Python 3.12 and uv.
For Node: Node 22. npm install compiles the naudiodon2 audio package, so you need a C++ toolchain.
A microphone, or a 24 kHz, 16-bit mono WAV file to stream instead.

Run it

git clone https://github.com/PolyAI-LDN/dialog-rsn-1-speech-to-text
cd dialog-rsn-1-speech-to-text
cp .env.example .env      # add POLY_API_KEY
cd python
uv sync
uv run transcribe.py

Speak, and each line prints when you finish the sentence. Add --reply to see the model’s “ok” under each line, and --out notes.txt to append each line to a file:

connecting to wss://api.studio.poly.ai/v1/realtime …
[connected to dialog-rsn-1]  speak when ready, Ctrl-C to quit

23:10:43  Hi, I'm calling about order N W one zero four eight two. It's under Maria Alvarez.
          ↳ ok
23:10:49  Can I still cancel it?
          ↳ ok

To test without a microphone, record a clip with the macOS say command and stream it at real time:

say -v Samantha -o clip.aiff "Table for two on Friday, please."
afconvert -f WAVE -d LEI16@24000 -c 1 clip.aiff clip.wav
uv run transcribe.py --file clip.wav

Step 1: Connect with the SDK

Pass the Dialog-RSN-1 base URL and key to the client. url is the full realtime URL, so drop /realtime and switch the scheme to get the base. The SDK builds wss://api.studio.poly.ai/v1/realtime?model=dialog-rsn-1, sends the key as Authorization: Bearer, and parses every event into its typed models.

# The SDK takes the base URL and appends /realtime itself.
base_url = url.removesuffix("/realtime").replace("wss://", "https://", 1)
client = AsyncOpenAI(api_key=key, base_url=base_url)
stop = asyncio.Event()
print(f"connecting to {url} …")
async with client.realtime.connect(model=MODEL) as conn:
    await conn.session.update(session=SESSION)

// The SDK builds wss://<host>/v1/realtime?model=... from the client's base URL and sends the
// key as Authorization: Bearer. Nothing else about the connection is ours.
const baseURL = url.replace(/\/realtime$/, "").replace(/^wss:/, "https:");
const client = new OpenAI({ apiKey, baseURL });
const rt = new OpenAIRealtimeWS({ model: MODEL }, client);

url defaults to wss://api.studio.poly.ai/v1/realtime. Set DIALOG_REALTIME_URL in .env, or pass --url, to point somewhere else.

Step 2: Configure the session

SESSION = {
    "type": "realtime",
    "output_modalities": ["text"],
    "instructions": INSTRUCTIONS,
    "audio": {
        "input": {
            "format": {"type": "audio/pcm", "rate": SAMPLE_RATE},
            "turn_detection": {"type": "server_vad", "create_response": True},
        }
    },
}

output_modalities: ["text"]. This program prints text, so it asks for text. A new session defaults to ["audio"], where the server speaks the reply and sends its words as response.output_audio_transcript.*. Text only keeps the reply on response.output_text.*, which the program reads.
rate: 24000. The program opens the microphone at 24 kHz, so nothing resamples on the client. The service resamples to the model’s own rate.
create_response: true. Dialog-RSN-1 decides when you’ve finished and starts the turn itself. That turn carries the transcript.

The instructions ask for the shortest possible reply:

INSTRUCTIONS = (
    "You are a silent transcription service. Whatever the caller says, reply with exactly "
    "one word: ok."
)

Step 3: Stream the microphone

sounddevice calls back with 100 ms of PCM16 at a time. The program base64-encodes each chunk and appends it to the input buffer:

def on_audio(indata, frames, time_info, status):
    if status:
        print(f"[audio] {status}", file=sys.stderr)
    loop.call_soon_threadsafe(queue.put_nowait, bytes(indata))

stream = sd.RawInputStream(
    samplerate=SAMPLE_RATE,
    channels=1,
    dtype="int16",
    blocksize=CHUNK_FRAMES,
    device=device,
    callback=on_audio,
)
with stream:
    while not stop.is_set():
        chunk = await queue.get()
        await conn.input_audio_buffer.append(audio=b64(chunk))

The program never commits the buffer or asks for a reply. Dialog-RSN-1 does both when it hears the end of the turn.

Step 4: Print the transcript

Two events carry the transcript. conversation.item.input_audio_transcription.delta streams it in pieces, and conversation.item.input_audio_transcription.completed carries the whole line:

elif kind == "conversation.item.input_audio_transcription.delta":
    if not partial:
        print("\r\033[K", end="")
    partial += event.delta
    print(event.delta, end="", flush=True)
elif kind == "conversation.item.input_audio_transcription.completed":
    line(event.transcript or partial)
    partial = ""
elif kind == "response.done" and partial:
    # A reply cut off by a barge-in, or one that failed, sends no .completed.
    line(partial)
    partial = ""

line() stamps the text with the time, prints it and appends it to the --out file. The response.done branch keeps what arrived when a reply ends without a .completed, so a line isn’t left half printed. Stop on invalid_request_error. It usually means the session.update was refused, which leaves the session on the server’s defaults:

elif kind == "error":
    print(f"[error] {event.error.message}", file=sys.stderr)
    if event.error.type == "invalid_request_error":
        stop.set()
        break

Turn it into a support agent

support_agent.py keeps the connection, the session shape and the audio path. It changes three things. The instructions. prompt.py holds everything Sam knows: three orders in different states, the return and cancellation policy, and a rule to confirm the order before giving details. There are no tools. The data sits in the prompt. A seeded greeting. The program prints the greeting itself and adds it to the conversation as an assistant message, so the model’s history starts with it:

await conn.conversation.item.create(
    item={
        "type": "message",
        "role": "assistant",
        "content": [{"type": "output_text", "text": GREETING}],
    }
)

rt.send({
  type: "conversation.item.create",
  item: {
    type: "message",
    role: "assistant",
    content: [{ type: "output_text", text: GREETING }],
  },
});

The reply waits for the transcript. Sam’s reply starts before the transcript of your line arrives. Printed in arrival order, the answer would sit above the question. The program buffers the reply until your line has printed:

elif kind == "conversation.item.input_audio_transcription.completed":
    print(f"\r\033[K{stamp()}  You:  {(event.transcript or heard).strip()}")
    heard = ""
    if reply:
        print(f"{stamp()}  Sam:  {reply}", end="", flush=True)
        reply_shown = True
elif kind == "response.output_text.delta":
    reply += event.delta
    if reply_shown:
        print(event.delta, end="", flush=True)

Run it, in either language:

cd python
uv run support_agent.py

cd node
npm install
npm start

Say “I’m calling about order N W one zero four eight two, it’s under Maria Alvarez”, then “Can I still cancel it?”. That order has shipped, so Sam refuses and points you to a return:

02:35  Sam:  Thanks for calling Northwind Outfitters, this is Sam. How can I help?

02:42  You:  Hi, I'm calling about order N, W, one zero four eight two. It's under Maria Alvarez.
02:42  Sam:  Got it, Maria. Your order shipped on September 15th via UPS and should arrive by September 19th. Is there anything specific you wanted to know about it?

02:47  You:  Can I still cancel it?
02:47  Sam:  I'm afraid not, Maria. Your order has already shipped on September 15th, so it can't be cancelled. You'll need to return it once it arrives. Would you like me to explain the return policy?

Both programs take --file call.wav to stream a recording instead of the microphone, and --list-devices and --device to pick a microphone.

What the timing looks like

We streamed recorded clips to us-1 production over 14 turns, 20 ms at a time as a microphone would, from London (about 90 ms round trip):

Measure	Time
Caller’s last word to `speech_stopped`	about 610 to 810 ms
`speech_stopped` to the first reply text	0 to 100 ms
`speech_stopped` to the finished transcript line	30 to 310 ms
Caller’s last word to the finished transcript line	about 750 to 1,020 ms

speech_stopped reports audio_end_ms, the point in your audio where Dialog-RSN-1 places the end of speech. That mark sits about 400 to 570 ms after the last word, so time your own measurements from the audio, not from audio_end_ms.

Troubleshooting

Symptom	Likely cause	Fix
`POLY_API_KEY is not set`	No `.env`, an empty key, or a `.env` that still uses an old name, `DIALOG_API_KEY` or `DIALOGUE_API_KEY`	Copy `.env.example` to `.env` at the repo root and paste the key after `POLY_API_KEY=`
HTTP 401 when connecting	A missing or wrong key, or a personal access token	Create a key in the API Keys tab at studio.poly.ai and set it as `POLY_API_KEY` in `.env`
No transcript ever arrives	`create_response` is `false`, so no turn runs	Set it to `true`, or send `response.create` after each `input_audio_buffer.committed`
`unsupported_parameter` after `session.update`	A field the service refuses, such as `idle_timeout_ms` or `interrupt_response: false` in `turn_detection`, or `reasoning`	Remove it. A rejected update applies none of its fields
`must be mono, 16-bit, 24000 Hz`	The WAV isn’t in the session’s format	Convert it with `afconvert -f WAVE -d LEI16@24000 -c 1 in.wav out.wav`
Sam’s answer prints above your question	Replies printed in arrival order	Hold the reply until `input_audio_transcription.completed`, as `support_agent.py` does
`npm install` fails building `naudiodon2`	No C++ toolchain	On a Mac, run `xcode-select --install`, then `npm install` again

What this isn’t

The transcriber shows that a stock OpenAI Realtime client works unchanged against Dialog-RSN-1. It isn’t a full speech-to-text service. It has no speaker labels, no punctuation settings, and no partial results while the caller speaks.

Where to go next

The OpenAI Realtime migration guide lists every field Dialog-RSN-1 accepts and rejects.
The streaming text-to-speech guide gives the support agent a voice.
The API reference lists every client and server event.

Overview

Quickstart

Prompt guide

Prompt cookbook

Evaluations

Reference

Guides

Transcribe speech with the OpenAI Realtime SDK

Full code on GitHub

What you’ll build

How transcription works on Dialog-RSN-1

Prerequisites

Run it

Step 1: Connect with the SDK

Step 2: Configure the session

Step 3: Stream the microphone

Step 4: Print the transcript

Turn it into a support agent

What the timing looks like

Troubleshooting

What this isn’t

Where to go next

Full code on GitHub

​What you’ll build

​How transcription works on Dialog-RSN-1

​Prerequisites

​Run it

​Step 1: Connect with the SDK

​Step 2: Configure the session

​Step 3: Stream the microphone

​Step 4: Print the transcript

​Turn it into a support agent

​What the timing looks like

​Troubleshooting

​What this isn’t

​Where to go next

What you’ll build

How transcription works on Dialog-RSN-1

Prerequisites

Run it

Step 1: Connect with the SDK

Step 2: Configure the session

Step 3: Stream the microphone

Step 4: Print the transcript

Turn it into a support agent

What the timing looks like

Troubleshooting

What this isn’t

Where to go next