> ## Documentation Index > Fetch the complete documentation index at: https://docs.poly.ai/llms.txt > Use this file to discover all available pages before exploring further. # Transcribe speech with the OpenAI Realtime SDK > Stream a microphone to Dialog-RSN-1 with the stock openai SDK in Python or Node and print a line of text per turn, then grow it into a voice-in, text-out support agent. Stream your microphone to Dialog-RSN-1 through the official `openai` SDK, and print each thing you say as a line of text. Then change the instructions and the same program becomes a support agent that hears the caller and answers in text. The SDK needs a new base URL and key, nothing else. A terminal transcriber in Python, and a voice-in, text-out support agent in Python and Node. ## What you'll build 1. A transcriber. Speak, and each finished sentence prints with a timestamp, optionally appended to a file. 2. A support agent. Speak, and Sam from a fictional outdoor store answers in text about three orders held in the prompt. 3. The same support agent in Node, on the npm `openai` package. ## How transcription works on Dialog-RSN-1 Dialog-RSN-1 has no separate speech-to-text model. It transcribes the caller as part of writing its reply, so every transcript belongs to a turn. That shapes the whole program: * **You need a reply to get a transcript.** Set `create_response: true` so each turn starts a reply on its own. With `create_response: false`, no transcript arrives until you send `response.create`. * **The transcriber asks for a one-word reply.** Its instructions say to answer "ok", which keeps each turn short. * **The transcript arrives when the turn ends.** You get no partial text while the caller is still talking. The whole line lands in a few events just after `input_audio_buffer.speech_stopped`. * **Only a completed reply sends `.completed`.** If the caller cuts in while the reply is still generating, or the reply fails, the turn gets at most some `.delta` events and no `.completed`. The transcript is the last thing the model writes, so a turn interrupted early may get no transcript at all. * **The reply starts before the transcript.** The first `response.output_text.delta` comes within about 100 ms of `speech_stopped`, and the transcript follows. The support agent holds the reply back so the terminal reads in order. * **`audio.input.transcription` changes nothing.** The service accepts the field and echoes it in `session.updated`, but transcripts arrive the same with or without it. ## Prerequisites 1. A Dialog-RSN-1 API key. Sign up or log in at [studio.poly.ai](https://studio.poly.ai) and create one in the **API Keys** tab. The [quickstart](/dialog-rsn-1/quickstart#getting-an-api-key) walks through it. 2. For Python: Python 3.12 and [uv](https://docs.astral.sh/uv/). 3. For Node: Node 22. `npm install` compiles the `naudiodon2` audio package, so you need a C++ toolchain. 4. A microphone, or a 24 kHz, 16-bit mono WAV file to stream instead. ## Run it ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} git clone https://github.com/PolyAI-LDN/dialog-rsn-1-speech-to-text cd dialog-rsn-1-speech-to-text cp .env.example .env # add POLY_API_KEY cd python uv sync uv run transcribe.py ``` Speak, and each line prints when you finish the sentence. Add `--reply` to see the model's "ok" under each line, and `--out notes.txt` to append each line to a file: ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} connecting to wss://api.studio.poly.ai/v1/realtime … [connected to dialog-rsn-1] speak when ready, Ctrl-C to quit 23:10:43 Hi, I'm calling about order N W one zero four eight two. It's under Maria Alvarez. ↳ ok 23:10:49 Can I still cancel it? ↳ ok ``` To test without a microphone, record a clip with the macOS `say` command and stream it at real time: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} say -v Samantha -o clip.aiff "Table for two on Friday, please." afconvert -f WAVE -d LEI16@24000 -c 1 clip.aiff clip.wav uv run transcribe.py --file clip.wav ``` ## Step 1: Connect with the SDK Pass the Dialog-RSN-1 base URL and key to the client. `url` is the full realtime URL, so drop `/realtime` and switch the scheme to get the base. The SDK builds `wss://api.studio.poly.ai/v1/realtime?model=dialog-rsn-1`, sends the key as `Authorization: Bearer`, and parses every event into its typed models. ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} # The SDK takes the base URL and appends /realtime itself. base_url = url.removesuffix("/realtime").replace("wss://", "https://", 1) client = AsyncOpenAI(api_key=key, base_url=base_url) stop = asyncio.Event() print(f"connecting to {url} …") async with client.realtime.connect(model=MODEL) as conn: await conn.session.update(session=SESSION) ``` ```javascript Node theme={"theme":{"light":"github-light","dark":"github-dark"}} // The SDK builds wss:///v1/realtime?model=... from the client's base URL and sends the // key as Authorization: Bearer. Nothing else about the connection is ours. const baseURL = url.replace(/\/realtime$/, "").replace(/^wss:/, "https:"); const client = new OpenAI({ apiKey, baseURL }); const rt = new OpenAIRealtimeWS({ model: MODEL }, client); ``` `url` defaults to `wss://api.studio.poly.ai/v1/realtime`. Set `DIALOG_REALTIME_URL` in `.env`, or pass `--url`, to point somewhere else. ## Step 2: Configure the session ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} SESSION = { "type": "realtime", "output_modalities": ["text"], "instructions": INSTRUCTIONS, "audio": { "input": { "format": {"type": "audio/pcm", "rate": SAMPLE_RATE}, "turn_detection": {"type": "server_vad", "create_response": True}, } }, } ``` * `output_modalities: ["text"]`. This program prints text, so it asks for text. A new session defaults to `["audio"]`, where the server speaks the reply and sends its words as `response.output_audio_transcript.*`. Text only keeps the reply on `response.output_text.*`, which the program reads. * `rate: 24000`. The program opens the microphone at 24 kHz, so nothing resamples on the client. The service resamples to the model's own rate. * `create_response: true`. Dialog-RSN-1 decides when you've finished and starts the turn itself. That turn carries the transcript. The instructions ask for the shortest possible reply: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} INSTRUCTIONS = ( "You are a silent transcription service. Whatever the caller says, reply with exactly " "one word: ok." ) ``` ## Step 3: Stream the microphone `sounddevice` calls back with 100 ms of PCM16 at a time. The program base64-encodes each chunk and appends it to the input buffer: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} def on_audio(indata, frames, time_info, status): if status: print(f"[audio] {status}", file=sys.stderr) loop.call_soon_threadsafe(queue.put_nowait, bytes(indata)) stream = sd.RawInputStream( samplerate=SAMPLE_RATE, channels=1, dtype="int16", blocksize=CHUNK_FRAMES, device=device, callback=on_audio, ) with stream: while not stop.is_set(): chunk = await queue.get() await conn.input_audio_buffer.append(audio=b64(chunk)) ``` The program never commits the buffer or asks for a reply. Dialog-RSN-1 does both when it hears the end of the turn. ## Step 4: Print the transcript Two events carry the transcript. `conversation.item.input_audio_transcription.delta` streams it in pieces, and `conversation.item.input_audio_transcription.completed` carries the whole line: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} elif kind == "conversation.item.input_audio_transcription.delta": if not partial: print("\r\033[K", end="") partial += event.delta print(event.delta, end="", flush=True) elif kind == "conversation.item.input_audio_transcription.completed": line(event.transcript or partial) partial = "" elif kind == "response.done" and partial: # A reply cut off by a barge-in, or one that failed, sends no .completed. line(partial) partial = "" ``` `line()` stamps the text with the time, prints it and appends it to the `--out` file. The `response.done` branch keeps what arrived when a reply ends without a `.completed`, so a line isn't left half printed. Stop on `invalid_request_error`. It usually means the `session.update` was refused, which leaves the session on the server's defaults: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} elif kind == "error": print(f"[error] {event.error.message}", file=sys.stderr) if event.error.type == "invalid_request_error": stop.set() break ``` ## Turn it into a support agent `support_agent.py` keeps the connection, the session shape and the audio path. It changes three things. **The instructions.** `prompt.py` holds everything Sam knows: three orders in different states, the return and cancellation policy, and a rule to confirm the order before giving details. There are no tools. The data sits in the prompt. **A seeded greeting.** The program prints the greeting itself and adds it to the conversation as an assistant message, so the model's history starts with it: ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} await conn.conversation.item.create( item={ "type": "message", "role": "assistant", "content": [{"type": "output_text", "text": GREETING}], } ) ``` ```javascript Node theme={"theme":{"light":"github-light","dark":"github-dark"}} rt.send({ type: "conversation.item.create", item: { type: "message", role: "assistant", content: [{ type: "output_text", text: GREETING }], }, }); ``` **The reply waits for the transcript.** Sam's reply starts before the transcript of your line arrives. Printed in arrival order, the answer would sit above the question. The program buffers the reply until your line has printed: ```python theme={"theme":{"light":"github-light","dark":"github-dark"}} elif kind == "conversation.item.input_audio_transcription.completed": print(f"\r\033[K{stamp()} You: {(event.transcript or heard).strip()}") heard = "" if reply: print(f"{stamp()} Sam: {reply}", end="", flush=True) reply_shown = True elif kind == "response.output_text.delta": reply += event.delta if reply_shown: print(event.delta, end="", flush=True) ``` Run it, in either language: ```bash Python theme={"theme":{"light":"github-light","dark":"github-dark"}} cd python uv run support_agent.py ``` ```bash Node theme={"theme":{"light":"github-light","dark":"github-dark"}} cd node npm install npm start ``` Say "I'm calling about order N W one zero four eight two, it's under Maria Alvarez", then "Can I still cancel it?". That order has shipped, so Sam refuses and points you to a return: ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} 23:02:35 Sam: Thanks for calling Northwind Outfitters, this is Sam. How can I help? 23:02:42 You: Hi, I'm calling about order N, W, one zero four eight two. It's under Maria Alvarez. 23:02:42 Sam: Got it, Maria. Your order shipped on September 15th via UPS and should arrive by September 19th. Is there anything specific you wanted to know about it? 23:02:47 You: Can I still cancel it? 23:02:47 Sam: I'm afraid not, Maria. Your order has already shipped on September 15th, so it can't be cancelled. You'll need to return it once it arrives. Would you like me to explain the return policy? ``` Both programs take `--file call.wav` to stream a recording instead of the microphone, and `--list-devices` and `--device` to pick a microphone. ## What the timing looks like We streamed recorded clips to us-1 production over 14 turns, 20 ms at a time as a microphone would, from London (about 90 ms round trip): | Measure | Time | | - | - | | Caller's last word to `speech_stopped` | about 610 to 810 ms | | `speech_stopped` to the first reply text | 0 to 100 ms | | `speech_stopped` to the finished transcript line | 30 to 310 ms | | Caller's last word to the finished transcript line | about 750 to 1,020 ms | `speech_stopped` reports `audio_end_ms`, the point in your audio where Dialog-RSN-1 places the end of speech. That mark sits about 400 to 570 ms after the last word, so time your own measurements from the audio, not from `audio_end_ms`. ## Troubleshooting | Symptom | Likely cause | Fix | | - | - | - | | `POLY_API_KEY is not set` | No `.env`, an empty key, or a `.env` that still uses an old name, `DIALOG_API_KEY` or `DIALOGUE_API_KEY` | Copy `.env.example` to `.env` at the repo root and paste the key after `POLY_API_KEY=` | | HTTP 401 when connecting | A missing or wrong key, or a personal access token | Create a key in the **API Keys** tab at [studio.poly.ai](https://studio.poly.ai) and set it as `POLY_API_KEY` in `.env` | | No transcript ever arrives | `create_response` is `false`, so no turn runs | Set it to `true`, or send `response.create` after each `input_audio_buffer.committed` | | `unsupported_parameter` after `session.update` | A field the service refuses, such as `idle_timeout_ms` or `interrupt_response: false` in `turn_detection`, or `reasoning` | Remove it. A rejected update applies none of its fields | | `must be mono, 16-bit, 24000 Hz` | The WAV isn't in the session's format | Convert it with `afconvert -f WAVE -d LEI16@24000 -c 1 in.wav out.wav` | | Sam's answer prints above your question | Replies printed in arrival order | Hold the reply until `input_audio_transcription.completed`, as `support_agent.py` does | | `npm install` fails building `naudiodon2` | No C++ toolchain | On a Mac, run `xcode-select --install`, then `npm install` again | ## What this isn't The transcriber shows that a stock OpenAI Realtime client works unchanged against Dialog-RSN-1. It isn't a full speech-to-text service. It has no speaker labels, no punctuation settings, and no partial results while the caller speaks. ## Where to go next * The [OpenAI Realtime migration guide](/dialog-rsn-1/guides/openai-realtime-migration) lists every field Dialog-RSN-1 accepts and rejects. * The [streaming text-to-speech guide](/dialog-rsn-1/guides/streaming-text-to-speech) gives the support agent a voice. * The [API reference](/dialog-rsn-1/reference) lists every client and server event.