Build a voice agent for the browser with JavaScript

What you’ll build

A browser page that captures microphone audio, streams it to Dialog-RSN-1 over a WebSocket, and renders the reply as text while it’s still arriving. You’ll use getUserMedia for the microphone, AudioWorklet to reach the raw samples, and a WebSocket for the wire protocol. There’s no SDK to install and no build step.

Text replies only. This guide shows the reply on the page, so it asks for text with output_modalities: ["text"] and stops there. Set the field explicitly: a session that leaves it out speaks its replies, and a page listening for text deltas then shows nothing. For a spoken reply, leave the default and play the server’s audio, or feed the accumulated text to a text-to-speech provider of your own.

Connecting from the browser

Your API key from studio.poly.ai is a bearer credential, and a browser tab is a poor place to keep one: a visitor can open dev tools and read it out of your JavaScript. Don’t connect from the browser to wss://api.studio.poly.ai/v1/realtime with the key embedded in client code. Put a small relay in front of it: a same-origin WebSocket endpoint on your own server that holds the real key, opens the upstream connection to Dialog-RSN-1, and pipes frames through in both directions. The browser talks to your relay and nothing else.

connect

Browser  →  wss://your-app.example.com/realtime-relay   (no key, same-origin)
Relay    →  wss://api.studio.poly.ai/v1/realtime        (holds the real key)

The companion repo includes a Node relay, relay.js, that also serves the page. If you write your own, set a User-Agent on the upstream connection. The API refuses a handshake without one with a bare 403, and Node’s ws package sends none by default. The code below assumes that relay exists and connects to it:

connect.js

const ws = new WebSocket("wss://your-app.example.com/realtime-relay");

Capturing microphone audio

The browser gives you microphone audio as floating-point samples; Dialog-RSN-1 wants base64-encoded 16-bit PCM. Do that conversion in an AudioWorkletProcessor, which runs off the main thread, so encoding audio doesn’t compete with rendering the page.

pcm-worklet.js

// Registered once, runs in the audio rendering thread
class PCMWorklet extends AudioWorkletProcessor {
  process(inputs) {
    const channel = inputs[0][0];
    if (!channel) return true;

    // Float32 [-1, 1] → Int16 PCM
    const pcm16 = new Int16Array(channel.length);
    for (let i = 0; i < channel.length; i++) {
      const s = Math.max(-1, Math.min(1, channel[i]));
      pcm16[i] = s < 0 ? s * 0x8000 : s * 0x7fff;
    }
    this.port.postMessage(pcm16.buffer, [pcm16.buffer]);
    return true;
  }
}
registerProcessor("pcm-worklet", PCMWorklet);

capture.js

function bufferToBase64(buffer) {
  let binary = "";
  const bytes = new Uint8Array(buffer);
  for (let i = 0; i < bytes.length; i++) binary += String.fromCharCode(bytes[i]);
  return btoa(binary);
}

async function startCapture(ws) {
  const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
  const audioContext = new AudioContext();
  await audioContext.audioWorklet.addModule("pcm-worklet.js");

  const source = audioContext.createMediaStreamSource(stream);
  const worklet = new AudioWorkletNode(audioContext, "pcm-worklet");

  worklet.port.onmessage = (event) => {
    if (ws.readyState !== WebSocket.OPEN) return;
    ws.send(JSON.stringify({
      type: "input_audio_buffer.append",
      audio: bufferToBase64(event.data),
    }));
  };

  source.connect(worklet);
  return { audioContext, stream };
}

The browser takes the AudioContext sample rate from the hardware, 48 kHz on most machines, where Dialog-RSN-1’s native rate is 16 kHz. Rather than resample in the client, declare your true rate with the poly_input_rate extension field on session.update and let the server resample on the way in:

session.update

{
  "type": "session.update",
  "session": {
    "output_modalities": ["text"],
    "audio": { "input": { "format": { "type": "audio/pcm" } } },
    "poly_input_rate": 48000
  }
}

Send the rate your AudioContext reports in audioContext.sampleRate. Hardcoding 48000 is wrong on hardware that runs at 44.1 kHz: the server then resamples from a rate your audio was never at, and hears it about 9% fast. You can read the rate before the user clicks anything. A context created on page load starts suspended, but its sampleRate is already set. Call audioContext.resume() first thing in the click handler, before you wait on the microphone: Safari only lets a context start during the click itself. The full example below does this.

Streaming audio and handling server VAD

Once you’ve wired up capture, input_audio_buffer.append frames flow without a break. The server decides when a turn starts and ends. Voice activity detection proposes a boundary and Dialog-RSN-1 judges it. speech_stopped always waits for Dialog-RSN-1’s judgment that the caller has finished. speech_started arrives as soon as the detector hears sound, unless a reply is playing, when Dialog-RSN-1 confirms the caller is talking first. A sound it then judges to be noise is dropped, so a speech_started can arrive with no speech_stopped after it. The server tells you with these events:

vad.js

ws.addEventListener("message", (event) => {
  const msg = JSON.parse(event.data);
  switch (msg.type) {
    case "input_audio_buffer.speech_started":
      console.log("caller started talking, item:", msg.item_id);
      break;
    case "input_audio_buffer.speech_stopped":
      console.log("caller stopped, turn committed");
      break;
  }
});

There’s no client-side VAD to configure or tune. If the caller starts talking while a response is still streaming, Dialog-RSN-1 confirms the audio came from the caller and then cancels the response for you. You get the same response.done with status: "cancelled" that response.cancel produces, with status_details.reason set to "turn_detected" rather than "client_cancelled".

Reconstructing the streamed reply

The reply arrives as a burst of small events instead of one block of text. Accumulate response.output_text.delta by the item_id it belongs to, and treat response.done as the signal the turn is over:

reply.js

const replies = new Map(); // item_id -> accumulated text

ws.addEventListener("message", (event) => {
  const msg = JSON.parse(event.data);

  if (msg.type === "response.output_text.delta") {
    const current = replies.get(msg.item_id) ?? "";
    replies.set(msg.item_id, current + msg.delta);
    renderPartial(msg.item_id, replies.get(msg.item_id));
  }

  if (msg.type === "response.output_text.done") {
    renderFinal(msg.item_id, replies.get(msg.item_id));
  }

  if (msg.type === "response.done") {
    console.log("turn finished:", msg.response.status);
  }

  if (msg.type === "error") {
    console.error(`${msg.error.code}: ${msg.error.message}`);
  }
});

function renderPartial(itemId, text) {
  document.getElementById("reply").textContent = text;
}
function renderFinal(itemId, text) {
  renderPartial(itemId, text);
}

Keying by item_id keeps each reply separate. Only one response runs at a time and each carries one text item, so deltas never interleave, but a new reply’s deltas start a new entry instead of appending to the last one. Keep the error branch. Every failure on this API arrives as an error event and leaves the connection open, as the quickstart explains. Without the branch, a rejected session.update looks like a page that connects and never answers.

Voice output is out of scope here. With output_modalities: ["text"] the server sends no audio events. For a spoken reply, either drop the output_modalities line and play response.output_audio.delta (base64 PCM16 at 24 kHz) through Web Audio, or hand the accumulated text from renderFinal to a text-to-speech provider of your own.

Full example

Three files: the page, its script, and the pcm-worklet.js from Capturing microphone audio saved next to them.

Full code on GitHub

These files plus the relay, ready to clone and run with npm start.

index.html

<!doctype html>
<html>
<body>
  <button id="start">Start talking</button>
  <p id="reply"></p>
  <script src="app.js"></script>
</body>
</html>

app.js

const replies = new Map();
const audioContext = new AudioContext(); // suspended until the click below, but sampleRate is known now
const ws = new WebSocket("wss://your-app.example.com/realtime-relay");

ws.addEventListener("open", () => {
  ws.send(JSON.stringify({
    type: "session.update",
    session: {
      instructions: "You are a helpful, concise assistant.",
      output_modalities: ["text"],
      audio: { input: { format: { type: "audio/pcm" } } },
      poly_input_rate: audioContext.sampleRate,
    },
  }));
});

ws.addEventListener("close", () => {
  // The server closes a connection that sends nothing for 5 minutes, so a page left
  // unclicked goes quiet. Say so rather than leave a button that does nothing.
  document.getElementById("reply").textContent = "Disconnected. Reload the page to start again.";
});

ws.addEventListener("message", (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type === "response.output_text.delta") {
    const current = replies.get(msg.item_id) ?? "";
    replies.set(msg.item_id, current + msg.delta);
    document.getElementById("reply").textContent = replies.get(msg.item_id);
  }
  if (msg.type === "error") {
    console.error(`${msg.error.code}: ${msg.error.message}`);
  }
});

const button = document.getElementById("start");
button.addEventListener("click", async () => {
  button.disabled = true; // a second click would add a second worklet and send every frame twice
  await audioContext.resume(); // first, while the click still counts as a user gesture
  const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
  await audioContext.audioWorklet.addModule("pcm-worklet.js");

  const source = audioContext.createMediaStreamSource(stream);
  const worklet = new AudioWorkletNode(audioContext, "pcm-worklet");

  worklet.port.onmessage = (e) => {
    if (ws.readyState !== WebSocket.OPEN) return;
    const bytes = new Uint8Array(e.data);
    let binary = "";
    for (let i = 0; i < bytes.length; i++) binary += String.fromCharCode(bytes[i]);
    ws.send(JSON.stringify({ type: "input_audio_buffer.append", audio: btoa(binary) }));
  };

  source.connect(worklet);
});

Next steps

Overview

Realtime streaming explained, plus how turn-taking, generation, and prompting fit together.

Dialog-RSN-1 reference

Every event, the session object, function calling, and error codes.

Python from scratch

The same protocol, built server-side with plain websockets.

Streaming to TTS

Where the accumulated text from renderFinal actually goes: barge-in, unspeakable text, and playback reports.

Overview

Quickstart

Prompt guide

Prompt cookbook

Evaluations

Reference

Guides

Build a voice agent for the browser with JavaScript

What you’ll build

Connecting from the browser

Capturing microphone audio

Streaming audio and handling server VAD

Reconstructing the streamed reply

Full example

Full code on GitHub

Next steps

Overview

Dialog-RSN-1 reference

Python from scratch

Streaming to TTS

​What you’ll build

​Connecting from the browser

​Capturing microphone audio

​Streaming audio and handling server VAD

​Reconstructing the streamed reply

​Full example

Full code on GitHub

​Next steps

Overview

Dialog-RSN-1 reference

Python from scratch

Streaming to TTS

What you’ll build

Connecting from the browser

Capturing microphone audio

Streaming audio and handling server VAD

Reconstructing the streamed reply

Full example

Next steps