How to Build a Low-Latency Voice AI App (2026): Start With WebRTC, Add a Chained Audio Stack Only When You Need Control

Quick Answer

If you are building a browser-based voice AI app in 2026, the default starting point is simpler than many teams assume:

1. Start with OpenAI Voice agents over WebRTC for the live conversation loop.
2. Keep your standard API key on the server and create the session from there.
3. Add a separate transcription stage only when you need durable text records, speaker labels, or downstream automation.
4. Add a separate TTS layer like ElevenLabs only when voice quality, cloning, or output format control changes the product outcome.

That is the practical lesson from OpenAI's 2026-05-04 engineering post on low-latency voice AI at scale, combined with the current Voice agents and Realtime WebRTC docs. The post is about OpenAI's internal relay and transceiver design, but the reader-facing takeaway is more basic: for a 1:1 live voice product, start with the direct realtime path. Do not begin with a three-stage speech-to-text -> text model -> text-to-speech stack unless you already know why you need that control.

Who This Tutorial Is For

This guide fits teams building:

• browser-based support or sales voice assistants
• AI tutors, interview coaches, or practice partners
• voice-enabled internal tools where interruptions and turn-taking matter
• mobile or web apps that need fast spoken replies

It is a weaker fit for batch transcription, meeting archives, or async media workflows. Those are usually text-first systems with audio attached, not live voice products.

Choose Between Live Audio and a Chained Pipeline

OpenAI's current voice-agent docs split the problem into two valid designs.

Architecture	Best for	What you gain	What you give up
Speech-to-speech live session	live browser or app conversations	lower interaction friction, native interruptions, fewer moving parts	less explicit control over every intermediate text step
Chained audio pipeline	support workflows, approvals, transcript-heavy systems, existing text agents	durable transcripts, deterministic routing, easier policy checkpoints	more latency, more orchestration, more failure points

The mistake is treating the chained stack as the "serious" architecture by default. In many products, it is just the slower one.

A good routing rule is:

• choose Realtime + WebRTC when the user is talking to the model directly
• choose transcription + text workflow + TTS when your application needs to inspect, store, approve, transform, or branch on the text between turns

Short version: if the product promise is "feels natural," start with live audio. If the product promise is "follows process," start with text control.

OpenAI's 2026-05-04 WebRTC Update

The news trigger is real, but this article is not a news rewrite.

On 2026-05-04, OpenAI published a detailed post explaining how it rebuilt its WebRTC stack for low-latency voice at scale. The post describes three production concerns that matter beyond OpenAI itself:

• connection setup speed
• low and stable media round-trip time
• keeping first-hop routing close to the user

Those infra details confirm a product-level point many teams still get backwards: low-latency voice is mostly an architecture decision before it becomes a prompt problem.

Measure Latency Before You Add Stages

Do one measurement pass before adding transcription, routing, or a second TTS provider. Otherwise you will not know whether the extra stage improved the product or just made the diagram look safer.

Use the same test script for at least 20 calls on the networks your users actually use:

Signal	Where to measure	Why it matters
session setup time	before session.connect() or pc.createOffer(), then after the connection is ready	catches slow session creation and first-hop routing problems
time to first assistant audio	user turn committed -> first remote audio frame or playing event	closest product proxy for "the app feels responsive"
media RTT, jitter, packet loss	RTCPeerConnection.getStats() on the WebRTC path	separates model delay from network quality
barge-in repair	user interruption -> assistant output cancelled or repaired	proves turn-taking works under real speech
fallback rate	count sessions that switch to text or async follow-up	tells you whether live audio is reliable enough for the workflow

For the Voice agents path, start by logging setup time, first assistant audio, interruption repair, and fallback rate. If those numbers are not enough to debug a problem, drop to the raw WebRTC path for a test build and sample WebRTC stats directly:

const startedAt = performance.now();
await pc.setRemoteDescription(answer);
console.log("webrtc_setup_ms", performance.now() - startedAt);

setInterval(async () => {
  const stats = await pc.getStats();

  for (const report of stats.values()) {
    if (report.type === "candidate-pair" && report.state === "succeeded") {
      console.log("current_rtt_s", report.currentRoundTripTime);
    }

    if (report.type === "inbound-rtp" && report.kind === "audio") {
      console.log("audio_jitter_s", report.jitter);
      console.log("packets_lost", report.packetsLost);
    }
  }
}, 5000);

Write down your own pass/fail thresholds before changing the architecture. A support bot on managed office Wi-Fi and a mobile tutor on public transit should not use the same latency budget.

The Fastest Safe Starting Point

Step 1: Start with one narrow live use case

Do not begin with a general-purpose "voice agent platform."

Pick one conversation shape with a clear success condition, such as:

• answer product questions from a fixed knowledge base
• collect intake details before handing off to a human
• run mock interview questions with spoken feedback
• guide a user through one workflow inside your app

If you cannot define what a good spoken answer looks like in one paragraph, your problem is not the audio stack yet.

Step 2: Start with Voice agents, then drop to raw WebRTC when you need control

OpenAI's current docs split the browser recommendation into two layers:

• for browser-based speech-to-speech apps, Voice agents are the fastest supported starting point
• when you need lower-level transport or session control, use the raw Realtime + WebRTC path

Keep the two paths separate. "Start with WebRTC" is an architecture choice. It does not always mean "start by wiring RTCPeerConnection yourself."

The highest-level browser path is its own SDK path. For a minimal npm app, install the Agents SDK and expose a tiny backend endpoint that mints the client secret:

npm install @openai/agents express

// server
import express from "express";

const app = express();

app.post("/realtime-client-secret", async (_req, res) => {
  const response = await fetch("https://api.openai.com/v1/realtime/client_secrets", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      session: {
        type: "realtime",
        model: "gpt-realtime-1.5",
        audio: { output: { voice: "marin" } },
      },
    }),
  });

  if (!response.ok) {
    res.status(response.status).send(await response.text());
    return;
  }

  res.json(await response.json());
});

app.listen(3000);

Then the browser connects through RealtimeSession:

import { RealtimeAgent, RealtimeSession } from "@openai/agents/realtime";

const agent = new RealtimeAgent({
  name: "Assistant",
  instructions: "You are a helpful voice assistant.",
});

const session = new RealtimeSession(agent, {
  model: "gpt-realtime-1.5",
});

const tokenResponse = await fetch("/realtime-client-secret", {
  method: "POST",
});
const { value: ephemeralKey } = await tokenResponse.json();

await session.connect({
  apiKey: ephemeralKey,
});

Expected result: the browser asks for microphone permission, session.connect() resolves, the remote audio output can speak back, and the standard OpenAI API key never appears in browser code.

If you need the lower-level route, OpenAI's current Realtime WebRTC docs recommend WebRTC rather than WebSockets for client-side realtime connections. The docs also show two server-assisted patterns:

• create the session on your server by POSTing SDP to https://api.openai.com/v1/realtime/calls
• or mint an ephemeral client secret from https://api.openai.com/v1/realtime/client_secrets

Either way, the standard API key stays on your backend. Do not mix the /session route below with RealtimeSession; the next example is the raw WebRTC unified-interface path for cases where you want to own RTCPeerConnection directly.

A minimal browser flow looks like this:

// browser client
const pc = new RTCPeerConnection();
const remoteAudio = document.createElement("audio");
remoteAudio.autoplay = true;

pc.ontrack = (event) => {
  remoteAudio.srcObject = event.streams[0];
};

const mic = await navigator.mediaDevices.getUserMedia({ audio: true });
pc.addTrack(mic.getTracks()[0]);

const events = pc.createDataChannel("oai-events");
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

const sdpResponse = await fetch("/session", {
  method: "POST",
  body: offer.sdp,
  headers: { "Content-Type": "application/sdp" },
});

await pc.setRemoteDescription({
  type: "answer",
  sdp: await sdpResponse.text(),
});

On the server, keep the setup equally narrow:

import express from "express";

const app = express();
app.use(express.text({ type: ["application/sdp", "text/plain"] }));

const sessionConfig = JSON.stringify({
  type: "realtime",
  model: "gpt-realtime",
  audio: { output: { voice: "marin" } },
});

app.post("/session", async (req, res) => {
  const form = new FormData();
  form.set("sdp", req.body);
  form.set("session", sessionConfig);

  const response = await fetch("https://api.openai.com/v1/realtime/calls", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
    },
    body: form,
  });

  res.send(await response.text());
});

This is enough to prove whether your product needs live voice at all.

Step 3: Add transcripts only when text is part of the business logic

A transcript stage is worth adding when you need one of these:

• searchable call logs
• speaker-aware records for review
• CRM or ticket updates after the call
• moderation or approval logic outside the live session
• analytics on what users asked and where the flow failed

If what you need is live captions or a transcription sidecar, OpenAI now also supports Realtime transcription sessions over WebRTC or WebSockets. Use that path for low-latency transcription-only flows. Keep /v1/audio/transcriptions for completed recordings, post-call processing, or cases where you explicitly need gpt-4o-transcribe-diarize, which is still not supported in the Realtime API.

OpenAI's current speech-to-text docs give you a few concrete constraints that are easy to design around:

• uploaded audio files are limited to 25 MB
• accepted formats include mp3, mp4, mpeg, mpga, m4a, wav, and webm
• gpt-4o-transcribe-diarize is available on /v1/audio/transcriptions but not yet supported in the Realtime API
• whisper-1 still supports verbose_json and word timestamps

If your app records calls for later processing, keep that path text-first and simple.

from openai import OpenAI

client = OpenAI()

with open("call.wav", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="gpt-4o-transcribe-diarize",
        file=audio_file,
        response_format="diarized_json",
        chunking_strategy="auto",
    )

for segment in transcript.segments:
    print(segment.speaker, segment.text, segment.start, segment.end)

That pattern is useful after the live session ends. It is usually not the first thing your user needs while speaking.

Step 4: Add a separate TTS layer only if voice quality changes the product

Many apps do not need a second voice provider. If OpenAI's built-in voice is already good enough, stop there.

Add ElevenLabs when you need one of these instead:

• a branded or cloned voice
• specific output formats for another system
• a voice library that your content or product team can control directly
• independent tuning of speech output without changing the reasoning model

ElevenLabs' current API keeps the integration small. For a low-latency output layer, use streaming TTS from your server instead of waiting for a complete file:

const voiceId = "JBFqnCBsd6RMkjVDRZzb";

const response = await fetch(
  `https://api.elevenlabs.io/v1/text-to-speech/${voiceId}/stream?output_format=mp3_44100_128`,
  {
    method: "POST",
    headers: {
      "xi-api-key": process.env.ELEVENLABS_API_KEY ?? "",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model_id: "eleven_flash_v2_5",
      text: "Your return has been approved. I just sent the label to your email.",
    }),
  }
);

if (!response.ok || !response.body) {
  throw new Error(await response.text());
}

const audioBody = response.body;
// Pipe audioBody to your web client, telephony layer, or audio playback path.

Two ElevenLabs details affect implementation:

• use the streaming endpoint and a Flash model when latency matters more than offline rendering; as of 2026-05-06, ElevenLabs lists eleven_flash_v2_5 as optimized for real-time use at about 75 ms model latency, excluding application and network latency
• optimize_streaming_latency still appears in the TTS API reference, but it is marked Deprecated, so do not build new guidance around it
• mp3_44100_192 requires at least the Creator tier, while pcm_44100 and wav_44100 require at least the Pro tier

Treat TTS as a later stage. Once you add a second audio provider, you inherit another pricing model, another failure surface, and another set of format constraints.

A Simple Decision Matrix

Use this table before you add more infrastructure.

Product shape	Start here	Add later if needed
browser voice tutor	OpenAI Realtime over WebRTC	transcript export after each session
support voice bot with QA review	chained pipeline with transcript storage	ElevenLabs if brand voice matters
internal voice form-filler	Realtime for the live exchange	diarized transcript for audit trail
AI meeting recap tool	transcription pipeline first	TTS only for optional playback
consumer companion app with a signature voice	Realtime for turn-taking	ElevenLabs for the branded output layer

The rule is simple: use the fewest stages that still satisfy the requirement.

The Most Useful Guardrails

Low-latency voice apps usually fail for operational reasons before they fail for model reasons.

Set up these guardrails early:

Keep your voice instructions short

Long system prompts cost time and make interruptions harder to reason about. For a live assistant, start with a compact instruction block like this:

You are a product support voice assistant.
Answer in short spoken sentences.
If the user asks for account actions, collect the minimum details first.
If you are uncertain, say what you need next instead of guessing.
Do not invent policies or order status.

Control session growth and transcription costs

OpenAI's current Realtime cost docs make three production constraints explicit:

• each new response reuses the accumulated conversation, so cost grows turn by turn
• input transcription is billed separately when you enable it
• prompt caching works best when the session history stays stable enough to reuse

In practice, set a token window for long sessions, summarize or delete stale turns, and avoid mutating instructions mid-session unless you need to. Low latency is not just transport speed. It is also keeping the session small enough that every turn stays cheap and predictable.

Decide where transcripts become durable

If you store nothing, debugging gets painful.

If you store everything by default, privacy review gets painful.

Define one rule up front:

• store only post-call transcripts
• or store only error cases
• or store user-approved transcripts only

Handle barge-in and repair explicitly

Users interrupt voice systems. They restart sentences. They change their mind halfway through a request.

Your app should support at least these repair moves:

• cancel the last answer
• let the user restate a question cleanly
• fall back to text input when audio quality is poor
• replay the final answer in text after the spoken turn

Add a fallback for bad networks

OpenAI's 2026-05-04 post is a reminder that network quality is part of product quality. You still need an app-level backup.

A practical fallback ladder is:

1. Realtime over WebRTC for the normal path
2. text transcript + typed reply when live audio degrades
3. asynchronous follow-up if the task cannot be completed live

What Not To Do

Avoid these three common mistakes:

Do not start with a full chained stack just because it feels safer

You will spend more time coordinating stages than learning whether users even want the voice interface.

Do not put the standard API key in the browser

OpenAI's current WebRTC setup still uses your backend for session creation or ephemeral secret minting. Keep it that way.

Do not pay for premium TTS until the product needs premium TTS

A better voice is useful only if voice quality changes conversion, retention, task completion, or brand perception. If it does not, the extra stage is just latency plus cost.

Recommended Rollout Order

If you want a practical implementation sequence, use this one:

1. ship one WebRTC-based live flow
2. measure whether users complete the spoken task
3. add transcript storage for review or automation
4. add a second TTS layer only if voice quality or format control becomes a blocker
5. add deeper analytics and routing after the core conversation works

That order keeps the system debuggable.

FAQ

Should I start with speech-to-speech or a chained audio pipeline?

Start with speech-to-speech over WebRTC if the product is a live browser conversation. Start with a chained pipeline if you need durable transcripts, approval steps, or deterministic logic between speech input and speech output.

Do I need Whisper if I already use a realtime voice model?

Not always. A realtime model can handle live audio directly. Add a transcription stage when you need searchable transcripts, speaker-aware records, or post-call processing outside the live session.

When is ElevenLabs worth adding to the stack?

Add ElevenLabs when the voice itself is part of the product requirement, such as brand narration, cloned voices, or specific output formats for downstream systems. Skip it if the built-in model voice is already good enough for the job.

Can I keep my API key in the browser for faster setup?

No. OpenAI's current WebRTC docs still route standard API keys through your server first, either to create a realtime call or to mint an ephemeral client secret.

Verification Note

Verified on 2026-05-06 against official sources:

• OpenAI engineering post on low-latency voice AI at scale
• OpenAI Realtime API guide
• OpenAI Voice agents guide
• OpenAI Realtime WebRTC guide
• OpenAI Realtime transcription guide
• OpenAI Speech-to-text guide
• OpenAI Realtime cost guide
• ElevenLabs streaming TTS API reference
• ElevenLabs model overview
• ElevenLabs TTS API reference
• ElevenLabs pricing

Checked items:

• OpenAI's current recommendation to start browser speech-to-speech with Voice agents, and to prefer WebRTC over WebSockets for client-side realtime voice
• server-side session creation endpoints for OpenAI Realtime
• realtime transcription-only support boundaries alongside /v1/audio/transcriptions
• speech-to-text upload limit, accepted file formats, and diarization availability
• Realtime session cost growth, separate transcription billing, and current truncation/caching guidance
• ElevenLabs streaming TTS endpoint, Flash model positioning, and deprecated status of optimize_streaming_latency
• ElevenLabs plan requirements for higher-quality output formats