Here's a test you can run on any voice agent in about ten seconds. Ask it a simple question, then interrupt it halfway through the answer. Then say something it has to look up. If the thing feels alive through all three, someone spent real time on latency. If it feels like a walkie-talkie with a bad connection, they didn't.

Latency is the single metric that decides whether a voice agent feels like a conversation or a phone tree. And it's the one most teams get wrong, because they optimize the number they can see on a dashboard instead of the delay the user actually feels.

So let's talk about where the delay really comes from, which levers move it, and the tradeoff that trips up almost everyone: the fastest transcript is not the fastest agent.

What "latency" even means in a voice agent

When people say "my agent is slow," they almost always mean one thing: the gap between when they stop talking and when they hear the agent start talking. Call it response latency. That's the number the user judges you on.

But response latency isn't one thing. It's a chain, and each link adds delay:

1. End-of-turn detection — figuring out the user is actually done speaking, not just pausing.

2. Speech-to-text — turning the final chunk of audio into text the model can read.

3. LLM inference — time to first token from your language model (plus any tool calls).

4. Text-to-speech — generating audio from the model's response.

5. Network and transport — every hop between the user, your servers, and each provider.

Humans expect a reply in roughly 200–300ms in natural conversation. You will not hit that with a full STT→LLM→TTS pipeline, and you don't need to. Somewhere around one second end-to-end feels responsive; north of two seconds feels broken. The whole game is spending your budget wisely across those five links.

Here's a realistic budget for a well-built agent targeting ~1s of perceived latency:

Notice something. The two stages people obsess over — the LLM and TTS — are real, but they're not where most teams are bleeding. The bleeding usually happens in the first stage, and in the seams between stages.

The biggest lever is the one nobody optimizes: turn detection

Ask an engineer how to cut agent latency and they'll say "faster model" or "stream the audio." Almost nobody says "fix endpointing." And endpointing is where the largest, cheapest wins live.

Endpointing — deciding when a turn is over — is usually done with a silence timeout. The user stops making noise for, say, 700ms, and the agent assumes they're finished. Simple, and quietly terrible. Set the timeout short and the agent interrupts people mid-thought, especially anyone who pauses to think, spell out an email, or read back a number. Set it long and you've just added most of a second to every single turn, whether the user needed it or not.

That's the trap: a fixed silence timeout taxes your fast turns to protect your slow ones.

The fix is turn detection that reads the speech itself — intonation, pacing, rhythm, whether the sentence sounds finished — instead of just counting milliseconds of silence. "My number is five five five..." ends on a rising, unfinished cadence; a model that hears that waits. "My number is five five five one two one two" lands flat and complete; the model can cut in fast. AssemblyAI's Universal-3.5 Pro Realtime does end-of-turn detection on tonality and rhythm rather than raw silence, and targets around 300ms to make the call. Twilio's own guidance on the subject makes the same point from the other direction: smart endpointing lowers median latency, but you have to watch the tail, because a confident wrong guess costs you an interruption.

Get this layer right and you often shave 300–500ms off the average turn without touching your models. That's a bigger win than swapping your LLM.

Streaming is table stakes — but stream everything, not just STT

The second-biggest source of avoidable latency is treating each stage as a batch step: wait for all the audio, send it all to STT, wait for the full transcript, send it all to the LLM, wait for the full response, send it all to TTS. Every "wait for all" is dead time.

Streaming collapses that. And most teams stream the STT input but stop there.

Stream all three:

• Audio into STT as the user speaks, so partial transcripts are ready the instant the turn ends. A good streaming API emits partials within a few hundred milliseconds and commits a final transcript almost immediately after end-of-turn. For reference, Universal-3 Pro Streaming runs a P50 around 150ms post-VAD and a P50 time-to-complete-transcript around 250ms — so by the time turn detection fires, the text is essentially already there.
• Tokens out of the LLM, so you can start speaking the first sentence while the model is still generating the second. Don't wait for the full completion.
• Audio out of TTS, so playback begins on the first generated chunk instead of the last.

Done well, these overlap instead of stacking. The user's last word, the committed transcript, the first LLM token, and the first audio chunk happen almost on top of each other. That overlap — not any single fast component — is what makes an agent feel instant.

Collapse the pipeline: every provider hop is a tax

Here's a cost most architecture diagrams hide. If your STT, LLM, and TTS are three separate vendors, every turn crosses three network boundaries, three auth layers, three regions that may not be co-located. You're paying round-trip latency three times, plus the jitter of whichever provider is having a slow morning.

Two ways to cut it:

Co-locate and pool connections. Keep your services in the same region, reuse warm connections instead of opening a new one per turn, and pre-warm models so the first request of a session doesn't eat a cold start. A common trick for tool-call latency is a "warm-up" or filler utterance — the agent says "let me check that" while the lookup runs — which doesn't reduce real latency but hides it, which is often good enough.

Or reduce the number of hops outright. This is the architectural version of the fix: fewer boundaries, less to go wrong. AssemblyAI's Voice Agent API, for example, runs the full STT→LLM→TTS path over a single WebSocket and lands around one second end-to-end, partly because there's one connection to manage instead of three. Whether you consolidate vendors or just tighten the ones you have, the principle holds — latency compounds at the seams, so remove seams.

Model choice, per stage — not globally

"Use a faster model" is real advice, but applied bluntly it costs you accuracy you didn't need to spend. The move is to find the stage that dominates your budget and optimize that one.

If your LLM time-to-first-token is 600ms and everything else is tight, a smaller or faster-serving model at that stage buys you more than anything else will. If TTS is your bottleneck, a lower-latency voice matters more than a marginally nicer one. Measure first, then cut the tall pole. Swapping every model for the fastest option available is how you end up with an agent that's quick and dumb.

Which brings us to the tradeoff nobody wants to talk about.

The tradeoff nobody talks about: fast-but-wrong is slower than accurate

Most latency guides treat speed and accuracy as separate knobs. They aren't. In a voice agent, a wrong transcript doesn't just hurt quality — it adds latency, and usually more than you saved by being fast.

Walk through it. Your STT mishears "my account is A-B-1-2" as "my account is maybe won too." The LLM now responds to the wrong input, confidently. The user has to catch the error, correct it, and wait for the agent to recover. You just turned one turn into three. Whatever milliseconds you shaved by running a lower-accuracy model got obliterated by two extra round-trips of the entire pipeline.

This is why input accuracy is a latency feature, not just a quality feature. The numbers back it up: on Pipecat's open STT benchmark of real agent conversations, Universal-3.5 Pro Realtime posts a 6.99% word error rate, versus 9–16% for common alternatives — and the entity error rate gap is wider still, which matters because names, account numbers, and addresses are exactly the tokens that trigger a correction loop when they're wrong. Feeding the model the agent's own question as context (so a one-word "yes" or a spelled-out email resolves correctly) cut WER by another 10.2% across 20,000 agent audio files in AssemblyAI's testing.

So the real optimization target isn't transcript speed. It's turns to resolution. An agent that's 150ms slower per turn but almost never forces a do-over will feel dramatically faster to a real user than one that's quick and keeps mishearing them.

How to measure it (or you're just guessing)

You can't fix what you don't instrument. And most teams measure the wrong statistic.

Instrument every stage, not just the total. Log a timestamp at end-of-turn, at final transcript, at first LLM token, at first audio out. You want per-stage numbers so you know which pole to cut. A single end-to-end number tells you that you're slow, not why.

Watch the tail, not the median. Your P50 can look great while your P95 quietly ruins the experience, because users remember the turn that hung for three seconds, not the fifty that were snappy. Report P50, P90, and P95 for each stage. Tail latency is where "smart" endpointing and cold starts hide.

Measure ASR latency honestly. Streaming transcript latency is easy to fool yourself about. Measure it as the gap between how much audio the model has actually heard and the transcript it's committed to — not wall-clock time from when you hit send. Cresta has written well on this; the short version is that "how far behind the audio is the text" is the number that matters for turn-taking.

The shortlist: ways to reduce voice agent latency

If you want the compressed version to work through in order:

1. Fix end-of-turn detection first — move off fixed silence timeouts to model-based turn detection; it's the biggest average win and costs no model changes.
2. Stream all three stages — audio into STT, tokens out of the LLM, audio out of TTS — so they overlap instead of stacking.
3. Reduce or co-locate provider hops — fewer network boundaries, warm connections, same region; a single-connection pipeline removes the seams entirely.
4. Optimize the dominant stage — measure, find the tall pole (usually LLM time-to-first-token), and cut that one.
5. Don't trade accuracy for speed at the STT layer — a mishear costs you a full correction loop; protect input accuracy, especially on names and numbers.
6. Use filler/thinking phrases for slow tool calls — hide unavoidable latency behind a natural "let me check that."
7. Instrument per stage and track the tail — P50/P90/P95 at every hop, and measure ASR latency against audio consumed, not wall clock.

None of these require a moonshot. Most teams can find 500ms in an afternoon just by fixing endpointing and streaming their TTS.

The takeaway

Chasing raw speed is the wrong frame. Users don't experience your STT latency or your TTS latency — they experience how long until the agent responds correctly, and how often they have to repeat themselves. Optimize turns-to-resolution, get the input right so the pipeline only has to run once, fix the turn-taking so you're not taxing every fast turn to cover the slow ones, and measure honestly enough to know which link in the chain is actually costing you.

Do that and the ten-second test takes care of itself.

Frequently asked questions

What is a good latency for a voice agent?

Aim for roughly one second of end-to-end response latency — the gap between the user finishing and the agent starting to speak. Under 1s feels responsive; over 2s feels broken. Natural human conversation runs ~200–300ms, which a full STT→LLM→TTS pipeline won't match, so the goal is "responsive," not "instant."

Why is my voice agent slow?

Usually one of three things: a fixed silence timeout adding delay to every turn, stages running in batch instead of streaming, or latency compounding across multiple provider hops. Instrument each stage before guessing — the bottleneck is often end-of-turn detection, not the model.

Does faster speech-to-text mean worse accuracy?

It can, and that's a bad trade for voice agents. A mis-transcribed name or number forces a correction loop that costs a full extra turn or two — far more time than a lower-accuracy model saves. Optimize for turns-to-resolution, not transcript speed.

What is end-of-turn detection, and why does it matter for latency?

It's how the agent decides the user is done speaking. Silence-timeout endpointing either interrupts people or adds fixed delay to every turn. Model-based turn detection that reads intonation and rhythm can cut 300–500ms off the average turn without changing any models.

How do you measure voice agent latency?

Log timestamps at each stage (end-of-turn, final transcript, first LLM token, first audio out), and report P50, P90, and P95 — not just the median. For streaming STT specifically, measure how far the committed transcript lags the audio the model has actually consumed, not wall-clock time.