At 10:14, a caller asks to move a repair appointment. The agent checks the schedule. Four seconds pass.
“Hello?” the caller says.
The agent begins its confirmation just as the scheduling service returns an error. The caller hangs up before hearing the correction. No appointment moves. The caller has no reason to know that.
Median latency will describe this system as fast. It is the wrong number for the call that matters.
The middle of the distribution is the easy part
Teams often quote the time between a caller finishing a sentence and the agent starting to speak. That is useful for a demo. It describes the uncomplicated call, when the network is clear, the model responds quickly, and the tool does not need to wait.
Customers do not judge a voice agent by its uncomplicated calls. They judge it during the pause before an answer, the interruption that does not land, or the moment the agent sounds certain after a slow dependency has failed.
P95 shows what happens in the slowest five calls out of a hundred. That is where people stop trusting that the agent is listening or able to help.
Time the whole turn the caller experiences
The caller does not experience model latency. They experience the combined time spent waiting for speech recognition, the model, tool calls, and the response to arrive over the phone.
For the repair-appointment call, track the full turn from the caller finishing their request to the agent delivering an answer. Then keep the component timings for the engineering team: speech recognition, model, scheduling service, and text-to-speech.
The distinction matters. The model may be fast while the scheduling service holds the call open. A dashboard that celebrates the model's median response time can hide the silence the customer hears.
Pair long waits with what the caller did next
Latency becomes an operating problem when it changes behaviour. Put these events beside the slowest calls:
- the caller says “hello,” repeats the request, or interrupts
- the caller hangs up while the agent waits on a tool
- the agent makes a completion claim after a delayed or failed response
- a transfer starts after a long silence
- the same caller contacts the team again
This turns an engineering graph into a customer-effort signal. The point is not to remove every pause. It is to find the pauses that lead a caller to give up, repeat themselves, or accept a promise the system cannot keep.
Test the tail before it reaches the queue
Run the repair-appointment scenario under the call volume you expect at busy times. Include a slow scheduling service, a late error, and a caller who interrupts after waiting.
An engineer should be able to see whether the agent waits, retries, or hands off. An operations lead should be able to see what happens to the caller if the booking does not complete. The test passes only when both answers are acceptable.
For this scenario, the agent must not confirm a new appointment until the scheduling service returns one. If the service fails, the fallback needs a person or queue that will contact the caller by a stated time.
The management question
When your slowest five percent of calls run long, how many end in a hangup, a repeat contact, or a promise your team later has to repair?
That is the latency number worth taking to the operations meeting.