Measuring
the Wait.

“The model is slow” is a hypothesis. “Our customers wait too long” is an observation. The work begins by connecting the two.

A document assistant might retrieve evidence, rerank results and assemble a prompt before it sends the final answer request. A model benchmark tells you little about those earlier steps. Your measurement needs to follow the customer’s request across the entire path.

Three clocks, three different questions.

1. API time to first token

Start when the API client submits the model request; stop when it receives the first non-empty output token. Record your exact instrumentation boundary. Queueing, prefill and network transit can all contribute.

2. Product time to first visible token

Start with the user action; stop when the first model-generated content is rendered. This includes application work, transit and client buffering that the API measurement may exclude.

3. Time to first useful output

Start with the same user action; stop when the user receives something that helps with the task. Define that milestone for the product: a grounded answer, an actionable recommendation, or a usable result.

A streamed “Let me look into that” can improve acknowledgement without delivering useful output. Internal reasoning or a tool-call token may also arrive before anything customer-facing. Keep these events separate in the trace.

For the underlying inference definitions, see NVIDIA’s latency metrics documentation ↗. It distinguishes TTFT, inter-token latency and total request latency.

Trace the journey before changing the engine.

Use a request identifier to connect browser, application, retrieval and model events. Record durations with monotonic clocks within each process; account for clock alignment when correlating events across machines.

1. Mark the user action and first content rendered in the browser.
2. Measure application preparation, retrieval and tools before the answer request.
3. Measure model request start, first content received and response completion.
4. Record model, input size, cache status where available, deployment region and concurrency.
5. Identify the critical path. For parallel work, use the branch that gates progress rather than adding all branch durations.

Managed APIs may not expose a separate queue or prefill duration. Record the observable interval as a combined stage instead of presenting estimates as measurements. The explorer separates them to explain the mechanisms.

A budget is useful only when its boundaries match the experience you intend to improve.

A typical wait can hide a painful one.

Inspect the median alongside p95 and p99: the thresholds below which 50%, 95% and 99% of observed requests fall. Segment by task, context size, cache state and load. Keep errors, timeouts and cancellations visible so a fast subset does not conceal failed experiences.

Do not add stage p95 values and call the result an end-to-end p95. Those slow stages may occur on different requests. Calculate percentiles from complete request traces, with a stated sample size and test window.

Also track the pace of tokens after the first and the time to complete the task. A fast start followed by a stalled stream is a different problem from a long initial wait.

Make the trade-off explicit.

Suppose an illustrative request takes 1,900 ms, of which prefill accounts for 700 ms. Halving prefill saves 350 ms, bringing the total to 1,550 ms: an 18.4% reduction. Even eliminating that stage entirely would save at most 36.8%, if every other stage stayed unchanged.

This is why the latency budget matters. It gives a proposed optimisation a ceiling before you spend engineering effort on it. In production, changes can also shift queueing, quality and cost; remeasure the whole request after each experiment.

Prefill processes the input, while subsequent decoding generates the remaining output. Their bottlenecks differ; Anyscale’s serving guide ↗ explains the distinction.

The decision to make

Choose a task-specific response target, a minimum acceptable quality level and a cost constraint. Then prioritise experiments by their likely impact on the end-to-end wait. The goal is a useful outcome delivered within a wait the customer will accept.