AI STRATEGY / TOPIC 01

Latency
Dynamics.

From the first token to the first useful outcome.

THE CENTRAL IDEA

Responsiveness is a property of the whole system. To make an AI product faster, first understand where the customer waits.

Explore the latency budget
GROWTHVECTOR FRAMEWORKInteractive exploration + reading seriesUpdated 3 November 2026

A faster model.
A faster product?

Before a model produces a token, your product may authenticate a user, retrieve documents, call tools and wait for capacity. After that token arrives, it still has to reach the screen.

Optimise the largest source of avoidable delay on the critical path. Then measure again. The explorer below makes that principle tangible.

01 / THE INTERACTIVE FRAMEWORK

Where does the time go?

ILLUSTRATIVE MODEL

Adjust the six stages of a single streamed response. Compare a change against your baseline, then replay the journey from click to first visible token.

What this model assumes—and leaves out

This is an editable, deterministic budget for one request, not a benchmark or a prediction. The preset timings are invented for explanation.

  • All six stages run sequentially. The total is their sum; real parallel branches require a critical-path calculation.
  • Retrieval & tools covers work before the final answer request, including earlier model calls if applicable. Prefill & first token covers prompt processing and any internal reasoning before that answer starts.
  • Network is the aggregate transit allowance across this simplified path. Rendering includes buffering and displaying the first content in the client.
  • Changing a slider holds all other stages constant. Real changes can affect quality, costs, cache behaviour and other timings.
  • No load distribution, cache hit probability, retries, streaming throughput or completion time is simulated. The model cannot estimate p95 or p99 latency.
  • A first visible token is not necessarily a useful answer. Never substitute a progress indicator for this measurement.

02 / GO DEEPER

Follow the thinking.

Start with measurement.
Then move through the system.

01
ARTICLE / FOUNDATIONS

Measuring the Wait

Define the boundaries, trace the request and choose a latency target that means something.

PLANNED EXPLORATIONS

02 / EXPERIENCE

The First Token Is Only the Beginning

Streaming, progressive disclosure and time to useful output.

03 / APPLICATION

Before the Model Even Starts

Retrieval, orchestration and the cost of serial work.

04 / INFERENCE

Inside the First Token

Context, caching, queueing and the prefill phase.

05 / ECONOMICS

How Fast Is Fast Enough?

Balancing responsiveness, quality and cost per successful task.

THE DECISION TO TAKE AWAY

Measure the whole wait.
Improve the part that matters.

The right latency target depends on the task, the customer and the economics. Start with a trace, test a change, and check that the answer is still worth waiting for.

Explore GV’s AI Strategy advisory