Track time to first token (TTFT), time per output token (TPOT), and inter-token latency (ITL) together, because each exposes a different aspect of inference effectiveness. TTFT reflects queueing and prompt processing, TPOT summarizes streaming speed, and ITL reveals stutter inside the stream.
Key takeaways:
- TTFT points to queueing and prefill, while TPOT and ITL point to decode.
- TPOT is an average, and ITL is per token gap, so only ITL exposes jitter.
- Alert on percentiles and per-request targets, not averages.
How TTFT, TPOT, and ITL relate
Inference runs in two phases:
- Prefill processes the whole prompt and is compute-bound.
- Decode generates one token at a time and is memory-bound.
TTFT lands on the prefill side, since it includes queue wait and prompt processing. TPOT and ITL land on decode.
With N output tokens, the math ties them together:
TPOT = (E2E latency - TTFT) / (N - 1)
E2E latency ≈ network overhead + TTFT + (TPOT × N)
On long responses, TPOT dominates total time even when the first token is fast.
When a healthy TPOT still feels slow
ITL measures the gap between two consecutive tokens, so it catches jitter that average TPOT smooths over. Our inference benchmarking guide lists common causes: large prefills monopolizing compute, PagedAttention overhead, and poor hardware topology—such as cross-NUMA traffic.
Read LLM inference observability metrics as a diagnostic
- High TTFT, normal TPOT: requests are queueing or prompts are long. Check prefill capacity.
- Rising TPOT: decode is memory-bound. Check GPU memory bandwidth and load.
- Healthy TPOT, spiky ITL: something is interrupting decode, such as prefill interference.
Alert on percentiles and goodput
Averages hide the slow tail. Track P95 and P99 for each metric, then define goodput: the share of requests that meet every target at once, such as a TTFT ceiling, an ITL ceiling, and an end-to-end budget. Set thresholds per use case.
Check how your tool defines each metric
Definitions vary. NVIDIA's benchmarking docs, for example, treat ITL as the average gap between tokens, also called TPOT, and warn that tools differ on whether TTFT is included. Confirm what each tool calculates before comparing numbers. On the DigitalOcean Inference Engine, Serverless Inference metrics cover TTFT, end-to-end latency, error rates, and per-request cost with token counts. You can derive TPOT from those with the formula shared earlier.
LLM inference observability FAQ
What is disaggregated prefill and decode in LLM inference?
It runs prompt processing (prefill, which is compute-bound) and token generation (decode, which is memory-bound) on separate workers. The KV cache must move between them, and depending on the implementation, that transfer can delay TTFT or the second token’s ITL. On the DigitalOcean Inference Engine, Serverless Inference metrics report TTFT and end-to-end latency, so you can check whether an architecture change affects first-token time.
What is the difference between TPOT and ITL?
TPOT is the average time per token after the first token, calculated across the entire response. ITL, in the stricter sense, is the gap between two specific consecutive tokens, so it exposes stutter that a healthy TPOT hides. You can derive it from TTFT and end-to-end latency with TPOT = (E2E latency - TTFT) / (N - 1).
Why track P95 and P99 instead of averages?
Averages hide outliers. P95 and P99 show what your slowest requests experience, which is what you compare against latency targets and use to define goodput. Serverless Inference metrics on DigitalOcean cover TTFT, end-to-end latency, error rates, and per-request cost with token counts, so you can set those targets against both speed and spend.
References & further reading
- Worth bookmarking: DigitalOcean's guide to LLM inference benchmarking, which covers the formulas above plus concurrency sweeps.
- Learn more about Serverless Inference metrics and the DigitalOcean Inference Engine.
- Worth checking directly before you compare tools: NVIDIA's LLM benchmarking metric definitions