WTF is Time To First Token?
How long a model takes to start responding, the latency metric that most shapes how fast an AI feels.
Also called: ttft, ttlt, time to last token, latency
TL;DR
Users judge speed by when text starts appearing, not when it finishes. TTFT measures that first moment; time to last token (TTLT) measures the whole response. Streaming makes a slow total feel fast as long as TTFT is low.
How it works
TTFT is dominated by queueing and processing the input (prefill), so long prompts raise it; prompt caching of a stable prefix can cut it sharply. TTLT adds per-token generation time multiplied by output length, so concise outputs and smaller models help. For agents, latency compounds across every loop iteration and tool call, which is why step count matters as much as model speed.
Origin
- Coined / introduced by
- LLM serving practice
- Story
- Standard serving metrics as streaming LLM APIs became common.