A token-per-second number describes one slice of inference. It does not tell you how long a request waited in a queue, how much time the model spent processing the prompt or whether the result completed your task. To compare LLM endpoints, start with the user’s experience and work backward to the measurements.
Put the clocks in order
A request has several observable milestones: it leaves the client, the server starts work, the first generated content arrives and the response finishes. Unless the server exposes internal timestamps, a client-side test cannot cleanly separate all the network and queueing components. Say what your clock measures.
Time to first token usually means elapsed time from request start to the first generated token. In a chat UI, the first meaningful answer text may be a better milestone than the first event in the stream. An initial event can contain metadata or an empty delta. If a model has a separate reasoning phase, state whether your measurement includes it.
Decode speed describes how quickly output is generated after generation starts. End-to-end latency measures the whole request. Those metrics answer different questions, so report them together.
A small arithmetic trap
Suppose a hypothetical response contains 600 output tokens. The client waits 2 seconds for the first token and receives the rest over another 3 seconds. Dividing all output tokens by the complete 5-second request gives 120 tokens/s. Dividing by the 3-second generation interval gives roughly 200 tokens/s.
Both calculations describe the same response. Neither is a lie if it is labelled correctly. Trouble starts when one service reports the first definition and another reports the second, and a comparison treats them as interchangeable.
Tokenizers add another complication. The same paragraph can produce different token counts across model families. If your concern is how quickly a person receives a readable answer, also compare completion time for a fixed task rather than only token throughput.
Compare these four measurements
| Measurement | Question it answers | What to record alongside it |
|---|---|---|
| First visible answer | How long does the person wait before reading? | Prompt length, streaming behavior, network location |
| Complete response | When is the whole answer available? | Output length and finish reason |
| p50 and p95 latency | What is typical, and what happens in the slow tail? | Number of requests and load pattern |
| Cost per accepted task | What did useful work actually cost? | Retries, rejected outputs, tool steps |
The p95 is the point below which about 95% of observed latencies fall. With very few samples it is unstable: the “tail” may effectively be one request. Publish the sample count and avoid making a service-wide claim from a handful of calls.
Load changes the answer
One request on an otherwise quiet endpoint measures a different condition from many concurrent requests. Batching may increase aggregate throughput while individual requests wait longer. Queueing may dominate at busy times. A benchmark should name its concurrency, request rate, prompt-length distribution and output cap.
Use two runs if both experiences matter: a single-user interactive run and a realistic workload run. Keep their results separate. A headline that combines the best first-token latency from one run with the best throughput from another describes no actual user experience.
The vLLM benchmarking documentation is a useful starting point for the vocabulary of serving measurements. An application comparison should go one step further and include whether the responses were accepted as useful.
Save enough to reproduce it
Keep a row per request with the UTC timestamp, endpoint, exact model ID, prompt hash, input/output token counts, streaming flag, start time, first content time, completion time, status and finish reason. Save the prompt set separately and strip credentials from all exports.
If you report a speed range, say whether it spans models, repeated runs, percentiles or minimum and maximum observations. “250–300 tokens/s depending on the model” is not the same claim as “p50 250, p95 300.” A range without that distinction is easy to misread.
For more evaluation structure, read how to build a model evaluation set. For the memory tradeoffs underneath serving, start with quantization without the folklore.
Questions, answered.
What does tokens per second measure?
It can describe generation throughput, but the denominator and token count must be defined. Per-request decode speed differs from aggregate server throughput.
What is time to first token?
Time to first token is the elapsed time between starting a request and receiving its first generated token. For user experience, first visible answer content can be a more useful milestone.
Published 4 November 2026. Source-check date: 4 November 2026. How we report.
