Why our tok/s are different
Why the estimates differ from vendor peak figures: batch, context, prefill and realistic bandwidth.
On this page · 3 sections
The tokens per second you see here are often lower than a vendor slide and different from another calculator. Usually neither number is wrong: they answer different questions. This page lists the common reasons and the three ways sizing tools produce speed figures.
Three approaches
| Approach | What it does | What it misses |
|---|---|---|
| Params-only calculators | VRAM from the parameter count. | Speed entirely; the KV cache; hybrid attention, where the memory error on 2026 models is 2–70×. |
| Benchmark lookup tables | Copied measurements for fixed model × GPU pairs. | Your configuration — another context, batch, precision or runtime — and new models before anyone benchmarks them. |
| Calibrated roofline (this site) | Compute- or bandwidth-bound time per stage × runtime coefficients, calibrated on published measurements and locked by regression tests. | Bands of ±15–30 % against a real deployment; tail latency under bursts; swings between runtime versions. |
Why the numbers differ from a peak figure
- Per user or per server. A headline "10,000 tok/s" is usually total output of a fully loaded server. The calculator shows both: tok/s per user, which falls as the batch grows, and total output, which rises with it.
- Batch and concurrency. Decode reads all active weights once per step. At batch 1 one user gets all of it; at batch 128 the read is shared, so throughput climbs while TPOT gets longer. A number without its batch is not comparable.
- Prompt processing. Many benchmarks report decode only. Here, at batch above 1 every output token carries its share of prefill time, so a RAG mix with 8K prompts produces far fewer output tokens per second than a chat mix on the same GPUs — as it does on real hardware.
- Context. Every decode step also reads the KV cache, which grows with context; long prompts slow decode, not only TTFT.
- Realistic bandwidth and compute. Memory bandwidth is scaled to about 80 % of the datasheet figure, TFLOPS are dense (no 2:4 sparsity) and runtimes realise a fitted share of both. Low-precision kernels do not reach their full byte saving at small batch.
- Runtime and parallelism. The same model differs by 10–30 % between runtimes, and tensor parallelism over PCIe costs more than over NVLink.
- MoE active parameters. Speed follows the parameters each token touches, memory follows all of them; tools that use the total for both understate MoE speed several times.
Comparing fairly
When you hold a published benchmark against an estimate, match the runtime, precision, batch or concurrency, prompt and answer length and GPU count first — the calculator takes all of them. If they still differ by more than 30 %, the point is worth reporting: that is how the calibration set grows. The current set and our error on each point are on the accuracy page; the method is in How we predict tok/s.