How we predict tok/s: a calibrated roofline

There are two easy ways to predict LLM inference speed, and both are wrong in instructive ways. Parameter heuristics ("a 70B model does X tok/s per GPU") ignore that the same model runs 4× differently across quantizations and frameworks. Benchmark lookup tables are accurate for the exact configuration someone measured — and silent about yours: a different context length, batch size, quantization, GPU count or engine version. This note describes the third approach this site uses: a roofline (Speed-of-Light) model, calibrated against published measurements and locked by regression tests.

The roofline

Transformer inference has two phases with opposite bottlenecks. Prefill (processing the prompt) is compute-bound: time ≈ FLOPs / achievable TFLOPS. Decode (generating tokens) is memory-bandwidth-bound: every new token re-reads the weights and the KV cache, so time ≈ bytes / achievable bandwidth. The roofline takes each phase at the maximum of its compute time and its memory time:

t_stage = max( FLOPs / (TFLOPS_dtype × efficiency),
               bytes_moved / (bandwidth × efficiency) )

decode bytes/token ≈ active_weights + KV_read(context, layer mix)
prefill FLOPs/token ≈ 2 × active_params

The two efficiency factors carry all the realism. Peak spec-sheet numbers are never achieved: we use an empirical memory-bandwidth factor of ~0.8 for HBM-class GPUs (a convention shared with NVIDIA's aiconfigurator SDK, from which our stage formulas are ported) and measured 0.65–0.85 on LPDDR5 edge devices — our Jetson calibration corpus of ~69 published data points puts dense ≥4 GB models at ~0.7 under llama.cpp and ~0.8 under tuned vLLM.

What the naive roofline misses

A textbook roofline still gets real deployments wrong. The corrections that matter:

• Framework awareness. vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama differ in static overhead, activation workspace coefficients (SGLang runs ~1.3× the TRT-LLM activation table), sliding-window billing and recurrent-state dtype (SGLang/TRT default the state to float32 — twice vLLM's footprint for the same model).
• Quantization that follows the hardware. NVFP4 is Blackwell-native only; FP8 is emulated (slower than INT8) on Ampere; weight-only 4-bit decodes fast but prefills at fp16 compute rates. A dtype dropdown is not enough — the GPU decides what the dtype is worth.
• MoE with active parameters. Speed follows the ~activated share (e.g. 5.1B of gpt-oss-120b's 117B), memory follows the full expert weights, and expert-parallel sharding changes both the per-GPU bytes and the communication bill.
• Attention that isn't uniform. MLA models cache a compact latent (~70× smaller than a naive GQA reading); 2026 hybrids cap most layers at a sliding window or a fixed recurrent state — decode re-reads far fewer bytes than layers × context suggests. See our companion note on hybrid attention memory.
• Parallelism is sublinear. Tensor-parallel decode hits collective-communication costs (we model a TP-tiered efficiency table, and treat NVLink vs PCIe differently); KV shards at most kv_heads ways; past that, ranks re-read replicas.
• Batching has a knee. Throughput grows with batch until the compute roof; per-user latency (TPOT) degrades past it. Our capacity planner solves for the largest batch that still meets a latency SLA rather than quoting best-case aggregate numbers.

Calibration, and staying honest

A model with this many terms can fit anything — unless it is pinned. Ours is pinned three ways:

• 16 datacenter anchors: published benchmark points (8B/70B dense at several batch sizes and TPs, DeepSeek-class MoE at EP8, single-stream and saturated) that the model must reproduce — enforced by a regression test, so a formula change that drifts the anchors fails CI.
• Control points for the hard cases: DeepSeek-R1 EP8 on 8×H200 (~103 tok/s single-stream against a ~100 target), and hybrid worked examples (gpt-oss-120b ≈ 4.8 GB KV @131K, Gemma 3 27B ≈ 10.4 GB) asserted within ±10% windows.
• An edge corpus: ~69 published Jetson measurements across power modes. They confirmed the bandwidth-bound decode model at 15W/25W within a few percent — and falsified it at 7W, where we apply a measured ×0.6 derate instead of pretending the roofline holds.

Honesty cuts both ways, so the same page that shows a prediction shows its limits: throughput carries a ±15–30% band (engine versions alone have moved a fixed model+GPU pair by up to 7× on young stacks — Jetson Thor's first-year vLLM containers being the extreme case); p99 latency under bursty load needs a queueing model we don't ship yet; and hybrid-attention throughput is labeled reduced-calibration while public benchmarks catch up. When the model can't know, the UI says so instead of printing a confident number.

Why bother, when you can benchmark?

You should benchmark — before you buy the fleet, not instead of sizing it. The calculator's job is the step before: shrinking a search space of 48 GPUs × quantizations × TP/PP/EP layouts × batch sizes to the two or three configurations worth measuring, with live rental prices attached. A calibrated roofline is exactly the right tool for that: directionally reliable everywhere, precise where it has anchors, and explicit about which is which.

Try the calculator — every throughput number links back to this methodology.

Sources

• Stage formulas ported from NVIDIA's aiconfigurator SDK (SOL backend, activation coefficient tables).
• Roofline model: Williams, Waterman & Patterson, “Roofline: An Insightful Visual Performance Model” (CACM 2009).
• Edge calibration corpus: jetson-ai-lab benchmarks, smolhub power-mode series, NVIDIA Jetson Thor vLLM container posts, llama.cpp discussions (full list in our Jetson note — forthcoming).
• Anchor values and tolerance windows are asserted in this site's public behavior — recompute any of them in the calculator.