Methodology & data sources

The tools compute rather than look up: memory from an explicit byte budget, speed from a calibrated roofline, cost from live market prices. This page lists how each part is built and where its inputs come from, so you can judge a number before you rely on it.

Memory: an explicit byte budget

VRAM per GPU is the sum of named terms, each from the model geometry, the precision and the runtime:

VRAM = weights / (TP or EP)
     + KV cache (per user × users, sharded by attention heads; MLA latent replicated)
     + activations (one scheduler step, bounded by the prefill chunk)
     + runtime overhead + NCCL workspace (repeated on every GPU)
     + LoRA adapters + draft model + linear-attention state (when present)

Activations use per-architecture coefficients (dense, MoE, MoE with MLA) and the tensor-parallel degree, taken from NVIDIA's TensorRT-LLM memory model. The KV cache bills three memory classes per layer: growing KV for full attention (or the compact latent for MLA), a window-capped cache for sliding-window layers and a fixed recurrent state for linear-attention layers. The NCCL workspace is a per-GPU, per-TP-size figure from the catalog. The result is pinned against 11 public memory logs — llama.cpp and vLLM, single GPU to 16-GPU tensor parallel.

Speed: a calibrated roofline

Each stage runs at the larger of its compute-bound and bandwidth-bound time. Decode reads the active weights and the KV cache every step and is usually bandwidth-bound; prefill is compute-bound. Bandwidth is scaled by an empirical 0.8 and by per-runtime efficiency factors; small-batch decode, tensor-parallel communication by interconnect class (NVLink, AMD xGMI, PCIe), low-precision kernels and a fixed per-step scheduler cost each have their own correction. At batch above 1, total throughput counts prompt processing as well as generation, because a busy server prefills new prompts while it decodes. The formulas started from NVIDIA's aiconfigurator SDK.

The constants are calibrated against public measurements and locked by regression tests, so they cannot drift silently: 11 VRAM measurements · 29 throughput benchmarks · 167 low-batch points (single-user and low-concurrency, batch 1–16), plus 4 llama.cpp time-to-first-token points and the TensorRT-LLM v0.14 performance table (534 rows: 7 models × 6 GPUs × 9 prompt/answer lengths) for the prefill model. Jetson power modes are fitted separately on published Jetson measurements, which are not part of the regression-locked set. Every point, with its source and our error, is on the accuracy page — How accurate is this?; the last recalibration was 2026-10-24. The long form is the note How we predict tok/s: a calibrated roofline.

Data sources

• GPU specs — 48 GPUs · 10 Jetson modules, hand-curated from vendor datasheets: VRAM, memory bandwidth, dense TFLOPS per precision (no sparsity), NVLink and intra-node bandwidth, TDP, MIG and vGPU support, release year. A test suite checks cross-field invariants (NVLink against form factor, TFLOPS ratios, enums, uniqueness) on every change.
• Models — 141 catalog models with geometry from their published configs, including MoE active parameters, MLA ranks and hybrid layer mixes; any other model is read live from its HuggingFace config.
• Prices — on-demand, spot and community rates collected once a day directly from provider price lists and APIs (14 providers with data in the last week); the calculator uses the market median per GPU, with a catalog estimate as fallback. See Cloud prices.
• Fine-tuning — memory calibrated against EleutherAI's Transformer Math, the QLoRA paper and Unsloth's VRAM tables; activations follow the Megatron-LM formula.

Where to read next

Why our tok/s are different explains the gap to vendor peak figures, What the model actually knows lists the architecture facts it uses, What is not modeled lists what it leaves out, and Accuracy and error bands gives the numbers.