What the model actually knows
The architecture facts the model uses per layer, and the ones it has to approximate.
On this page · 2 sections
A sizing estimate is only as good as what it knows about the model, the GPU and the runtime. These are the facts the calculators use explicitly — each one a place where a simpler tool is off by a large factor — and the few they have to approximate.
What it uses
- Roofline per stage. Prefill is compute-bound, decode is bandwidth-bound; bandwidth is scaled to about 0.8 of the datasheet figure, and each runtime realises a fitted share of it.
- The runtime. vLLM, SGLang, TensorRT-LLM, Dynamo, llama.cpp and Ollama each get their own overhead, activation coefficients, prefill chunk, per-step cost, sliding-window billing and recurrent-state dtype (Frameworks).
- MoE. Active parameters drive speed, all expert weights drive memory; expert-parallel sharding, attention data parallelism in prefill and the extra routing workspace of DeepSeek-style models are modeled. For HuggingFace models, active parameters come from the expert shapes in the config, not from a share of the total.
- The interconnect. NVLink / NVSwitch, AMD xGMI (NVLink-class) and PCIe peer-to-peer each get their own parallelism limits and communication factors — PCIe tensor parallelism up to 4 GPUs at reduced efficiency; consumer GeForce cards stay single-GPU.
- MLA (DeepSeek, Kimi). The compact latent cache is billed — 576 values per token per layer for DeepSeek V3 instead of 2 × 128 heads × 128 — about 57× less than the naive multi-head reading. It is replicated, not sharded, across tensor-parallel GPUs.
- GQA and MQA. KV size follows the KV heads, not the attention heads.
- Hybrid attention. Three memory classes per layer: growing KV, a sliding-window cache capped at the window, and the fixed recurrent state of linear-attention layers (Gated DeltaNet, KDA, Mamba). Gemma 3 27B at 128K needs about 11 GB of KV cache instead of the roughly 67 GB a dense reading gives. Details: Hybrid attention memory.
- Quantization as hardware sees it. NVFP4 and MXFP4 compute only on Blackwell, FP8 is emulated on Ampere and older, small-batch kernels do not realise their full byte saving, GGUF files are billed at their whole-file bits per weight (Precision).
- Jetson power modes. Each
nvpmodelmode scales compute and bandwidth separately, which exposes the traps — on an Orin NX 16GB the 25 W mode has the same memory bandwidth as 15 W but a lower compute scale, so prefill and TTFT are slower while bandwidth-bound decode is unchanged. - Calibration. 11 VRAM measurements · 29 throughput benchmarks · 167 low-batch points, plus 4 TTFT points, locked by regression tests (How accurate is this?). Jetson devices are fitted separately on published measurements across power modes.
What it approximates
- Activation memory — fitted coefficients per architecture family, not a trace of the actual kernels.
- Runtime overhead — a fixed part plus CUDA-graph buffers that grow with layers × hidden size; the real figure moves with engine version and flags.
- Per-layer KV compression of the newest DeepSeek releases, and throughput of hybrid layers, which have few public measurements.
- AMD multi-GPU — xGMI is modeled at NVLink-class efficiency without a measured point yet.
What is left out entirely is in What is not modeled.