41 terms used across the calculator, TCO, Optimize, fine-tuning and edge tools — precision formats, memory, parallelism, latency and cost. Every term is underlined where it appears in an article.
8 terms have a long form with formulas and worked examples
Cost to generate a million output tokens with the serving unit busy 24/7: GPU $/hr ÷ output tokens per hour. Prompt processing is included — each output token carries its share of prefill time — so long-prompt workloads such as RAG cost more per output token. The most comparable cost metric across GPUs and configs.
$/1M tokens folds GPU count, hourly price and speed into one number, which makes configurations with different cards and GPU counts comparable. It is the cost of a million output tokens with the serving unit busy around the clock.
Prompt processing is included: every output token carries its share of the prefill time, so a RAG request with an 8K prompt and a 256-token answer costs about twice as much per output token as a 1K chat turn on the same unit — as it does on real hardware. At batch 1 the figure is the decode-only rate. It assumes full utilization; divide by your average load for a budget, and swap in your contracted rate for the market median. Read more: Reading a result.
What stops a replica from taking one more stream: VRAM (KV pool), decode latency (TPOT), prefill (chunked-prefill load pushes TPOT/TTFT over the SLA) or the batch cap.
Share of a turn's history already resident in the prefix cache (vLLM APC / SGLang RadixAttention). Agentic mode predicts it from free VRAM vs the idle sessions' working set, and reports the smallest hit rate at which the fleet needs no replicas beyond its decode-only sizing (or that no hit rate removes them).
An agent summarizing its own context when it nears the window (Claude Code, Cursor). Modeled as a sawtooth: occupancy drops to prefix + summary and the summary is re-prefilled.
Independent copies of the whole serving unit, each handling a share of users. The Capacity Planner scales replicas to meet the user count once one replica hits its latency or VRAM cap.
Share of each GPU's memory a configuration uses. Underused — under 50 % of memory: more GPU than the workload needs; a smaller card would do. Optimal — 50–70 % of memory: headroom for longer context, no paying for empty memory. Good — 70–80 % of memory: well used, with some room to grow. Tight — 80–95 % of memory: fits, little room for longer prompts or more users. Above 95 % the configuration is excluded.
<50% underused · 50–70% optimal · 70–80% good · 80–95% tight · >95% excludedused in Calculator · What fits
The llama.cpp / Ollama file format: Q4_K_M (≈4.9 bits per weight, the community default), Q5_K_M (≈5.7), Q8_0 (8.5). Every tensor is quantized, embeddings included. Pick llama.cpp or Ollama as the framework to see them.
Grouped / Multi-Query Attention — several query heads share one KV head, shrinking the KV cache (the dominant long-context cost) versus full multi-head attention.
Attention needs the key and value vectors of every earlier token. Engines store them once instead of recomputing them each step, so the cache grows by one entry per token per layer — for every request in flight. It is usually the second-largest term of the memory budget after the weights, and the largest at long context or high concurrency.
Formula · dense / GQA
bytes per token = 2 (K and V) × layers × KV heads × head dim × bytes per element
KV cache = bytes per token × context × concurrent users
The calculator reserves a full context window per concurrent user; the Capacity Planner sizes the pool for how full contexts really get, at a chosen preemption risk. Levers: FP8 or INT8 KV halves it, GQA and MLA shrink it at the architecture level, sliding-window layers cap it. Tensor parallelism shards it by KV heads. Read more: Context length and what drops out · Hybrid attention memory.
Low-Rank Adaptation. Freeze the base weights, train small low-rank adapter matrices. Gradients + optimizer state live only on the adapter, so VRAM drops dramatically.
Model FLOPs Utilization — fraction of a GPU's peak compute actually used during training (typically ~30-50%). Drives the wall-clock and cost-to-train estimates.
Multi-head Latent Attention (DeepSeek V3, Kimi K2). Stores compact latent vector per token instead of full K/V — drastically reduces KV memory at long context.
kv_lora_rank + qk_rope_head_dim per tokenused in Calculator
Multi-head Latent Attention, introduced with DeepSeek V2 and used by DeepSeek V3, R1 and Kimi K2, compresses keys and values into one low-rank latent vector per token plus a small rotary part. The cache stores the latent, not per-head K and V, so it is a fraction of the size a head count suggests.
Formula · MLA cache per token
bytes per token = layers × (kv_lora_rank + qk_rope_head_dim) × bytes per element
Worked example · DeepSeek V3, 128K context, FP16 cache
MLA: 61 × (512 + 64) × 2 B ≈ 69 KiB per token → ≈ 9.2 GB
naive: 61 × 2 × 128 heads × 128 × 2 B ≈ 3.8 MiB per token → ≈ 524 GB
Two consequences for sizing: tools that read the head count overstate the cache about 57×, and the latent is replicated on every tensor-parallel GPU rather than split by heads, so TP does not shrink it. Catalog and HuggingFace models carry the MLA fields; a manual entry cannot. Read more: What the model actually knows.
OCP microscaling FP4 (4.25 bits per weight) — the native format of gpt-oss: MoE experts in FP4, attention and embeddings in BF16. FP4 tensor-core speed on Blackwell.
NVIDIA 4-bit floating point with one FP8 scale per 16 weights. FP4 compute only on Blackwell (B200/B300/GB200/GB300/RTX 5090/RTX PRO 6000 Blackwell/DGX Spark); elsewhere it saves memory at BF16-equivalent compute. Better quality than INT4.
NVFP4 stores weights as 4-bit floats (E2M1) in blocks of 16 that share one FP8 (E4M3) scale, plus a per-tensor FP32 scale. The small block and the floating-point scale keep more precision than INT4 formats with coarser groups, which is why NVIDIA ships NVFP4 checkpoints of large models as a production format.
Size
16 × 4 bits + 8-bit scale = 72 bits per 16 weights = 4.5 bits = 0.5625 B per param
70B model ≈ 42 GB of weights with BF16 embeddings (BF16: 140 GB · FP8: ≈ 72 GB)
The speed benefit needs Blackwell FP4 tensor cores: B200, B300, GB200, GB300, RTX 5090, RTX PRO 6000 Blackwell and DGX Spark, where the tensor cores' peak FP4 rate is up to about 4× BF16 (the catalog's FP4 rate where available). The calculator credits about half of the gain beyond BF16 — ≈ 2.5× at a 4× peak ratio — the share real kernels reach on compute-bound work. On Hopper, Ada and Ampere the calculator falls back to the BF16-equivalent compute rate: the memory saving remains, the prefill speed-up does not, and AWQ or INT8 are the better choices there. Embeddings and the LM head stay in 16 bit. NVFP4 needs vLLM, SGLang, TensorRT-LLM or Dynamo. Read more: Weight and KV-cache precision.
Capacity Planner knob (0.1/1/5%): the KV pool is reserved at the (1−ε) quantile of total demand over the occupancy distribution (statistical multiplexing), so at most ε% of moments exceed it. Lower risk → more GPUs; the buffer shrinks as the fleet grows (√batch).
reserve at the (1−ε) quantile; ε ∈ {0.1, 1, 5}%used in TCO
Requests rarely fill their context window, and they do not all peak at once. The Capacity Planner uses this: instead of reserving a full window per stream, it reserves the KV pool at a high quantile of the total demand of all streams on a replica. The preemption risk is how often that total may exceed the pool — moments when the engine has to pause or evict a request.
Because the safety margin grows with √batch while the mean grows with batch, the buffer per stream shrinks as replicas get larger — big replicas multiplex better. Lower risk means more reserve and possibly more GPUs; when the fleet is bound by TPOT rather than memory, the setting changes nothing. The occupancy distribution comes from a workload preset or your p50 / p90 / p99 request lengths. Read more: KV risk gauge and occupancy trend.
Share of a replica's time spent in chunked prefill at the recommended batch. High values mean the fleet is prefill-bound — fix the cache hit rate or the prompt size before adding decode GPUs.
LoRA on top of a 4-bit (NF4) quantized frozen base. ~0.52 byte/param for the linear layers (embeddings and lm_head stay bf16); adapters train in FP32. A 70B run at rank 16 and 2K sequence needs about 49 GB per GPU — an 80 GB card.
≈ 0.52 byte/param for linear layersused in Fine-tuning
Speed-of-Light model: each stage runs at max(compute-bound, bandwidth-bound) time. Throughput/latency here are roofline estimates scaled by an empirical bandwidth factor (~0.8).
max(compute-bound, bandwidth-bound) timeused in Calculator · TCO
Common prompt prefix (system prompt) stored once per serving unit thanks to prefix caching (vLLM APC, SGLang RadixAttention). The Capacity Planner subtracts it from per-stream KV reservations; VRAM effect only.
Service-Level target you size against — here a max TPOT (per-token latency) and an optional max TTFT. The Capacity Planner finds the largest batch that still meets it.
Tensor parallelism splits every layer's weight matrices across GPUs: each GPU holds 1/TP of the weights and computes its slice, and after the attention and MLP blocks of each layer the partial results are combined with an all-reduce. It cuts per-GPU weight memory and the weight read per decode step by TP, at the price of two collective operations per layer on every step.
What TP changes
weights per GPU = weights / TP
KV per GPU = KV / TP (by KV heads; an MLA latent is replicated)
added per GPU = NCCL workspace (per TP size) + runtime overhead (repeated)
decode time ≈ step time / (TP × communication efficiency)
The efficiency depends on the fabric. On NVLink / NVSwitch and AMD xGMI the calculator uses calibrated tables up to the node size; on datacenter PCIe it allows TP up to 4 at about 0.80 (TP2) and 0.70 (TP4); consumer GeForce cards without peer-to-peer stay single-GPU. At batch 1 an all-reduce latency model applies instead, which is why a wider TP can cut TPOT sharply for a few users — the "Lowest latency" line of a result. Across nodes, the calculator adds pipeline stages rather than stretching TP over the network. Read more: How the calculator picks TP / PP / EP.
Each decode step produces one token for every stream in the batch, and to do it the GPU reads all active weights plus each stream's KV cache. At small and medium batch that read, not arithmetic, sets the step time — which is why TPOT tracks memory bandwidth far more than TFLOPS.
Formula · bandwidth floor of one decode step
TPOT ≈ (active weight bytes + KV bytes read) / (bandwidth × 0.8 × runtime efficiency)
+ fixed scheduler cost per step
tok/s per user ≈ 1000 / TPOT (ms)
Example floor: 70 GB of FP8 weights on one H100 (3.35 TB/s, scaled to about 2.7 TB/s) is about 26 ms per step before runtime efficiency and the KV read — roughly 38 tok/s per user at best. Tensor parallelism divides the weight read across GPUs and pays an all-reduce per layer; speculative decoding produces several tokens per target step. 50 ms (≈ 20 tok/s) reads as fluent typing, 20–30 ms feels instant. TPOT is what the Capacity Planner's SLA is written against. Read more: Why our tok/s are different.
Time To First Token. Prefill latency — how long before generation starts, per request on an idle replica (queueing not included). Dominated by compute on the input prompt.
Before the first output token the engine runs the whole prompt through the model in one or a few prefill passes. Prefill processes thousands of tokens per weight read, so unlike decode it is compute-bound: its time follows the prompt length and the GPU's TFLOPS in the precision you serve.
Example: Llama 3.1 8B in BF16 with an 8K prompt on one H100 comes out at about 0.6 s. The estimate is per request on an idle replica: in a burst the last prompt waits behind the others, and chunked prefill interleaves prompts with decode, so TTFT under load sits between the two. Agentic turns with a warm prefix cache prefill only the new tokens. Typical targets: under 200 ms to a few seconds for interactive chat, depending on how the product streams. Read more: latency targets.
Sharding strategies that split training state across GPUs. ZeRO-1 shards optimizer states, ZeRO-2 adds gradients, ZeRO-3 / FSDP also shards the weights. Single/DDP replicate everything (no memory saving).