Weight and KV-cache precision

Two precision choices shape every result. Weight precision sets the bytes per parameter of the model itself — usually the largest term of the memory budget and, at small batch, the bytes every decode step has to read. KV-cache precision sets the bytes per element of the attention cache, which grows with users and context. They are separate deck cells because they are separate engine settings.

Weight formats

The list depends on the runtime: safetensors engines (vLLM, SGLang, TensorRT-LLM, Dynamo) and GGUF runtimes (llama.cpp, Ollama) load different files.

The bytes above include scale overhead. Formats that keep embeddings or the LM head in 16 bit (INT8, AWQ, NVFP4, MXFP4) bill those tensors at 2 bytes, which is why a small 4-bit model is more than a quarter of its BF16 size. The Precision panel lists each format with its total VRAM, the change against BF16, a quality and a relative-speed label and the smallest GPU that holds it; above them, the quantization advisor offers cheaper combinations (weights and KV cache together) when one exists.

Where a format is native

A lower-precision format always saves memory; whether it also runs faster depends on the GPU.

• FP8 has tensor-core support on Hopper, Blackwell, Ada Lovelace and AMD MI300-class GPUs. On Ampere, Turing and Volta (A100, A10, A30, A40, RTX 3090, T4, V100) and on the MI250X it is emulated: the weights still take half the memory, but compute-bound work runs at about 0.7× BF16, slower than INT8. The calculator notes this when the answer GPU has no FP8 path.
• NVFP4 and MXFP4 get the FP4 compute rate only on Blackwell: B200, B300, GB200, GB300, RTX 5090, RTX PRO 6000 Blackwell and DGX Spark. Elsewhere they run at the BF16-equivalent compute rate, so they save memory and weight-read bandwidth but give no prefill speed-up; on Hopper, Ada and Ampere pick AWQ or INT8 for 4- and 8-bit instead.
• At small batch decode is a stream of weight reads, and 8- and 4-bit kernels do not realise the full byte saving: the model credits about 70 % of it at batch 1, rising to the full saving as the batch grows. Public measurements put FP8 decode on an H100 at about 1.4× BF16 for a single user, not 2×.
• GGUF formats run on llama.cpp and Ollama; vLLM-class engines get a warning, because their GGUF loading is experimental. AWQ and FP8 are not offered on GGUF runtimes.

KV-cache precision

The KV cache takes three modes: FP16 (2 bytes per element, the default, also covering BF16), FP8 and INT8 (1 byte each). Halving the KV cache halves the memory every user reserves, which at long context or high concurrency is often a bigger saving than quantizing the weights. FP8 KV is native on Hopper and Blackwell and supported by vLLM, SGLang, TensorRT-LLM and Dynamo; llama.cpp and Ollama do not support FP8 KV and quantize their cache through their own GGUF cache types, which INT8 approximates. Quality impact is usually small for FP8 and needs calibration for INT8.

KV bytes per token = 2 × layers × KV heads × head dim × bytes per element
MLA: layers × (kv_lora_rank + qk_rope_head_dim) × bytes per element

Choosing

• Start from the format the model is published in — re-quantizing is a project of its own. Catalog entries offer only their published formats; the HuggingFace path detects the native one.
• On Hopper and Blackwell, FP8 weights with FP8 KV is the common production pair; on Blackwell, NVFP4 is the next step.
• When the fit is tight because of users or context, try FP8 or INT8 KV before a smaller weight format. The Precision panel says so when the KV cache is more than half the budget.
• The calculator sizes memory and speed, not output quality; validate a 4-bit model on your own evaluation set.