Frequently asked questions
11 short answers; each links to the article that explains it in full.
How accurate are the estimates?
Memory is an explicit byte budget — weights, KV cache, activations, runtime overhead and NCCL workspace — and lands close to measurements: 11 of 11 public memory logs within −10…+20 %, median error 2.1 %.
Throughput, TTFT and TPOT come from a calibrated roofline: 20 of 29 published benchmarks and 107 of 167 low-concurrency measurements (batch 1–16) within ±20 %. Against your own deployment expect ±15–30 %, depending on runtime version, drivers, batching and the prompt mix. Every point with its source is on the accuracy page.
What is not modeled?
Inference: queueing and p99 latency under bursts (TTFT is per request on an idle replica), disaggregated prefill/decode, prefix-cache reuse in the calculator (TCO models it), runtime-specific speculative decoding, CPU offload and multi-tenant contention.
Fine-tuning: CPU/NVMe optimizer offload, communication losses in multi-GPU wall-clock time (scaling is linear) and a percentage-based overhead (a flat 1.5 GB per GPU is used). LoRA and QLoRA compute is about 4N FLOPs per token, not the 6N of full fine-tuning, because the frozen base skips its weight gradients. Activation memory is an estimate; weights, gradients and optimizer states are exact.
How does the calculator pick TP/PP/EP?
Dense models: the smallest serving unit that fits within 95 % of every card, using tensor parallelism first — up to the NVLink domain (8 on SXM nodes, up to 72 on rack-scale systems) or 4 over PCIe — and pipeline stages only when TP alone cannot hold the model, because pipeline bubbles cost latency.
MoE models: expert parallelism first, the smallest power-of-two degree that fits; a GPU on which no EP degree fits falls back to the dense TP/PP path.
Why does my pre-quantized HuggingFace model not pre-fill correctly?
The native format is read from quantization_config.quant_method, then repo tags and file names (awq, fp8, nvfp4, gptq, GGUF), then the safetensors tensor types and finally torch_dtype. A repo that uses none of these signals is treated as BF16. Pick the precision yourself in the Precision cell if the detection is wrong.
Why are some rack-scale GPUs missing from recommendations?
Rack-scale systems (GB200 NVL36/72, GB300 NVL72, GH200 NVL32) appear only when a model needs more than 8 GPUs — for MoE models, an expert-parallel degree above 8 — because a partial rack cannot realistically be rented or bought. Very large models (DeepSeek V3 in BF16, Llama 3.1 405B at long context) and high user counts bring them in.
Read more: Multi-node, rack-scale and recommendation order →
Can I use this for fine-tuning?
Yes — the fine-tuning calculator models training memory for full fine-tuning, LoRA and QLoRA: weights, the FP32 master copy, gradients, optimizer states, adapters and activations, with ZeRO-1/2/3 and FSDP sharding. Enter training tokens to also get tokens per second, wall-clock time and cost to train per recommended GPU. Memory is calibrated against EleutherAI's Transformer Math, the QLoRA paper and Unsloth's VRAM tables; activation memory is an estimate.
How does the Capacity Planner differ from the Inference Calculator?
The Inference calculator answers "for this load, what fits and how fast?". The Capacity Planner inverts it: you give a user count and a latency SLA (TPOT, optional TTFT), and it solves for the largest batch that still meets the SLA, then scales data-parallel replicas to cover all users — returning total GPUs, topology and $/1M for each candidate card. Latency-bound sizing carries wider uncertainty, so treat it as a starting estimate.
How is $/1M tokens calculated?
The hourly price of the serving unit (market median per GPU × the number of GPUs) divided by its output tokens per hour, with the unit busy 24/7. Output throughput includes prompt processing: at batch above 1 each output token carries its share of the prefill time (prompt ÷ answer length at the batched-prefill rate), so an 8K-prompt RAG request costs about twice as much per output token as a 1K chat turn — as it does on real hardware. It is the most comparable cost metric across cards because it folds GPU count, price and speed into one figure. Hourly rates are indicative (Estimates, not quotes) — swap in your contracted price for budgeting.
Do you support AMD GPUs?
Yes — the GPU catalog (48 GPUs · 10 Jetson modules) includes AMD Instinct cards (MI250X, MI300X, MI325X, MI350X, MI355X), modeled with the same byte budget and roofline as NVIDIA. The throughput calibration set is NVIDIA-heavy, so AMD figures carry wider uncertainty; multi-GPU over xGMI is modeled at NVLink-class efficiency. FP8 is native on MI300-class cards, NVFP4 is NVIDIA-Blackwell-only, and TensorRT-LLM and Dynamo do not run on AMD.
Why is NVFP4 only available on Blackwell?
NVFP4 is a 4-bit float format with an FP8 scale per 16 weights whose compute runs on Blackwell FP4 tensor cores only (B200/B300/GB200/GB300, RTX 5090, RTX PRO 6000 Blackwell, DGX Spark). On older GPUs the calculator keeps its memory saving but uses the BF16-equivalent compute rate, so there is no prefill speed-up — pick AWQ or INT8 for 4- and 8-bit on Hopper, Ada and Ampere instead.