Question 1

How accurate are the estimates?

Accepted Answer

VRAM estimates are tight (within ~5%) because they are derived from explicit byte budgets: weights, KV cache, activations, framework overhead, and NCCL workspace. Throughput, TTFT, and TPOT are roofline (Speed-of-Light) estimates scaled by an empirical bandwidth factor; real performance depends on framework version, driver stack, batching, and prompt distribution, so expect plus/minus 15-30% variance.

Question 2

How does the calculator pick TP/PP/EP?

Accepted Answer

For dense models it tries tensor parallelism first up to max_nvlink_gpus (8 for SXM, 36/72 for rack-scale), then adds pipeline parallelism across nodes if a single node cannot hold the model, avoiding PP when TP alone is enough because pipeline bubbles hurt latency. For MoE models it tries expert parallelism up to max_nvlink_gpus.

Question 3

Can I use this for fine-tuning?

Accepted Answer

Yes. The Fine-tuning Calculator models training VRAM for full fine-tune, LoRA, and QLoRA (weights, fp32 master copy, gradients, optimizer states, adapter, and activations) with ZeRO-1/2/3 and FSDP sharding. Enter training tokens to also get wall-clock time, tokens/sec, and cost-to-train per recommended GPU.

Question 4

How does the Capacity Planner differ from the Inference Calculator?

Accepted Answer

The Inference Calculator answers, for a given batch size, what fits and how fast. The Capacity Planner inverts it: you give a user count and a latency SLA (TPOT, optional TTFT), and it solves for the largest batch that still meets the SLA, then scales data-parallel replicas to cover all users, returning total GPUs, topology, and dollars per million tokens for each candidate card.

Question 5

How is $/1M tokens calculated?

Accepted Answer

GPU on-demand dollars-per-hour (times the number of GPUs) divided by aggregate throughput converted to tokens-per-hour. It is the most comparable cost metric across cards because it normalizes away GPU count and price differences. Hourly rates are indicative; swap in your contracted price for budgeting.

Question 6

Do you support AMD GPUs?

Accepted Answer

Yes. The catalog includes AMD Instinct cards (e.g. MI300X and MI250X) alongside NVIDIA. VRAM and roofline throughput are modeled the same way, though the throughput calibration set is NVIDIA-heavy, so AMD figures carry somewhat wider uncertainty, and quantization availability differs (FP8 is supported; NVFP4 is NVIDIA-Blackwell-only).

Question 7

Why is NVFP4 only available on Blackwell?

Accepted Answer

NVFP4 is a hardware-native 4-bit float format with per-block scaling, accelerated only by Blackwell Tensor Cores (B200/B300/GB200/GB300, RTX 5090, RTX PRO 6000 Blackwell, DGX Spark). On older GPUs the calculator falls back to a bf16-equivalent rate, so pick AWQ or INT8 for 4-/8-bit on Hopper, Ada, or Ampere instead.

How many GPUs do you need for LLM inference?