Inference calculator — overview

The Inference calculator answers the first sizing question for a model: which GPUs hold it, in what configuration, how fast and at what cost per token. It checks every GPU in the catalog that is offered as a sizing target against the model and workload you describe and returns the configurations that fit, with the memory budget behind each one.

What it needs from you

The input deck across the top of the page holds one cell per decision. Click a cell to open its panel; every change re-runs the estimate once.

• Model — one of 141 catalog models, a HuggingFace repo ID, or a manual architecture (parameters, layers, heads, KV heads, experts). The catalog and HuggingFace paths pre-fill the geometry, including MoE and MLA fields. See Model input.
• Precision — the weight format. The panel lists every format your runtime supports with its VRAM, relative speed and the smallest GPU that holds it. KV cache is a separate cell: FP16, FP8 or INT8.
• Users — concurrent active requests. Each one reserves a context window of KV cache, so this is usually the biggest memory lever after the weights.
• Context — the maximum tokens one request can hold (prompt + answer), capped at the model's limit.
• Traffic — the typical prompt and answer length: Chat (1K → 256), RAG (8K → 256), Long answers (1K → 2K) or Agents (16K → 512). It drives throughput, TTFT and cost, not memory.
• Runtime — vLLM, SGLang, TensorRT-LLM, Dynamo, llama.cpp or Ollama. It changes overhead, available precisions and speed; see Frameworks.
• Advanced — batch, prompt and answer tokens, latency targets, LoRA adapters and speculative decoding. See The deck and Advanced.

The answer and the GPU list

The right-hand panel states the answer in one sentence — for example "runs on 2× H100 (TP2) for 16 users at 8K context" — with the memory bar of that configuration: weights, KV cache, activations and overhead against the card's VRAM. Below it are the speed-versus-cost chart, the share actions and the "What if more people use it?" and "What if prompts get longer?" panels, which re-price the answer at other user counts and context lengths.

The table lists every configuration that fits. Each row shows the GPU, how many cards (n), memory use with its fit band, output tok/s per user, TTFT, $/hr and $/1M tokens. Open a row for the full readout: TPOT, total output throughput, the parallel layout, why this GPU was chosen, the fastest alternative on more cards of the same model, and live cloud prices. Use as answer makes that row the headline; Pin as baseline keeps it on screen so other rows show their difference against it.

How the list is ordered

The default sort, Fewest GPUs, keeps the backend order: single-node before multi-node before rack-scale, live cards before end-of-life ones, then fewest GPUs, then fit band, then $/hr. A single card that fits is always the top recommendation. Best fit and $/hr re-sort the same rows. Tier chips (datacenter, prosumer, consumer) and Single GPU only filter them. The reasoning behind the order is in Multi-node, rack-scale and recommendation order.

Where to go next

• Optimize map — every GPU × precision × parallel shape that fits this model on one cost-versus-speed chart (Optimize).
• Size it in TCO — turn the answer into a fleet for a user count and a latency target (Capacity Planner vs calculator).
• Share — the address bar encodes the whole configuration; the share sheet adds a summary, JSON, a PNG card and an embed code.