Capacity Planner — step-by-step sizing guide

The Capacity Planner answers the question every platform and presales team gets: how many of which GPUs do we need to serve N concurrent users at our latency target? You describe the workload and the SLA; it returns complete fleet options — cards, topology, throughput and cost — ready to paste into a proposal. In the navigation the page is called TCO: fleet costs are rentals at live market rates, so the hourly price already carries power, hosting and the hardware itself.

1Pick the model and serving precision

Same model picker as the calculator: a catalog model, a HuggingFace ID or a manual architecture. Weight precision matters twice here — it shrinks the weights and speeds up decode, so FP8 against BF16 can halve the fleet. The context window you set is the maximum a request may use; how full it actually gets is handled by the occupancy model in step 3.

2Set the SLA: users, TPOT, optional TTFT

• Users at once — simultaneously active decode streams at your peak, not registered accounts. A rule of thumb for chat products: peak concurrency is about 5–10 % of daily active users.
• Target TPOT — per-token decode latency. About 50 ms reads like fluent typing (≈ 20 tok/s), 100 ms is acceptable for long-form generation, 20–30 ms feels instant. The planner finds the largest batch each serving unit can run while staying under it.
• Target TTFT (optional) — time to first token for one request. Empty by default, so it is not a sizing constraint; set it if your product shows nothing until generation starts (for chat typically 1–2 s). It gates long-context setups where prefill dominates. Both latencies are reported under the replica's prefill load, so they grow with the batch.

3Describe KV-cache occupancy — the money lever

The default, Worst case — full context, reserves the whole context window for every stream. That is safe but usually 3–5× too expensive: real requests fill a fraction of it. Two ways to tell the planner what your traffic looks like:

• Workload preset — Chat, Code completion, RAG / summarization and Long-context / agent set a mix of short, medium and long streams. The three sliders are shares of all users and must total 100 %; Normalize to 100% fixes any mix in one click.
• From your logs — the accurate way once you are in production: enter p50, p90 and p99 of total tokens per request (prompt + answer) from your gateway or vLLM metrics. The planner fits a lognormal to the three numbers and waits until all three are set.

Preemption risk is the explicit safety knob: the KV pool is sized so that at most 0.1 %, 1 % or 5 % of moments exceed it. When the pool does overflow, the engine preempts a stream and recomputes it — a latency spike, not an outage. Pick 0.1 % for strict customer-facing P99 targets, 1 % (default) for normal production, 5 % for batch and internal tools. The buffer shrinks as the fleet grows (√batch statistics), so on large fleets even 0.1 % costs little. How the gauge reserves KV.

Shared prefix tokens — if every request carries the same system prompt (agents and RAG commonly 2–10k tokens), enter its length: prefix caching (vLLM Automatic Prefix Caching, SGLang RadixAttention) stores it once per serving unit instead of once per stream. In chat mode this is a VRAM effect only; TTFT gains from cache hits are modeled in agentic mode.

4Read the results

• Total GPUs — the whole fleet: replicas × GPUs per serving unit. This is the number for the budget line.
• Topology — R identical replicas, each a tensor × pipeline unit, or EPn for MoE models (expert parallelism). For example 3×(4×1) = three replicas of 4-way tensor parallel = 12 cards.
• Batch/rep — concurrent streams one replica carries inside the TPOT target and the KV-cache VRAM cap. Users ÷ batch, rounded up, gives R.
• Agg tok/s — fleet-wide output throughput at the batch actually run.
• Prefill % — the share of each replica's time spent in chunked prefill; high values mean prompts, not decode, limit the replica.
• $/1M tok — fleet $/hr ÷ output tokens per hour: the most comparable cost metric across GPUs, and the default sort.
• Binding constraint — what stops a replica from taking one more stream: VRAM (KV pool), decode latency (TPOT), prefill or the batch cap.
• You save — occupancy-aware sizing in money: GPUs, $/month and $/year against reserving the full context window for every user, priced at live market rates.

Use Sort by to rank by cost, card count or throughput; options that cannot meet the SLA always sink to the bottom with the reason. Pin an anchor GPU — for example the one your customer already owns — to keep it as the headline row while alternatives are compared against it. A GPU detail page opens the planner with that card anchored.

Worked example

Llama 3.3 70B, FP8, 8K context, 400 concurrent users, TPOT 50 ms, Chat preset, 1 % risk, a 2k shared system prompt. Read the binding constraint first. If it says TPOT, the replica is limited by decode speed and the occupancy settings cannot lower the count — re-running at 0.1 % and 5 % risk will return the same fleet. If it says VRAM, the risk setting moves the count: re-run at 0.1 % and at 5 % to see what your risk appetite is worth in dollars. That spread is the business conversation. The presets in the card below open this example at the three risk levels.