Capacity Planner — modeling assumptions

The Capacity Planner turns a user count and a latency target into a fleet. These are the assumptions behind that step — read them before you put a number in a proposal, because each one is a place where your production traffic can differ from the model.

Load and latency

• Concurrent users = active decode streams. The user count is that many simultaneously active streams at a steady-state worst case. There is no request-rate or queueing model, so treat it as the peak you must hold, not average traffic.
• TPOT drives the sizing; prefill shares the replica. Per-token latency (TPOT) sets the largest batch each replica can run. Every stream takes its next prompt as soon as its answer is done, so a replica also spends part of its time prefilling prompts (chunked prefill, priced at the batched-prefill rate). TTFT and TPOT are reported under that load — on prompt-heavy traffic (long RAG prompts, short answers) prefill can be what limits a replica, and both latencies grow with the batch.
• Topology minimizes total cards, not per-stream fit. Each GPU's tensor, pipeline or (for MoE) expert parallelism is chosen to minimize the whole fleet for your SLA, so counts are comparable across GPUs. A conservative throughput margin is applied above 4 GPUs and at very large batch — the least-calibrated regime — so the count errs slightly high rather than low.

KV-cache reservation

• The worst-case default reserves the full context window per stream (100 % KV fill): no preemption, the most conservative count. Pick a workload preset or enter p50/p90/p99 from your logs for risk-aware, cheaper sizing.
• Risk-aware sizing reserves a quantile, not the mean. The total KV demand of a replica's batch is treated as approximately normal (central limit theorem over the streams) and the pool is reserved at its (1 − ε) quantile, where ε is the preemption risk. Streams are assumed independent; correlated bursts (everyone pasting a long document at once) are not modeled. The figures are in KV risk gauge and occupancy trend.
• A shared prefix is stored once per serving unit. In chat mode this lowers the per-stream reservation only; prefill savings from prefix-cache hits are modeled in agentic mode.

Fleet and cost

• Homogeneous fleet. Every replica is the same GPU and topology; mixing cards is not modeled.
• Costs assume a fleet busy 24/7 at on-demand rates — the median of the providers we track, or the catalog price when no market price exists. Real utilization and negotiated pricing lower both the $/hr and the $/1M-token figures proportionally.

Agentic mode

• Agentic mode sizes sessions, not users. Concurrent sessions × (1 + sub-agents) × active share × burst = the decode streams the fleet must hold at peak. A random active stream is at any turn of its session, so the KV pool is reserved over the whole trajectory (prefix cached once per unit, compaction sawtooth if you set it). Sub-agents are short fresh-context sessions sharing only the prefix.
• Prefill counts only new tokens. Generated tokens are already in the KV cache; each turn prefills the new prompt plus the uncached share of the history. Chunked prefill shares scheduler steps with decode, so TPOT and TTFT are reported under that load, and replicas are added when prefill binds (at most 85 % of replica time in prefill).
• The cache hit rate is predicted, not assumed. Idle sessions keep their KV only while it fits in free VRAM (plus host offload, reloaded over PCIe) — LRU at session granularity, solved together with the batch. Override it under Advanced if you measured it. The "required" marker is the smallest hit rate at which the fleet needs no replicas beyond its decode-only sizing; the gauge says so when no hit rate removes them.

Uncertainty

What the planner does not model at all — queueing under burst load, multi-tenant contention, disaggregated prefill and decode — is listed in What is not modeled.