KV risk gauge and occupancy trend

Two charts on a chat-mode capacity result explain how much KV cache the planner reserves and why: the risk gauge shows the demand distribution of one replica against its reserved pool, and the occupancy trend shows how the per-stream reservation falls as the fleet grows. Both come from the same model; this article explains it on fixed examples.

How the risk model reserves KV — the gauge

Every serving replica runs many streams at once, and each stream's KV-cache footprint is a random variable: some requests are short, some long. By the central limit theorem the total KV demand of a full batch is approximately normal — a bell curve. The planner reserves the pool at the (1 − ε) quantile of that curve, where ε is your chosen preemption risk (0.1, 1 or 5 %). The shaded tail to the right of the reservation line is exactly ε: the fraction of moments where demand exceeds the pool and a few streams get preempted — evicted and recomputed, a latency spike rather than an outage.

The same workload (Chat, 64 streams per replica) at three risk appetites. Watch the dashed reservation line move: lower risk pushes it right (bigger pool → more GPUs, smaller tail), higher risk pulls it left (cheaper, but a fatter shaded tail of preemption moments).

Conservative · 0.1 % risk

Balanced · 1 % risk

Aggressive · 5 % risk

On a result, the gauge also carries the "×N more users per GPU vs worst-case" line: how many more streams each GPU holds than it would if every stream reserved the full context window.

Why bigger fleets are cheaper per user — the occupancy trend

The per-stream reservation is the mean occupancy plus a safety buffer that scales with 1/√b:

eff_ctx(b) = S/b + max(μ − S, 0) + z(ε) · σ / √b

S     shared prefix (stored once per replica)
μ, σ  mean and standard deviation of per-stream occupancy, tokens
z(ε)  normal quantile for the preemption risk ε

On a tiny fleet the reservation sits near the full context window — you cannot average away variance across a handful of streams — and as the batch grows it falls toward the mean. The gap below the dashed 100 % line is what statistical multiplexing saves, and it widens as the fleet grows. Telecoms size trunk lines and banks compute value-at-risk with the same effect.

Three workloads on the same 8k window at 1 % risk. A Chat load (short requests, about 19 % mean fill) sits far below the 100 % line — large savings. A Long-context load (about 83 % mean fill) hugs the line — little to gain, because there is little slack to reclaim. The savings on a real result depend entirely on how full your requests actually run.

Chat · 19 % mean fill

RAG / summarize · 69 % mean fill

Long-context · 83 % mean fill

The You save card on your capacity result turns this gap into GPUs and dollars for your workload.

When the gauge does not move the count

The risk setting changes the KV reservation, so it only changes the fleet when VRAM is what limits a replica. If the binding constraint is TPOT — decode speed — the replica runs out of latency budget before it runs out of KV pool, and 0.1 % and 5 % return the same number of GPUs. Read the binding constraint on the result before spending time on the risk setting.