Workload: context, users, batch, traffic

After the model and its precision, four inputs describe the load: Context, Users, Batch and Traffic. They look similar but move different terms — two of them set memory, two set speed and cost — and mixing them up is the most common reason a sizing comes out too big or too small.

Context: the window each request may use

Context is the maximum tokens one request can hold, prompt plus answer. The calculator reserves a full window of KV cache for every concurrent user, as an engine does when it must guarantee that any admitted request can run to the end. Quick values go from 4K to 1M tokens, capped at the model's architectural limit, which is added as a stop of its own.

Context is a memory input: it does not change speed directly, but it decides how many users fit. A 128K window for 32 users is 4M tokens of KV cache; for most models that is far more than the weights. If your requests are much shorter than the window, the result carries a note with the KV your typical request would need. Models with GQA, MLA or sliding-window layers grow more slowly with context — see Context length and what drops out.

Users: concurrent requests, not accounts

Users is the number of requests the GPUs serve at the same moment — active streams at your peak, not registered users or daily actives. A common rule of thumb for chat products is that peak concurrency is 5–10 % of daily active users; agents and batch jobs can be far higher per user. Users multiplies the KV reservation linearly, and by default it also sets the batch.

Batch: how many requests share a step

Batch (under Advanced) is how many requests the engine processes in one forward pass, up to 4,096. It follows Users until you change it. Speed is computed at the smaller of the two, so a batch below the user count models an engine that admits fewer requests per step than it holds in memory — the per-user speed rises, total throughput falls, memory stays the same. Batch never changes the KV reservation; Users does.

Batch is also why the same model on the same GPU shows very different speeds: at batch 1 decode is a pure weight read and per-user speed is at its highest; at batch 64 the weight read is shared across 64 streams, so total throughput is many times higher while each user's tokens arrive more slowly.

Traffic: what a typical request looks like

Traffic sets the typical prompt and answer length of one request. It drives TTFT, throughput and cost, not memory.

The prompt is clamped so that prompt plus answer fits the context. Exact lengths go into Prompt tokens and Answer tokens; the cell then reads "Custom". A long prompt with a short answer is the expensive case: at batch above 1 every output token carries its share of prefill time, so RAG costs noticeably more per output token than chat on the same configuration (how $/1M is computed).

The same load in TCO

The Capacity Planner takes the load the other way round. You give concurrent users and a latency target (TPOT, optionally TTFT) as the SLA; it finds the largest batch one serving unit can run under that target and adds data-parallel replicas until every user is covered. Instead of reserving the full window for everyone, it models how full contexts really get — a workload preset or your own p50 / p90 / p99 request lengths — and reserves KV at a chosen preemption risk. Sessions with many turns, tools and sub-agents use agentic mode (session fields). Step by step: Capacity Planner guide.

Getting the numbers

• Users — peak concurrent requests from your gateway or engine metrics (running requests), not request rate.
• Context — the window you will configure (max_model_len and similar), not the model's maximum.
• Traffic — median input and output tokens per request from your logs.
• Batch — leave it coupled to Users unless your engine caps running requests (--max-num-seqs) below your concurrency.