Agentic mode — sizing session-based workloads

The Chat / Agentic switch on the TCO page changes the unit of planning from a concurrent user to a session: a task that runs for many turns, whose context grows turn by turn, that may spawn sub-agents, and that spends part of its life waiting on tools or people. It fits coding agents, support assistants, RAG chats and multi-agent research alike — anything where prefill and the cache hit rate, not decode speed, decide the fleet.

1Pick the closest preset

Preset	Shape of a session	Why it matters for GPUs
Coding agent	40 turns, 6k prefix (system prompt + tools), heavy-tailed tool results (p90 6k), compaction at 80 %	Prefill-heavy; the cache hit rate decides whether you need extra GPUs.
Support agent	8 short turns, waits on the human 75 % of the time	Few active streams per session; idle KV must survive the pauses.
RAG chat	6 turns, 3k–8k retrieved tokens arrive on every turn	New context each turn is never cached — pure prefill load.
Multi-agent research	20 turns, 3 sub-agents live per session, long answers	Sub-agents multiply active streams; each shares only the prefix.

Every value is editable; editing switches the preset to Custom. The four core fields are always visible; the rest sit under Advanced and keep the preset's values until you change them.

2Every session parameter, explained

A turn is one model call: the model reads the whole context (prefill), writes an answer or a tool call (decode), and the result of that tool — or the user's reply — comes back as the next turn's prompt. The context grows by prompt + output every turn. Each entry says what the field means, where to get the number and what it moves in the sizing.

Core

#	Core — the shape of one task	Where to get it	Effect on GPUs
01	Turns per sessionHow many model calls one task takes from start to finish. Every tool call whose result goes back to the model is a turn; so is every human reply.	Agent traces: calls per completed task (median). Presets: coding 40, research 20, support 8, RAG 6.	Length of the context-growth curve: more turns → longer history → higher peak KV per session and more prefill per turn when the cache misses. Also multiplies the session wall time, so it drives cost per session.
02	Prefix tokens (system prompt + tools)The part every session shares verbatim: system prompt, tool/function schemas, fixed instructions. It opens every context.	Tokenize your system prompt plus the tool definitions you send. Tool-heavy agents are typically 3–10k.	Counted in every session's KV, but prefilled only once per replica — it is assumed always resident in the prefix cache. A big prefix raises KV per session; it does not raise prefill load.
03	Prompt tokens per turn (p50)New tokens that arrive each turn: the user message, a tool result, or retrieved documents. The median, not the mean.	Gateway logs: input tokens minus the previous turn's context. RAG: chunks per query × chunk size.	Grows the context every turn, and is prefilled every turn even at a 100% cache hit rate — new tokens can never be cached. The floor of your prefill load.
04	Output tokens per turnTokens the model generates per turn: reasoning, the tool call or the answer.	Gateway logs: completion tokens per call (median). Reasoning models are much higher.	Decode work per turn, so it sets how long a turn holds its stream under the TPOT target. Also appended to the context — but generated tokens are already in the KV cache, so they are never prefilled again.

Rhythm

#	Rhythm — how many streams are live at once	Where to get it	Effect on GPUs
05	Concurrent sessionsTasks alive at the same time — parents only; sub-agents come from the profile. Set next to the SLA fields, not in the profile.	Peak concurrent tasks from your product metrics (not requests per second).	Multiplied by (1 + sub-agents) × active share × burst it gives the active decode streams the fleet is sized for — the number shown as "Active streams".
06	Active share %Share of a session's wall time spent waiting on the model. The rest is tools running, a human reading, or network.	Trace timing: Σ(model latency) ÷ task duration. Autonomous coding agent ~60%, human-in-the-loop support ~25%.	Scales concurrent sessions down to active streams: 200 sessions at 60% ≈ 120 streams before sub-agents and bursts. The idle 40% still hold KV the cache would like to keep — this is what the hit-rate prediction works with.
07	Burst factorHow peaky the load is: the p99 of simultaneously active turns divided by their mean. 1 = perfectly even.	Per-second active-request counts over a busy hour: p99 ÷ mean. 1.2–1.5 is typical; batch-triggered agents run higher.	Straight multiplier on active streams — the fleet is sized for the burst peak, not the average.
08	Sub-agents per sessionAverage number of sub-agents alive at once per parent session (0.5 = half the time one is running). Each is a short, fresh-context session that shares only the prefix with its parent.	Count spawned sub-agent sessions × their duration ÷ parent duration. Orchestrator/worker research setups: 2–4.	Multiplies active streams by (1 + sub-agents). Their KV is small (short, fresh context), so they add decode and prefill load more than VRAM.
09	Sub-agent turnsHow many turns one sub-agent runs before returning. Blank = a third of the parent's turns; never more than the parent.	Median call count per sub-agent task in your traces.	Length of the sub-agents' own context curve; mixed with the parent's to form the per-stream KV distribution.

Context

#	Context — how big the history gets	Where to get it	Effect on GPUs
10	Prompt tokens per turn (p90)The heavy tail of new tokens per turn: the occasional 10k-token file read or search dump. Blank = every turn is the p50.	Same gateway logs as p50, 90th percentile. Coding agents: 4–6× the median.	Fitted as a lognormal with the p50: raises the mean growth per turn and its variance. The variance widens the KV reserve at the chosen preemption risk; the tail sets TTFT p90 — the turn that prefills a big tool result.
11	Compact context at % of windowThe fill level of the context window at which the agent summarizes its history and starts over (Claude Code, Cursor and most coding agents do this). Blank = never.	Your agent framework's auto-compaction setting; 80% is common.	Caps peak KV per session and produces the sawtooth in the trajectory chart. Blank means the history grows until the context window and is clamped there.
12	Summary tokens after compactionSize of the summary that replaces the history. After a compaction the context is prefix + summary.	Length of your compaction summaries; 2–4k is typical.	Sets the floor of each sawtooth tooth. A larger summary keeps more memory but restarts higher, so the next compaction comes sooner and the average KV per session is higher.

Cache

#	Cache — what does not have to be prefilled again	Where to get it	Effect on GPUs
13	Cache hit rate % (override)Share of the previous turn's history still in the prefix cache when the next turn arrives. Blank = predicted by the planner.	Measured prefix-cache hits on your serving stack (vLLM / SGLang metrics, or your gateway) — use it once you have it.	Blank: predicted from the VRAM each replica has left for idle sessions (LRU at session granularity), and the fleet is sized at the largest hit rate its own free VRAM delivers. Set: the fleet is sized at your value. Lower hit rate → more prefill → possibly extra replicas.
14	KV offload to host RAM (GB per unit)Host memory per serving unit that keeps idle sessions' KV when VRAM runs out (LMCache, Dynamo KVBM, SGLang HiCache).	What you configure in the offload layer; bounded by the server's RAM.	Adds to the capacity the hit-rate prediction sees, so more history survives the pauses. Offloaded KV reloads over PCIe; that reload time is added to TTFT.

Generated tokens are never prefilled again. They are already in the KV cache when the next turn starts. Each turn prefills only the new prompt tokens plus the share of the history that fell out of the cache.

3Set the load and the SLA

Sessions at once = tasks alive at the same time (parents only — sub-agents come from the profile). TPOT and the optional TTFT work as in chat mode, but both are reported under the prefill load of the recommended batch: chunked prefill shares scheduler steps with decode. When you switch to agentic mode or pick a preset and the context window is too small for the profile — one turn must fit, and compaction must leave room for prefix, summary and one turn — the page raises it to the smallest context stop that fits, never above the model's maximum.

4Read the Agentic sessions card

Binding constraint — what stops a replica from taking one more stream: VRAM (KV pool), decode latency (TPOT), prefill (the actionable one: fix the cache or the prompt before buying decode GPUs) or the batch cap.
KV per session, turn by turn — the occupancy of one session over its turns; dashed verticals are compactions, the dashed ceiling is the context window, the dot is the peak.
Prefill / turn — mean tokens prefilled per turn with the cache against without it. Only new prompt tokens plus the uncached share of the history are prefilled.
Cache-hit gauge — the bar is the hit rate the fleet was sized at (predicted, or your override); the marker is the required hit rate: the smallest one at which no replicas beyond the decode-only sizing are needed. "No cache needed" means prefill fits regardless; "no cache hit rate removes them" means the extra replicas come from new tokens that can never be cached.
Cost / session — fleet $/hr ÷ sessions completed per hour, using the loaded turn time and the active share; also shown per turn and as a table column. The options table sorts by it by default in this mode, and the prefill share column shows how busy each replica is with prompts.

5What to do with the answer

Prefill-bound with a required hit rate below the predicted one → you are fine; keep prefix caching on (vLLM APC / SGLang RadixAttention).
Required above predicted → add host KV offload, pin sessions to replicas (KV-aware routing) or shrink tool results; re-plan to see the replica count drop.
"No cache hit rate removes them" → the new tokens themselves are the load: shorter tool outputs, a smaller prefix, or accept the extra replicas.
Compare $/session across GPUs rather than $/1M tokens — for agents the session is the unit you sell.

The Pareto explorer is not available in agentic mode; the options table and the card above are the agentic readout. The field reference without the walkthrough is in Agentic session parameters.

Treat the output as an estimate, not a quote. The session model is built from first principles and has few public agentic benchmarks to calibrate against. Use it to compare GPUs and to find the binding constraint, then load-test the shortlisted configuration with your own traces. How accurate is this?