The input deck and Advanced: batch, targets, LoRA, speculative decoding

The calculator has no separate Basic and Advanced modes any more. The input deck — Model, Precision, KV cache, Users, Context, Traffic, Runtime — covers what most sizing questions need, and the last cell, Advanced, holds the inputs that only matter for some deployments. Its value reads "Defaults" until you change something, then "N set". On a desktop it opens as a popover under its cell, like every other step; on a phone as a sheet. This article covers the four Advanced groups and when each is worth opening.

What the deck already decides

The Runtime cell, a separate deck cell rather than an Advanced field, changes the framework overhead, the precisions on offer and the speed model — see Frameworks. Traffic sets the prompt and answer length from four presets, which is enough for most estimates. If those two cells describe your service, you can leave Advanced alone: the batch then follows the user count, there are no latency targets, and no adapters or draft model are loaded.

Batch and lengths

• Batch size — requests processed together in one forward pass, up to 4,096. By default it equals Users. The speed model runs at the smaller of batch and users, so a batch below the user count models a server that admits fewer requests per step than it holds. Memory does not follow the batch: every concurrent user still reserves a context window of KV cache. Lower the batch to see per-user speed at light load; raise Users to see the memory of a full server.
• Prompt tokens / Answer tokens — exact lengths when no Traffic preset matches; the Traffic cell then reads "Custom". The prompt drives TTFT and the prefill share of throughput; the answer drives total output and TPOT. At batch above 1, throughput counts prompt processing too, so a long prompt with a short answer costs more per output token.
• Chunked prefill — on by default, as in every current engine: the scheduler caps tokens per step (8,192 on vLLM, SGLang and TensorRT-LLM), which bounds activation memory. Turn it off only for engines that run without it; activations are then reserved for one whole maximum-length prompt — several GB per GPU at 128K context.

Targets

Three optional pass/fail checks shown on each GPU row: Max ms / token (TPOT), Max first token, ms (TTFT) and Min total tok/s. They do not change which GPUs are listed or how they are ordered; they mark which rows meet the target. All three are empty by default (no check). For TTFT, set the time your product can show nothing for one request — for chat typically 1–2 s, longer for long-prompt and batch work. If the target is the question — "how many GPUs to meet 50 ms per token for 400 users" — use the Capacity Planner, which sizes against it instead of checking it.

LoRA adapters

Serve adapters adds the memory of LoRA adapters loaded next to the base weights, for multi-tenant serving where each request may use a different adapter (vLLM --max-loras, SGLang --max-loras-per-batch). Switching it on starts at rank 64 and one adapter.

• Rank — common values 8 to 256; memory grows linearly with it.
• Adapters loaded — adapters resident in VRAM at the same time.
• Target modules — attention (Q, K, V, O) or all-linear (also gate, up and down — the PEFT and vLLM default, about 3× the parameters per adapter on Llama 3.1 8B; the ratio depends on the model's MLP width and KV heads).

Adapters are stored in BF16 and sharded with the weights across tensor-parallel GPUs. Example: Llama 3.1 8B, rank 64, ten adapters is about 3.4 GB extra with all-linear targets and 1.1 GB attention-only. The estimate covers memory; the extra compute of applying adapters is not modeled. Training adapters is a different calculation — see the fine-tuning calculator.

Speculative decoding

Draft model turns on speculative decoding: a small draft model proposes K tokens, the target model verifies them in one pass. Switching it on starts at K = 5 and an acceptance rate of 0.8; set the Draft size in billions of parameters.

E[accepted] = (1 − α^(K+1)) / (1 − α)
speed-up    = E[accepted] / (1 + K × draft / target)

α = per-token acceptance rate, K = draft tokens per step
target = active parameters for MoE models

With α = 0.8, K = 5 and a draft a tenth of the target, the formula gives about 2.5×. The gain is a per-stream latency effect: at batch 4 and above the weight read is already shared across the batch, so the bonus fades toward 1× as the batch grows. The draft adds its own weights, a share of KV cache and activations to the memory budget. Acceptance depends on how well the draft matches the target and on your prompts; 0.7–0.9 is typical for a well-matched pair such as a Llama 3.2 1B draft for a Llama 3.1 70B target.

Auto-detected architecture notes

The Advanced panel also shows what the model picker detected: an MLA note for DeepSeek-style latent attention, a hybrid-attention note for sliding-window and linear-state layers, and an MoE note with experts and active parameters. They are read-only; to change the architecture, edit the model in the Model cell.