Frameworks — what each runtime changes

The serving runtime is not a label on the result: it changes how much memory the engine keeps for itself, how many tokens one scheduler step may process, which weight formats load at all, how close decode gets to the memory-bandwidth limit and whether the server can batch. Pick the runtime you will deploy with — the Runtime cell of the calculator, What fits and Optimize (Optimize offers all but Dynamo); the default is vLLM.

What the runtime changes

• Overhead is memory the engine holds besides weights, KV cache and activations — CUDA context, allocator slack, non-torch allocations and, on vLLM and SGLang, CUDA-graph buffers. It is repeated on every GPU of a multi-GPU unit, like the NCCL workspace.
• Tokens per step is the chunked-prefill budget (vLLM max_num_batched_tokens, SGLang chunked_prefill_size, TensorRT-LLM max_num_tokens, llama.cpp n_ubatch, Ollama num_batch). Activation memory is billed for the smaller of all prompts in flight and this budget — never below the decode batch — which is why a busy server does not need workspace for every prompt at once.
• Step cost is a fixed scheduler cost added to every decode step on top of the roofline time. It matters at small batch and on small models, where a step is only a few milliseconds; consumer GeForce hosts carry 3 ms for vLLM and 2 ms for Dynamo.

Decode efficiency — the share of memory bandwidth a runtime realises — is fitted per engine on saturated batches. For the four serving engines it blends toward a common batch-1 value as the batch shrinks, because at batch 1 every current engine streams weights at much the same rate; llama.cpp and Ollama keep their own fitted factors. Batched prefill, which sets total throughput at batch above 1, runs at 55 % of peak compute on the four serving engines; llama.cpp and Ollama prefill prompts one at a time.

vLLM

PagedAttention and continuous batching; the default, and the runtime most public benchmarks are run on. Overhead is 1.0 GB plus 6.5 × 10⁻⁶ GB per layer × hidden size for CUDA graphs, capped at 6 GB — about 1.9 GB for an 8B model with 32 layers of 4,096 and about 5.3 GB for a 70B with 80 layers of 8,192, the same on every tensor-parallel rank. Decode reaches about 0.75 of the scaled bandwidth limit at high batch; the per-step cost of 1.5 ms reflects the V1 engine with async scheduling and full CUDA graphs. Supports every safetensors format, including FP8 KV cache, NVFP4 on Blackwell, LoRA serving and speculative decoding; runs on AMD Instinct through ROCm builds. Linear-attention state follows the model dtype (BF16). Above 100 users the result reminds you to tune --max-num-seqs and --gpu-memory-utilization.

SGLang

RadixAttention prefix caching and FlashInfer kernels. Same overhead model as vLLM, but activation coefficients 30 % higher for the same shapes. Decode efficiency at high batch is higher (0.83) and the overlap scheduler keeps the per-step cost at 0.5 ms, so SGLang decode usually comes out somewhat faster than vLLM on the same configuration while its activation memory is larger. Recurrent state of linear-attention layers (Gated DeltaNet, KDA, Mamba) is billed in FP32, as SGLang allocates it by default. Its prefix-cache advantage on shared-prefix workloads is real but not runtime-specific in the model: prefix reuse is sized in TCO through the shared prefix and agentic cache hit rate.

TensorRT-LLM

NVIDIA's compiled-engine backend, also inside NIM and Triton; the memory formulas here originate from its model (via NVIDIA's aiconfigurator). Flat 5.0 GB overhead, the highest high-batch decode efficiency (0.85) and 0.5 ms per step. Engines must be built per GPU, model and precision before serving — typically 30–90 minutes. NVIDIA-only: on an AMD answer the result carries an error. Consumer RTX cards and speculative decoding get warnings for limited support. Linear-attention state is billed in FP32, as on SGLang.

Dynamo

NVIDIA's distributed serving layer for large multi-GPU and multi-node deployments, with vLLM-class workers. Modeled with 5.5 GB overhead, 0.78 decode efficiency and 1.5 ms per step. The estimate is aggregated serving — prefill and decode on the same GPUs. Disaggregated prefill/decode, Dynamo's main feature, is not modeled: dedicated prefill workers improve TTFT and throughput at the cost of more GPUs, so treat the result as the baseline before disaggregation. Single-GPU answers and consumer GPUs get warnings; NVIDIA-only.

llama.cpp

GGUF files, CPU/GPU hybrid, aimed at one GPU and local use. The lightest overhead (0.5 GB) and a 512-token micro-batch, which keeps the compute buffers small. Weight formats are the GGUF ones — Q8_0, Q5_K_M, Q4_K_M — plus BF16 and MXFP4; FP8 weights and FP8 KV cache are not supported, and INT8 KV stands in for its q8_0 cache type. Its CUDA kernels are tuned for consumer parts: decode efficiency is 0.85 on GeForce cards and 0.40 on datacenter GPUs, and prefill on GeForce runs close to the card's BF16 rate. Multi-GPU answers and more than 8 concurrent users are flagged as errors — the model treats llama.cpp as a single-GPU runtime without continuous batching — and no wider-TP "lowest latency" alternative is offered. Models of 30B and up get a note that layers can be offloaded to CPU RAM at a large speed cost; offload itself is not modeled.

Ollama

A llama.cpp wrapper with model management, for local prototyping. Same formats, limits and 512-token batch as llama.cpp, with 1.0 GB overhead, slightly lower efficiencies (0.75 decode on consumer cards, 0.35 on datacenter GPUs) and a 0.5 ms HTTP-layer cost per step. One memory difference matters: Ollama allocates KV for the full declared context on sliding-window layers, so models such as Gemma 3 or gpt-oss do not get the window saving they get on other runtimes — set a smaller num_ctx.

Choosing for an estimate

• Size with the runtime you will run; the differences are 10–30 % in speed and a few GB per GPU in memory, enough to change a fit band.
• Comparing runtimes for the same model is a one-cell change; the precision list follows the runtime, so a GGUF runtime switches the weights to a GGUF format.
• Runtime versions move faster than any model of them; the calibration of each engine is on the accuracy page, and What is not modeled lists the gaps.