How the calculator picks TP / PP / EP

When a model does not fit on one card, the calculator splits it. For each GPU it looks for the smallest serving unit that holds the model within 95 % of every card's memory, using tensor parallelism first, pipeline stages only when TP alone cannot do it, and expert parallelism for MoE models. The fabric between the cards sets how far each can go.

Dense models: TP first, PP only when needed

The memory budget is split into the part that shards across GPUs — weights, LoRA adapters, the draft model and, for GQA and MHA models, the KV cache by attention head — and the part every GPU repeats: activations, framework overhead and the NCCL workspace. The steps:

1. If everything fits on one card within 95 %, the answer is one GPU.
2. Otherwise the minimum card count is the sharded memory divided by what each card has left after the replicated part.
3. The TP degree must divide the number of attention heads evenly. The calculator takes the smallest valid TP at or above the minimum, up to the fabric's limit.
4. The memory is recomputed for that TP — activation coefficients and the NCCL workspace depend on it. If a card now lands above 95 %, the next valid TP is tried.
5. Only when no TP inside the fabric is enough does it keep the largest valid TP and add pipeline stages: PP = ⌈minimum ÷ TP⌉.

Pipeline stages are avoided because they cost latency: a batch waits for the micro-batches travelling through the stages (the pipeline bubble), which the speed model charges as batch ÷ (batch + PP − 1). Two KV-cache details matter here. KV heads replicate when TP exceeds their number, so TP 16 on a model with 8 KV heads is legal but does not shrink the KV cache further. And MLA models (DeepSeek, Kimi) keep their compact latent KV on every GPU — attention TP does not shard it.

The fabric sets the limit

PCIe, bridge and xGMI multi-GPU results are outside the benchmark-calibrated range and are labelled as reduced calibration. On PCIe, TP above 4 loses to TP 4 × PP 2: all-reduce traffic crosses the root complex, so a pair on one switch does better than a quad.

MoE models: expert parallelism

For mixture-of-experts models the calculator tries EP first. With EP = N, every GPU holds all non-expert weights (attention, norms, embeddings) plus 1/N of the experts. KV cache is split by request rather than by head — attention data parallel, each rank keeps the KV of its share of the streams, ⌈streams ÷ N⌉ — so one stream's KV never splits across GPUs. EP degrees come from a power-of-two ladder up to the smaller of the expert count and the fabric's EP domain; the smallest one that fits wins. If no EP degree fits on a GPU, that GPU falls back to the dense TP/PP path. Rack-scale systems are used for EP only beyond 8 GPUs.

Reading the result

Each row states its layout: TP4, TP2×PP2 or EP8, with the explanation in the expanded row. The expanded row also shows the fastest alternative: the lowest-TPOT single-node TP unit of the same GPU with more cards than the recommendation — the answer when latency matters more than card count. llama.cpp and Ollama split layers across GPUs rather than tensors, so extra GPUs give them memory but not speed, and no faster alternative is offered. To see every TP, PP and EP shape that fits side by side, open the Optimize map.