Reading the tok/s per user × $/1M map

The Optimize map plots every setup that fits one model on two logarithmic axes. Once you know what the position, colour, size and outline of a mark encode, the map answers most "which GPU and precision" questions at a glance.

The axes

The horizontal axis is output tokens per second for each user — right is faster. Under each tick the same value is shown as milliseconds per token (TPOT = 1000 ÷ tok/s), so you can read latency directly. The vertical axis is $ per million output tokens — down is cheaper. The corner label "cheaper & faster ↘" points to where the best setups sit. Both axes are logarithmic because setups span orders of magnitude: a 4-bit single card and an 8-GPU BF16 node differ several-fold on both.

The second view, Tokens per GPU, replaces cost with output tokens per second per GPU — the axis NVIDIA and SemiAnalysis's InferenceX use. It ignores price and shows hardware efficiency: up and to the right means more tokens from each card at the same speed per user.

$/1M = unit $/hr ÷ (output tok/s × 3600) × 10⁶

unit $/hr = (market median, else catalog price) per GPU × GPUs in the unit; output tok/s counts prefill time

What a mark encodes

• Colour = precision: BF16, FP8, INT8, 4-bit (AWQ), NVFP4, MXFP4 and the GGUF formats on llama.cpp and Ollama. Tap a precision in the Precision filter to hide it.
• Size = how many GPUs: small for one GPU, medium for 2–8 GPUs in one node, large for multi-node and rack-scale units.
• Filled or hollow: filled marks are inside the benchmark-calibrated range; hollow marks are physics-modeled only — pipeline stages, TP above 8, PCIe or AMD multi-GPU, hybrid attention. Treat their position as approximate.
• Badges on the pick card: calibrated or modeled, FP8 emulated on GPUs without FP8 tensor cores, end of life, MoE via TP.

The trade-off line and the three views

The stepped line is the trade-off line — the Pareto front of the visible setups: no other setup is both cheaper and faster than a point on it. It is recomputed whenever you hide a precision or set a limit. The Show control picks which marks are drawn at full strength:

• Best per GPU (default) — the trade-off line plus the best setup for each GPU and precision pair, so every card stays visible without the clutter of every TP degree.
• Trade-off line — only the front.
• All — every setup that fits at this context.

Setups outside the current view stay as faint grey marks for context. When the front mixes bit widths — for example FP8 and 4-bit — a note says so: lower precisions trade some quality for speed and price, so hide a precision to compare like with like.

The pick card and the list

The card beside the map shows the best setup for your goal, or the one you selected: GPU count and name, precision and parallel layout, $/1M, tok/s per user with ms per token, $/hr, first-token time, memory per GPU (weights, KV cache, runtime and activations) and the longest context that setup holds. Against the goal winner it states the trade-off, for example "4.1× faster · 7.6× the price · +3 GPUs". Apply to calculator opens the setup in the calculator; Size a fleet in TCO takes it to the Capacity Planner.

Click a mark to select it (tap to inspect on a phone, then tap its card). With the keyboard, ← and → step along the trade-off line and Enter applies. The top-5 list ranks setups by the goal; Table view shows the same rows as the map as a ranked table.

The footnote

Under the map a mono line reports how many GPU × precision × context combinations were evaluated, how many setups are drawn at this context, how many sit on the trade-off line, how many have no price (not drawn — no cost coordinate) and the date of the market prices used.