howmanygpus.ai / accuracy
How accurate is howmanygpus.ai?
Recomputed by the live calculator · model calibrated 2026-10-23
Every number below is a public measurement — a vLLM or llama.cpp log, an NVIDIA benchmark table, an MLPerf submission — next to what this calculator predicts for exactly that configuration. The predictions are produced by the same code that answers your query, so this page cannot drift from the product. Each row links to its source, and the rows we get wrong are left in.
GPU memory
11/11
within −10 % … +20 % · median error 2.1 %
Throughput
19/29
within ±20 % · median -5.5 %
TensorRT-LLM table
56 %
of 534 rows within ±20 %
Time to first token
4/4
within ±25 % (llama.cpp)
How to read the tables
- Error = prediction ÷ measurement − 1. For memory, a negative error is the dangerous one — it means we would tell you a config fits when it runs out of memory — so the memory band is asymmetric (−10 % … +20 %). For throughput a negative error means we are conservative.
- calibration — at least one model constant (an efficiency factor, a tensor-parallel tier, the per-step cost) was tuned on this point, so the error shows how well the model reproduces public data, not how it does on a setup nobody measured. check — no constant was tuned on it.
- Real deployments vary with framework version, driver and settings: expect ±15–30 % against your own load test and validate the shortlist before you buy.
GPU memory — 11 public logs
Memory the configuration needs at one request: weights, KV cache for the context, activations, runtime overhead (CUDA graphs, buffers). vLLM's pre-allocated KV pool is excluded — it fills whatever is left. Multi-GPU rows are per unit at the measured tensor-parallel degree.
| Case | Model | GPUs | Context | Measured | Ours | Error | Fit | Source |
|---|---|---|---|---|---|---|---|---|
| C1 | LLaMA 3.1 8BQ4_K_M · llama.cpp | A100 PCIe 80GB | 8,192 | 5.96 GB | 6.53 GB | +9.6% | check | llama.cpp #10664 |
| C2 | LLaMA 3.2 3BQ4_K_M · llama.cpp | RTX 4090 24GB | 74,000 | 10.79 GB | 11.02 GB | +2.1% | check | llama.cpp #15664 |
| C3 | GPT OSS 20BMXFP4 · llama.cpp | RTX 3090 24GB | 32,768 | 12.72 GB | 14.17 GB | +11.4% | check | llama.cpp #15789 |
| C4 | LLaMA 3.1 8BINT8 · vLLM | RTX 4090 24GB | 65,536 | 20.12 GB | 20.37 GB | +1.2% | calibration | vLLM #14286 |
| C5 | Qwen 2.5 14B4-bit · vLLM | RTX 3090 24GB | 32,768 | 20.24 GB | 19.94 GB | -1.5% | calibration | vLLM #11289 |
| C6 | Qwen3 30B-A3B4-bit · vLLM | RTX 4090 24GB | 16,384 | 20.38 GB | 20.51 GB | +0.6% | calibration | vLLM PR #24217 |
| C7 | Qwen 2.5 72B4-bit · vLLM | 2× L40S 48GB | 16,000 | 59.44 GB | 59.35 GB | -0.2% | calibration | vLLM #15228 |
| C8 | Qwen 2.5 32BBF16 · vLLM · no chunked prefill, eager | 4× L40 48GB | 131,072 | 150.75 GB | 140.88 GB | -6.5% | calibration | vLLM #16883 |
| C9 | Qwen 2.5 32BBF16 · vLLM | 2× RTX 6000 Ada Generation 48GB | 12,288 | 73.10 GB | 76.82 GB | +5.1% | calibration | vLLM #12664 |
| C10 | GPT OSS 120BMXFP4 · vLLM | 2× H100 SXM5 80GB | 131,072 | 81.74 GB | 74.47 GB | -8.9% | calibration | vLLM #46933 |
| C11 | DeepSeek R1FP8 (block) · vLLM | 16× H100 SXM5 80GB | 16,384 | 833.80 GB | 849.70 GB | +1.9% | calibration | vLLM #16766 |
Throughput — 29 published benchmarks
Output tokens per second for the whole serving unit. "max" = the benchmark ran at the highest concurrency that fits; we then report our best rate over a batch sweep under the same memory limit. Since v0.32.0 the prediction counts prompt processing as well as generation.
| Model | GPUs | In/Out | Concurrency | Measured | Ours | Error | Fit | Source |
|---|---|---|---|---|---|---|---|---|
| LLaMA 3.1 8BFP8 · TensorRT-LLM | H100 SXM5 80GB | 1000/1000 | max | 14,992 | 14,102 | -5.9% | calibration | TensorRT-LLM performance overview |
| LLaMA 3.1 8BBF16 · TensorRT-LLM | A100 SXM4 80GB | 1000/1000 | max | 3,634 | 3,526 | -3.0% | calibration | TensorRT-LLM v0.14 performance overview |
| LLaMA 3.1 8BFP8 · TensorRT-LLM | L40S 48GB | 1000/1000 | max | 3,727 | 3,617 | -2.9% | calibration | TensorRT-LLM v0.14 performance overview |
| LLaMA 3.1 8BBF16 · vLLM | A100 SXM4 80GB | 2048/128 | 4 | 230 | 173 | -24.9% | calibration | SqueezeBits — vLLM vs TensorRT-LLM |
| LLaMA 3.1 70BFP8 · TensorRT-LLM | 4× H100 SXM5 80GB TP4 | 1000/1000 | max | 6,933 | 6,950 | +0.3% | calibration | TensorRT-LLM v0.14 performance overview |
| LLaMA 3.1 70BFP8 · TensorRT-LLM | 8× H200 SXM 141GB TP8 | 1000/1000 | max | 13,081 | 14,940 | +14.2% | calibration | TensorRT-LLM v0.14 performance overview |
| DeepSeek R1FP8 (block) · TensorRT-LLM | 8× H200 SXM 141GB EP8 | 1024/2048 | 1024 | 11,489 | 9,556 | -16.8% | calibration | TensorRT-LLM DeepSeek-R1 best-practice blog |
| DeepSeek R1-0528NVFP4 · TensorRT-LLM | 8× B200 SXM 180GB EP8 | 1024/2048 | 7168 | 43,146 | 70,903 | +64.3% | calibration | TensorRT-LLM DeepSeek-R1 best-practice blog |
| Mixtral 8x7BFP8 · TensorRT-LLM | 2× H100 SXM5 80GB TP2 | 1000/1000 | max | 15,932 | 13,831 | -13.2% | calibration | TensorRT-LLM v0.14 performance overview |
| LLaMA 3.1 8BFP8 · TensorRT-LLM | L40S 48GB | 1000/1000 | 250 | 3,676 | 3,600 | -2.1% | calibration | NVIDIA NIM benchmarking docs (NIM 1.8.0) |
| LLaMA 3.1 8B4-bit · vLLM · soft setup | RTX 4090 24GB | 64/256 | 7 | 579 | 685 | +18.3% | calibration | Community vLLM benchmark on RTX 4090 |
| LLaMA 3.1 8BBF16 · vLLM · soft setup | A100 SXM4 80GB | 2048/128 | 16 | 639 | 361 | -43.6% | calibration | SqueezeBits — vLLM vs TensorRT-LLM |
| Qwen3 32BBF16 · vLLM · soft setup | H100 SXM5 80GB | 217/201 | max | 1,132 | 1,402 | +23.9% | calibration | GPUStack performance lab — Qwen3-32B on H100 |
| LLaMA 3.3 70BFP8 · TensorRT-LLM | 2× H100 SXM5 80GB TP2 | 1000/1000 | max | 4,181 | 4,155 | -0.6% | calibration | TensorRT-LLM performance overview |
| LLaMA 2 70BNVFP4 · TensorRT-LLM | B200 SXM 180GB | 1024/1024 | max | 11,264 | 7,995 | -29.0% | calibration | NVIDIA blog — MLPerf Inference v4.1 |
| LLaMA 2 70BNVFP4 · TensorRT-LLM | 8× B200 SXM 180GB (replicas) | 1024/1024 | max | 98,858 | 81,912 | -17.1% | calibration | NVIDIA blog — MLPerf Inference v5.0 |
| Mixtral 8x7BNVFP4 · TensorRT-LLM · mean prompt length unpublished — decode rate | 8× B200 SXM 180GB EP8 | 2048/145 | max | 128,148 | 93,641 | -26.9% | calibration | NVIDIA blog — MLPerf Inference v5.0 |
| Mixtral 8x7BFP8 · TensorRT-LLM · mean prompt length unpublished — decode rate | 8× H200 SXM 141GB EP8 | 2048/145 | max | 62,630 | 54,568 | -12.9% | calibration | NVIDIA blog — MLPerf Inference v5.0 |
| DeepSeek R1NVFP4 · TensorRT-LLM | 72× GB300 NVL72 EP72 | 3136/3880 | max | 420,624 | 342,059 | -18.7% | calibration | NVIDIA blog — Blackwell Ultra in MLPerf Inference v5.1 |
| LLaMA 3.3 70BNVFP4 · TensorRT-LLM | B200 SXM 180GB | 1024/2048 | max | 6,547 | 8,456 | +29.2% | calibration | TensorRT-LLM performance overview |
| LLaMA 3.1 405BNVFP4 · TensorRT-LLM | 4× B200 SXM 180GB TP4 | 128/2048 | max | 7,178 | 9,596 | +33.7% | calibration | TensorRT-LLM performance overview |
| LLaMA 3.3 70BFP8 · TensorRT-LLM | 2× H200 SXM 141GB TP2 | 128/2048 | max | 7,467 | 12,575 | +68.4% | calibration | TensorRT-LLM performance overview |
| DeepSeek V3FP8 (block) · SGLang · decode workers only (disaggregated) | 72× H100 SXM5 80GB EP72 | 2000/100 | 4096 | 200,520 | 185,403 | -7.5% | calibration | LMSYS — large-scale expert parallelism |
| DeepSeek R1NVFP4 · SGLang · decode workers only (disaggregated) | 48× GB200 NVL72 EP48 | 2000/100 | 1408 | 642,528 | 220,582 | -65.7% | calibration | LMSYS — DeepSeek on GB200 NVL72, part 2 |
| LLaMA 2 7B4-bit · llama.cpp | RTX 4090 24GB | 512/128 | 1 | 188 | 180 | -3.8% | calibration | llama.cpp CUDA scoreboard (discussion #15013) |
| LLaMA 2 7B4-bit · llama.cpp | RTX 5090 32GB | 512/128 | 1 | 295 | 320 | +8.6% | calibration | llama.cpp CUDA scoreboard (discussion #15013) |
| LLaMA 2 7B4-bit · llama.cpp | A100 SXM4 80GB | 512/128 | 1 | 196 | 172 | -12.4% | calibration | llama.cpp CUDA scoreboard (discussion #15013) |
| LLaMA 3.1 405BFP8 · TensorRT-LLM | 8× H200 SXM 141GB TP8 | 2048/128 | 1 | 56 | 53 | -5.5% | calibration | NVIDIA blog — Llama 3.1 405B on H200 + NVLink Switch |
| LLaMA 3.1 405BFP8 · TensorRT-LLM | 8× H200 SXM 141GB TP8 | 2048/128 | max | 506 | 428 | -15.4% | check | NVIDIA blog — Llama 3.1 405B on H200 + NVLink Switch |
Beyond these points the prefill model was fitted on the whole TensorRT-LLM v0.14 performance table: 534 rows (7 models × 6 GPUs × 9 prompt/answer lengths). 56 % of them land within ±20 % — before the prefill fix it was 23 %.
Time to first token — llama.cpp
512-token prompt, Llama 2 7B Q4_0, from the llama.cpp CUDA scoreboard (TTFT = 512 ÷ the pp512 rate).
| GPU | Measured | Ours | Error | Fit |
|---|---|---|---|---|
| RTX 4090 24GB | 42.7 ms | 45.7 ms | +7.0% | calibration |
| RTX 5090 32GB | 36.4 ms | 36.1 ms | -0.8% | calibration |
| RTX 3090 24GB | 98.9 ms | 106.2 ms | +7.3% | calibration |
| H100 SXM5 80GB | 51.6 ms | 58.1 ms | +12.5% | calibration |
Where the model is off
- LLaMA 3.3 70B on 2× H200 SXM 141GB TP2 (128/2048, TensorRT-LLM): +68.4%.
- DeepSeek R1 on 48× GB200 NVL72 EP48 (2000/100, SGLang): -65.7% — decode workers only (disaggregated).
- DeepSeek R1-0528 on 8× B200 SXM 180GB EP8 (1024/2048, TensorRT-LLM): +64.3%.
- LLaMA 3.1 8B on A100 SXM4 80GB (2048/128, vLLM): -43.6% — soft setup.
- LLaMA 3.1 405B on 4× B200 SXM 180GB TP4 (128/2048, TensorRT-LLM): +33.7%.
- Very large decode batches on H200 / B200 with long answers run high: the benchmark engines' schedulers cap the batch below what fits in memory, our sweep does not.
- Datacenter time-to-first-token has no reliable public ground truth yet — the TTFT table above is llama.cpp only.
- The calibration set is NVIDIA-heavy: AMD Instinct figures carry wider uncertainty.
Method: Help → Methodology · How we predict tok/s · Try a configuration in the calculator.