Best LLMs to run on GH200 NVL32 (141 GB)
datacenter 141 GB HBM3e · 4900 GB/s · 989 TFLOPS BF16 · updated 2026-10-23
Headline answer
Of 141 models in our catalog, 113 fit on a single GH200 NVL32 at FP8, 119 at 4-bit (AWQ) and 97 at BF16 (8K context, batch 1). Largest that fits: Nemotron-H 56B Base at BF16; Devstral 2 123B at FP8; DeepSeek Coder V2 at 4-bit (AWQ). Rental: $1.49/hr catalog estimate.
Change context, concurrency or framework in the calculator → GH200 NVL32 full specs
Every model on GH200 NVL32
Fit band and share of VRAM per precision; N× GPUs = minimum count inside one node (tensor parallel); — = does not fit within one 8-GPU node. Speed and $/1M tokens for the highest-quality precision that fits on one GPU (BF16, else FP8, else 4-bit), single stream.
| Model | Params | BF16 | FP8 | 4-bit (AWQ) | tok/s | $ / 1M tok |
|---|---|---|---|---|---|---|
| DeepSeek Coder V2 | 236.0B (21.0B active) | 8× GPUs | 2× GPUs | tight 92% | 152 4-bit | $2.73 |
| Qwen3 235B-A22B | 235.0B (22.0B active) | 8× GPUs | 2× GPUs | tight 92% | 147 4-bit | $2.81 |
| MiniMax M2.5 | 230.0B (10.0B active) | 4× GPUs | 2× GPUs | tight 90% | 210 4-bit | $1.97 |
| MiniMax M2.7 | 230.0B (10.0B active) | 4× GPUs | 2× GPUs | tight 90% | 210 4-bit | $1.97 |
| Mixtral 8x22B | 141.0B (39.0B active) | 4× GPUs | 2× GPUs | optimal 57% | 103 4-bit | $4.01 |
| DBRX 132B | 132.0B (36.0B active) | 4× GPUs | 2× GPUs | optimal 54% | 109 4-bit | $3.79 |
| Devstral 2 123B | 123.0B | 2× GPUs | tight 94% | optimal 54% | 22 FP8 | $18.56 |
| Mistral Large 2 123B | 123.0B | 2× GPUs | tight 94% | optimal 51% | 22 FP8 | $18.56 |
| Nemotron-3 Super 120B-A12B | 120.0B (12.0B active) | 2× GPUs | tight 90% | optimal 49% | 141 FP8 | $2.94 |
| Mistral Small 4 119B | 119.0B (11.0B active) | 2× GPUs | tight 89% | optimal 48% | 148 FP8 | $2.79 |
| GPT OSS 120B | 117.0B (5.1B active) | 2× GPUs | tight 88% | optimal 48% | 210 FP8 | $1.97 |
| Llama 4 Scout (109B) | 109.0B (17.0B active) | 2× GPUs | tight 83% | optimal 46% | 113 FP8 | $3.66 |
| Command R+ 104B | 104.0B | 2× GPUs | good 80% | optimal 49% | 26 FP8 | $15.92 |
| Qwen3-Next 80B-A3B | 80.0B (3.0B active) | 2× GPUs | optimal 61% | optimal 34% | 247 FP8 | $1.67 |
| Qwen 2 72B | 72.0B | 2× GPUs | optimal 57% | optimal 34% | 36 FP8 | $11.40 |
| Qwen 2.5 72B | 72.0B | 2× GPUs | optimal 57% | optimal 34% | 36 FP8 | $11.40 |
| CodeLlama 70B | 70.0B | 2× GPUs | optimal 55% | optimal 31% | 37 FP8 | $11.13 |
| DeepSeek R1 Distill 70B | 70.0B | 2× GPUs | optimal 55% | optimal 33% | 37 FP8 | $11.13 |
| LLaMA 2 70B 4K ctx | 70.0B | 2× GPUs | optimal 54% | optimal 30% | 37 FP8 | $11.13 |
| LLaMA 3.1 70B | 70.0B | 2× GPUs | optimal 55% | optimal 33% | 37 FP8 | $11.13 |
| LLaMA 3.3 70B | 70.0B | 2× GPUs | optimal 55% | optimal 33% | 37 FP8 | $11.13 |
| Llama-3.1-Nemotron-70B | 70.0B | 2× GPUs | optimal 55% | optimal 33% | 37 FP8 | $11.13 |
| Nemotron-H 56B Base | 56.0B | tight 85% | optimal 46% | optimal 29% | 24 BF16 | $17.03 |
| Llama-3.1-Nemotron-51B | 51.0B | good 76% | optimal 41% | optimal 25% | 26 BF16 | $15.62 |
| Llama-3.3-Nemotron Super 49B | 49.0B | good 74% | optimal 40% | optimal 24% | 28 BF16 | $15.05 |
| Kimi Linear 48B-A3B | 48.0B (3.0B active) | good 71% | optimal 38% | optimal 22% | 198 BF16 | $2.09 |
| Mixtral 8x7B | 46.7B (12.9B active) | optimal 70% | optimal 37% | optimal 21% | 85 BF16 | $4.89 |
| Command R 35B | 35.0B | optimal 60% | optimal 36% | optimal 28% | 37 BF16 | $11.25 |
| Qwen3.6 35B-A3B | 35.0B (3.0B active) | optimal 53% | optimal 29% | optimal 17% | 198 BF16 | $2.10 |
| CodeLlama 34B | 34.0B | optimal 52% | optimal 29% | optimal 17% | 38 BF16 | $10.83 |
| Yi-1.5 34B 4K ctx | 34.0B | optimal 52% | optimal 28% | optimal 17% | 38 BF16 | $10.83 |
| DeepSeek R1 Distill 32B | 32.0B | optimal 50% | optimal 28% | optimal 18% | 40 BF16 | $10.27 |
| Nemotron-Terminal-32B | 32.0B | optimal 50% | optimal 28% | optimal 18% | 40 BF16 | $10.27 |
| OLMo-2 32B 4K ctx | 32.0B | optimal 49% | optimal 27% | optimal 16% | 40 BF16 | $10.27 |
| Qwen 2.5 32B | 32.0B | optimal 50% | optimal 28% | optimal 18% | 40 BF16 | $10.27 |
| Qwen 2.5 Coder 32B | 32.0B | optimal 50% | optimal 28% | optimal 18% | 40 BF16 | $10.27 |
| Qwen3 32B | 32.0B | optimal 50% | optimal 28% | optimal 18% | 40 BF16 | $10.27 |
| Nemotron-3 Nano 30B-A3B | 31.6B (3.2B active) | optimal 48% | optimal 26% | optimal 15% | 193 BF16 | $2.14 |
| Muse Glimmer 30B | 30.0B | optimal 46% | optimal 25% | optimal 16% | 43 BF16 | $9.69 |
| Qwen3 30B-A3B | 30.0B (3.0B active) | optimal 46% | optimal 25% | optimal 15% | 197 BF16 | $2.10 |
| Gemma 2 27B | 27.0B | optimal 43% | optimal 25% | optimal 18% | 47 BF16 | $8.88 |
| Gemma 3 27B | 27.0B | optimal 44% | optimal 25% | optimal 19% | 46 BF16 | $8.90 |
| Qwen3.6 27B | 27.0B | optimal 43% | optimal 24% | optimal 17% | 47 BF16 | $8.86 |
| Devstral 24B | 24.0B | optimal 38% | optimal 21% | optimal 14% | 52 BF16 | $8.02 |
| Devstral Small 2 24B | 24.0B | optimal 38% | optimal 21% | optimal 14% | 52 BF16 | $8.02 |
| Magistral Small 24B | 24.0B | optimal 38% | optimal 21% | optimal 14% | 52 BF16 | $8.02 |
| Mistral Small 3.1 24B | 24.0B | optimal 38% | optimal 21% | optimal 14% | 52 BF16 | $8.02 |
| Codestral 22B | 22.0B | optimal 35% | optimal 20% | optimal 13% | 56 BF16 | $7.46 |
| GPT OSS 20B | 21.0B (3.6B active) | optimal 33% | optimal 18% | optimal 12% | 183 BF16 | $2.26 |
| InternLM 2.5 20B | 20.0B | optimal 32% | optimal 18% | optimal 13% | 60 BF16 | $6.90 |
| DeepSeek Coder V2 Lite | 16.0B (2.4B active) | optimal 26% | optimal 15% | optimal 9% | 216 BF16 | $1.92 |
| StarCoder2 15B | 16.0B | optimal 26% | optimal 15% | optimal 10% | 72 BF16 | $5.74 |
| DeepSeek R1 Distill 14B | 14.8B | optimal 25% | optimal 15% | optimal 11% | 76 BF16 | $5.42 |
| Qwen 2.5 14B | 14.8B | optimal 25% | optimal 15% | optimal 11% | 76 BF16 | $5.42 |
| Qwen3 14B | 14.8B | optimal 25% | optimal 14% | optimal 11% | 76 BF16 | $5.42 |
| Phi 3 Medium | 14.0B | optimal 24% | optimal 14% | optimal 10% | 80 BF16 | $5.21 |
| Phi-4 14B | 14.0B | optimal 24% | optimal 14% | optimal 10% | 80 BF16 | $5.21 |
| Qwen 2.5 Coder 14B | 14.0B | optimal 24% | optimal 14% | optimal 11% | 80 BF16 | $5.21 |
| OLMo-2 13B 4K ctx | 13.7B | optimal 25% | optimal 15% | optimal 11% | 80 BF16 | $5.19 |
| LLaMA 2 13B 4K ctx | 13.0B | optimal 24% | optimal 15% | optimal 10% | 83 BF16 | $4.99 |
| Gemma 3 12B | 12.0B | optimal 22% | optimal 14% | optimal 12% | 89 BF16 | $4.66 |
| Mistral Nemo 12B | 12.0B | optimal 21% | optimal 12% | optimal 10% | 89 BF16 | $4.64 |
| Nemotron-Nano-12B-v2 | 12.0B | optimal 21% | optimal 13% | optimal 10% | 89 BF16 | $4.65 |
| StableLM 2 12B 4K ctx | 12.0B | optimal 20% | optimal 12% | optimal 9% | 89 BF16 | $4.64 |
| Falcon3 10B | 10.0B | optimal 18% | optimal 11% | optimal 8% | 102 BF16 | $4.07 |
| GLM-4 9B | 9.0B | optimal 16% | optimal 10% | optimal 8% | 110 BF16 | $3.78 |
| Gemma 2 9B | 9.0B | optimal 18% | optimal 11% | optimal 10% | 109 BF16 | $3.81 |
| Nemotron-Nano-9B-v2 | 9.0B | optimal 17% | optimal 11% | optimal 9% | 109 BF16 | $3.80 |
| Yi-1.5 9B 4K ctx | 9.0B | optimal 16% | optimal 10% | optimal 7% | 109 BF16 | $3.79 |
| InternLM3 8B | 8.8B | optimal 16% | optimal 10% | optimal 8% | 111 BF16 | $3.73 |
| Qwen3-VL 8B | 8.8B | optimal 16% | optimal 10% | optimal 8% | 111 BF16 | $3.73 |
| MiniCPM-V 4.5 | 8.7B | optimal 16% | optimal 10% | optimal 8% | 112 BF16 | $3.71 |
| Ministral-3 8B | 8.5B | optimal 16% | optimal 10% | optimal 8% | 113 BF16 | $3.65 |
| Qwen2.5-VL 7B | 8.3B | optimal 15% | optimal 9% | optimal 7% | 116 BF16 | $3.58 |
| DeepSeek R1 Distill 8B | 8.0B | optimal 15% | optimal 9% | optimal 8% | 118 BF16 | $3.51 |
| Granite 3.3 8B | 8.0B | optimal 15% | optimal 10% | optimal 7% | 118 BF16 | $3.51 |
| LLaMA 3.1 8B | 8.0B | optimal 15% | optimal 9% | optimal 8% | 118 BF16 | $3.51 |
| Ministral 8B | 8.0B | optimal 15% | optimal 10% | optimal 8% | 118 BF16 | $3.51 |
| Qwen3 8B | 8.0B | optimal 15% | optimal 10% | optimal 8% | 118 BF16 | $3.51 |
| Qwen 2.5 7B | 7.6B | optimal 14% | optimal 9% | optimal 7% | 122 BF16 | $3.39 |
| Falcon3 7B | 7.5B | optimal 14% | optimal 9% | optimal 7% | 123 BF16 | $3.35 |
| Phi-3 Small 7B | 7.4B | optimal 14% | optimal 9% | optimal 7% | 124 BF16 | $3.34 |
| InternLM 2.5 7B | 7.0B | optimal 14% | optimal 9% | optimal 7% | 128 BF16 | $3.23 |
| LLaMA 2 7B 4K ctx | 7.0B | optimal 14% | optimal 10% | optimal 7% | 127 BF16 | $3.27 |
| Mistral 7B | 7.0B | optimal 14% | optimal 9% | optimal 6% | 128 BF16 | $3.23 |
| OLMo-2 7B 4K ctx | 7.0B | optimal 14% | optimal 10% | optimal 8% | 127 BF16 | $3.27 |
| Qwen 2 7B | 7.0B | optimal 13% | optimal 8% | optimal 7% | 129 BF16 | $3.22 |
| Qwen 2.5 Coder 7B | 7.0B | optimal 13% | optimal 8% | optimal 7% | 129 BF16 | $3.22 |
| StarCoder2 7B | 7.0B | optimal 13% | optimal 8% | optimal 6% | 129 BF16 | $3.22 |
| Yi-1.5 6B 4K ctx | 6.0B | optimal 12% | optimal 7% | optimal 6% | 141 BF16 | $2.94 |
| Qwen3-VL 4B | 4.4B | optimal 10% | optimal 7% | optimal 6% | 165 BF16 | $2.51 |
| Gemma 3 4B | 4.3B | optimal 10% | optimal 7% | optimal 7% | 168 BF16 | $2.47 |
| Nemotron-Mini-4B 4K ctx | 4.0B | optimal 9% | optimal 6% | optimal 6% | 174 BF16 | $2.38 |
| Qwen3 4B | 4.0B | optimal 9% | optimal 7% | optimal 6% | 174 BF16 | $2.38 |
| Ministral 3B | 3.8B | optimal 9% | optimal 6% | optimal 6% | 178 BF16 | $2.32 |
| Phi 3.5 Mini | 3.8B | optimal 11% | optimal 8% | optimal 7% | 176 BF16 | $2.35 |
| Phi-4 Mini | 3.8B | optimal 9% | optimal 6% | optimal 6% | 178 BF16 | $2.33 |
| Qwen2.5-VL 3B | 3.8B | optimal 8% | optimal 6% | optimal 5% | 180 BF16 | $2.30 |
| Ministral-3 3B | 3.4B | optimal 8% | optimal 6% | optimal 6% | 187 BF16 | $2.21 |
| LLaMA 3.2 3B | 3.2B | optimal 8% | optimal 6% | optimal 6% | 192 BF16 | $2.16 |
| Falcon3 3B | 3.0B | optimal 8% | optimal 6% | optimal 5% | 198 BF16 | $2.10 |
| Qwen 2.5 3B | 3.0B | optimal 7% | optimal 5% | optimal 5% | 198 BF16 | $2.09 |
| Qwen 2.5 Coder 3B | 3.0B | optimal 7% | optimal 5% | optimal 5% | 198 BF16 | $2.09 |
| StarCoder2 3B | 3.0B | optimal 7% | optimal 5% | optimal 4% | 198 BF16 | $2.09 |
| Qwen3-VL 2B | 2.1B | optimal 7% | optimal 5% | optimal 5% | 223 BF16 | $1.85 |
| Granite 3.1 2B | 2.0B | optimal 6% | optimal 5% | optimal 4% | 228 BF16 | $1.81 |
| Qwen3 1.7B | 1.7B | optimal 6% | optimal 5% | optimal 5% | 239 BF16 | $1.73 |
| SmolLM2 1.7B | 1.7B | optimal 6% | optimal 5% | optimal 5% | 238 BF16 | $1.74 |
| StableLM 2 1.6B 4K ctx | 1.6B | optimal 6% | optimal 5% | optimal 4% | 242 BF16 | $1.71 |
| Qwen 2 1.5B | 1.5B | optimal 5% | optimal 4% | optimal 4% | 248 BF16 | $1.67 |
| Qwen 2.5 1.5B | 1.5B | optimal 5% | optimal 4% | optimal 4% | 248 BF16 | $1.67 |
| Qwen 2.5 Coder 1.5B | 1.5B | optimal 5% | optimal 4% | optimal 4% | 248 BF16 | $1.67 |
| LLaMA 3.2 1B | 1.2B | optimal 5% | optimal 4% | optimal 4% | 260 BF16 | $1.59 |
| Falcon3 1B 4K ctx | 1.0B | optimal 4% | optimal 4% | optimal 4% | 270 BF16 | $1.53 |
| Gemma 3 1B | 1.0B | optimal 4% | optimal 4% | optimal 4% | 271 BF16 | $1.53 |
| Qwen3 0.6B | 0.6B | optimal 4% | optimal 4% | optimal 4% | 291 BF16 | $1.42 |
| Qwen 2 0.5B | 0.5B | optimal 4% | optimal 3% | optimal 3% | 299 BF16 | $1.38 |
| Qwen 2.5 0.5B | 0.5B | optimal 4% | optimal 3% | optimal 3% | 299 BF16 | $1.38 |
| Qwen 2.5 Coder 0.5B | 0.5B | optimal 4% | optimal 3% | optimal 3% | 299 BF16 | $1.38 |
| Kimi K3 | 2800.0B (104.0B active) | — | — | 16× GPUs | 64 4-bit, 16× GPUs | $103.63 |
| DeepSeek V4-Pro | 1600.0B (49.0B active) | — | 16× GPUs | 8× GPUs | 70 FP8, 16× GPUs | $94.74 |
| Kimi K2 | 1000.0B (32.0B active) | 32× GPUs | 16× GPUs | 8× GPUs | 117 FP8, 16× GPUs | $56.50 |
| Kimi K2.5 | 1000.0B (32.0B active) | 32× GPUs | 16× GPUs | 8× GPUs | 117 FP8, 16× GPUs | $56.50 |
| GLM-5 744B | 744.0B (40.0B active) | 16× GPUs | 8× GPUs | 4× GPUs | 77 FP8, 8× GPUs | $42.95 |
| GLM-5.1 744B | 744.0B (40.0B active) | 16× GPUs | 8× GPUs | 4× GPUs | 77 FP8, 8× GPUs | $42.95 |
| DeepSeek R1 | 671.0B (37.0B active) | 16× GPUs | 8× GPUs | 4× GPUs | 80 FP8, 8× GPUs | $41.29 |
| DeepSeek R1-0528 | 671.0B (37.0B active) | 16× GPUs | 8× GPUs | 4× GPUs | 80 FP8, 8× GPUs | $41.29 |
| DeepSeek V3 | 671.0B (37.0B active) | 16× GPUs | 8× GPUs | 4× GPUs | 80 FP8, 8× GPUs | $41.29 |
| DeepSeek V3-0324 | 671.0B (37.0B active) | 16× GPUs | 8× GPUs | 4× GPUs | 80 FP8, 8× GPUs | $41.29 |
| DeepSeek V3.2 | 671.0B (37.0B active) | 16× GPUs | 8× GPUs | 4× GPUs | 80 FP8, 8× GPUs | $41.29 |
| MiniMax-M1 | 456.0B (45.9B active) | 16× GPUs | 8× GPUs | 4× GPUs | 72 FP8, 8× GPUs | $45.99 |
| MiniMax-Text-01 | 456.0B (45.9B active) | 16× GPUs | 8× GPUs | 4× GPUs | 72 FP8, 8× GPUs | $45.99 |
| LLaMA 3.1 405B | 405.0B | 8× GPUs | 4× GPUs | 2× GPUs | 19 FP8, 4× GPUs | $87.60 |
| Llama 4 Maverick (400B) | 400.0B (17.0B active) | 8× GPUs | 4× GPUs | 2× GPUs | 97 FP8, 4× GPUs | $17.09 |
| Qwen3.5 397B-A17B | 397.0B (17.0B active) | 8× GPUs | 4× GPUs | 2× GPUs | 145 FP8, 4× GPUs | $11.39 |
| Nemotron-4 340B 4K ctx | 340.0B | 8× GPUs | 4× GPUs | 2× GPUs | 22 FP8, 4× GPUs | $74.24 |
| MiMo V2-Flash | 309.8B (15.0B active) | 8× GPUs | 4× GPUs | 2× GPUs | 145 FP8, 4× GPUs | $11.44 |
| DeepSeek V4-Flash | 284.0B (13.0B active) | 8× GPUs | 4× GPUs | 2× GPUs | 142 FP8, 4× GPUs | $11.63 |
| Llama-3.3-Nemotron Ultra 253B | 253.0B | 4× GPUs | 4× GPUs | 2× GPUs | 29 FP8, 4× GPUs | $56.50 |
| Falcon 40B | 40.0B | — | — | — | — | — |
| Falcon 7B | 7.0B | — | — | — | — | — |
Frequently asked questions
What is the largest LLM that fits on a single GH200 NVL32?
At 8K context and batch 1, the largest models in our catalog that fit on a GH200 NVL32 (141 GB) are: Nemotron-H 56B Base (56B) at BF16; Devstral 2 123B (123B) at FP8; DeepSeek Coder V2 (236B) at 4-bit (AWQ). Larger models need tensor parallelism across several GPUs or lower-precision weights.
Can a GH200 NVL32 run LLaMA 3.3 70B?
at BF16: needs 2× GH200 NVL32; at FP8: fits on one GH200 NVL32 (55% of VRAM, ~37 tok/s); at 4-bit (AWQ): fits on one GH200 NVL32 (33% of VRAM, ~67 tok/s). 8K context, batch 1, vLLM — the calculator sizes other contexts and concurrency.
Can a GH200 NVL32 serve DeepSeek R1?
at BF16: needs 16× GH200 NVL32; at FP8: needs 8× GH200 NVL32; at 4-bit (AWQ): needs 4× GH200 NVL32. 8K context, batch 1, vLLM — the calculator sizes other contexts and concurrency.
How fast is a GH200 NVL32 for LLM inference?
Single-stream decode for LLaMA 3.1 8B at FP8 is about 174 tokens/s on one GH200 NVL32; decode is bound by memory bandwidth (4900 GB/s), so larger models run proportionally slower. Batching raises total throughput far above the single-stream figure.
How much does it cost to rent a GH200 NVL32?
For on-demand rental our catalog estimate is $1.49/hr. The $/1M tokens column above uses that price.
Methodology: every cell comes from the same engine as our GPU Lookup — weights, KV cache (including MLA latent and hybrid / sliding-window layers), activations and framework overhead per model architecture, decode speed from a calibrated memory-bandwidth roofline. Reference scenario: 8K context (4K for models with a shorter native window), batch 1, vLLM, fp16 KV cache. Fit bands: optimal 50–70% of VRAM, good 70–80%, tight 80–95%. Engineering estimates, not a guarantee — validate against your serving framework before buying hardware.