Best LLMs to run on RTX PRO 6000 Blackwell Server Edition 96GB
datacenter 96 GB GDDR7 · 1597 GB/s · 500 TFLOPS BF16 · updated 2026-10-23
Headline answer
Of 141 models in our catalog, 106 fit on a single RTX PRO 6000 Blackwell Server Edition 96GB at FP8, 115 at 4-bit (AWQ) and 92 at BF16 (8K context, batch 1). Largest that fits: Command R 35B at BF16; Qwen3-Next 80B-A3B at FP8; Mixtral 8x22B at 4-bit (AWQ). Rental: market median $2.47/hr across 3 providers (from $1.80/hr).
Change context, concurrency or framework in the calculator → RTX PRO 6000 Blackwell Server Edition 96GB full specs Rental price history
Every model on RTX PRO 6000 Blackwell Server Edition 96GB
Fit band and share of VRAM per precision; N× GPUs = minimum count inside one node (tensor parallel); — = does not fit within one 8-GPU node. Speed and $/1M tokens for the highest-quality precision that fits on one GPU (BF16, else FP8, else 4-bit), single stream.
| Model | Params | BF16 | FP8 | 4-bit (AWQ) | tok/s | $ / 1M tok |
|---|---|---|---|---|---|---|
| Mixtral 8x22B | 141.0B (39.0B active) | 4× GPUs | 2× GPUs | tight 83% | 42 4-bit | $16.12 |
| DBRX 132B | 132.0B (36.0B active) | 4× GPUs | 2× GPUs | good 79% | 46 4-bit | $15.03 |
| Devstral 2 123B | 123.0B | 4× GPUs | 2× GPUs | good 79% | 15 4-bit | $46.30 |
| Mistral Large 2 123B | 123.0B | 4× GPUs | 2× GPUs | good 75% | 15 4-bit | $46.30 |
| Nemotron-3 Super 120B-A12B | 120.0B (12.0B active) | 4× GPUs | 2× GPUs | good 71% | 108 4-bit | $6.37 |
| Mistral Small 4 119B | 119.0B (11.0B active) | 4× GPUs | 2× GPUs | good 71% | 114 4-bit | $6.00 |
| GPT OSS 120B | 117.0B (5.1B active) | 4× GPUs | 2× GPUs | good 70% | 175 4-bit | $3.92 |
| Llama 4 Scout (109B) | 109.0B (17.0B active) | 4× GPUs | 2× GPUs | optimal 68% | 83 4-bit | $8.25 |
| Command R+ 104B | 104.0B | 4× GPUs | 2× GPUs | good 73% | 17 4-bit | $39.38 |
| Qwen3-Next 80B-A3B | 80.0B (3.0B active) | 2× GPUs | tight 90% | optimal 49% | 161 FP8 | $4.26 |
| Qwen 2 72B | 72.0B | 2× GPUs | tight 84% | optimal 50% | 13 FP8 | $53.54 |
| Qwen 2.5 72B | 72.0B | 2× GPUs | tight 84% | optimal 50% | 13 FP8 | $53.54 |
| CodeLlama 70B | 70.0B | 2× GPUs | tight 81% | optimal 46% | 13 FP8 | $52.31 |
| DeepSeek R1 Distill 70B | 70.0B | 2× GPUs | tight 81% | optimal 48% | 13 FP8 | $52.31 |
| LLaMA 2 70B 4K ctx | 70.0B | 2× GPUs | tight 80% | optimal 44% | 13 FP8 | $52.31 |
| LLaMA 3.1 70B | 70.0B | 2× GPUs | tight 81% | optimal 48% | 13 FP8 | $52.31 |
| LLaMA 3.3 70B | 70.0B | 2× GPUs | tight 81% | optimal 48% | 13 FP8 | $52.31 |
| Llama-3.1-Nemotron-70B | 70.0B | 2× GPUs | tight 81% | optimal 48% | 13 FP8 | $52.31 |
| Nemotron-H 56B Base | 56.0B | 2× GPUs | optimal 68% | optimal 42% | 16 FP8 | $42.30 |
| Llama-3.1-Nemotron-51B | 51.0B | 2× GPUs | optimal 60% | optimal 37% | 18 FP8 | $38.72 |
| Llama-3.3-Nemotron Super 49B | 49.0B | 2× GPUs | optimal 58% | optimal 36% | 18 FP8 | $37.24 |
| Kimi Linear 48B-A3B | 48.0B (3.0B active) | 2× GPUs | optimal 56% | optimal 32% | 162 FP8 | $4.22 |
| Mixtral 8x7B | 46.7B (12.9B active) | 2× GPUs | optimal 55% | optimal 31% | 60 FP8 | $11.35 |
| Command R 35B | 35.0B | tight 88% | optimal 53% | optimal 41% | 13 BF16 | $52.71 |
| Qwen3.6 35B-A3B | 35.0B (3.0B active) | good 78% | optimal 42% | optimal 26% | 107 BF16 | $6.39 |
| CodeLlama 34B | 34.0B | good 77% | optimal 42% | optimal 25% | 14 BF16 | $50.76 |
| Yi-1.5 34B 4K ctx | 34.0B | good 76% | optimal 41% | optimal 25% | 14 BF16 | $50.76 |
| DeepSeek R1 Distill 32B | 32.0B | good 73% | optimal 40% | optimal 26% | 14 BF16 | $47.92 |
| Nemotron-Terminal-32B | 32.0B | good 73% | optimal 40% | optimal 26% | 14 BF16 | $47.92 |
| OLMo-2 32B 4K ctx | 32.0B | good 72% | optimal 39% | optimal 24% | 14 BF16 | $47.92 |
| Qwen 2.5 32B | 32.0B | good 73% | optimal 40% | optimal 26% | 14 BF16 | $47.92 |
| Qwen 2.5 Coder 32B | 32.0B | good 73% | optimal 40% | optimal 26% | 14 BF16 | $47.92 |
| Qwen3 32B | 32.0B | good 73% | optimal 40% | optimal 26% | 14 BF16 | $47.92 |
| Nemotron-3 Nano 30B-A3B | 31.6B (3.2B active) | good 70% | optimal 38% | optimal 22% | 103 BF16 | $6.63 |
| Muse Glimmer 30B | 30.0B | optimal 67% | optimal 37% | optimal 24% | 15 BF16 | $45.08 |
| Qwen3 30B-A3B | 30.0B (3.0B active) | optimal 68% | optimal 37% | optimal 22% | 107 BF16 | $6.40 |
| Gemma 2 27B | 27.0B | optimal 64% | optimal 36% | optimal 26% | 17 BF16 | $40.79 |
| Gemma 3 27B | 27.0B | optimal 65% | optimal 37% | optimal 28% | 17 BF16 | $41.03 |
| Qwen3.6 27B | 27.0B | optimal 63% | optimal 35% | optimal 25% | 17 BF16 | $40.79 |
| Devstral 24B | 24.0B | optimal 56% | optimal 31% | optimal 21% | 19 BF16 | $36.45 |
| Devstral Small 2 24B | 24.0B | optimal 56% | optimal 31% | optimal 21% | 19 BF16 | $36.45 |
| Magistral Small 24B | 24.0B | optimal 56% | optimal 31% | optimal 21% | 19 BF16 | $36.45 |
| Mistral Small 3.1 24B | 24.0B | optimal 56% | optimal 31% | optimal 21% | 19 BF16 | $36.45 |
| Codestral 22B | 22.0B | optimal 52% | optimal 30% | optimal 19% | 20 BF16 | $33.59 |
| GPT OSS 20B | 21.0B (3.6B active) | optimal 48% | optimal 27% | optimal 18% | 95 BF16 | $7.23 |
| InternLM 2.5 20B | 20.0B | optimal 48% | optimal 27% | optimal 18% | 22 BF16 | $30.73 |
| DeepSeek Coder V2 Lite | 16.0B (2.4B active) | optimal 38% | optimal 22% | optimal 14% | 124 BF16 | $5.50 |
| StarCoder2 15B | 16.0B | optimal 38% | optimal 22% | optimal 14% | 28 BF16 | $24.92 |
| DeepSeek R1 Distill 14B | 14.8B | optimal 37% | optimal 22% | optimal 16% | 29 BF16 | $23.31 |
| Qwen 2.5 14B | 14.8B | optimal 37% | optimal 22% | optimal 16% | 29 BF16 | $23.31 |
| Qwen3 14B | 14.8B | optimal 36% | optimal 21% | optimal 16% | 29 BF16 | $23.31 |
| Phi 3 Medium | 14.0B | optimal 35% | optimal 21% | optimal 14% | 31 BF16 | $22.18 |
| Phi-4 14B | 14.0B | optimal 35% | optimal 21% | optimal 15% | 31 BF16 | $22.18 |
| Qwen 2.5 Coder 14B | 14.0B | optimal 35% | optimal 21% | optimal 16% | 31 BF16 | $22.18 |
| OLMo-2 13B 4K ctx | 13.7B | optimal 36% | optimal 22% | optimal 17% | 31 BF16 | $22.11 |
| LLaMA 2 13B 4K ctx | 13.0B | optimal 35% | optimal 22% | optimal 15% | 32 BF16 | $21.09 |
| Gemma 3 12B | 12.0B | optimal 33% | optimal 20% | optimal 17% | 35 BF16 | $19.41 |
| Mistral Nemo 12B | 12.0B | optimal 31% | optimal 18% | optimal 14% | 36 BF16 | $19.30 |
| Nemotron-Nano-12B-v2 | 12.0B | optimal 31% | optimal 19% | optimal 15% | 35 BF16 | $19.36 |
| StableLM 2 12B 4K ctx | 12.0B | optimal 30% | optimal 18% | optimal 13% | 35 BF16 | $19.36 |
| Falcon3 10B | 10.0B | optimal 26% | optimal 16% | optimal 12% | 42 BF16 | $16.43 |
| GLM-4 9B | 9.0B | optimal 23% | optimal 14% | optimal 11% | 46 BF16 | $14.96 |
| Gemma 2 9B | 9.0B | optimal 26% | optimal 17% | optimal 15% | 45 BF16 | $15.13 |
| Nemotron-Nano-9B-v2 | 9.0B | optimal 25% | optimal 16% | optimal 13% | 46 BF16 | $15.06 |
| Yi-1.5 9B 4K ctx | 9.0B | optimal 23% | optimal 14% | optimal 10% | 46 BF16 | $15.00 |
| InternLM3 8B | 8.8B | optimal 23% | optimal 14% | optimal 11% | 47 BF16 | $14.67 |
| Qwen3-VL 8B | 8.8B | optimal 24% | optimal 15% | optimal 12% | 47 BF16 | $14.67 |
| MiniCPM-V 4.5 | 8.7B | optimal 24% | optimal 15% | optimal 12% | 47 BF16 | $14.58 |
| Ministral-3 8B | 8.5B | optimal 23% | optimal 14% | optimal 12% | 48 BF16 | $14.28 |
| Qwen2.5-VL 7B | 8.3B | optimal 22% | optimal 14% | optimal 11% | 49 BF16 | $13.96 |
| DeepSeek R1 Distill 8B | 8.0B | optimal 22% | optimal 14% | optimal 11% | 50 BF16 | $13.57 |
| Granite 3.3 8B | 8.0B | optimal 22% | optimal 14% | optimal 11% | 50 BF16 | $13.60 |
| LLaMA 3.1 8B | 8.0B | optimal 22% | optimal 14% | optimal 11% | 50 BF16 | $13.57 |
| Ministral 8B | 8.0B | optimal 22% | optimal 14% | optimal 12% | 50 BF16 | $13.57 |
| Qwen3 8B | 8.0B | optimal 22% | optimal 14% | optimal 12% | 50 BF16 | $13.57 |
| Qwen 2.5 7B | 7.6B | optimal 21% | optimal 13% | optimal 10% | 53 BF16 | $12.98 |
| Falcon3 7B | 7.5B | optimal 21% | optimal 13% | optimal 10% | 54 BF16 | $12.79 |
| Phi-3 Small 7B | 7.4B | optimal 21% | optimal 13% | optimal 11% | 54 BF16 | $12.69 |
| InternLM 2.5 7B | 7.0B | optimal 20% | optimal 13% | optimal 10% | 56 BF16 | $12.15 |
| LLaMA 2 7B 4K ctx | 7.0B | optimal 21% | optimal 14% | optimal 11% | 56 BF16 | $12.35 |
| Mistral 7B | 7.0B | optimal 20% | optimal 13% | optimal 10% | 56 BF16 | $12.15 |
| OLMo-2 7B 4K ctx | 7.0B | optimal 21% | optimal 14% | optimal 12% | 56 BF16 | $12.35 |
| Qwen 2 7B | 7.0B | optimal 19% | optimal 12% | optimal 10% | 57 BF16 | $12.11 |
| Qwen 2.5 Coder 7B | 7.0B | optimal 19% | optimal 12% | optimal 10% | 57 BF16 | $12.11 |
| StarCoder2 7B | 7.0B | optimal 19% | optimal 12% | optimal 9% | 57 BF16 | $12.11 |
| Yi-1.5 6B 4K ctx | 6.0B | optimal 17% | optimal 11% | optimal 9% | 64 BF16 | $10.67 |
| Qwen3-VL 4B | 4.4B | optimal 15% | optimal 10% | optimal 9% | 81 BF16 | $8.49 |
| Gemma 3 4B | 4.3B | optimal 14% | optimal 10% | optimal 10% | 83 BF16 | $8.29 |
| Nemotron-Mini-4B 4K ctx | 4.0B | optimal 13% | optimal 9% | optimal 9% | 87 BF16 | $7.85 |
| Qwen3 4B | 4.0B | optimal 14% | optimal 10% | optimal 9% | 87 BF16 | $7.86 |
| Ministral 3B | 3.8B | optimal 13% | optimal 9% | optimal 8% | 91 BF16 | $7.54 |
| Phi 3.5 Mini | 3.8B | optimal 16% | optimal 12% | optimal 10% | 89 BF16 | $7.71 |
| Phi-4 Mini | 3.8B | optimal 13% | optimal 9% | optimal 9% | 91 BF16 | $7.56 |
| Qwen2.5-VL 3B | 3.8B | optimal 12% | optimal 8% | optimal 8% | 92 BF16 | $7.44 |
| Ministral-3 3B | 3.4B | optimal 12% | optimal 9% | optimal 8% | 98 BF16 | $6.98 |
| LLaMA 3.2 3B | 3.2B | optimal 12% | optimal 9% | optimal 8% | 102 BF16 | $6.71 |
| Falcon3 3B | 3.0B | optimal 11% | optimal 8% | optimal 8% | 107 BF16 | $6.40 |
| Qwen 2.5 3B | 3.0B | optimal 11% | optimal 8% | optimal 7% | 108 BF16 | $6.37 |
| Qwen 2.5 Coder 3B | 3.0B | optimal 11% | optimal 8% | optimal 7% | 108 BF16 | $6.37 |
| StarCoder2 3B | 3.0B | optimal 11% | optimal 8% | optimal 7% | 108 BF16 | $6.36 |
| Qwen3-VL 2B | 2.1B | optimal 10% | optimal 8% | optimal 7% | 133 BF16 | $5.16 |
| Granite 3.1 2B | 2.0B | optimal 9% | optimal 7% | optimal 6% | 138 BF16 | $4.96 |
| Qwen3 1.7B | 1.7B | optimal 9% | optimal 7% | optimal 7% | 151 BF16 | $4.55 |
| SmolLM2 1.7B | 1.7B | optimal 10% | optimal 8% | optimal 7% | 149 BF16 | $4.60 |
| StableLM 2 1.6B 4K ctx | 1.6B | optimal 8% | optimal 7% | optimal 7% | 154 BF16 | $4.45 |
| Qwen 2 1.5B | 1.5B | optimal 8% | optimal 6% | optimal 6% | 162 BF16 | $4.22 |
| Qwen 2.5 1.5B | 1.5B | optimal 8% | optimal 6% | optimal 6% | 162 BF16 | $4.22 |
| Qwen 2.5 Coder 1.5B | 1.5B | optimal 8% | optimal 6% | optimal 6% | 162 BF16 | $4.22 |
| LLaMA 3.2 1B | 1.2B | optimal 7% | optimal 6% | optimal 6% | 178 BF16 | $3.85 |
| Falcon3 1B 4K ctx | 1.0B | optimal 7% | optimal 6% | optimal 6% | 194 BF16 | $3.53 |
| Gemma 3 1B | 1.0B | optimal 7% | optimal 6% | optimal 6% | 196 BF16 | $3.50 |
| Qwen3 0.6B | 0.6B | optimal 6% | optimal 6% | optimal 6% | 230 BF16 | $2.98 |
| Qwen 2 0.5B | 0.5B | optimal 5% | optimal 5% | optimal 5% | 247 BF16 | $2.78 |
| Qwen 2.5 0.5B | 0.5B | optimal 5% | optimal 5% | optimal 5% | 247 BF16 | $2.78 |
| Qwen 2.5 Coder 0.5B | 0.5B | optimal 5% | optimal 5% | optimal 5% | 247 BF16 | $2.78 |
| MiniMax-M1 | 456.0B (45.9B active) | — | — | 4× GPUs | 33 4-bit, 4× GPUs | $84.08 |
| MiniMax-Text-01 | 456.0B (45.9B active) | — | — | 4× GPUs | 33 4-bit, 4× GPUs | $84.08 |
| LLaMA 3.1 405B | 405.0B | — | 8× GPUs | 4× GPUs | 4 FP8, 8× GPUs | $1218.27 |
| Llama 4 Maverick (400B) | 400.0B (17.0B active) | — | — | 4× GPUs | 52 4-bit, 4× GPUs | $52.31 |
| Qwen3.5 397B-A17B | 397.0B (17.0B active) | — | — | 4× GPUs | 87 4-bit, 4× GPUs | $31.58 |
| Nemotron-4 340B 4K ctx | 340.0B | 8× GPUs | 8× GPUs | 4× GPUs | 5 FP8, 8× GPUs | $1034.38 |
| MiMo V2-Flash | 309.8B (15.0B active) | — | 4× GPUs | 2× GPUs | 50 FP8, 4× GPUs | $55.15 |
| DeepSeek V4-Flash | 284.0B (13.0B active) | — | 4× GPUs | 2× GPUs | 48 FP8, 4× GPUs | $56.63 |
| Llama-3.3-Nemotron Ultra 253B | 253.0B | 8× GPUs | 4× GPUs | 2× GPUs | 7 FP8, 4× GPUs | $380.71 |
| DeepSeek Coder V2 | 236.0B (21.0B active) | — | 4× GPUs | 2× GPUs | 29 FP8, 4× GPUs | $95.84 |
| Qwen3 235B-A22B | 235.0B (22.0B active) | — | 4× GPUs | 2× GPUs | 36 FP8, 4× GPUs | $75.93 |
| MiniMax M2.5 | 230.0B (10.0B active) | — | 4× GPUs | 2× GPUs | 75 FP8, 4× GPUs | $36.35 |
| MiniMax M2.7 | 230.0B (10.0B active) | — | 4× GPUs | 2× GPUs | 75 FP8, 4× GPUs | $36.35 |
| Kimi K3 | 2800.0B (104.0B active) | — | — | — | — | — |
| DeepSeek V4-Pro | 1600.0B (49.0B active) | — | — | — | — | — |
| Kimi K2 | 1000.0B (32.0B active) | — | — | — | — | — |
| Kimi K2.5 | 1000.0B (32.0B active) | — | — | — | — | — |
| GLM-5 744B | 744.0B (40.0B active) | — | — | — | — | — |
| GLM-5.1 744B | 744.0B (40.0B active) | — | — | — | — | — |
| DeepSeek R1 | 671.0B (37.0B active) | — | — | — | — | — |
| DeepSeek R1-0528 | 671.0B (37.0B active) | — | — | — | — | — |
| DeepSeek V3 | 671.0B (37.0B active) | — | — | — | — | — |
| DeepSeek V3-0324 | 671.0B (37.0B active) | — | — | — | — | — |
| DeepSeek V3.2 | 671.0B (37.0B active) | — | — | — | — | — |
| Falcon 40B | 40.0B | — | — | — | — | — |
| Falcon 7B | 7.0B | — | — | — | — | — |
Compare RTX PRO 6000 Blackwell Server Edition 96GB
Frequently asked questions
What is the largest LLM that fits on a single RTX PRO 6000 Blackwell Server Edition 96GB?
At 8K context and batch 1, the largest models in our catalog that fit on an RTX PRO 6000 Blackwell Server Edition 96GB (96 GB) are: Command R 35B (35B) at BF16; Qwen3-Next 80B-A3B (80B) at FP8; Mixtral 8x22B (141B) at 4-bit (AWQ). Larger models need tensor parallelism across several GPUs or lower-precision weights.
Can an RTX PRO 6000 Blackwell Server Edition 96GB run LLaMA 3.3 70B?
at BF16: needs 2× RTX PRO 6000 Blackwell Server Edition 96GB; at FP8: fits on one RTX PRO 6000 Blackwell Server Edition 96GB (81% of VRAM, ~13 tok/s); at 4-bit (AWQ): fits on one RTX PRO 6000 Blackwell Server Edition 96GB (48% of VRAM, ~25 tok/s). 8K context, batch 1, vLLM — the calculator sizes other contexts and concurrency.
Can an RTX PRO 6000 Blackwell Server Edition 96GB serve DeepSeek R1?
at BF16: does not fit within one 8-GPU node; at FP8: does not fit within one 8-GPU node; at 4-bit (AWQ): does not fit within one 8-GPU node. 8K context, batch 1, vLLM — the calculator sizes other contexts and concurrency.
How fast is an RTX PRO 6000 Blackwell Server Edition 96GB for LLM inference?
Single-stream decode for LLaMA 3.1 8B at FP8 is about 87 tokens/s on one RTX PRO 6000 Blackwell Server Edition 96GB; decode is bound by memory bandwidth (1597 GB/s), so larger models run proportionally slower. Batching raises total throughput far above the single-stream figure.
How much does it cost to rent an RTX PRO 6000 Blackwell Server Edition 96GB?
For on-demand rental the market median is $2.47/hr across 3 providers (lowest $1.80/hr). The $/1M tokens column above uses that price.
Methodology: every cell comes from the same engine as our GPU Lookup — weights, KV cache (including MLA latent and hybrid / sliding-window layers), activations and framework overhead per model architecture, decode speed from a calibrated memory-bandwidth roofline. Reference scenario: 8K context (4K for models with a shorter native window), batch 1, vLLM, fp16 KV cache. Fit bands: optimal 50–70% of VRAM, good 70–80%, tight 80–95%. Engineering estimates, not a guarantee — validate against your serving framework before buying hardware.