howmanygpus.ai

Best LLMs to run on AMD MI325X 256GB

datacenter 256 GB HBM3e · 6000 GB/s · 1307 TFLOPS BF16 · updated 2026-10-23

Headline answer

Of 141 models in our catalog, 117 fit on a single AMD MI325X 256GB at FP8, 126 at 4-bit (AWQ) and 110 at BF16 (8K context, batch 1). Largest that fits: Mistral Small 4 119B at BF16; MiniMax M2.5 at FP8; LLaMA 3.1 405B at 4-bit (AWQ). Rental: $3.80/hr catalog estimate.

Change context, concurrency or framework in the calculator → AMD MI325X 256GB full specs

Every model on AMD MI325X 256GB

Fit band and share of VRAM per precision; N× GPUs = minimum count inside one node (tensor parallel); — = does not fit within one 8-GPU node. Speed and $/1M tokens for the highest-quality precision that fits on one GPU (BF16, else FP8, else 4-bit), single stream.

Model Params BF16FP84-bit (AWQ) tok/s $ / 1M tok
LLaMA 3.1 405B 405.0B 4× GPUs2× GPUstight 88% 17 4-bit $69.57
Llama 4 Maverick (400B) 400.0B (17.0B active) 4× GPUs2× GPUstight 85% 185 4-bit $6.31
Qwen3.5 397B-A17B 397.0B (17.0B active) 4× GPUs2× GPUstight 84% 186 4-bit $6.30
Nemotron-4 340B 4K ctx 340.0B 4× GPUs2× GPUsgood 77% 20 4-bit $58.73
MiMo V2-Flash 309.8B (15.0B active) 4× GPUs2× GPUsoptimal 66% 196 4-bit $5.98
DeepSeek V4-Flash 284.0B (13.0B active) 4× GPUs2× GPUsoptimal 60% 207 4-bit $5.64
Llama-3.3-Nemotron Ultra 253B 253.0B 4× GPUs2× GPUsoptimal 57% 26 4-bit $44.78
DeepSeek Coder V2 236.0B (21.0B active) 4× GPUs2× GPUsoptimal 50% 169 4-bit $6.93
Qwen3 235B-A22B 235.0B (22.0B active) 4× GPUs2× GPUsoptimal 51% 164 4-bit $7.13
MiniMax M2.5 230.0B (10.0B active) 2× GPUstight 94%optimal 50% 172 FP8 $6.81
MiniMax M2.7 230.0B (10.0B active) 2× GPUstight 94%optimal 50% 172 FP8 $6.81
Mixtral 8x22B 141.0B (39.0B active) 2× GPUsoptimal 58%optimal 31% 72 FP8 $16.23
DBRX 132B 132.0B (36.0B active) 2× GPUsoptimal 55%optimal 30% 77 FP8 $15.24
Devstral 2 123B 123.0B 2× GPUsoptimal 52%optimal 30% 27 FP8 $43.45
Mistral Large 2 123B 123.0B 2× GPUsoptimal 52%optimal 28% 27 FP8 $43.45
Nemotron-3 Super 120B-A12B 120.0B (12.0B active) 2× GPUsoptimal 50%optimal 27% 158 FP8 $7.42
Mistral Small 4 119B 119.0B (11.0B active) tight 95%optimal 49%optimal 26% 110 BF16 $10.65
GPT OSS 120B 117.0B (5.1B active) tight 93%optimal 48%optimal 26% 171 BF16 $6.83
Llama 4 Scout (109B) 109.0B (17.0B active) tight 87%optimal 46%optimal 26% 80 BF16 $14.59
Command R+ 104B 104.0B tight 84%optimal 44%optimal 27% 16 BF16 $71.27
Qwen3-Next 80B-A3B 80.0B (3.0B active) optimal 64%optimal 34%optimal 18% 213 BF16 $5.48
Qwen 2 72B 72.0B optimal 59%optimal 31%optimal 19% 23 BF16 $50.38
Qwen 2.5 72B 72.0B optimal 59%optimal 31%optimal 19% 23 BF16 $50.38
CodeLlama 70B 70.0B optimal 57%optimal 30%optimal 17% 24 BF16 $49.11
DeepSeek R1 Distill 70B 70.0B optimal 57%optimal 30%optimal 18% 24 BF16 $49.11
LLaMA 2 70B 4K ctx 70.0B optimal 57%optimal 30%optimal 17% 24 BF16 $49.11
LLaMA 3.1 70B 70.0B optimal 57%optimal 30%optimal 18% 24 BF16 $49.11
LLaMA 3.3 70B 70.0B optimal 57%optimal 30%optimal 18% 24 BF16 $49.11
Llama-3.1-Nemotron-70B 70.0B optimal 57%optimal 30%optimal 18% 24 BF16 $49.11
Nemotron-H 56B Base 56.0B optimal 47%optimal 26%optimal 16% 29 BF16 $40.03
Llama-3.1-Nemotron-51B 51.0B optimal 42%optimal 23%optimal 14% 32 BF16 $36.64
Llama-3.3-Nemotron Super 49B 49.0B optimal 40%optimal 22%optimal 13% 33 BF16 $35.42
Kimi Linear 48B-A3B 48.0B (3.0B active) optimal 39%optimal 21%optimal 12% 214 BF16 $5.46
Mixtral 8x7B 46.7B (12.9B active) optimal 38%optimal 21%optimal 12% 98 BF16 $11.91
Command R 35B 35.0B optimal 33%optimal 20%optimal 15% 44 BF16 $26.56
Qwen3.6 35B-A3B 35.0B (3.0B active) optimal 29%optimal 16%optimal 10% 214 BF16 $5.47
CodeLlama 34B 34.0B optimal 29%optimal 16%optimal 9% 46 BF16 $25.63
Yi-1.5 34B 4K ctx 34.0B optimal 29%optimal 16%optimal 9% 46 BF16 $25.63
DeepSeek R1 Distill 32B 32.0B optimal 27%optimal 15%optimal 10% 48 BF16 $24.35
Nemotron-Terminal-32B 32.0B optimal 27%optimal 15%optimal 10% 48 BF16 $24.35
OLMo-2 32B 4K ctx 32.0B optimal 27%optimal 15%optimal 9% 48 BF16 $24.35
Qwen 2.5 32B 32.0B optimal 27%optimal 15%optimal 10% 48 BF16 $24.35
Qwen 2.5 Coder 32B 32.0B optimal 27%optimal 15%optimal 10% 48 BF16 $24.35
Qwen3 32B 32.0B optimal 27%optimal 15%optimal 10% 48 BF16 $24.35
Nemotron-3 Nano 30B-A3B 31.6B (3.2B active) optimal 26%optimal 14%optimal 8% 209 BF16 $5.59
Muse Glimmer 30B 30.0B optimal 25%optimal 14%optimal 9% 51 BF16 $23.01
Qwen3 30B-A3B 30.0B (3.0B active) optimal 25%optimal 14%optimal 8% 213 BF16 $5.48
Gemma 2 27B 27.0B optimal 24%optimal 14%optimal 10% 55 BF16 $21.13
Gemma 3 27B 27.0B optimal 24%optimal 14%optimal 10% 55 BF16 $21.17
Qwen3.6 27B 27.0B optimal 24%optimal 13%optimal 9% 55 BF16 $21.10
Devstral 24B 24.0B optimal 21%optimal 12%optimal 8% 61 BF16 $19.13
Devstral Small 2 24B 24.0B optimal 21%optimal 12%optimal 8% 61 BF16 $19.13
Magistral Small 24B 24.0B optimal 21%optimal 12%optimal 8% 61 BF16 $19.13
Mistral Small 3.1 24B 24.0B optimal 21%optimal 12%optimal 8% 61 BF16 $19.13
Codestral 22B 22.0B optimal 20%optimal 11%optimal 7% 66 BF16 $17.84
GPT OSS 20B 21.0B (3.6B active) optimal 18%optimal 10%optimal 7% 200 BF16 $5.86
InternLM 2.5 20B 20.0B optimal 18%optimal 10%optimal 7% 71 BF16 $16.53
DeepSeek Coder V2 Lite 16.0B (2.4B active) optimal 14%optimal 8%optimal 5% 230 BF16 $5.07
StarCoder2 15B 16.0B optimal 14%optimal 8%optimal 5% 84 BF16 $13.90
DeepSeek R1 Distill 14B 14.8B optimal 14%optimal 8%optimal 6% 89 BF16 $13.15
Qwen 2.5 14B 14.8B optimal 14%optimal 8%optimal 6% 89 BF16 $13.15
Qwen3 14B 14.8B optimal 14%optimal 8%optimal 6% 89 BF16 $13.13
Phi 3 Medium 14.0B optimal 13%optimal 8%optimal 5% 92 BF16 $12.65
Phi-4 14B 14.0B optimal 13%optimal 8%optimal 6% 92 BF16 $12.65
Qwen 2.5 Coder 14B 14.0B optimal 13%optimal 8%optimal 6% 92 BF16 $12.65
OLMo-2 13B 4K ctx 13.7B optimal 14%optimal 8%optimal 6% 93 BF16 $12.62
LLaMA 2 13B 4K ctx 13.0B optimal 13%optimal 8%optimal 6% 96 BF16 $12.15
Gemma 3 12B 12.0B optimal 12%optimal 8%optimal 6% 103 BF16 $11.39
Mistral Nemo 12B 12.0B optimal 12%optimal 7%optimal 5% 103 BF16 $11.34
Nemotron-Nano-12B-v2 12.0B optimal 12%optimal 7%optimal 6% 103 BF16 $11.36
StableLM 2 12B 4K ctx 12.0B optimal 11%optimal 7%optimal 5% 103 BF16 $11.35
Falcon3 10B 10.0B optimal 10%optimal 6%optimal 5% 116 BF16 $10.04
GLM-4 9B 9.0B optimal 9%optimal 5%optimal 4% 125 BF16 $9.36
Gemma 2 9B 9.0B optimal 10%optimal 6%optimal 6% 124 BF16 $9.43
Nemotron-Nano-9B-v2 9.0B optimal 9%optimal 6%optimal 5% 124 BF16 $9.41
Yi-1.5 9B 4K ctx 9.0B optimal 9%optimal 5%optimal 4% 125 BF16 $9.37
InternLM3 8B 8.8B optimal 9%optimal 5%optimal 4% 127 BF16 $9.23
Qwen3-VL 8B 8.8B optimal 9%optimal 6%optimal 5% 126 BF16 $9.24
MiniCPM-V 4.5 8.7B optimal 9%optimal 6%optimal 5% 127 BF16 $9.20
Ministral-3 8B 8.5B optimal 9%optimal 5%optimal 4% 129 BF16 $9.06
Qwen2.5-VL 7B 8.3B optimal 8%optimal 5%optimal 4% 131 BF16 $8.91
DeepSeek R1 Distill 8B 8.0B optimal 8%optimal 5%optimal 4% 134 BF16 $8.74
Granite 3.3 8B 8.0B optimal 8%optimal 5%optimal 4% 134 BF16 $8.74
LLaMA 3.1 8B 8.0B optimal 8%optimal 5%optimal 4% 134 BF16 $8.74
Ministral 8B 8.0B optimal 8%optimal 5%optimal 4% 134 BF16 $8.74
Qwen3 8B 8.0B optimal 8%optimal 5%optimal 4% 134 BF16 $8.74
Qwen 2.5 7B 7.6B optimal 8%optimal 5%optimal 4% 138 BF16 $8.47
Falcon3 7B 7.5B optimal 8%optimal 5%optimal 4% 140 BF16 $8.38
Phi-3 Small 7B 7.4B optimal 8%optimal 5%optimal 4% 140 BF16 $8.34
InternLM 2.5 7B 7.0B optimal 8%optimal 5%optimal 4% 145 BF16 $8.08
LLaMA 2 7B 4K ctx 7.0B optimal 8%optimal 5%optimal 4% 143 BF16 $8.18
Mistral 7B 7.0B optimal 8%optimal 5%optimal 4% 145 BF16 $8.08
OLMo-2 7B 4K ctx 7.0B optimal 8%optimal 5%optimal 4% 143 BF16 $8.18
Qwen 2 7B 7.0B optimal 7%optimal 5%optimal 4% 145 BF16 $8.07
Qwen 2.5 Coder 7B 7.0B optimal 7%optimal 5%optimal 4% 145 BF16 $8.07
StarCoder2 7B 7.0B optimal 7%optimal 5%optimal 4% 145 BF16 $8.07
Yi-1.5 6B 4K ctx 6.0B optimal 6%optimal 4%optimal 3% 158 BF16 $7.42
Qwen3-VL 4B 4.4B optimal 6%optimal 4%optimal 3% 182 BF16 $6.43
Gemma 3 4B 4.3B optimal 5%optimal 4%optimal 4% 184 BF16 $6.33
Nemotron-Mini-4B 4K ctx 4.0B optimal 5%optimal 3%optimal 4% 190 BF16 $6.14
Qwen3 4B 4.0B optimal 5%optimal 4%optimal 3% 190 BF16 $6.14
Ministral 3B 3.8B optimal 5%optimal 3%optimal 3% 195 BF16 $5.99
Phi 3.5 Mini 3.8B optimal 6%optimal 4%optimal 4% 192 BF16 $6.07
Phi-4 Mini 3.8B optimal 5%optimal 4%optimal 4% 195 BF16 $6.01
Qwen2.5-VL 3B 3.8B optimal 5%optimal 3%optimal 3% 196 BF16 $5.95
Ministral-3 3B 3.4B optimal 5%optimal 3%optimal 3% 204 BF16 $5.74
LLaMA 3.2 3B 3.2B optimal 4%optimal 3%optimal 3% 208 BF16 $5.62
Falcon3 3B 3.0B optimal 4%optimal 3%optimal 3% 213 BF16 $5.48
Qwen 2.5 3B 3.0B optimal 4%optimal 3%optimal 3% 214 BF16 $5.46
Qwen 2.5 Coder 3B 3.0B optimal 4%optimal 3%optimal 3% 214 BF16 $5.46
StarCoder2 3B 3.0B optimal 4%optimal 3%optimal 2% 214 BF16 $5.46
Qwen3-VL 2B 2.1B optimal 4%optimal 3%optimal 3% 238 BF16 $4.92
Granite 3.1 2B 2.0B optimal 3%optimal 3%optimal 2% 242 BF16 $4.83
Qwen3 1.7B 1.7B optimal 3%optimal 3%optimal 3% 252 BF16 $4.64
SmolLM2 1.7B 1.7B optimal 4%optimal 3%optimal 3% 251 BF16 $4.66
StableLM 2 1.6B 4K ctx 1.6B optimal 3%optimal 2%optimal 2% 254 BF16 $4.59
Qwen 2 1.5B 1.5B optimal 3%optimal 2%optimal 2% 260 BF16 $4.49
Qwen 2.5 1.5B 1.5B optimal 3%optimal 2%optimal 2% 260 BF16 $4.49
Qwen 2.5 Coder 1.5B 1.5B optimal 3%optimal 2%optimal 2% 260 BF16 $4.49
LLaMA 3.2 1B 1.2B optimal 3%optimal 2%optimal 2% 271 BF16 $4.32
Falcon3 1B 4K ctx 1.0B optimal 2%optimal 2%optimal 2% 280 BF16 $4.17
Gemma 3 1B 1.0B optimal 2%optimal 2%optimal 2% 281 BF16 $4.16
Qwen3 0.6B 0.6B optimal 2%optimal 2%optimal 2% 298 BF16 $3.92
Qwen 2 0.5B 0.5B optimal 2%optimal 2%optimal 2% 305 BF16 $3.83
Qwen 2.5 0.5B 0.5B optimal 2%optimal 2%optimal 2% 305 BF16 $3.83
Qwen 2.5 Coder 0.5B 0.5B optimal 2%optimal 2%optimal 2% 305 BF16 $3.83
Kimi K3 2800.0B (104.0B active) ——8× GPUs 65 4-bit, 8× GPUs $143.63
DeepSeek V4-Pro 1600.0B (49.0B active) —8× GPUs4× GPUs 71 FP8, 8× GPUs $131.88
Kimi K2 1000.0B (32.0B active) —8× GPUs4× GPUs 116 FP8, 8× GPUs $80.95
Kimi K2.5 1000.0B (32.0B active) —8× GPUs4× GPUs 116 FP8, 8× GPUs $80.95
GLM-5 744B 744.0B (40.0B active) 8× GPUs4× GPUs2× GPUs 81 FP8, 4× GPUs $57.50
GLM-5.1 744B 744.0B (40.0B active) 8× GPUs4× GPUs2× GPUs 81 FP8, 4× GPUs $57.50
DeepSeek R1 671.0B (37.0B active) 8× GPUs4× GPUs2× GPUs 85 FP8, 4× GPUs $55.00
DeepSeek R1-0528 671.0B (37.0B active) 8× GPUs4× GPUs2× GPUs 85 FP8, 4× GPUs $55.00
DeepSeek V3 671.0B (37.0B active) 8× GPUs4× GPUs2× GPUs 85 FP8, 4× GPUs $55.00
DeepSeek V3-0324 671.0B (37.0B active) 8× GPUs4× GPUs2× GPUs 85 FP8, 4× GPUs $55.00
DeepSeek V3.2 671.0B (37.0B active) 8× GPUs4× GPUs2× GPUs 85 FP8, 4× GPUs $55.00
MiniMax-M1 456.0B (45.9B active) 8× GPUs4× GPUs2× GPUs 75 FP8, 4× GPUs $62.17
MiniMax-Text-01 456.0B (45.9B active) 8× GPUs4× GPUs2× GPUs 75 FP8, 4× GPUs $62.17
Falcon 40B 40.0B ——— — —
Falcon 7B 7.0B ——— — —

Frequently asked questions

What is the largest LLM that fits on a single AMD MI325X 256GB?

At 8K context and batch 1, the largest models in our catalog that fit on an AMD MI325X 256GB (256 GB) are: Mistral Small 4 119B (119B) at BF16; MiniMax M2.5 (230B) at FP8; LLaMA 3.1 405B (405B) at 4-bit (AWQ). Larger models need tensor parallelism across several GPUs or lower-precision weights.

Can an AMD MI325X 256GB run LLaMA 3.3 70B?

at BF16: fits on one AMD MI325X 256GB (57% of VRAM, ~24 tok/s); at FP8: fits on one AMD MI325X 256GB (30% of VRAM, ~44 tok/s); at 4-bit (AWQ): fits on one AMD MI325X 256GB (18% of VRAM, ~78 tok/s). 8K context, batch 1, vLLM — the calculator sizes other contexts and concurrency.

Can an AMD MI325X 256GB serve DeepSeek R1?

at BF16: needs 8× AMD MI325X 256GB; at FP8: needs 4× AMD MI325X 256GB; at 4-bit (AWQ): needs 2× AMD MI325X 256GB. 8K context, batch 1, vLLM — the calculator sizes other contexts and concurrency.

How fast is an AMD MI325X 256GB for LLM inference?

Single-stream decode for LLaMA 3.1 8B at FP8 is about 190 tokens/s on one AMD MI325X 256GB; decode is bound by memory bandwidth (6000 GB/s), so larger models run proportionally slower. Batching raises total throughput far above the single-stream figure.

How much does it cost to rent an AMD MI325X 256GB?

For on-demand rental our catalog estimate is $3.80/hr. The $/1M tokens column above uses that price.

Methodology: every cell comes from the same engine as our GPU Lookup — weights, KV cache (including MLA latent and hybrid / sliding-window layers), activations and framework overhead per model architecture, decode speed from a calibrated memory-bandwidth roofline. Reference scenario: 8K context (4K for models with a shorter native window), batch 1, vLLM, fp16 KV cache. Fit bands: optimal 50–70% of VRAM, good 70–80%, tight 80–95%. Engineering estimates, not a guarantee — validate against your serving framework before buying hardware.