How much VRAM does CodeLlama 70B need?

About CodeLlama 70B

At 70B parameters, CodeLlama 70B sits firmly in the territory where quantization choice, not just raw parameter count, decides which GPU you end up buying or renting. In bf16 at a reference 8K context and batch size 1, the model needs 146.8GB of VRAM — computed the same way our interactive calculator computes it, from CodeLlama's published architecture rather than a generic per-parameter estimate — and the smallest configuration that clears that bar is 1× B200 SXM 180GB. Compare that to fp8, which roughly halves the weight footprint at essentially no measurable quality cost on most tasks, or AWQ and NVFP4, which push weight-only compression to 4 bits and can drop the requirement by roughly three-quarters relative to bf16 — often enough to move the model down an entire GPU tier or free up room for a much larger batch of concurrent requests. The full breakdown for all four dtypes is in the table above, alongside the exact GPU configuration each one unlocks. None of this accounts for context length, though: KV cache scales with how many tokens you keep resident, so a long-context deployment needs meaningfully more headroom than this 8K reference — the context-sensitivity table shows the same bf16 model at 32K and 128K wherever its native window supports it, so you can see the real cost of long-context serving before you commit to hardware.