Fine-tuning calculator — full, LoRA, QLoRA

The fine-tuning calculator estimates training memory: what has to live on each GPU while a model is trained, which card holds it and how many of them. It covers the three methods most teams choose between — full fine-tuning, LoRA and QLoRA — and shows them side by side for your model, so the choice is a number rather than a rule of thumb.

What lives on the GPU

The result breaks the per-GPU memory into seven components:

• Base weights — the model at its base precision (BF16 or FP16 2 bytes per parameter, FP8 about 1), or NF4 for QLoRA.
• Master weights (fp32) — full fine-tuning only: an FP32 copy the optimizer updates, 4 bytes per parameter.
• Gradients — one per trainable parameter.
• Optimizer — optimizer state per trainable parameter: 8 bytes for AdamW.
• Adapter — LoRA and QLoRA only: the low-rank matrices being trained.
• Activations — what the forward pass stores for the backward pass; depends on sequence length, batch and the memory savers.
• Overhead — a flat 1.5 GB for the CUDA context and framework buffers.

Weights, gradients and optimizer states are exact byte counts. Activations are an estimate from the Megatron-LM formula; the real peak depends on the framework, FlashAttention, fused kernels and gradient-accumulation settings.

The three methods

The 8B figures are at rank 16, AdamW, batch 1 with gradient checkpointing and FlashAttention on; open the presets below to see them recomputed.

• Full fine-tuning carries 16 bytes per parameter before activations: BF16 weights, the FP32 master copy, BF16 gradients and two FP32 AdamW moments. It is the heaviest method and the one that needs ZeRO or FSDP sharding beyond small models.
• LoRA freezes the base and trains adapters on the attention and MLP projections (all-linear). Gradients and optimizer state exist only for the adapters, which PEFT keeps in FP32 — 16 bytes per adapter parameter — but adapters are a fraction of a percent of the model, so the frozen base dominates. Rank sets the adapter size; the calculator offers quick values and a free field.
• QLoRA stores the frozen base in NF4: 4.127 bits per parameter with double quantization (the bitsandbytes default), 4.5 bits without. Embeddings and the LM head stay in BF16, as bitsandbytes leaves them, which is why an 8B base is about 5.7 GB rather than 4.1. A 70B QLoRA run is about 49 GB per GPU at these settings — an 80 GB card, not a 48 GB one.

Memory savers

• Gradient checkpointing (on by default) stores only each layer's input and recomputes the rest in the backward pass — a large activation cut for about a third more compute, modeled as a lower effective MFU. On the 8B full run it takes activations from about 11 GB to about 2 GB.
• FlashAttention (on by default) removes the term that grows with the square of the sequence length.
• Fused cross-entropy (Liger, Unsloth; off by default) drops the logits activation — batch × sequence × vocabulary, upcast to FP32 for the loss. For models with a 128K+ vocabulary this is often the single largest activation.

The calculator warns when checkpointing is off at a sequence length of 8K or more.

Reading the result

The headline shows the per-GPU and all-GPU totals for the recommended GPU, with the breakdown bar. The GPU table lists up to six options — the card, how many, memory per GPU and its fit band, interconnect, $/hr and, if you set training tokens, time and cost to train. Options are ranked by how fully they use the card: closest to about 85 % first, then fewer GPUs, with cost only as the final tie-break. Live cards come before end-of-life ones. The "what if" panel compares the three methods, a full fine-tune with ZeRO-3 and other sequence lengths at your settings.

AMD GPUs are not recommended for training: the kernels the model assumes — bitsandbytes NF4, fused optimizers — are CUDA-first. For MoE models all experts are treated as trainable.