What is not modeled
The effects the calculators leave out, for inference and fine-tuning.
On this page · 4 sections
An estimate is only as useful as its stated limits. These are the effects the calculators leave out or approximate today, grouped by tool. Where a gap matters for your decision, measure it on your own stack before you commit.
Inference
- Queueing and tail latency. TTFT is the prefill time of one request on an idle replica. Waiting behind other prompts in a burst, and p99 latency under load, need a queueing model the calculator does not have; the result notes how far a simultaneous burst could stretch it.
- Disaggregated prefill and decode. Dynamo and similar setups run prefill and decode on separate workers; every estimate here is aggregated serving.
- Prefix-cache reuse in the calculator. The calculator prefills every prompt in full. Reuse of a shared system prompt or of conversation history is modeled only in TCO — the shared prefix and the agentic cache hit rate.
- Speculative decoding by implementation. The speed-up follows a generic formula from draft size, draft length and acceptance rate; EAGLE-style heads, multi-token prediction and runtime-specific schedulers are not modeled separately.
- LoRA serving compute. Adapters are billed in memory; the extra compute of applying them per request is not.
- CPU offload. llama.cpp and Ollama can keep layers in system RAM; the estimate assumes everything is on the GPU.
- Multi-tenant contention on shared clusters, noisy neighbours, power capping and thermal throttling (Jetson power modes are the exception).
- Runtime versions. Kernel choices, CUDA-graph capture and scheduler defaults change between releases; the calibration describes recent versions, not every one.
- Output quality. Precision choices are sized for memory and speed; the quality loss of a 4-bit or FP8 model is yours to evaluate.
- Approximations flagged in the result. Per-layer KV compression of the newest DeepSeek models, hybrid-layer throughput and multi-GPU AMD over xGMI carry reduced calibration and say so.
Capacity Planner
- Sizing is for a steady peak: the user count, burst factor and occupancy distribution you give. Autoscaling, diurnal load and cold starts are outside it.
- Requests are assumed independent; correlated spikes (a batch job that sends thousands of identical long prompts at once) need their own margin.
- Network, load-balancer and gateway latency are not part of TTFT or TPOT.
The modeling choices themselves are listed in Capacity Planner — modeling assumptions.
Fine-tuning
- CPU and NVMe optimizer offload (ZeRO-Offload, paged optimizers), which would lower per-GPU VRAM in exchange for host RAM and speed.
- Overhead is a flat 1.5 GB per GPU plus, for multi-GPU DDP, the gradient all-reduce bucket — not a percentage that grows with model or NCCL scale.
- Multi-GPU scaling of wall-clock time is linear in the GPU count; real clusters lose some of it to communication, more over PCIe and across nodes.
- LoRA and QLoRA compute uses about 4 × active parameters FLOPs per token instead of the 6 of full fine-tuning, because the frozen base skips its weight-gradient matmuls. The adapters' own small share is not added.
- Activation memory is an estimate that depends on the trainer, FlashAttention and fused kernels; weights, gradients and optimizer states are exact.
- Recommendations cover NVIDIA GPUs only; the training kernels involved (bitsandbytes, 8-bit optimizers) assume CUDA.
Edge
- Sustained thermals: the top power mode of some modules is a burst mode; long runs settle lower.
- The DLA and PVA accelerators — speeds are GPU-only.
- Camera pipelines, video decode and pre-processing time outside the model.