⌕ search /
Knowledge base

Knowledge base

How to size GPUs for LLM inference, serving and fine-tuning with howmanygpus.ai — 39 articles, 41 glossary terms, 11 answers.

TryTPOTNVFP4preemption riskZeRO-3agentic

Start here

  1. 1

    Pick a model. Catalog, HuggingFace ID, or manual architecture.

  2. 2

    Set the workload. Context, users; Advanced: batch, framework, LoRA, speculative decoding.

  3. 3

    Read the answer. VRAM, configs, TTFT / TPOT, $/1M. Copy the URL to share.

Open calculator Full quick start

Theoretical estimates (±15–30 %) · prices indicative — estimates, not quotes

Keyboard shortcuts

/
Focus the search · here: the knowledge-base search
?
Open this knowledge base
Esc
Blur the current field / close a menu

Popular

curated · page-view order follows
  1. 01Capacity Planner — step-by-step sizing guideTCO6 min
  2. 02Quick start — three stepsGetting started2 min
  3. 03Weight and KV-cache precisionParameters & inputs4 min
  4. 04Reading a result: VRAM, fit band, $/1MGetting started5 min
  5. 05Agentic mode — sizing session-based workloadsTCO10 min
  6. 06Frameworks — what each runtime changesFrameworks6 min
Browse all 39 articles →

How accurate is this?

memory
11/11 in band
throughput
20/29 ±20 %
low-concurrency (batch 1–16)
107/167 ±20 %
Accuracy page →

Recently updated

Inference calculator — overviewInference calculator · 2026-10-26 The input deck and Advanced: batch, targets, LoRA, speculative decodingInference calculator · 2026-10-26 How the calculator picks TP / PP / EPInference calculator · 2026-10-26 Multi-node, rack-scale and recommendation orderInference calculator · 2026-10-26 GPU Lookup — models that fit a GPUWhat fits my GPU · 2026-10-26

Calculator modes

21 articles · one category per tool
Inference calculator4

VRAM, fit, throughput and parallelism for one model on real GPUs.

Inference calculator — overviewThe input deck and Advanced: batch, targets, LoRA, speculative decodingHow the calculator picks TP / PP / EPMulti-node, rack-scale and recommendation order
What fits my GPU3

Start from a card and see which models run on it.

GPU Lookup — models that fit a GPUFit bands: underused · optimal · good · tightContext length and what drops out
Optimize3

Every setup that fits, on one tok/s per user by $/1M map.

Optimize — every setup that fitsReading the tok/s per user × $/1M mapGoals, draggable limits and the context slider
TCO · Capacity Planner6

GPUs to serve N users at a latency SLA, chat or agentic.

Capacity Planner — step-by-step sizing guideAgentic mode — sizing session-based workloadsPareto explorer — the idea, on an exampleCapacity Planner — modeling assumptionsKV risk gauge and occupancy trendCapacity Planner vs Inference calculator
Fine-tuning3

Training memory and cost for full, LoRA and QLoRA runs.

Fine-tuning calculator — full, LoRA, QLoRAOptimizers and ZeRO / FSDP shardingTraining tokens → wall-clock time and cost-to-train
Edge / Jetson2

Fit and speed on Jetson Orin and Thor.

Edge fit & speed — text mode, power modesVision / camera mode — max fps, frame latency

Reference

Getting started5

Quick start, reading a result, sharing, limits of the estimates.

Quick start — three stepsReading a result: VRAM, fit band, $/1MSharing a configurationEstimates, not quotesKeyboard shortcuts
Parameters & inputs4

What every input means and how it moves the result.

Model input: catalog, HuggingFace, manualWeight and KV-cache precisionAgentic session parametersWorkload: context, users, batch, traffic
Frameworks1

What each serving runtime changes in the numbers.

Frameworks — what each runtime changes
Methodology & accuracy5

How the numbers are built and where they are wrong.

Methodology & data sourcesWhy our tok/s are differentWhat is not modeledWhat the model actually knowsAccuracy and error bands
GPU catalog & prices3

The GPU catalog, side-by-side compare and live cloud prices.

GPU catalog: NVIDIA and AMD GPUs, filtersCompare 2–4 GPUs side by sideCloud prices, history and price index
Glossary41 FAQ11

Engineering notes

All engineering notes →
published 2026-08-28Hybrid attention memory: why calculators are 2–70× wrongKDA, Gated DeltaNet, Mamba and sliding-window attention break the classic KV-cache formula. The exact per-layer state math, worked examples for Kimi K3 / Qwen3-Next / Gemma 3 / gpt-oss, and how vLLM, llama.cpp and SGLang actually bill the memory. published 2026-08-28 · updated 2026-10-23How we predict tok/s: a calibrated rooflineParameter heuristics and benchmark tables both fail in instructive ways. The Speed-of-Light model behind this site: prefill vs decode rooflines, the corrections that matter (frameworks, quantization, MoE, hybrid attention, TP sublinearity), and the three-way calibration that keeps it honest — including where it is wrong.

Can't find it? Ask @howmanygpus or on LinkedIn. No accounts, no cookies — privacy.