Hybrid attention memory: why calculators are 2–70× wrong

Every popular VRAM calculator computes the KV cache the same way: 2 × layers × kv_heads × head_dim × context × bytes — memory that grows linearly with context length on every layer. The 2026 model generation broke that formula. Kimi K3, Qwen3-Next and Qwen3.5, Ling, Gemma 3, gpt-oss, Nemotron 3, Granite 4.0 and Jamba replace most of their attention layers with mechanisms whose memory does not grow with context at all, or grows only to a small window. Calculators that don't know this overshoot by 2× on gpt-oss, 6× on Gemma 3, and up to ~70× when a Multi-head-Latent-Attention model is read as plain GQA. For Kimi K3 the public estimates we surveyed span a 200× range — from "~6,060 GB" (a parameter-count heuristic) to the correct ~8 GB of KV at 256K context.

Three memory classes, one formula

A hybrid model's per-sequence attention memory is the sum of three classes (this is also exactly how vLLM's hybrid KV-cache manager allocates it):

per_seq = Σ full-attention layers   × per_token_bytes × context      # grows with context
        + Σ sliding-window layers   × per_token_bytes × min(context, W)   # capped at the window
        + Σ linear/SSM layers       × state_bytes                          # constant — never grows

per_token_bytes is the classic 2 × kv_heads × head_dim × dtype_bytes (or the MLA latent kv_lora_rank + rope_dim), and state_bytes is a fixed recurrent state each sequence owns — it scales with the number of concurrent users, not with tokens. That last property changes capacity planning: at high concurrency the state term is a per-slot price, while the token-KV term is a per-token price.

State formulas per mechanism

All variable names map directly to fields in each model's config.json.

Two traps worth knowing: Falcon-H1 is a parallel hybrid — every layer runs attention and SSM, so there is no KV saving at all; and MiniMax M2 abandoned its predecessor's linear attention entirely — it is a classic full-attention model.

Naive vs actual — six models

fp16 KV cache, one sequence. "Naive" bills every layer as full GQA attention over the whole context.

The Muse Glimmer row explains a famous thread: the model "actually fits on a single RTX 3090 at 128K context" precisely because 39 of its 52 layers cap at a 2048-token window and its full layers use only 2 KV heads — a naive reading predicts it can't.

What the runtimes actually allocate

The formula above is the physics; serving engines add their own billing rules:

• vLLM (v1 hybrid KV manager, ≥~0.21): SWA layers are billed at the window (older versions billed the full context — upgrading gave Gemma-class models a measured 9.5× KV-capacity jump); state pages are padded to a unified page size, which on small hybrids can over-allocate up to ~7× (open issue #37121).
• llama.cpp: window-sized SWA cache by default (--swa-full restores full-context allocation for context reuse); recurrent states live per sequence slot and are not covered by KV-cache quantization.
• SGLang / TensorRT-LLM: recurrent state defaults to float32 — twice the footprint of vLLM's bf16 for the same model.
• MTP (multi-token prediction, shipped by DeepSeek, Qwen3.5, Ling): the draft head is a real attention block — one extra layer of KV plus its weights when speculation is on.

Try it yourself

Our calculator models all three memory classes (and shows the naive-vs-actual delta on every hybrid result). A pre-configured example — gpt-oss-120b at 131K context — or import any HuggingFace model: the hybrid layer mix is auto-detected from config.json for all five encoding styles the 2026 model zoo uses.

Sources

• Kimi Linear / KDA: arXiv:2510.26692; model configs on HuggingFace (moonshotai/Kimi-Linear-48B-A3B, moonshotai/Kimi-K3)
• Gated DeltaNet geometry: Qwen/Qwen3-Next-80B-A3B and Qwen3.5/3.6 config.json (linear_num_value_heads family)
• Mamba-2 hybrids: nvidia/Nemotron-H & Nemotron-3, ibm-granite/granite-4.0-h, tiiuae/Falcon-H1 configs
• vLLM hybrid KV cache manager: design doc; PyTorch blog “Hybrid Models as First-Class Citizens in vLLM”; issues #37121, #43308
• llama.cpp iSWA: PR #13194, discussion #24543; SGLang hybrid support: PyTorch blog “Hybrid Models Meet SGLang”
• DeepSeek V3.2 DSA byte layout: vLLM & LMSYS day-0 posts (2025-09-29)
• All formulas are regression-tested in this site's calculator against the worked examples above.