Context length and what drops out

Raise the context on What fits and the list gets shorter. Two different things remove models: the KV cache grows until a model no longer fits, and some models simply do not support the context you asked for. The summary panel counts the second group as "hidden by context"; the first group moves to the multi-GPU table or disappears.

The KV cache grows with every token

For each token in the window, every attention layer keeps a key and a value vector per KV head. The tools reserve the full window for every concurrent user, because a server must be able to hold a request that uses all of it:

KV bytes = 2 × layers × KV heads × head dim × bytes per element × context × users

Llama 3.3 70B has 80 layers, 8 KV heads and a head dimension of 128 — about 0.33 MB per token at FP16. One user needs about 2.7 GB at 8K context, 10.7 GB at 32K and 43 GB at 128K. At 128K a single user's cache is more than half the weights of the model at FP8, and sixteen users would need 690 GB. Weights do not change with context; the cache does, linearly, so long context is where KV becomes the dominant term.

What drops out, and why

• Models that stop fitting. A model at a good fit at 8K can be tight at 32K and over 95 % at 128K. It then leaves the single-card table; if a multi-GPU layout holds it, it reappears in Needs more than one card with its minimum GPU count.
• Models whose native context is shorter. A model trained for 32K is not listed at 128K at all, even if its weights are small. These are counted as "hidden by context" so a vanishing model does not read as a bug. The calculator, by contrast, warns about the same condition and still sizes it.
• Users multiply the effect. The cache is per user. Doubling the users doubles the KV cache, so a long context and many users together remove models fastest.

Architectures that grow slower

The formula above is for full attention. Several architectures change it, and the tools model each of them:

• GQA and MQA — fewer KV heads than query heads; most current models use them, and it is already in the KV-head term.
• MLA (DeepSeek V3, Kimi K2) — a compact latent vector per token instead of full keys and values, several times smaller at long context.
• Sliding-window layers (Gemma 3, gpt-oss) — those layers keep only the last window of tokens, so their share of the cache stops growing.
• Linear-attention and state-space layers (Qwen3-Next, Kimi Linear, Mamba hybrids) — a fixed state per user that does not grow with context at all.

That is why a hybrid model can stay in the list at 128K while a dense model of the same size drops out. The engineering note Hybrid attention memory has the formulas and worked examples.

Keeping a model in the list

• FP8 or INT8 KV cache — halves the cache against FP16, usually with little quality loss; FP8 is native on Hopper, Ada and Blackwell.
• Size the context to the workload — the window is a per-request maximum. If 95 % of your requests are under 16K, a 128K window reserves memory you will rarely use; the Capacity Planner can size for the real occupancy instead of the maximum.
• Lower-precision weights free memory for the cache.
• More GPUs — tensor parallelism shards the KV cache by head across cards (except MLA, whose latent cache is repeated on every GPU).

The presets below open the same H100 at 8K, 32K and 128K so you can watch the list change.