Edge fit & speed — text mode, power modes

Edge sizes LLMs for NVIDIA Jetson. Jetson modules share one pool of memory between the CPU and the GPU, so the question is not only "does the model fit" but "does it fit after the operating system takes its share", and speed depends on the power mode the module runs at. The tool checks every one of the 10 Orin and Thor modules in the catalog at once and gives decode speed and TTFT per power mode. It shows power and compute figures only — no prices.

Inputs

• Model — a catalog model, a HuggingFace repo or a manual architecture, as in the calculator.
• Quantization — Q4 (GGUF Q4_K_M, about 0.58 bytes per parameter, the edge default), Q8 (GGUF Q8_0, about 1.07), FP16 (2 bytes) or NVFP4 (about 0.58, Jetson Thor only). The GGUF sizes are whole-file averages, embeddings included.
• Context — 2K to 128K tokens, and the prompt length — 128 to 8,192 tokens — that TTFT is computed for.
• Mode — Text for chat and agents; Vision for camera workloads with a vision-language model (Vision / camera mode).

Fit in unified memory

usable  = unified memory − OS reserve
needed  = params × quant bytes + KV cache (FP16, one stream, full context) + 0.9 GB runtime
fit %   = needed ÷ usable

The OS reserve is what JetPack, the desktop and system services keep: 1.5 GB on Orin Nano and NX, 2.5 GB on AGX Orin, 3–5 GB on Thor. The 0.9 GB runtime term covers llama.cpp context and compute buffers including the CUDA context. The KV cache is FP16 for one stream at the full context (the llama.cpp default); MLA models store their compact latent instead. The usual fit bands apply, and above 95 % a device is marked as not fitting. NVFP4 is excluded on Orin, which has no FP4 tensor cores. End-of-life modules are not evaluated.

Speed per power mode

Every module lists its nvpmodel power modes — for example 7W, 15W, 25W and MAXN_SUPER on Orin Nano 8GB, up to MAXN on AGX Thor. A mode scales memory bandwidth and compute against the maximum, and the result has one row per mode:

decode tok/s  = derate × 1000 ÷ (traffic ms + per-token overhead ms)
traffic ms    = (active weights + ½ × KV) ÷ (bw scale × memory GB/s × efficiency) × 1000
prefill tok/s = compute scale × dense TFLOPS × 0.25 ÷ (2 × active params)
TTFT          = prompt tokens ÷ prefill tok/s

efficiency = 0.70 dense, 0.55 MoE

• Decode is bounded by memory bandwidth. Each token reads the active weights and, on average, half of the KV cache. The bandwidth efficiency is calibrated on a corpus of Jetson measurements: 0.70 for dense models, 0.55 for MoE models, whose expert reads are less contiguous.
• A per-token overhead (about 5 ms on Orin) fixes what a pure roofline gets wrong on small models: below about 2 GB of weights a roofline over-predicts speed 2–6×, because kernel launches and sampling dominate.
• A derate lowers the lowest power modes (the 7W modes) where measurements fall below the scaled roofline.
• Prefill is bounded by compute: the module's dense FP16 TFLOPS (FP4 for NVFP4 on Thor) at an efficiency of 0.25, coarse for llama.cpp on Jetson. All compute figures are GPU-only dense TOPS — the DLA accelerators cannot run LLMs, so they are left out.

Decode speeds come with ±20 % bands. Modes that cannot be held indefinitely — MAXN on Jetson Thor T4000 and T5000 — are marked as not sustained; plan a product on a sustained mode.

The device lineup

The "what if" panel re-runs the same model at every quantization and at other context lengths, so you can see which modules a change brings into range.