Vision / camera mode — max fps, frame latency

Switch Edge to Vision mode and the question changes from "how many tokens per second" to "how many camera frames per second can this module describe, and how long does one frame take". The tool turns a camera resolution into image tokens, adds the frames it must keep in memory, and computes the maximum frame rate and per-frame latency for every Jetson power mode.

Inputs

• Model — a vision-language model from the catalog: Gemma 3, Qwen2.5-VL, Qwen3-VL or MiniCPM-V. The tool needs the model's vision encoder and image-token scheme, so a model without one is rejected with "This model has no vision encoder — pick a VLM".
• Camera — 1080p, 720p, a document page at A4, a VGA sensor, or any width × height.
• Target fps — the frame rate you need — quick values from 0.5 to 30, or any value up to 240; 0 means single frames (a photo or a document, no stream).
• Window — how many recent frames stay in the context (1, 4, 8 or 16), for questions about what changed between frames.
• Answer tokens — the length of the model's reply per frame: 16 for a label, 64 for a sentence, 256 for a description.

Quantization and the text context work as in text mode; the text context is added on top of the frame window.

From pixels to tokens

A vision-language model does not read pixels: its vision encoder turns each image into a number of LLM tokens, and that number decides both memory and latency. Each model family does it differently, and the tool implements four schemes:

The difference is large. At 1080p, Qwen2.5-VL 7B produces about 2,700 tokens per frame, Qwen3-VL about 2,000, Gemma 3 always 256. At 720p the Qwen models drop to about 1,200 and 920; at VGA to about 420 and 300. Lowering the camera resolution is often the cheapest way to a higher frame rate. Slice and tile counts are area-based approximations; the exact grids differ per implementation by about one tile.

For a video stream some models pack consecutive frames into one frame's worth of tokens — Qwen2.5-VL and Qwen3-VL two frames, MiniCPM-V six — which the tool applies whenever the target fps is above 0.

Memory

On top of the text-mode budget, vision mode adds the encoder weights in FP16, a 0.25 GB transient encoder workspace, and the KV cache of the frame window: window frames × tokens per frame, held like text tokens. A 16-frame window of 1080p Qwen frames is over 40,000 tokens of KV — more than a long chat. If a module no longer fits, shorten the window or lower the resolution before changing the model.

Frame latency and maximum fps

frame latency = prefill(frame tokens ÷ pack) + encoder + decode(answer tokens)
encoder       = 2 × encoder params × frame tokens × 4 FLOPs at the mode's FP16 compute
max fps       = 1000 ÷ frame latency (ms)

• Prefill of the image tokens is usually the largest term; it is compute-bound and scales with the power mode, as in text mode.
• The encoder processes about four times the final token count before merging, at the same coarse efficiency as prefill.
• Decode of the answer is bandwidth-bound: a 256-token description can take longer than the image itself.

The device table shows, per module, whether it keeps up with the target — "Keeps up with 2 fps on N of M that fit" — and the panel for the selected module shows max fps, frame latency and headroom against the target per power mode. The numbers are for one stream: two cameras on one module halve the frame rate each can get.