Why memory bandwidth sets generation speed

Two phases, two bottlenecks

A model answers in two steps. First it reads your prompt, all of it at once, and that is a large batch of matrix multiplication: the chip’s compute decides how long it takes. Then it writes the answer one token at a time, and for each token it has to pull every active weight out of memory. The arithmetic per weight is tiny; the waiting for bytes is not. Tables on this site call the first figure prompt speed and the second generation speed.

The arithmetic

Llama 3.3 70B Instruct at 4-bit is a 39.6 GB file and every one of its weights is used for every token. A machine with 256 GB/s of memory bandwidth can read that file at most 6.0 times a second, so about that many tokens a second is the ceiling before any overhead. Real runs reach a share of it. On discrete GPUs, measured llama.cpp runs reach a median of 66% of the ceiling on dense models (82 runs) but only 31% on mixture-of-experts models (55 runs), where each token jumps between experts scattered through memory. Unified-memory machines do better on the same kind of model: 55% (86 runs).

What the measurements show

The same model at 4-bit, single user, plain decoding, median of published runs:

The order matches bandwidth. It is not exactly proportional: software maturity on each chip, how the backend handles the model’s expert layers and the size of the run all move the result.

What that means when buying

For chatting with a model, compare GB/s first and memory size second; memory decides whether a model runs, bandwidth decides how fast it talks. Compute matters when you paste long documents or code, because the prompt phase is where a Ryzen AI Max box or a Mac falls well behind a discrete NVIDIA card with the same bandwidth.