Method
How we compare
Two questions per machine and model: does the model fit in the memory the GPU can use, and how many tokens a second does it generate. The first is arithmetic on published sizes. The second is a measurement where one exists and a labelled estimate where it does not.
Does it fit
A model fits when its quant file, plus the memory its context needs, plus 1.5 GB for compute buffers and the runtime, is no larger than the memory the GPU can address. File sizes are the published byte counts of the GGUF files on Hugging Face. Sizes here are in binary gigabytes (GiB), the unit memory is sold in, so the same file reads about 7% smaller here than on Hugging Face. Headline checks use the 4-bit file (Q4_K_M, or the dynamic 4-bit file where a repo skips it) and 8,192 tokens of context; the calculator takes any published quant and context length.
Context memory is the model card’s bytes per token times the context length. Models that mix full attention with sliding-window or linear layers publish no single figure; for those we count the full-attention layers only and mark the result “context memory estimated”. Models with a compressed cache and no figure at all are checked on weights alone and marked “context memory not counted”. More on context memory.
How much memory the GPU can use
Where a maker publishes the figure, we use it. Where it does not, we apply a stated assumption and mark it “assumption” everywhere it appears:
- Apple: macOS caps GPU memory by default at a share of unified memory that Apple does not publish. We use the share owners read from the system (two thirds up to 36 GB, three quarters above) and also show the limit raised with
sysctl iogpu.wired_limit_mb, as the MLX documentation describes, leaving 8 GB for macOS. - AMD Ryzen AI Max: AMD documents 96 GB of GPU memory on 128 GB machines; smaller configurations get the same three-quarter share.
- NVIDIA GB10 and other coherent-memory machines: total memory less 8 GB for the operating system.
- DIY GPU builds: the VRAM of every card added together. Builds that offload to system RAM show a second “GPU + RAM offload” figure, which is VRAM plus RAM less 8 GB. CPU-only builds use system RAM less the same reserve.
Measured speeds
The catalog holds 1044 published runs from 433 sources: hardware reviews, benchmark collections, maker material and owner benchmark threads, most of them in the llama.cpp project on GitHub. Every run records its model, quant file, backend and source link, and each one is listed on the machine or GPU page it belongs to.
Runs are pooled by chip or GPU set rather than by product, because two machines on the same chip read memory at the same speed. A run taken on a different chassis with the same chip says so in the table. The headline figure is the median generation speed over runs with one user, plain decoding and no more than 4,096 tokens already in context, at the same bit width (a Q8 run is never averaged with a Q4 one). Runs with speculative decoding, several requests at once, cards working in tensor parallel or deep context are shown in their own tables and kept out of the median, because each describes a different workload.
Estimates
Where nobody has published a run, generation speed is estimated from memory bandwidth. The ceiling is bandwidth divided by the bytes read per token, which for a mixture-of-experts model is the active-parameter share of the file. Real runs reach only part of that ceiling, and how much depends on the hardware: a discrete GPU, a unified-memory machine and a CPU server each reach a different share. So the ceiling is scaled by the share that measured runs reach on the same backend, kind of hardware and kind of model, and shown as a range covering the middle half of those runs. A factor needs at least 10 runs behind it; with fewer, or none (CPU-only servers, so far), there is no estimate:
| Backend | Hardware | Model type | Share of ceiling | Middle half | Runs |
|---|---|---|---|---|---|
| llama.cpp | discrete GPU | dense | 66% | 49%–77% | 82 |
| llama.cpp | discrete GPU | mixture of experts | 31% | 25%–36% | 55 |
| llama.cpp | unified memory | dense | 76% | 68%–81% | 31 |
| llama.cpp | unified memory | mixture of experts | 55% | 47%–60% | 86 |
| MLX | unified memory | dense | 83% | 73%–91% | 53 |
| MLX | unified memory | mixture of experts | 46% | 33%–59% | 90 |
Estimates are drawn with a dashed outline and the word “estimate”, and they never enter a measured median. For builds that spill layers into system RAM, each pool’s share of the weights is timed at its own bandwidth and the times added. Multi-GPU builds use one card’s bandwidth, because llama.cpp’s default split sends each token through the cards in turn. Prompt speed is not estimated: it depends on compute and software too much for a one-line formula.
Prices
Complete machines carry the maker’s list price for each memory configuration. A DIY build is the sum of its parts at maker prices, but only when every part has a price. Used GPUs have no price here yet, so those builds show the priced parts and name what is missing, rather than a total that leaves out the most expensive part. Every price is approximate and shops will differ.
There are no shop links on this site and nobody pays for a place in it.
What this does not tell you
- Quality. A model that fits at 2-bit may answer noticeably worse than at 4-bit.
- Speed deep into a long conversation, beyond the runs shown in the long-context tables.
- Batch serving for many users, except where a source measured it.
- Noise, heat, and how much work a DIY build takes to keep running.
Found a figure that is wrong, or a run we should include? Tell us.