Bits per weight
Released weights are usually 16-bit: two bytes per parameter. Quantization rounds them to fewer bits, in blocks that share a scale factor, so a “4-bit” GGUF averages a little more than 4 bits once the scales and a few layers kept at higher precision are counted. The names come from llama.cpp: Q4_K_M, Q5_K_M, Q6_K and Q8_0 are the common steps, and the dynamic quants (UD-Q2_K_XL and similar) keep sensitive layers wider than the rest.
One model at four sizes
| Llama 3.3 70B Instruct | File |
|---|---|
| Q4_K_M | 39.6 GB |
| Q5_K_M | 46.5 GB |
| Q6_K | 53.9 GB |
| Q8_0 | 69.8 GB |
Because generation reads the whole active file per token, the 8-bit file is also the slow one: roughly in proportion to its size. The fit check on every page here uses the 4-bit file; the calculator takes any quant the model’s GGUF repo publishes.
How much quality goes
Published comparisons tend to find 8-bit indistinguishable from the original, 5- and 6-bit very close, and 4-bit close enough for chat and most coding. Below 4 bits the loss becomes noticeable on smaller models; very large MoE models tolerate 2- and 3-bit files better, which is why those files exist for them. This site does not score quality, only memory and speed.
MLX files on a Mac
Apple machines often run MLX instead of llama.cpp. MLX 4-bit and 8-bit files are close in size to their GGUF counterparts and are listed on each model’s page. Measured speeds group both formats by bit width.