Learn

The five things that decide what a machine can run

• Why memory bandwidth sets generation speed

  Each new token reads every active weight once. Compute matters for reading the prompt; bandwidth matters for writing the answer.

• Quantization: what 4-bit and 8-bit files cost and save

  A quant stores each weight in fewer bits. File size, and so memory and speed, fall almost in step with the bit width.

• Context memory: why long conversations need more RAM

  The KV cache stores every token already read. On older models it can outgrow the weights at long context.

• Mixture-of-experts models: total versus active parameters

  An MoE model needs memory for all of its weights but reads only the active ones per token, so it runs far faster than its size suggests.

• Unified memory: how much of it the GPU can use

  A 128 GB Mac or Ryzen AI Max box cannot hand all 128 GB to the GPU by default. What each platform documents, and what this site assumes.