What the cache holds
Attention compares each new token with every earlier one. Rather than recompute the earlier ones, the model keeps their keys and values, the KV cache. Its size per token is 2 (keys and values) × layers × KV heads × head dimension × bytes per number.
A standard model
Llama 3.3 70B Instruct has 80 layers and 8 KV heads of dimension 128, so at 16-bit each token costs 327,680 bytes. That is 2.5 GB at 8,192 tokens and 40.0 GB at 131,072, on top of a 39.6 GB 4-bit file.
Newer layouts cache less
Many 2026 models mix a few full-attention layers with linear-attention or sliding-window layers that keep a small fixed state. Qwen3.8 27B caches only its full-attention layers: about 65,536 bytes per token, or 8.0 GB at 131,072 tokens. Pages here use that full-attention figure and mark the result “context memory estimated”, because the fixed state is left out.
Models with compressed latent caches, such as DeepSeek V3.2, publish no per-token figure that survives the compression. For those the fit check counts weights and overhead only and says “context memory not counted”; leave several GB spare.
Ways to shrink it
llama.cpp and MLX can store the cache at 8-bit or 4-bit, roughly halving or quartering it. The fit figures on this site assume the 16-bit cache, which is the default, so they lean cautious.