Where VRAM actually goes during inference

Working on a single 8 GB laptop GPU has been unexpectedly useful. The constraint is small enough that the accounting has to be right, and it removes any temptation to assume a compression result will transfer to a machine I do not have.

Parameter count is a lower bound, not a budget

Weights are the obvious term: parameters multiplied by bytes per parameter. It is also the only term that compression reduces directly, which is why it dominates the way people talk about model size.

The rest of the footprint is made of things that do not shrink when the weights do.

• Activations. Whatever has to be held while a forward pass is in flight. Bounded by the widest intermediate tensor and the batch size rather than by parameter count.
• KV cache for autoregressive decoding. Grows with sequence length and batch size, and keeps growing during generation. For long contexts it can rival the weights.
• Runtime and context. The CUDA context, cuDNN and cuBLAS workspaces, the allocator's own reserved blocks. Several hundred megabytes that exist before the model is loaded.
• Fragmentation. Reserved memory that cannot be handed out because it is the wrong shape. Not visible in a parameter count and entirely capable of causing an allocation failure.

So a model whose weights fit in 8 GB with room to spare can still refuse to run, and a model that runs at one sequence length can fail at another with no change to the weights.

Why this shapes what compression is worth

If weights are 4.4 GB of an 8 GB budget and everything else takes 2.3 GB, then quantizing weights to a quarter of their size frees roughly 3.3 GB. That is a real gain — it is the difference between one model fitting and two, or between a short context and a long one.

But it also means the returns stop. Once weights are a small fraction of the footprint, further weight compression buys almost nothing, and the remaining pressure comes from the cache and the activations. That is a different problem with different tools, and confusing the two is how compression work ends up solving something the deployment was not blocked on.

What I actually record

For any configuration I benchmark, I want peak allocated and peak reserved memory, the sequence length and batch size they were measured at, and whether the run was close enough to the ceiling that the allocator started reusing aggressively. A single "memory: 3.2 GB" number without those conditions is not reproducible even by me, a month later.

Open questions

• How predictable is the KV cache term in practice, given cache layouts that pack or page memory rather than allocating it contiguously?
• At what point does memory bandwidth, rather than capacity, become the binding constraint on a consumer GPU — and does low-bit weight storage help there for the same reason it helps capacity?
• Whether the fragmentation behaviour I see near the ceiling is stable enough to be part of a reported result, or whether it is an artifact of one allocator's policy.