What makes INT4 quantization difficult

Going from 32 bits to 8 feels like a compression problem. Going from 8 to 4 feels like a representation problem. The arithmetic is the same in both cases — pick a scale, round onto a grid, keep the integers — but the second step removes something the first one did not, and the failure mode changes character.

Sixteen levels is not many

A 4-bit signed grid has sixteen reconstruction points. Whatever the distribution of a weight tensor looks like, it has to be summarised by sixteen values plus a scale. For a well-behaved, roughly bell-shaped tensor that is survivable: the levels land where the mass is, and the average error stays small relative to the signal.

The problem is that the scale is set by the extremes, not by the mass. One weight three times larger than the rest of the tensor stretches the grid so that the levels near zero — where most of the weights actually live — are spaced far too coarsely. The tensor is quantized correctly and represented badly at the same time.

At 8 bits there is enough resolution to absorb this. At 4 bits there is not, which is why so much of the low-bit literature is really about controlling the range rather than about rounding.

Three responses that keep appearing

Reading through recent post-training quantization work, most methods I find convincing are attacking the same problem from different directions.

• Shrink the group. Instead of one scale per tensor, use one per row, per channel, or per block of 64 or 128 weights. Outliers then only damage their own neighbourhood. The cost is metadata and slightly awkward kernels.
• Change the grid. Uniform spacing assumes a uniform distribution, which weights do not have. Formats like NF4 place levels according to an assumed normal distribution, so resolution follows density.
• Move the difficulty somewhere cheaper. Smoothing and rotation-style methods rescale activations and weights against each other, or rotate the representation, so that the extreme values are spread out before quantization sees them.

What these have in common is that none of them makes 4 bits into 8 bits. They make the distribution easier to describe with sixteen numbers.

Activations are the harder half

Weights are static, so they can be inspected offline and quantized carefully. Activations are produced at runtime, they depend on the input, and in transformers a small number of feature dimensions carry values far outside the rest of the distribution. Those dimensions matter — the model relies on them — so clipping them away costs accuracy immediately.

This is what makes me sceptical of any single-number claim about a low-bit method. Whether INT4 works depends on which tensors are in INT4, whether activations are included, what the group size is, and whether a kernel exists that reads the format without unpacking it first.

What I am still unsure about

• How much of the observed degradation is genuinely about numerical error, and how much is about specific circuits in the model losing a value they depend on.
• Whether the outlier structure is a property of the architecture, of the training recipe, or of the data — and therefore whether it can be trained away.
• Whether error metrics computed per tensor predict end-task behaviour well enough to drive a search, which is the assumption most sensitivity analyses quietly make.

The last one is the question my own work runs into first, because if per-tensor error does not predict task loss, then a sensitivity-guided precision allocation is optimising a proxy and not the thing I care about.