Quantization stores a model's weights (and sometimes its activations) in a lower-precision number format so the model takes less memory and runs faster. INT8 cuts weight storage to one byte per number with usually small quality loss; INT4 cuts it to half a byte and saves more, but the damage becomes visible and needs careful measurement. The trade is memory and speed against fidelity, and where that trade stops being acceptable depends on the model and the task.
What Quantization Does to a Model
A neural network is mostly a pile of weights: numbers learned during training that get multiplied by inputs at every layer. By default those numbers are stored in floating point, usually 32-bit (FP32) or 16-bit (FP16/BF16). Quantization replaces them with low-precision integers, typically 8-bit (INT8) or 4-bit (INT4), plus a small amount of shared metadata called a scale (and sometimes a zero-point) that maps the integers back onto the original numeric range. A single FP32 weight costs 4 bytes; an INT8 weight costs 1; an INT4 weight costs half a byte before metadata. For a model with billions of parameters, that difference is the whole ballgame.
Two details matter for understanding quality loss. First, quantization is lossy compression: many distinct float values collapse onto the same integer, so the model can no longer represent fine distinctions. Second, most schemes are "block-wise": weights are grouped (say, 32 or 128 at a time) and each group gets its own scale, which limits how badly outliers distort the rest. Some methods also quantize activations, the intermediate values flowing through the network at runtime, which is harder because activations vary with every input; others keep activations in higher precision and quantize only weights. The common umbrella term is PTQ, post-training quantization: compressing an already-trained model without retraining it.
Why Float32 Weights Are Wasteful at Inference
Training needs high precision because gradients are tiny and accumulate over many steps; inference does not. At inference you mostly do one thing: multiply a weight by an input and sum the results. The weight's useful information is roughly its magnitude and sign at a coarse resolution, and research practice has long shown that networks tolerate noticeably noisier weights at prediction time than during training. Storing 32 bits per weight to preserve precision the forward pass never exploits is wasted memory and wasted bandwidth.
The waste is concrete. Memory traffic, not arithmetic, usually limits LLM inference: generating each token requires reading essentially every weight once, so halving the bytes halves the bytes moved per token. A rough estimate: a 7-billion-parameter model needs about 28 GB in FP32, about 14 GB in FP16, about 7 GB in INT8, and roughly 3.5–4 GB in INT4 (plus overhead for scales, embeddings and the KV cache, the stored attention keys and values that grow with context length). Notice the pattern: dropping precision shifts the bottleneck from raw weight storage toward the KV cache and activations, which is why 4-bit weights alone don't make very long contexts cheap.
Lower precision also unlocks hardware. Many CPUs and accelerators execute 8-bit integer matrix multiplies much faster than 32-bit float ones, and 4-bit weights shrink models enough to fit on consumer GPUs or single devices where the full-precision version would not fit at all. That "fits vs. doesn't fit" threshold is often a bigger practical win than the raw speedup.
INT8: How It Works and What It Costs
INT8 quantization maps each weight (or block of weights) from its float range onto the integers -128 to 127 using a scale factor. In code, the mapping looks like:
q = round(w / s) # quantize
w_approx = q * s # dequantize
s = max(|w|) / 127 # per-tensor or per-block scale
The scale s is stored alongside the integers, so reconstruction is a multiply. With per-block scales, outliers in one group don't force every weight in the tensor onto a coarse grid.
Quality cost is usually small. In practice INT8 weight-only quantization is widely treated as near-lossless for most chat and generation workloads, with degradation typically showing up as small changes in benchmark scores rather than obvious breakage; per-channel or per-block schemes are safer than naive per-tensor ones. The costs are more about engineering than accuracy: some layers (embeddings, the output head, normalization layers) are often kept in higher precision, kernels must support the format, and quantizing activations as well as weights (e.g., for INT8 matrix-multiply acceleration) is touchier because activation ranges shift with input. If you want maximum safety with meaningful memory savings, INT8 is the conservative choice.
INT4 and the Quality Cliff: Measuring the Loss
INT4 squeezes weights onto 16 levels (roughly -8 to 7), a fourfold reduction versus FP32 per weight. That is where the "quality cliff" language comes from: the grid is coarse enough that naive rounding visibly hurts, and the field responded with smarter schemes. Group-wise scales shrink the effective range per block; GPTQ-style methods choose rounding errors layer by layer to minimize output error; AWQ-style methods protect a small fraction of salient weights; and newer 4-bit float formats (such as the FP4 variants used on recent accelerators) spend their few bits on exponent rather than uniform spacing. Modern 4-bit weight-only quantization is often usable for chat, but it is not free: expect degradation to concentrate in reasoning, math, code and long-context tasks, and in instruction-following consistency.
Measuring the loss properly means more than one perplexity number. Perplexity (how surprised the model is by held-out text) is cheap and sensitive but correlates imperfectly with task quality. Better: run task benchmarks relevant to your use (multiple-choice knowledge tests, code execution tests, math word problems), check long-context retrieval, and do side-by-side human or model-graded comparisons on your own prompts. A defensible workflow is to quantize, then verify that scores stay within a tolerance you set in advance, per task. Don't trust a single aggregate score, and treat vendor claims of "no quality loss" as marketing until you reproduce them on your data. As a rule of thumb rather than a measured fact: INT8 weight-only is usually within noise of the original; 4-bit is often close on general chat but can visibly trail on harder reasoning, with the gap growing as bit-width falls further (3-bit and below are experimental for most uses).
FAQ
What is the difference between INT8 and INT4 quantization? INT8 stores each weight as an 8-bit integer (256 levels) plus a scale; INT4 stores 4 bits (16 levels). INT4 halves memory versus INT8 but quantizes far more aggressively, so it relies on grouping and error-aware rounding to stay usable, and it loses more quality.
How much quality is lost when quantizing an LLM to 4 bits? It depends on the model, the scheme and the task. Expect little change on casual chat and general knowledge, and the largest drops on math, code, reasoning and long-context work. Measure on your own tasks; don't assume a universal number.
Does quantization make inference faster and cheaper? Usually yes for memory and cost: smaller weights mean less memory traffic per token and the model may fit on cheaper hardware. Speedups are real only when your hardware and kernels support the low-precision format; otherwise dequantization overhead can eat the gain.
Which quantization method should I use for my deployment? Start with INT8 weight-only if you want safety, or a well-supported 4-bit group-wise method (GPTQ/AWQ-family or FP4 on hardware that supports it) if memory is tight. Validate on your own evaluation set before shipping, and keep the full-precision model for comparison.
Can quantization be applied after instruction tuning and alignment? Yes. Post-training quantization is normally applied to the final, aligned model, and the alignment behavior (refusals, tone, format-following) is part of what you must re-check, since it can shift slightly at 4 bits. Quantizing before fine-tuning is possible but then you'd typically retrain or do quantization-aware training to recover quality.
INT8: How It Works and What It Costs
Quantization maps a range of floating-point values onto a small set of integers. INT8 uses 8 bits, so 256 distinct levels. The standard approach is affine (asymmetric) quantization: for each group of weights you store a scale and a zero-point, then compute
q = round(x / scale) + zero_point
x_approx = (q - zero_point) * scale
scale = (max - min) / 255
Weights are quantized once, offline. Activations (the intermediate values flowing through the network at runtime) are quantized on the fly, which is why INT8 inference needs calibration data: a few hundred representative prompts are run through the model to observe activation ranges, and those ranges set the scales. Calibration is the main source of INT8 error — a badly chosen range clips outliers and hurts more than the rounding itself.
The cost in quality is usually small. For most models, INT8 weight-and-activation quantization lands within noise of the float baseline on perplexity and standard benchmarks, and the failure mode is subtle: slightly worse instruction-following on edge cases rather than obvious breakage. The gain is concrete — 4× less memory than FP32 and roughly 2× less than FP16, plus faster matrix multiplies on hardware with INT8 kernels.
INT4 and the Quality Cliff: Measuring the Loss
INT4 gives 16 levels. Naively quantizing every weight to 4 bits does not work; the trick is groupwise quantization: weights are split into small groups (commonly 32, 64, or 128 values), each with its own scale. This is what GPTQ, AWQ, and GGUF's K-quants do. A related family, weight-only quantization, keeps activations in FP16 and dequantizes weights on the fly — less speedup, but far more forgiving.
Worked example. Take a weight group of 64 values with min = −0.80 and max = +0.80.
- INT8:
scale = 1.6 / 255 ≈ 0.00627. A weight of 0.10 becomesround(0.10/0.00627) ≈ 16, reconstructed as16 × 0.00627 ≈ 0.1003— error about 0.3%. - INT4:
scale = 1.6 / 15 ≈ 0.1067. The same weight becomesround(0.10/0.1067) ≈ 1, reconstructed as0.1067— error about 6.7%, roughly 20× larger.
Errors compound across layers. Expect INT4 to be usable but measurably worse: small perplexity increases, more degradation on reasoning and code, and occasional repetition or format failures. Aggressive 3-bit and 2-bit quants degrade sharply.
Measure, don't guess. Compare the quantized model against the float original on your own task set, tracking exact-match or pass rate, not just perplexity. A practical checklist:
- Run the same 50–200 prompts through both models
- Score outputs with a rubric or programmatic check
- Watch for repetition, truncation, and broken formatting
- Test the longest contexts you actually use
- Keep INT8 as the fallback if INT4 fails your bar
The honest summary: INT8 is close to free quality-wise; INT4 is a real trade you should verify on your workload before shipping.
Frequently Asked Questions
What is the difference between INT8 and INT4 quantization?
INT8 stores each weight in 8 bits, giving 256 possible values; INT4 uses 4 bits, giving only 16. INT8 is therefore a milder compression (roughly 4x smaller than the 32-bit original versus roughly 8x for INT4) and usually stays close to the full-precision model, while INT4 trades noticeably more accuracy for a smaller memory footprint. In short: INT8 is the conservative choice, INT4 the aggressive one.
How much quality is lost when quantizing an LLM to 4 bits?
There is no single number, because loss depends on the model, the method, and the task. As a rough estimate, well-executed 4-bit quantization (for example group-wise rounding with a good scale scheme) tends to preserve general quality closely enough that many users notice little difference, while naive or very aggressive 4-bit schemes can degrade reasoning and long-form coherence measurably. Always validate on your own tasks rather than trusting a headline figure.
Does quantization make inference faster and cheaper?
It mainly cuts memory: a smaller model fits on cheaper or fewer GPUs and reduces memory-bandwidth pressure, which often speeds up generation because decoding is bandwidth-bound. Whether you see a wall-clock speedup depends on hardware support for low-precision math (INT8 tensor cores, for instance); compute-bound workloads may benefit less. So expect real savings in memory and cost, with speed gains that vary by setup.
Which quantization method should I use for my deployment?
Start with INT8 if you want a low-risk reduction, and move to INT4 only when memory or cost forces it and you can test quality on representative prompts. Prefer methods that quantize weights in small groups with per-group scales and, where supported, keep activations at higher precision. Whatever you choose, benchmark accuracy and latency on your actual hardware before committing.
Can quantization be applied after instruction tuning and alignment?
Yes. Quantization is a post-training step applied to the finished model, so you quantize after instruction tuning and alignment (RLHF or similar) rather than before. One caveat: alignment can make the model more sensitive to precision loss, so re-check safety and refusal behavior after quantizing, not just raw accuracy.