A trained model is often bigger than it needs to be β stored in 32-bit floats, carrying weights that barely matter. That's fine on a training cluster and painful everywhere else: a phone, a browser, an edge device, or a server bill that scales with every request. Quantization shrinks each weight to fewer bits; pruning removes weights entirely. Together they make a model smaller, faster, and cheaper β usually with only a sliver of accuracy lost.
The one-line idea. Quantization lowers the precision of the numbers (FP32 β INT8 β INT4). Pruning lowers the count of the numbers (zero out or delete weights). They attack model size from two orthogonal directions, they compose, and both pair naturally with knowledge distillation to squeeze a network down without letting its quality collapse.
Foundations Two tools, one goal
Both techniques serve model compression: shrink a network so it fits and runs where you need it. A model has two things you can trim β how many bits each weight uses, and how many weights there are. Quantization does the first, pruning the second.
The wins are practical and stack up: less memory, less memory-bandwidth traffic (often the real bottleneck), faster arithmetic on hardware that supports low precision, lower energy, and a lower per-request cost that you pay on every inference forever. The art is taking those wins without meaningfully hurting accuracy.
Different from distillation. Distillation trains a new, smaller model to imitate a big one. Quantization and pruning instead modify an existing trained model β cheaper to apply, and they compose with distillation rather than competing with it (Β§8).
Fewer bits Quantization β lower the precision
Networks train in FP32 (32-bit floating point) for numerical headroom. But inference rarely needs that range. Quantization maps those floats to a lower-precision format β cutting memory and unlocking faster integer or half-precision math.
Format
Bits
Size vs FP32
Typical use
FP32
32
1Γ
Training; the baseline
FP16 / BF16
16
Β½Γ
Mixed-precision training & inference β near-lossless
INT8
8
ΒΌΓ
The workhorse for deployment β big win, small loss
INT4 / NF4
4
β Γ
Fitting large LLMs in limited VRAM
The mapping is an affine transform. A real value r is represented by an integer q through a scaleS and a zero-pointZ:
# Quantize a float to int8, and dequantize back.
q = round(r / S) + Z # float β int (S = range / 255)
r β S * (q - Z) # int β float (approx: rounding loses a little)# Symmetric quant: Z = 0, range centred on zero.# Asymmetric quant: Z β 0, uses the full int range for skewed values.# Per-tensor: one S,Z for the whole tensor. Per-channel: one per row β more accurate.
The enemy is the outlier. A single huge weight or activation stretches the range, so the scale S gets coarse and every ordinary value loses precision. This is why per-channel scales, and (for LLMs) tricks that isolate outliers, matter so much β and why activations, which are more dynamic than weights, are harder to quantize than weights.
How to quantize PTQ vs QAT (and dynamic vs static)
Two questions define a quantization recipe: do you retrain? and when are activation ranges computed?
β‘ Post-training quantization (PTQ)
Quantize an already-trained model β no retraining. Fast and cheap; often just a few lines. For static PTQ you run a small calibration set through the model to observe activation ranges. Great down to INT8; accuracy can wobble at INT4 and below.
π― Quantization-aware training (QAT)
Simulate quantization during training with "fake-quant" nodes, so the weights learn to be robust to the rounding. More accurate β the go-to when PTQ drops too much quality or you need aggressive low-bit β but it costs a training run and data.
Dynamic
Static
Weights
Quantized ahead of time
Quantized ahead of time
Activations
Quantized on the fly, per inference
Pre-quantized using calibration ranges
Setup
Easiest β no calibration data
Needs a calibration pass
Speed
Good (great for Transformers/RNNs)
Fastest β ranges fixed, no runtime cost
Where to start. Try dynamic PTQ to INT8 first β it's a one-liner and often nearly free in accuracy. If you need static's speed or lower bits and the accuracy slips, reach for QAT. Always re-measure on your real task β see evaluation.
Fewer weights Pruning β remove what doesn't matter
Trained networks are over-parameterised: many weights contribute almost nothing. Pruning finds and removes them, leaving a sparse (or smaller) network. The usual heuristic is magnitude β a weight near zero is doing little, so zero it out.
# Magnitude pruning: keep the largest weights, drop the rest.
threshold = percentile(abs(weights), 40) # drop the smallest 40%
mask = abs(weights) >= threshold # 1 = keep, 0 = prune
weights = weights * mask # pruned weights are now exactly 0
But there's a catch that trips people up: zeroing a weight doesn't make the model faster by itself. The matrix is the same size β it just has zeros in it. Whether pruning buys speed depends entirely on how you prune.
Sparsity β speed, unless the hardware agrees. A model with 90% of its weights zeroed still does the same dense matrix multiply β the zeros are multiplied too. You only get a real speedup if (a) you prune in a structured way that shrinks the actual dimensions, or (b) you have hardware/kernels that exploit sparsity. That distinction is the whole game (next section).
The key distinction Structured vs unstructured
Unstructured
Structured
Removes
Individual weights, anywhere
Whole units β neurons, channels, filters, heads, layers
Result
Sparse matrix, same shape
Genuinely smaller, dense matrix
Accuracy
Very high sparsity with little loss
Coarser β more accuracy risk per unit removed
Real speedup?
Only with sparse hardware/kernels
Yes β on any standard hardware
Unstructured pruning is precise and can reach extreme sparsity (95%+) with minimal accuracy loss β but the leftover matrix is the same size, so without special support it only saves storage, not compute. Structured pruning removes entire rows/channels, producing a smaller dense model that runs faster everywhere β at the cost of being a blunter instrument.
The hardware-friendly middle ground: N:M sparsity. Modern GPUs (NVIDIA Ampere+) accelerate a 2:4 pattern β 2 of every 4 weights are zero β giving a real ~2Γ speedup from a semi-structured sparsity that keeps accuracy high. It's the pragmatic sweet spot between the two extremes.
The recipe The prune β fine-tune loop
Pruning aggressively in one shot usually hurts. The reliable recipe restores what was lost:
Train the full model to convergence.
Prune β remove the least important weights/units (by magnitude, or a smarter importance score).
Fine-tune β retrain the survivors so they compensate for what's gone. This is where most of the accuracy comes back.
Iterate β repeat pruneβfine-tune in small steps (iterative magnitude pruning) to reach high sparsity gently rather than in one damaging cut.
The Lottery Ticket Hypothesis. Frankle & Carbin (2019) showed a dense network contains a sparse subnetwork β a "winning ticket" β that, trained from the original initialization, can match the full model's accuracy. It's strong evidence that most of the parameters were never needed, and that the pruned network isn't just a degraded copy but a capable model in its own right.
Today Compressing LLMs
For large language models, compression isn't a nice-to-have β it's what lets a model run on a single GPU or a laptop at all. Quantization dominates here because LLM inference is memory-bandwidth bound: fewer bits per weight means less data to move, which directly means more tokens per second.
π’ Low-bit weight quantization
GPTQ and AWQ are PTQ methods that push weights to 4-bit with clever per-layer error correction and outlier handling. GGUF (llama.cpp) packages models at various bit widths for CPU/edge. LLM.int8() / bitsandbytes does 8-bit with outlier-aware paths. SmoothQuant shifts the quantization difficulty from activations to weights so both can go INT8.
Load the base model in 4-bit (NF4) and train only small LoRA adapters on top. You fine-tune a huge model on a single consumer GPU because the frozen 4-bit backbone barely uses memory β a marriage of quantization and parameter-efficient fine-tuning.
Low-bit isn't free. 8-bit is usually near-lossless; 4-bit is often a great trade; below that, quality degrades faster and unevenly β some capabilities (reasoning, rare knowledge) suffer more than perplexity suggests. Structured pruning of LLMs is harder still and can cost more accuracy than quantization for the same size cut. Always evaluate the compressed model on your tasks, not just a benchmark β see what's inside the model.
They compose The compression stack
These aren't rival techniques β they're layers you apply in sequence, each cutting size a different way:
Remove units the student doesn't need, then fine-tune to recover accuracy.
3 Β· Quantize
Fewer bits
Drop the surviving weights to INT8/INT4 for the final memory & bandwidth cut.
A common production pipeline is exactly this: distill to a smaller architecture, structurally prune it, fine-tune, then quantize to INT8 β compounding the reductions. Order matters (quantize last, since pruning and fine-tuning want full precision), and every step is followed by measurement.
Diminishing returns are real. Each layer takes more effort for less marginal gain and more accuracy risk. Many teams stop after INT8 quantization alone β it's the highest reward-to-effort step. Add pruning and distillation only when the deployment target demands it.
The trade-off When to use β and its limits
Goal
Reach for
Why
Smaller memory footprint, easy win
INT8 quantization (PTQ)
4Γ smaller, minutes to apply, tiny accuracy cost
Fit a big LLM in limited VRAM
4-bit quantization (GPTQ/AWQ/NF4)
8Γ smaller; the only way it fits at all
Lower latency on standard hardware
Structured pruning
Produces a genuinely smaller, dense, faster model
Max compression, keep accuracy
QAT and/or the full stack
Worth the extra training when the target is strict
What it can't do. Compression removes redundancy β it doesn't add capability, and past a point it removes ability, not just fat. Aggressive low-bit or high-sparsity can quietly break specific skills while average metrics look fine, so a compressed model needs the same evaluation rigor as the original. And "smaller" only becomes "faster" when the hardware and runtime actually exploit the format β a sparse or INT4 model on unsupported hardware may be no quicker at all.
End to end Worked example: INT8 + magnitude pruning in PyTorch
Take a trained model and apply the two cheapest wins β dynamic INT8 quantization and unstructured magnitude pruning. See the PyTorch tutorial for the surrounding code.
import torch
import torch.nn.utils.prune as prune
# --- 1. Pruning: zero out the smallest 40% of weights in each Linear layer ---for module in model.modules():
if isinstance(module, torch.nn.Linear):
prune.l1_unstructured(module, name="weight", amount=0.4)
prune.remove(module, "weight") # bake the mask in permanently# ...then FINE-TUNE the pruned model for a few epochs to recover accuracy...# --- 2. Quantization: dynamic PTQ to int8 (weights int8, activations on the fly) ---
quantized = torch.quantization.quantize_dynamic(
model, {torch.nn.Linear}, dtype=torch.qint8
)
# The result: ~4Γ smaller Linear weights + a sparser model, ready to ship.
torch.save(quantized.state_dict(), "model.int8.pt")
Step
Choice
Why
Prune method
l1_unstructured, 40%
Simple magnitude pruning; start conservative
After pruning
Fine-tune
Where accuracy comes back β never skip it
Quant type
Dynamic INT8
No calibration data needed; great for Linear-heavy nets
Target modules
Linear
Where the parameters (and the wins) mostly live
Order
Prune β fine-tune β quantize
Quantize last; pruning & training want full precision
Then measure both axes. Compression is only a win if you check the trade: model size and latency on the target device, against accuracy on your real task. A change that halves size but breaks a key capability isn't a win β it's a regression you shipped.
Cheat sheet
Question
Short answer
Quantization vs pruning?
Fewer bits per weight vs fewer weights. Orthogonal; they compose.
Common quant targets?
FP16/BF16 (Β½Γ), INT8 (ΒΌΓ), INT4 (β Γ).
The affine mapping?
r β SΒ·(q β Z): scale S, zero-point Z.
PTQ vs QAT?
PTQ = quantize after training (fast). QAT = simulate during training (accurate).
Dynamic vs static?
Activation ranges computed at runtime vs pre-computed via calibration.
Unstructured vs structured pruning?
Sparse same-shape matrix vs a genuinely smaller dense model.
Why prune β faster?
Zeros still get multiplied unless structured or on sparse hardware.
N:M sparsity?
e.g. 2:4 β hardware-accelerated semi-structured sparsity, ~2Γ speedup.