← All tutorials
Quantization & Pruning

Quantization & pruning

A trained model is often bigger than it needs to be β€” stored in 32-bit floats, carrying weights that barely matter. That's fine on a training cluster and painful everywhere else: a phone, a browser, an edge device, or a server bill that scales with every request. Quantization shrinks each weight to fewer bits; pruning removes weights entirely. Together they make a model smaller, faster, and cheaper β€” usually with only a sliver of accuracy lost.

The one-line idea. Quantization lowers the precision of the numbers (FP32 β†’ INT8 β†’ INT4). Pruning lowers the count of the numbers (zero out or delete weights). They attack model size from two orthogonal directions, they compose, and both pair naturally with knowledge distillation to squeeze a network down without letting its quality collapse.

Foundations  Two tools, one goal

Both techniques serve model compression: shrink a network so it fits and runs where you need it. A model has two things you can trim β€” how many bits each weight uses, and how many weights there are. Quantization does the first, pruning the second.

Quantization reduces bits per weight; pruning removes weights On the left, a weight stored in 32-bit floating point is mapped down to an 8-bit integer, cutting memory fourfold. On the right, a small neural network has some of its connections and one neuron removed, leaving a sparser, smaller network. Quantization β€” fewer bits weight = 3.14159… Β· FP32 Β· 32 bits (4 bytes) q = 78 Β· INT8 Β· 8 bits (1 byte) r β‰ˆ S Β· (q βˆ’ Z) β€” scale S, zero-point Z continuous range β†’ 256 discrete levels Β· 4Γ— smaller Pruning β€” fewer weights dashed = removed Β· a whole neuron gone (structured)

The wins are practical and stack up: less memory, less memory-bandwidth traffic (often the real bottleneck), faster arithmetic on hardware that supports low precision, lower energy, and a lower per-request cost that you pay on every inference forever. The art is taking those wins without meaningfully hurting accuracy.

Different from distillation. Distillation trains a new, smaller model to imitate a big one. Quantization and pruning instead modify an existing trained model β€” cheaper to apply, and they compose with distillation rather than competing with it (Β§8).

Fewer bits  Quantization β€” lower the precision

Networks train in FP32 (32-bit floating point) for numerical headroom. But inference rarely needs that range. Quantization maps those floats to a lower-precision format β€” cutting memory and unlocking faster integer or half-precision math.

FormatBitsSize vs FP32Typical use
FP32321Γ—Training; the baseline
FP16 / BF1616Β½Γ—Mixed-precision training & inference β€” near-lossless
INT88ΒΌΓ—The workhorse for deployment β€” big win, small loss
INT4 / NF44β…›Γ—Fitting large LLMs in limited VRAM

The mapping is an affine transform. A real value r is represented by an integer q through a scale S and a zero-point Z:

# Quantize a float to int8, and dequantize back.
q = round(r / S) + Z          # float  β†’ int    (S = range / 255)
r β‰ˆ S * (q - Z)               # int    β†’ float  (approx: rounding loses a little)

# Symmetric quant: Z = 0, range centred on zero.
# Asymmetric quant: Z β‰  0, uses the full int range for skewed values.
# Per-tensor: one S,Z for the whole tensor. Per-channel: one per row β€” more accurate.
The enemy is the outlier. A single huge weight or activation stretches the range, so the scale S gets coarse and every ordinary value loses precision. This is why per-channel scales, and (for LLMs) tricks that isolate outliers, matter so much β€” and why activations, which are more dynamic than weights, are harder to quantize than weights.

How to quantize  PTQ vs QAT (and dynamic vs static)

Two questions define a quantization recipe: do you retrain? and when are activation ranges computed?

⚑ Post-training quantization (PTQ)

Quantize an already-trained model β€” no retraining. Fast and cheap; often just a few lines. For static PTQ you run a small calibration set through the model to observe activation ranges. Great down to INT8; accuracy can wobble at INT4 and below.

🎯 Quantization-aware training (QAT)

Simulate quantization during training with "fake-quant" nodes, so the weights learn to be robust to the rounding. More accurate β€” the go-to when PTQ drops too much quality or you need aggressive low-bit β€” but it costs a training run and data.

DynamicStatic
WeightsQuantized ahead of timeQuantized ahead of time
ActivationsQuantized on the fly, per inferencePre-quantized using calibration ranges
SetupEasiest β€” no calibration dataNeeds a calibration pass
SpeedGood (great for Transformers/RNNs)Fastest β€” ranges fixed, no runtime cost
Where to start. Try dynamic PTQ to INT8 first β€” it's a one-liner and often nearly free in accuracy. If you need static's speed or lower bits and the accuracy slips, reach for QAT. Always re-measure on your real task β€” see evaluation.

Fewer weights  Pruning β€” remove what doesn't matter

Trained networks are over-parameterised: many weights contribute almost nothing. Pruning finds and removes them, leaving a sparse (or smaller) network. The usual heuristic is magnitude β€” a weight near zero is doing little, so zero it out.

# Magnitude pruning: keep the largest weights, drop the rest.
threshold = percentile(abs(weights), 40)   # drop the smallest 40%
mask = abs(weights) >= threshold             # 1 = keep, 0 = prune
weights = weights * mask                     # pruned weights are now exactly 0

But there's a catch that trips people up: zeroing a weight doesn't make the model faster by itself. The matrix is the same size β€” it just has zeros in it. Whether pruning buys speed depends entirely on how you prune.

Sparsity β‰  speed, unless the hardware agrees. A model with 90% of its weights zeroed still does the same dense matrix multiply β€” the zeros are multiplied too. You only get a real speedup if (a) you prune in a structured way that shrinks the actual dimensions, or (b) you have hardware/kernels that exploit sparsity. That distinction is the whole game (next section).

The key distinction  Structured vs unstructured

UnstructuredStructured
RemovesIndividual weights, anywhereWhole units β€” neurons, channels, filters, heads, layers
ResultSparse matrix, same shapeGenuinely smaller, dense matrix
AccuracyVery high sparsity with little lossCoarser β€” more accuracy risk per unit removed
Real speedup?Only with sparse hardware/kernelsYes β€” on any standard hardware

Unstructured pruning is precise and can reach extreme sparsity (95%+) with minimal accuracy loss β€” but the leftover matrix is the same size, so without special support it only saves storage, not compute. Structured pruning removes entire rows/channels, producing a smaller dense model that runs faster everywhere β€” at the cost of being a blunter instrument.

The hardware-friendly middle ground: N:M sparsity. Modern GPUs (NVIDIA Ampere+) accelerate a 2:4 pattern β€” 2 of every 4 weights are zero β€” giving a real ~2Γ— speedup from a semi-structured sparsity that keeps accuracy high. It's the pragmatic sweet spot between the two extremes.

The recipe  The prune β†’ fine-tune loop

Pruning aggressively in one shot usually hurts. The reliable recipe restores what was lost:

  1. Train the full model to convergence.
  2. Prune β€” remove the least important weights/units (by magnitude, or a smarter importance score).
  3. Fine-tune β€” retrain the survivors so they compensate for what's gone. This is where most of the accuracy comes back.
  4. Iterate — repeat prune→fine-tune in small steps (iterative magnitude pruning) to reach high sparsity gently rather than in one damaging cut.
The Lottery Ticket Hypothesis. Frankle & Carbin (2019) showed a dense network contains a sparse subnetwork β€” a "winning ticket" β€” that, trained from the original initialization, can match the full model's accuracy. It's strong evidence that most of the parameters were never needed, and that the pruned network isn't just a degraded copy but a capable model in its own right.

Today  Compressing LLMs

For large language models, compression isn't a nice-to-have β€” it's what lets a model run on a single GPU or a laptop at all. Quantization dominates here because LLM inference is memory-bandwidth bound: fewer bits per weight means less data to move, which directly means more tokens per second.

πŸ”’ Low-bit weight quantization

GPTQ and AWQ are PTQ methods that push weights to 4-bit with clever per-layer error correction and outlier handling. GGUF (llama.cpp) packages models at various bit widths for CPU/edge. LLM.int8() / bitsandbytes does 8-bit with outlier-aware paths. SmoothQuant shifts the quantization difficulty from activations to weights so both can go INT8.

🧩 QLoRA β€” quantize and fine-tune

Load the base model in 4-bit (NF4) and train only small LoRA adapters on top. You fine-tune a huge model on a single consumer GPU because the frozen 4-bit backbone barely uses memory β€” a marriage of quantization and parameter-efficient fine-tuning.

Low-bit isn't free. 8-bit is usually near-lossless; 4-bit is often a great trade; below that, quality degrades faster and unevenly β€” some capabilities (reasoning, rare knowledge) suffer more than perplexity suggests. Structured pruning of LLMs is harder still and can cost more accuracy than quantization for the same size cut. Always evaluate the compressed model on your tasks, not just a benchmark β€” see what's inside the model.

They compose  The compression stack

These aren't rival techniques β€” they're layers you apply in sequence, each cutting size a different way:

1 Β· Distill

Better small weights

Train a compact student to imitate a big teacher β€” a good, dense starting model.

2 Β· Prune

Fewer weights

Remove units the student doesn't need, then fine-tune to recover accuracy.

3 Β· Quantize

Fewer bits

Drop the surviving weights to INT8/INT4 for the final memory & bandwidth cut.

A common production pipeline is exactly this: distill to a smaller architecture, structurally prune it, fine-tune, then quantize to INT8 β€” compounding the reductions. Order matters (quantize last, since pruning and fine-tuning want full precision), and every step is followed by measurement.

Diminishing returns are real. Each layer takes more effort for less marginal gain and more accuracy risk. Many teams stop after INT8 quantization alone β€” it's the highest reward-to-effort step. Add pruning and distillation only when the deployment target demands it.

The trade-off  When to use β€” and its limits

GoalReach forWhy
Smaller memory footprint, easy winINT8 quantization (PTQ)4Γ— smaller, minutes to apply, tiny accuracy cost
Fit a big LLM in limited VRAM4-bit quantization (GPTQ/AWQ/NF4)8Γ— smaller; the only way it fits at all
Lower latency on standard hardwareStructured pruningProduces a genuinely smaller, dense, faster model
Max compression, keep accuracyQAT and/or the full stackWorth the extra training when the target is strict
What it can't do. Compression removes redundancy β€” it doesn't add capability, and past a point it removes ability, not just fat. Aggressive low-bit or high-sparsity can quietly break specific skills while average metrics look fine, so a compressed model needs the same evaluation rigor as the original. And "smaller" only becomes "faster" when the hardware and runtime actually exploit the format β€” a sparse or INT4 model on unsupported hardware may be no quicker at all.

End to end  Worked example: INT8 + magnitude pruning in PyTorch

Take a trained model and apply the two cheapest wins β€” dynamic INT8 quantization and unstructured magnitude pruning. See the PyTorch tutorial for the surrounding code.

import torch
import torch.nn.utils.prune as prune

# --- 1. Pruning: zero out the smallest 40% of weights in each Linear layer ---
for module in model.modules():
    if isinstance(module, torch.nn.Linear):
        prune.l1_unstructured(module, name="weight", amount=0.4)
        prune.remove(module, "weight")      # bake the mask in permanently

# ...then FINE-TUNE the pruned model for a few epochs to recover accuracy...

# --- 2. Quantization: dynamic PTQ to int8 (weights int8, activations on the fly) ---
quantized = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8
)

# The result: ~4Γ— smaller Linear weights + a sparser model, ready to ship.
torch.save(quantized.state_dict(), "model.int8.pt")
StepChoiceWhy
Prune methodl1_unstructured, 40%Simple magnitude pruning; start conservative
After pruningFine-tuneWhere accuracy comes back β€” never skip it
Quant typeDynamic INT8No calibration data needed; great for Linear-heavy nets
Target modulesLinearWhere the parameters (and the wins) mostly live
OrderPrune β†’ fine-tune β†’ quantizeQuantize last; pruning & training want full precision
Then measure both axes. Compression is only a win if you check the trade: model size and latency on the target device, against accuracy on your real task. A change that halves size but breaks a key capability isn't a win β€” it's a regression you shipped.

Cheat sheet

QuestionShort answer
Quantization vs pruning?Fewer bits per weight vs fewer weights. Orthogonal; they compose.
Common quant targets?FP16/BF16 (Β½Γ—), INT8 (ΒΌΓ—), INT4 (β…›Γ—).
The affine mapping?r β‰ˆ SΒ·(q βˆ’ Z): scale S, zero-point Z.
PTQ vs QAT?PTQ = quantize after training (fast). QAT = simulate during training (accurate).
Dynamic vs static?Activation ranges computed at runtime vs pre-computed via calibration.
Unstructured vs structured pruning?Sparse same-shape matrix vs a genuinely smaller dense model.
Why prune β‰  faster?Zeros still get multiplied unless structured or on sparse hardware.
N:M sparsity?e.g. 2:4 β€” hardware-accelerated semi-structured sparsity, ~2Γ— speedup.
The prune recipe?Train β†’ prune β†’ fine-tune β†’ iterate.
Compressing LLMs?GPTQ/AWQ/GGUF (4-bit), bitsandbytes (8-bit), QLoRA (4-bit + LoRA).

Knowledge check

Six questions. Pick an answer for each, then check your score.

1. What's the difference between quantization and pruning?

2. Quantizing FP32 β†’ INT8 makes the weights roughly…

3. What is the key difference between PTQ and QAT?

4. Why doesn't unstructured pruning automatically make a model faster?

5. Which pruning style yields a genuinely smaller, faster model on standard hardware?

6. In the compression stack, what's the usual order?

Next steps

βš—οΈ Knowledge Distillation

The third compression tool β€” train a small student that these techniques then shrink further.

🧠 How LLMs & Transformers Work

What's being compressed β€” the weights and activations quantization & pruning act on.

πŸ”₯ PyTorch

Apply it β€” the framework APIs for quantization and pruning.

πŸ§ͺ LLM Evaluation

Prove the trade β€” measure accuracy against size & latency after compressing.