Quantization — Technologies & Tricks

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

Contents

1. Foundations
2. Number Formats
3. Quantization Basics
4. Activation Outliers: The Central Problem
5. Weight Quantization Methods
6. Activation Quantization
7. KV Cache Quantization
8. Diffusion Model Quantization
9. MoE Quantization
10. Vision Model Quantization
11. FP8 / FP4 Training (Native Low-Precision)
12. Hardware Support
13. Inference Frameworks and Quantization Formats
14. LoRA + Quantization
15. Quantization for Inference Acceleration
16. Failure Modes and Mitigations
17. Calibration Best Practices
18. Production Patterns
19. Open Models with Quantized Releases
20. Recent Frontier 2025–2026
21. Production Stack 2026

• Appendix A: Twenty-Five Things to Know
• Appendix B: Decision Tree — "How to Quantize?"
• Appendix C: Year-by-Year Quantization Milestones

1. Foundations

1.1 What is quantization?

Replacing high-precision numbers (FP32 / BF16) with lower-precision (INT8, INT4, FP8, FP4) representations to reduce memory, bandwidth, and compute cost.

\[\hat{x} = s \cdot \operatorname{round}(x/s) + z, \quad s = \text{scale},\; z = \text{zero-point}.\]

1.2 Why quantize?

• Memory: 4× (BF16 →INT4) or 8× (FP32 →FP4) reduction.
• Bandwidth: every byte saved doubles effective throughput on memory-bound workloads.
• Throughput: native low-precision tensor cores are 2–4× faster.
• Energy: lower-precision MACs use less power per op.
• Cost: serve large models on smaller hardware.

1.3 Quantization vs distillation vs pruning

• Quantization: same architecture, fewer bits per weight.
• Distillation: smaller architecture, full precision.
• Pruning: sparser architecture, full precision.
• Often combined: distill + quantize, prune + quantize, LoRA + quantize.

1.4 The Pareto frontier (2026 defaults)

1.5 The big idea

Key

For modern LLMs, weights compress easily; activations are harder (because of channel-wise outliers). KV cache compresses too. Most quality issues come from naive per-tensor activation quantization; per-channel / per-token / per-block scaling fixes most issues.

2. Number Formats

2.1 Floating-point formats

2.2 Integer formats

• INT8: −128 to 127, symmetric uses ±127.
• INT4: −8 to 7, symmetric ±7.
• INT2: 4 levels, used for extreme.
• INT1 / Ternary: BitNet-style; needs special training.

2.3 Block FP / Microscaling (MX)

Microsoft's MX format: per-block scale (typically 32 elements) of FP4 or FP6 mantissas. Standardized as MXFP4, MXFP6. Hardware-supported on B100/B200.

2.4 Why BF16 won training

Same range as FP32 (8 exp bits) but lower precision (7 mantissa). Range matters most for gradients / activations during training; precision can be made up by accumulating in FP32. No loss scaling needed (unlike FP16).

2.5 Why FP8 (over INT8) for training

• E4M3 / E5M2 keep wider dynamic range than INT8 with the same byte cost.
• Outliers don't saturate as catastrophically.
• Per-block scaling brings INT8-like quality with FP8 robustness.
• Standard since H100; default in DeepSeek-V3 + most 2025 frontier training.

2.6 Why FP4 is the next frontier

Blackwell (B100/B200) supports FP4 natively at \(\sim 2\times\) FP8 throughput. Combined with double quantization (per-block scale stored at 8-bit, scaled itself), gives 4-bit forward at training quality.

3. Quantization Basics

3.1 Linear / affine quantization

\[\hat{x} = s \cdot q + z, \quad q = \operatorname{round}((x - z)/s), \quad s \in \mathbb{R}_+,\; z \in \mathbb{Z}.\]

Symmetric: \(z = 0\), range \([-Q_{\max}, Q_{\max}]\). Asymmetric: \(z \neq 0\), range \([Q_{\min}, Q_{\max}]\).

3.2 Scale computation

Min-max: \(s = (\max - \min)/(Q_{\max} - Q_{\min})\).

Symmetric min-max: \(s = \max |x|/Q_{\max}\).

Percentile: drop top/bottom 0.1% before computing range; reduces outlier impact.

MSE / KL minimization: search for \(s\) minimizing reconstruction error or KL divergence between fp and quantized distributions.

3.3 Granularity

• Per-tensor: one \((s, z)\) for the whole tensor. Cheapest; outlier-sensitive.
• Per-channel (per-row / per-column for weights): one \((s, z)\) per channel. Outlier-robust for weights.
• Per-token (for activations): one \((s, z)\) per token (sequence position). Adapts online.
• Per-block / per-group (e.g., 128 elements): scales within a block share. Good middle ground.
• Per-element FP-like: FP formats give per-element exp; effectively per-element scale.

3.4 Calibration

A small dataset (\(\sim\) 128–1024 samples) used to compute scales for activations / static quantization. Critical:

• Match deployment distribution (in-domain prompts).
• Diverse enough to capture activation range.
• Too small →poor scales; too large →slow.

3.5 Static vs dynamic

• Static: scales pre-computed via calibration; fixed at inference.
• Dynamic: scales computed per batch / per token at runtime. More accurate; more compute.

3.6 PTQ vs QAT

• PTQ (Post-Training Quantization): quantize a pretrained model; cheap; minutes to hours.
• QAT (Quantization-Aware Training): simulate quantization during training; expensive; hours to days. Higher quality at low bits (W4A4 and below).

3.7 Straight-Through Estimator (STE)

For QAT: round operation has zero gradient. STE passes gradient through as identity:

\[\frac{\partial\,\operatorname{round}(x)}{\partial x} \approx 1.\]

4. Activation Outliers: The Central Problem

4.1 The observation (LLM.int8(), Dettmers et al.)

LLMs' activations have a small number of channels with > 100× the typical magnitude. These outliers:

• Concentrate in specific feature dimensions.
• Persist across tokens / layers.
• Carry disproportionate information.
• Break per-tensor quantization.

4.2 Why outliers exist

Pretrained Transformers learn to use a few high-norm channels for attention sink behavior. Cannot be removed without retraining; must be handled.

4.3 Per-tensor disaster

Per-tensor scale is set by the outlier →quantization grid stretched →all non-outlier values bucketed together.

INT8 effectively becomes \(\sim\) INT4 for non-outlier channels.

4.4 Five solutions

1. Per-channel (weights): outliers isolated to their own channel scale.
2. Per-token (activations): outliers don't dominate other tokens.
3. Mixed-precision: keep outlier channels in FP16; rest in INT8 (LLM.int8()).
4. Migration: shift outlier scale from activations to weights (SmoothQuant).
5. Rotation: orthogonal rotation distributes outlier energy across channels (QuaRot, SpinQuant).

5. Weight Quantization Methods

5.1 Round-to-Nearest (RTN) baseline

Simplest: quantize each weight to nearest grid point. Per-channel symmetric. Works fine to INT8; degrades at INT4.

5.2 GPTQ (Frantar et al. 2022)

Layer-wise reconstruction with the inverse Hessian:

\[H = 2XX^\top + \lambda I, \quad H \in \mathbb{R}^{d \times d}.\]

Quantize columns one at a time; after quantizing column \(k\), update remaining columns to compensate:

\[\Delta W_{:,\,j>k} = -\frac{(W_q - W)_{:,k} \cdot [H^{-1}]_{k,\,j>k}}{[H^{-1}]_{kk}}.\]

INT4 or INT3 weight quant with minimal quality loss.

5.3 AWQ (Lin et al. 2023)

Activation-aware: identify "salient" weights connected to large activations; protect them. Per-channel scaling \(s \in \mathbb{R}^d\):

\[\min_{s}\; \big\| W\operatorname{diag}(s^{-1})\,(\operatorname{diag}(s)X) - WX \big\|.\]

Then symmetric quantize \(W\operatorname{diag}(s^{-1})\). Faster than GPTQ; no per-layer Hessian.

5.4 SmoothQuant (Xiao et al. 2023)

Migrate scale from activations to weights:

\[W' = W \cdot \operatorname{diag}(s), \quad X' = X \cdot \operatorname{diag}(s^{-1}),\]

with \(s_c = (\max_i |X_{i,c}|)^\alpha / (\max_i |W_{i,c}|)^{1-\alpha}\), \(\alpha \in [0.4, 0.6]\). Activations become flatter (easier to quantize); weights absorb the scale.

Enables INT8 W & A inference where naive PTQ would fail.

5.5 QuaRot (Ashkboos et al. 2024)

Apply random rotations \(Q \in O(d)\) to weight + activation pairs:

\[W' = QW, \quad X' = XQ^\top.\]

\(W'X' = QWXQ^\top\) is invariant under rotations of input/output, but the rotated activations have outliers spread across channels (no longer concentrated). Per-channel quantization becomes much easier.

Hadamard rotations chosen for FFT-fast matmul. Used in W4A4 inference.

5.6 SpinQuant

Learned rotations + per-channel scales. Better than random Hadamard for some models.

5.7 HQQ (Half-Quadratic Quantization)

Closed-form half-quadratic optimization for per-block scales. No calibration data required; extremely fast.

5.8 OmniQuant (Shao et al.)

Learnable per-layer + per-channel quantization parameters via gradient descent (lightweight). Better than GPTQ at very low bits (INT3, INT2).

5.9 LLM-QAT (Liu et al.)

QAT for LLMs using teacher-generated synthetic data (no need for original training corpus). Recovers quality at W4A4 and below.

5.10 QUIP / QUIP#

Incoherence preprocessing: rotate weights to make them "incoherent" (similar magnitudes), then quantize. Lossless 2-bit compression with sparse codebooks.

5.11 BitNet b1.58 (1.58-bit)

Ternary weights \(\{-1, 0, +1\}\) trained from scratch. \(\log_2 3 \approx 1.58\) bits per weight. Matches BF16 quality at > 3B scale; \(\sim 5\times\) faster inference. Requires retraining.

5.12 Comparison table

★ 2026 SOTA update — Rotation successors: FlatQuant, DuQuant

• FlatQuant: learnable per-layer affine transforms (Kronecker-decomposed, fused single kernel) that flatten weights AND activations beyond Hadamard; SOTA W4A4 with <1% drop on Llama-3-70B, up to 2.3x prefill / 1.7x decode (ICML 2025).
• DuQuant: rotation + zigzag permutation removes both 'normal' and 'massive' outliers (first to flag massive outliers at FFN down_proj); new W4A4 SOTA beating QuaRot/SpinQuant (NeurIPS 2024 Oral).

★ 2026 SOTA update — Trellis and extreme low-bit weights

• QTIP: trellis-coded quantization (TCQ) + incoherence processing; a stateful 'bitshift trellis' decoder decouples codebook size from bitrate/dimension, enabling ultra-high-dimensional VQ; SOTA weight-only 2-3 bit and fast inference. Successor to QuIP#.
• ParetoQ (Meta): unified 1 / 1.58 / 2 / 3 / 4-bit QAT scaling laws; finds a sharp learning transition between 2 and 3 bits; ternary 600M beats prior 3B ternary SOTA; 2-bit sits on the best memory-accuracy Pareto frontier.

6. Activation Quantization

6.1 The challenge

Activations are dynamic (depend on input), have outliers, and need per-token scaling.

6.2 Static activation quantization

Calibrate scales offline; freeze. Cheap; good for INT8 if outliers handled.

6.3 Dynamic per-token quantization

Compute scale per token at runtime: \(s_t = \max_c |x_{t,c}|/Q_{\max}\). Standard for INT8 / FP8 activations in production.

6.4 Per-channel activation

Different channels have different ranges. Combine per-token + per-channel: each scalar gets its own scale via a (per-token) · (per-channel) outer product.

6.5 Mixed-precision activations

LLM.int8(): detect outlier channels at runtime; route those through FP16 matmul; rest through INT8. Two parallel matmuls; merge results.

\[y = \operatorname{INT8}(W_{\text{rest}}) \cdot \operatorname{INT8}(X_{\text{rest}}) + \operatorname{FP16}(W_{\text{out}}) \cdot \operatorname{FP16}(X_{\text{out}}).\]

6.6 SmoothQuant activations

Migrating scale to weights flattens activation distribution; per-tensor or per-token quantization then works at INT8 quality.

6.7 FP8 activations

Hopper / Blackwell: FP8 (E4M3 forward, E5M2 backward) tensor cores.

Per-tensor or per-block (per 128) scaling. Matmul accumulator in FP32. Natural fit for activations because outliers fit FP8's wider exp range.

6.8 Activation quantization granularity choice

7. KV Cache Quantization

7.1 Why KV separately?

KV cache often dominates inference memory at long context.

Reducing per-element bytes is direct savings without affecting weight precision.

7.2 Per-channel for K, per-token for V (KIVI)

KIVI insight: K has channel-wise outliers; V is token-wise stable.

• K: per-channel scale.
• V: per-token scale.
• INT4 viable; INT2 with quality loss.

7.3 INT8 / INT4 KV cache

7.4 Outlier-aware KV quantization

Some channels of K have persistent outliers. Solutions:

• Keep outlier channels in FP16; rest in INT4.
• Apply rotation (QuaRot-style) to spread outliers.
• Per-block scaling instead of per-channel.

7.5 FP8 KV (modern frontier)

H100 / B200: FP8 KV native; per-block scale stored alongside. 2× memory saving with negligible quality loss.

Standard in production serving frameworks.

7.6 Implementation

Frameworks: vLLM (KV INT8 / FP8), TensorRT-LLM (FP8 KV), SGLang, llama.cpp (Q4 / Q5 / Q8 KV), KIVI-2 reference impl.

★ 2026 SOTA update — Vector-quantized KV cache

• CommVQ: additive/vector quantization of the KV cache with a lightweight encoder + learned codebook designed commutative with RoPE (decode via a single matmul); reaches 1-2 bit KV (-87.5% at 2-bit), running 128K-context Llama-3.1-8B on one RTX 4090 (ICML 2025).

8. Diffusion Model Quantization

8.1 Why harder than LLMs

• Activation distributions shift dramatically across timesteps.
• Cross-attention introduces additional outliers.
• Quality measured perceptually (FID, human pref), not perplexity — harder to validate small changes.
• Sampling many steps amplifies any per-step error.

8.2 Q-Diffusion (Li et al. 2023)

First systematic PTQ for diffusion. Time-aware calibration: split timesteps into bins; calibrate scales per bin.

8.3 PTQ4DM (He et al.)

Symmetric channel-wise weight quantization + dynamic activation. INT8 with < 1% FID loss.

8.4 Q-DiT (DiT-specific)

PTQ for DiT (Sora, SD3, FLUX backbones). Handles adaLN-induced outliers via per-block scaling.

8.5 SVDQuant (Li et al. 2024)

Low-rank decomposition + 4-bit quantization for FLUX:

\[W = Q + BA, \quad Q \in \operatorname{INT4},\; B, A \in \text{FP16 low-rank}.\]

The low-rank residual absorbs hard-to-quantize structure; INT4 covers the bulk.

4-bit FLUX with quality matching FP16.

8.6 EfficientDM, Q-DM, BitsFusion

Various PTQ approaches for SD-class diffusion. INT8 W & A standard; INT4 W achievable.

8.7 INT4 W + FP8 A

A common production setting for SDXL: 4-bit weights, 8-bit activations. \(\sim 4\times\) memory savings; mild quality loss.

8.8 LCM / DMD2 + quantization

Distillation reduces NFEs; quantization reduces per-step cost.

Combining: 4-step distilled SDXL with 4-bit weights \(\sim 100\times\) faster than baseline.

★ 2026 SOTA update — FP4 diffusion and video

• SANA-Video: NVFP4 / SVDQuant W4A4 quantization of a linear-attention video DiT for edge deployment; selectively quantizes attention + FFN projections while keeping norms, temporal convs, and KV projections at higher precision to preserve fidelity.

9. MoE Quantization

9.1 Why MoE quantization is harder

• Per-expert calibration: each expert has its own activation distribution.
• Cold experts (rarely used) have less calibration data.
• Routing depends on FP-precision logits; must keep router precision.
• Memory savings the most valuable since MoE total params dominate.

9.2 Per-expert calibration

Calibrate each expert independently; require enough hits per expert. Sample more for popular experts; less for cold (or use shared calibration).

9.3 Variable bits per expert

• Hot experts: INT8 / INT4.
• Cold experts: INT2 / INT3 (lower precision OK because rarely accessed).
• Total memory minimized while preserving frequently-used experts' quality.

9.4 Router FP precision

Keep the router (small linear) at FP16 / BF16. Quantizing the router degrades expert selection.

9.5 ktransformers MoE serving

Combines:

• Q4 quantization on experts.
• CPU offload of cold experts.
• GPU loading of active.

Enables 671B DeepSeek-V3 on a single workstation with \(\sim\) 96 GB VRAM.

9.6 DeepSeek-V3's FP8 native MoE

Trained natively in FP8 (E4M3 forward, with per-128-element block scaling). MoE expert FFNs in FP8; shared layers BF16. 2× throughput vs BF16; minimal quality loss.

10. Vision Model Quantization

10.1 ViT and ConvNet PTQ

• PTQ-ViT: ViTs are sensitive to LayerNorm + GELU quantization; need careful per-layer scales.
• Q-ViT: per-block ViT quantization with attention map preservation.
• FQ-ViT: fully-quantized ViT.

10.2 Mobile vision quantization

MobileNet / EfficientNet / MobileViT: INT8 PTQ standard; can run on phone NPUs (Apple ANE, Qualcomm AI Engine).

10.3 Detection + segmentation

Q-DETR: quantization-aware DETR with attention map distillation. MobileSAM / EfficientSAM: smaller + INT8 for mobile.

10.4 VLM quantization

• Vision encoder: quantize separately; usually less sensitive than LLM.
• LLM backbone: standard LLM quantization (GPTQ, AWQ).
• Projector (small MLP): often kept FP.
• Image-token activations: per-token quantization works; image patches don't have the same outlier structure as text tokens.

★ 2026 SOTA update — Modality-balanced VLM quantization

• MBQ (Modality-Balanced Quantization): vision and language tokens differ sharply in quantization sensitivity, so per-modality reconstruction weighting during calibration avoids over-fitting insensitive tokens; +4.4% (W3) / +11.6% (W4A8) on 7B-70B VLMs, with a fused W3 GEMV kernel (CVPR 2025).

11. FP8 / FP4 Training (Native Low-Precision)

11.1 The mixed-precision training stack

Standard recipe (BF16 era):

• Forward + backward: BF16.
• Weights master copy: FP32.
• Optimizer states: FP32.
• Gradient accumulator: FP32.

11.2 FP8 training (NVIDIA Transformer Engine, H100)

• Forward: E4M3 (more precision, ±448 range).
• Backward gradients: E5M2 (more range, ±57344).
• Per-tensor or per-block (per-128) scale.
• Matmul accumulator: FP32.
• Master weights: FP32.

2× throughput vs BF16; matched quality with proper scale management.

11.3 Scale management

• Delayed scaling: track recent max-amax; update scale to keep within FP8 range.
• Just-in-time scaling: compute amax this step; apply.
• Per-block scaling: each block of 128 elements has its own scale (DeepSeek-V3 default).

11.4 Loss scaling (FP16 only)

For FP16 (not BF16 / FP8): scale loss by \(S\), divide gradients by \(S\) before optimizer. Prevents underflow. Not needed for BF16.

11.5 Stochastic rounding

Round to nearest with probability proportional to fractional part:

\[\hat{x} = \begin{cases} \lceil x \rceil & \text{prob } x - \lfloor x \rfloor \\ \lfloor x \rfloor & \text{else} \end{cases}\]

Unbiased in expectation; better than nearest rounding for very low precision (FP4, INT2). Especially useful for accumulators.

11.6 DeepSeek-V3 FP8 recipe

• Per-block (per-128) scaling.
• Per-block group scale also FP-quantized (double quantization).
• Mixed: shared layers + LayerNorm BF16; routed experts FP8.
• Custom kernels: FP8 GEMM with FP32 accumulator + per-block dequant.
• Result: trained 671B-MoE / 37B-active in \(\sim\) 5.5M H800-hours.

11.7 FP4 training (Blackwell, B100/B200)

FP4 (E2M1) at 2× FP8 throughput. Requires:

• Per-block scaling (typically per-32 or per-16).
• Double quantization (scale itself stored compactly).
• Carefully managed tensor stats.

Early production results: matched BF16 / FP8 quality for some workloads; still maturing for general training.

11.8 Microscaling (MX) formats

Standardized per-block FP formats: MXFP4 (per-32 block of FP4), MXFP6, MXFP8. Hardware-supported on Blackwell / AMD MI300+. Becoming the standard low-precision format.

★ 2026 SOTA update — Native FP4 training (2025)

• NVFP4 (NVIDIA): micro-block (16-element) FP4 with 2D block scaling, Random Hadamard transforms, stochastic-rounded gradients, and a few high-precision-kept sensitive layers; a 12B model on 10T tokens ~matches FP8 (MMLU-pro 62.58 vs 62.62). Finer blocks than MXFP4.
• Quartet: near-lossless fully-native MXFP4 training (all matmuls in MXFP4) with optimized Blackwell CUDA kernels; derives the accuracy-vs-efficiency-optimal FP4 recipe at billion scale (NeurIPS 2025).

12. Hardware Support

12.1 NVIDIA

• H100 / H200: FP8 native (E4M3, E5M2); \(\sim\) 1979 TFLOPs FP8 vs 989 BF16.
• B100 / B200 / GB200 (Blackwell): FP4 native; \(\sim\) 4500 TFLOPs FP8 / \(\sim\) 9000 TFLOPs FP4.
• A100: BF16 native; INT8 supported but not FP8.
• Tensor Memory Accelerator (TMA) for async low-precision data movement.

12.2 AMD

MI300X / MI325X: BF16 + FP8 (E4M3, E5M2).

192 / 256 GB HBM. Competitive on memory-bound workloads.

12.3 Apple Silicon

M1/M2/M3/M4: ANE (Apple Neural Engine) optimized for INT8. CoreML supports INT4 / INT8 weight quant. MLX framework: Apple's MoE / quantized inference framework.

12.4 Mobile NPUs

Qualcomm AI Engine (Hexagon NPU): INT8 / INT4 weight, INT8 activation. MediaTek APU: similar.

Google Tensor / TPU mobile: INT8.

12.5 Hardware-specific sparsity

2:4 sparsity on H100 / B200: 2 of 4 weights zero →2× matmul throughput. Combined with quantization for \(\sim 4\times\) effective speedup.

12.6 Edge accelerators

Coral TPU, Hailo, Jetson: INT8 PTQ standard. Groq LPU: deterministic int / FP inference at very low latency.

12.7 TPU FP8 / int8

TPU v5p / v6e (Trillium): BF16 + INT8. Different tensor-core programming model.

13. Inference Frameworks and Quantization Formats

13.1 The framework lineup (2026)

13.2 GGUF / llama.cpp K-quants

• Q4 K S: 4-bit weights, small.
• Q4 K M: 4-bit weights, medium (popular default).
• Q5 K M: 5-bit weights, more quality.
• Q6 K: 6-bit weights.
• Q8 0: 8-bit weights (close to FP16).
• Q3 K M, Q2 K: aggressive; quality risk.
• IQ-quants: importance-weighted (IQ4 NL, IQ3 XXS, etc.); better than K-quants at same bits.

K-quants vary precision per layer / matrix automatically: critical layers (e.g., attention output) at higher bits.

13.3 NF4 (NormalFloat 4)

4-bit format optimized for normally-distributed weights. Used in QLoRA. Better than uniform INT4 for typical model weights.

13.4 Marlin / Machete kernels

Fast W4 GEMM kernels for INT4 weight quantization. Used in vLLM, TensorRT-LLM. 2–4× speedup over FP16 GEMM.

13.5 Format conversion

• AutoAWQ / llm-compressor: HF →AWQ.
• AutoGPTQ: HF →GPTQ.
• llama.cpp quantize: HF (via GGML) →GGUF.
• NVIDIA Model Optimizer: HF →TensorRT-LLM.

14. LoRA + Quantization

14.1 QLoRA (Dettmers et al. 2023)

• Base model in 4-bit (NF4).
• LoRA adapters in BF16 for training.
• Paged optimizer states (CPU offload on overflow).
• Double quantization (quantize the per-block scales themselves).
• Result: fine-tune 65B model on a single 48 GB GPU.

14.2 Standard QLoRA recipe

• Load base in NF4 with double-quant.
• Add LoRA on attention + MLP linears (rank 8–256).
• Train LoRA only; base frozen.
• Merge or keep separate at inference.

14.3 Q-Galore, GaLore variants

Memory-efficient training of full-precision base via low-rank gradient projection. Combined with quantization for further savings.

14.4 Unsloth optimizations

• Custom kernels for QLoRA training; 2–5× speedup.
• Bit-packed memory layouts.
• Triton-based dispatch.

15. Quantization for Inference Acceleration

15.1 Memory bandwidth analysis

For decode (memory-bound): per-step time \(\approx\) (model bytes) / (HBM bandwidth).

Llama-3-70B BF16 on H100 (3 TB/s HBM3): 140 GB / 3000 GB/s = 47 ms/step. INT4: 35 GB / 3000 GB/s = 12 ms/step. \(\sim 4\times\) faster decode.

15.2 Compute bandwidth analysis

For prefill (compute-bound): per-step time \(\approx\) FLOPs / (peak FLOPs).

INT8 matmul \(\sim 2\times\) BF16; FP8 \(\sim 2\times\); FP4 \(\sim 4\times\). Speedup applies primarily to prefill.

15.3 Speedup table (typical)

15.4 Per-deployment selection

• Cloud API (max throughput): FP8 / W8A8.
• Cloud API (memory-tight): W4A16 + FP8 KV.
• Consumer GPU: GGUF Q4 K M (sweet spot).
• Consumer CPU: GGUF Q4 K M or Q5 K M.
• Mobile: INT8 + 2:4 sparsity if hardware supports.
• Edge / NPU: INT8 PTQ.

16. Failure Modes and Mitigations

16.1 Quality cliff at INT4

Quality OK at INT8, suddenly bad at INT4. Fix: per-channel for weights; per-token for activations; consider AWQ / GPTQ over RTN.

16.2 Outlier sensitivity

Per-tensor activation quant collapses. Fix: SmoothQuant migration, QuaRot rotation, mixed-precision (LLM.int8()).

16.3 Long-context quality degradation

Quantized model fine on short context; degrades on long.

Fix: don't quantize KV too aggressively; check perplexity at long context; switch to FP8 KV.

16.4 Tokenizer / language regression

Some tokens / languages disproportionately affected. Fix: more diverse calibration data; per-language eval.

16.5 Calibration overfit

Model great on calibration distribution, poor elsewhere. Fix: diverse calibration; use percentile / MSE-based scales rather than max.

16.6 Diffusion FID drift

Quantized diffusion has higher FID despite low MSE. Fix: per-timestep calibration; SVDQuant for FLUX; check perceptual quality not just numeric.

16.7 Quantized MoE expert atrophy

Cold experts get poor quantization; quality drops. Fix: per-expert calibration with adequate samples; or keep cold experts FP.

16.8 Speculative-decoding mismatch

Draft and target models quantized differently; rejection rate spikes. Fix: match precision; or lower acceptance threshold.

16.9 QAT-PTQ gap

PTQ quality far below QAT. Fix: try AWQ / GPTQ / SpinQuant / OmniQuant; if still bad, consider QAT (LLM-QAT).

17. Calibration Best Practices

17.1 Calibration data

• 128–1024 samples typical.
• Match deployment domain (in-domain prompts).
• Diverse: multiple tasks, languages, lengths.
• Long-context examples for long-context deployment.

17.2 Scale-determination strategies

• Min-max: simplest; outlier-sensitive.
• Percentile (e.g., 99.9%): drop top outliers; better.
• MSE-min: search for scale minimizing reconstruction MSE.
• KL-min: search for scale minimizing KL between FP and quantized output.
• Smooth-search (AWQ): grid-search for \(s\) minimizing weight-level error after activation flattening.

17.3 Eval after quantization

• Perplexity on held-out data.
• Task accuracy on benchmarks (MMLU, GSM8K, HumanEval, etc.).
• Long-context tasks (NIAH, RULER) for long-context deployment.
• Safety / refusal rate (quantization sometimes unsafes a model).
• Human eval on representative prompts.

17.4 Per-layer sensitivity analysis

Quantize one layer at a time; measure quality drop. Layers with high sensitivity (often first/last attention) kept at higher precision.

17.5 Mixed-bit allocation

Given a target average bits, allocate per layer / module by sensitivity. Critical layers higher; less-critical lower.

Used in K-quants automatically.

18. Production Patterns

18.1 Tier-based serving

• Premium tier: BF16 / FP8 (best quality).
• Standard tier: W8A8 (mid).
• Free tier: W4A16 (cheap).
• Route by user / task.

18.2 Distillation + quantization

Distill into smaller model + quantize the student.

Compounding gains: 10× from distillation + 4× from quantization \(= 40\times\) inference cost reduction.

18.3 Quantization-aware deployment

• Test multiple quant configs; pick Pareto-optimal.
• Maintain FP and quant versions side-by-side.
• Canary new quantization; rollback ready.

18.4 Mixed deployment

• Hot path: FP8 / BF16.
• Cold path: INT4 / Q4 K M (acceptable latency loss).
• Spillover: route to cheaper tier on overload.

18.5 Quantize then continue training

Some workflows: PTQ to INT8 →fine-tune to recover quality →re-quantize. Approaches QAT quality with PTQ simplicity.

18.6 Continuous re-quantization

As the base model evolves (RLHF iterations), re-quantize. Maintain calibration data hash for reproducibility.

19. Open Models with Quantized Releases

19.1 LLMs

• Llama 3 / 4: official + community AWQ, GPTQ, GGUF, FP8.
• Mistral / Mixtral: native quant releases, GGUF.
• Qwen 2.5 / 3: official AWQ, GPTQ, GGUF.
• DeepSeek-V3 / R1: FP8 native; community Q4 GGUF / ktransformers.
• Phi family: AWQ + INT4.
• Gemma 2 / 3: official quant releases.

19.2 Vision encoders

• EVA / DINOv2 / SigLIP: ONNX INT8 widely available.
• SAM / SAM 2: INT8 / FP16 deployment.

19.3 VLMs

• LLaVA-OneVision / Qwen2.5-VL / InternVL3: AWQ + GGUF.
• Pixtral, Molmo: FP8 / AWQ.

19.4 Diffusion

• SDXL / FLUX / SD3: INT8 + FP8 widely deployed.
• SVDQuant FLUX: INT4 weights.
• LCM / Hyper-SD distilled + quantized for production.

20. Recent Frontier 2025–2026

20.1 Trends

• FP8 mainstream for training (DeepSeek-V3 demonstrated viability).
• FP4 emerging for inference (and select training) on Blackwell.
• AWQ + GPTQ both production-grade; choice often hardware-driven.
• KV cache quantization (FP8 / INT8) standard.
• MoE-aware quantization (per-expert calibration, variable bits).
• Microscaling formats becoming hardware standard (MXFP4, MXFP6).

20.2 BitNet b1.58 follow-ups

Pure-ternary LLMs trained from scratch; matches BF16 quality at scale; requires retraining (no PTQ path).

Hardware support emerging.

20.3 Research directions

• Lower-bit (INT2, ternary) without retraining.
• Joint quant + sparsity (2:4 + Q4).
• Activation quant for non-LLM tasks (vision-action, video gen).
• KV cache rotation (QuaRot for KV).
• Native FP4 training maturation.

★ 2026 SOTA update — Native 1.58-bit LLMs ship

• BitNet b1.58 2B4T: first open-weight native 1.58-bit LLM (ternary weights, INT8 activations) at 2B params, trained from scratch on 4T tokens; matches full-precision peers of similar size with ~6x lower RAM and 2-6x faster CPU decode via bitnet.cpp kernels.

21. Production Stack 2026

Appendix A: Twenty-Five Things to Know

1. Affine quant: \(\hat{x} = s \cdot \operatorname{round}((x - z)/s) + z.\)
2. Symmetric vs asymmetric (\(z = 0\) vs not).
3. Granularity: per-tensor / per-channel / per-token / per-block.
4. BF16 has same range as FP32 (8 exp bits).
5. FP8: E4M3 forward, E5M2 backward.
6. FP4 (E2M1) on Blackwell at 2× FP8 throughput.
7. Microscaling (MXFP4, MXFP6): per-block FP standard.
8. GPTQ: Hessian-based per-column quantization with update.
9. AWQ: salient-channel scaling formula.
10. SmoothQuant: migrate scale from activations to weights.
11. QuaRot: orthogonal rotation spreads outliers.
12. LLM.int8(): outlier channels in FP16, rest INT8.
13. KIVI: per-channel K, per-token V.
14. NF4: 4-bit format for normal-distributed weights.
15. GGUF K-quants: per-layer adaptive precision.
16. QLoRA: NF4 base + BF16 LoRA adapters.
17. Marlin / Machete: fast W4 GEMM kernels.
18. DeepSeek-V3 FP8 recipe: per-block + double-quant + custom kernels.
19. BitNet b1.58: ternary weights; matches BF16 at > 3B with retraining.
20. PTQ vs QAT: post-training cheap vs aware-training quality.
21. STE: gradient passes through round.
22. SVDQuant: low-rank residual + INT4 for FLUX.
23. Per-expert calibration for MoE quantization.
24. ktransformers: GGUF Q4 + offload for 671B on workstation.
25. Quantization × distillation × sparsity = compounding gains.

Appendix B: Decision Tree — "How to Quantize?"

1. Cloud serving with H100/B200 hardware? → FP8 (W8A8 + FP8 KV); TensorRT-LLM or vLLM.
2. Cloud serving on A100 / older? → \(\operatorname{INT8}\) (SmoothQuant) or \(\operatorname{W4A16}\) (AWQ).
3. Memory-tight / want max throughput? → \(\operatorname{W4A16}\) (AWQ + Marlin) + FP8 KV.
4. Single consumer GPU (24–80 GB)? → GGUF Q4 K M (llama.cpp) or AWQ via vLLM.
5. Fine-tuning on single GPU? → QLoRA (NF4 + LoRA via bitsandbytes / Unsloth).
6. Apple Silicon device? → MLX INT4 / INT8.
7. Mobile / NPU (Qualcomm, ANE, MediaTek)? → INT8 PTQ via OpenVINO / Core ML / QNN.
8. Diffusion model serving? → INT8 W & A; SVDQuant for FLUX 4-bit.
9. MoE serving on workstation? → ktransformers + GGUF Q4 + CPU offload.
10. Training a frontier model from scratch? → FP8 (Transformer Engine) or experimental FP4.

Appendix C: Year-by-Year Quantization Milestones

• 2018–2019: Mixed-precision training (Apex / FP16); BERT INT8 PTQ.
• 2020–2021: TensorRT INT8 PTQ + QAT mature; ZeroQuant; Q-BERT.
• 2022: BF16 mainstream for LLM training; ZeroQuant-V2; SmoothQuant; LLM.int8().
• 2023: GPTQ; AWQ; QLoRA / NF4; bitsandbytes 4-bit; KIVI; FP8 on H100 mainstream.
• 2024: SpinQuant; QuaRot; SVDQuant; HQQ; OmniQuant; GGUF K-quants + IQ-quants; AutoAWQ + AutoGPTQ standardized; FP8 KV cache mainstream; MXFP4 standard.
• 2025: DeepSeek-V3 FP8 native MoE training; BitNet b1.58 follow-ups; FP4 inference on Blackwell; ktransformers consumer MoE; KV cache quantization standard in serving frameworks.
• 2026: FP4 training maturing; microscaling (MXFP) standard hardware; per-expert MoE quantization standard; INT2 / 1.58-bit research; quantization compositions (Q × distill × sparsity) Pareto-frontier.