Tokenization & Context Treatment — Technologies & Tricks

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

Why Tokenization Matters
Text Tokenization
Vision Tokenization: Continuous (Patch Embedding)
Vision Tokenization: Discrete (VQ Family)
Video Tokenization
Audio Tokenization
Action and Robot Tokenization
Multi-modal Token Interleaving
Position Encoding for Long Context
Long-Context Training Recipes
Memory and Recurrence Mechanisms
Context Compression and Distillation
Sequence Packing and Batch-Level Tricks
Inference-Time Context Tricks
Retrieval-Augmented Context (RAG)
Long Video and Multi-Image Context for VLMs
Diffusion-Specific Context Treatment

Appendix A: Twenty-Five Things to Memorize

1. Why Tokenization Matters

1.1 The big picture

A model never sees raw bytes / pixels / waveform; it sees tokens: a vocabulary-indexed sequence whose embedding lookup feeds the first layer. Tokenization is the bridge between the world and the model, and it is the most under-appreciated source of capability and bug in modern systems.

Three properties to optimize:

Compression: fewer tokens per unit content \(\Rightarrow\) longer effective context, lower cost.
Reconstructability: for generation, the tokens must decode back losslessly (or approximately).
Inductive bias: tokens that align with semantic / syntactic units help the model.

1.2 Modality cheat sheet

Modality	Common scheme (2026)	Tokens per unit
Text (English)	BPE / SentencePiece	~1 token / 4 chars
Text (CJK)	BPE / SentencePiece	~1 token / 1.5 chars
Text (code)	BPE w/ code-aware vocab	~1 token / 3.5 chars
Image (continuous)	Patch embedding	\((H/p)(W/p)\), \(p \in \{14, 16\}\)
Image (discrete)	VQ / FSQ / LFQ	256–4096 / image
Image (1D)	TiTok	32 / image
Video (latent)	Causal 3D VAE	\((T/4)\cdot(H/8)\cdot(W/8)\)
Audio (24 kHz)	EnCodec / SoundStream	75 / sec
Action (robot)	Discrete bins or FM	7–14 / step

2. Text Tokenization

2.1 The fundamental algorithms

2.1.1 BPE (Byte-Pair Encoding)

Greedy bottom-up merge of most-frequent symbol pairs:

Initial vocab = all characters / bytes in corpus.
Count adjacent pair frequencies in the corpus.
Merge the most frequent pair into a new symbol; add to vocab.
Repeat until target vocab size reached.

At inference, apply learned merges to input string in the same order.

Used by: GPT-2/3/4 (tiktoken), RoBERTa, Llama 2 (modified), most modern LLMs.

2.1.2 Byte-level BPE (BBPE)

Operate on bytes (256 initial symbols), not Unicode characters. Guarantees no <UNK> ever; covers any text.

Standard since GPT-2.

2.1.3 WordPiece (BERT)

Like BPE but the merge criterion is the likelihood-ratio score:

\[\text{score}(a, b) = \frac{\text{count}(ab)}{\text{count}(a)\cdot\text{count}(b)}.\]

Picks the merge that increases corpus likelihood the most under the unigram model assumption.

Inference: greedy longest-match (## prefix marks subwords).

2.1.4 Unigram (SentencePiece, T5, XLNet)

Define a unigram language model over a candidate vocabulary; train via EM, removing low-probability tokens until target vocab size:

\[P(\mathbf{x}) = \prod_i P(x_i),\]

\[\mathcal{L} = \sum_{\mathbf{w}} \log \sum_{\mathbf{x}\in S(\mathbf{w})} P(\mathbf{x}),\]

with \(S(w)\) = all valid tokenizations of word \(w\). Multiple valid tokenizations per string \(\to\) at training, sample from \(P\); at inference, take the argmax.

2.1.5 SentencePiece (the framework)

Treats input as a raw character / byte stream (no pre-tokenization). Supports BPE and Unigram modes. Handles arbitrary languages without language-specific rules. Used by: T5, Llama 1/2/3, Mistral, Gemma.

2.2 Modern tokenizer specifics

2.2.1 Tiktoken (OpenAI)

cl100k base: GPT-3.5/4, ~100k vocab.
o200k base: GPT-4o, ~200k vocab. Larger vocab = better compression on rare languages and code.

Implementation: byte-level BPE with regex pre-splitting (numbers split into individual digits, words split on whitespace).

2.2.2 Llama 3 / Llama 4 tokenizer

~128k vocab (Llama 3) up from ~32k (Llama 2). Big improvement for non-English languages. SentencePiece \(\to\) BPE.

2.2.3 Gemma, Mistral, Qwen, DeepSeek

Mostly SentencePiece BPE with _32k–200k vocabularies. Vocabulary expansion (continued pretraining) when adding new languages or code support.

2.3 Special tokens to know

[BOS], <s>: beginning-of-sequence.
[EOS], </s>, <|endoftext|>: end-of-sequence.
[PAD]: padding token for batch alignment.
[UNK]: unknown (avoided in byte-level / SentencePiece).
[CLS]: classification (BERT).
[SEP]: separator (BERT).
[MASK]: mask token for MLM.
<image>, <image pad>, <vision start>, <vision end>: VLM placeholders.
Tool / function-calling tokens: <tool call>, <tool response>.
Reasoning tokens: <think>, </think>, <answer>.

2.4 Chat / instruction templates

2.4.1 ChatML (OpenAI)

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is 2+2?<|im_end|>
<|im_start|>assistant
4<|im_end|>

2.4.2 Llama 3 template

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

...<|eot_id|><|start_header_id|>user<|end_header_id|>

...<|eot_id|><|start_header_id|>assistant<|end_header_id|>

2.4.3 Why templates matter

A chat-tuned model is trained to expect exactly this format. Off-template prompts produce noticeably worse outputs. Always use the model's official Jinja template (tokenizer.apply_chat_template).

2.5 Tokenization pitfalls

2.5.1 Numbers

Naive BPE merges "100", "1000" into single tokens. Disastrous for arithmetic. Modern tokenizers (Llama 3, GPT-4o) split into individual digits ("1", "0", "0") via regex pre-split.

2.5.2 Code tokenization

Python-aware regex helps (split on operators, keywords, indentation). StarCoder/CodeLlama tokenizers preserve common indents (" " as one token).

2.5.3 Multilingual fairness

A tokenizer trained on English-heavy corpus emits 2–5× more tokens for the same content in low-resource languages. Costs 2–5× more inference dollars per request and burns context budget. Larger / more balanced vocabularies (Llama 3, GPT-4o) partially fix.

2.5.4 Glitch / unspeakable tokens

Rare BPE merges that occur in vocab-build but rarely in pretraining have undefined behavior (SolidGoldMagikarp-style). Identifiable by extremely high logit variance; production filters should reject.

2.5.5 Detokenization issues

Some tokenizers strip leading whitespace; others encode it. "hello" vs " hello" are different tokens. Streaming generation often requires a decode that handles spacing.

Watch out

Tokenizer choice is an architectural decision with multi-year consequences. Vocab is locked once pretrained; expansion (adding tokens later) is fragile because new embeddings start at zero.

★ 2026 SOTA update — Tokenizer-free byte models

BLT: Byte Latent Transformer groups raw bytes into entropy-based dynamic patches (small byte encoder/decoder + large latent transformer); matches token LLMs at 8B/4T, ~50% inference FLOP savings, robust to noise.
H-Net: end-to-end hierarchical dynamic chunking learns content-dependent segmentation jointly with the model, no tokenizer/heuristics; multi-stage hierarchy matches Transformers of 2x size, strong char-level robustness and multilingual gains.

★ 2026 SOTA update — Superword and adaptive tokenizers

SuperBPE: pretokenization curriculum first learns subwords then 'superwords' bridging whitespace; up to 33% fewer tokens at 200k vocab, +4.0% avg (+8.2% MMLU) and 27% less inference compute vs BPE.
FlexiTokens: byte-level LM with a learnable boundary predictor for variable-length segments; adapts to new domains/languages without retokenization, cuts over-fragmentation, up to +10% downstream.

3. Vision Tokenization: Continuous (Patch Embedding)

3.1 ViT patch embedding

Image \(X \in \mathbb{R}^{H\times W \times 3}\) split into \(N = (H/p)(W/p)\) patches of size \(p \times p\).

Each patch flattened and linearly projected:

\[z_i = W_p \cdot \text{vec}(X_{\text{patch}_i}) + b_p, \qquad W_p \in \mathbb{R}^{d\times 3p^2}.\]

Equivalently: a \(p \times p\) stride-\(p\) convolution with \(d\) output channels.

Standard patch sizes: \(p = 16\) (ViT-B/16 default), \(p = 14\) (DINOv2/CLIP-L), \(p = 32\) (smaller variants).

3.2 Class token

Prepend learnable token \(z_{[\text{CLS}]} \in \mathbb{R}^d\) to the patch sequence. After encoding, \(z^{(L)}_{[\text{CLS}]}\) is used for classification.

Modern alternative: average-pool all patch tokens.

3.3 Position embeddings on patches

Learnable absolute (original ViT): \(L \times d\) table, one per spatial position.
Sinusoidal absolute: as in original Transformer.
2D sinusoidal: independent sin/cos along \(x\) and \(y\), concatenated.
2D-RoPE: modern (Qwen2-VL, FLUX).

3.4 Convolutional stem

ViT-22B and others replace single-layer patchify with a small conv stem (e.g., 3 stride-2 convs). Empirically more stable; trades a small spatial bias for training robustness.

3.5 Pixel shuffle / unshuffle for token compression

Reshape \((H, W, C) \to (H/r, W/r, Cr^2)\); linearly project to target channel: tokens reduced by \(r^2\).

Standard in InternVL family (\(r = 2\), 4× fewer tokens, manageable for high-res).

3.6 AnyRes (LLaVA-NeXT)

For high-resolution inputs:

Resize to a global thumbnail (\(336^2\)).
Tile the original into chunks of \(336^2\).
Encode thumbnail + each tile separately.
Concatenate token sequences with separators.

\(N\) tiles \(\to N + 1\) sub-images; token count grows linearly, OCR / dense reasoning improves dramatically.

3.7 Native dynamic resolution (Qwen2-VL, InternVL3)

Process the image at its native aspect ratio: assign 2D-RoPE coordinates to each patch from absolute (row, col).

Token count \(\approx HW/p^2\). Combined with pixel unshuffle for compression.

3.8 Image cropping for training (RandomResizedCrop)

Augmentation: random crop with random scale (0.08–1.0) and aspect ratio (3/4–4/3), then resize to fixed input size. Standard for ImageNet/CLIP/DINO training; less common for VLM pretraining.

3.9 Register tokens

Append \(K \sim 4\) extra learnable "register" tokens to the patch sequence. They absorb high-norm artifacts that otherwise dump on random patches, cleaning attention maps and improving dense-task quality (DINOv2, V-JEPA).

4. Vision Tokenization: Discrete (VQ Family)

4.1 Why discrete tokens?

Required for autoregressive generation (Parti, MUSE, MAGVIT, Chameleon, Emu3).
Enables shared vocab with text in native multimodal models.
Enables efficient masked image modeling (MaskGIT, MAGE).

4.2 VQ-VAE basics

Encoder \(E : X \mapsto z_e \in \mathbb{R}^{H'\times W' \times d}\). Codebook \(\{e_1, \ldots, e_K\} \subset \mathbb{R}^d\). Quantize each spatial position by nearest neighbor:

\[z_q^{(i,j)} = e_{k^*}, \qquad k^* = \arg\min_k \left\| z_e^{(i,j)} - e_k \right\|_2.\]

Decoder \(D : z_q \mapsto \hat{X}\). Loss with stop-grad straight-through estimator:

\[\mathcal{L}_{\text{VQ}} = \| X - D(z_q) \|^2 + \| \text{sg}(z_e) - e_{k^*} \|^2 + \beta \| z_e - \text{sg}(e_{k^*}) \|^2.\]

Last term (\(\beta \sim 0.25\)) is the commitment loss that pulls encoder output toward codebook entries.

4.3 VQ-VAE-2 (multi-scale)

Hierarchical: top-level codes capture global structure (e.g. pose), bottom-level codes capture fine detail (e.g. texture). Generated autoregressively top-to-bottom.

4.4 VQ-GAN

VQ-VAE + adversarial loss + perceptual (LPIPS) loss:

\[\mathcal{L}_{\text{VQGAN}} = \mathcal{L}_{\text{VQ}} + \lambda\,\mathcal{L}_{\text{GAN}} + \lambda_p\,\mathcal{L}_{\text{LPIPS}}.\]

Sharper reconstructions; standard backbone for Stable Diffusion's KL-VAE alternative.

4.5 Codebook-collapse mitigations

Persistent problem: many codes go unused.

EMA codebook: update codes via \(e_k \leftarrow \alpha e_k + (1 - \alpha)\bar{z}_k\) instead of gradient.
Dead-code resampling: replace unused codes with sampled features periodically.
Random restart: re-init dead codes from current encoder outputs.
Code dropout: at training, randomly drop codes to force distribution.

4.6 LFQ (Lookup-Free Quantization, MAGVIT-v2)

Project encoder output to dimension \(L\), sign-quantize to \(\{-1, +1\}^L\):

\[q = \text{sgn}(z), \qquad \text{vocab size} = 2^L.\]

No codebook. No collapse by construction. Decoder receives the sign vector. Loss combines reconstruction + entropy regularizer that promotes uniform usage:

\[\mathcal{L}_{\text{LFQ}} = \mathcal{L}_{\text{recon}} + \lambda_e\,\mathcal{L}_{\text{ent}}, \qquad \mathcal{L}_{\text{ent}} = \mathbb{E}[H(P(q|z))] - H(\mathbb{E}_z[P(q|z)]).\]

Vocabularies up to \(2^{18}\) practical. Used in MAGVIT-v2, Cosmos.

4.7 FSQ (Finite Scalar Quantization)

Each dim of \(z \in \mathbb{R}^d\) rounded to a small set \(\{-K_i, \ldots, K_i\}\):

\[q_i = \text{round}(K_i \cdot \tanh(z_i)), \qquad |\text{vocab}| = \prod_i (2K_i + 1).\]

No codebook, no entropy reg. Simpler than LFQ. Common: 8 dims with levels [8, 8, 8, 5, 5, 5], vocab ~64k.

4.8 BSQ (Binary Spherical Quantization)

Project to unit sphere then sign-quantize per coordinate.

Combines LFQ's binary structure with spherical normalization for stability.

4.9 TiTok (1D tokenization)

Learn a 1D sequence of ~32 latent tokens per \(256^2\) image (vs 256+ for 2D). Cross-attention from learnable queries over the encoder's 2D features. Trade spatial structure for sequence brevity; surprisingly strong downstream generation quality.

4.10 Cosmos Tokenizer (NVIDIA)

Joint image + video tokenizer with continuous and discrete variants. Causal 3D structure (temporal causal, spatial non-causal). Supports up to 8K resolution.

4.11 Comparison summary

Method	Vocab	Codebook	Collapse-prone	Use
VQ-VAE	1k–16k	yes	yes	legacy
VQ-GAN	8k–16k	yes	yes	SD-VAE alt
MAGVIT (VQ)	1k	yes	yes	video gen
LFQ (MAGVIT-v2)	\(2^{10}\)–\(2^{18}\)	no	no	video / image
FSQ	up to ~64k	no	no	general
BSQ	\(2^L\)	no	no	general
TiTok	varies	yes	some	1D fast gen
Cosmos	varies	yes/no	no	video at scale

5. Video Tokenization

5.1 Frame-by-frame 2D tokenization

Apply image tokenizer per frame independently. Simplest; no temporal compression. Used in early video gen (ModelScope T2V).

5.2 Tubelet patches (3D ViT)

Patches are 3D blocks of size \(p_t \times p_h \times p_w\). Token count: \((T/p_t)(H/p_h)(W/p_w)\). Standard in ViViT, VideoMAE.

5.3 Causal 3D VAE

Encoder: 3D convolutions with causal temporal conv (current frame depends only on past). Compression typically \(4\times\) temporal, \(8\times\) spatial. Decoder symmetric.

\[T \times H \times W \; \xrightarrow{\text{causal 3D VAE}} \; (T/4) \times (H/8) \times (W/8) \times C.\]

Standard in Sora, Open-Sora, CogVideoX, Wan, MovieGen.

5.4 First-frame special handling

Causal VAE often handles the first frame asymmetrically (it has no past). Common trick: pad first frame as its own past, or use a separate first-frame encoder.

5.5 MAGVIT-v2 video tokenization

Joint image + video LFQ tokenizer. A still image is the \(T = 1\) case. Same tokenizer trains both modalities; downstream model is unified.

5.6 Cosmos Video Tokenizer

Continuous (CV) and discrete (DV) variants. Continuous gives ~8× spatial × 8× temporal compression for diffusion. Discrete uses FSQ at ~64k vocab for autoregressive video models.

5.7 Token budget per second of video

At 24 fps, \(H = W = 512\), \(p = 8\) spatial ×4 temporal compression:

\[\text{tokens / sec} = 24 \cdot 512^2 / (8^2 \cdot 4) \approx 24{,}576.\]

1-minute clip ~1.5M tokens at this rate. Compression remains a major frontier.

6. Audio Tokenization

6.1 Why audio tokenization?

Generative audio (MusicGen, AudioLM, Suno, Udio), TTS (NaturalSpeech, VALL-E), audio understanding (Whisper variants), joint audio-video generation (MovieGen, Veo 3) all require tokenized audio.

6.2 SoundStream

RVQ (Residual VQ) on raw waveform. Encoder compresses 24 kHz waveform to 75 Hz latents; \(N\) levels of VQ quantize the residual at each step:

\[z_0 = z_e, \qquad q_n = Q_n(z_{n-1}), \qquad z_n = z_{n-1} - q_n.\]

Reconstruct: \(\hat{z} = \sum_n q_n\). Typical \(N = 8\), vocab 1024 per level.

6.3 EnCodec (Meta)

Like SoundStream with adversarial loss. 75 tokens/sec at 24 kHz, 8-level RVQ. Used in MusicGen, AudioLM.

6.4 WaveTokenizer (single-codebook)

One codebook with very large vocab (e.g., 4096). Simpler downstream modeling (no RVQ stacking) at slight quality cost.

6.5 Mel-spectrogram tokens

Some pipelines tokenize the mel-spectrogram instead of raw waveform (more parameter-efficient). Vocoder (HiFi-GAN, BigVGAN) decodes back to waveform.

7. Action and Robot Tokenization

7.1 Discrete action tokens (RT-2, OpenVLA)

Each action dimension binned into \(K\) discrete tokens (typically \(K = 256\)). For a 7-DoF arm: 7 tokens per timestep. Embedded into the LLM vocab so the same model predicts text or actions.

7.2 Action chunks (ACT, π0)

Predict \(H\) future actions per step (action chunk). Reduces compounding error and allows temporal smoothing via overlapping ensemble at inference.

7.3 Continuous action via flow matching (π0)

Small flow expert head produces continuous action vectors \(a_t \in \mathbb{R}^d\) via flow matching:

\[\mathcal{L}_{\text{FM}} = \mathbb{E}\left\| v_\theta(a_t, t, o) - (a_1 - a_0) \right\|^2.\]

Avoids discretization loss for fine manipulation; integrates with VLM backbone.

7.4 Multi-embodiment vocab

For cross-embodiment training (RT-X, OpenVLA), align action dimensions across robots via a canonical action space; each embodiment has its own tokenizer head.

8.1 Interleaved sequences

A modern VLM input is a single token sequence with image / text / video tokens interleaved. Common patterns:

LLaVA: [image tokens] What is in this image? → assistant: ...
Multi-image / interleaved: Compare [img1] and [img2]. → The first ...
Video as image sequence: [frame1] [frame2] ... [frameT] Caption: → ...

8.2 Modality-specific embedding tables

Three options:

Shared vocab (Chameleon): image tokens occupy reserved vocab range \([V_{\text{text}}, V_{\text{text}} + V_{\text{img}})\); one embedding table.
Separate tables: text tokens and image tokens have separate embeddings, summed or concatenated by modality flag.
Projection from continuous (LLaVA / InternVL): image patches stay continuous, projected to LLM hidden via MLP / Q-Former; no image vocab.

8.3 Modality boundary tokens

<image>, </image>: Qwen-VL.
<vision start>, <vision end>: Qwen2-VL.
<image pad>: placeholder used inside templates, replaced by encoded patch features.

These help the model learn to switch attention patterns at modality boundaries.

8.4 Native multimodal training

Chameleon: discrete image tokens via VQ; train next-token on mixed text + image sequences.
Show-o: text AR + image discrete diffusion in same backbone.
Transfusion: text AR + image continuous diffusion in same backbone.
Janus / Janus-Pro: decoupled image encoders for understanding vs generation, shared LLM.
Emu3: pure next-token prediction over text + image + video tokens.

9. Position Encoding for Long Context

9.1 The extrapolation problem

A model trained at context length \(L_{\text{train}}\) generally degrades at \(L_{\text{eval}} > L_{\text{train}}\). Extending context cleanly requires position encoding that extrapolates.

9.2 Position Interpolation (PI)

Linearly compress positions: \(p \to p \cdot L_{\text{train}}/L_{\text{eval}}\). Cheap, quick, but sacrifices high-frequency detail. Needs short fine-tuning (~1B tokens) to recover.

9.3 NTK-aware RoPE

Rather than uniform interpolation, scale the base wavelength so high-frequency components stay almost intact and low-frequency ones stretch:

\[\theta'_i = \theta_i \cdot \left(\frac{L_{\text{eval}}}{L_{\text{train}}}\right)^{-2i/(d-2)}.\]

Essentially keeps fine-grained position info while extending coarse-grained.

9.4 YaRN (Yet another RoPE extensioN)

Piecewise rescale by frequency band:

High-freq dims: keep unchanged (preserve local info).
Low-freq dims: linearly interpolate (extend range).
Mid-freq dims: NTK-style smooth ramp.

Plus an inverse-temperature term in attention. Achieves 10–32× context extension with light fine-tuning.

9.5 LongRoPE

Search for per-dimension rescaling factors via evolutionary algorithm on a long-context evaluation.

Pushes Llama-2 to 2M context.

9.6 Self-Extend

Inference-time only: bin positions in groups of \(G\) at distant ranges, keep fine positions only locally. No fine-tuning.

Effective for \(2\)–\(4\times\) extension.

9.7 ALiBi for extrapolation

ALiBi inherently extrapolates because the bias is a continuous function of distance, not a learned table. Trained at 2K, runs at 16K reasonably. Lower asymptotic quality than RoPE+YaRN at same training.

★ 2026 SOTA update — Near-lossless RoPE scaling

LongRoPE2: fixes under-trained high RoPE dims via needle-driven perplexity evolutionary search + mixed-context-window training; extends LLaMA3-8B to 128K keeping >98.5% short-context with only 10B tokens (80x cheaper). Used in Phi-4-mini.

10. Long-Context Training Recipes

10.1 Continued pretraining at longer context

Standard recipe: pretrain at \(L_0\) (e.g., 4K), then continued-pretrain at \(L_1 \in \{16K, 32K, 128K\}\) on a curated long-context corpus. Often two stages: 4K \(\to\) 32K \(\to\) 128K.

10.2 Long-context data curation

Sources: code repositories, books, scientific papers, web pages with long threads. Filter by length and remove templated repetition. Shuffle to balance topics.

10.3 Synthetic long-context data

Concatenation: pack multiple short documents with delimiters.
Question-over-document: generate Q+A pairs about specific spans of long docs (forces the model to retrieve).
Distractor injection: insert irrelevant documents between query-relevant ones.

10.4 Sequence parallelism for training long sequences

Training at 100K+ tokens requires sequence parallelism (and often ring attention). Activation memory dominates without it.

10.5 Long-context evaluation

Needle-in-a-Haystack (NIAH): insert a fact at random position in long context; model must retrieve. Tests retrieval ability.
RULER: extends NIAH with multiple keys, multi-hop, aggregation, QA.
LongBench, ∞Bench: real long-document QA, summarization, code completion.
LV-Eval: multi-hop QA at lengths up to 256K.

Watch out

A model can pass NIAH and still fail real long-document tasks. NIAH measures retrieval; real tasks need synthesis, reasoning, and selective ignoring of distractors. Use multiple eval suites.

11. Memory and Recurrence Mechanisms

11.1 Sliding window + sink (recap)

Mistral, StreamingLLM: keep first \(k \sim 4\) "sink" tokens always in cache plus sliding window of \(w \sim 4096\) recent.

Effective unbounded streaming.

11.2 Memorizing Transformer

Attention augmented with a kNN-retrieval over a large external memory of past KVs. At inference, query the memory; concat top-k to local KVs.

11.3 Recurrent Memory Transformer (RMT)

Chunk input into segments; pass a small set of memory tokens between segments. Read-write memory similar to Neural Turing Machine, simpler.

11.4 Compressive Transformer

Maintain two memories: short (recent KVs) and long (compressed older KVs via pooling / conv).

Trades exactness for capacity.

11.5 RWKV / RetNet / Mamba

Linear-recurrence architectures with constant per-token state \(\Rightarrow\) effectively unbounded context at \(O(d^2)\) memory.

RWKV / Mamba have parallel-train, recurrent-inference duality. Trade-off: weaker exact retrieval than attention at the same scale.

11.6 Token-merge / pruning during inference

Periodically merge or drop tokens that have low attention from current queries (DiffPlanner, H2O).

Keeps cache small without full compression.

★ 2026 SOTA update — Trainable sparse attention

NSA: Native Sparse Attention, hardware-aligned + natively trainable; three parallel branches (coarse compressed tokens, fine-grained selected blocks, sliding local), matches/beats full attention with large long-context speedups end-to-end.
MoBA: Mixture of Block Attention applies MoE routing to attention blocks (learned, less-structure), toggles between full and sparse; powers Kimi long-context serving.

12. Context Compression and Distillation

12.1 LLMLingua / LLMLingua-2

Prompt compression by token deletion. A small classifier scores each token; drop low-importance tokens. Achieves 5–20× compression with minor quality loss for long instructions / RAG contexts.

12.2 AutoCompressors

Train a small adapter to compress a long context into \(K\) summary tokens that are concatenated to a short query.

Effectively a learned context summarizer.

12.3 Activation compression

Compress KV cache via:

Quantization (INT8, INT4, FP8, KIVI).
Low-rank decomposition (MLA).
Token pruning (H2O, FastV).
Streaming (sliding window + sink).

12.4 Hierarchical / multi-scale context

Process long context at multiple granularities: coarse summaries at top, detailed chunks below.

Used in MovieChat (long video), MA-LMM, Goldfish.

★ 2026 SOTA update — Query-agnostic KV compression

KVzip: scores KV importance by how well the LLM can reconstruct the original context from cache, then evicts; query-agnostic so a compressed cache is reusable across many queries. 3-4x smaller cache, ~2x faster decoding, near-lossless to 170K (LLaMA3.1/Qwen2.5/Gemma3). Contrast query-aware SnapKV/PyramidKV/Ada-KV which degrade under multi-query reuse.

13. Sequence Packing and Batch-Level Tricks

13.1 Naive padding

Pad each sequence to batch max length. Wastes compute on padding tokens.

13.2 Bucket batching

Sort sequences by length; batch similar lengths together. Reduces padding waste.

13.3 Sequence packing

Concatenate multiple short sequences into one row up to max length, separated by EOS. Use block-diagonal attention mask so each example only attends to itself:

\[M_{ij} = \begin{cases} 0 & i, j \text{ same example} \\ -\infty & \text{else} \end{cases}.\]

Eliminates padding; standard in modern training (Megatron, Flash-Attention's varlen API).

13.4 Document packing for pretraining

Pack documents to fixed length, separated by EOS. Common variants:

No special handling: documents may be split mid-sentence; loss on EOS prediction trains the boundary.
Within-document attention: block-diagonal mask so sequences across doc boundaries don't attend.

13.5 Sample packing for SFT

For multi-turn instruction tuning, pack multiple short conversations into one row with proper masking and per-conversation loss masking (don't compute loss on user turns).

13.6 Loss masking in SFT

Compute loss only on assistant tokens (not user / system).
Mask is the labels array with \(-100\) on positions to ignore.
In packed batches, must mask boundaries between independent examples.

14. Inference-Time Context Tricks

14.1 Prefix caching

The KV cache for a fixed prefix (e.g. system prompt) is computed once and reused across requests.

Linear speedup proportional to prefix length.

14.2 Prompt caching (Anthropic API)

First ~5 minutes after a prompt is sent, the API caches the KV; subsequent requests with the same prefix benefit from up to 90% cost reduction. Granular checkpoints supported.

14.3 Paged KV cache (vLLM)

KV stored in fixed-size blocks; per-request page table maps logical \(\to\) physical. Enables shared prefix across requests, no fragmentation, and continuous batching.

14.4 Continuous batching

At each token step, accept new requests / drop completed ones. Per-token batching at the kernel level. Bumps throughput 5–10× over static batching for variable-length workloads.

14.5 Speculative decoding

Draft model proposes \(k\) tokens; target verifies all in one forward. Effective speedup 2–3×. Requires the target's KV cache to support the new tokens; tree attention generalizes (Medusa, EAGLE).

14.6 KV reuse across edits / rewrites

Tools like Cursor / Cline cache the file's KV; minor edits invalidate only the suffix. Same idea: incremental computation when prefix is stable.

★ 2026 SOTA update — Cache-augmented generation

CAG: preload the whole knowledge base into context and precompute its KV cache once, then answer queries with zero retrieval step; simpler/faster than RAG when the corpus fits the (long) context window. Hybrid CAG+RAG adds selective retrieval for large/updating corpora.

15. Retrieval-Augmented Context (RAG)

15.1 The RAG pattern

Encode query with embedding model.
Retrieve top-k relevant chunks from a vector index.
Inject chunks into prompt: <chunk1> <chunk2> ... <query>.
Generate answer.

15.2 Chunking strategies

Fixed-size (e.g., 512 tokens): simple, OK for most cases.
Semantic chunking: split at topic boundaries (using a chunker model or sentence embeddings).
Hierarchical: chunks + parent-doc summaries.
Sliding window with overlap: avoids boundary loss.

15.3 Embedding models

BGE, E5, GTE: open-source dense retrievers.
OpenAI text-embedding-3-large, Voyage AI, Cohere Embed v3.
Multi-vector / late interaction (ColBERT, ColPali): one vector per token, max-similarity per query token.
Multi-modal (CLIP, SigLIP): text \(\leftrightarrow\) image retrieval.

15.4 Re-ranking

After top-k retrieval, re-rank with a cross-encoder (BGE-Reranker, Cohere Rerank).

Returns higher-quality top-N for the LLM. Latency cost: \(k\) extra small-model passes.

15.5 Long-context vs RAG

Long-context wins: cohesive reasoning across entire context, no chunking artifacts, simpler stack.
RAG wins: massive corpora that don't fit, dynamic / freshly updated content, lower per-query cost.
In practice, hybrid: RAG to fetch candidates, then long-context to reason.

15.6 Query rewriting and decomposition

Multi-hop questions: decompose into sub-questions; retrieve per sub-question; aggregate. Used in Self-RAG, LangGraph patterns, agentic RAG.

15.7 GraphRAG

Build a knowledge graph from corpus (entities + relations), retrieve sub-graphs by query similarity, inject as context. Better for multi-hop reasoning over structured knowledge.

16. Long Video and Multi-Image Context for VLMs

16.1 Token budget management

At 256 tokens/frame (after pixel unshuffle), 1 fps for 1 hour = 920k tokens. Fitting requires aggressive compression.

16.2 Temporal pooling

Average / max pool frame tokens across \(T\) frames before feeding LLM. Crude but cheap; loses temporal detail.

16.3 Token merging across time (ToMe)

Apply Token Merging across the temporal dim: merge tokens with high cosine similarity to neighbors. Adaptive compression per-clip.

16.4 Q-Former summarization (BLIP-2 style)

\(n_q \sim 32\) queries per frame chunk; cross-attend over the chunk's tokens; output fixed-size summary. Then concat summaries across chunks.

16.5 Hierarchical memory (MovieChat, MA-LMM)

Short-term memory: recent frames at full resolution.
Long-term memory: older frames merged via similarity-based clustering.
On query: retrieve relevant long-term entries + short-term, feed LLM.

16.6 Caption-then-reason (Goldfish, MM-VID)

Generate captions per frame / clip with a small VLM, store in text. On query, retrieve relevant captions; reason in text-only mode. Cheap, scalable to multi-hour video.

16.7 Sparse temporal sampling

For most VLMs, sample 8–64 frames uniformly from a video; encode each; concat. Loses fine-grained motion but works for global understanding.

16.8 Token compression via 3D-RoPE

With 3D-RoPE on \((t, h, w)\), the model can natively handle variable resolution / fps without per-token modality flags. Combined with adaptive sampling, scales to longer videos.

16.9 Long-context video VLMs (2026)

Qwen2.5-VL: 1-hour video native.
Gemini 2.5: multi-hour native.
LongVA, LongVU, LongVILA: open with ring attention + curated long video data.

17. Diffusion-Specific Context Treatment

17.1 Text conditioning in U-Net diffusion

Cross-attention from image tokens to frozen text encoder (CLIP for SD 1.x; CLIP+OpenCLIP for SDXL; CLIP+T5 for SD 3 / FLUX).
Text embedding length: 77 tokens (CLIP), up to 512 tokens (T5).
Negative prompt: a separate text embedding fed via CFG.

17.2 MM-DiT joint context

SD3, FLUX: text + image tokens concatenated and processed by joint self-attention. Text uses learnable position; image uses 2D-RoPE.

17.3 Long-prompt handling

CLIP encoder caps at 77 tokens; longer prompts use chunked encoding then concat (SDXL, A1111 long-prompt mode).
T5 (in SD3 / FLUX) supports 512 tokens, much better long-prompt fidelity.
Compel / weighted prompts: per-token weighting of text embeddings to emphasize tokens.

17.4 Image as context

Img2img / SDEdit: encode source, partial-noise, denoise.
IP-Adapter: image encoded to CLIP, injected via parallel cross-attn.
ReferenceNet: trainable U-Net copy on reference, KVs concatenated.
InstantID, PuLID: face encoder with attention / feature injection.

17.5 ControlNet conditioning

Trainable copy of encoder takes the control input (canny, depth, pose, segmentation, scribble); outputs added to base U-Net via zero-conv connections.

17.6 Long video diffusion: context across frames

Joint denoising: full clip in one forward pass; expensive.
Chunked + autoregressive: generate clip, condition next clip on last frames.
Causal generation: train with causal temporal mask so the model can extend.
Anchor-frame conditioning: keyframes provided; in-between frames generated.

17.7 Diffusion KV / activation cache (DeepCache, TGATE, Block-Cache)

Cache U-Net (or DiT) intermediate activations across consecutive denoising steps; refresh every \(k\) steps. 2–4× inference speedup with negligible quality loss.

17.8 Reference-only conditioning patterns

Concatenate noisy + reference encoded as "parallel batch," allow self-attn to mix them.
Used for character consistency in animation, identity preservation in editing.