RL Training Strategies & Practical Recipes — Every recipe and engineering trick that turns a policy-optimization paper into a model that actually trains

22 sections · classical RL through 2026 RLHF hyperparameters, stability, distributed, debugging

April 2026 · Version 1.0

1. Pre-Training and Cold-Start

1.1 The four canonical starting points

1. Pure RL from scratch: random init policy. Classical control, AlphaZero-style self-play.
2. Pure RL from a base model (R1-Zero): pretrained LM, no SFT, jump straight to RL on verifiable rewards. Works only above a critical scale.
3. SFT → RL: supervised warm-start, then RL. Classical InstructGPT recipe.
4. SFT → DPO/RL → rejection-sampling SFT → RL again (DeepSeek-R1): multi-stage interleaved SFT and RL.

1.2 When pure RL works (R1-Zero)

• Base model strong enough that random sampling occasionally produces correct outputs.
• Reward must be verifiable (math, code, format) so noise is zero.
• GRPO (or REINFORCE-family) handles long sequences without value head.
• Patience: takes thousands of steps for behavior to emerge.

1.3 When SFT-first wins

• Domain shift from base (medical, legal, niche language).
• Output format is unusual (require strict JSON, multi-step structured reasoning).
• Cold-start needed for stability.
• RM-based RL where exploration cost is high.

1.4 R1-style 4-stage recipe

Key

DeepSeek-R1 multi-stage pipeline.

1. Cold-start SFT: thousands of curated long-CoT examples.
2. RL with GRPO: verifiable math/code rewards + format + language-consistency rewards.
3. Rejection-sampling SFT: collect \(\sim 600\) k (correct, well-reasoned) traces from the RL'd model + \(\sim 200\) k general SFT data, retrain.
4. Final RL: another GRPO pass for safety / helpfulness / general reasoning.

Each stage strengthens different capabilities; interleaving avoids the brittleness of pure-RL or pure-SFT.

1.5 The cold-start data choice

For RLHF cold start: \(\sim 10\) k–100k high-quality demos. For reasoning cold start: \(\sim 1\) k–10k carefully curated long-CoT exemplars. Quality > quantity by orders of magnitude.

2. Rollout Strategies and Data Collection

2.1 On-policy vs off-policy

On-policy (PPO, A2C): trajectories must come from the current policy or a very recent one. Sample-inefficient; clean theory.

Off-policy (DQN, SAC, TD3): trajectories from any policy work, with replay buffer. Sample-efficient; trickier stability.

Near-on-policy (GRPO, PPO with \(\sim 1\) epoch): one or two updates per rollout; effectively on-policy. Modern LLM RL leans here.

2.2 Vectorized environments

Run \(N\) parallel envs in one process; step them together. \(N\) typically 8–64 for classic Atari, 1024+ for IsaacGym.

Key

Vectorized rollout. Increase \(N\) env parallelism until either the GPU is saturated (large policy) or env step is the bottleneck. Then switch to multiprocessing or distributed actors.

2.3 Asynchronous actor-learner (IMPALA / A3C)

Many actors collect data in parallel, push to a central learner. V-trace corrects for the policy lag:

\[\bar{\rho}_t = \min(\rho_t, \bar{\rho}), \qquad \hat{V}(s_t) = V(s_t) + \sum_{k\ge t} \gamma^{k-t}\Big(\prod_{i<k} c_i\Big)\bar{\rho}_k\,\delta_k V.\]

\(\bar{\rho}\) is the truncation level (1.0 typical). Necessary at large scale.

2.4 Synchronous distributed actors (PPO-distributed)

All actors collect a fixed number of timesteps; learner waits for sync; one update; broadcast new weights. Easier to reason about; standard in modern PPO infra (Ray RLlib, OpenRLHF).

2.5 Massively-parallel sim (Isaac Lab, MJX, Genesis)

\(10^4+\) parallel envs on a single GPU. Trains quadruped locomotion in hours. Required for modern robotics RL.

2.6 Sample efficiency knobs

• Replay buffer size (off-policy): \(10^5\)–\(10^7\) transitions.
• Update-to-data ratio (UTD): how many gradient steps per env step. SAC/TD3: 1–4. "REDQ"-style high-UTD: 20+.
• Multiple PPO epochs per rollout: 4–10.
• Mini-batch size: a fraction of rollout size.

2.7 Trajectory length per rollout

• Atari/Mujoco PPO: 128–2048 steps per env \(\times\) \(N\) envs.
• LLM RLHF: prompt length + max generation (usually 512–4096 tokens).
• GRPO: \(G = 8\)–64 responses per prompt, generations of 512–16384 tokens (long-CoT).

3. Batch Composition and Update Frequency

3.1 The key ratios

• Effective batch: rollout size \(\times\) epochs / mini-batch size.
• Update frequency: gradient steps per environment step.
• KL budget per update: how far the policy can drift before we stop.

3.2 PPO default ratios

• Rollout: \(T = 2048\) steps, \(N = 8\) envs \(\Rightarrow\) 16k samples.
• Epochs: 4–10 over the rollout.
• Mini-batch: 64 (continuous control), 256–1024 (Atari).

3.3 LLM RLHF default ratios

• Per step: \(B\) prompts, sample 1 response each (PPO) or \(G\) responses each (GRPO).
• PPO: 1–4 epochs over the rollout.
• Mini-batch: typically the full rollout (1 epoch) at large \(B\), or partition.
• KL coefficient \(\beta\): 0.001–0.1 for PPO; 0.0–0.04 for GRPO.

3.4 GRPO group size \(G\)

• Small \(G\) (4–8): cheaper per prompt; higher variance, slower convergence.
• Large \(G\) (32–64): better baseline, slower wall-clock; better for hard prompts.
• DeepSeek-R1: \(G = 16\) as default; some recipes go to 64.

Watch out

A group where all responses succeed (or all fail) gives zero advantage signal. Filter these prompts or use dynamic sampling (DAPO) to re-sample.

4. Hyperparameter Recipes

4.1 Discount factor \(\gamma\)

• \(\gamma = 0.99\): most continuous control, Atari (effective horizon \(\sim 100\)).
• \(\gamma = 0.999\): long-horizon (locomotion across minutes).
• \(\gamma = 1\): episodic with no time bonus; handle bootstrapping carefully.
• LLM PPO: episodic, \(\gamma = 1\) (or \(\gamma = 0.99\) on per-token rewards with KL bonus).

4.2 GAE \(\lambda\)

• \(\lambda = 0\): TD(0); high bias, low variance.
• \(\lambda = 1\): full Monte Carlo; low bias, high variance.
• Default: \(\lambda = 0.95\) for PPO across most domains.

4.3 PPO clip \(\epsilon\)

• \(\epsilon = 0.2\) default; works for most things.
• \(\epsilon = 0.1\): very stable, slow.
• \(\epsilon = 0.3\): faster, riskier.
• DAPO: asymmetric \(\epsilon^+ > \epsilon^-\) for upward latitude.

4.4 KL coefficient \(\beta\)

• Adaptive (KL-Penalty PPO): target KL \(\sim 0.01\), double \(\beta\) if exceeded, halve if much lower.
• Fixed: \(\beta = 0.01\)–0.1 for RLHF.
• GRPO often uses \(\beta = 0.0\)–0.04 (lighter regularization).
• Per-token KL accumulates fast; per-sequence KL is more stable.

4.5 Learning rate

• Atari/MuJoCo PPO: \(3 \times 10^{-4}\), linear or cosine anneal to 0.
• Continuous control SAC: \(3 \times 10^{-4}\) for both actor and critics.
• LLM RLHF (full FT): \(1 \times 10^{-6}\)–\(5 \times 10^{-6}\).
• LLM RLHF (LoRA): \(5 \times 10^{-5}\)–\(1 \times 10^{-4}\).
• DPO: \(5 \times 10^{-7}\)–\(5 \times 10^{-6}\) (lower than SFT).

4.6 Adam epsilon (the famous one)

\(\epsilon_{\mathrm{Adam}} = 10^{-5}\) for RL (default \(10^{-8}\) destabilizes Atari PPO!). Documented in CleanRL.

4.7 Entropy coefficient

• Discrete actions (Atari): \(c_e = 0.01\) standard.
• Continuous (Gaussian policy): \(c_e = 0\) usually OK because Gaussian is already entropic.
• LLM RL: \(c_e = 0\) typically; KL anchor handles regularization.
• Anneal down over training as policy specializes.

4.8 Value function coefficient

\(c_v = 0.5\) for PPO. Higher (1–2) when value loss is dominating; lower if value-loss training is unstable.

5. Stability Tricks

5.1 Reward normalization / scaling

• Running mean/std of returns; divide rewards by std (not by mean) for scale invariance.
• Avoid normalizing by mean: introduces bias.
• Reward clipping (\(\pm 10\)): bound spike effects.

5.2 Observation normalization

Maintain running mean / std of observations; subtract mean, divide by std, clip \(\pm 5\). Necessary for many MuJoCo / Atari policies.

5.3 Value function clipping

Key

Clipped value loss.

\[V_t^{\mathrm{clip}} = V_{\mathrm{old}}(s_t) + \mathrm{clip}\big(V_\theta(s_t) - V_{\mathrm{old}}(s_t),\, -\epsilon,\, \epsilon\big),\]

\[\mathcal{L}_V = \max\big((V_\theta - R)^2,\ (V^{\mathrm{clip}} - R)^2\big).\]

Mirrors the policy clip, prevents value-net overshoot.

5.4 Gradient clipping

Global norm clip: \(g \leftarrow g \cdot \min(1, \tau / \|g\|)\), \(\tau = 0.5\) standard. Prevents single-batch divergence.

5.5 EMA / target networks

• Off-policy critics (DQN, SAC, TD3): hard target sync every \(K\) steps, or Polyak \(\bar{\theta} \leftarrow \tau\theta + (1 - \tau)\bar{\theta}\), \(\tau \sim 0.005\).
• Reference policy in DPO/GRPO: keep frozen (no EMA needed); or use slow EMA in iterative DPO.
• Generative SSL targets (DINOv2, BYOL): EMA \(\tau \sim 0.999\).

5.6 Loss spike mitigation

• Skip-bad-batches: if gradient norm \(> K\sigma\) above EMA, skip the step.
• Save checkpoint every \(N\) updates; on persistent spike, rollback.
• Detect entropy collapse and halve learning rate.

5.7 Adaptive KL regularization

Key

Adaptive \(\beta\) (PPO-Penalty).

\[\beta \leftarrow \begin{cases} 2\beta & \text{if } \mathbb{E}[\mathrm{KL}(\pi_{\mathrm{old}}\|\pi_\theta)] > 1.5\,d_{\mathrm{target}} \\[2pt] \beta/2 & \text{if } \mathbb{E}[\mathrm{KL}] < d_{\mathrm{target}}/1.5 \\[2pt] \beta & \text{else} \end{cases}\]

\(d_{\mathrm{target}} \sim 0.01\) for most LLM RL.

5.8 Trust-region and natural gradient

TRPO uses CG to compute \(F^{-1}g\) + line search for guaranteed improvement; PPO approximates with clip. ACKTR uses K-FAC for cheaper natural gradient.

5.9 Whitening advantages

Per minibatch: \(\hat{A} \leftarrow (\hat{A} - \mathrm{mean}(\hat{A})) / (\mathrm{std}(\hat{A}) + 10^{-8})\). Standard for PPO; not for GRPO (whose advantage is already group-relative).

6. Exploration Strategies

6.1 Action-space exploration

• \(\epsilon\)-greedy with \(\epsilon\) annealed from \(1.0 \to 0.05\) over \(10^6\) steps (DQN).
• Boltzmann (softmax over \(Q\) values) with annealed temperature.
• Gaussian noise on actions (DDPG): \(\sigma\) from \(0.5 \to 0.05\).
• Ornstein-Uhlenbeck noise (DDPG original): correlated; mostly replaced by Gaussian.
• Parameter noise (NoisyNet): factorized noise on weights; better state-dependent exploration.

6.2 Entropy bonus annealing

Key

Entropy schedule. Start \(c_e \sim 0.01\) early to encourage exploration; anneal to 0 over training. For LLM RLHF, \(c_e = 0\) is fine because per-token sampling temperature provides exploration.

6.3 Sampling temperature for LLM RL

• PPO-RLHF: \(T = 1.0\) (or slightly higher) during rollouts; \(T = 0\) at eval.
• GRPO: \(T = 0.6\)–1.0; lower temp gives less diverse groups.
• Some recipes (R1-style) sample at \(T = 0.6\)–0.7 for math/code.

6.4 Intrinsic motivation

• RND for hard exploration (Montezuma, etc.).
• ICM for general curiosity.
• Combine: \(r = r_{\mathrm{ext}} + \beta r_{\mathrm{int}}\), anneal \(\beta\) down as \(r_{\mathrm{ext}}\) becomes informative.

6.5 Population-based exploration

• PBT (Population-Based Training): many policies with different hyperparameters; periodically copy + perturb the best.
• Quality-Diversity: maintain a behavior archive; encourage novel behavior.
• Self-play with past versions for non-stationarity.

7. Curriculum Learning

7.1 Forward curriculum

Start at easy difficulty; advance when success rate exceeds threshold (e.g., 80%). Common in robotics manipulation, Atari with shaped progression.

7.2 Reverse curriculum

Start agent near the goal; expand starting state distribution backward. Effective for sparse-reward goal-reaching tasks.

7.3 Adaptive curriculum

Per-sample difficulty based on current policy success. Variants:

• Goal GAN: adversary proposes goals near the policy's frontier of competence.
• Asymmetric self-play (Sukhbaatar): Alice proposes tasks for Bob; Alice rewarded for hard-but-solvable tasks.
• ALP-GMM: maintain a GMM over task parameters; sample from regions of recent learning progress.

7.4 Self-play as curriculum

For zero-sum games (AlphaZero, AlphaStar): play against past versions or current opponent. Auto-generates appropriate difficulty.

7.5 Hindsight curriculum

HER + curriculum: use achieved goals as the curriculum's task distribution. Effectively a free curriculum without any difficulty engineering.

7.6 Domain randomization as curriculum

Start with narrow randomization range; widen as policy succeeds. Eventual: full real-world distribution.

8. Sim-to-Real for Robotics

8.1 Domain randomization

• Visual: textures, lighting, camera noise, occlusions.
• Dynamics: mass, friction, damping, motor noise, gravity.
• Sensor: latency, dropouts, calibration error.
• Initial conditions: pose perturbation, object placement noise.

Trained policy is invariant to nuisances \(\to\) transfers.

8.2 Asymmetric actor-critic

Critic uses privileged info (full state \(s\), terrain map \(\xi\)); actor uses only egocentric observations \(o\):

\[\pi_\theta(a|o), \qquad V_\phi(s, \xi).\]

Variance reduction without leaking privileged info to the deployable policy.

8.3 Privileged-to-vision distillation

After RL, distill the privileged actor (which sees \(s, \xi\)) into a vision-only student (sees \(o\)):

\[\mathcal{L}_{\mathrm{distill}} = \mathbb{E}\big[\|\pi_S(o) - \pi_T(s, \xi)\|^2 + \alpha\,\mathrm{KL}(\pi_T\|\pi_S)\big].\]

Student deployed in real world.

8.4 System identification (SysID)

Learn dynamics parameters from a few seconds of real-world data. Use to narrow domain randomization or to retune the real-world policy in-place.

8.5 Online adaptation

RMA (Rapid Motor Adaptation): an adaptation module estimates env parameters from recent observation history, feeds them to the policy.

8.6 Curriculum + DR

Key

Sim-to-real recipe (locomotion).

1. Train teacher with privileged info, full DR range, in \(10^4\) parallel envs.
2. Distill to vision-only student in same envs.
3. Deploy on real robot; collect \(\sim 1\) minute of data.
4. SysID + retune student LR; brief online adaptation.

8.7 Eureka / DrEureka for reward + DR design

LLM proposes reward functions and DR ranges; train with massive parallel sim; evaluate; refine. Useful when the search space is large.

9. Offline-to-Online Fine-Tuning

9.1 Why?

Offline RL is safe but bounded by data quality. Online fine-tune leverages further interaction. Pre-training is the same idea as in supervised learning: cheaper, faster overall.

9.2 Standard recipe

1. Pre-train policy + critic offline (CQL, IQL).
2. Continue online with SAC / TD3.
3. Mix offline data into online replay buffer with ratio \(\rho\).
4. Anneal conservatism during online phase.

9.3 Catastrophic forgetting mitigations

• Keep offline buffer in replay (50% mix).
• Regularize new policy to offline-trained baseline (KL anchor).
• Slow critic update (low UTD initially).
• "Cal-QL" (Calibrated Q-Learning): explicit calibration before transition.

9.4 For LLM RLHF

Offline-to-online RLHF: SFT on demonstrations \(\to\) DPO on offline preferences \(\to\) online iterative DPO with model-generated samples.

10. Distributed RL Training

10.1 Architectures

• Single-process vectorized: \(N\) envs, one GPU policy. Simplest.
• Sync distributed actors: \(N\) remote workers; gather, broadcast.
• Async actor-learner (IMPALA): actors push to learner; V-trace correction.
• Parameter server: legacy; mostly replaced by all-reduce.

10.2 For LLM RL

• Tensor parallelism for model.
• Sequence parallelism for long contexts.
• Pipeline parallelism for very large models.
• Reference-policy sharding (separate GPU group serving the frozen ref).
• KV cache sharing across reference and policy if architecturally compatible.

10.3 Reference model handling

• Full ref copy: simplest; doubles memory.
• Frozen LoRA target: train policy as LoRA on frozen base; the base is the reference. Massive memory savings.
• Sharded ref on dedicated nodes: best for largest models.

10.4 Actor / learner separation in LLM RL

• Generation servers (vLLM / SGLang): serve the policy for rollouts.
• Trainer nodes (Megatron / DeepSpeed): backprop on collected rollouts.
• Periodically push policy weights from trainer \(\to\) generation servers.
• Reward / verifier servers: separate microservice.

10.5 Frameworks for distributed RL

11. LLM RLHF Training Pipelines

11.1 Stage 1: SFT

• Data: \(\sim 10\) k–500k high-quality demos / instructions.
• LR: \(1 \times 10^{-5}\)–\(5 \times 10^{-5}\) for full FT, higher for LoRA.
• Batch: largest that fits; 64–512 tokens per global batch typical.
• Epochs: 1–3.
• Loss masking: only on assistant turns.

11.2 Stage 2: Reward Model training

• Data: tens of thousands to millions of pairwise comparisons.
• Architecture: SFT-finetuned model with extra scalar head; freeze most of body, fine-tune last layers.
• Loss: BT \(\log\sigma(r(y_w) - r(y_l))\). Optionally length normalize.
• Eval: held-out preference accuracy (\(\sim 65\)–75% strong; \(> 80\%\) suspicious of overfit).

11.3 Stage 3: PPO

Key

PPO-RLHF default hyperparameters.

• LR: \(1 \times 10^{-6}\)–\(5 \times 10^{-6}\).
• Batch: 64–512 prompts.
• Sample 1 response per prompt; max-tokens 256–2048.
• PPO epochs: 1–4.
• Clip \(\epsilon = 0.2\).
• KL coefficient \(\beta\): adaptive, target KL \(\sim 0.02\).
• Value head: shared backbone, separate scalar head.
• Critic LR: same as policy.
• Reward whitening: per-batch z-score.

11.4 Common issues in PPO-RLHF

• Reward hacking: see length explosion, repetition, sycophancy.
• KL spike: model diverges; check ref-policy implementation.
• Value not tracking reward: increase value coefficient or use separate value network.
• Reward saturation: RM has ceiling; switch to harder RM or human labels.

11.5 Iterative RLHF

After PPO converges:

1. Sample new responses from RL'd model.
2. Collect preferences (or RM scores).
3. Update RM.
4. Run PPO again from current policy.
5. Repeat 2–4 times. Each iteration tightens the alignment.

12. DPO Training Pipelines

12.1 Standard DPO recipe

Key

DPO defaults.

• Pre-stage: SFT on demos.
• Data: \(\sim 10\) k–100k preference pairs.
• LR: \(5 \times 10^{-7}\)–\(5 \times 10^{-6}\) (lower than SFT!).
• \(\beta\): 0.1 standard; lower (0.01) for stronger optimization, higher (1.0) for more conservative.
• Batch: 8–64 pairs.
• Epochs: 1 (more risks overfitting).
• Linear warmup over 10% of steps; cosine to 0.

12.2 ORPO (combined SFT + preference)

Skip the SFT stage; one-step training with combined loss:

\[\mathcal{L}_{\mathrm{ORPO}} = \mathcal{L}_{\mathrm{SFT}}(y_w) - \lambda\log\sigma\!\left(\log\frac{\mathrm{odds}(\pi_\theta(y_w))}{\mathrm{odds}(\pi_\theta(y_l))}\right).\]

\(\lambda \sim 0.1\). No reference policy, half the memory.

12.3 SimPO (no reference policy)

Length-normalized log-prob, drops reference. Saves memory; good for resource-constrained settings.

12.4 KTO (single-response thumbs)

Useful when you only have \(+/-\) flags, not pairs. Up to \(5\times\) more sample-efficient with binary feedback.

12.5 Iterative / online DPO

1. Start with SFT model.
2. Sample \(K\) responses per prompt with current \(\pi_\theta\).
3. Score with RM (or LLM judge); form preference pairs.
4. Run DPO for one epoch.
5. Repeat 2–5 times.

Closes the gap with PPO at much lower complexity.

12.6 Mixing prompts

For DPO data: cover the prompt distribution you want at inference. Pure helpfulness pairs \(\to\) helpful but maybe unsafe. Mix safety + helpfulness pairs at \(\sim 70/30\) ratio.

12.7 Self-play DPO (SPIN)

Generate responses from current model; the previous-iteration model's responses are losing; current model's responses are winning. Iterative bootstrap. Effective when starting from a strong SFT.

13. GRPO Training Recipes

13.1 Default GRPO setup (DeepSeek-style)

Key

GRPO defaults.

• Group size \(G\): 8–64 (16 typical).
• Sampling temperature: 0.6–1.0.
• Max generation: 4096–16384 tokens for reasoning (long CoT).
• PPO-style clip \(\epsilon\): 0.2.
• KL coefficient \(\beta\): 0.001–0.04.
• LR: \(1 \times 10^{-6}\)–\(5 \times 10^{-6}\).
• Reward normalization: group-relative (built into GRPO).
• Filter trivial-pass groups: skip prompts where all responses succeed or fail.

13.2 R1-style verifiable-reward GRPO

• Reward components: format (+1 for valid <think> structure), accuracy (+1 for correct math/code), language (\(+r_{\mathrm{lang}}\) for target-language fraction).
• Pre-compute rewards in CPU (regex / sympy / sandboxed code execution).
• Sample temperature 0.7 default; lower for code (0.5).
• Anti-degeneracy: reject empty / nonsense responses with \(-1\).

13.3 DAPO improvements

• Decoupled clip: \(\epsilon^+ = 0.28\), \(\epsilon^- = 0.2\).
• Token-level loss aggregation (sum per token, not mean per response).
• Dynamic sampling: re-sample for prompts with no signal.
• Overlong reward shaping: gradual penalty rather than hard cap.

13.4 Dr. GRPO

Drop length normalization to remove length / difficulty bias; otherwise identical.

13.5 Common GRPO failure modes

• Length explosion: model rewarded for longer responses (length bias). Fix with Dr. GRPO or explicit length penalty.
• Mode collapse: all \(G\) responses identical. Fix with higher temp, more \(G\), or temperature schedule.
• Reward gaming: model emits exact-format reward shortcuts. Fix with anti-hacking filters.
• Format-only optimization: model maximizes format reward without solving task. Fix with weighted reward and curriculum.

13.6 R1-Zero specifics

Pure RL from base model:

• No SFT cold start.
• Allow exploration via higher temp early.
• Patience: \(\sim 10\) k–100k steps for behavior emergence.
• Reasoning length grows monotonically over training (a healthy signal).
• Add language-consistency reward to suppress mixed-language CoT.

14. Diffusion Alignment Training Recipes

14.1 Diffusion-DPO standard recipe

Key

SDXL/SD3/FLUX DPO defaults.

• Data: 10k–100k preferred-vs-rejected image pairs at the same prompt.
• LR: \(1 \times 10^{-7}\)–\(1 \times 10^{-6}\) (very low!).
• \(\beta\): 1000–5000 (much larger than text DPO due to per-step diffusion loss scaling).
• Batch: 1–4 (large memory cost).
• Epochs: 1.
• Optimizer: AdamW, weight decay 0.01.
• LoRA training (rank 8–64) often used for efficiency.

14.2 DDPO / DPOK (PPO on diffusion)

• Treat each denoising step as an action.
• Sample \(\sim 50\)-step DDIM rollouts.
• Reward = aesthetic / preference score on final image.
• PPO with very low LR (\(10^{-7}\)) and small clip (\(\epsilon = 0.1\)).
• Memory-heavy; gradient checkpointing required.

14.3 Reward backprop (DRaFT, AlignProp, ReFL)

• Differentiable evaluator (e.g., aesthetic predictor) on final image.
• Backprop \(\partial r / \partial\theta\) through the entire sampler.
• Memory: train only LoRA layers; checkpoint every \(k\) denoising steps.
• Works for short samplers (25–50 steps); doesn't scale to thousands.

14.4 Composite reward for diffusion

\[r = w_{\mathrm{HPS}}\,r_{\mathrm{HPS}} + w_{\mathrm{CLIP}}\,r_{\mathrm{CLIP}} + w_{\mathrm{aes}}\,r_{\mathrm{aes}} - w_{\mathrm{NSFW}}\,r_{\mathrm{NSFW}}.\]

15. Robotics Training Recipes

15.1 Locomotion (ANYmal, Cassie, H1, G1)

Key

Quadruped/biped locomotion recipe.

1. Isaac Lab / MJX with \(\sim 4096\) parallel envs.
2. PPO, LR \(3 \times 10^{-4}\), \(\gamma = 0.99\), \(\lambda = 0.95\), clip 0.2.
3. Composite reward: track velocity, stability, smoothness penalties.
4. Domain randomization: terrain, friction, mass, motor delay, sensor noise.
5. Train teacher with privileged info (terrain map).
6. Distill teacher \(\to\) vision-only student.
7. Sim-to-real with brief on-robot adaptation.

15.2 Manipulation (table-top tasks)

• ManiSkill 3 / Isaac Lab / Genesis with smaller env count (256).
• PPO with much higher reward shaping (sparse rarely works).
• Curriculum: easier object positions first.
• HER for goal-reaching subtasks.
• Often: SFT on demos (BC) first, then RL fine-tune.

15.3 Bimanual / dexterous (ALOHA, ShadowHand)

• Diffusion Policy or ACT trained via BC on \(\sim 50\)–500 demos per task.
• RL fine-tune via residual policy on top.
• Can use Eureka to design rewards from task descriptions.

15.4 VLA training (RT-2, OpenVLA, \(\pi_0\))

1. Base: pretrained VLM.
2. Co-train with mix of robot trajectories + web image-text data.
3. Discretize actions (RT-2/OpenVLA) or use FM head (\(\pi_0\)).
4. Massive batch size; gradient checkpointing; mixed precision.
5. For RL fine-tune: sparse success + Q-chunking.

15.5 Humanoid mocap RL (DeepMimic, AMP, ASE)

• Reward = Gaussian on pose / velocity error.
• Reference Motion Imitation (RMI): track reference trajectories.
• Adversarial Motion Prior (AMP): discriminator that distinguishes generated from reference motion; reward from discriminator score.
• Combine with task reward (e.g., move toward target while staying upright).

16. Eval Strategies During Training

16.1 What to log

• Mean / max / min reward per rollout.
• Per-component reward.
• \(\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{old}})\) (per-step KL drift).
• \(\mathrm{KL}(\pi_\theta\|\pi_{\mathrm{ref}})\) (anchor drift).
• Entropy / per-token entropy.
• Value loss, value mean, value variance.
• Gradient norm.
• Sampling temperature / noise scale.
• For LLM: generation length distribution.
• For LLM: sample n-gram diversity.

16.2 Held-out eval cadence

• Cheap eval (held-out RM, automatic metrics): every 100–500 steps.
• Full benchmark suite: every 1k–10k steps or at milestones.
• Human eval: at major milestones / weekly.
• "Vibes" set: small hand-curated, cover edge cases.

16.3 Benchmarks

• LLM general: MMLU, BBH, HellaSwag, ARC, GPQA, MMLU-Pro.
• LLM math: MATH, AIME, MathBench, GSM8K.
• LLM code: HumanEval, MBPP, LiveCodeBench, SWE-bench.
• LLM reasoning: ARC-AGI, MuSR, BigBench-Hard.
• LLM safety: HarmBench, JailbreakBench, TruthfulQA.
• LLM preference: AlpacaEval 2, Arena-Hard, MT-Bench.
• Multimodal: MMMU, MathVista, ChartQA, MMVet, BLINK.
• Robotics: LIBERO, RoboCasa, SimplerEnv.

16.4 Pass@k for code/reasoning

\[\mathrm{pass@}k = 1 - \prod_{i=1}^{k}(1 - p_i),\]

where \(p_i\) is success of \(i\)-th independent sample. Standard for LiveCodeBench / HumanEval.

16.5 Win rate (Arena-style)

For DPO/RLHF eval: pairwise comparison against a strong baseline (GPT-4, Claude) judged by another LLM. Compute win rate or Elo.

17. Debugging Failed Training Runs

17.1 Diagnostic tree

1. Reward not increasing? \(\to\) verify reward computation manually on samples.
2. Reward increasing, eval flat? \(\to\) reward hacking; check for length explosion, repetition, format-only optimization.
3. KL exploding? \(\to\) check reference model frozen; reduce LR; increase \(\beta\); reduce PPO epochs.
4. Entropy collapsing too fast? \(\to\) increase entropy bonus; raise sampling temperature.
5. Value loss not decreasing? \(\to\) value head undertrained; increase \(c_v\); consider separate value net.
6. Gradient spikes? \(\to\) tighten clip \(\tau\); investigate problematic batches; add reward clipping.
7. Mode collapse in DPO? \(\to\) lower LR; reduce epochs; increase \(\beta\).
8. Length exploding in GRPO? \(\to\) switch to Dr. GRPO or add explicit length penalty.

17.2 Sanity checks before scaling

• Train on tiny subset (100 samples); should overfit.
• Train on a single fixed prompt; reward should converge to maximum.
• Sample policy outputs throughout training; read them.
• Trajectory videos for robotics; visualize what the agent does.
• Per-component reward dashboard.

17.3 Ablations to run

• Without KL anchor: does it explode? (It should.)
• Without reward normalization: how unstable?
• With smaller LR: same final reward?
• With smaller batch / smaller \(G\): how much variance?
• Without curriculum: does it still learn?

18. Compute and Memory Engineering

18.1 Memory math (LLM RLHF)

For a 7B model in BF16 + Adam states + gradients:

\[\mathrm{Mem} \approx 14 \cdot 7\mathrm{B} \cdot 1\,\mathrm{B} = 98\ \mathrm{GB},\]

plus activations + KV cache. Fits with FSDP / ZeRO-3 across 8 GPUs.

18.2 PPO doubles memory

PPO needs the reference policy too: \(\sim 2\times\) the model footprint. Solutions:

• LoRA: train as low-rank delta; base model serves as reference.
• Sharded reference on dedicated nodes.
• Reference-free methods (SimPO, ORPO).

18.3 Generation acceleration

• vLLM / SGLang for high-throughput rollout generation.
• Continuous batching across rollout prompts.
• KV cache reuse for shared system prompts.
• Speculative decoding for evaluation.

18.4 Gradient checkpointing

Save memory by recomputing activations in backward pass at the cost of \(\sim 30\%\) extra compute. Standard for full-FT large models.

18.5 Mixed precision and FP8

• BF16 default for training.
• FP8 (E4M3 forward, E5M2 backward) on H100/B200; \(2\times\) throughput.
• Loss scaling for FP16 only; not needed for BF16.

18.6 LoRA / PEFT for RLHF

• Train rank-8 to rank-256 LoRA on attention + MLP.
• Base model = reference policy = frozen.
• \(10\times\) less memory; comparable quality at small/medium scale.
• Stack with QLoRA (4-bit base) for further savings.

18.7 Reference model sharing tricks

• DPO/GRPO: pre-compute and cache reference log-probs once per epoch.
• Stream: compute reference log-probs in batches alongside policy.
• Quantize reference to INT8 (small accuracy loss).

19. Multi-Task and Generalization

19.1 Task sampling

• Uniform: simplest baseline.
• Difficulty-weighted: emphasize tasks where the policy is improving.
• Loss-weighted: emphasize high-loss tasks (a la PCGrad).
• Curriculum-based: progressive difficulty.

19.2 Multi-task PPO

Single policy across \(K\) tasks. Per-task reward normalization to prevent dominance. Sometimes one-hot task ID concatenated to observation.

19.3 Distillation across tasks

After per-task experts, distill into a single multi-task policy:

\[\mathcal{L}_{\mathrm{multi}} = \sum_k \lambda_k\,\mathbb{E}_{(s,a)\sim\pi_k^*}\big[-\log\pi_\theta(a|s, k)\big].\]

19.4 Meta-RL recipes

• MAML: train initialization that fast-adapts; few-shot fine-tune at test.
• PEARL: probabilistic embedding of task; sample \(z\), condition policy.
• VariBAD: variational Bayes over MDPs; condition on belief.
• RL\(^2\): train RNN policy across many MDPs; let recurrence amortize adaptation.

20. Common Failure Modes Catalogued

20.1 For PPO

• Reward not learnt: GAE bug, value not training, advantage not normalized.
• Catastrophic forgetting: too aggressive update, KL too large; reduce epochs / LR.
• Entropy collapse: clip too tight, no entropy bonus, deterministic actions; raise entropy coefficient.
• Reward hacking: shaping is wrong; sparse it.

20.2 For DPO

• Both probabilities decreasing: known DPO failure; lower LR, fewer epochs, higher \(\beta\).
• Length explosion: switch to SimPO (length-norm) or add explicit length penalty.
• Forgetting SFT capability: mix SFT loss back in (hybrid DPO+SFT).
• Overfitting: 1 epoch only; pair quality matters.

20.3 For GRPO / R1-style

• No emergent reasoning: base model too small (under \(\sim 7\) B usually fails); cold-start with SFT.
• Length exploding: Dr. GRPO or explicit penalty.
• Format-only optimization: anti-hacking filters; weight task-accuracy higher.
• Slow convergence: increase \(G\), raise temperature, decrease \(\beta\).

20.4 For robotics

• Sim-to-real gap: widen DR; add SysID; brief on-robot adaptation.
• Energy waste: add action smoothness; energy penalty.
• Standing still (passive collapse): reduce survival bonus; add forward-progress reward.
• Tipping over: explicit roll/pitch penalty; lower terrain difficulty in curriculum.

21. Production Best Practices (2026)

21.1 LLM RLHF stack

• Cold-start SFT on \(\sim 50\) k demos.
• Reward model on \(\sim 200\) k pairs, ensemble of 3.
• PPO with KL anchor, length norm, value clip.
• OR: DPO / SimPO / ORPO for cheaper alignment.
• Iterate: 2–4 rounds of preference collection \(\to\) DPO.
• Eval: AlpacaEval 2 + Arena-Hard + held-out human prefs.

21.2 LLM reasoning stack

• Either: cold-start SFT on long-CoT demos \(\to\) GRPO; or pure R1-Zero from base.
• Verifiable reward: math / code / format.
• GRPO with \(G = 16\), low \(\beta\), dynamic sampling.
• Eval: AIME, MATH, LiveCodeBench, ARC-AGI.
• Distill larger reasoner \(\to\) smaller for deployment.

21.3 Diffusion alignment stack

• Diffusion-DPO with HPS-based pair selection.
• LoRA fine-tune at very low LR.
• Composite reward (HPS + ImageReward + aesthetic).
• Eval: Image Arena + held-out human.

21.4 Robotics stack

• Massive parallel sim (Isaac Lab).
• PPO + asymmetric A-C + DR.
• Privileged-to-vision distillation.
• Eureka for reward design.
• Brief on-robot fine-tune.

Appendix A: Hyperparameter Cheat Card

Appendix B: Pre-Run Checklist

1. Reward computed correctly on hand-checked samples.
2. Reference policy frozen (verify gradient flow).
3. KL anchor active (log \(\mathrm{KL}(\pi\|\pi_{\mathrm{ref}})\) from step 1).
4. Sample prompts read for sanity.
5. Hyperparameters from a known-good recipe (no exotic settings).
6. Tiny-scale overfitting test passes (100 samples, model memorizes).
7. Eval pipeline runs end-to-end on a checkpoint.
8. Logging: per-component reward, KL, entropy, gradient norm, value loss.
9. Checkpoint cadence + max keep set.
10. Failure-recovery: skip-bad-batches and rollback ready.

Appendix C: Twenty Things a Senior RL Engineer Should Just Know

1. Adam \(\epsilon = 10^{-5}\) for RL.
2. Reward normalization by std, not mean.
3. Observation normalization is necessary for MuJoCo.
4. GAE with \(\lambda = 0.95\).
5. KL anchor is non-negotiable in RLHF.
6. Length normalize RM data.
7. Use ensembles of RMs.
8. Filter trivial-pass prompts in GRPO.
9. Dr. GRPO removes length bias.
10. DPO needs only 1 epoch.
11. DPO LR is much smaller than SFT LR.
12. For diffusion DPO, \(\beta \sim 1000\), not 0.1.
13. Sample-and-read your policy outputs every checkpoint.
14. Privileged-to-vision distillation closes most sim-to-real gaps.
15. Eureka can design rewards better than humans.
16. Vectorized envs first; distributed only when GPU saturated.
17. LoRA + frozen base saves \(2\times\) memory in DPO/GRPO.
18. Pre-compute reference log-probs once per epoch.
19. AlpacaEval 2 + Arena-Hard correlate well with human prefs.
20. The held-out human eval is the only ground truth.