Reward Functions — All Variants & Design Patterns

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

What Is a Reward Function?
Reward Shaping
Sparse Rewards: Curriculum and Hindsight
Inverse Reinforcement Learning (IRL)
Preference-Based Reward Modeling (RLHF)
Process Reward Models (PRMs) vs Outcome Reward Models (ORMs)
Verifiable / Programmatic Rewards (R1 Style)
Reward Functions for Vision Tasks
Reward Functions for Image / Video Generation
Reward Functions for Robotics
Reward Functions for Autonomous Driving
Reward Functions for Game Playing
Multi-Objective Reward Aggregation
Risk-Sensitive and Distributional Rewards
Intrinsic Motivation / Curiosity Rewards
Reward Hacking and Overoptimization
AI Feedback (RLAIF) and Constitutional Methods
Reward Design for Safety & Alignment
Reward Shaping for Diffusion-Based Policies
Reward Functions for World Models / RL via Imagination
Reward Function Engineering: A Practical Recipe
Production Recipes (2026)

Appendix A: Twenty-Five Key Equations and Patterns

Appendix B: Decision Tree for "What Reward Should I Use?"

1. What Is a Reward Function?

1.1 Formal definition

A reward function \(r : S \times A \to \mathbb{R}\) (or \(S \times A \times S \to \mathbb{R}\)) maps state-action(-next-state) tuples to scalar feedback.

The agent's goal is to maximize expected discounted return:

\[J(\pi) = \mathbb{E}_\pi\!\left[\sum_{t=0}^{\infty} \gamma^t r_t\right].\]

1.2 Reward vs return

Reward \(r_t\) is per-step; return \(G_t = \sum_{k \ge 0} \gamma^k r_{t+k+1}\) is cumulative. Most policy-optimization quantities (advantages, value functions) are derived from returns, not raw rewards.

1.3 Markovian assumption

RL theory assumes \(r_t\) depends only on \((s_t, a_t)\). Non-Markovian rewards (depending on history) require state augmentation or non-Markovian policies (e.g., RNN, Transformer).

1.4 Reward as the spec for behavior

Key

"You get what you measure." The reward function is the entire specification of agent behavior. Every other component (policy class, optimizer, exploration strategy) is in service of maximizing it. Bugs in the reward \(\Rightarrow\) bugs in the policy.

1.5 Goodhart's law

"When a measure becomes a target, it ceases to be a good measure." All reward design must anticipate that the agent will exploit any discrepancy between the proxy reward and the true intent. This is the central design problem.

1.6 Reward Engineering vs Reward Modeling vs RL from Verifiers

Three regimes:

Engineered: human designs \(r\) as code (classical control, robotics).
Learned: \(r_\phi\) trained from human preferences (RLHF) or demonstrations (IRL).
Verifier-based: programmatic check (math correctness, code passes test, format adherence).

The R1 / GRPO regime.

2. Reward Shaping

2.1 The motivation

Sparse rewards (e.g., +1 at the end of an episode if the goal is reached, 0 otherwise) provide little gradient signal until the agent stumbles upon the goal. Shaping adds intermediate rewards to guide learning.

2.2 Naive shaping is dangerous

Adding any auxiliary reward \(\tilde{r}\) changes the optimal policy:

\[J^{\text{shaped}}(\pi) = J(\pi) + \mathbb{E}_\pi\!\left[\sum_t \gamma^t \tilde{r}_t\right].\]

If \(\tilde{r}\) doesn't preserve the ordering over policies, the agent optimizes the wrong objective.

2.3 Potential-Based Reward Shaping (Ng, Harada, Russell 1999)

Key

Theorem. A shaping reward of the form

\[F(s, a, s') = \gamma\, \Phi(s') - \Phi(s)\]

for any potential \(\Phi : S \to \mathbb{R}\) provably preserves the optimal policy under \(r' = r + F\).

Intuition: \(\sum_t \gamma^t F_t\) telescopes to \(\gamma^T \Phi(s_T) - \Phi(s_0)\), a horizon-dependent constant. The advantage function is unchanged.

2.4 Choices of Φ

Distance to goal: \(\Phi(s) = -\|s - s_{\text{goal}}\|^2\).
Estimated value: \(\Phi(s) = \hat{V}(s)\) from a (possibly partial) value estimate.
Demonstration-based: \(\Phi(s) = -d(s, \mathcal{D}_{\text{expert}})\), distance to nearest expert state.

2.5 Beyond pure potential shaping

Action-dependent potential: extends to \(F = \gamma\Phi(s', a') - \Phi(s, a)\) with similar invariance.
Plan-based shaping: use a heuristic planner's value as \(\Phi\).
Look-ahead shaping: \(\Phi(s)\) from \(n\)-step sim rollouts.

2.6 Common engineered shaping terms

Time penalty: \(r_t = -c\) each step to encourage shorter episodes.
Energy penalty: \(r_t \mathrel{-}= \lambda \|a_t\|^2\) for smooth control.
Smoothness penalty: \(r_t \mathrel{-}= \lambda \|a_t - a_{t-1}\|^2\).
Velocity reward: \(r_t = v_t \cdot \hat{n}_{\text{forward}}\) for locomotion.
Survival bonus: \(r_t = +c\) each step alive (encourages stability).
Collision penalty: \(r_t = -C\) on contact with obstacle.

Watch out

Shaping terms are often non-potential-based and do change the optimal policy. Always test: does the agent optimize what you want, or does it sit in a corner collecting your survival bonus?

3. Sparse Rewards: Curriculum and Hindsight

3.1 When sparse is right

Sparse rewards are the cleanest specification: "did you achieve X?" Avoids reward hacking by construction. The cost is harder optimization.

3.2 Hindsight Experience Replay (HER, Andrychowicz et al. 2017)

Relabel a failed trajectory's goal to whatever was actually achieved. Each trajectory now contains some "successful" transitions for the relabeled goal, yielding gradient signal. Standard for goal-conditioned RL. Strategies for relabeling:

Final: relabel goal to terminal state.
Future: relabel goal to a future state in the same trajectory.
Episode: relabel from any state in the episode.
Random: from any state in the buffer.

3.3 Curriculum learning

Forward: start with easy task; gradually increase difficulty.
Reverse: start the agent near the goal; expand starting region backward.
Adaptive: per-task / per-sample difficulty based on success rate.
Goal-GAN / asymmetric self-play: an adversary proposes goals at the frontier of the agent's ability.

3.4 Self-imitation learning

At each iteration, save trajectories with above-mean returns; SFT on those. Free amortized improvement signal.

4. Inverse Reinforcement Learning (IRL)

4.1 The IRL setup

Given expert demonstrations \(\mathcal{D} = \{\tau_i\}\), recover the reward function \(r\) that explains them. Then run forward RL.

4.2 Linear IRL (Ng & Russell 2000)

Assume \(r(s) = w^\top \phi(s)\) for features \(\phi\). Find \(w\) such that the expert is optimal:

\[\mathbb{E}_{\pi^*}\!\left[\sum_t \gamma^t r(s_t)\right] \ge \mathbb{E}_\pi\!\left[\sum_t \gamma^t r(s_t)\right] \quad \forall \pi.\]

Solve as a linear program. Classical but ill-posed (many \(r\) are consistent with \(\pi^*\)).

4.3 Maximum Entropy IRL (Ziebart et al. 2008)

Pick the reward whose Boltzmann-distributed policy best matches expert frequencies:

\[\pi(a|s) \propto \exp(Q_{\text{soft}}(s, a)), \quad \mathcal{L}(w) = \log Z(w) - w^\top \mathbb{E}_{\mathcal{D}}[\phi].\]

Resolves the ill-posedness via maximum entropy.

4.4 GAIL (Generative Adversarial Imitation Learning, Ho & Ermon 2016)

Skip explicit reward; train a discriminator \(D_\phi\) to distinguish expert from policy. Policy reward:

\[r(s, a) = -\log(1 - D_\phi(s, a)).\]

Train via PPO.

4.5 AIRL (Adversarial IRL, Fu et al. 2017)

Discriminator structured to recover reward:

\[D_\phi(s, a) = \frac{\exp(f_\phi(s, a))}{\exp(f_\phi(s, a)) + \pi(a|s)},\]

\(f_\phi\) approximates the advantage. Decomposable as \(f_\phi = g_\phi(s) + \gamma h_\phi(s') - h_\phi(s)\) for transferable \(g_\phi\).

4.6 GCL, Variational IRL, f-IRL

Other estimation routes; share the goal of recovering a reward consistent with expert behavior.

4.7 When IRL is right

Useful when:

The reward is hard to specify but easy to demonstrate.
You want a transferable reward (re-use across tasks).
Behavior is the data, but you need a handle on optimization.

4.8 Limitations

Many rewards are consistent with the same expert policy.
Suboptimal demonstrations confound the inferred reward.
Computational cost: nested optimization (RL inside IRL).

5. Preference-Based Reward Modeling (RLHF)

5.1 The Bradley-Terry model

Pairwise comparisons \((x, y_w, y_l)\), \(y_w\) preferred. Assume a latent reward \(r\):

\[\mathbb{P}(y_w \succ y_l \mid x) = \sigma\!\left(r(x, y_w) - r(x, y_l)\right).\]

5.2 Reward model loss

Key

\[\mathcal{L}_{\text{RM}} = -\mathbb{E}_{(x, y_w, y_l)}\!\left[\log \sigma\!\left(r_\phi(x, y_w) - r_\phi(x, y_l)\right)\right].\]

Trained on tens of thousands to millions of human preference pairs.

5.3 Listwise / K-wise: Plackett–Luce

For a ranking \(y_1 \succ y_2 \succ \cdots \succ y_K\):

\[\mathbb{P}(\text{ranking}) = \prod_{i=1}^{K-1} \frac{\exp(r(x, y_i))}{\sum_{j \ge i} \exp(r(x, y_j))}.\]

Generalizes BT to arbitrary list sizes; more sample-efficient when listwise data is available.

5.4 Rating-scale rewards

Direct numerical scores (1–5, 1–7). Train RM via MSE. Cheaper to collect; noisier per item; less calibrated.

5.5 Conditional / context-dependent rewards

The same response can be preferred under one persona and dispreferred under another. Condition the RM on the persona / system prompt:

\[r_\phi(x, y) \to r_\phi(x, y, c_{\text{persona}}).\]

5.6 Reward model best practices

Length normalization: regress on \(r(x, y) - \bar{r}_{|y|}\) where \(\bar{r}_{|y|}\) is the mean reward at that length, to remove length bias.
Margin loss: replace BT with \(\max(0, \delta - (r(y_w) - r(y_l)))\) for cleaner training.
Ensembles: train \(K\) RMs; take min/median for conservative reward.
Calibration: reward should track empirical preference rate; calibrate via Platt scaling.
Active learning: select new labeling pairs with high disagreement among the ensemble.

6. Process Reward Models (PRMs) vs Outcome Reward Models (ORMs)

6.1 The dichotomy

ORM: scores only the final answer (correct/incorrect). PRM: scores each intermediate reasoning step.

6.2 Why PRMs?

Outcome rewards are noisy: a correct final answer can come from flawed reasoning that won't generalize. PRMs train better verifiers and better RL signal for long CoT.

6.3 PRM training data

6.3.1 PRM800K (OpenAI)

800K human-labeled steps in math reasoning. Expensive to collect.

6.3.2 Math-Shepherd auto-labeling

For each prefix \(s_{\le t}\), sample \(K\) continuations under the model. Label step \(t\) as "good" if \(> \tau\) fraction of continuations succeed. Cheaper, scalable.

6.3.3 OmegaPRM

Tree-search-based labeling with binary success per leaf; assign step labels via Monte Carlo.

6.4 PRM training loss

Per-step binary labels \(y_t \in \{0, 1\}\):

\[\mathcal{L}_{\text{PRM}} = -\sum_t \Big[ y_t \log p_\phi(y_t = 1 \mid s_{\le t}) + (1 - y_t) \log(1 - p_\phi(\cdot)) \Big].\]

6.5 Implicit PRM (process from outcome)

Score function:

\[r_\phi(s_{\le t}) = \log \frac{\pi_\phi(y^* \mid s_{\le t})}{\pi_{\text{ref}}(y^* \mid s_{\le t})} \cdot \beta,\]

recovers process-level rewards from outcome-only training using the DPO closed-form trick.

6.6 Generative reward models (LLM-as-judge)

Prompt an LLM: "Score this response 1–10 with reasoning." Cheap, flexible, but biased (position bias, length bias, self-preference). Practical patterns:

Pairwise comparison better than absolute rating.
Two-pass with order swap to debias position.
Chain-of-thought before score (better calibration).
Constitutional / rubric-guided scoring.

6.7 Constitutional AI (Anthropic)

Use an LLM with a written constitution to critique and revise responses, then train a preference model on these self-generated comparisons. RLAIF without humans in the labeling loop.

6.8 RLAIF

Replace human labelers with LLM judges throughout the RM training. Cheap, scalable; quality depends on judge model. Best practice: filter out judge-disagreement cases or weight by judge confidence.

6.9 Self-Rewarding LMs (Yuan et al. 2024)

The same model generates responses, judges them, and trains on its own preference data. Iterative bootstrap.

Effective when the base model is already strong.

★ 2026 SOTA update — Generative reasoning reward models (2025)

RM-R1: reward modeling as reasoning; self-generates chain-of-rubrics then judges, trained by distillation + RLVR; beats larger GPT-4o/70B judges.
Reward Reasoning Model: emits long CoT before scoring, RL-trained with no reasoning traces; spends adaptive test-time compute for harder queries.
DeepSeek-GRM: pointwise generative RM via Self-Principled Critique Tuning (SPCT); generates principles+critiques, meta-RM voting for inference-time scaling.

★ 2026 SOTA update — Generative & data-efficient PRMs (2025)

ThinkPRM: generative long-CoT verifier that labels every step; 8K labels beat discriminative PRMs trained on ~100x more data; scales verification compute.
Qwen2.5-Math-PRM (Lessons): MC-estimation step labels are noisy; consensus filtering of MC + LLM-as-judge, plus response+step-level eval, fixes it.

★ 2026 SOTA update — Reward-model & PRM benchmarks (2025)

RewardBench 2: harder multi-skill RM benchmark (6 subsets, 1-chosen/3-rejected); ~20 pts lower than v1, correlates with BoN + PPO downstream gains.
ProcessBench: 3,400 math solutions with first-error step annotated; measures step-error identification for PRMs and LLM critics.

7. Verifiable / Programmatic Rewards (R1 Style)

7.1 The verifiable-reward paradigm

For tasks with checkable answers, programmatic rewards remove the need for a learned RM:

Math problem \(\to\) check final number / equation.
Code problem \(\to\) run unit tests.
Logic puzzle \(\to\) check constraints satisfied.
Format \(\to\) regex match.
Tool use \(\to\) correct API call.

Key

This is the regime that powers DeepSeek-R1, Open-R1, TinyZero, and the entire 2025–2026 reasoning revolution. Programmatic rewards are noiseless, scale infinitely, and resist reward hacking by design.

7.2 Common verifiable reward components

7.2.1 Math accuracy

\[r_{\text{math}}(y) = \begin{cases} +1 & \text{extracted answer matches GT} \\ 0 & \text{else} \end{cases}\]

Extraction via regex on \boxed{...} or final-line number. Symbolic equivalence (sympy) for algebraic answers.

7.2.2 Code correctness

\[r_{\text{code}}(y) = \frac{\#\text{tests passed}}{\#\text{tests total}}.\]

Or binary all-pass. Sandboxed execution required. Speed: parallelize across many test cases.

7.2.3 Format reward

\[r_{\text{fmt}}(y) = \begin{cases} +1 & y \text{ contains } \texttt{<think>...</think><answer>...</answer>} \\ 0 & \text{else} \end{cases}\]

7.2.4 Length penalty

Mild penalty for excessive length to prevent unbounded scaling:

\[r_{\text{len}} = -\lambda \cdot \max(0, |y| - L_{\max}).\]

7.2.5 Language consistency (R1's fix)

DeepSeek-R1 added a reward for the response being in the same language as the question, suppressing mixed-language CoT:

\[r_{\text{lang}} = \frac{\#\text{tokens in target language}}{|y|}.\]

7.2.6 Tool-use success

For agentic tasks: reward = task-evaluator on whether the tool call achieved the subgoal. Binary or graded.

7.3 Composing rewards

\[r_{\text{total}} = w_{\text{acc}} r_{\text{acc}} + w_{\text{fmt}} r_{\text{fmt}} + w_{\text{len}} r_{\text{len}} + w_{\text{lang}} r_{\text{lang}} + \dots\]

With GRPO using group-relative advantages, the absolute scale of each term matters less than the relative differences across responses.

7.4 Anti-hacking patterns

Filter trivial-pass cases (where all \(G\) responses succeed; no signal).
Cap individual reward components.
Detect repetition; penalize.
Anti-degeneracy: penalize empty / nonsense responses.

★ 2026 SOTA update — Rubric-based rewards beyond verifiable

Rubrics as Rewards: instance-specific checklist rubrics (LLM-generated) give structured GRPO reward for non-verifiable tasks; +28% on HealthBench vs Likert.

★ 2026 SOTA update — Rewards for agents & tool use (2025)

Agentic Reward Modeling: combines human-preference RM with verifiable correctness/factuality+instruction-following checkers into one reliable reward.
ToolRM: reward model for tool-use agents that critiques tool calls via efficient reasoning, scaling feedback beyond verified trajectories.

8. Reward Functions for Vision Tasks

8.1 Detection

IoU reward:

\[r_{\text{det}}(\hat{b}, b^*) = \text{IoU}(\hat{b}, b^*) - \alpha \cdot \mathbb{1}[\text{class wrong}].\]

For multi-object: Hungarian-matched per-object IoUs; aggregate via mean or count of matched.

8.2 Segmentation

Mask IoU or Dice:

\[r_{\text{seg}}(\hat{m}, m^*) = \frac{2 |\hat{m} \cap m^*|}{|\hat{m}| + |m^*|}.\]

For panoptic: per-instance Dice averaged.

8.3 Counting / numerical answers

\[r_{\text{count}}(\hat{n}, n^*) = \begin{cases} +1 & \hat{n} = n^* \\ 1 - |\hat{n} - n^*| / n^* & \text{near miss} \end{cases}\]

8.4 OCR / text extraction

\[r_{\text{ocr}} = 1 - \text{CER}(\hat{t}, t^*),\]

where CER = character error rate. Or BLEU / token-F1 for paragraphs.

8.5 Captioning metrics

CIDEr-D, SPICE, METEOR: classical caption metrics. CLIP-Score or BLIP-Score: learned alignment.

VLM-as-judge for quality + factuality.

8.6 VQA exact match

\[r_{\text{vqa}} = \mathbb{1}[\text{normalize}(\hat{a}) = \text{normalize}(a^*)].\]

With normalization: lowercase, strip punctuation, articles. For multi-answer: max over GT options.

8.7 Grounding / referring expressions

IoU between predicted box / mask and GT for the referred region. Combined with linguistic accuracy.

8.8 VLM-R1 / MM-EUREKA reward composition

Typical multimodal-R1 reward:

\[r = w_{\text{fmt}}\, \mathbb{1}[\text{format ok}] + w_{\text{acc}}\, r_{\text{task}} + w_{\text{vis}}\, r_{\text{visual-grounded}}.\]

9. Reward Functions for Image / Video Generation

9.1 CLIP-Score

\[\text{CLIP-Score}(x, c) = \cos(\text{CLIP}_I(x), \text{CLIP}_T(c)) \cdot 100.\]

Cheap and standard. Gameable; correlates weakly with human preference in modern era.

9.2 ImageReward (Xu et al. 2023)

137k human-rated image-text pairs \(\to\) trained reward model. Bradley-Terry on triplet rankings:

\[\mathcal{L}_{\text{IR}} = -\mathbb{E}[\log \sigma(r_\phi(x_i, c) - r_\phi(x_j, c))].\]

Better correlation than CLIP-Score.

9.3 HPSv2 / HPSv3 (Human Preference Score)

Trained on web-scraped preferences.

v2: 798k pairs, ranks SDXL-level.

v3: scales further.

Standard for SDXL/FLUX alignment.

9.4 PickScore

Trained on Pick-a-Pic (1M+ comparisons from real users on a stable-diffusion playground).

9.5 VQAScore

Use a frozen VLM to answer compositional questions about the generated image. Score = answer probability:

\[r(x, c) = P_{\text{VLM}}(\text{``Yes''} \mid \text{Question}(c), x).\]

Tests fine-grained compositional fidelity (counting, attribute binding, spatial relations).

9.6 Aesthetic predictors

LAION-Aesthetic and successors: small MLP on CLIP features regressing aesthetic ratings. Cheap secondary signal.

9.7 Video reward signals

VBench (16+ axes): subject consistency, motion smoothness, dynamic degree, scene coherence, etc. Composable into a single reward via weighted sum or learned aggregation. VideoScore: end-to-end learned video reward.

9.8 Composite generation reward (in production)

Common pattern for diffusion alignment:

\[r = w_{\text{HPS}} r_{\text{HPS}} + w_{\text{CLIP}} r_{\text{CLIP}} + w_{\text{aes}} r_{\text{aes}} - w_{\text{NSFW}} r_{\text{NSFW}}.\]

Used in Diffusion-DPO / DDPO / DPOK pipelines.

10. Reward Functions for Robotics

10.1 Locomotion (quadruped, biped, humanoid)

Composite reward typical of ANYmal, OmniH2O:

\[r = r_{\text{vel}} + r_{\text{stab}} + r_{\text{height}} - r_{\text{torque}} - r_{\text{action-rate}} - r_{\text{joint-limits}}.\]

\(r_{\text{vel}} = -\|v_{xy} - v^*\|^2\): track commanded velocity.
\(r_{\text{stab}} = -\|\omega_{xy}\|^2\): penalize roll/pitch rate.
\(r_{\text{height}} = -|h - h^*|\): maintain target height.
\(r_{\text{torque}} = \lambda \|\tau\|^2\): energy cost.
\(r_{\text{action-rate}} = \lambda \|a_t - a_{t-1}\|^2\): smoothness.

10.2 Manipulation

Sparse: +1 on success, 0 otherwise.
Distance to target: \(-\|p_{ee} - p_{\text{target}}\|^2\).
Grasp success: +1 if object lifted off table.
Insertion: +1 if peg inside hole within tolerance; dense via \(-d_{\text{contact}}\).
Mimic: \(-\|\text{robot pose} - \text{demo pose}\|^2\) at each frame for motion-tracking RL.

10.3 Eureka and DrEureka (LLM-designed rewards)

10.3.1 Eureka loop

LLM proposes \(K\) candidate reward functions as Python code from task description.
Train PPO with each in massively parallel sim (\(10^4\) envs).
Evaluate on held-out task evaluator (success rate, performance metric).
LLM analyzes results; proposes refined rewards.
Repeat.

Successful for shadow-hand pen spinning, ANYmal locomotion variants, humanoid walking.

10.3.2 DrEureka

Extends Eureka to also design domain randomization ranges.

The LLM picks both rewards and which dynamics nuisance to randomize, with what magnitude.

10.4 Imitation-shaped rewards

Use a learned discriminator (GAIL/AIRL) or distance-to-demo (DTW) as a dense reward atop sparse task success.

Stabilizes hard manipulation tasks.

10.5 Trajectory-tracking rewards (humanoid mocap RL)

DeepMimic, AMP (Adversarial Motion Prior): imitate reference motion via:

\[r_t = w_p e^{-\alpha_p \|q_t - q_t^*\|^2} + w_v e^{-\alpha_v \|\dot{q}_t - \dot{q}_t^*\|^2} + w_{\text{ee}} e^{-\alpha_{\text{ee}} d_{\text{ee}}^2}.\]

The Gaussian-style \(e^{-\alpha \cdot \text{error}^2}\) is the standard "mimic" reward shape.

11. Reward Functions for Autonomous Driving

11.1 Composite driving reward

\[r = r_{\text{progress}} + r_{\text{lane}} + r_{\text{speed}} - r_{\text{collision}} - r_{\text{comfort}} - r_{\text{rule}}.\]

11.2 Components

\(r_{\text{progress}}\): forward distance along route per step.
\(r_{\text{lane}} = -d_{\text{lane-center}}\) penalty for drifting.
\(r_{\text{speed}} = -(v - v_{\text{limit}})^2\) to track speed limit.
\(r_{\text{collision}} = -C\) on contact with another agent.
\(r_{\text{comfort}} = -\lambda(|a_{\text{long}}| + |a_{\text{lat}}| + |\text{jerk}|)\).
\(r_{\text{rule}}\): penalize traffic-rule violations (red light, stop sign).
\(r_{\text{TTC}}\): time-to-collision below threshold \(\to\) penalty.

11.3 End-to-end driving reward (UniAD style)

\[\mathcal{L}_{\text{plan}} = \|\hat{\tau} - \tau^*\|_2 + \lambda_c\, \text{CollisionPenalty}(\hat{\tau}).\]

Imitation against expert trajectory + learned collision penalty against perception-detected agents.

11.4 Closed-loop simulator rewards

CARLA, Bench2Drive, NAVSIM, DriveArena: standardized reward designs combining route completion, infraction count, comfort. Used to RL fine-tune end-to-end planners.

12. Reward Functions for Game Playing

12.1 Zero-sum games

AlphaGo / AlphaZero / MuZero: +1 win, −1 loss. No shaping. The game tree provides natural curriculum via self-play.

12.2 StarCraft II (AlphaStar)

Composite reward including: win/loss, supply efficiency, killed enemy units, Lost own units, build orders. Mix of binary outcome and dense shaping.

12.3 Dota 2 (OpenAI Five)

Per-step shaping: gold, XP, kills, deaths, last-hits, denies, tower damage. The most heavily-shaped reward in famous RL applications. Required because pure win/loss has 30-min episodes and 5-agent credit assignment.

12.4 General principle for games

The richer the action space and the longer the horizon, the more shaping is needed. The trade-off: shaping accelerates training but biases the policy toward gameable proxies.

13. Multi-Objective Reward Aggregation

13.1 Linear scalarization

\[r_{\text{total}} = \sum_i w_i r_i.\]

Simple, but the weights are interdependent: changing one component's scale changes the relative importance of all others. Sensitive to scale; brittle.

13.2 Lexicographic ordering

Optimize \(r_1\) first; among policies tied on \(r_1\), optimize \(r_2\); etc. Used when there's a strict priority order (e.g., \(\text{safety} \gg \text{comfort} \gg \text{speed}\)).

13.3 Constrained MDP / Lagrangian

Maximize \(r_1\) subject to \(\mathbb{E}[r_2] \ge c\). Lagrangian:

\[\max_\pi \mathbb{E}[r_1 + \lambda(r_2 - c)], \quad \lambda \ge 0,\]

update \(\lambda\) via gradient ascent on the constraint violation.

Reward Constrained Policy Optimization (RCPO), Lagrangian SAC.

13.4 Pareto-optimal policies

For applications wanting an explicit trade-off frontier: train multiple policies with different weight vectors; report the Pareto front; let the user pick. Computationally expensive.

13.5 Reward decomposition

\[r = r_1 + r_2 + \cdots + r_K, \quad Q^\pi(s, a) = \sum_k Q^\pi_k(s, a),\]

train one \(Q\) per component. Easier credit assignment, interpretable. Hybrid Reward Architecture (HRA), multi-head Q learning.

13.6 Normalization tricks

Per-component running mean / std.
Per-component z-score normalization.
Reward clipping after scaling.
Adaptive weight tuning (e.g., GradNorm).

14. Risk-Sensitive and Distributional Rewards

14.1 CVaR (Conditional Value-at-Risk)

Optimize the average over the worst \(\alpha\) fraction of returns:

\[\text{CVaR}_\alpha(R) = \mathbb{E}[R \mid R \le \text{VaR}_\alpha(R)].\]

Used in finance and safety-critical control.

14.2 Variance penalty

\[J = \mathbb{E}[R] - \lambda\, \text{Var}(R).\]

Trades expected reward for lower variance.

Approximate via per-trajectory moments; leads to "risk-averse" policies.

14.3 Distributional RL targets

Predict the full return distribution rather than a point estimate; choose action via expectation, CVaR, or other risk measure of the distribution. C51, QR-DQN, IQN.

15. Intrinsic Motivation / Curiosity Rewards

15.1 Why intrinsic rewards

Sparse-reward environments need exploration. Intrinsic rewards provide a signal even when extrinsic reward is silent.

15.2 Curiosity (ICM, Pathak et al. 2017)

Forward-model prediction error in feature space:

\[r^{\text{int}}_t = \|\phi(s_{t+1}) - f_\theta(\phi(s_t), a_t)\|^2.\]

\(\phi\) trained via inverse model to ignore reward-irrelevant features.

15.3 Random Network Distillation (RND)

Reward = prediction error of a learned network on a fixed random target network's features:

\[r^{\text{int}}_t = \|f_\theta(s_{t+1}) - f^*_{\text{rnd}}(s_{t+1})\|^2.\]

Decreases as states are visited (the predictor learns them). Successful on hard-exploration Atari (Montezuma's Revenge).

15.4 NovelD (Novelty Difference)

\[r^{\text{int}}_t = \max(0, \text{RND}(s_{t+1}) - \text{RND}(s_t)).\]

Reward only positive novelty transitions. Cleaner signal than raw RND.

15.5 BYOL-Explore

Curiosity in BYOL feature space (more stable representations than ICM's inverse model).

15.6 Empowerment

\(r^{\text{int}} \propto I(A_{t:t+k}; S_{t+k} \mid S_t)\), the mutual information between actions and future state. Encourages reaching states from which the agent has many controllable futures.

15.7 Information gain

Reward = how much the agent's belief over dynamics changes after observing a transition. VIME, Plan2Explore, Disagreement.

15.8 Combining intrinsic and extrinsic

\[r = r^{\text{ext}} + \beta\, r^{\text{int}}.\]

\(\beta\) annealed down as extrinsic reward becomes informative. Or train two value functions (Burda et al.).

16. Reward Hacking and Overoptimization

16.1 Specification gaming

The agent satisfies the literal reward in unintended ways. Famous examples: a boat-racing agent looping in circles to collect respawning power-ups instead of finishing the race; a robot arm bumping the table to make the camera think it succeeded.

16.2 Reward overoptimization scaling laws (Gao et al. 2023)

For RLHF, true reward (held-out RM) initially rises with policy KL from base, peaks, then declines:

\[R_{\text{true}}(\text{KL}) = a\sqrt{\text{KL}} - b\, \text{KL},\]

empirically fitted. The policy's reward (according to the proxy RM) keeps rising because it's exploiting the proxy.

16.3 KL anchor as a regularizer

\[J = \mathbb{E}[r_\phi(x, y)] - \beta\, \text{KL}(\pi\, \|\, \pi_{\text{ref}}).\]

The KL term limits how far the policy can drift from the trusted reference; bounds proxy-reward exploitation.

All RLHF / DPO / GRPO recipes use this.

16.4 Reward model ensembles

Train \(K\) RMs with different data subsets / random seeds:

Mean: standard RM ensemble; lower variance.
Min / Worst-Case: conservative; upper-bounds proxy exploitation.
Standard deviation as uncertainty: penalize high-variance reward (epistemic).

16.5 Conservative reward training

WARM (Weight Averaged Reward Models, Rame et al.): average multiple RMs trained on shuffled data to reduce reward spuriousness.

16.6 Length bias in RM

RMs heavily reward longer responses (humans confuse length with quality). Mitigations:

Length-normalize log-probs (SimPO).
Subtract per-length mean reward.
Length-balanced training data.
Compose reward with explicit length penalty.

16.7 Sycophancy bias

RM rewards responses that agree with the user (even when wrong).

Mitigation: actively-curated counterexamples in training data; constitutional checks.

16.8 Verbosity / hedging bias

RM rewards safe-sounding hedges ("it's complex...") over confident answers. Counter via diverse prompt templates and balanced data.

16.9 Anti-hacking patterns checklist

Always anchor with KL penalty to a trusted reference.
Use RM ensembles; take the conservative aggregator.
Log per-component reward over training; spot anomalies.
Hold-out a "vibes" eval that the RM hasn't been trained on.
Sample policy outputs frequently and read them.

Watch out

The biggest mistake in RLHF practice is not running a held-out human evaluation. The proxy reward (RM) will keep rising while the true quality declines. Always test with humans on a fresh prompt set.

★ 2026 SOTA update — Spurious & misleading RLVR signals

Spurious Rewards: random/format/even incorrect rewards lift Qwen2.5-Math ~20%+ on MATH-500 via GRPO clipping bias amplifying priors; gains fail on Llama/OLMo. Beware reward-signal illusions.

17. AI Feedback (RLAIF) and Constitutional Methods

17.1 The basic RLAIF recipe

Replace human preferences with LLM judge preferences. Use a strong instruct model (or your own model after some SFT) as the judge.

17.2 Constitutional AI (Bai et al. 2022)

Two-step:

Critique-revise: model writes a response, critiques itself against a constitution (list of principles), revises. SFT on revised responses.
Preference labeling: model picks preferred response from pairs by checking against constitution. Train RM on these.

Eliminates human preference labeling; humans only write the constitution.

17.3 LLM-as-judge prompting

Pairwise: "Which response is better?" Better than absolute rating.
Order-swap to debias position bias.
Chain-of-thought before judgment: "Explain why, then rate."
Rubric-guided: provide explicit criteria.
Multiple judges + majority vote.

17.4 Self-Rewarding LMs (Yuan et al.)

Iterate: model generates responses, judges them via LLM-as-judge prompt, trains via DPO on its own preferences.

Bootstraps without external labels.

17.5 Pros and cons of RLAIF

Pros: cheap, scalable, fast iteration. Cons: judge biases inherited (length, position, self-preference), capped by judge quality, can drift if iterated.

★ 2026 SOTA update — RL-trained reasoning judges (2025)

J1: RL-trains LLM-as-judge by casting verifiable+non-verifiable prompts into verifiable judgment tasks; incentivizes thinking, mitigates position bias, beats larger distilled judges.

★ 2026 SOTA update — Label-free self-rewarding RL (2025)

TTRL: test-time RL with no ground truth; majority-vote consensus over sampled outputs is the reward; +211% pass@1 on AIME24 for Qwen2.5-Math-7B.

18. Reward Design for Safety & Alignment

18.1 The helpfulness vs harmlessness trade-off

Human preferences can pull in opposite directions: maximally helpful answers may include harmful content; maximally safe answers may refuse legitimate requests. Standard practice: separate reward heads + Lagrangian weighting.

18.2 Refusal rewards

Train a classifier on (prompt, response) pairs labeled as "should refuse vs not". Use as auxiliary reward to discourage refusals on legitimate prompts and encourage them on truly harmful ones.

18.3 Truthfulness rewards

Use a fact-checking model or retrieval-grounded checker. Reward = TruthfulQA-style metric or grounded-answer match.

18.4 Constitutional rewards

Score responses against an explicit policy (constitution). Each principle becomes a reward dimension; weighted aggregation.

18.5 Red-team rewards

Reward = whether a response violates a safety policy when tested by an adversarial prober. Used to harden the model against jailbreaks.

18.6 Rule-based rewards (RBR, OpenAI)

LLM-as-judge with structured criteria for safety: refusal style, comprehensive disclaimers, encrypted-content detection. Replaces human safety preferences for scalable iteration.

19. Reward Shaping for Diffusion-Based Policies

19.1 The setup

Diffusion policies (Diffusion Policy, \(\pi_0\), RDT-1B) generate continuous actions via diffusion. RL fine-tuning needs the reward signal to flow back through the sampler.

19.2 DPPO (Diffusion Policy PO)

Apply DDPO-style multi-step MDP framing to a diffusion policy. Each denoising step is an action; reward is task success at episode end.

19.3 Q-chunking

Treat each action chunk (size \(H\)) as a single decision. Reward for the chunk = task success / sub-goal achieved during the chunk's horizon. Easier credit assignment than per-step.

19.4 Reward backprop through sampler

With a differentiable evaluator (e.g., goal classifier on rendered observation), backprop \(\partial r / \partial \theta\) through the diffusion denoising. Memory-heavy; works for short samplers.

20. Reward Functions for World Models / RL via Imagination

20.1 Learned reward heads

World models (Dreamer V3, IRIS, TD-MPC2) include a reward head \(r_\phi(h_t)\) trained on observed rewards. At policy training time, imagined rollouts use the learned reward.

20.2 Symlog reward target (Dreamer V3)

Train the reward head with two-hot symlog targets:

\[\text{symlog}(x) = \text{sgn}(x) \log(|x| + 1).\]

Stabilizes across reward scales (Atari \(\sim 10^4\) vs continuous \(\sim 1\)).

20.3 Discounted-sum reward heads

Some world models predict the discounted return directly rather than per-step reward; cleaner signal but loses interpretability.

20.4 Self-supervised reward signals from latent dynamics

LEXA, SMIRL: rewards derived from world-model uncertainty / surprise.

Free-form exploration without human-specified extrinsic reward.

21. Reward Function Engineering: A Practical Recipe

21.1 Step-by-step design loop

State the goal in English. "The robot should pick up the cup."
Define success. Binary: cup off the table, no collisions.
Choose a sparse reward first. \(r = +1\) on success.
Train with strong exploration / curriculum. HER, intrinsic motivation, easier task variants.
Diagnose. If learning fails, identify the credit-assignment gap.
Add potential-based shaping. \(\Phi(s) = -\|p_{ee} - p_{\text{cup}}\|\) or similar.
Test for hacking. Run rollouts; check the agent isn't gaming the shape.
Iterate weights. Use Eureka-style LLM evolution if the search space is large.
Lock the spec. Once working, freeze; document; turn into eval.

21.2 The five most common bugs

Sign error: reward for the wrong direction (positive for collisions).
Scale mismatch: one term dominates; effective weights wrong.
Sparse-only: agent never finds reward; nothing to learn.
Reward hacking: agent satisfies the proxy without satisfying the goal.
Non-Markovian without state augmentation: agent oscillates.

21.3 Diagnostics to log

Per-component reward over episode (mean, max, min).
Reward distribution over rollouts at fixed checkpoints.
Episodes where reward \(\ne 0\).
Held-out task success rate (decoupled from training reward).
Trajectory videos (visualize what the agent does).

22. Production Recipes (2026)

Use case	Default reward	Notes
LLM helpfulness alignment	RM trained on ~50k–500k preferences + KL anchor	PPO or DPO, length-normalized
LLM safety	Constitutional + Rule-Based Rewards (RBR)	Lagrangian to cap refusal rate
LLM reasoning	Verifiable: math exact-match, code tests, format	GRPO; programmatic, no RM needed
LLM truthfulness	Retrieval-grounded fact-check + TruthfulQA-style	Caps hallucination, often fights helpfulness
Image diffusion alignment	HPSv2/v3 + ImageReward + aesthetic + KL	Diffusion-DPO standard
Video gen alignment	VBench composite + VideoScore + human PRM	Early but emerging
Robotics locomotion	Composite engineered + Eureka-refined	Massive parallel sim, asymmetric A-C
Robotics manipulation	Sparse success + dense distance + grasp	Curriculum + HER
Autonomous driving	Composite (progress, safety, comfort, rule)	Imitation-anchored
Game playing	Win/loss + minimal shaping	Self-play handles credit assignment
Visual reasoning (multimodal)	IoU / mask-IoU / VQA EM + format	VLM-R1, MM-EUREKA
Robot via VLA + RL fine-tune	Sparse success + Q-chunking	Residual RL on pretrained \(\pi_0\)