01The policies and the reward, on one axis
The reference $\pi_{\mathrm{ref}}$ is what the base language model would have output. The reward $r(a)$ is the human-preferred behavior — high where we want the model to spend its probability mass. The KL-regularized optimum $\pi^\star$ is the answer to "how should the policy reshape itself to chase $r$ without straying too far from $\pi_{\mathrm{ref}}$?" — analytically,
$$\pi^\star(a) \;\propto\; \pi_{\mathrm{ref}}(a)\,\exp\!\left(r(a)/\beta\right).$$
As you shrink $\beta$, $\pi^\star$ sharpens onto the reward; as you grow $\beta$, it relaxes back to $\pi_{\mathrm{ref}}$.
02The loss landscape
| Quantity | Value |
|---|---|
| $\mathbb{E}_{\pi_\theta}\!\left[r(a)\right]$ expected reward of $\pi_\theta$ | — |
| $\mathrm{KL}\!\left(\pi_\theta \,\Vert\, \pi_{\mathrm{ref}}\right)$ distance from the base model | — |
| $J(\pi_\theta)$ $\mathbb{E}[r] - \beta\,\mathrm{KL}$ — what RLHF maximizes | — |
| $\left\Vert \nabla_{\!\theta}\, J \right\Vert$ gradient magnitude | — |
| $J(\pi^\star)$ unrestricted objective ceiling | — |
| $J(\pi_G^\star)$ best objective achievable by one Gaussian | — |
| $\mu^\star,\ \sigma^\star$ moments of unrestricted $\pi^\star$ | — |
| $\mu_G^\star,\ \sigma_G^\star$ best Gaussian for the displayed $J$ surface | — |
The red dot is the current $\pi_\theta$. The open green star records the moments of the unrestricted, generally non-Gaussian $\pi^\star$; the blue diamond is the Gaussian that actually maximizes this displayed surface. Press Play under Exact RLHF gradient to roll the Gaussian policy toward that constrained optimum.
Six algorithms, one landscape. Click any tab below to swap the bottom half of the page — including the sidebar's method hyperparameters — to that algorithm. PG, PPO, and GRPO estimate the Gaussian RLHF direction; DPO and DRO fit related offline objectives that recover the same unrestricted $\pi^\star$ only under matched temperatures and their ideal data assumptions. Restricting every learned policy to one Gaussian exposes how those objectives can project that ideal target differently. The last tab, Compare, runs all six from the same starting point and overlays their trajectories on a single map.
03RLHF — climbing the exact gradient
In this idealized setting both $\mathbb{E}_{\pi_\theta}[r(a)]$ and $\mathrm{KL}(\pi_\theta\Vert\pi_{\mathrm{ref}})$ have closed forms, so the full gradient $\nabla_{\!\theta} J$ is computable. Each step rolls the policy uphill on the objective surface above — no sampling, no clipping. This is the limit that GRPO approximates as $G \to \infty$.
$$\theta \;\leftarrow\; \theta \,+\, \eta\,\nabla_{\!\theta}\!\left[\,\mathbb{E}_{\pi_\theta}[r(a)] \,-\, \beta\,\mathrm{KL}(\pi_\theta\,\Vert\,\pi_{\mathrm{ref}})\,\right]$$
Press Play in the sidebar to roll $\pi_\theta$ uphill on $J$. The blue path is the analytic field $\nabla_{\!\theta} J$ — what an infinite-batch RLHF run would do. Compare it to the GRPO and DPO tabs to see how stochastic estimators bend this idealized trajectory.
03Policy Gradient — the simplest sampled estimator
Before GRPO, there's just policy gradient (REINFORCE with a mean baseline). At each step, draw a fresh batch of $N$ actions from the current $\pi_\theta$ — no frozen $\pi_{\mathrm{old}}$, so no importance ratio and no clipping. Score them with $r$, subtract the in-batch mean as a control variate, and follow the resulting Monte-Carlo estimate of $\nabla_{\!\theta}J$.
$$\widehat{\nabla_{\!\theta} J} \;=\; \frac{1}{N}\sum_{i=1}^{N}\big(r_i \,-\, \bar r\big)\,\nabla_{\!\theta}\log\pi_\theta(a_i) \;-\; \beta\,\nabla_{\!\theta}\mathrm{KL}\!\left(\pi_\theta\,\Vert\,\pi_{\mathrm{ref}}\right),$$
and then $\theta \leftarrow \theta + \eta\,\widehat{\nabla_{\!\theta} J}$. As $N \to \infty$ this estimator converges to the exact RLHF gradient from the previous tab; for finite $N$ the path is noisier and small-$N$ batches can overshoot. GRPO (next tab) is essentially this estimator, except it pins $\pi_{\mathrm{old}}$ across multiple updates (importance ratios), clips those ratios, and normalizes the advantage by its in-batch standard deviation.
04PG vs. exact gradient — Monte-Carlo noise around the analytic path
Run both Play buttons. The blue path is the analytic gradient field; the rose path is PG with the current batch size. Shrink $N$ — the rose trace gets visibly jittery; grow $N$ — it converges onto the blue. Compare to GRPO (next tab): GRPO adds three modifications (group normalization, importance ratios, ratio clipping) on top of this same idea.
03PPO — clipped updates with a learned baseline
PPO (Schulman et al., 2017) is the clipped surrogate that GRPO inherits. From a frozen $\pi_{\mathrm{old}}$ draw a batch of $N$ samples, score them, form per-sample advantages $\hat A_i = r_i - V$ against a learned scalar baseline $V$, and maximize the clipped objective
$$\mathcal{L}_{\mathrm{PPO}}(\theta) \;=\; \frac{1}{N}\sum_{i=1}^{N} \min\!\Big(\rho_i\,\hat A_i,\ \mathrm{clip}(\rho_i,\,1{-}\varepsilon,\,1{+}\varepsilon)\,\hat A_i\Big) \;-\; \beta\,\mathrm{KL}\!\left(\pi_\theta \,\Vert\, \pi_{\mathrm{ref}}\right),$$
with $\rho_i = \pi_\theta(a_i)/\pi_{\mathrm{old}}(a_i)$. The baseline $V$ is trained jointly via $V \leftarrow V + \eta_V(\bar r - V)$ — gradient descent on $\tfrac{1}{2}(\bar r - V)^2$. Compared to PG: PPO adds importance ratios and clipping, and replaces the in-batch mean with a learned scalar. Compared to GRPO: the only difference is that GRPO uses the in-batch mean as the baseline and divides the advantage by the in-batch standard deviation.
04PPO vs. PG vs. exact gradient — clipping tames the variance
Blue is the analytic gradient field; rose is plain PG; brown is PPO with the current $N$, $\varepsilon$, and learned $V$. Tightening $\varepsilon$ shortens each step but reduces variance; loosening it lets PPO approach PG. GRPO (next tab) replaces $V$ with the in-batch mean and divides $\hat A_i$ by the in-batch std — the only differences from PPO.
03GRPO — one step, in slow motion
Real RL on language models can't compute $\mathbb{E}_{\pi_\theta}[r(a)]$ in closed form. Instead, it samples a group of $G$ completions $a_1,\ldots,a_G \sim \pi_{\mathrm{old}}$, scores each, and uses the in-group mean as the baseline. GRPO then takes a clipped policy-gradient step that maximizes
$$\mathcal{L}_{\mathrm{GRPO}}(\theta) \;=\; \frac{1}{G}\sum_{i=1}^{G} \min\!\Big(\rho_i\,\hat A_i,\ \mathrm{clip}(\rho_i,\,1{-}\varepsilon,\,1{+}\varepsilon)\,\hat A_i\Big) \;-\; \beta\,\mathrm{KL}\!\left(\pi_\theta \,\Vert\, \pi_{\mathrm{ref}}\right),$$
where $\rho_i = \pi_\theta(a_i)/\pi_{\mathrm{old}}(a_i)$ is the importance ratio and $\hat A_i$ is the group-normalized advantage.
04GRPO vs. exact gradient — same landscape, different paths
Run the Play buttons across tabs. The blue path is the analytic limit; rose is plain PG (REINFORCE with an in-batch mean baseline); brown is PPO (the same clipped surrogate as GRPO, but with a learned scalar baseline $V$ instead of the group mean); red dotted is GRPO with the current $G$, $\varepsilon$, and group-normalized advantages. PG, PPO, and GRPO are three Monte-Carlo estimates of the same gradient — PPO adds importance ratios and clipping on top of PG; GRPO additionally normalizes the advantage by the in-batch standard deviation.
03DPO — learning from preferences, without a reward model
DPO (Rafailov et al., 2023) sidesteps the reward model entirely. Given a dataset of preference pairs $(a_w \succ a_l)$ — typically labeled by humans — it trains $\pi_\theta$ directly via a logistic loss whose implicit reward is $\beta_{\mathrm{DPO}}\,\log\pi_\theta(a)/\pi_{\mathrm{ref}}(a)$. With a sufficiently expressive policy and matched data model, its population target is the same unrestricted $\pi^\star$ as RLHF; the best member of a restricted Gaussian family need not coincide.
$$\mathcal{L}_{\mathrm{DPO}}(\theta) \;=\; -\,\mathbb{E}_{(a_w,\,a_l)\sim\mathcal{D}}\!\left[\log\sigma\!\left(\underbrace{\beta_{\mathrm{DPO}}\,\log\dfrac{\pi_\theta(a_w)}{\pi_{\mathrm{ref}}(a_w)} \;-\; \beta_{\mathrm{DPO}}\,\log\dfrac{\pi_\theta(a_l)}{\pi_{\mathrm{ref}}(a_l)}}_{h\;=\;\text{implicit margin}}\right)\right]$$
Here we synthesize preferences: draw pairs from $\pi_{\mathrm{ref}}$ and label them with a noisy Bradley-Terry oracle on $r(a)$, so we know the ground truth and can compare DPO's trajectory to RLHF's.
Background is still $J(\theta) = \mathbb{E}[r] - \beta\,\mathrm{KL}$ — what RLHF maximizes. DPO doesn't see $J$; it minimizes the logistic loss above. Its purple path can still improve $J$, but finite preference data, label noise, temperature mismatch, and the Gaussian restriction can give it a different endpoint from the blue Gaussian-$J$ optimum.
Press Step or Play in the sidebar and watch the violet curve climb as DPO reorders the fixed preference set. The implicit-margin accuracy ($h_i>0$) is what the loss directly drives; the red curve drops the $\pi_{\mathrm{ref}}$ correction and asks only whether $\pi_\theta$ itself ranks $a_w$ above $a_l$. The gray line — how often $\pi_{\mathrm{ref}}$ alone gets the pair right — is flat because the reference never moves; it's the baseline DPO starts from. With label noise ($\nu>0$) accuracy plateaus below 100%: some pairs are mislabeled and can't all be satisfied at once. Resample draws a new preference set without clearing the curve, so you can see the accuracy jump as the data changes.
03DRO — fitting reward with a policy log-ratio
Direct Reward Optimisation (Richemond et al., 2024) sidesteps preference pairs and pairwise loss entirely. Given offline scalar-reward data $(a_i, r_i)$ — here, $a_i \sim \pi_{\mathrm{ref}}$ and $r_i = r(a_i)$ — DRO jointly fits the policy $\pi_\theta$ and a scalar value $V$ by minimizing
$$\mathcal{L}_{\mathrm{DRO}}(\theta, V) \;=\; \tfrac{1}{2}\,\mathbb{E}_{a\sim\pi_{\mathrm{ref}}}\!\left[\,\Big(\underbrace{r(a) \,-\, V \,-\, \beta_{\mathrm{DRO}}\log\dfrac{\pi_\theta(a)}{\pi_{\mathrm{ref}}(a)}}_{\delta(a)\;=\;\text{residual}}\Big)^{2}\,\right].$$
$V$ is another learned parameter, not a hand-tuned constant. Raw MSE descent uses
$$\theta \;\leftarrow\; \theta \,+\, \eta_\theta\;\beta_{\mathrm{DRO}}\,\mathbb{E}\!\left[\delta(a)\,\nabla_{\!\theta}\log\pi_\theta(a)\right], \qquad V \;\leftarrow\; V \,+\, \eta_V\,\mathbb{E}\!\left[\delta(a)\right].$$
The practical DRO-V option divides the policy update by $\beta_{\mathrm{DRO}}$ for conditioning and performs the scale update in $\log\sigma$; the value update is unchanged. At the DRO temperature, the unrestricted target is $\pi^\star_{\mathrm{DRO}}(a)=\pi_{\mathrm{ref}}(a)e^{r(a)/\beta_{\mathrm{DRO}}}/Z_{\mathrm{DRO}}$. If the policy class could represent it, choosing $V=\beta_{\mathrm{DRO}}\log Z_{\mathrm{DRO}}$ would make the fitted reward equal the target reward at every action: $\widehat r_{\theta,V}(a)=r(a)$, hence $\delta(a)=0$ everywhere — not merely zero on average. Here the ratio of two Gaussian densities has only a quadratic log, so it generally cannot reproduce the bumpy reward and the loss stops at a positive approximation-error floor.
Population means the ideal infinite-data loss: average the squared residual over all actions, weighted by $\pi_{\mathrm{ref}}$, rather than estimating it from the current sampled batch. In this toy it is computed analytically, so it has no sampling noise. Profiled out means that every $(\mu,\sigma)$ pixel uses whichever scalar $V$ minimizes that pixel's loss. The batch surface performs the same minimization using only the fixed offline samples. Because DRO loss explodes near $\sigma=0$, the two loss views use a disclosed nonlinear color scale that expands the useful basin and saturates the darkest 10% of cells; colorbar labels and hover values remain the exact loss.
Switch surfaces without changing the trajectory. The three target markers separate the best Gaussian for $J$, the best Gaussian for population DRO, and the optimum fitted to this finite offline batch. Their separation is policy-class error plus sampling error — not an optimizer bug.
Both conditional targets move as the policy moves. Their gap measures finite-batch sampling error; the lag from the learned $V$ to the batch target measures value-optimizer error. The unrestricted value $\beta_{\mathrm{DRO}}\log Z_{\mathrm{DRO}}$ is shown only as a reference and has no fixed ordering relative to the targets of an arbitrary Gaussian policy.
The batch curve is what the optimizer sees; the population curve is an analytic expectation under $\pi_{\mathrm{ref}}$. A falling batch loss with rising population loss exposes finite-dataset overfitting. The positive floor that remains in both curves is approximation error from representing a bumpy reward with a quadratic Gaussian log-ratio.
03Compare — six trajectories, one landscape
Everything we've seen so far, one map. Press Run all in the sidebar to launch each algorithm from the same initial point $(\mu_\theta,\sigma_\theta)$ — the current red dot — with fixed default hyperparameters. Each algorithm runs for the chosen number of optimizer steps; the trajectories are then drawn on top of the shared objective surface. This is a controlled visual comparison, not an equal-data-budget benchmark: the online methods refresh rewards, while DPO and DRO reuse fixed offline data.
A few things to look for: (1) the blue exact path is deterministic; everything else depends on sampled data. (2) PG, PPO, and GRPO estimate the Gaussian $J$ direction with different variance and clipping. (3) DPO and DRO never see $J$ directly; a rise in $J$ is useful evidence, while a different plateau can be a genuine objective projection rather than failed optimization. (4) Switch reward presets to see when that distinction becomes large.
The same runs as the map above, scored by the RLHF objective $J(\theta) = \mathbb{E}[r] - \beta\,\mathrm{KL}$ at every step. This answers what the 2D paths can't: which method climbs fastest, which plateaus, and how far below the best Gaussian each one settles. The dotted blue line is the achievable Gaussian ceiling; the dashed gray $J(\pi^\star)$ line is the generally higher unrestricted ceiling. DPO and DRO never see $J$ — watching their curves rise anyway is the point. Set Runs per algorithm > 1 to get a median line with a min–max band, separating an algorithm's typical behavior from a lucky draw. The toggles in the map's title hide curves here too.