RL post-training for language models: from human preferences to verifier farms.

Reinforcement learning for language models gets presented as a modeling story. A new advantage estimator, a new regularizer, a new way of shaping the reward. When I read the actual technical reports, the modeling change is usually twenty lines of loss function, and the remaining forty pages are about generating rollouts fast enough to keep the GPUs busy, keeping thousands of sandboxes alive, working out whether a failed episode was the model's fault or a dead network, and reconciling two copies of the same weights that disagree about what they predict.

This post is the background I wish someone had written down for me in one place: where the objective came from, what changed when the reward stopped being a learned model and became a program, and which parts of the pipeline actually decide whether a run produces a usable model. I assume you know what a transformer is and what supervised fine-tuning does, and nothing past that.

How the objective got here

The core idea predates language models by decades. A policy takes actions, a scalar reward comes back, and you nudge the parameters so that high-reward actions become more likely. Plain policy gradients are high variance and easy to break, since one large update can move the policy somewhere it cannot recover from. Proximal Policy Optimization (Schulman et al., 2017) handled that with a clipped surrogate objective that stops rewarding an update once it moves past a fixed ratio of change from the sampling policy. It is cheap, it is stable enough, and it became the default.

The other half of the story is where the reward comes from. Hand-written reward functions work for games, where AlphaGo Zero (Silver et al., 2017) could self-play against an exact rulebook. They do not work for "write a helpful answer". Christiano et al. (2017) showed you can learn the reward instead: ask humans which of two trajectories is better, fit a reward model to those comparisons, then optimize the policy against the model. Ziegler et al. (2019) ported this to text, Stiennon et al. (2020) applied it to summarization and found the resulting summaries were preferred over the human reference summaries, and InstructGPT (Ouyang et al., 2022) turned it into the three-stage recipe everyone copied: supervised fine-tuning, a reward model on pairwise preferences, then PPO against that reward model with a KL penalty pulling back toward the SFT policy. Labellers preferred the 1.3B InstructGPT outputs over those of 175B GPT-3.

There is a branch off this path that skips the loop entirely. Direct Preference Optimization (Rafailov et al., 2023) shows that the reward model and the policy can be the same object, which collapses the procedure into a classification loss on preference pairs with no sampling at all. DPO is far easier to run and works well for style and helpfulness tuning. It does not help when the thing you care about involves the model interacting with something, because a fixed dataset of preference pairs contains no interaction.

Reward models break when you optimize them

A learned reward model is a fit to a finite sample of human judgments, and the policy is an effective search procedure looking for inputs that score highly. Those two facts do not coexist peacefully. Gao et al. (2022) measured it directly by treating a large reward model as synthetic ground truth and a smaller one as the trainable proxy. As optimization proceeds, the proxy score climbs steadily while the gold score rises, peaks, and then falls, with the turning point predictable as a function of the KL distance travelled from the starting policy. The proxy stops being a measurement and becomes a target.

Skalse et al. (2022) give the formal version: a proxy reward is hackable relative to a true reward when two policies exist that the proxy ranks one way and the truth ranks the other, and the conditions under which a pair of rewards is unhackable turn out to be close to trivial. Do not expect to write a proxy that survives arbitrary optimization pressure. The KL penalty in the InstructGPT recipe is doing exactly this job. It is a budget on how far you are allowed to exploit the reward model, set by whoever is reading the samples.

When the reward became a program

Mathematics and code have a property that helpfulness does not: you can check the answer. Cobbe et al. (2021) released GSM8K and trained verifiers to rank sampled solutions, which beat fine-tuning at equivalent cost. Lightman et al. (2023) asked whether to supervise the final answer or every intermediate step, collected 800K step-level labels, and found process supervision clearly better on MATH. Those verifiers were still learned models. The step that changed the field was dropping the model and using the checker itself.

The Tülu 3 report (Lambert et al., 2024) names the approach reinforcement learning with verifiable rewards: reward is 1 when a string-matched answer or an executed test passes and 0 otherwise, with no reward model anywhere in the loop. DeepSeek-R1 (DeepSeek-AI, 2025) took the extreme version, running RL with rule-based accuracy and format rewards directly on a base model with no supervised warm start. They reported long chains of thought, self-checking, and backtracking appearing on their own, alongside output unreadable enough that the released model added a cold-start SFT stage to fix it. Kimi k1.5 (Kimi Team, 2025) reported a comparable result with long-context rollouts as the central scaling axis.

A program is harder to exploit than a reward model, because there is no gradient pointing at its weak spots and no smooth surface to climb. It is not immune. If the check is a test suite, holes in the test suite are the reward, and models find them.

GRPO, and what removing the critic costs

PPO needs a value function to compute advantages, which for an LLM means a second network of comparable size, its own optimizer state, and its own training loop, all to fit a critic against a reward that arrives once at the end of a long sequence. DeepSeekMath (Shao et al., 2024) introduced Group Relative Policy Optimization, which deletes it. Sample a group of G completions for the same prompt, use the group's own reward statistics as the baseline, and give every token in a completion the same advantage.

rewards = [verify(prompt, r) for r in completions]   # G completions, one prompt
                    adv = (rewards - rewards.mean()) / (rewards.std() + 1e-6)
                    # every token of completion i is trained with advantage adv[i]

The memory saving is real and the change fits on a slide, so GRPO spread quickly. What it buys is paid for elsewhere. You now need G rollouts per prompt instead of one, so generation cost multiplies by the group size, and generation was already the slow part. The signal comes entirely from spread within the group, so a prompt where all G completions succeed and a prompt where all G fail both produce an advantage of zero and contribute nothing. Batch composition stops being a detail. Keeping prompts near intermediate success rates is the difference between a batch that teaches and a batch that costs a few thousand GPU-seconds to produce no gradient, which is why serious pipelines carry an explicit curriculum that promotes and retires prompts based on measured pass rates.

The normalization has consequences that are easy to miss. Liu et al. (2025) point out that dividing by the group standard deviation and by response length introduces biases into the optimization, one of which pushes incorrect responses to get longer. That is a plausible explanation for at least part of the response-length growth people report as emergent reasoning.

Two engines holding the same weights

Generation and gradient computation want opposite things from a GPU. Generation wants a paged KV cache, continuous batching, fused decode kernels, and often reduced precision. Training wants sharded parameters, activation checkpointing, and optimizer state. So every real RL framework runs two stacks: an inference engine such as vLLM or SGLang produces rollouts, a training engine such as FSDP or Megatron computes updates, and weights are synchronized between them. HybridFlow (Sheng et al., 2024) describes the architecture most current frameworks follow, a single controller expressing the RL dataflow on top of multi-controller distributed compute.

Splitting the work creates a subtle problem. The two engines assign different probabilities to the same token sequence even when they hold identical weights, because kernel implementations, batching nondeterminism, and quantized rollouts all perturb the logits slightly. Your gradient is therefore computed under a policy that is not the one that generated the data, on every step, permanently. The usual corrections are an importance sampling ratio between sampler and trainer with clipping or truncation on the tail, plus higher-precision rollouts when the divergence grows. Anyone who lets the sampler run asynchronously ahead of the trainer to keep hardware busy is widening that same gap deliberately and needs the same machinery to hold it together.

The reason people accept the complexity is wall clock. A math rollout is a few thousand tokens. An agent rollout is minutes of tool calls, file edits, and waiting on a network, and if the trainer sits idle through that, most of your cluster does nothing most of the time.

The environment is the hard part

Classic RL research had Gym (Brockman et al., 2016), a shared interface where reset and step meant the same thing everywhere and a benchmark was a package you installed. There is no equivalent for LLM agents, so everyone builds environments and most do not release them. The two that did become shared infrastructure show what the job involves. SWE-bench (Jimenez et al., 2023) pairs real GitHub issues with repository state and a hidden test suite, which makes it a reward function as much as a benchmark. WebArena (Zhou et al., 2023) runs self-hosted clones of a shopping site, a forum, a code host, and a wiki with programmatic validators, precisely because you cannot train an agent against the live internet and get reproducible episodes.

Once you run these at training scale, operational failures dominate. Environments hang. Containers leak state between episodes, so a task passes because of something the previous rollout left behind. A page fails to load and the policy gets punished for a network timeout, which is the worst of the three, because it writes noise into the gradient with the same sign as a real mistake. The countermeasures are unglamorous, and they separate a pipeline that trains from one that thrashes: a classifier that labels each failed episode as task failure, model failure, or environment failure; health checks that pull sick workers out of the pool; snapshot and reset paths fast enough that resetting is not the bottleneck; and audits of the verifiers themselves, since a test suite with false positives is a reward function aimed somewhere you did not intend.

What the training is actually doing

It helps to be precise about the size of the effect. Yue et al. (2025) compared base models against their RLVR-trained versions across math, code, and vision tasks at varying sampling budgets. The RL-trained models win comfortably at pass@1, and the gap closes as k grows until, at large k, the base model matches or overtakes them. Their reading is that RLVR raises the probability of solutions the base model could already reach and narrows the output distribution, rather than adding new reasoning behaviour, while distillation from a stronger model does extend the reachable set. The result has been argued over and the measurement is delicate, but it matches what the loop does mechanically, since every solution that gets reinforced was sampled from the current policy in the first place.

Three things I would carry into a project. The base model dominates the outcome, so spend effort there before tuning the RL. Report pass@k across a range of k rather than pass@1 alone, or you will not see the distribution narrowing. And log the diagnostics that reveal a failing run before the reward curve does: KL from the reference policy, policy entropy, mean response length, the fraction of groups with zero reward variance, and the agreement between sampler and trainer log-probabilities on the same sequences. When an RL run goes wrong, the reward is usually the last number to admit it.

Sources and further reading

Policy optimization and learning from preferences

• Schulman et al. (2017). Proximal Policy Optimization Algorithms. The clipped surrogate objective that most RLHF pipelines still use.
• Christiano et al. (2017). Deep Reinforcement Learning from Human Preferences. NeurIPS 2017.
• Ziegler et al. (2019). Fine-Tuning Language Models from Human Preferences.
• Stiennon et al. (2020). Learning to Summarize from Human Feedback. NeurIPS 2020.
• Ouyang et al. (2022). Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022. The InstructGPT paper, and the origin of the three-stage recipe.
• Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.
• Silver et al. (2017). Mastering the Game of Go without Human Knowledge. Nature 550.

Verifiers and verifiable rewards

• Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems. Introduces GSM8K along with solution-ranking verifiers.
• Lightman et al. (2023). Let's Verify Step by Step. ICLR 2024. Process supervision against outcome supervision, with the PRM800K label set.
• Lambert et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. Names and specifies reinforcement learning with verifiable rewards.
• Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. The paper that introduced GRPO.
• DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.
• Kimi Team (2025). Kimi k1.5: Scaling Reinforcement Learning with LLMs.

How reward and optimization go wrong

• Gao et al. (2022). Scaling Laws for Reward Model Overoptimization. ICML 2023. Measures the point at which proxy reward and gold reward part company.
• Skalse et al. (2022). Defining and Characterizing Reward Hacking. NeurIPS 2022.
• Liu et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. Identifies biases in GRPO's normalization terms.
• Yue et al. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. The pass@k analysis behind the sharpening argument.

Systems and environments

• Brockman et al. (2016). OpenAI Gym. The shared environment interface that LLM agent research still lacks an equivalent of.
• Sheng et al. (2024). HybridFlow: A Flexible and Efficient RLHF Framework. The hybrid controller design underlying several current RL training stacks.
• Jimenez et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. ICLR 2024.
• Zhou et al. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR 2024.

All posts