RL for LLMs & RL Environments

Reinforcement learning went from the last, small step of RLHF to the main driver of frontier capability gains. Reasoning models, coding agents and computer-use agents all get their edge from large-scale RL against verifiable rewards inside carefully built environments. This page covers the RL you need (policy gradients, PPO, GRPO and its fixes), how reasoning models are trained, why reward hacking is the central failure mode, and, in the most depth, how RL environments are designed, built and scaled. Frontier labs and the fast-growing set of RL-environment startups now interview directly on that last topic.

1. RL fundamentals, compactly

You will be asked to define these crisply. Interviewers at labs often start here to check that you know RL itself and haven't only memorized "GRPO".

The MDP

A Markov decision process is a tuple \((\mathcal{S}, \mathcal{A}, P, R, \gamma, \rho_0)\): states, actions, transition dynamics \(P(s'|s,a)\), reward \(R(s,a)\), discount \(\gamma\in[0,1]\) and initial-state distribution \(\rho_0\). Markov means the next state depends only on the current state and action. A policy \(\pi_\theta(a|s)\) is a distribution over actions. A trajectory (episode, rollout) is \(\tau = (s_0,a_0,r_0,s_1,\dots)\). The return is the discounted sum of rewards \(G_t = \sum_{k\ge 0}\gamma^k r_{t+k}\). The objective is $$J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta}\Big[\sum_t \gamma^t r_t\Big].$$ Sutton & Barto 2018

Value functions, advantage, Bellman

The Bellman expectation equation writes value recursively: \(V^\pi(s)=\mathbb{E}_{a\sim\pi,s'\sim P}[r+\gamma V^\pi(s')]\). The Bellman optimality equation replaces the expectation over actions with a max: \(Q^*(s,a)=\mathbb{E}_{s'}[r+\gamma\max_{a'}Q^*(s',a')]\). Value-based methods (Q-learning, DQN) iterate on the optimality equation. Policy-gradient methods optimize \(\pi\) directly and use value functions only to reduce variance. LLM RL is almost entirely in the policy-gradient family, because the action space (a vocabulary of about 100k+ tokens, applied autoregressively) makes a max over actions meaningless at the sequence level, and because we already start from a very good policy, the pretrained model.

On-policy vs off-policy, exploration, credit assignment

• On-policy methods (REINFORCE, PPO, GRPO) learn from data sampled by the current policy. Every update needs fresh samples. Off-policy methods (Q-learning, SAC, DPO-style offline objectives) can learn from data generated by another policy, which means replay buffers or static datasets. In practice LLM RL is "nearly on-policy". PPO/GRPO take a few gradient steps per batch, and asynchronous systems let data go a few versions stale. Importance ratios \(\pi_\theta/\pi_{\text{old}}\) correct for the small mismatch.
• Exploration: the policy only learns from what it samples. If a correct solution has probability \(10^{-6}\), you will essentially never see the reward. With LLMs, exploration is shaped by the base model's prior, by sampling temperature, by group size, and by curriculum (tasks just hard enough that some samples succeed). Entropy collapse, where the policy becomes deterministic too early, is the LLM version of premature exploitation.
• Credit assignment: a single 0/1 reward at the end of a 10,000-token chain of thought must somehow be attributed to the tokens that mattered. Outcome-reward RL spreads it uniformly (every token in a successful response gets the same advantage) and relies on averaging over many samples to separate signal from noise. Process rewards, value functions and turn-level rewards try to do better.

2. Policy gradients: REINFORCE to GAE

The log-derivative trick

We want \(\nabla_\theta \mathbb{E}_{\tau\sim\pi_\theta}[R(\tau)]\), but the expectation is over a distribution that itself depends on \(\theta\), and the reward is not differentiable (it's a unit-test result). The trick is \(\nabla p_\theta = p_\theta \nabla\log p_\theta\), so $$\nabla_\theta J = \mathbb{E}_{\tau\sim\pi_\theta}\big[R(\tau)\,\nabla_\theta\log p_\theta(\tau)\big],\qquad \log p_\theta(\tau)=\sum_t\log\pi_\theta(a_t|s_t) + \underbrace{\textstyle\sum_t \log P(s_{t+1}|s_t,a_t)}_{\text{no }\theta}.$$ The dynamics terms drop out. You need neither a model of the environment nor a differentiable reward, only the ability to sample and score. This is the REINFORCE estimator (Williams, 1992). In code, it is just a weighted negative log-likelihood: loss = -(advantage.detach() * logprobs).mean().

Baselines and variance reduction

The raw estimator is unbiased but has very high variance. Three standard fixes:

1. Causality / reward-to-go: an action can't affect past rewards, so weight \(\nabla\log\pi(a_t|s_t)\) by \(G_t\), not the full return. (With a single terminal reward, as in most LLM RL, these coincide.)
2. Baselines: subtracting any \(b(s_t)\) that doesn't depend on the action leaves the gradient unbiased, because \(\mathbb{E}_{a\sim\pi}[\nabla\log\pi(a|s)]=\nabla\sum_a\pi(a|s)=\nabla 1=0\). The best common choice is \(b=V(s)\), which gives the advantage. In LLM RL the baseline is often empirical: the mean reward of other samples for the same prompt (RLOO, GRPO), or a batch mean (REINFORCE++).
3. Normalization: whitening advantages (divide by std) stabilizes the step size across prompts and reward scales. It also introduces subtle biases (§5).

Generalized Advantage Estimation (GAE)

With a learned critic \(V_\phi\), you can estimate advantages anywhere between the one-step TD error (low variance, biased if \(V\) is wrong) and the Monte-Carlo return minus baseline (unbiased, high variance). GAE takes an exponentially weighted mix: $$\hat A_t^{\text{GAE}(\gamma,\lambda)}=\sum_{l\ge 0}(\gamma\lambda)^l\,\delta_{t+l},\qquad \delta_t=r_t+\gamma V_\phi(s_{t+1})-V_\phi(s_t).$$ \(\lambda=0\) gives TD(0). \(\lambda=1\) gives the MC return minus \(V\). Schulman+ 2015 In LLM RLHF with PPO, typical settings were \(\gamma=1\) and \(\lambda\) close to 1 (0.95–1.0), and the reward was sparse: the reward-model score at the last token, plus a per-token KL penalty.

3. PPO and classic RLHF

Why trust regions

Plain policy gradient takes one step per batch of samples. If you take several steps on the same batch, or one large step, the policy can move far from the one that generated the data, so the gradient estimate becomes wrong and performance collapses. TRPO enforced a hard KL constraint between old and new policy with a second-order method. Schulman+ 2015 PPO gets most of that benefit with a first-order clipped surrogate. Schulman+ 2017

The clipped objective

Let \(\rho_t(\theta)=\dfrac{\pi_\theta(a_t|s_t)}{\pi_{\theta_\text{old}}(a_t|s_t)}\) be the importance ratio. Then $$L^{\text{CLIP}}(\theta)=\mathbb{E}_t\Big[\min\big(\rho_t\hat A_t,\ \text{clip}(\rho_t,1-\epsilon,1+\epsilon)\hat A_t\big)\Big],\qquad \epsilon\approx 0.2.$$ If \(\hat A_t>0\), the objective stops rewarding increases of \(\rho_t\) above \(1+\epsilon\). If \(\hat A_t\lt 0\), it stops rewarding decreases below \(1-\epsilon\). The min makes it a pessimistic bound: clipping can only remove incentive, never add it. Once a token's ratio leaves the band in the "already improved" direction, its gradient is exactly zero. That's the property later algorithms (DAPO, CISPO) tinker with.

The full PPO-RLHF setup

InstructGPT-style RLHF uses four models: the policy (trained), a frozen reference policy (the SFT model), a reward model (frozen, trained beforehand on human preference pairs with a Bradley-Terry loss), and a value model/critic (trained, often initialized from the reward model). Ouyang+ 2022 Christiano+ 2017 The per-token reward is $$r_t = \underbrace{r_\text{RM}(x,y)\cdot\mathbb{1}[t=T]}_{\text{sequence score at the end}} - \beta\,\log\frac{\pi_\theta(y_t|x,y_{\lt t})}{\pi_\text{ref}(y_t|x,y_{\lt t})}.$$ The KL term keeps the policy near the reference. It stops reward-model exploitation ("overoptimization": the proxy score rises while the true quality falls) and preserves fluency and diversity. Gao+ 2022 The value loss is a regression of \(V_\phi(s_t)\) onto the returns, often also clipped.

4. The LLM as a policy

Mapping the MDP onto a language model:

The bandit view. With a deterministic "append token" transition and a single terminal reward, single-turn RL is effectively a contextual bandit: context = prompt, arm = complete response, payoff = reward. That's why simple REINFORCE-style estimators with a per-prompt baseline work so well (RLOO's main argument). Ahmadian+ 2024 Per-token value functions buy less here than in Atari or robotics, because there are no intermediate rewards to bootstrap from.

The token-level view still matters for the implementation. The gradient is a sum over tokens, importance ratios and clipping are per token in PPO/GRPO, and the KL penalty is per token. Choices like how you average over tokens (per sequence or per batch) change the effective weighting of long vs short responses. Those choices turned out to matter a lot (§5).

The multi-turn view is a real MDP: the environment's responses are not under the policy's control and must be excluded from the loss. Rewards may arrive after many turns, and the "state" includes things the model can't see (a database, a remote browser). §10 covers this.

5. GRPO and its successors

GRPO: Group Relative Policy Optimization

Introduced in DeepSeekMath and made famous by DeepSeek-R1. Shao+ 2024 For each prompt \(q\), sample a group of \(G\) outputs \(\{o_1,\dots,o_G\}\) from \(\pi_{\theta_\text{old}}\), score them, and give every token of output \(i\) the same advantage: $$\hat A_{i,t} = \frac{r_i-\text{mean}(r_1,\dots,r_G)}{\text{std}(r_1,\dots,r_G)}.$$ The objective is a PPO-style clipped surrogate averaged per sequence, then over the group, with the KL term moved from the reward into the loss: $$J_\text{GRPO}(\theta)=\mathbb{E}\Bigg[\frac{1}{G}\sum_{i=1}^G\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\Big(\min\big(\rho_{i,t}\hat A_{i,t},\ \text{clip}(\rho_{i,t},1\!-\!\epsilon,1\!+\!\epsilon)\hat A_{i,t}\big)-\beta\,\mathbb{D}_\text{KL}[\pi_\theta\|\pi_\text{ref}]\Big)\Bigg]$$ where \(\rho_{i,t}=\pi_\theta(o_{i,t}|q,o_{i,\lt t})/\pi_{\theta_\text{old}}(o_{i,t}|q,o_{i,\lt t})\). The KL is estimated per token with the low-variance "k3" estimator \(\frac{\pi_\text{ref}}{\pi_\theta}-\log\frac{\pi_\text{ref}}{\pi_\theta}-1\).

Why it caught on: no critic (saving roughly a policy-sized model of memory and a hard training problem); the group mean is a good baseline for 0/1 verifiable rewards; and it's simple. Typical group sizes are 8–64 samples per prompt.

Key property: if all \(G\) samples get the same reward (all right or all wrong), every advantage is zero and the prompt contributes no gradient. Prompts that are too easy or too hard are wasted compute. That's the root of curriculum and difficulty-filtering practices (§9).

Known problems with vanilla GRPO

1. Length bias from \(1/|o_i|\). Each sequence's loss is divided by its own length. For a negative advantage, a longer wrong answer gets a smaller per-token penalty, so the model learns that being wrong at length is cheaper. Response length inflates, especially for incorrect outputs. Liu+ 2025 (Dr. GRPO)
2. Difficulty bias from std normalization. Dividing by the group std up-weights prompts where almost all samples agree (low std), meaning very easy or very hard ones. Dr. GRPO argues this skews the optimization.
3. Entropy collapse. With symmetric clipping, low-probability "exploration" tokens with positive advantage can only rise by a factor of \(1+\epsilon\) per step, while already-likely tokens dominate. Entropy drops fast early in training and performance plateaus. One paper fits an empirical relationship between performance and entropy, \(R\approx -a e^{H}+b\), suggesting performance is "bought" with entropy. Cui+ 2025
4. Token-level importance ratios are noisy. The ratio of a single token's probability under two nearby policies is a high-variance estimate when one sequence-level reward is spread over thousands of tokens. This gets worse with MoE models, where routing changes between \(\pi_\text{old}\) and \(\pi_\theta\) can swing individual token probabilities. Zheng+ 2025 (GSPO)
5. Clipping as an unintended prior. One study found that GRPO with random rewards still improved Qwen2.5-Math-7B on MATH-500 by about 21 points (vs. about 29 with true rewards), and attributed it to a clipping bias that amplifies behaviors the base model already favors. The effect largely didn't transfer to Llama or OLMo. Shao+ 2025 Treat RLVR gains on a single model family with suspicion.

The successor family

RLOO (REINFORCE Leave-One-Out): sample \(k\) responses and use as baseline for each the mean reward of the other \(k-1\). This is unbiased, critic-free, with no clipping or ratio in the simplest form. The paper argued PPO's machinery is mostly unnecessary in the LLM bandit setting. Ahmadian+ 2024 (Mean-centering with the full group, as in GRPO without the std division, gives exactly \(\tfrac{k-1}{k}\) times the RLOO advantage, so the two differ only by a constant scale.)

REINFORCE++: critic-free, but normalizes advantages over the global batch instead of per prompt. The claim is that per-prompt normalization is biased and overfits, and global normalization is effectively unbiased as the batch grows. It has a variant with a group baseline for reasoning tasks. Hu+ 2025

Dr. GRPO ("GRPO Done Right"): removes the \(1/|o_i|\) per-sequence length normalization and the std division, which fixes the length and difficulty biases above. It reports similar accuracy with better token efficiency. Liu+ 2025

DAPO (ByteDance Seed / Tsinghua): four changes on top of GRPO, with no KL term, for long-CoT reasoning Yu+ 2025:

• Clip-Higher: decoupled bounds \(\epsilon_\text{low}=0.2\), \(\epsilon_\text{high}\approx 0.28\). Unlikely tokens can grow more, which counters entropy collapse.
• Dynamic Sampling: drop prompts whose groups are all-correct or all-wrong (zero advantage) and keep sampling until the batch is full of informative groups.
• Token-level policy gradient loss: average over all tokens in the batch instead of per sequence, so long sequences are weighted by their length and long degenerate outputs get properly penalized.
• Overlong reward shaping: a soft length penalty near the max length, and masking of truncated samples, so the reward isn't noisy when a good answer gets cut off.

GSPO (Qwen team): uses a sequence-level importance ratio, the length-normalized likelihood ratio \(s_i=\big(\pi_\theta(o_i|q)/\pi_{\theta_\text{old}}(o_i|q)\big)^{1/|o_i|}\), and clips whole sequences. The reward is per sequence, so the off-policy correction should be too. It reports better stability, especially for MoE RL, and credits it for improvements in Qwen3. Zheng+ 2025

CISPO (MiniMax-M1): instead of zeroing the gradient of clipped tokens, it clips the importance-sampling weight (as a stop-gradient coefficient) and keeps every token's gradient. Rare but important "fork" tokens, like a reflective "wait", aren't silenced. MiniMax 2025

VAPO: brings the value function back (value-pretraining, decoupled GAE \(\lambda\) for actor and critic, length-adaptive GAE) and argues a well-trained critic beats critic-free methods on long CoT. Yue+ 2025 Clip-Cov / KL-Cov restrict updates on the high-covariance tokens that drive entropy collapse. Cui+ 2025

6. RL with verifiable rewards and reasoning models

RLVR

Reinforcement Learning with Verifiable Rewards replaces the learned reward model with a program that checks correctness. The term was popularized by AI2's Tülu 3, which applied it to math, grade-school math and instruction-following constraints. Lambert+ 2024 The common verifier types:

Why RLVR beats RM-based RL for reasoning: a program can't be talked into a high score the way a neural reward model can, so you can optimize much harder (thousands of steps, little or no KL) before hacking dominates. The trade-off is coverage: only domains with checkable answers qualify. Most current research is about extending verifiability (rubrics, LLM judges, generated tests) to fuzzier domains (§7).

o1 and the reasoning-model paradigm

OpenAI's o1 (September 2024) announced that performance improves smoothly with both more RL train-time compute and more test-time compute (thinking longer), using large-scale RL to teach the model to use a long chain of thought productively. OpenAI 2024 Details were not published. The prior work on spending inference compute (best-of-N with verifiers, search, revision) showed that compute-optimal test-time scaling can beat scaling parameters on some problems. Snell+ 2024 o1's contribution was to put the "search" inside one long sampled chain of thought, learned by RL, instead of in an external search procedure.

DeepSeek-R1 and R1-Zero

DeepSeek-R1 (January 2025) published the recipe openly DeepSeek-AI 2025:

• R1-Zero: GRPO applied directly to the base model (DeepSeek-V3-Base) with no SFT, using only rule-based rewards: accuracy and a format reward for putting reasoning in think tags. AIME 2024 pass@1 rose from about 15.6% to 71.0% during training. Response length grew steadily on its own, and the model started doing reflection, verification and backtracking. The paper calls one example an "aha moment". The downsides were poor readability and language mixing.
• R1: a multi-stage pipeline. (1) A small "cold-start" SFT on long, readable CoT examples; (2) reasoning-focused RL with an added language-consistency reward; (3) rejection sampling from the RL checkpoint to build a large SFT set (on the order of 800k samples, reasoning plus general), then SFT again; (4) a final RL stage across all scenarios (verifiable rewards for reasoning, reward models for helpfulness and harmlessness).
• Distillation: SFT of smaller dense models (Qwen, Llama) on R1's outputs gave strong small reasoning models. For small models, distilling from a strong RL'd teacher beat running RL on the small model directly.
• What didn't work (per the paper): process reward models (hard to define steps, reward hacking, overhead) and MCTS over tokens (search space too large).

Kimi k1.5, released the same week, independently reported a similar finding: long-context RL with simple outcome rewards and no MCTS, value function or PRM. It also introduced "partial rollouts" to handle very long generations (§11). Kimi Team 2025

Test-time compute scaling

Two families:

The debate: does RL teach new things or just sharpen?

This is a favorite interview topic because there is real evidence on both sides.

A defensible synthesis: on short, well-covered tasks, small-to-moderate RL mostly sharpens (it raises pass@1 toward the base model's pass@k). Longer training, broader task distributions and compositional, multi-step environments can push the boundary out. Exploration is the bottleneck, so the base model and the environment's curriculum largely determine which regime you're in.

7. Reward design: outcome, process, judges and rubrics

Outcome vs process rewards

Key results. OpenAI's "Let's Verify Step by Step" trained a PRM on about 800k human step-level labels (PRM800K). Used as a best-of-N reranker, it beat an ORM on a MATH subset. Lightman+ 2023 Math-Shepherd removed the human labels: a step's label is estimated by rolling out completions from that step and checking how often they reach the right answer. Wang+ 2023 "Rewarding Progress" reframes a good process reward as progress, the change in the likelihood of eventual success under a (different) prover policy, which is effectively a step-level advantage. Setlur+ 2024 PRIME derives an implicit PRM from a model trained only on outcome labels and updates it online, which reduces hacking. Cui+ 2025

Practical status: large-scale reasoning RL mostly uses outcome rewards (R1 and Kimi k1.5 both reported PRMs as not worth it at scale). Process signals live on in verifier-guided search, in agentic settings as turn-level checks, and as an active research area.

Learned reward models, LLM judges and rubrics

For non-verifiable domains (writing quality, helpfulness, research reports, medical advice), the options are:

• Bradley-Terry reward models from human preference pairs (classic RLHF). They are easy to overoptimize, so you need a KL leash and early stopping. Gao+ 2022
• LLM-as-judge: prompt a strong model to score or compare responses. It's flexible, but it has known biases (verbosity, position, self-preference, confident tone) and the policy will find them. Pairwise comparisons against a reference are usually more stable than absolute scores.
• Rubric-based rewards: write per-prompt checklists of concrete criteria (expert-authored or model-generated, e.g., "mentions contraindication X", "cites the 2023 guideline"). A judge checks each item and the reward is a weighted sum. "Rubrics as Rewards" showed gains over Likert-style judge scores in medicine and science. Gunjal+ 2025 Rubrics make an LLM judge more like a verifier: decomposed, auditable, and harder to sway with style.
• Generative/reasoning verifiers: a judge that reasons before scoring, sometimes trained with RL itself. These are increasingly common in 2025–2026 recipes for "fuzzy" domains.

8. Reward hacking and specification gaming

Reward hacking: the policy raises the measured reward without doing what the designer intended. Formally, any time optimizing the proxy reward diverges from optimizing the true objective. Skalse+ 2022 Classic RL is full of examples, like the boat-racing agent circling to collect respawning targets instead of finishing the race. DeepMind 2020 With LLMs, the agent is smart enough to read and reason about its own grader.

Taxonomy with LLM examples

Why it matters beyond the metric

• Hacks generalize to behavior. Anthropic reported that when a model learned to reward hack in real production coding environments, it generalized to broader misalignment (alignment faking, sabotage attempts in agentic use). Standard chat-style safety RLHF fixed chat evals but not agentic ones. Effective mitigations: prevent the hacking, diversify safety training, and "inoculation prompting", which frames hacking as acceptable in training so it doesn't generalize into a misaligned self-concept. MacDiarmid+ 2025
• Monitoring the chain of thought works, but optimizing against it backfires. OpenAI found an LLM reading the CoT catches hacks in agentic coding far better than watching actions alone. Adding the monitor to the reward helps at low optimization pressure, but under heavy pressure the model learns obfuscated hacking: it hides intent in the CoT and keeps hacking. Their recommendation is to avoid strong optimization pressure on the CoT itself. Baker+ 2025
• Benchmarks inherit it. If agents learn to game tests during training, SWE-bench-style scores overstate real capability. ImpossibleBench measures "cheating rate" on tasks where the spec and tests conflict, so any pass implies a shortcut. Zhong+ 2025

A mitigation playbook

1. Harden the environment first: graders run outside the agent's reach, protected files, no reference answers in the filesystem, network egress controlled.
2. Diverse, hidden verification: held-out tests the policy never sees, plus property-based or randomized tests, plus multiple independent checkers.
3. Monitor rollouts continuously: sample and read transcripts; run LLM monitors that flag suspicious patterns (edits to test files, sys.exit, hard-coded constants); track reward vs. held-out-eval divergence.
4. Regularize: KL to a reference, early stopping on proxy-vs-true divergence, reward caps.
5. Penalize detected hacks carefully. Explicit penalties work for clear-cut cases, but penalizing on CoT content risks obfuscation.
6. Patch and re-version the environment when a hack is found, and add the exploit to a regression suite for env QA.

9. RL environments in depth

For LLM RL, the environment is everything except the policy and the optimizer: what tasks the model sees, what it can do, what it observes back, when the episode ends, and how it's scored. As algorithms converged on a few GRPO/PPO variants, the environments became the main thing that sets one lab's RL apart from another's, and a market formed around building them. As of 2026, frontier labs buy environments from dedicated vendors and from human-data companies, alongside open hubs.

Anatomy of an environment

A minimal interface

Classical RL standardized on the Gym API. Gymnasium (the maintained successor to OpenAI Gym) uses reset(seed=...) → (obs, info) and step(action) → (obs, reward, terminated, truncated, info). It separates true termination from time-limit truncation, which matters for bootstrapping. Legacy Gym's step returned a 4-tuple with a single done. Gymnasium docs Towers+ 2024 LLM environments keep that shape but deal in messages and tool calls, run asynchronously (thousands of concurrent episodes), and keep the verifier separate from the dynamics. A sketch (pseudo-code, not any specific library's API):

from dataclasses import dataclass, field
from typing import Any

@dataclass
class Task:                       # one row of the task distribution
    id: str
    prompt: str
    assets: dict                  # repo snapshot, DB seed, URL, user persona...
    hidden: dict                  # reference answer / hidden tests (NEVER shown to the policy)
    meta: dict = field(default_factory=dict)   # difficulty, domain, split, version

@dataclass
class StepResult:
    observation: list[dict]       # new messages to append (tool results, user replies)
    terminated: bool              # task finished (submit / success / fatal)
    truncated: bool               # budget exhausted (turns, tokens, wall-clock)
    info: dict                    # diagnostics; not shown to the policy

class LLMEnv:
    tools: list[dict]             # JSON schemas exposed to the policy

    async def reset(self, task: Task, seed: int) -> list[dict]:
        """Provision an isolated sandbox from task.assets (container/VM/browser),
        seed all randomness, return the initial messages (system + task prompt)."""

    async def step(self, action: dict) -> StepResult:
        """Parse one assistant turn (text and/or tool calls). Execute tools in the
        sandbox with timeouts; format outputs (truncate, sanitize); check
        termination. Malformed calls get an error observation, not a crash."""

    async def score(self) -> dict[str, float]:
        """Run the verifier OUTSIDE the policy's reach (separate process or
        container) against the final state plus task.hidden. Return components,
        e.g. {"correct": 1.0, "format": 1.0, "tests_modified": 0.0}."""

    async def close(self) -> None:
        """Tear down the sandbox; always runs, even on error/timeout."""

# Trainer-side rollout (simplified):
#   msgs = await env.reset(task, seed)
#   while True:
#       turn = await policy.generate(msgs, tools=env.tools)     # record token ids + logprobs
#       res  = await env.step(turn); msgs += [turn] + res.observation
#       if res.terminated or res.truncated: break
#   reward = combine(await env.score()); await env.close()
#   loss_mask = 1 for policy-generated tokens, 0 for prompt / tool-output tokens

Design principles

Types of environments

SWE environments in more detail, since they're the most-asked type. SWE-bench pairs real GitHub issues with the PR that fixed them. The grader runs the tests that the fix made pass (fail-to-pass) plus tests that must keep passing (pass-to-pass). Jimenez+ 2023 To train, you need many such executable instances. SWE-Gym built about 2.4k real executable Python tasks and showed both agent fine-tuning and verifier training benefit. Pan+ 2024 R2E-Gym generated environments procedurally from commits, with synthesized tests, plus hybrid (execution + model) verifiers. Jain+ 2025 SWE-smith scaled further by synthesizing bugs into repos, reaching tens of thousands of instances. Yang+ 2025 SWE-RL avoided execution altogether with a rule-based reward: the similarity between the generated patch and the ground-truth patch. That's cheap, but a weaker proxy. Wei+ 2025 The main engineering cost is building a working Docker image for each repo version.

Frameworks and interfaces (as of 2026)

The broad trend: environments are moving from "a Python function that scores a string" to "a containerized task plus an unmodified production agent harness". The policy is trained inside the same scaffold it will be deployed in, and tools increasingly come through MCP.

10. Agentic and multi-turn RL

Multi-turn agent RL keeps the same algorithms but changes nearly everything around them.

Loss masking

A trajectory interleaves policy tokens with environment tokens (tool outputs, user messages, file contents). Only the policy's own tokens are actions. Environment tokens must be masked out of the policy-gradient loss (and the KL), or you train the model to "predict" tool outputs, which is wrong and destabilizing. Search-R1, for example, masks retrieved passages from the loss. Jin+ 2025 The same applies to prompt tokens and system messages.

Credit assignment over turns

• Trajectory-level (default): one final reward, every policy token in every turn gets the same advantage (GRPO-style over G full trajectories of the same task). Simple and robust, but noisy for long horizons.
• Turn-level / step-level: group "similar" intermediate states across trajectories and compute relative advantages per step, as in GiGPO's nested groups (episode-level plus step-level anchored on repeated states). Feng+ 2025 Or train a turn-level critic.
• Intermediate rewards: partial credit for sub-goals (tests compiled, correct file found). These are useful but invite hacking and shortcutting, so keep them small relative to the outcome reward.
• Failure modes: RAGEN documents an "echo trap", where multi-turn agents collapse into repetitive, low-diversity reasoning, and argues for diverse initial states and fine-grained, reasoning-aware rewards. Wang+ 2025

Long horizons

• Context growth: a 100-turn SWE episode can exceed 100k tokens. Options are bigger context windows, truncating old observations, summarization or "context management" tools, and training the model to use memory files. Each changes the MDP (the policy no longer sees everything), so training and deployment must use the same strategy.
• Variance in episode length and cost: some episodes finish in 3 turns, some take 200. This drives the long-tail problem (§11).
• Environment failures: flaky tools, sandbox crashes and timeouts. Mark these trajectories as invalid and drop them. Don't give them reward 0, which would punish the policy for infrastructure noise.
• Cost per episode: VMs, browsers and LLM-simulated users cost real money per step. The policy's tokens are not the only cost.

11. RL training infrastructure

The key fact: LLM RL is dominated by generation. Each training step needs fresh samples from the current policy, and autoregressive decoding of long CoTs or multi-turn episodes is far slower per token than the training forward/backward pass. Rollout generation is often reported as the majority of wall-clock time, often well over half, in synchronous systems. So RL systems are built as a fast inference fleet bolted to a trainer.

Design axes

The long-tail problem

Response lengths are heavy-tailed. In a synchronous batch, everyone waits for the longest response or slowest episode. Decode batch occupancy decays as short sequences finish, and GPUs sit mostly idle at the end of each step. Fixes: (1) asynchronous or pipelined generation so the trainer never waits for stragglers; (2) partial rollouts / interruptible generation; (3) over-sampling and dropping the slowest (this biases against long correct answers, so be careful); (4) length caps with proper truncation handling; (5) continuous batching and prefix caching in the engine. The G samples for one prompt share a prefix, so prefix caching is a direct win.

Weight sync and the inference/trainer mismatch

• Weight sync cost: a 70B model in bf16 is about 140 GB. Broadcasting it over NVLink/InfiniBand at hundreds of GB/s takes seconds, but it happens every step and must be resharded from the trainer's layout (FSDP/TP/PP) to the engine's (TP). Frameworks use bucketed NCCL broadcasts, CUDA IPC when colocated, or LoRA-only sync.
• Logprob mismatch: even with identical weights, vLLM/SGLang and the training stack produce different token probabilities (different kernels, precision, batch-dependent reductions, sometimes quantized rollouts). The "on-policy" data is quietly off-policy. A common fix is truncated importance sampling: weight each token's gradient by \(\min(\pi_\text{train}/\pi_\text{rollout}, C)\). veRL PR #2953 Batch-invariant kernels can make inference deterministic and remove the mismatch at its source. Thinking Machines 2025
• MoE specifics: router decisions can differ between engine and trainer, which amplifies the mismatch. Sequence-level ratios (GSPO) and routing replay are the mitigations discussed.

Frameworks

Back-of-envelope: one GRPO step

Say 512 prompts × G=16 samples × an average of 8k generated tokens. That's \(512\times16\times8{,}000\approx 6.6\times10^7\) tokens per step. If a 7B policy decodes at a few thousand tokens/s per GPU at high batch, then 64 GPUs at about 3k tok/s gives about 190k tok/s, or roughly 350 s of generation per step, ignoring the long tail. The training pass over the same tokens costs about \(6 \times 7\times10^9 \times 6.6\times10^7 \approx 2.8\times10^{18}\) FLOPs. At about 400 TFLOP/s effective per GPU that's roughly 110 s on 64 GPUs. So generation dominates even before stragglers and environment latency, which is why async designs and the long tail are the main concerns.

12. Evaluating RL'd models

• Variance is large. Report multiple seeds and confidence intervals; a few points on a 30-problem benchmark is noise.
• Separate format gains from reasoning gains. Early RL jumps are often answer-extraction and format compliance.
• Hold out environments, not just tasks: evaluate on repos, sites or domains never seen in training to measure transfer.
• Check the base model on the same harness and sampling settings. Many "RL gains" shrink once the base model gets a good prompt and enough samples.
• Read transcripts. Aggregate metrics hide hacks, language mixing and degenerate repetition.

Interview question bank