A Practical Recipe for Production RL

Introduction

Enterprises rely on LLMs for a diverse set of applications, with models processing trillions of tokens a day. Production data from these deployments contains experiences from completing economically valuable work, but current methods for converting this data into training signal are limited. In past work (e.g. RMSD⌝, MOPD⌝) we have explored variations of self-distillation, which applies when we have textual feedback from users or offline labels. In this blog post we discuss a new paradigm, “production RL (prod RL),” which creates a continuous feedback loop where models learn from positive and negative real world experiences, improve, and then get redeployed to collect new experiences.

We present a training recipe for prod RL that steadily and efficiently improves the model using a data stream directly from production inference deployments, without a model harness or synthetic environment. This recipe combines special metadata recorded during production inference with run stabilizing techniques like score centering and KL penalties. We also show an asynchronous version of the recipe remains stable while providing a win on speed.

One natural way to view prod RL is a repeated cycle of collecting deployment traffic, constructing a dataset, training the model, and redeploying the updated checkpoint. In this view, we are doing a series of independent training runs that build on top of each other.

However, because successive rounds of deployment updates and training runs are coupled, we instead consider prod RL as one long training run where we take multiple gradient steps at a time between redeployments. The coupling between rounds includes both training dynamics (carrying over optimizer state, policy changes in one run leading to instability in the next) and practical constraints (users expecting similar or better performance across updates).

In light of these dependencies, any recipe we use for prod RL must be amenable to training under several unusual constraints, such as:

1. we are deploying to real customers: runs should not result in noticeable degradations from one checkpoint to the next;
2. an end-to-end run is expensive: stability is important since the process is much more time-consuming and resource-intensive than an ordinary RL run;
3. train-rollout mismatch is high: inference deployments and training engines differ significantly and the staleness distribution is non-stationary;
4. no harness or task replay: data arrives in the form of single rollouts rather than tasks.

Solving these challenges unlocks benefits including:

• very large volumes of data, up to the entirety of an inference endpoint’s workload;
• gold-standard feedback from real users every time we redeploy a trained model on in-distribution data;
• a simple solution that integrates with customer inference workloads with minimal setup cost.

The Prod RL Loop

The prod RL loop consists of many rounds. Each round collects interactions from a deployed checkpoint, takes $S$ optimizer steps on minibatches of $B$ trajectories, and deploys the updated checkpoint. Write $\theta_k$ for the $k$-th deployed checkpoint and $\mu_k$ for its deployed policy (to distinguish it from the same checkpoint in our training engine). Within the round, $\theta$ denotes the evolving trainer parameters.

1. Build the dataset: collect $SB$ trajectories from $\mu_k$: tasks $x_i$ and rewards $R_i$

2. Train on minibatches: initialize $\theta\leftarrow\theta_k$, then take $S$ Adam steps on minibatches of $B$ trajectories.

• Compute advantages $A_i$ from the rewards $R_i$ using some baseline (e.g. FlashREINFORCE). For each model-generated token $a$ in context $h$, start with the score $s=\nabla_\theta\log\pi_\theta(a\mid h)$ and weight $w=1$.
• Apply some train-rollout mismatch correction to $w$ and/or $s$ (such as importance sampling or score centering)
• Aggregate the contributions $g=wA_i s$ over the minibatch, optionally apply a KL penalty, and apply Adam:

Here $T_i$ counts the model-generated tokens in trajectory $i$, and $s_{i,t}$ includes score centering when enabled.

3. Redeploy the model: save $\theta_{k+1}\leftarrow\theta$, then export and quantize that checkpoint to produce the next deployed policy $\mu_{k+1}$.

A Prod RL Laboratory

Our experiments simulate prod RL using Qwen3.6-35B-A3B on multi-turn chess puzzles from Lichess. The model receives a textual board position and submits moves through a tool; the environment checks them against the recorded solution and automatically plays the opponent’s replies.

We run this loop through our AC2 platform: we collect scored trajectories from a model deployment into a dataset, train on that dataset, and redeploy the updated checkpoint for the next round. We use separate evaluation jobs on a held-out puzzle set to track the model’s progress throughout training, and intentionally match the deployment and eval setup with the deployments that we use for production with customers.

The Importance of Faithful Replay

Training on data from a production endpoint requires a representation of this data which preserves the model’s original inputs and actions. Although recording these interactions is conceptually straightforward, the representation determines which training algorithms are possible, and inaccuracies at this first stage can severely destabilize learning.

A straightforward approach is to record text transcripts or the structured JSON responses returned by the deployment. However, the retokenizing and chat processing that are required to turn a response object back into a training example can change the model’s original inputs and actions. Some examples:

• Retokenizing concatenated text can change token boundaries. Further complicating this, in multi-turn trajectories, the same conversation history could be tokenized differently as prompts across completion requests;
• Chat processing can remove or rewrite formatting such as special tokens and newlines, changing the character sequence used for training. Therefore, token IDs must be gathered before the chat processor is applied;
• It is also possible for harness components such as compaction tools or dynamically-injected hints to modify the conversation history between model calls

While these differences may appear minor at first glance, they have a large impact in practice. We compare two prod RL runs where one run uses retokenized traces and the other uses the original input/output tokens from the deployment, and find that retokenization causes the run to collapse:

Prod RL runs with 16 weight updates per redeployment at batch size 64, with vs. without recording deployment tokens.

In fact, even in a run where we take just one optimizer step per deployment (hence removing all staleness), the run using recorded tokens learns, but the run using retokenized text degrades. Note that the run with tokenization preserved is equivalent to a synchronous FlashREINFORCE⌝ run.

Prod RL runs with 1 weight update per redeployment at batch size 64, with vs. without recording deployment tokens.

These results reveal the importance of preserving exact prompt and response token IDs. Interestingly, this differs from the case of self-distillation which we have previously investigated, where retokenizing traces still provides lift. The simplest explanation for this behavior is that running a structured-JSON trace through a chat processor and retokenizing is actually a huge source of train-rollout mismatch. Because RL works by encouraging or penalizing behavior based on its outcome, a policy update on a retokenized trace reinforces or penalizes a different sequence of actions than those which were really taken.

Breaking Down Train-Rollout Mismatch

Stable learning from minibatches of stale data requires reliable policy gradient estimates. A central obstacle to this is train-rollout policy mismatch, which has two components:

• engine mismatch: differences between the kernels we use for training/serving, as well as between our BF16 training and NVFP4 deployment weights.
• staleness: the more steps we take before redeploying, the farther away our trainer policy gets from the policy that generated the data.

The classic method to correct for train-rollout mismatch is token-level importance sampling: each token contributes $w\,A\,\nabla_\theta\log\pi_\theta(a\mid h)$ to the policy gradient. Since we are taking multiple optimizer steps on data from the same deployment, we use the TIS + PPO-clip correction (Yao et al., 2025)⌝, which decomposes the importance weight into

Writing $r$ for that second ratio, the per-token objective we maximize is

Using these settings, we observe stable training runs with 16 weight updates between deployments on a batch size of 64 trajectories:

Reward curves for deployment-IS prod RL runs with 16 weight updates per deployment at batch size 64; three random seeds.

We can also gain the following intuition about our importance sampling ratios by plotting the mean absolute logprob differences $|\log \pi_{\theta_k} - \log \mu_k|$ and $|\log \pi_\theta - \log \pi_{\theta_k}|$. The first term measures the difference between the last trainer checkpoint policy and the rollout policy (engine mismatch), and it decreases gradually over the course of the entire run as the policy concentrates on successful behaviors. The second term measures the difference between the current trainer policy and the last trainer checkpoint policy (staleness), and it increases over the course of every series of minibatch updates while resetting every redeployment.

Train/inference IS ratio $|\log \pi_{\theta_k} - \log \mu_k|$ over a deployment-IS prod RL run with 16 weight updates per deployment at batch size 64.

Staleness IS ratio $|\log \pi_\theta - \log \pi_{\theta_k}|$ over a deployment-IS prod RL run with 16 weight updates per deployment at batch size 64.

These compose to give the net train-rollout mismatch over the course of the run $|\log \pi_\theta - \log \mu_k|$:

Train/inference IS ratio $|\log \pi_\theta - \log \mu_k|$ over a deployment-IS prod RL run with 16 weight updates per deployment at batch size 64.

Under these settings, the staleness term is a noticeable but relatively small component of train-rollout mismatch. If we were to take significantly more optimizer steps in between deployments, the relative contribution of staleness would be much higher and destabilize the run.

Requesting logprobs from an inference deployment may slow it down by ~5% due to computation and communication overheads, so we wondered if this extra information was really necessary. For instance, we might instead try computing the importance sampling denominator directly inside the training engine according to a reference model with upcasted deployment weights. This only corrects some of our train-rollout mismatch: it successfully corrects for staleness, only partially corrects for quantization, and does not account for engine mismatch. In practice, on the same settings as before, this hillclimbs but is very unstable and prone to collapse over longer horizons.

Reward curves for reference model importance sampling prod RL runs with 16 weight updates per deployment at batch size 64; three random seeds.

The effect of run collapse on reward seems sudden, so a natural hypothesis is that some bad data or training step in the most recent training run led to the degradation. However, restarting the run from checkpoints one or two deployments earlier still leads to immediate collapse, even when we shuffle new data, reset the optimizer state, or increase batch size. This suggests that there is a gradual loss of plasticity in the model rather than a phase change from stable to unstable. As such, controlling run stability is a problem scoped to the end-to-end prod RL run, since we cannot continue training when we detect regressions by simply “rolling back” a few checkpoints.

Score Centering Outperforms Importance Sampling

We also consider score centering⌝, an alternative RL stabilization technique which modifies each token’s contribution to the policy update:

The subtracted term is the average direction the trainer would move for a randomly-sampled inference policy token; hence, it “centers” the score $\nabla_\theta \log \pi_\theta (a|h)$ from the trainer. The score centering term can be similarly estimated by collecting extra metadata from the deployment (this can be done with little to no overhead). At a high level, we can view score centering as a way to estimate the gradient with respect to the rollout policy.

We empirically find that score centering outperforms importance sampling, hillclimbing both faster and higher. We do see increased run variance; we will explore mitigations for this later. This fits with prior results we have showing that score centering is superior to importance sampling in cases where train-rollout mismatch is high and has a non-stationary distribution (in this case, a sawtooth pattern from $0$ to $S$).

Reward curves for score centering (orange) vs. importance sampling (gray) prod RL runs with 16 weight updates per deployment at batch size 64; three random seeds each.

Accommodating Sampling Parameters

Inference deployments frequently employ sampling parameters such as top-$p$ and top-$k$ sampling to restrict the support of the serving policy distribution to a smaller set of tokens than the full vocabulary. These are not traditionally considered in standard RL settings, where we set top-$p=1$ and top-$k=\infty$ to minimize the train-rollout mismatch. However, in the prod RL case, we must take these sampling settings as given. Importance sampling must correct for a smaller support $S \subset V$ by renormalizing both deployment and the trainer probabilities over $S$. Meanwhile, our implementation of score centering does not require this renormalization. While these sampling parameters do further increase train-rollout mismatch, empirically with top-$p = 0.95$ and top-$k=20$, we do not see significant impacts on training performance.

Reward curves for score centering prod RL runs with 16 weight updates per deployment at batch size 64, on deployments with no top-p/top-k (gray) vs. top-$p = 0.95$/top-$k=20$ (orange); three random seeds each.

Further Optimizing the Recipe

Asynchronous Prod RL

The prod RL pipeline we described above alternates between data collection and training. However, factors such as large batches, long-context tasks, or frequent history edits that require repeated prefill computation could cause a training step to take substantial time. In a synchronous loop, the interactions we collect from the deployment during this period are discarded without ever being used for training. In such a scenario, we can get a speedup in time between redeployments by overlapping these stages, where the deployment collects the next batch while the trainer processes the previous one.

In the one-round asynchronous schedule considered here, data is already one training round old when its first update begins. Staleness therefore ranges from $S$ to $2S$ optimizer steps for asynchronous prod RL, compared to just $0$ to $S$ in the synchronous version. In practice, we found that this extra staleness difference causes importance sampling to collapse altogether. Score centering only exhibits a minor regression:

Reward curves for synchronous (gray) and one-round asynchronous (orange) score centering prod RL runs with 16 weight updates per deployment at batch size 64, on deployments with top-$p = 0.95$/top-$k=20$; three random seeds each.

Sampled KL Regularization Reduces Run Variance

We previously noted that score centering, though it seems to perform better than importance sampling, also has higher inter-run variance and seems less stable. We find that this can be mitigated by regularizing the current trainer policy $\pi_\theta$ toward the round-start reference $\pi_{\theta_k}$ by estimating the reverse-KL loss $D_{\mathrm{KL}}(\pi_\theta\Vert\pi_{\theta_k})$. This reference stays fixed for the $S$ optimizer steps within each round and is refreshed at the next round.

Computing this quantity exactly at each observed prefix requires comparing their distributions over the full vocabulary. Instead, we use a sampled-token estimator that requires only the probabilities assigned to the recorded token, which reduces memory and computation at the cost of sampling variance. We need to account for the differences between the sampling distribution $\mu$, the trainer policy $\pi_\theta$, and the initial checkpoint policy $\pi_{\theta_k}$:

Here, the deployment policy $\mu$ is $\mu_k$ for synchronous prod RL and $\mu_{k-1}$ for one-round asynchronous prod RL. We comment that, when the deployment has top-$p$ and top-$k$ filtering, this is actually a biased estimator of the KL loss: the identity is only valid when $\mu$ has full-vocabulary support, and also should theoretically require the renormalized probabilities $\tilde\mu(a)$ over the top token set.

In practice, we find that even with top-$p$ and top-$k$ sampling parameters, the KL penalty significantly reduces run variance and increases the stability of the plateau:

Reward curves for one-round asynchronous score centering prod RL runs with 16 weight updates per deployment at batch size 64, on deployments with top-$p = 0.95$/top-$k=20$, with (orange) vs. without (gray) a KL penalty; three random seeds each.

Conclusion

Every day, LLMs process trillions of tokens while working on tasks in production. There is a trove of information content in this production data, yet current methods of converting this to training signal are quite limited. Our recipe for prod RL provides a powerful method to hillclimb from this underutilized data source.

We find that learning from production data requires great care in faithfully replaying the model's experience and accounting for all sources of train-rollout mismatch. Our experiments show how these details determine the overall stability of the end-to-end prod RL run, with token recording, score centering, and KL penalties proving to be critical for running prod RL in practice. We also show that an asynchronous version of the prod RL recipe can improve data efficiency.

Prod RL closes the loop from inference deployments back to model training. AC2 brings deployment, production data collection, training, and evaluation into one platform; if you are interested in applying prod RL to turn production data into model improvements, reach out!