Growing demand for models that behave correctly and efficiently inside a company’s own tools and processes has made self-distillation a core part of our post-training work at Applied Compute. We’ve used it to learn from non-replayable production traces, reduce tool-call failures, teach models internal formats, and correct specific mistakes in long agent trajectories.
Production agents already generate a rich stream of feedback: retries, corrected tool calls, accepted edits, comments, and subsequent user messages with clarifications. The problem is that this feedback is attached to specific points in long traces, expressed in natural language, and difficult to turn into a scalar, training reward.
On-policy self-distillation turns that feedback into training signal directly, using nothing but the trace.
Running OPSD is relatively straightforward, but choosing relevant hints and getting visibility and confidence that the model's behavior is learning from them is not. The heavy lifting is deciding which behavior to change, what feedback will help, where that feedback belongs in the trajectory, and how to tell whether the model learned the intended behavior. Evaluating the changes to qualitative behaviors in the model requires a level of granularity beyond just the eval score.
With native support for OPSD and RMSD, AC2 gives researchers insight into the trace, the hint, teacher / student disagreement, and which tokens are responsible for the behavior divergence used for training.
The goal of our platform is to productize model customization but research today still requires taste, nuance, and most importantly looking at the data. Loss curves are nice, but there is no replacement for building intuition by looking at traces. Every week, improvements to AC2 makes this easier and easier.
Where OPSD is useful
OPSD is not a replacement for RL, it is a complementary tool with a different set of strengths and tradeoffs.
RL works well for replayable tasks with verifiable graders and enough variance to provide learning signal. SFT can teach a model from good demonstrations, but the supervision only covers the states the demonstration passes through. The model’s own samples leave those states within a few tokens, so the learning stops transferring.
OPSD supervises the model’s own samples, so the feedback lands on states it actually visits. The student and teacher models are the same, the only difference being that the teacher has access to privileged information such as a hint, a correction, a piece of human feedback, or some other prior about what should have happened. The student then learns from how that extra information changes the teacher's preferences over the response it already produced.
Concretely, the student sees a prompt x and produces a response y. The teacher sees x′, the same prompt with the feedback added, and we ask what it would have preferred at each token of y. The loss is the reverse KL between those two distributions, optionally restricted to the positions the judge selects⌝, which come from the same weights and differ only in one extra piece of context.
A simple way to think about it is: instead of giving the model a reference answer, give it useful feedback on its own work.
OPSD is most useful when there is a clear prior on the failure mode. Maybe a model likes to include an undesirable quirk in the code that it generates. Maybe it ignores a company-specific convention. Maybe it makes the same bad tool call across a large set of production traces. You can find those examples, attach targeted feedback, and train on the specific behavior you want to change.
There are two settings where this is especially compelling:
- You cannot replay the task. Production interactions are often expensive, stateful, or impossible to rewind. Group-based RL methods want multiple attempts. Self-distillation can learn from the rollout you already have.
- The feedback is qualitative. A lawyer’s redline, a designer’s critique, or a user’s explanation of why an answer felt wrong can be very useful without mapping cleanly to a scalar reward.
However, while existing self-distillation use cases are already exciting, there is still more to be proven out. Climbing the ladder from known failure modes to broader capability improvements is still an open research problem we’re actively pursuing.
Tooling to build intuition
Every OPSD project has the same workflow: find the failing behavior, construct a hint, place it near the mistake, and check that the update landed on the behavior and nothing else. The loop never changes, so we built it into AC2.
Verifying that the update landed on the desired behavior requires visibility into the data: a run can look healthy while learning the wrong thing entirely. We have seen a target behavior improve within ten steps, but the gains did not translate because the model memorized a narrow dataset or picked up an unrelated teacher behavior. The KL loss curve doesn’t disambiguate the good learnings from the bad learnings, so we made sure that AC2 shows the token-level effect of every hint next to the trace it came from.
Below is that view for a rollout of Qwen3-4B on an airline task. The user asks to cancel all of their upcoming flights and offers a user ID. The model calls get_reservation_details without the reservation ID anyway, and the task scores zero. We then inject the tool schema hint just before the mistake and see which tokens the teacher prefers over the student.
Green tokens are positions where the hinted teacher prefers the token more than the student did, red where the teacher prefers it less. Most of these differences are noise or style, for example the teacher prefers “I” over “My” in the third sentence, and training on them teaches nothing about the tool call. The relevance mask narrows the update to the differences that matter. Among the judge selected positions the teacher prefers “_user” over “_reservation” and “ask” instead of “call”, which is the failure the hint was written for.
The view ties the update back to the full trace. It shows the turn being trained, the hint given to the teacher, the teacher-student difference at each token, and the positions the mask kept. If a hint meant to fix a tool call is spending its update on wording, that is visible mid-run. In AC2, a researcher can start from a regression, open the trace, inspect the update, adjust the hint, and launch the run from the same place.
Three ways to run it
The platform also centralizes the launch paths. Our first OPSD runs each looked like different infrastructure: one replayed stored transcripts, one re-sampled a single turn, and one drove a full environment. However, the core requirement for OPSD is a response from the student and a teacher prompt carrying the feedback. We built that common ground into AC2, while retaining three distinct entry points:
- Offline, from production traces. Replay a stored transcript. There is no environment and no inference engine. The response was generated once in production, and training reads it back. By that point a trace is just a list of messages, so it does not matter what harness produced it. This is the mode for tasks that cannot be replayed.
- One step, re-sampling a single turn. The student re-samples the turn where the feedback belongs, and learns from how the hint shifts the teacher’s preferences over what it just produced. Fresh on-policy data without replaying the whole task and reconstructing the environment and tool surface.
- Online, in a replayable environment. Run the full task, grade it, then distill. The hint can be fixed per task, or written at rollout time by a judge that reads the graded trajectory and decides what the model should have done differently. This is the setting where RL is also available, and OPSD can run beside it or in place of it. It uses the customer’s exact agent harness, which AC2 can drive as a black box for both distillation and RL. More on that soon.
We generally focus initially on the first to show lift directly from production traces, then move towards the second and third to compound the gains into a continual learning pipeline.
Above is an example of the offline path end to end. These traces were replayed from stored transcripts for training, and each trace carries a full token level view.
What makes a good hint?
The quality of the hints determines the quality of the training run, and the detailed visualizers on the platform let us iterate on these hints quickly. The cheapest test is to simply give the model the task with and without the feedback in context. If the answer does not improve, the hint does not have useful training signal. Further iteration on hints requires the granularity of information provided by the platform to analyze what kind of signal each hint provides. Below, we share some of our initial learnings.
First, the content of the hint matters more than the exact wording. In our experiments, rephrasing the same hint does not meaningfully change the update. This is a very desirable property of the training algorithm, since it is robust to small changes in the hinting methodology.
This does not mean all hints are equivalent. A vague hint and a targeted priors-based hint behave differently. “Do better” is less useful than “You still have two attempts left; call the checker before finalizing”.
In addition, we have found turn-level feedback to be much more useful than attaching one broad comment (even the correct answer) to an entire trajectory. The closer the hint is to the decision you want to change, the easier it is to understand what the model is learning.
Above, we measure the KL divergence between the student and teacher at various token positions after the hint. We find that the disagreement is highest immediately following the hint, where the teacher provides the strongest signal based on the hint. This signal decays as you move further away from the location of the hint.
Leveraging self-distillation
New techniques only become broadly useful after they fit into the rest of the research workflow; we used AC2 to make it fast, easy and repeatable. Our forward deployed researchers are able to innovate with our customers and then bring learnings and techniques back into the platform. The next researcher should start with better tools than the last one had.
OPSD and RMSD are a good example of that motion. The original research gave us a promising way to teach specific behaviors from rich feedback. Applying it to real agent tasks exposed the harder problems around hints, data, and observability. We synthesized the common abstractions across different engagements to build a foundation on the platform for repeatable results. Those problems shaped the products we are using now and releasing soon.
The opportunity
Production systems create a huge amount of useful feedback: retries, corrected tool calls, accepted edits, user reactions, human comments, and moments where a person knew what the model should have done. Almost none of that feedback makes it to model weights.
Turning this feedback into model improvement requires finding the failure, writing a hint that carries real information, placing it where the decision went wrong, and stopping before a narrow correction becomes overfitting. Currently, no single algorithm solves that loop: learning comes from combining the algorithms with the tooling to see what they are actually learning and guide research judgement.
Up until now, user feedback usually affects the context, not the model. With OPSD and related techniques, we’re one step closer to building online continual learning systems that just get better the more you use them.