Scaling On-Policy Self-Distillation with Hint Auto-Optimization

Introduction

In On-Policy Self-Distillation (OPSD), a student’s on-policy traces receive dense supervision from a teacher, often the same base model given privileged information: a hint. The training signal therefore depends heavily on the hints provided to the teacher.

There are two challenges to constructing hints: 1) manually iterating on hints over training runs is impractical and 2) trying to attribute improvements in a training run to individual hints is difficult. We ask whether we can instead automatically optimize hints using cheap proxies for their eventual training value.

We build an agentic optimizer that proposes hints, evaluates them based on teacher rollouts (auto-regressively decoding the hinted teacher model), and revises them using grader feedback.

On an internal enterprise task for tool-call failures where our base model has 47% accuracy, manually optimized hints improve the trained model’s pass@1 from 71.68% (researcher tuned hints) to 83.23% (automated hints).

We share this algorithm as an inexpensive way to scale OPSD performance. This is an early step in closing Applied Compute’s autonomous loop between identifying failures in production and fixing them.

Methodology

Our procedure has two stages: optimize hints with a frozen teacher, then train the student using those hints as teacher-only context.

Each training sample consists of a failed student trace, a target step, a replay-able serving model, and an executable grader. Let $c=x_{<t}$ denote the interaction history immediately before the failed step, including the tool information and environment state needed to resume execution.

1. Propose a hint.

The optimizer examines the failure and grader feedback, then writes a hint describing how the teacher should approach the next step.

2. Test the hint.

We restore the original prefix, append the hint to the final tool response as privileged teacher context, and generate a fresh teacher continuation. Each candidate starts from the same prefix; it does not continue from the previous candidate’s attempt.

3. Grade and revise.

The executable grader scores the continuation. The optimizer sees the result and feedback and revises its hint for up to 15 rounds. In our internal enterprise tool-call recovery setting, passing also requires correcting the intended tool action; abandoning or substituting that action does not count as recovery.

The default objective is expected teacher re-rollout reward:

Each candidate initially receives one seeded rollout. Promising candidates receive two additional rollouts on fresh seeds. We score each hint by its mean observed reward and retain the best candidate tested:

This concentrates inference on promising candidates. For binary rewards, three successful rollouts serve as a stopping heuristic. When no candidate achieves full credit, we retain the best hint and its score; whether such examples enter training is a separate filtering choice.

Anti-cheating instructions. To avoid cheating, we prompt the optimizer to inspect the recorded failure to diagnose it, but that its hint must be forward-looking and grounded in information available in the original prefix. It must not reveal a hindsight answer or refer to the grader or recorded failure.

4. Train with the selected hints.

During OPSD, the student receives the original context prefix up to the failed step, while the teacher additionally receives the hint and provides token-level supervision on student rollouts.

By default, we optimize hints before training and keep them fixed. In the refreshed variant, we periodically collect hint-free student rollouts, identify tasks with low pass rates, and re-optimize their hints using the new failures.

Experiments

We distinguish hint validation reward, which measures the frozen teacher’s behavior with a hint, from held-out hint-free pass@1, which measures the trained student’s performance without one.

Setup Details

We use a 800 sample dataset of enterprise tool call failures, as well as a dataset of 600 enterprise chat response failures. Our experiments are on Qwen3-4B and Qwen3.8-27B, respectively, on 200 task evals ran on 4 random seeds. After choosing hints for every train task, we use OPSD with a fixed best chosen learning rate and batch size that we found over several ablations for each model. We show the best eval result across training steps. Our graders for each failure mode are pre-defined as binary classifiers on whether or not this assistant step had any specified issue or not.

Human Optimized Hints

For a more realistic baseline, we used an independently optimized set of hints for this task that a researcher at Applied Compute had generated after looking through failure modes and tuning prompts for the tool failure tasks.

Teacher Rollout Reward Proxies Transfer to Training Performance

We compare teacher re-rollout grader reward with alternatives:

1. Response-edit correctness, which grades a self-reflected edit of the visible failed response given the hint (rather than a fresh hinted rollout) to enable parity of the hint with its targeted failure.
2. Sparse next-token-probability movement, which displays the new most-likely token in the teachers logits on the student trace on the top 10% highest KL loss token positions to the agent. This is how human researchers tend to observe the quality of OPSD hints and what the training signal directly corresponds to.

Fresh re-rollout optimization reaches 79.85% trained pass@1, compared with 71.68% for researcher-optimized hints and 50.30% for the next-token-probability proxy. Since this proxy is what our human researchers previously relied on to optimize hints, its clear that current frontier models are far poorer at understanding this. We hypothesize for why this direct metric is unstable.

• Over training, the student converges more contiguous logits to be on-policy of the teacher. This means the 1-token updates become stale. We show these dynamics in the Appendix.
• There are many times the student trace is "off-policy" which initially causes many high entropy predictions for the teacher model, so the discrete next-token probability shifts initially can be highly variant.

Additionally, response editing reaches 79.08%, which is an insignificant difference for a more complex system, so we find the teacher rollouts are our best metric.

2. Online Hint Optimization Improves Performance

Hints optimized for the initial student need not remain the most useful as it learns.

We thus try re-optimizing hints for tasks with low sampled successes rates (<0.5). This is mildly successful, raising held-out pass@1 to 83.23%, compared with 79.85% for fixed optimized hints though requires on avg. 2.3 re-rollouts with the student (rather than the frozen base model) given the hint during training, which costs additional training time.

Rather than optimizing individual hints we can optimize a singular hinter prompt. This changes the optimizer updates as follows:

1. Use the current hinter prompt to generate hints for each of N failed steps.
2. Use the grader to evaluate the N graded re-rollouts and inject the re-rollouts into the the agent
3. Reflect and optimize the hinter prompt, continuing to next batch of N failed steps.

We compare two different prompts, one probing for minimal changes to the hint generator after reflection, and one without this additional input.

Unfortunately, a singular hinter prompt tends to be far too constrained to give good feedback across widely diverse failure modes, these results may change with future, more capable models. While minimizing step size does improve performance over the dataset by ensuring reasonable step-size in prompt updates, it still underperforms per-case optimization.

3. Hint validation grades can guide training-data filtering

We compare training on all optimized hints against training only on tasks whose hints pass validation. This tests whether, when enough training tasks are available, excluding unsuccessful teacher interventions is more useful than retaining the additional task coverage.

Despite using fewer training tasks, the filtered method outperforms unfiltered. As we spend compute on evaluating optimized hints anyways, this is a free addition to performance that could be used in training runs with or without hint optimization.

4. Favoring minimal impact reduces downstream performance

A large concern of OPSD is inducing bias in the model with a poor hint. We ask whether reducing this bias should be a core metric of optimization, rather than just an unchecked constraint given to the optimizer.

Using an LLM-as-a-judge, fully natural response gates favor corrections that preserve the task, plan, and non-target behavior; embedding minimality favors corrected responses close to the original response in embedding space using `openai-embedding-large`.

We additionally try simple computational metrics, such as chasing hints that make the teacher’s logits minimally shifted from the students decoded logits.

Correctness-only selection transfers best in the reported comparisons. Additional naturally and minimality criteria reduce the final student’s held-out performance. Small or plausible-looking changes are therefore not necessarily the most useful training targets; these criteria can reject corrections that would otherwise help the student with higher pass rates.

5. Search Topology Comparison

We compare final mean re-rollout grade on pre-defined search regimes against an agent with a max-budget tool. This measures how efficiently each regime discovers useful teacher interventions, separately from downstream training performance. We use a separate random set of particularly hard 132 tool-fix failures on an Internal Enterprise agent.

We describe the details and empirical results of several simple optimizations to search strategies.

We find that confidence-based search strongly outperforms all other methods as it wastes the least number of rollouts in optimizing. While the exact setup of 1 run with 2 validation brings out strong results in this domain, these hyperparameters may depend heavily based on the entropy of the model as well as difficulty of task.

Additionally, we compare which model is required to back the search between GPT-5.6 Luna, Terra, and Sol. Despite being the cheapest and least capable overall, Luna gets the strongest performance on this hard-subset of tasks, requiring the least number of hints tested along with the lowest cost per hint. This may be correlated with Luna being the most similar in capacity to Qwen3-4b.

Why This Matters

What might automated hint optimization look like in production? If we see an enterprise agent fails to make an action, makes an incorrect tool call, or mishandles certain kinds of requests, then that interaction becomes a training opportunity. We can replay the failed step, search for guidance that helps the teacher recover, and distill the corrected behavior into the student. As its failures change, the system can refresh its supervision, reducing the manual work required to turn production feedback into model improvements.

Applied Compute works at the intersection of reinforcement learning and enterprise agents, where useful post-training depends on understanding what went wrong and providing a precise learning signal for how to fix it. Automating that signal requires reproducible environments, graders that capture the intended behavior, and infrastructure that can search over corrections at scale. Hint optimization connects these pieces and takes a step toward autonomous post-training.

Future Work

There are additional fruitful directions which we defer to future work.

• Direct gradient based prompt optimization such as Greedy Coordinate Gradients⌝ may improve performance on tasks where hint optimizers get no success.
• Optimizing a more loosely defend hinter python script rather than individual hints or a hinter prompt. We find that current frontier models are not capable of this and cannot out-perform individual hints.

Appendix A: Teacher Depth During OPSD

We analyze teacher depth over training to measure how much of a teacher continuation is locally compatible with the student. For teacher continuation $y^T_{1:L}$ from prefix $c$, define student surprisal as

Thus, $D_\tau$ counts consecutive teacher tokens from the start of the continuation before the first high-surprisal token. During training, its distribution lightly shifts right: longer teacher continuations become compatible with the student’s next-token predictions.

Surprisingly, while the models depth increases in early training steps, over time it does not completely converge. We hypothesize this is due to natural constraints baked into pre-training and mid-training of the model which constrain hint-following to a significant degree. However, the early gains do explain that the teacher rollouts may be a strong proxy.

Appendix B: Degradation Checks

Below we show that there is be some small degradation on a disjoint task (IFBench), but this is not statistically significant across methods. Specifically, we find no difference in observed degradation between fully natural gated and purely grader correctness optimized hints.