Enriching Domain-Specific Post-Training with Multi-Teacher On-Policy Distillation

Introduction

Many of the best open-source frontier models, including DeepSeek, Kimi, Nemotron, and GLM, use Multi-Teacher On-Policy Distillation (MOPD) to combine the capabilities of specialists trained extensively in different domains into a single model. At Applied Compute, we were interested in ways we could leverage this approach for specializing models on specific tasks.

We introduce Spork, a simple post-training recipe which combines GRPO and MOPD to bring impressive gains into smaller-scale task-specific training. On DAPO-MATH and multi-step chess puzzles, Spork outperforms our strongest GRPO baselines, improving math pass@4 by 20% and chess pass@1 by 80%. Performing Spork with a second round brings the chess score to a nearly 100% improvement over GRPO.

Spork uses a short, full-dataset RL warmup to establish a common foundation. Afterwards, separate teachers specialize on disjoint partitions of the training set. On-policy distillation then consolidates those teachers into a single student initialized from the original base model.

Like chess puzzles, many of the enterprise tasks that we train on at Applied Compute are less familiar to existing models, and thus difficult to generate positive RL samples for. Spork serves as a new post-training paradigm to unlock these capabilities.

Background

Multi-Teacher On-Policy Distillation

Below we describe the classic MOPD algorithm:

Train specialists. Start all $N$ experts from checkpoint $\pi_0$ . For each $i \in \{1, 2, \ldots, N\}$ , perform supervised fine-tuning with expert traces in domain dataset $D_i$ , and finish with RL (typically GRPO or PPO) on environments from $D_i$ to produce checkpoint $\pi_i$ .
Consolidate with MOPD. Initialize the student $\pi_\theta$ from the shared checkpoint and freeze the experts. Sample a task $x\in D_i$ , roll out a complete trajectory $\tau$ with the current student, and use expert $\pi_i$ to supervise the student at student-generated history $\tau_{<t}$ .
(Optional) Repeat. Repeat using the latest consolidated student as the shared initialization for both the specialists and the next student.

\mathcal{L}_{\mathrm{MOPD}}(\theta) = \mathbb{E}_{\substack{ i \sim w,\; x \sim D_i,\; \tau \sim \pi_\theta(\cdot \mid x) }} \left[ \frac{1}{T_\tau} \sum_{t=1}^{T_\tau} D_{\mathrm{KL}}\!\left( \pi_\theta(\cdot \mid x, \tau_{<t}) \,\Vert\, \pi_i(\cdot \mid x, \tau_{<t}) \right) \right]

Here, $T_\tau$ counts student-generated tokens, and $w$ controls partition sampling. Environment-generated tokens are excluded from the loss. For efficiency, our implementation uses top- $k$ logits for token-level supervision, not full-vocabulary logits.

Challenges with Specialization

Part of the motivation for pursuing MOPD is that when specializing models with GRPO, low pass-rate tasks provide little useful learning signal. On the other hand, methods such as supervised fine-tuning can cause catastrophic forgetting, degrading existing capabilities while teaching new ones (1⌝, 2⌝). MOPD provides a route to getting the best of both worlds: creating teachers with dense learning signal, while avoiding off-policy training.

Spork: A Practicle MOPD Recipe for Individual Domains

We present the Spork recipe.

Partition the data. Starting with a task dataset, use predefined, disjoint, and exhaustive definitions of $N$ partitions. Then, use an LLM judge to partition RL environments $D$ into the partition datasets $D_1,\ldots,D_N$ .
Train a warmup RL checkpoint. Starting from $\pi_0$ , run online GRPO on $D$ to obtain $\pi_{\mathrm{RL}}$ . We use a small number of steps with a high learning rate to efficiently climb performance across the entire task. This stage allows all specialists to inherit strong base capabilities, especially where partitions share similar skillsets.
Specialize on each partition. Initialize each expert $\pi_i$ from $\pi_{\mathrm{RL}}$ , then continue online GRPO on its assigned subset $D_i$ .
Consolidate with MOPD. Freeze the specialists and initialize the student $\pi_\theta$ from $\pi_0$ . Sample tasks from $D$ and generate online trajectories using the student. Compute $\mathcal{L}_{\mathrm{MOPD}}(\theta)$
(Optional) Repeat. Repeat steps 3–4 for additional rounds, using the latest consolidated student as the shared initialization for both the specialists and the next student.

The resulting pipeline is:

\pi_0 \xrightarrow{\text{few-step GRPO on }D} \pi_{\mathrm{RL}} \xrightarrow{\text{specialist RL on }D_i} \{\pi_i\}_{i=1}^{N} \xrightarrow{\text{MOPD into }\pi_0} \pi_{\mathrm{Spork}}.

Methodology

RL Algorithms

We sweep over several GRPO hyperparameters on our datasets and find that AdamW with a learning rate of $10^{-6}$ and batches of 64 prompts with 16 rollouts per prompt achieves the strongest validation performance. We train for up to 4 epochs and evaluate checkpoints every 10 steps.

For the warmup stage in Spork, we also use GRPO with batches of 8 prompts with 8 rollouts per prompt for just 40 training steps.

Datasets

We use two tasks:

Lichess puzzles⌝ with 8k training samples with Qwen3.5-9B
- These are several-step chess puzzles with one optimal move to be submitted at every step with three allowed attempts. The final reward is based on how many of the optimal moves the agent solves.
DAPO-MATH using 2k training samples with Qwen3-4B

For chess, we have a held-out validation and eval set of 400 tasks each, evaluated with 4 random seeds. For math, we have a held-out validation and eval set of 256 tasks each, also evaluated with 4 random seeds.

We chose Lichess as a task because current frontier models have had little specialized mid-training or post-training on chess. This mirrors a lot of Applied Compute’s customer-specific tasks.

Partitions

For each task, we defined N=4 partitions. In math these were algebra/analysis, geometry/trigonometry, number theory, and combinatorics/probability. In chess, we chose king attack/mate, forcing tactics, line/material tactics, and quiet conversion.

We then used GPT-5.6-Sol to classify the tasks between categories and form our specialist subsets. More details are in the Appendix.⌝

Results

For each of the following methods, we select the best validation checkpoint and evaluate it on the eval set:

GRPO
GRPO-only MOPD, which is Spork without a GRPO warmup stage.
Spork

We find strong improvements on task success rates across models and datasets.

Spork raises the performance ceiling, reaching 80% higher pass@1 and 52% higher mean reward than the best GRPO checkpoint.

On DAPO, we see a +20% improvement in Pass@4 over our best GRPO checkpoint.

We confirm that without warmup, the model largely underperforms GRPO, and that our relatively inexpensive warmup substantially improves performance. For a clear illustration of the stages of Spork, we show the validation scores (not evaluation) over training samples seen.

The GRPO plot shown has the highest validation score over a hyperparameter sweep of learning rate, batch size, and group size. The warmup stage appears to climb far faster but has a much lower ceiling, as it has a lower batch size and higher learning rate. We show its continuation in the Appendix⌝.

Outlearning the Teachers

To understand this strong gain, we investigate the teacher performance on each respective partition. Notably, the base-initialized teachers, while training only on their partition, performed worse than the GRPO baseline over all the partitions.

One possible explanation is that the complementarity between partitions bolsters each partition’s performance. Thus, the warmed-up teachers which have learned capabilities across partitions strongly outperform the base-initialized teachers, and they are able to increase the performance of the student further. For instance, to specialize on number theory, it’s helpful to improve at problem solving, a skill common to all partitions.

Most interestingly, the Spork checkpoint is even sometimes able to beat its own teachers across categories.

We also saw the Spork model have gains over its own teachers on DAPO. We flag that this is in contrast with the results for general MOPD across frontier post-training runs, where the final model is worse than each individual teacher. This reinforces the leveraged complementarity of the teachers; for example, in chess, mating attacks may use strategies similar to those used in forcing tactics.

Ablations

Below we ablate over several algorithmic choices, in order to maximize our understanding of these performance gains.

How Far Can We Partition?

On the same fixed training set, we compare Spork’s performance when partitioning the training set into $N \in \{1, 2, 4, 8\}$ partitions as defined in the Appendix.⌝

At $N=1$ , we observe no improvements over GRPO. This shows that the expert training and on-policy distillation component of learning did not explain the improvements of Spork over GRPO.

On chess, $N=2$ specialists work dramatically better than $N=1$ , and nearly as well as $N=4$ . This implies that a large portion of the gains in Spork are attributed to specialization, and that we may not require many specialists to achieve strong gains on a task.

We additionally observe that it isn’t clearly possible to continue scaling final performance by increasing $N$ . For example, comparing $N=2$ with $N=8$ , we observe that the teacher model aggregate is much higher in $N=8$ , but the eventual learned student model performs similarly. This suggests that consolidation becomes a limiting factor.

A reasonable explanation for this phenomenon is that the teachers have become too distinct. This forces the student to learn with constant distribution shift, which may cause each learning step to be unstable.

We highlight that these results likely vary from task to task and model to model, and we defer to future work to explore compute-efficient algorithms for finding optimal values of $N$ .

Does the choice of partitions matter?

We compare our $N=4$ semantic partitions with a random partition of the data into $N$ subsets, to measure the value of semantic partitioning.

Random partitioning also significantly improves performance over GRPO, which implies that valuable specialization can still naturally occur in identically sampled subsets, and each specialist becomes stronger learning from its own, smaller, more saturable train which gets distilled back into the student model.

To ensure this was truly specialization rather than expected variance between trained specialists, we additionally compared simply using different random seeds between each specialist training run on the whole dataset. The only difference is that there is no guarantee that data is not shared between specialists. As shown, this is still a 51% improvement over GRPO, but is a 3% reduction from randomly partitioned Spork. This implies that partitioning is a clear improvement to performance, and natural variance in the specialists is another large contributor.

Both methods far underperform strategic partitioning, implying that models are better able to specialize and provide cohesive learning signals when the partitions have larger distributional differences.

How can we leverage repititions?

In frontier post-training runs, MOPD is often applied repeatedly. We compare applying Spork repeatedly with and without applying GRPO warmup atop the best Spork validation checkpoint.

We observe strong gains in performance without warmup, and limited gains with a warmup. This implies that improving general capabilities is no longer helpful in improving specialist performance, and may cause unnecessary entropy reduction.

The diminished gains as compared with the first round can be further explained by more marginal teacher performance improvements. While the distilled final model roughly matched or improved its teachers across partitions, the teachers on certain partitions did not improve over the initial Spork checkpoint.

A more dynamic approach would be to continue repeating the GRPO specialization and MOPD steps on only the partitions that are still unsaturated; we defer this to future work. We expect that on more difficult tasks with larger models, several partitions can be repeated several times. Subsequent rounds of training could re-balance partitions such that each partition is equally unsaturated.

GPU Optimization

In the final MOPD step, keeping every teacher resident on the GPU at once is wasteful. On-policy rollout time per step stays constant regardless of $N$ (the number of teachers), while each teacher only works on its own partition's samples. To exploit this, we split each training step into per-expert sub-steps and stream teachers through a single GPU slot.

We assume no staleness in training. This method can be done faster by allowing for staleness, by simply always continuing the rollout engine on the next batch without waiting for the optimizer step.

Our solution is to repeatedly load and unload the teacher models, by using a sufficiently large batch size and ordering the rollouts submitted to the rollout engine by specialist group.

Per-expert sub-steps:

Each step generates $B = \sum_{k=1}^{N} B_k$ rollouts, grouped by partitions and generated one group after another. For each group, in order:

Keep teacher $k$ loaded as early as possible for its entire specialist group.
Score top- $m$ logits on completed trajectories in $B_k$ in teacher $k$ ’s microbatches.
Run student forwards and backwards on ready microbatches, accumulating gradients, computing the top- $m$ approximation of the reverse KL, $D_{\mathrm{KL}}\!\left(\pi_\theta(\cdot\mid h_t)\,\Vert\,\pi_k(\cdot\mid h_t)\right)$ .
Switch to teacher $i+1$ once teacher $i$ ’s scoring is complete and its outputs are safely retained.
Apply one optimizer update after the complete global batch.

We show how to choose hyperparameters to optimize utilization in the Appendix⌝.

Conclusion

These experiments show that partitioning, specializing, and distilling, can produce state-of-the-art individual task performance, outperforming GRPO baselines and the base MOPD recipe. We show that these gains can be achieved with minimal additional compute costs and strong GPU utilization.

We believe these results may be most applicable to tasks with minimal prior mid-training and post-training, such as multi-step chess puzzles and many of the enterprise tasks we train on at Applied Compute.

Future work will test how these findings generalize across tasks and models, develop compute-efficient methods for choosing the number of specialists, and explore adaptive training that continues specializing partitions with remaining headroom while leaving saturated ones unchanged. We also hypothesize that we can leverage the specialists to better distill into even weaker or smaller base models.

Appendix

Compute Bounds

A. Sizing a synchronous MOPD step

Setup

We model three independent GPU pools: a rollout pool that generates trajectories with the current student, a training pool that accumulates student gradients, and a teacher pool with space for one resident specialist. The remaining specialists are stored in pinned host memory.

We assume balanced specialist groups, approximately equal microbatch costs, and sufficient buffering. We analyze the saturated-rollout regime, where trajectories arrive fast enough that teacher scoring does not wait for generation. While this is unrealistic, optimizing for this regime ensures that in all regimes there is maximal overlap with rollouts, teacher scoring, and training.

For simplicity, teacher and student microbatches are aligned, and each student forward–backward begins once its teacher targets are available.

Step-time model

The first specialist group requires a teacher load followed by a pipeline of $m$ scoring and training microbatches:

s + t + u + (m - 1)\max(t,u).

Each subsequent group contributes the larger of teacher loading-plus-scoring time and student training time. Therefore,

\begin{aligned} T_{\mathrm{batch}} &= s + t + u + (m - 1)\max(t,u) \\ &\quad + (N - 1)\max(s + mt,\; mu). \end{aligned}

Final gradient synchronization, optimizer execution, and rollout-weight synchronization are a constant addition, and they can overlap with the loading of the first teacher in the next step.

Hiding teacher switches

When $u > t$ , the expression simplifies to

\begin{aligned} T_{\mathrm{batch}} &= \underbrace{Nmu}_{\text{student computation}} + \underbrace{s+t}_{\text{startup}} \\ &\quad + \underbrace{(N-1)[s-m(u-t)]_+}_{\text{exposed switching overhead}}, \end{aligned}

where $[x]_+ = \max(x,0)$ . Each microbatch provides $u-t$ seconds of spare teacher capacity. Recurring switching overhead is hidden when

m(u-t) \ge s.

The corresponding minimum group and global batch sizes are

\begin{aligned} m_{\min} &= \max\!\left(1,\left\lceil\frac{s}{u-t}\right\rceil\right), \\ B_{\min} &= Nb\,m_{\min}. \end{aligned}

Above this threshold, $T_{\mathrm{batch}} = Nmu+s+t$ : student computation determines the sustained processing rate. If $u \le t$ and $s > 0$ , increasing the group size amortizes switching but cannot eliminate the teacher bottleneck.

B. Total compute

With $L$ tokens per sample and $P$ active parameters per token, the per-step FLOPs are

\begin{aligned} F_{\text{GRPO}}(B) &= 8PLB \quad (2PL \text{ generation} + 6PL \text{ training}), \\ F_{\text{MOPD}}(B) &= 10PLB \quad (\text{plus } 2PL \text{ teacher scoring}), \end{aligned}

ignoring attention FLOPs. Each distillation sample is scored by exactly one routed teacher, so a distillation step costs 1.25× a GRPO step for any $N$ .

The main cost comes from the amount of steps given to each specialist. We find that in certain cases, the specialists plateau quite early, and in others they are able to train for extended periods of time. This adds a lot of variance in total cost.

The number of experts enters cost only through the specialist runs:

\text{Cost}(N) = W F_{\text{GRPO}}(B_W) + \sum_{k=1}^{N} S_k F_{\text{GRPO}}(B_S) + M F_{\text{MOPD}}(B_M).

† Round 2 starts from the round-1 N=4 model, so its warmup is all of round 1. These are the costs of our runs, not an efficiency comparison. GRPO configurations or variants that converge faster may cost less.

Lichess puzzle partitions

For N=4, we define four training partitions. We acknowledge that these labels may have expected error, though these are not of large importance to our results, as the partitions are arbitrarily chosen anyways.

N=2 Ablation:

We merge the N=4 partitions.

tactical = King attack /Mate + Forcing tactics (3,872)
positional = Line / material tactics + Quiet conversion (4,128)

N=8 Ablation:

DAPO-MATH partitions

Continued GRPO Run from Warmup

Below we include the rest of the run from the GRPO training run used for warmup. As we can see, it reached a much lower validation (and evaluation) ceiling than the reported GRPO baseline.

References

Ma, W., et al. (2026). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training⌝.
Lu, K., and Thinking Machines Lab. (2025). On-Policy Distillation⌝.
Xiaomi LLM-Core Team. (2026). MiMo-V2-Flash Technical Report⌝.
DeepSeek-AI. (2026). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence⌝.
NVIDIA. (2026). Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning⌝.
Shenfeld, I., Pari, J., & Agrawal, P. (2025). RL’s Razor: Why Online Reinforcement Learning Forgets Less⌝.

Cite this work

Get our latest research

Product news, customer stories, and new posts, straight to your inbox.