Multi-Rollout On-Policy Distillation via Peer Successes and Failures

Chen Henry Wu; Gaurav Mittal; Haixin Wang; Matt Fredrikson; Ruowang Zhang; Weichen Yu; Xiaomin Li; Xiaoze Liu; Yinyi Luo; Yizhou Zhao

arxiv: 2605.12652 · v2 · pith:43MK434Fnew · submitted 2026-05-12 · 💻 cs.LG · cs.AI

Multi-Rollout On-Policy Distillation via Peer Successes and Failures

Weichen Yu , Xiaomin Li , Yizhou Zhao , Xiaoze Liu , Ruowang Zhang , Haixin Wang , Yinyi Luo , Chen Henry Wu

show 3 more authors

Gaurav Mittal Matt Fredrikson Yu Hu

This is my paper

Pith reviewed 2026-06-30 21:59 UTC · model grok-4.3

classification 💻 cs.LG cs.AI

keywords on-policy distillationmulti-rolloutpeer conditioningsuccess-failure conditioningverifier rewardslanguage model post-trainingLLM reasoning

0 comments

The pith

Conditioning teacher signals on both successful and failed peer rollouts improves on-policy distillation for language models.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard on-policy distillation trains language models on student-generated trajectories but treats each rollout in isolation, missing opportunities to learn from multiple attempts on the same prompt. The paper proposes Multi-Rollout On-Policy Distillation, which conditions the teacher on the full local group of rollouts to supply positive evidence from successes and structured negative evidence from failures. Two variants are examined: positive peer imitation and contrastive success-failure conditioning. Experiments across competitive programming, mathematical reasoning, scientific question answering, and tool-use tasks show consistent gains over standard on-policy baselines. Teacher-signal analysis further indicates that mixed success-failure contexts produce scores that align more closely with verifier rewards, supporting the claim that the gains stem from more faithful instance-adaptive supervision.

Core claim

MOPD uses the student's local rollout group to build teacher signals by conditioning on both successful and failed peer attempts, so that valid reasoning patterns receive positive reinforcement while plausible mistakes receive explicit contrastive signals, yielding supervision that better matches verifier rewards than isolated-rollout distillation.

What carries the argument

Peer-conditioned distillation framework that constructs teacher signals via positive peer imitation and contrastive success-failure conditioning on the student's multi-rollout group.

If this is right

MOPD yields consistent performance gains over standard on-policy baselines on competitive programming, mathematical reasoning, scientific question answering, and tool-use benchmarks.
Mixed success-failure contexts produce teacher scores that align more closely with verifier rewards than single-rollout or success-only contexts.
Effective on-policy distillation should treat the student's multi-rollout trial-and-error behavior as a source of structured supervision rather than isolated samples.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The approach could be extended by varying the number of peer rollouts per prompt to test how many failures are needed for effective contrastive signals.
Similar peer conditioning might transfer to other post-training regimes that already sample multiple trajectories per prompt.
Instance-adaptive teacher signals derived from student peers could reduce reliance on a separate, stronger teacher model.

Load-bearing premise

Performance gains arise specifically because the local rollout group supplies more faithful instance-adaptive supervision rather than from unrelated factors such as extra compute or altered training dynamics.

What would settle it

An experiment applying the same multi-rollout setup but finding no improvement in either benchmark performance or alignment between teacher scores and verifier rewards would falsify the central claim.

Figures

Figures reproduced from arXiv: 2605.12652 by Chen Henry Wu, Gaurav Mittal, Haixin Wang, Matt Fredrikson, Ruowang Zhang, Weichen Yu, Xiaomin Li, Xiaoze Liu, Yinyi Luo, Yizhou Zhao, Yu Hu.

**Figure 1.** Figure 1: MOPD Illustration. To directly examine whether peer conditioning improves the self-teacher signal itself, we introduce an analysis of self-teacher signal quality. For each prompt, we fix a set of student-generated rollouts containing both successful and failed attempts, vary only the context shown to the self-teacher, and compare the self-teacher’s normalized logits or scores with ground-truth verifier r… view at source ↗

**Figure 2.** Figure 2: MOPD Pipeline. to the successes and failures observed in the other rollouts. This prevents the teacher from exploiting local, instance-specific evidence contained in the rollout group. 4 Multi-Rollout On-Policy Distillation We propose Multi-Rollout On-Policy Distillation (MOPD), a peer-conditioned distillation framework that exploits the local structure of multiple on-policy rollouts generated for the same… view at source ↗

**Figure 3.** Figure 3: Number of training data that have ever generated a correct answer in the N rollout during training. Case Study. During training, we save the generated rollouts and compare them on the same question across training steps to provide a case study. Additionally, after training for the same number of steps, we save checkpoints from both SDPO and MOPD, then sample from these checkpoints to evaluate whether each … view at source ↗

**Figure 4.** Figure 4: Self-teacher-signal quality across seven context conditions. Each panel reports an averaged prompt [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

**Figure 5.** Figure 5: Diversity Analysis. evidence sharpens decision boundaries that positive evidence alone leaves blurred. 4) Combining both types yields the best results: the “2 success + 1 failure” context achieves the highest score on 5 of the 6 ranking and discrimination metrics in the signal-quality analysis, with a competitive Brier score, and the highest LCB downstream mean@8 among the compact peer-context settings. 5)… view at source ↗

read the original abstract

Large language models are often post-trained with sparse verifier rewards, which indicate whether a sampled trajectory succeeds but provide limited guidance about where reasoning succeeds or fails. On-policy distillation (OPD) offers denser token-level supervision by training on student-generated trajectories, yet existing methods typically distill each rollout independently and ignore the other attempts sampled for the same prompt. We introduce Multi-Rollout On-Policy Distillation (MOPD), a peer-conditioned distillation framework that uses the student's local rollout group to construct more informative teacher signals. MOPD conditions the teacher on both successful and failed peer rollouts: successes provide positive evidence for valid reasoning patterns, while failures provide structured negative evidence about plausible mistakes to avoid. We study two peer-context constructions: positive peer imitation and contrastive success-failure conditioning. Experiments on competitive programming, mathematical reasoning, scientific question answering, and tool-use benchmarks show that MOPD consistently improves over standard on-policy baselines. Further teacher-signal analysis shows that mixed success-failure contexts better align teacher scores with verifier rewards, indicating that the gains arise from more faithful, instance-adaptive supervision. These results indicate that effective on-policy distillation should exploit the student's multi-rollout trial-and-error behavior rather than treating rollouts as isolated samples.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

MOPD adds peer conditioning from multi-rollout groups to on-policy distillation and reports gains on reasoning benchmarks, but the gains could trace to higher sample volume rather than the conditioning itself.

read the letter

The main point is that this paper extends on-policy distillation by conditioning the teacher on a local group of student rollouts for the same prompt, pulling positive signals from successes and negative signals from failures.

What is new is the explicit peer-context construction, with two variants: positive peer imitation and contrastive success-failure conditioning. The work runs experiments on competitive programming, mathematical reasoning, scientific question answering, and tool-use tasks, and it finds consistent improvements over the independent per-rollout baselines described in prior OPD work. The teacher-signal analysis is a useful addition; it shows that mixed success-failure contexts produce scores that line up better with verifier rewards.

The soft spot is the comparison to baselines. The abstract frames the baselines as distilling rollouts independently, but it does not state whether those baselines match the total number of rollouts used by MOPD. If the baseline draws one rollout per prompt while MOPD draws K, any uplift could come from extra data volume or reduced variance alone. The alignment analysis is post-hoc and does not directly test against a same-K independent-distillation control, so it leaves the confound open. If the full paper includes that control, the central claim strengthens; otherwise the evidence for the peer mechanism specifically is weaker than presented.

The paper is aimed at people working on LLM post-training and distillation for reasoning tasks. A reader who cares about denser supervision signals from student-generated data would find the method and the benchmark results worth looking at. It shows clear engagement with the literature on sparse verifier rewards and offers a concrete, testable extension.

I would send it to peer review, with the main request being clarification on total rollout counts in the baselines.

Referee Report

2 major / 0 minor

Summary. The paper introduces Multi-Rollout On-Policy Distillation (MOPD), which extends standard on-policy distillation by conditioning the teacher on both successful and failed peer rollouts within the student's local group for the same prompt. It defines two peer-context constructions (positive peer imitation and contrastive success-failure conditioning) to supply denser token-level signals that exploit trial-and-error behavior. Experiments across competitive programming, mathematical reasoning, scientific QA, and tool-use benchmarks report consistent gains over independent-rollout OPD baselines, supported by post-hoc analysis showing that mixed success-failure contexts align teacher scores more closely with verifier rewards.

Significance. If the reported gains are shown to arise specifically from the peer-conditioned signals rather than from increased rollout volume, the approach would provide a practical way to obtain more faithful instance-adaptive supervision from sparse verifiers, strengthening on-policy methods for reasoning and tool-use tasks.

major comments (2)

[Abstract] Abstract: the central claim that performance gains 'arise from more faithful, instance-adaptive supervision' is not yet supported, because the manuscript provides no same-K independent-distillation control; without it, any uplift could be explained by the simple fact that MOPD draws K>1 rollouts per prompt while the described baselines distill rollouts independently.
[Abstract] Abstract (teacher-signal analysis paragraph): the post-hoc observation that mixed success-failure contexts align better with verifier rewards does not rule out the sampling-volume confound, as it lacks a direct comparison against an independent multi-rollout baseline using the same total number of trajectories.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful reading and for identifying the sampling-volume confound. We agree that the current experimental design does not fully isolate the contribution of peer-conditioned signals from the use of K>1 rollouts per prompt. We will add the requested same-K independent multi-rollout baseline and revise the abstract and analysis sections accordingly.

read point-by-point responses

Referee: [Abstract] Abstract: the central claim that performance gains 'arise from more faithful, instance-adaptive supervision' is not yet supported, because the manuscript provides no same-K independent-distillation control; without it, any uplift could be explained by the simple fact that MOPD draws K>1 rollouts per prompt while the described baselines distill rollouts independently.

Authors: We acknowledge the point. The reported baselines distill each rollout independently (standard single-rollout OPD), whereas MOPD conditions on a group of K rollouts. To address the confound, we will introduce an additional control that performs K independent distillations per prompt (same total trajectories, no peer conditioning) and compare it directly to MOPD. This will allow us to quantify how much of the gain is attributable to the peer success/failure construction versus rollout volume. revision: yes
Referee: [Abstract] Abstract (teacher-signal analysis paragraph): the post-hoc observation that mixed success-failure contexts align better with verifier rewards does not rule out the sampling-volume confound, as it lacks a direct comparison against an independent multi-rollout baseline using the same total number of trajectories.

Authors: We agree that the current teacher-signal analysis cannot rule out the volume confound. In the revision we will rerun the alignment analysis on the new independent multi-rollout baseline (K independent trajectories) and report whether the improved alignment with verifier rewards is specific to the mixed success-failure peer contexts or appears under any multi-rollout regime. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical method proposal with independent experimental validation

full rationale

The paper introduces MOPD as an algorithmic extension to on-policy distillation that conditions on peer rollouts for positive imitation and contrastive failure signals. No equations, derivations, or predictions are presented that reduce by construction to fitted parameters, self-definitions, or self-citation chains. The central claims rest on benchmark experiments comparing against standard on-policy baselines, with post-hoc analysis of teacher signals; these are falsifiable empirical results rather than tautological reductions. No load-bearing self-citations, uniqueness theorems, or ansatzes imported from prior author work are invoked in the provided text. This is a standard case of a self-contained empirical contribution.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are identifiable. The framework builds on standard assumptions in reinforcement learning and distillation without introducing new postulated entities.

pith-pipeline@v0.9.1-grok · 5789 in / 1203 out tokens · 35959 ms · 2026-06-30T21:59:29.822040+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages · 1 internal anchor

[1]

arXiv preprint arXiv:2402.00782 (2024) 3, 8

URL https://openreview.net/forum?id=bx24KpJ4Eb. Survey Certification, Featured Certification. Alex J Chan, Hao Sun, Samuel Holt, and Mihaela Van Der Schaar. Dense reward for free in reinforcement learning from human feedback.arXiv preprint arXiv:2402.00782, 2024. 11 Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning ...

work page arXiv 2024
[2]

A Survey on Knowledge Distillation of Large Language Models

URLhttps://arxiv.org/abs/2402.13116. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to Reason under Off-Policy Guidance.arXiv preprint arXiv:2504.14945, 2025. URL https://arxiv.org/abs/ 2504.14945. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, ...

work page internal anchor Pith review Pith/arXiv arXiv 2025

[1] [1]

arXiv preprint arXiv:2402.00782 (2024) 3, 8

URL https://openreview.net/forum?id=bx24KpJ4Eb. Survey Certification, Featured Certification. Alex J Chan, Hao Sun, Samuel Holt, and Mihaela Van Der Schaar. Dense reward for free in reinforcement learning from human feedback.arXiv preprint arXiv:2402.00782, 2024. 11 Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning ...

work page arXiv 2024

[2] [2]

A Survey on Knowledge Distillation of Large Language Models

URLhttps://arxiv.org/abs/2402.13116. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to Reason under Off-Policy Guidance.arXiv preprint arXiv:2504.14945, 2025. URL https://arxiv.org/abs/ 2504.14945. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, ...

work page internal anchor Pith review Pith/arXiv arXiv 2025