Pith. sign in

REVIEW 3 major objections 2 minor 5 cited by

On-policy self-distillation fails when privileged information is instance-specific because the student learns an averaged policy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 22:12 UTC pith:VFLFREZP

load-bearing objection The paper breaks down OPD/OPSD failures via instance-specific vs shared PI and names three mechanisms, but the math-reasoning experiments leave open whether those mechanisms are primary outside the tested regime. the 3 major comments →

arxiv 2605.11182 v2 pith:VFLFREZP submitted 2026-05-11 cs.AI

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Siqi Zhu , Xuyan Ye , Hongyu Lu , Weiye Shi , Ge Liu This is my paper
classification cs.AI
keywords on-policy distillationself-distillationprivileged informationmathematical reasoninglarge language modelsdistribution mismatchoptimization instability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines on-policy distillation and on-policy self-distillation for large language models. It shows that these methods are sensitive to teacher models and loss choices in mathematical reasoning tasks. On-policy self-distillation breaks down when the privileged information varies with each instance and is missing at test time. It works when the privileged information is a shared rule such as a system prompt. Three mechanisms explain the failures, and specific fixes restore performance.

Core claim

On-policy self-distillation fails in tested settings because the student learns a privileged-information-free policy that aggregates multiple privileged-information-conditioned teachers, which cannot match the behavior needed when each instance has its own privileged information unavailable at test time. In contrast, the method succeeds when the privileged information encodes a shared latent rule.

What carries the argument

The distinction between instance-specific privileged information and shared latent rules, which determines whether the aggregated policy from on-policy self-distillation remains effective.

Load-bearing premise

The tested mathematical reasoning settings and chosen loss formulations represent the typical conditions for applying on-policy distillation.

What would settle it

An experiment applying OPSD with instance-specific privileged information made available at test time and measuring whether performance degradation disappears.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • OPD on mathematical reasoning is highly sensitive to teacher choice and loss formulation.
  • Three failure mechanisms are distribution mismatch from student prefixes, biased TopK reverse-KL gradients, and the OPSD-specific aggregation of PI-conditioned teachers.
  • Stop-gradient TopK objectives, RLVR-adapted teachers, and SFT-stabilized students mitigate the failures.
  • OPSD succeeds when PI represents shared rules like system prompts or alignment preferences.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Distillation approaches may need to preserve or simulate access to per-instance context during inference to avoid performance loss.
  • Similar issues could arise in other post-training techniques that rely on self-generated trajectories without full context.
  • Testing the fixes on additional domains beyond mathematical reasoning would clarify their generality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper claims that on-policy distillation (OPD) and on-policy self-distillation (OPSD) exhibit mixed effectiveness in post-training of LLMs. OPD is highly sensitive to teacher choice and loss formulation in mathematical reasoning tasks. OPSD fails when privileged information (PI) is instance-specific at test time due to three mechanisms: (1) distribution mismatch from conditioning on student-generated prefixes, (2) optimization instability from biased TopK reverse-KL gradients, and (3) learning a PI-free policy that aggregates PI-conditioned teachers. OPSD succeeds when PI is a shared latent rule like a system prompt. Proposed fixes include stop-gradient TopK objectives, RLVR-adapted teachers, and SFT-stabilized students.

Significance. If the mechanisms and fixes are shown to generalize, the work provides practical guidance on when OPD/OPSD succeed or fail by distinguishing instance-specific versus shared PI. This could help avoid instability in LLM post-training applications such as alignment and knowledge internalization. The empirical identification of concrete failure modes and mitigations adds value even without theoretical derivations.

major comments (3)
  1. [§4] §4 (Mathematical reasoning experiments): The attribution of OPSD failures specifically to the three mechanisms (distribution mismatch, biased TopK reverse-KL, and PI aggregation) is derived from experiments on mathematical reasoning with particular teacher models and loss formulations. Without additional experiments in other domains or with varied model scales to demonstrate that these mechanisms are primary rather than due to unexamined factors, the general claim that OPSD fails due to test-time absence of instance-specific PI does not fully hold.
  2. [§3] §3 (Failure mechanisms): The description of the three mechanisms would be strengthened by quantitative ablations or metrics isolating each one's contribution (e.g., measuring distribution mismatch via KL divergence on prefixes or gradient norms for the TopK bias), as the current evidence rests on observed instabilities without isolating their relative impact.
  3. [Experimental sections] Experimental sections: The manuscript does not report the number of runs, statistical tests, or full ablation results, which is load-bearing for verifying that the observed degradations and mitigations are consistent and not artifacts of the chosen settings.
minor comments (2)
  1. [Abstract] Abstract: The claim that OPSD 'is effective when PI represents a shared latent rule' would benefit from a forward reference to the specific results or section demonstrating this contrast.
  2. [Notation] Notation: Ensure 'PI' is defined at first use and used consistently; some passages refer to 'privileged information' without the acronym after initial definition.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below, indicating planned revisions where the manuscript can be strengthened without overclaiming generality.

read point-by-point responses

  1. Referee: [§4] §4 (Mathematical reasoning experiments): The attribution of OPSD failures specifically to the three mechanisms (distribution mismatch, biased TopK reverse-KL, and PI aggregation) is derived from experiments on mathematical reasoning with particular teacher models and loss formulations. Without additional experiments in other domains or with varied model scales to demonstrate that these mechanisms are primary rather than due to unexamined factors, the general claim that OPSD fails due to test-time absence of instance-specific PI does not fully hold.

    Authors: The manuscript already qualifies its claims to the tested settings (see abstract: 'in our tested settings' and 'OPSD fails in our tested settings'). We agree that additional domains or scales would further support generality but lie outside this work's scope. We will revise §4 and add a limitations paragraph to explicitly bound the claims, highlight that the mechanisms were identified in mathematical reasoning, and note the value of future cross-domain validation. revision: partial

  2. Referee: [§3] §3 (Failure mechanisms): The description of the three mechanisms would be strengthened by quantitative ablations or metrics isolating each one's contribution (e.g., measuring distribution mismatch via KL divergence on prefixes or gradient norms for the TopK bias), as the current evidence rests on observed instabilities without isolating their relative impact.

    Authors: We will strengthen §3 by adding quantitative support: KL divergence between teacher and student next-token distributions conditioned on student prefixes (to isolate distribution mismatch), and per-step gradient norm statistics under TopK reverse-KL (to illustrate bias). These will be reported alongside the existing qualitative observations. revision: yes

  3. Referee: [Experimental sections] Experimental sections: The manuscript does not report the number of runs, statistical tests, or full ablation results, which is load-bearing for verifying that the observed degradations and mitigations are consistent and not artifacts of the chosen settings.

    Authors: We will revise the experimental sections to state that all main results use 3 random seeds, report mean ± std, and include paired t-tests for key comparisons. Full per-seed ablation tables will be moved to the appendix. revision: yes

Circularity Check

0 steps flagged

Empirical study with no circular derivations or self-referential reductions

full rationale

This is a purely empirical paper whose central claims rest on experimental observations across mathematical reasoning tasks, teacher models, and loss formulations. No equations, derivations, or fitted parameters are presented that reduce to their own inputs by construction. The three failure mechanisms are identified and illustrated via ablation experiments rather than proven via self-citation chains or ansatzes. Self-citations, if present, are not load-bearing for the core attribution of failures to instance-specific PI absence. The work is therefore self-contained against external benchmarks and receives the default non-circularity finding.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper is an empirical investigation of existing distillation methods and does not introduce new theoretical constructs, free parameters, or postulated entities.

axioms (1)
  • domain assumption Standard assumptions in machine learning about on-policy sampling and trajectory distributions in LLM training
    The study relies on typical on-policy data generation from the model's current policy without stating deviations.

pith-pipeline@v0.9.1-grok · 5773 in / 1199 out tokens · 27350 ms · 2026-06-30T22:12:24.305654+00:00 · methodology

0 comments
read the original abstract

On-policy distillation (OPD) and on-policy self-distillation (OPSD) have emerged as promising post-training methods for large language models, offering dense token-level supervision on trajectories sampled from the model's own policy. However, existing results on their effectiveness remain mixed: while OP(S)D has shown promise in system prompt and knowledge internalization, recent studies also report instability and degradation. In this work, we present a comprehensive empirical study of when OPD and OPSD work, when they fail, and why. We find that OPD on mathematical reasoning is highly sensitive to teacher choice and loss formulation, whereas OPSD fails in our tested settings due to test-time absence of instance-specific privileged information (PI). In contrast, OPSD is effective when PI represents a shared latent rule, such as a system prompt or alignment preference. We identify three failure mechanisms: (1) distribution mismatch between teacher and student caused by conditioning on student-generated prefixes, (2) optimization instability from biased TopK reverse-KL gradients, and (3) an OPSD-specific limitation where the student learns a PI-free policy that aggregates PI-conditioned teachers, which is insufficient when PI is instance-specific. We further show that stop-gradient TopK objectives, RLVR-adapted teachers, and SFT-stabilized students mitigate these failures.

Figures

Figures reproduced from arXiv: 2605.11182 by Ge Liu, Hongyu Lu, Siqi Zhu, Weiye Shi, Xuyan Ye.

Figure 1
Figure 1. Figure 1: Overview. We map the OP(S)D design space (left, top) and its task-dependent success/fail￾ure behavior (left, bottom), identify three failure mechanisms—prefix-distorted teacher state, biased Top-K reverse-KL, and PI-marginalized OPSD policy (middle), and propose practical fixes: stable Top-K losses, SFT stabilization, and RLVR-adapted teachers (right). In this paper, we present a comprehensive empirical st… view at source ↗
Figure 2
Figure 2. Figure 2: (Left) On-Policy (Self-)Distillation. In OPSD, the teacher is constructed from the student itself and privileged information (PI) is necessary. In OPD, the teacher is a stronger model and PI is optional. (Right) p: teacher distribution, q: student distribution. Reverse KL is mode-seeking, whereas forward KL is mode-covering. Reinforcement Learning from Textual Feedback. Another related direction augments r… view at source ↗
Figure 3
Figure 3. Figure 3: Qwen3-1.7B, trained on OpenThoughts. OPSD fails to improve student. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Collapse under unnormalized Top-20 reverse KL. The model first becomes verbose, then degenerates into repetitive “maybe” outputs as response length reaches the limit and evaluation accuracy drops. Token statistics show that repetitive tokens dominate as the repeat ratio approaches one. 4 Experiments We evaluate OPD and OPSD on reasoning, system-prompt internalization, and alignment, covering both failure a… view at source ↗
Figure 5
Figure 5. Figure 5: Training reward (left) and evaluation score (right) curves for OPSD, GRPO, and PPO on [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of GRPO and OPSD on Qwen3-8B (thinking mode) trained with DAPO-Math [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Train and evaluate Qwen3-1.7B (nothink) on Wildguardmix using their original train and [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Effectiveness of OPSD depends on the structure of privileged information I. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: PI does not improve OPD on math reasoning with a stronger teacher. Using a Qwen3- 8B teacher and a Qwen3-1.7B student on OpenThoughts, both final-answer PI and full-response PI underperform vanilla OPD. PI-conditioned OPD leads to higher KL loss. This form indicates that OPSD can distill behavior that is consistently supported under different PI. Outputs that receive high probability under some PI but low… view at source ↗
Figure 11
Figure 11. Figure 11: Teacher: Qwen3-1.7B-GRPO (nothink), Student: Qwen3-1.7B (nothink), DAPO, TopK=5. [PITH_FULL_IMAGE:figures/full_fig_p008_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Whether to put the distillation loss in policy gradient? sampled token KL in policy gradient [PITH_FULL_IMAGE:figures/full_fig_p008_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: dataset: OpenThoughts. Left: Qwen3-8B and Qwen3-1.7B-GRPO have similar math reasoning performance. Middle: In OPD, Qwen3-1.7B-GRPO is a more effective teacher. Right: Qwen3-1.7B-GRPO’s Top20 vocabulary distribution is more aligned with the Qwen3-1.7B student. 0 50000 100000 150000 200000 250000 Number of Training Samples 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Task Reward (Pass@1) MATH-500 Task Reward Direct OPD … view at source ↗
Figure 14
Figure 14. Figure 14: Qwen3-4B teacher, Qwen3-1.7B-Base student, OpenThoughts. [PITH_FULL_IMAGE:figures/full_fig_p009_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Teacher: Qwen3-1.7B-GRPO (nothink), Student: Qwen3-1.7B (nothink), training data: [PITH_FULL_IMAGE:figures/full_fig_p019_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Comparison of teacher signal on responses generated by different student models. [PITH_FULL_IMAGE:figures/full_fig_p020_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Comparison of token-level KL supervision distributions for correct and incorrect student [PITH_FULL_IMAGE:figures/full_fig_p020_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: We show token-level heatmap of ∆logprob on last 128 tokens. The experiment is based on openthoughts [22], we show an example question. PI strengthens supervision for the same teacher, yet the sampled-token supervision distribution is based more on teacher capability (as shown in the figure, 3 experiments using Qwen3-8B teacher show similar distribution, while 2 experiments using Qwen3-1.7B teacher show an… view at source ↗
Figure 19
Figure 19. Figure 19: General reasoning results of OPD training. The experiment uses the Science subset of [PITH_FULL_IMAGE:figures/full_fig_p021_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Comparison of teacher signals on general reasoning trajectories. [PITH_FULL_IMAGE:figures/full_fig_p022_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Next-token log probs (left), truncated ratio (middle) and evaluation results (right) curves [PITH_FULL_IMAGE:figures/full_fig_p022_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: An example of thinking mode hacking during OPSD. The student is trained with thinking mode disabled, while the teacher is queried with reasoning enabled. During training, the student gradually learns to emit explicit thinking-mode control tokens in its response, even though such tokens are not intended to appear at inference time. 1.7b and is trained on dapo [20]. We observe a failure mode that we term th… view at source ↗
Figure 23
Figure 23. Figure 23: ∆logprob - token entropy. Teacher: Qwen3-8B w/ PI. and imagery to tell true human stories. I hold fast to poetic meter, seek no popular applause, and ask only that everyone who hears me feels understood. When asked why I do not try new forms, I say: true innovation is not breaking tradition, but letting tradition be reborn in new breath. Each of my recitations guards and transmits ancient wisdom -- not to… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

    cs.LG 2026-06 unverdicted novelty 6.0

    RLCSD contrasts teacher-student distributional gaps under correct versus wrong hints to suppress privilege-induced style drift and concentrate supervision on task tokens, outperforming GRPO and prior OPSD on Qwen3 and...

  2. Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions

    cs.CV 2026-06 unverdicted novelty 6.0

    Z-Reward trains a 27B reasoning teacher VLM on score distributions via GDSO and distills it via RISD into a 9B student, reaching 89.6% and 88.6% human preference accuracy with 41.3% optimization gain over SFT baseline.

  3. Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents

    cs.LG 2026-06 unverdicted novelty 5.0

    SGCD improves held-out scores on AppWorld and tau^3-airline by using LLM-summarized sibling contrasts to reshape GRPO advantages while keeping policy gradient in charge of the actor update.

  4. A Formula-Driven Survey and Research Agenda for On-Policy Distillation

    cs.AI 2026-06 unverdicted novelty 4.0

    A survey creates a taxonomy for on-policy distillation in LLMs that separates temporal credit assignment from vocabulary-level probability routing.

  5. Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents

    cs.LG 2026-06 unverdicted novelty 4.0

    SGCD uses LLM-summarized contrasts from successful/failed sibling rollouts to adjust token advantages in GRPO, reporting modest gains on AppWorld and τ³-airline benchmarks.

Reference graph

Works this paper leans on

28 extracted references · 2 canonical work pages · cited by 4 Pith papers · 1 internal anchor

  1. [1]

    On-policy distillation of language models: Learning from self- generated mistakes, 2024

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes, 2024

    2024

  2. [2]

    Minillm: On-policy distillation of large language models, 2026

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: On-policy distillation of large language models, 2026

    2026

  3. [3]

    Deepseek-v4: Towards highly efficient million-token context intelligence, 2026

    DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026

    2026

  4. [4]

    Mimo-v2-flash technical report, 2026

    Core Team, Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, Gang Xie, Hailin Zhang, Hanglong Lv, Hanyu Li, Heyu Chen, Hongshen Xu, Houbin Zhang, Huaqiu Liu, Jiangshan Duo, Jianyu Wei, Jiebao Xiao, Jinhao Dong, Jun Shi, Junhao Hu, Kainan Bao, Kang Zhou, Lei Li, Liang Zhao, Linghao Zhang,...

    2026

  5. [5]

    On-policy distillation.Thinking Machines Lab: Connec- tionism, 2025

    Kevin Lu and Thinking Machines Lab. On-policy distillation.Thinking Machines Lab: Connec- tionism, 2025. https://thinkingmachines.ai/blog/on-policy-distillation

    2025

  6. [6]

    Self-distillation enables continual learning, 2026

    Idan Shenfeld, Mehul Damani, Jonas H¨ubotter, and Pulkit Agrawal. Self-distillation enables continual learning, 2026

    2026

  7. [7]

    Self-distilled reasoner: On-policy self-distillation for large language models, 2026

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models, 2026

    2026

  8. [8]

    On-policy self-distillation for reasoning compression, 2026

    Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. On-policy self-distillation for reasoning compression, 2026

    2026

  9. [9]

    On-policy context distillation for language models, 2026

    Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models, 2026

    2026

  10. [10]

    Revisiting on-policy distillation: Empirical failure modes and simple fixes, 2026

    Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes, 2026

    2026

  11. [11]

    Why does self-distillation (sometimes) degrade the reasoning capability of llms?, 2026

    Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of llms?, 2026

    2026

  12. [12]

    Learning by distilling context, 2022

    Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context, 2022

    2022

  13. [13]

    Andrew Bagnell, Aarti Singh, and Andrea Zanette

    Yuda Song, Lili Chen, Fahim Tajwar, Remi Munos, Deepak Pathak, J. Andrew Bagnell, Aarti Singh, and Andrea Zanette. Expanding the capabilities of reinforcement learning via text feedback, 2026

    2026

  14. [14]

    Pope: Learning to reason on hard problems via privileged on-policy exploration, 2026

    Yuxiao Qu, Amrith Setlur, Virginia Smith, Ruslan Salakhutdinov, and Aviral Kumar. Pope: Learning to reason on hard problems via privileged on-policy exploration, 2026. 11

    2026

  15. [15]

    Rupam Mahmood, and Martha White

    Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A. Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse kl divergences, 2022

    2022

  16. [16]

    Characterbench: Benchmarking character customization of large language models, 2024

    Jinfeng Zhou, Yongkang Huang, Bosi Wen, Guanqun Bi, Yuxuan Chen, Pei Ke, Zhuang Chen, Xiyao Xiao, Libiao Peng, Kuntian Tang, Rongsheng Zhang, Le Zhang, Tangjie Lv, Zhipeng Hu, Hongning Wang, and Minlie Huang. Characterbench: Benchmarking character customization of large language models, 2024

    2024

  17. [17]

    Jen tse Huang, Man Ho Lam, Eric John Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, and Michael R. Lyu. Emotionally numb or empathetic? evaluating how llms feel using emotionbench, 2024

    2024

  18. [18]

    Crisp: Compressed reasoning via iterative self-policy distillation, 2026

    Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. Crisp: Compressed reasoning via iterative self-policy distillation, 2026

    2026

  19. [19]

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms, 2024

    Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms, 2024

    2024

  20. [20]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, W...

    2025

  21. [21]

    Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe, 2026

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe, 2026

    2026

  22. [22]

    Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang, Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, ...

    2025

  23. [23]

    Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models

    Feng Luo, Yu-Neng Chuang, Guanchu Wang, Zicheng Xu, Xiaotian Han, Tianyi Zhang, and Vladimir Braverman. Demystifying opd: Length inflation and stabilization strategies for large language models.arXiv preprint arXiv:2604.08527, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  24. [24]

    Reinforcement learning via self-distillation, 2026

    Jonas H ¨ubotter, Frederike L ¨ubeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation, 2026

    2026

  25. [25]

    Openclaw-rl: Train any agent simply by talking, 2026

    Yinjie Wang, Xuyang Chen, Xiaolong Jin, Mengdi Wang, and Ling Yang. Openclaw-rl: Train any agent simply by talking, 2026

    2026

  26. [26]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017

    2017

  27. [27]

    Persuasion for good: Towards a personalized persuasive dialogue system for social good

    Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. Persuasion for good: Towards a personalized persuasive dialogue system for social good. arXiv preprint arXiv:1906.06725, 2019

    work page arXiv 1906

  28. [28]

    L(θ, y)∇θ logπ θ(y|x) + 1 T TX t=1 ∇θℓt(θ, y<t) # . 17 Usinglogπ θ(y|x) = PT t=1 logπ θ(yt |x, y <t), the rollout term becomes Ex,y

    Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models, 2026. 12 A Appendix A.1 Experiment Setup We use a maximum response length of16384, temperature1.0, and top-p0.95for evaluation. Table 1 summarizes the main hyperparameters for OPD,...

    2026