pith. sign in

arxiv: 2605.07804 · v3 · pith:IBATJDLOnew · submitted 2026-05-08 · 💻 cs.LG · cs.AI

Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning

Zhicheng Yang , Zhijiang Guo , Yifan Song , Minrui Xu , Yongxin Wang , Yiwei Wang , Xiaodan Liang , Jing Tang This is my paper

Pith reviewed 2026-06-30 23:00 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords on-policy distillationprefix driftreward reliabilitylong-horizon reasoningdynamic truncationteacher-student compatibilitymath reasoning benchmarks
0
0 comments X

The pith

Prune-OPD detects when student prefixes drift from teacher reasoning and truncates unreliable rewards to cut training time by 37.6 to 68 percent while preserving benchmark performance.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that on-policy distillation for long-horizon reasoning breaks down once the student's generated prefix diverges from the teacher's path, because the teacher's dense rewards then lose local usefulness. Prune-OPD watches the overlap between the top-k tokens the student and teacher would pick at each step to catch this drift as soon as it happens. When overlap drops sharply, the method lowers the weight of later rewards and stops generating further tokens on that trajectory. Compute is then redirected only to segments where the teacher's signal remains reliable. On math reasoning benchmarks the approach shortens training runs substantially yet keeps or raises final scores across several teacher-student size combinations.

Core claim

Prune-OPD aligns computation with supervision reliability by continuously checking top-k overlap to detect prefix-drift events in real time; upon severe drift it monotonically down-weights subsequent rewards and triggers dynamic rollout truncation so that training halts on unexploitable trajectories and reallocates budget to locally reliable teacher signals.

What carries the argument

Top-k overlap monitoring between student and teacher next-token predictions, used as a real-time detector that triggers reward down-weighting and rollout truncation.

If this is right

  • When prefix drift is detected, training time drops 37.6-68.0 percent with no loss, and sometimes gains, on AMC, AIME, and HMMT.
  • When student-teacher predictions stay aligned, the method automatically keeps longer rollouts to retain full supervision.
  • The savings come from reallocating compute to reliable segments rather than applying a fixed shorter length to every rollout.
  • The same detection logic works across multiple teacher-student size pairs without retuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same overlap check could be tried in other sequential decision settings where reward quality degrades along a trajectory, such as long-horizon planning or code generation.
  • An experiment that replaces top-k overlap with a different local similarity measure (for example, embedding cosine) would test whether the particular indicator is essential.
  • If the method is combined with curriculum ordering of training examples, early high-compatibility rollouts might further reduce total wall-clock time.

Load-bearing premise

That a drop in top-k overlap between student and teacher predictions is enough to know the teacher's reward has become locally useless and that cutting the rollout at that point does not throw away recoverable learning.

What would settle it

Run the same teacher-student pairs on AMC/AIME/HMMT but continue full rollouts even after low top-k overlap; if final accuracy falls below the pruned version, the truncation rule would be falsified.

Figures

Figures reproduced from arXiv: 2605.07804 by Jing Tang, Minrui Xu, Xiaodan Liang, Yifan Song, Yiwei Wang, Yongxin Wang, Zhicheng Yang, Zhijiang Guo.

Figure 1
Figure 1. Figure 1: Conceptual overview of PRUNE-OPD. PRUNE-OPD monitors local student-teacher compatibility along the student rollout, monotonically attenuates OPD rewards after low-overlap drift events, and truncates the response once reliable teacher supervision is exhausted. However, the same on-policy design creates a new reliability problem. The teacher is queried not only on prefixes where its local distribution offers… view at source ↗
Figure 2
Figure 2. Figure 2: High-compatibility training dynamics for DeepSeek-R1-Distill-Qwen-7B / Skywork-OR1-7B. Left: effective response length and maximum OPD length versus training step. Middle: overlap ratio versus training step. Right: AMC23 accuracy over training, comparing OPD, OPD (Truncate 4k), and PRUNE-OPD. suggesting that exact sampled-token acceptability can be too strict for reasoning traces, whereas candidate-space o… view at source ↗
Figure 3
Figure 3. Figure 3: Training-step accuracy dynamics for DeepSeek-R1-Distill-Qwen-1.5B distilled from JustRL￾DeepSeek-1.5B. The five panels report benchmark accuracy over training steps on AMC23, AIME24, AIME25, HMMT24, and HMMT25, comparing OPD and PRUNE-OPD [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training-dynamics diagnostics for DeepSeek-R1-Distill-Qwen-1.5B distilled from JustRL-DeepSeek￾1.5B. The panels report mean Prune-OPD weight by token position with curves every 20 training steps from 0 to 200; effective response length and maximum OPD length over training; and overlap ratio over training. 4.6 Ablation Study OPD with simple truncation. We include OPD (Truncate 4k) as a fixed-budget baseline… view at source ↗
Figure 4
Figure 4. Figure 4: Training-step accuracy dynamics for DeepSeek-R1-Distill-Qwen-1.5B distilled from JustRL￾DeepSeek-1.5B. The five panels report benchmark accuracy over training steps on AMC23, AIME24, AIME25, HMMT24, and HMMT25, comparing OPD and PRUNE-OPD [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Accuracy over wall-clock time for the 4 DeepSeek student-teacher pairs. Each panel uses wall-clock time as the x-axis and benchmark accuracy as the y-axis, comparing OPD and PRUNE-OPD. A successful curve should match or exceed OPD accuracy while reaching comparable checkpoints earlier in time [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Short effective OPD windows in the low-overlap Qwen3 distillation pairs. For Qwen3-1.7B-Base / Qwen3-4B (Non-thinking) and Qwen3-4B-Base / Qwen3-4B (Non-thinking), low overlap causes PRUNE￾OPD to concentrate OPD supervision within a few hundred reliable tokens, whereas the OPD baseline keeps training on responses up to 12,288 tokens. and PRUNE-OPD therefore keeps the effective OPD length at only a few hund… view at source ↗
Figure 7
Figure 7. Figure 7: OPD baseline overlap-ratio training dynamics for DeepSeek-R1-Distill-Qwen-1.5B / JustRL￾DeepSeek-1.5B. Each panel plots overlap ratio versus training step for a token-position band: 0–1K, 2–3K, 4–6K, and 7–8K. This diagnostic shows how local student-teacher compatibility evolves at different trajectory depths under unpruned OPD. 0 2K 4K 6K Token position 0.6 0.9 1.2 1.5 Token Weight (a) Token Weight γ=0.6 … view at source ↗
Figure 8
Figure 8. Figure 8: Prune-OPD threshold diagnostics for DeepSeek-R1-Distill-Qwen-1.5B / JustRL-DeepSeek-1.5B. Left: mean Prune-OPD weight as a function of token position under three overlap thresholds, γ = 0.6, 0.7, 0.8; for each threshold, the curves are taken at training steps 100, 120, 140, 160, 180, and 200. Right: maximum OPD response length over training steps under the same thresholds. Together, these diagnostics show … view at source ↗
read the original abstract

On-policy distillation (OPD) leverages dense teacher rewards to enhance reasoning models. However, scaling OPD to long-horizon tasks exposes a critical flaw: as the student's generated prefix inevitably diverges from the teacher's thought process, the teacher's dense reward loses local exploitability. Continuing to generate and evaluate tokens on these ``drifted'' trajectories not only degrades reward quality but also incurs massive computational waste. To address this, we introduce \textbf{Prune-OPD}, a framework that dynamically aligns training budgets with supervision quality. By continuously monitoring the local compatibility between student and teacher predictions (e.g., via top-$k$ overlap), Prune-OPD detects prefix-drift events in real time. Upon detecting severe drift, it monotonically down-weights subsequent unreliable rewards and triggers dynamic rollout truncation. This allows the training process to halt futile generation and reallocate compute strictly to reliable teacher supervision. Across diverse teacher-student combinations, Prune-OPD consistently aligns computation with supervision reliability. When prefix drift makes dense teacher rewards unreliable, it reduces training time by 37.6\%--68.0\% while preserving, and often improving, performance on challenging benchmarks (AMC, AIME, HMMT). When student-teacher compatibility remains high, it automatically preserves long-context supervision by expanding the training window. These results suggest that Prune-OPD improves OPD not by blindly shortening rollouts, but by reallocating computation toward locally exploitable teacher rewards.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces Prune-OPD, an extension to on-policy distillation (OPD) for long-horizon reasoning. It identifies prefix drift as a source of unreliable dense teacher rewards and computational waste, proposing to monitor local student-teacher compatibility (via top-k overlap) in real time. Upon detecting severe drift, the method applies monotonic down-weighting of subsequent rewards and dynamic rollout truncation to reallocate compute toward reliable supervision segments. When compatibility remains high, it expands the training window. Empirical claims include 37.6%–68.0% training-time reductions while preserving or improving performance on AMC, AIME, and HMMT across diverse teacher-student pairs.

Significance. If the core mechanism proves reliable, Prune-OPD offers a practical way to scale OPD to longer reasoning trajectories by avoiding wasted rollouts on non-exploitable prefixes. The dynamic, compatibility-driven budget allocation is a concrete engineering contribution that could reduce the cost of distilling reasoning capabilities without requiring changes to the underlying teacher or loss formulation.

major comments (2)
  1. [Section 3.2] Section 3.2: The central design decision treats top-k overlap (or similar local compatibility metrics) as a sufficient real-time proxy for when prefix drift renders the teacher's dense reward non-exploitable. No ablation, correlation analysis, or diagnostic is presented showing that low overlap predicts degraded gradient signal (e.g., reward variance, advantage magnitude, or downstream policy improvement on the drifted prefix). Without this link, the truncation and down-weighting rules risk either premature termination or continued training on low-value trajectories.
  2. [Experimental section] Experimental section (presumably §4–5): The abstract reports concrete speed-ups (37.6%–68.0%) and benchmark gains, yet the provided text contains no description of baselines, number of random seeds, variance estimates, or controls that isolate the effect of the overlap-triggered truncation from other implementation choices. This makes it impossible to assess whether the claimed efficiency gains are robust or attributable to the proposed mechanism.
minor comments (2)
  1. [Abstract] The abstract states performance numbers without any accompanying experimental protocol; the main text should include a concise experimental-setup paragraph early in the results section to allow readers to interpret the reported percentages.
  2. [Section 3] Notation for the compatibility metric and the exact functional form of the monotonic down-weighting schedule should be defined with an equation rather than described only in prose.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the justification of our core metric and the experimental reporting standards. We address each major comment below and commit to revisions that strengthen the manuscript without altering its core claims.

read point-by-point responses

  1. Referee: [Section 3.2] Section 3.2: The central design decision treats top-k overlap (or similar local compatibility metrics) as a sufficient real-time proxy for when prefix drift renders the teacher's dense reward non-exploitable. No ablation, correlation analysis, or diagnostic is presented showing that low overlap predicts degraded gradient signal (e.g., reward variance, advantage magnitude, or downstream policy improvement on the drifted prefix). Without this link, the truncation and down-weighting rules risk either premature termination or continued training on low-value trajectories.

    Authors: We agree that an explicit empirical link between top-k overlap and gradient quality would strengthen the justification. The current manuscript motivates the metric via the observed divergence in reasoning trajectories but does not include the requested correlation or ablation studies. In the revised version we will add (i) a diagnostic plot correlating overlap scores with reward variance and advantage magnitude on drifted prefixes and (ii) an ablation comparing performance when truncation is triggered by overlap versus by a direct gradient-quality threshold. These additions will directly address the concern about premature or delayed truncation. revision: yes

  2. Referee: [Experimental section] Experimental section (presumably §4–5): The abstract reports concrete speed-ups (37.6%–68.0%) and benchmark gains, yet the provided text contains no description of baselines, number of random seeds, variance estimates, or controls that isolate the effect of the overlap-triggered truncation from other implementation choices. This makes it impossible to assess whether the claimed efficiency gains are robust or attributable to the proposed mechanism.

    Authors: The referee correctly identifies missing experimental details. The manuscript text does not report the number of random seeds, variance estimates, or explicit controls isolating the truncation mechanism. In the revision we will (i) specify all baselines and implementation choices, (ii) report mean and standard deviation over at least three seeds, and (iii) add an ablation that disables the overlap-triggered truncation while keeping all other components fixed. These changes will allow readers to evaluate the robustness and attribution of the reported speed-ups. revision: yes

Circularity Check

0 steps flagged

No circularity; heuristic engineering intervention with no derived or fitted quantities

full rationale

The paper describes Prune-OPD as a monitoring-and-truncation framework that uses local compatibility metrics (e.g., top-k overlap) to detect prefix drift and adjust reward weighting/rollout length. No equations, parameter fits, predictions, or uniqueness theorems appear in the provided text. No self-citations are used to justify core premises. The method is presented as an empirical engineering choice rather than a derivation that reduces to its own inputs by construction; performance claims rest on benchmark results, not on any self-referential reduction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are described in the abstract.

pith-pipeline@v0.9.1-grok · 5821 in / 985 out tokens · 27183 ms · 2026-06-30T23:00:38.329029+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Blockwise Policy-Drift Gating for On-Policy Distillation

    cs.LG 2026-06 unverdicted novelty 5.0

    Blockwise policy-drift gating raises mean pass@8 from 0.4978 to 0.5160 on four math benchmarks by reweighting OPD losses with detached mean-normalized gates from student policy drift over 64-token blocks.

  2. Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts

    cs.LG 2026-06 unverdicted novelty 5.0

    PG-OPD uses early prefix overlap to selectively continue only high-compatibility rollouts in on-policy distillation, reporting up to 4.8 accuracy points gained and 2.46x less training time on math benchmarks.

Reference graph

Works this paper leans on

47 extracted references · 4 canonical work pages · cited by 2 Pith papers · 4 internal anchors

  1. [1]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. InInternational Conference on Learning Representations, 2024

    2024

  2. [2]

    Scheduled sampling for sequence prediction with recurrent neural networks.Advances in neural information processing systems, 28, 2015

    Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks.Advances in neural information processing systems, 28, 2015

    2015

  3. [3]

    Distillation scaling laws

    Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russell Webb. Distillation scaling laws. InInternational Conference on Machine Learning, pages 5977–6045. PMLR, 2025

    2025

  4. [4]

    On the efficacy of knowledge distillation

    Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. InProceedings of the IEEE/CVF international conference on computer vision, pages 4794–4802, 2019

    2019

  5. [5]

    Scaling instruction-finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024

    2024

  6. [6]

    Hdpo: Hybrid distillation policy optimization via privileged self-distillation, 2026

    Ken Ding. Hdpo: Hybrid distillation policy optimization via privileged self-distillation, 2026

    2026

  7. [7]

    Revisiting on-policy distillation: Empirical failure modes and simple fixes, 2026

    Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes, 2026

    2026

  8. [8]

    Co-evolving policy distillation, 2026

    Naibin Gu, Chenxu Yang, Qingyi Si, Chuanyu Qin, Dingyu Yao, Peng Fu, Zheng Lin, Weiping Wang, Nan Duan, and Jiaqi Wang. Co-evolving policy distillation, 2026

    2026

  9. [9]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InThe twelfth international conference on learning representations, 2024

    2024

  10. [10]

    Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature, 645(8081):633–638, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature, 645(8081):633–638, 2025

    2025

  11. [11]

    Justrl: Scaling a 1.5b llm with a simple rl recipe, 2025

    Bingxiang He, Zekai Qu, Zeyuan Liu, Yinghao Chen, Yuxin Zuo, Cheng Qian, Kaiyan Zhang, Weize Chen, Chaojun Xiao, Ganqu Cui, et al. Justrl: Scaling a 1.5b llm with a simple rl recipe, 2025

    2025

  12. [12]

    How far can unsupervised rlvr scale llm training?, 2026

    Bingxiang He, Yuxin Zuo, Zeyuan Liu, Shangziqi Zhao, Zixuan Fu, Junlin Yang, Cheng Qian, Kaiyan Zhang, Yuchen Fan, Ganqu Cui, et al. How far can unsupervised rlvr scale llm training?, 2026

    2026

  13. [13]

    Distilling the Knowledge in a Neural Network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  14. [14]

    Reinforcement learning via self-distillation, 2026

    Jonas Hubotter, Frederike Lubeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation, 2026

    2026

  15. [15]

    Stable on-policy distillation through adaptive target reformulation, 2026

    Ijun Jang, Jewon Yeom, Juan Yeo, Hyunggu Lim, and Taesup Kim. Stable on-policy distillation through adaptive target reformulation, 2026

    2026

  16. [16]

    Tinybert: Distilling bert for natural language understanding

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174, 2020

    2020

  17. [17]

    Entropy-aware on-policy distillation of language models, 2026

    Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models, 2026

    2026

  18. [18]

    Why does self-distillation (sometimes) degrade the reasoning capability of llms?, 2026

    Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of llms?, 2026. 10

    2026

  19. [19]

    Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, 2016

    2016

  20. [20]

    Scaling reasoning efficiently via relaxed on-policy distillation, 2026

    Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling reasoning efficiently via relaxed on-policy distillation, 2026

    2026

  21. [21]

    Unifying group-relative and self-distillation policy optimization via sample routing, 2026

    Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, and Tat-Seng Chua. Unifying group-relative and self-distillation policy optimization via sample routing, 2026

    2026

  22. [22]

    Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  23. [23]

    Small models struggle to learn from strong reasoners

    Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ra- masubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. InFindings of the Association for Computational Linguistics: ACL 2025, pages 25366–25394, 2025

    2025

  24. [24]

    On-policy distillation.Thinking Machines Lab: Connec- tionism, 2025

    Kevin Lu and Thinking Machines Lab. On-policy distillation.Thinking Machines Lab: Connec- tionism, 2025. https://thinkingmachines.ai/blog/on-policy-distillation

    2025

  25. [25]

    Improved knowledge distillation via teacher assistant

    Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. InProceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5191–5198, 2020

    2020

  26. [26]

    Crisp: Compressed reasoning via iterative self-policy distillation, 2026

    Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. Crisp: Compressed reasoning via iterative self-policy distillation, 2026

    2026

  27. [27]

    Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter, 2019

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter, 2019

    2019

  28. [28]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. Multitask prompted training enables zero-shot task generalization. InInternational Conference on Learning Representations, 2021

    2021

  29. [29]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  30. [30]

    Self-distillation enables continual learning, 2026

    Idan Shenfeld, Mehul Damani, Jonas Hubotter, and Pulkit Agrawal. Self-distillation enables continual learning, 2026

    2026

  31. [31]

    EHRAG: Bridging Semantic Gaps in Lightweight GraphRAG via Hybrid Hypergraph Construction and Retrieval

    Yifan Song, Xingjian Tao, Zhicheng Yang, Yihong Luo, and Jing Tang. Ehrag: Bridging semantic gaps in lightweight graphrag via hybrid hypergraph construction and retrieval.arXiv preprint arXiv:2604.17458, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  32. [32]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.Advances in Neural Information Processing Systems, 33:5776–5788, 2020

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.Advances in Neural Information Processing Systems, 33:5776–5788, 2020

    2020

  33. [33]

    Dai, and Quoc V

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V . Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021

    2021

  34. [34]

    Mimo-v2-flash technical report, 2026

    Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report, 2026

    2026

  35. [35]

    Envfactory: Scaling tool-use agents via executable environments synthesis and robust rl, 2026

    Minrui Xu, Zilin Wang, Mengyi DENG, Zhiwei Li, Zhicheng Yang, Xiao Zhu, Yinhong Liu, Boyu Zhu, Baiyu Huang, Chao Chen, Heyuan Deng, Fei Mi, Lifeng Shang, Xingshan Zeng, and Zhijiang Guo. Envfactory: Scaling tool-use agents via executable environments synthesis and robust rl, 2026. 11

    2026

  36. [36]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report, 2025

    2025

  37. [37]

    Self-distilled rlvr, 2026

    Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr, 2026

    2026

  38. [38]

    Learning beyond teacher: Generalized on-policy distillation with reward extrapolation, 2026

    Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation, 2026

    2026

  39. [39]

    Accordion-thinking: Self-regulated step summaries for efficient and readable llm reasoning

    Zhicheng Yang, Zhijiang Guo, Yinya Huang, Yongxin Wang, Wenlei Shi, Yiwei Wang, Xiaodan Liang, and Jing Tang. Accordion-thinking: Self-regulated step summaries for efficient and readable llm reasoning. InForty-Third International Conference on Machine Learning, 2026

    2026

  40. [40]

    Depth-breadth synergy in rlvr: Unlocking llm reasoning gains with adaptive exploration

    Zhicheng Yang, Zhijiang Guo, Yinya Huang, Yongxin Wang, Dongchun Xie, Hanhui Li, Yiwei Wang, Xiaodan Liang, and Jing Tang. Depth-breadth synergy in rlvr: Unlocking llm reasoning gains with adaptive exploration. InForty-Third International Conference on Machine Learning, 2026

    2026

  41. [41]

    Optibench meets resocratic: Measure and improve LLMs for optimization modeling

    Zhicheng Yang, Yiwei Wang, Yinya Huang, Zhijiang Guo, Wei Shi, Xiongwei Han, Liang Feng, Linqi Song, Xiaodan Liang, and Jing Tang. Optibench meets resocratic: Measure and improve LLMs for optimization modeling. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  42. [42]

    Online experiential learning for language models, 2026

    Tianzhu Ye, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. Online experiential learning for language models, 2026

    2026

  43. [43]

    On-policy context distillation for language models, 2026

    Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models, 2026

    2026

  44. [44]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale, 2025

    2025

  45. [45]

    Glm-5: From vibe coding to agentic engineering, 2026

    Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chengxing Xie, Cunxiang Wang, et al. Glm-5: From vibe coding to agentic engineering, 2026

    2026

  46. [46]

    Self- distillation for multi-token prediction, 2026

    Guoliang Zhao, Ruobing Xie, An Wang, Shuaipeng Li, Huaibing Xie, and Xingwu Sun. Self- distillation for multi-token prediction, 2026

    2026

  47. [47]

    reward hacking

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models, 2026. A Appendix A.1 Broader Impact This research contributes to the development of more efficient and reliable training protocols for LLMs specializing in complex reasoning. By introdu...

    2026