pith. sign in

arxiv: 2605.18643 · v2 · pith:74DINWF6new · submitted 2026-05-18 · 💻 cs.LG · cs.AI· cs.CL

Post-Trained MoE Can Skip Half Experts via Self-Distillation

Xingtai Lv , Li Sheng , Kaiyan Zhang , Yichen You , Siyan Gao , Xueheng Luo , Yuxin Zuo , Yuchen Fan
show 7 more authors
Junlin Yang Ganqu Cui Bingning Wang Fan Yang Youbang Sun Ning Ding Bowen Zhou
This is my paper

Pith reviewed 2026-06-30 18:20 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CL
keywords mixture of expertsself-distillationdynamic routinginference efficiencymodel adaptationlarge language models
0
0 comments X

The pith

Post-trained static MoE models can be turned dynamic to skip over half their experts using zero-output additions and self-distillation.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that fully trained Mixture-of-Experts models do not need to be rebuilt from scratch to gain dynamic routing. Instead, parameter-free experts that always output zero can be inserted into each layer, after which the model undergoes two-stage self-distillation from its own frozen original version plus a balancing loss. This produces input-dependent routing that lets many tokens bypass experts. On Qwen3-30B-A3B and GLM-4.7-Flash the resulting models cut expert FLOPs by more than 50 percent across math, code, and instruction benchmarks while accuracy stays nearly unchanged. The same models also beat prior dynamic MoE baselines by 4 to 6 points and deliver measurable end-to-end speedups.

Core claim

ZEDA converts a post-trained static MoE into an efficient dynamic version by injecting parameter-free zero-output experts into every MoE layer and adapting the augmented model through two-stage self-distillation that treats the original MoE as a frozen teacher together with a group-level balancing loss, yielding over 50 percent reduction in expert FLOPs at marginal accuracy loss on eleven benchmarks.

What carries the argument

The ZEDA framework of zero-output expert injection followed by two-stage self-distillation from a frozen teacher that learns stable input-dependent routing.

If this is right

  • Already-trained static MoE models become candidates for dynamic conversion at low additional cost.
  • Inference serving can route easy tokens past more than half the experts without retraining the base weights.
  • The same conversion works on models of different sizes and across math, code, and instruction tasks.
  • Dynamic MoE performance can exceed prior routing baselines when the teacher remains the original static model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The zero-expert trick might extend to other sparse activation patterns to encourage skipping without new parameters.
  • If the balancing loss proves critical, similar group constraints could be tested in non-MoE sparse models.
  • End-to-end speedup of 1.2x suggests the method could be combined with quantization or KV-cache optimizations for further gains.

Load-bearing premise

Adding parameter-free zero-output experts and performing two-stage self-distillation from a frozen teacher will produce stable routing that preserves performance without task-specific fine-tuning or retraining from scratch.

What would settle it

Apply ZEDA to a third post-trained MoE model on a held-out benchmark suite and measure whether expert FLOPs drop below 50 percent or accuracy falls more than the marginal loss reported on the original two models.

read the original abstract

Mixture-of-Experts (MoE) scales language models efficiently through sparse expert activation, and its dynamic variant further reduces computation by adjusting the activated experts in an input-dependent manner. Existing dynamic MoE methods usually rely on pre-training from scratch or task-specific adaptation, leaving the practical conversion of fully trained MoE underexplored. Enabling such adaptation would directly alleviate the inference costs by allowing easy tokens to bypass unnecessary expert during serving. This paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework that transforms post-trained static MoE models into efficient dynamic ones. To stabilize this architectural conversion, ZEDA injects parameter-free zero-output experts into each MoE layer and adapts the augmented model through two-stage self-distillation, utilizing the original MoE as a frozen teacher and applying a group-level balancing loss. On Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks spanning math, code, and instruction following, ZEDA eliminates over 50% of expert FLOPs at marginal accuracy loss. It outperforms the strongest dynamic MoE baseline by 6.1 and 4.0 points on the two models, and delivers ~1.20$\times$ end-to-end inference speedup.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework to convert post-trained static MoE models into dynamic ones. ZEDA injects parameter-free zero-output experts into each MoE layer and adapts the model via two-stage self-distillation using the original static MoE as a frozen teacher together with a group-level balancing loss. On Qwen3-30B-A3B and GLM-4.7-Flash evaluated across 11 benchmarks in math, code, and instruction following, ZEDA eliminates over 50% of expert FLOPs at marginal accuracy loss, outperforms the strongest dynamic MoE baseline by 6.1 and 4.0 points respectively, and yields approximately 1.20× end-to-end inference speedup.

Significance. If the empirical results hold, the work supplies a practical post-training route to dynamic expert skipping in already-trained MoE models, avoiding the cost of pre-training dynamic MoE from scratch or task-specific adaptation. The reported FLOP reductions and benchmark gains on two distinct models indicate a potentially useful contribution to efficient inference for large-scale MoE deployments.

major comments (2)
  1. [Experiments] Experiments section: the reported accuracy and FLOP numbers on Qwen3-30B-A3B and GLM-4.7-Flash are given without error bars, standard deviations, or results from multiple random seeds, which is required to assess whether the 6.1- and 4.0-point gains over the strongest baseline and the claimed marginal accuracy loss are statistically reliable.
  2. [Methods] Methods section: the precise interaction between the group-level balancing loss and the two-stage distillation objective (including loss coefficients and scheduling) is not specified in sufficient detail to allow reproduction of the claimed stable routing behavior.
minor comments (1)
  1. [Abstract] Abstract: the 11 benchmarks are not enumerated; listing the concrete tasks would improve clarity without lengthening the abstract.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the positive evaluation and recommendation for minor revision. The comments highlight important aspects of experimental reporting and methodological clarity, which we address point by point below. We will revise the manuscript accordingly to improve reproducibility and transparency.

read point-by-point responses

  1. Referee: [Experiments] Experiments section: the reported accuracy and FLOP numbers on Qwen3-30B-A3B and GLM-4.7-Flash are given without error bars, standard deviations, or results from multiple random seeds, which is required to assess whether the 6.1- and 4.0-point gains over the strongest baseline and the claimed marginal accuracy loss are statistically reliable.

    Authors: We agree that the absence of error bars or multi-seed results limits the ability to assess statistical reliability, particularly for the reported gains and marginal accuracy drops. Due to the high computational cost of full training and evaluation on 30B+ parameter models, experiments were performed with a single random seed. In the revised manuscript we will add an explicit statement in the Experiments section acknowledging this limitation, noting that consistent trends were observed across two distinct base models and 11 benchmarks. We will also consider adding multi-seed results on a smaller proxy model if space permits. revision: partial

  2. Referee: [Methods] Methods section: the precise interaction between the group-level balancing loss and the two-stage distillation objective (including loss coefficients and scheduling) is not specified in sufficient detail to allow reproduction of the claimed stable routing behavior.

    Authors: We thank the referee for pointing out this gap in reproducibility. The current manuscript describes the overall two-stage self-distillation and group-level balancing loss at a high level but omits the exact coefficients, stage durations, and their combined scheduling. In the revised Methods section we will provide the precise loss weights (e.g., distillation loss coefficient λ_dist and balancing loss coefficient λ_bal), the number of steps per stage, and the scheduling rule that governs when each term is active, thereby enabling exact reproduction of the reported routing stability. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper introduces an empirical adaptation method (ZEDA) consisting of zero-output expert injection followed by two-stage self-distillation on a frozen teacher, then reports accuracy and FLOP numbers on held-out benchmarks across 11 tasks. No equations, uniqueness theorems, or self-citations are invoked to derive the performance claims; the central results are measured outcomes rather than quantities forced by construction from the training procedure or prior author work. The derivation chain is therefore self-contained against external evaluation.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no equations or implementation details; therefore no free parameters, axioms, or invented entities can be extracted beyond the high-level description of zero-output experts.

pith-pipeline@v0.9.1-grok · 5807 in / 1233 out tokens · 24023 ms · 2026-06-30T18:20:36.304628+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 36 canonical work pages · 20 internal anchors

  1. [1]

    GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computa- tion and automatic sharding.arXiv preprint arXiv:2006.16668,

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  2. [2]

    Mixtral of Experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    arXiv preprint arXiv:2410.07348 (2024)

    Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan. Moe++: Accelerating mixture-of-experts methods with zero-computation experts.arXiv preprint arXiv:2410.07348,

    work page arXiv

  4. [4]

    arXiv preprint arXiv:2509.01322 , year=

    Meituan LongCat Team, Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang, Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, et al. Longcat-flash technical report.arXiv preprint arXiv:2509.01322,

    work page arXiv

  5. [5]

    Grove moe: Towards efficient and superior moe llms with adjugate experts.arXiv preprint arXiv:2508.07785,

    Haoyuan Wu, Haoxing Chen, Xiaodong Chen, Zhanchao Zhou, Tieyuan Chen, Yihong Zhuang, Guoshan Lu, Zenan Huang, Junbo Zhao, Lin Liu, et al. Grove moe: Towards efficient and superior moe llms with adjugate experts.arXiv preprint arXiv:2508.07785,

    work page arXiv

  6. [6]

    arXiv preprint arXiv:2405.14297 (2024)

    Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu, and Tao Lin. Dynamic mixture of experts: An auto-tuning approach for efficient transformer models.arXiv preprint arXiv:2405.14297,

    work page arXiv

  7. [7]

    Moe lens–an expert is all you need.arXiv preprint arXiv:2603.05806,

    Marmik Chaudhari, Idhant Gulati, Nishkal Hundia, Pranav Karra, and Shivam Raval. Moe lens–an expert is all you need.arXiv preprint arXiv:2603.05806,

    work page arXiv

  8. [8]

    Adamoe: Token-adaptive routing with null experts for mixture-of-experts language models

    Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, and Zhijie Deng. Adamoe: Token-adaptive routing with null experts for mixture-of-experts language models. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 6223–6235,

    2024

  9. [9]

    GLM-5: from Vibe Coding to Agentic Engineering

    Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    MiniLLM: On-Policy Distillation of Large Language Models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models.arXiv preprint arXiv:2306.08543,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Qwen3 Technical Report

    doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.64434/tml.20251026

  12. [12]

    GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models

    Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    16 Post-Trained MoE Can Skip Half Experts via Self-Distillation Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Instruction-Following Evaluation for Large Language Models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Generalizing Verifiable Instruction Following

    Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following.arXiv preprint arXiv:2507.02833,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    arXiv preprint arXiv:2506.13284 , year =

    Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-nemotron 1.1: Advancing math and code reasoning through sft and rl synergy. arXiv preprint arXiv:2506.13284,

    work page arXiv

  18. [18]

    Llama-nemotron: Efficient reasoning models

    Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al. Llama-nemotron: Efficient reasoning models. arXiv preprint arXiv:2505.00949,

    work page arXiv

  19. [19]

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053,

    work page internal anchor Pith review Pith/arXiv arXiv 1909

  20. [20]

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and 1 others

    Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling reasoning efficiently via relaxed on-policy distillation.arXiv preprint arXiv:2603.11137,

    work page arXiv

  21. [21]

    Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning.arXiv preprint arXiv:2506.01939,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologi...

    2025

  23. [23]

    GPQA: A Graduate-Level Google-Proof Q&A Benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022,

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

    17 Post-Trained MoE Can Skip Half Experts via Self-Distillation Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538,

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [25]

    ST-MoE: Designing Stable and Transferable Sparse Expert Models

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models.arXiv preprint arXiv:2202.08906,

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2407.00945 , year=

    Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs.arXiv preprint arXiv:2407.00945,

    work page arXiv

  27. [27]

    arXiv preprint arXiv:2310.01334 , year=

    Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy.arXiv preprint arXiv:2310.01334,

    work page arXiv

  28. [28]

    arXiv preprint arXiv:2410.08589 , year=

    I Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, Chun-Yi Lee, et al. Retraining- free merging of sparse moe via hierarchical clustering.arXiv preprint arXiv:2410.08589,

    work page arXiv

  29. [29]

    Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts

    Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts. InFindings of the Association for Computational Linguistics: ACL 2025, pages 86–102,

    2025

  30. [30]

    Lightmoe: Reducing mixture-of-experts redundancy through expert replacing.arXiv preprint arXiv:2603.12645,

    JiaweiHao,ZhiweiHao,JianyuanGuo,LiShen,YongLuo,HanHu,andDanZeng. Lightmoe: Reducing mixture-of-experts redundancy through expert replacing.arXiv preprint arXiv:2603.12645,

    work page arXiv

  31. [31]

    Expert threshold routing for autoregressive language modeling with dynamic computation allocation and load balancing.arXiv preprint arXiv:2603.11535,

    Hanchi Sun, Yixin Liu, Yonghui Wu, and Lichao Sun. Expert threshold routing for autoregressive language modeling with dynamic computation allocation and load balancing.arXiv preprint arXiv:2603.11535,

    work page arXiv

  32. [32]

    Distilling the Knowledge in a Neural Network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    work page internal anchor Pith review Pith/arXiv arXiv

  33. [33]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. InProceedings of the 2016 conference on empirical methods in natural language processing, pages 1317–1327,

    2016

  34. [34]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  35. [35]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. InFindings of the Association for Computa- tional Linguistics: ACL 2023, pages 8003–8017,

    2023

  36. [36]

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734,

    work page internal anchor Pith review Pith/arXiv arXiv

  37. [37]

    Self-Distillation Enables Continual Learning

    Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897,

    work page internal anchor Pith review Pith/arXiv arXiv

  38. [38]

    Reinforcement Learning via Self-Distillation

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802,

    work page internal anchor Pith review Pith/arXiv arXiv

  39. [39]

    Rad: Redundancy- aware distillation for hybrid models via self-speculative decoding.arXiv preprint arXiv:2505.22135,

    Yuichiro Hoshino, Hideyuki Tachibana, Muneyoshi Inahara, and Hiroto Takegawa. Rad: Redundancy- aware distillation for hybrid models via self-speculative decoding.arXiv preprint arXiv:2505.22135,

    work page arXiv

  40. [40]

    Hybrid linear attention done right: Efficient distillation and effective architectures for extremely long contexts, 2026

    Yingfa Chen, Zhen Leng Thai, Zihan Zhou, Zhu Zhang, Xingyu Shen, Shuo Wang, Chaojun Xiao, Xu Han, and Zhiyuan Liu. Hybrid linear attention done right: Efficient distillation and effective architectures for extremely long contexts.arXiv preprint arXiv:2601.22156,

    work page arXiv

  41. [41]

    Ladimo: Layer-wise distillation inspired moefier.arXiv preprint arXiv:2408.04278,

    Sungyoon Kim, Youngjun Kim, Kihyo Moon, and Minsung Jang. Ladimo: Layer-wise distillation inspired moefier.arXiv preprint arXiv:2408.04278,

    work page arXiv

  42. [42]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901,

    2023

  43. [43]

    Nevertheless, even at8𝑘, a commonly used long-context setting, ZEDA still achieves approximately 20% speedup, demonstrating its practical usability

    The speedup gradually diminishes as sequence length increases. Nevertheless, even at8𝑘, a commonly used long-context setting, ZEDA still achieves approximately 20% speedup, demonstrating its practical usability. Furthermore, ZEDA exhibits greater potential for advanced communication frameworks like DeepEP [Zhao et al., 2025], which we aim to integrate in ...

    2025

  44. [44]

    Shared MoE Cost Decomposition The MoE FFN and router costs have the same form in both stages; the only difference is the number of tokens processed in the current forward pass

    D.1. Shared MoE Cost Decomposition The MoE FFN and router costs have the same form in both stages; the only difference is the number of tokens processed in the current forward pass. Let𝑛 denote that token count. For the original 22 Post-Trained MoE Can Skip Half Experts via Self-Distillation Table 11|Notation used in the theoretical FLOP analysis. Symbol ...

    2023

  45. [45]

    Table 12|Architectural parameters of Qwen3-30B-A3B used in the FLOP analysis. Symbol𝐻 𝐻 attn 𝑔kv 𝐻𝑒 𝑁 𝑁 𝑍 𝐾 Value2048 4096 1/8 768 128 64 8 To facilitate direct comparison with empirical measurements, we convert the FLOP ratios in Equa- tions (14) and (18) into theoretical speedups by taking their reciprocals. Table 13 reports the resulting prefill and de...

    2048

  46. [46]

    24 Post-Trained MoE Can Skip Half Experts via Self-Distillation Table 13| Comparison between theoretical speedups derived from the FLOP analysis and measured empirical speedups on Qwen3-30B-A3B across different sequence lengths. Length Prefill Speedup Decode Speedup Theoretical Empirical Theoretical Empirical 1024 1.403x 1.141x 1.443x 1.233x 2048 1.341x 1...

    2048