Pith. sign in

REVIEW 2 major objections 1 cited by

Looped-MoE models scale better than standard transformers because different experts activate on each loop pass.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-01 07:24 UTC pith:R3NPBOX7

load-bearing objection Looped-MoE models show better scaling than dense looped ones in the reported experiments, but the paper does not isolate routing divergence as the cause. the 2 major comments →

arxiv 2605.09165 v2 pith:R3NPBOX7 submitted 2026-05-09 cs.LG cs.CL

Sparse Layers are Critical to Scaling Looped Language Models

Ryan Lee , Jacob Biloki , Edward J. Hu , Jonathan May This is my paper
classification cs.LG cs.CL
keywords looped language modelsmixture of expertsmodel scalingearly exitrouting divergenceparameter efficiencytransformer layers
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Looped language models reuse the same transformer layers across depth to cut memory use and create natural early-exit points. Plain dense looped models fail to scale as favorably as standard transformers with unique layers. Adding mixture-of-experts sparsity fixes this by producing routing divergence: each pass through the shared layers activates different experts, which restores expressivity without adding parameters. The same loop boundaries also serve as stronger early-exit locations than arbitrary points in standard models because outputs converge earlier there. These two properties together let a looped MoE model surpass standard transformers at scale while cutting memory and inference cost with little quality loss.

Core claim

Looped-MoE models scale better than the standard baseline while dense looped models do not. The performance gap traces to routing divergence between loops: in Looped-MoE models, different experts are activated on each pass through the same shared layers, recovering expressivity without additional parameters. Looped models also deliver better compute-quality trade-offs with early exits than standard models because each loop ends with the same layers that produce the final output, making loop boundaries superior exit points confirmed by earlier output convergence.

What carries the argument

Routing divergence between loops in Looped-MoE models, where distinct experts are selected on successive passes through the shared layers.

Load-bearing premise

The observed scaling advantage of Looped-MoE over dense looped models is driven primarily by routing divergence rather than other unexamined differences in training dynamics or model capacity.

What would settle it

A controlled comparison in which expert selection is forced to be identical across loops in a Looped-MoE model, resulting in scaling curves that match those of dense looped models rather than standard transformers.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Looped-MoE models can exceed standard transformers in scaling performance.
  • Early exits at loop boundaries yield superior compute-quality curves compared with standard models.
  • Memory and inference costs drop substantially with minimal quality degradation when using looped MoE plus early exits.
  • Dense looped models remain inferior in scaling to models that vary layers with depth.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same sparsity pattern may help other parameter-sharing architectures that repeat layers.
  • Early-exit benefits at loop boundaries could extend to non-transformer looped designs if routing divergence is introduced.
  • The interaction between looping and expert choice suggests that sparsity may be broadly necessary for repeated-layer scaling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper compares standard and MoE transformers with and without layer looping. It claims that Looped-MoE models scale better than both standard transformers and dense looped models because routing divergence activates different experts on successive passes through the shared layers, recovering expressivity without extra parameters. It further claims that looped models provide superior compute-quality trade-offs via early exits at loop boundaries due to earlier output convergence.

Significance. If the scaling and early-exit results hold after proper controls, the work identifies a concrete architectural direction for memory-efficient scaling of looped transformers that could reduce inference costs while matching or exceeding dense baselines. The emphasis on routing divergence as the operative mechanism, if isolated, would also inform MoE design in recurrent settings.

major comments (2)
  1. [Abstract] The central attribution of the scaling advantage to routing divergence (Abstract) is load-bearing yet unsupported by an isolating ablation: no experiment is described that holds total active parameters, router architecture, and per-token compute fixed while suppressing cross-loop expert variation (e.g., by tying router weights across iterations or forcing identical top-k selections).
  2. [Abstract] The claim that Looped-MoE models beat standard transformers at scale (Abstract) cannot be evaluated because the manuscript provides no model sizes, dataset descriptions, training details, or statistical evidence; the reader's assessment of soundness is therefore limited to 3.0.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback identifying areas where additional evidence and detail would strengthen the claims. We address each major comment below and commit to revisions that directly respond to the concerns raised.

read point-by-point responses

  1. Referee: [Abstract] The central attribution of the scaling advantage to routing divergence (Abstract) is load-bearing yet unsupported by an isolating ablation: no experiment is described that holds total active parameters, router architecture, and per-token compute fixed while suppressing cross-loop expert variation (e.g., by tying router weights across iterations or forcing identical top-k selections).

    Authors: We agree that an explicit isolating ablation would provide stronger causal evidence for routing divergence as the mechanism. The current results show the scaling benefit only emerges in the Looped-MoE case (not dense looped), and we report observed differences in expert activation across loops. To address the gap, the revised manuscript will include a new controlled ablation that ties router weights across loop iterations (or forces identical top-k selections) while matching total active parameters, router architecture, and per-token compute exactly. This will directly test whether suppressing cross-loop variation eliminates the advantage. revision: yes

  2. Referee: [Abstract] The claim that Looped-MoE models beat standard transformers at scale (Abstract) cannot be evaluated because the manuscript provides no model sizes, dataset descriptions, training details, or statistical evidence; the reader's assessment of soundness is therefore limited to 3.0.

    Authors: We acknowledge that the manuscript as submitted does not present these details with sufficient prominence or completeness for independent evaluation of the scaling claims. The revision will expand the experimental section and add a dedicated appendix containing exact model sizes and configurations, full dataset descriptions, training hyperparameters, number of random seeds, and statistical evidence (including error bars and significance tests) so that the abstract claims can be rigorously assessed. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical comparisons with interpretive attribution

full rationale

The manuscript reports experimental scaling results across standard, dense-looped, and Looped-MoE transformers, then offers an interpretive explanation for the observed performance gap. No equations, derivations, fitted parameters renamed as predictions, or self-citation chains appear in the provided text. The statement that routing divergence 'recovers expressivity' is presented as a post-hoc tracing of empirical outcomes rather than a definitional or self-referential reduction. The paper therefore contains no load-bearing step that collapses to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no free parameters, axioms, or invented entities are identifiable from the provided text.

pith-pipeline@v0.9.1-grok · 5725 in / 1074 out tokens · 33430 ms · 2026-07-01T07:24:30.623757+00:00 · methodology

0 comments
read the original abstract

Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as favorably as standard transformers with unique layers. We compare standard and Mixture-of-Experts (MoE) transformers, with and without looping, and find two main results. First, we find Looped-MoE models scale better than the standard baseline while dense looped models do not. We trace this to routing divergence between loops: in Looped-MoE models, different experts are activated on each pass through the same shared layers, recovering expressivity without additional parameters. Our second finding is that looped models have better compute-quality trade-offs with early exits than standard models. Because each loop ends with the same layers that produce the final output, loop boundaries are superior exit points, as confirmed by earlier output convergence at these points. In sum, we provide a clear direction for scaling looped models: a Looped-MoE model with early exits can not only beat standard transformers at scale, but also enable significant memory and inference savings with minimal degradation in quality.

Figures

Figures reproduced from arXiv: 2605.09165 by Edward J. Hu, Jacob Biloki, Jonathan May, Ryan Lee.

Figure 1
Figure 1. Figure 1: Overview of our models and main results. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: µP Transfer test. The best learning rate for the smallest model remains optimal across larger sizes, validating our µP implementation. If the optimal learning rate for a given width does differ from the base learning rate, the loss difference is < 1% [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Left: IsoFLOP curves for Base. Right: IsoFLOP curves for Looped-MoE. Stars mark compute-optimal model sizes at each budget; solid lines show fitted L ∝ N −α (α = 0.076 for Base, 0.077 for Looped-MoE). The dashed line shows Kaplan et al. [4] scaling exponent (α = 0.076) [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Left: Schematic of entropy-based early exit. For Looped/Looped-MoE, tokens exit at loop boundaries. Right: Looped/Looped-MoE models have the best compute-quality trade-offs. 6 Analysis In this section, we conduct experiments to understand why replacing the dense FFN of a looped transformer with a MoE layer results in better scaling laws and early-exit trade-offs. At a high level we find the reasons are: (1… view at source ↗
Figure 5
Figure 5. Figure 5: Expert assignment overlap between loop passes 1 and 2 across physical layers in a Looped [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distributional analysis of intermediate layer outputs relative to the final layer output (JSD), [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Left: More loops improve the Looped compute-quality tradeoff, though not strictly at all savings levels. Right: For Looped-MoE, more loops yield a strictly better compute-quality tradeoff, with all configurations better than non-looped MoE. Looped models converge faster at loop boundaries [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The pre-norm transformer layer, which we use in this study across all models. For our [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Left: IsoFLOP curves for Looped. Right: IsoFLOP curves for MoE. Power law fit to compute budgets from 5 × 1016 to 1018 FLOPs. Dashed lines show fitted power-law scaling relations. Kaplan et al. scaling exponent is shown in dotted line [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Combined scaling laws for all models in the study. Lower test loss is better. All [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models

    cs.LG 2026-06 unverdicted novelty 5.0

    Dense per-loop cross-entropy in looped transformers fails to control hidden-state scale with scale-invariant readouts like RMSNorm, driving norms to thousands, while scale-visible readouts or norm penalties keep norms...

Reference graph

Works this paper leans on

34 extracted references · 27 canonical work pages · cited by 1 Pith paper · 14 internal anchors

  1. [1]

    Universal Transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal Transformers, March 2019. URL http://arxiv.org/abs/1807.03819. arXiv:1807.03819 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  2. [2]

    Scaling Latent Reasoning via Looped Language Models

    Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, Lu Li, Jiajun Shi, Kaijing Ma, Shanda Li, Taylor Kergan, Andrew Smith, Xingwei Qu, Mude Hui, Bohong Wu, Qiyang Min, Hongzhi Huang, Xun Zhou, Wei Ye, Jiaheng Liu, Jian Yang, Yunfeng Shi, Chenghua Lin, Enduo Zhao, Tianle Cai, Ge Zhang, Wenhao Huang,...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  3. [3]

    Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bar- toldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach, February 2025. URL http://arxiv.org/abs/2502.05171. arXiv:2502.05171 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    Scaling Laws for Neural Language Models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models, January 2020. URL http://arxiv.org/abs/2001.08361. arXiv:2001.08361 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  5. [5]

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, January 2017. URLhttp://arxiv.org/abs/1701.06538. arXiv:1701.06538 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  6. [6]

    MoEUT: Mixture-of-Experts Universal Transformers

    Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber, Christopher Potts, and Christopher D Manning. MoEUT: Mixture-of-Experts Universal Transformers

  7. [7]

    Surat Teerapittayanon, Bradley McDanel, and H.T. Kung. BranchyNet: Fast inference via early exiting from deep neural networks. In2016 23rd International Conference on Pattern Recognition (ICPR), pages 2464–2469, Cancun, December 2016. IEEE. ISBN 978-1-5090-4847-

    2016

  8. [8]

    URL http://ieeexplore.ieee.org/document/ 7900006/

    doi: 10.1109/ICPR.2016.7900006. URL http://ieeexplore.ieee.org/document/ 7900006/

    work page doi:10.1109/icpr.2016.7900006 2016

  9. [9]

    Deebert: Dynamic early exiting for accelerating bert inference

    Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference, April 2020. URL http://arxiv.org/abs/2004. 12993. arXiv:2004.12993 [cs]

    work page arXiv 2020

  10. [10]

    Tran, Yi Tay, and Donald Metzler

    Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident Adaptive Language Modeling, October 2022. URL http: //arxiv.org/abs/2207.07061. arXiv:2207.07061 [cs]

    work page arXiv 2022

  11. [11]

    RoFormer: Enhanced Transformer with Rotary Position Embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding, November 2023. URL http:// arxiv.org/abs/2104.09864. arXiv:2104.09864 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  12. [12]

    GLU Variants Improve Transformer

    Noam Shazeer. GLU Variants Improve Transformer, February 2020. URL http://arxiv. org/abs/2002.05202. arXiv:2002.05202 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  13. [13]

    Root Mean Square Layer Normalization

    Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ 1e8a19426224ca89e83cef47f1e7f53b-Abstract.html

    2019

  14. [14]

    Switch transformers: scaling to trillion parameter models with simple and efficient sparsity.J

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity.J. Mach. Learn. Res., 23(1):120:5232– 120:5270, January 2022. ISSN 1532-4435. URL https://dl.acm.org/doi/10.5555/ 3586589.3586709

    work page arXiv 2022

  15. [15]

    ST-MoE: Designing Stable and Transferable Sparse Expert Models, April

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. ST-MoE: Designing Stable and Transferable Sparse Expert Models, April

  16. [16]

    ST-MoE: Designing Stable and Transferable Sparse Expert Models

    URLhttp://arxiv.org/abs/2202.08906. arXiv:2202.08906 [cs]. 10

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer, March 2022. URL http: //arxiv.org/abs/2203.03466. arXiv:2203.03466 [cs]

    work page arXiv 2022

  19. [19]

    \( \) -parametrization for mixture of experts

    Jan Mała´snicki, Kamil Ciebiera, Mateusz Boru ´n, Maciej Pióro, Jan Ludziejewski, Maciej Stefaniak, Michał Krutul, Sebastian Jaszczur, Marek Cygan, Kamil Adamczewski, and Jakub Krajewski. µ-parametrization for mixture of experts, October 2025. URL http://arxiv. org/abs/2508.09752. arXiv:2508.09752 [cs]

    work page arXiv 2025

  20. [20]

    Training Compute-Optimal Large Language Models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  21. [21]

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

    Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, and Thomas Wolf. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, October 2024. URL http://arxiv.org/abs/2406.17557. arXiv:2406.17557 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [22]

    MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. MiniCPM: Unveiling the Potential of Small Langu...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler, August 2025

    Aleksandr Dremov, Alexander Hägele, Atli Kosson, and Martin Jaggi. Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler, August 2025. URL http://arxiv.org/abs/2508.01483. arXiv:2508.01483 [cs]

    work page arXiv 2025

  24. [24]

    2025 , url=

    Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. OLMES: A Standard for Language Model Evaluations, February 2025. URL http://arxiv. org/abs/2406.08446. arXiv:2406.08446 [cs]

    work page arXiv 2025

  25. [25]

    Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, and Daniel Y . Fu. Parcae: Scaling Laws For Stable Looped Language Models, 2026. URL https://arxiv.org/abs/2604. 12946. Version Number: 1

    2026

  26. [26]

    interpreting GPT: the logit lens — LessWrong

    nostalgebraist. interpreting GPT: the logit lens — LessWrong. August

  27. [27]

    URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens

  28. [28]

    Approximating Two-Layer Feedforward Networks for Efficient Transformers, November 2023

    Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. Approximating Two-Layer Feedforward Networks for Efficient Transformers, November 2023. URL http://arxiv.org/abs/2310. 10837. arXiv:2310.10837 [cs]

    work page arXiv 2023

  29. [29]

    SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention, September 2024

    Róbert Csordás, Piotr Pi˛ ekos, Kazuki Irie, and Jürgen Schmidhuber. SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention, September 2024. URL http://arxiv.org/ abs/2312.07987. arXiv:2312.07987 [cs]

    work page arXiv 2024

  30. [30]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization, July 2016. URLhttp://arxiv.org/abs/1607.06450. arXiv:1607.06450 [stat]. 11

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  31. [31]

    Aly, Beidi Chen, and Carole-Jean Wu

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A. Aly, Beidi Chen, and Carole-Jean Wu. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...

    work page doi:10.18653/v1/2024.acl-long.681 2024

  32. [32]

    Mixture-of-Depths: Dynamically allocating compute in transformer-based language models

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-Depths: Dynamically allocating compute in transformer-based lan- guage models, April 2024. URL http://arxiv.org/abs/2404.02258. arXiv:2404.02258 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  33. [33]

    Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, and Se-Young Yun

    Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, and Se-Young Yun. Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation, October 2025. URLhttp://arxiv.org/abs/2507.10524. arXiv:2507.10524 [cs]

    work page arXiv 2025

  34. [34]

    Don't be lazy: CompleteP enables compute-efficient deep transformers

    Nolan Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don’t be lazy: CompleteP enables compute- efficient deep transformers, January 2026. URL http://arxiv.org/abs/2505.01618. arXiv:2505.01618 [cs]. 12 A Appendix Compute.All experiments were run on NVIDIA H100 GPUs (80GB). Appr...

    work page arXiv 2026