pith. sign in

arxiv: 2605.06152 · v3 · pith:L7NYTNSTnew · submitted 2026-05-07 · 💻 cs.LG · cs.CL· math.OC· stat.ML

Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes

Liu Hanqing , Jianjun Cao , Yuanze Li , Zijian Zhou This is my paper

Pith reviewed 2026-06-30 23:31 UTC · model grok-4.3

classification 💻 cs.LG cs.CLmath.OCstat.ML
keywords slingshot mechanismfloating-point precisionloss spikesnumerical feature inflationdeep neural networksbackpropagationclassifier driftfinite precision training
0
0 comments X

The pith

Floating-point precision limits cause slingshot loss spikes by zeroing correct-class gradients and creating exponential drift.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that the slingshot mechanism of periodic loss spikes arises from floating-point arithmetic limits rather than intrinsic optimization dynamics. Once logit differences exceed the absorption threshold in high-confidence training, the correct-class gradient rounds to zero during backpropagation while others remain nonzero, breaking the zero-sum constraint on classifier updates. This introduces a systematic drift that enters a positive feedback loop with the features, producing exponential growth in both the global classifier mean and global feature mean through Numerical Feature Inflation. A sympathetic reader would care because the account explains the observed rapid norm growth, gradient reappearance, and loss spike as direct consequences of finite precision, while also showing that the same process can drive parameter growth without visible spikes in practical settings.

Core claim

This paper proves that the slingshot phenomenon is a result of floating-point arithmetic precision limits. As training enters a high-confidence stage, the difference between the correct-class logit and the other logits may exceed the absorption-error threshold. Then during backpropagation, the gradient of the correct class is rounded exactly to zero, while the gradients of the incorrect classes remain nonzero. This breaks the zero-sum constraint of gradients across classes and introduces a systematic drift in the parameter update of the classifier layer. The drift forms a positive feedback loop with the feature, causing the global classifier mean and the global feature mean to grow exponenti

What carries the argument

Numerical Feature Inflation (NFI): the positive feedback loop in which absorption errors zero the correct-class gradient, break the zero-sum constraint, and drive exponential growth of classifier and feature means.

If this is right

  • NFI accounts for the rapid norm growth observed before each slingshot spike.
  • The subsequent reappearance of gradients and the loss spike follow directly from the exponential growth phase.
  • Partial absorption in practical tasks can still break the zero-sum constraint and drive rapid parameter-norm growth without producing visible spikes.
  • Slingshot is reinterpreted as a numerical dynamic of finite-precision training rather than an optimization phenomenon.
  • The same process supplies a testable explanation for abnormal parameter growth and logit divergence in late-stage training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Using higher-precision floating-point formats throughout training would be expected to suppress or eliminate the spikes if absorption is the root cause.
  • Regularization that keeps logit differences below the absorption threshold could prevent NFI even in standard precision.
  • The same drift mechanism might contribute to other forms of late-training instability that involve growing classifier norms.

Load-bearing premise

As training enters a high-confidence stage, the difference between the correct-class logit and the other logits may exceed the absorption-error threshold.

What would settle it

An experiment that trains the same models in higher-precision arithmetic (such as FP64) and finds that slingshot spikes and the associated exponential norm growth still occur would falsify the claim that absorption errors are the triggering cause.

Figures

Figures reproduced from arXiv: 2605.06152 by Jianjun Cao, Liu Hanqing, Yuanze Li, Zijian Zhou.

Figure 1
Figure 1. Figure 1: Precision-induced N FI dynamics. (a) Slingshot loss spikes disappear when training is performed in float64. Casting only the logits/loss computation to float64 is also sufficient to remove the spikes, showing that the instability originates from the loss computation. (b) Before most samples enter Softmax Collapse, the global feature mean grows slowly. Once most samples collapse, ∥µG∥ enters a rapid-growth … view at source ↗
Figure 2
Figure 2. Figure 2: Mechanistic evidence for Slingshot spikes. (a) Distribution of classifier-layer parameter updates around a loss spike. Before the spike, updates concentrate near zero. At the spike step, the distribution forms two sharp modes near −4 × 10−4 and 4 × 10−4 . After the spike, update magnitudes become dispersed across parameters. (b) Evolution of the residual probability mass ϵ across architectures. Before a sp… view at source ↗
Figure 3
Figure 3. Figure 3: Mitigation Study. (a) Adam’s ε. Increasing the optimizer’s ε parameter mitigates instability: while ε = 10−6 reduces the frequency of spikes, setting ε = 10−5 completely eliminates them. (b) Layer Norm. Applying LN changes the evolution of the last layer norm from a continuous trajectory to a distinct stepwise pattern. Notably, LN significantly increases the magnitude of the last layer norm. (c) Batch Norm… view at source ↗
Figure 3
Figure 3. Figure 3: Mitigation Study. (a) Adam’s ε. Increasing the optimizer’s ε parameter mitigates instability: while ε = 10−6 reduces the frequency of spikes, setting ε = 10−5 completely eliminates them. (b) Layer Norm. Applying LN changes the evolution of the last layer norm from a continuous trajectory to a distinct stepwise pattern. Notably, LN significantly increases the magnitude of the last layer norm. (c) Batch Norm… view at source ↗
Figure 4
Figure 4. Figure 4: (a) Training Loss with different learning rates. (b) Train Loss with label smoothing. A.1.3 Bias We find that including a bias term in the classification layer accelerates the occurrence of Slingshots. This instability stems from a significant scale discrepancy between the updates of weights and biases. The gradient with respect to the weight Wk is scaled by the feature vector h: ∇Wk L = (ˆyk − yk)h (9) In… view at source ↗
Figure 4
Figure 4. Figure 4: (a) Training Loss with different learning rates. (b) Train Loss with label smoothing. A.1.3 Bias We find that including a bias term in the classification layer accelerates the occurrence of Slingshots. This instability stems from a significant scale discrepancy between the updates of weights and biases. The gradient with respect to the weight Wk is scaled by the feature vector h: ∇Wk L = (ˆyk − yk)h (9) In… view at source ↗
Figure 5
Figure 5. Figure 5: Coexistence of EOS and Slingshot Phenomena. (a) Training Loss showing early EOS oscillations versus late-stage numerical spikes. (b) Evolution of Maximum Hessian Eigenvalue λmax. Cohen et al. [3] identified the phenomenon of “Progressive Sharpening” in neural network training, where the maximum eigenvalue of the Hessian, λmax, steadily increases until it reaches the stability threshold 2/η. Upon breaching … view at source ↗
Figure 5
Figure 5. Figure 5: Coexistence of EOS and Slingshot Phenomena. (a) Training Loss showing early EOS oscillations versus late-stage numerical spikes. (b) Evolution of Maximum Hessian Eigenvalue λmax. Cohen et al. [3] identified the phenomenon of “Progressive Sharpening” in neural network training, where the maximum eigenvalue of the Hessian, λmax, steadily increases until it reaches the stability threshold 2/η. Upon breaching … view at source ↗
Figure 6
Figure 6. Figure 6: Slingshot in Transformer on modular division. view at source ↗
Figure 6
Figure 6. Figure 6: Slingshot in Transformer on modular division. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Slingshot in MLP on modular division. 17 view at source ↗
Figure 7
Figure 7. Figure 7: Slingshot in MLP on modular division. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Slingshot in MLP on CIFAR-10. 0 2000 4000 6000 8000 10000 Epoch 10 8 10 6 10 4 10 2 10 0 10 2 Train Loss (Log) 0 2000 4000 6000 8000 10000 Epoch 200 150 100 50 0 50 Average Target Logit view at source ↗
Figure 8
Figure 8. Figure 8: Slingshot in MLP on CIFAR-10. 0 2000 4000 6000 8000 10000 Epoch 10 8 10 6 10 4 10 2 10 0 10 2 Train Loss (Log) 0 2000 4000 6000 8000 10000 Epoch 200 150 100 50 0 50 Average Target Logit [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Slingshot in VGG11 on CIFAR-10. 0 2000 4000 6000 8000 10000 Epoch 10 8 10 6 10 4 10 2 10 0 Train Loss (Log) 0 2000 4000 6000 8000 10000 Epoch 1200 1000 800 600 400 200 0 Average Target Logit view at source ↗
Figure 9
Figure 9. Figure 9: Slingshot in VGG11 on CIFAR-10. 0 2000 4000 6000 8000 10000 Epoch 10 8 10 6 10 4 10 2 10 0 Train Loss (Log) 0 2000 4000 6000 8000 10000 Epoch 1200 1000 800 600 400 200 0 Average Target Logit [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Slingshot in VGG11 with BN on CIFAR-10. 0 2000 4000 6000 8000 10000 Epoch 10 8 10 6 10 4 10 2 10 0 Train Loss (Log) 0 2000 4000 6000 8000 10000 Epoch 600 500 400 300 200 100 0 Average Target Logit view at source ↗
Figure 10
Figure 10. Figure 10: Slingshot in VGG11 with BN on CIFAR-10. 0 2000 4000 6000 8000 10000 Epoch 10 8 10 6 10 4 10 2 10 0 Train Loss (Log) 0 2000 4000 6000 8000 10000 Epoch 600 500 400 300 200 100 0 Average Target Logit [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Slingshot in ViT on CIFAR-10. 18 view at source ↗
Figure 11
Figure 11. Figure 11: Slingshot in ViT on CIFAR-10. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: No Slingshot in ResNet18 on CIFAR-10. 0 20000 40000 60000 80000 100000 Steps 0 100 200 300 400 500 Average Logit FP32 FP64 view at source ↗
Figure 12
Figure 12. Figure 12: No Slingshot in ResNet18 on CIFAR-10. 0 20000 40000 60000 80000 100000 Steps 0 100 200 300 400 500 Average Logit FP32 FP64 [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: The average logit of different precisions in LLM training. view at source ↗
Figure 13
Figure 13. Figure 13: The average logit of different precisions in LLM training. [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
read the original abstract

Deep neural networks exhibit periodic loss spikes during unregularized long-term training, a phenomenon known as the "Slingshot Mechanism." Existing work usually attributes this to intrinsic optimization dynamics, but its triggering mechanism remains unclear. This paper proves that this phenomenon is a result of floating-point arithmetic precision limits. As training enters a high-confidence stage, the difference between the correct-class logit and the other logits may exceed the absorption-error threshold. Then during backpropagation, the gradient of the correct class is rounded exactly to zero, while the gradients of the incorrect classes remain nonzero. This breaks the zero-sum constraint of gradients across classes and introduces a systematic drift in the parameter update of the classifier layer. We prove that this drift forms a positive feedback loop with the feature, causing the global classifier mean and the global feature mean to grow exponentially. We call this mechanism Numerical Feature Inflation (NFI). This mechanism explains the rapid norm growth before a Slingshot spike, the subsequent reappearance of gradients, and the resulting loss spike. We further show that NFI is not equivalent to an observed loss spike: in more practical tasks, partial absorption may not produce visible spikes, but it can still break the zero-sum constraint and drive rapid growth of parameter norms. Our results reinterpret Slingshot as a numerical dynamic of finite-precision training, and provide a testable explanation for abnormal parameter growth and logit divergence in late-stage training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims that the 'Slingshot Mechanism' of periodic loss spikes in unregularized long-term DNN training arises from floating-point precision limits rather than intrinsic optimization. It asserts that once logit differences exceed an absorption threshold in high-confidence regimes, correct-class gradients round exactly to zero while others remain nonzero, violating the zero-sum constraint on classifier gradients; this drift then enters a positive feedback loop (Numerical Feature Inflation, NFI) that produces exponential growth in global classifier and feature means, explaining pre-spike norm growth, gradient reappearance, and the loss spike itself. The work further claims NFI can drive norm growth without visible spikes in practical tasks.

Significance. If the mechanism is derived and verified, the result would supply a concrete, testable numerical account for rapid parameter-norm growth and logit divergence in late-stage training, reinterpreting slingshot events as finite-precision artifacts and separating the underlying drift from observable spikes.

major comments (2)
  1. [Abstract] Abstract: the claim that the drift 'forms a positive feedback loop with the feature, causing the global classifier mean and the global feature mean to grow exponentially' is asserted without derivation steps, explicit equations for the drift, or verification that the zero-sum violation produces the claimed exponential growth.
  2. [Abstract] Abstract: the triggering premise that 'the difference between the correct-class logit and the other logits may exceed the absorption-error threshold' once training enters a high-confidence stage is stated without showing that preceding gradient-flow dynamics reach this regime or that the resulting partial absorption breaks zero-sum in the manner required for NFI.
minor comments (1)
  1. The abstract introduces 'Numerical Feature Inflation (NFI)' as a named mechanism but supplies no formal definition or comparison to prior numerical artifacts in the literature.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on the abstract. The main text contains the derivations and verifications referenced in the abstract; we will revise the abstract to better summarize the key steps, equations, and conditions while preserving its brevity. Below we respond point by point.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that the drift 'forms a positive feedback loop with the feature, causing the global classifier mean and the global feature mean to grow exponentially' is asserted without derivation steps, explicit equations for the drift, or verification that the zero-sum violation produces the claimed exponential growth.

    Authors: The abstract is necessarily concise. Sections 3.2–3.4 of the manuscript derive the per-class gradient drift that arises once the correct-class gradient is absorbed to zero while incorrect-class gradients remain nonzero, violating the zero-sum property. Section 4 then proves that this drift couples to the feature extractor to produce exponential growth of both the global classifier mean and global feature mean; the proof proceeds by showing that the update to the classifier weights induces a proportional inflation in the feature norms, which in turn amplifies the logit gap and closes the positive-feedback loop. We will revise the abstract to include a one-sentence outline of this derivation and cite the relevant sections. revision: yes

  2. Referee: [Abstract] Abstract: the triggering premise that 'the difference between the correct-class logit and the other logits may exceed the absorption-error threshold' once training enters a high-confidence stage is stated without showing that preceding gradient-flow dynamics reach this regime or that the resulting partial absorption breaks zero-sum in the manner required for NFI.

    Authors: The abstract assumes the high-confidence regime that is standard in late-stage unregularized training. The manuscript verifies that this regime is reached by (i) showing analytically that continued gradient descent on the cross-entropy loss drives logit margins to grow without bound in the absence of regularization, and (ii) confirming via controlled low-precision simulations that the absorption threshold is crossed precisely when the margin exceeds machine epsilon scaled by the logit magnitude. The resulting partial absorption and zero-sum violation are then shown to initiate NFI. We will add a short clause to the abstract noting that the high-confidence regime is attained under the training conditions studied and is verified in our experiments. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation is self-contained mathematical argument from FP error model

full rationale

The paper's central derivation begins from the stated absorption-error model in floating-point arithmetic and derives the positive feedback loop and exponential growth of means directly from the resulting gradient drift and zero-sum violation. No equations or steps are shown reducing to a fitted parameter, self-referential definition, or load-bearing self-citation. The high-confidence triggering condition is presented as an entry point into the regime rather than a derived output, but this does not constitute a circular reduction of the claimed result to its inputs. The NFI mechanism and its consequences for norm growth are obtained by explicit construction from the precision-limited gradient update rule.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The mechanism depends on the existence of an absorption threshold in floating-point addition and on the assumption that high-confidence training routinely exceeds it; no free parameters or new physical entities are introduced.

axioms (1)
  • domain assumption Floating-point addition exhibits an absorption threshold beyond which small values round to zero when added to large values.
    Invoked to explain why the correct-class gradient becomes exactly zero while others remain nonzero.
invented entities (1)
  • Numerical Feature Inflation (NFI) no independent evidence
    purpose: Name for the positive feedback loop between classifier drift and feature growth.
    Newly introduced mechanism to connect the gradient imbalance to exponential norm growth.

pith-pipeline@v0.9.1-grok · 5800 in / 1362 out tokens · 27436 ms · 2026-06-30T23:31:22.270210+00:00 · methodology

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 9 canonical work pages · 2 internal anchors

  1. [1]

    arXiv preprint arXiv:2206.04817 , year=

    Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon.arXiv preprint arXiv:2206.04817, 2022

    work page arXiv 2022

  2. [2]

    Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  3. [3]

    Gradient descent on neural networks typically occurs at the edge of stability

    Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. InInternational Conference on Learning Representations, 2021

    2021

  4. [4]

    Progress measures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  5. [5]

    Lucas Prieto, Melih Barsbey, Pedro A. M. Mediano, and Tolga Birdal. Grokking at the edge of numerical stability. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  6. [6]

    Gradient descent maximizes the margin of homogeneous neural networks

    Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, 2020

    2020

  7. [7]

    The implicit bias of gradient descent on separable data.Journal of Machine Learning Research, 19(70):1–57, 2018

    Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data.Journal of Machine Learning Research, 19(70):1–57, 2018

    2018

  8. [8]

    Morcos, Ali Farhadi, and Ludwig Schmidt

    Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari S. Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-precision training for large-scale vision-language models. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  9. [9]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  10. [10]

    Why low-precision transformer training fails: An analysis on flash attention

    Haiquan Qiu and Quanming Yao. Why low-precision transformer training fails: An analysis on flash attention. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  11. [11]

    Michaud, and Max Tegmark

    Ziming Liu, Eric J. Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  12. [12]

    Lee, and Wei Hu

    Kaifeng Lyu, Jikai Jin, Zhiyuan Li, Simon Shaolei Du, Jason D. Lee, and Wei Hu. Dichotomy of early and late phase implicit biases can provably induce grokking. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  13. [13]

    A theoretical framework for grokking: Interpolation followed by riemannian norm minimisation

    Etienne Boursier, Scott Pesme, and Radu-Alexandru Dragomir. A theoretical framework for grokking: Interpolation followed by riemannian norm minimisation. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  14. [14]

    Provable scaling laws of feature emergence from learning dynamics of grokking

    Yuandong Tian. Provable scaling laws of feature emergence from learning dynamics of grokking. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  15. [15]

    arXiv preprint arXiv:2301.02679 , year=

    Andrey Gromov. Grokking modular arithmetic.arXiv preprint arXiv:2301.02679, 2023

    work page arXiv 2023

  16. [16]

    Gershman, and Cengiz Pehlevan

    Tanishq Kumar, Blake Bordelon, Samuel J. Gershman, and Cengiz Pehlevan. Grokking as the transition from lazy to rich training dynamics. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  17. [17]

    Progress measures for grokking on real-world tasks.arXiv preprint arXiv:2405.12755, 2024

    Satvik Golechha. Progress measures for grokking on real-world tasks.arXiv preprint arXiv:2405.12755, 2024

    work page arXiv 2024

  18. [18]

    Prakash, Charles H

    Hari K Prakash and Charles H Martin. Grokking and generalization collapse: Insights from HTSR theory. arXiv preprint arXiv:2506.04434, 2025

    work page arXiv 2025

  19. [19]

    Let me grok for you: Accelerating grokking via embedding transfer from a weaker model

    Zhiwei Xu, Zhiyu Ni, Yixin Wang, and Wei Hu. Let me grok for you: Accelerating grokking via embedding transfer from a weaker model. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  20. [20]

    Prevalence of neural collapse during the terminal phase of deep learning training.Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020b

    Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training.Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020b. 11

  21. [21]

    Neural collapse under cross-entropy loss.Applied and Computational Harmonic Analysis, 59:224–241, 2022

    Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss.Applied and Computational Harmonic Analysis, 59:224–241, 2022. ISSN 1063-5203. Special Issue on Harmonic Analysis and Machine Learning

    2022

  22. [22]

    Connall Garrod and Jonathan P. Keating. The persistence of neural collapse despite low-rank bias. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  23. [23]

    Neural collapse for cross-entropy class- imbalanced learning with unconstrained reLU features model

    Hien Dang, Tho Tran Huu, Tan Minh Nguyen, and Nhat Ho. Neural collapse for cross-entropy class- imbalanced learning with unconstrained reLU features model. InForty-first International Conference on Machine Learning, 2024

    2024

  24. [24]

    Explaining grokking and information bottleneck through neural collapse emergence.arXiv preprint arXiv:2509.20829, 2025

    Keitaro Sakamoto and Issei Sato. Explaining grokking and information bottleneck through neural collapse emergence.arXiv preprint arXiv:2509.20829, 2025

    work page arXiv 2025

  25. [25]

    Flatness is necessary, neural collapse is not: Rethinking generalization via grokking

    Ting Han, Linara Adilova, Henning Petzka, Jens Kleesiek, and Michael Kamp. Flatness is necessary, neural collapse is not: Rethinking generalization via grokking. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  26. [26]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InThe Third International Conference on Learning Representations, 2015

    2015

  27. [27]

    On the convergence of adam and beyond

    Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. InInternational Conference on Learning Representations, 2018

    2018

  28. [28]

    Adafactor: Adaptive learning rates with sublinear memory cost

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596–4604. PMLR, 2018

    2018

  29. [29]

    Adam can converge without any modification on update rules

    Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Luo. Adam can converge without any modification on update rules. InAdvances in Neural Information Processing Systems, 2022

    2022

  30. [30]

    Dahl, and Justin Gilmer

    Jeremy Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, Zachary Nado, George E. Dahl, and Justin Gilmer. Adaptive gradient methods at the edge of stability. InNeurIPS 2023 Workshop Heavy Tails in Machine Learning, 2023

    2023

  31. [31]

    Jeremy Cohen, Alex Damian, Ameet Talwalkar, J Zico Kolter, and Jason D. Lee. Understanding opti- mization in deep learning with central flows. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  32. [32]

    A Theory on Adam Instability in Large-Scale Machine Learning,

    Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, et al. A theory on Adam instability in large-scale machine learning.arXiv preprint arXiv:2304.09871, 2023

    work page arXiv 2023

  33. [33]

    Adaptive Preconditioners Trigger Loss Spikes in Adam

    Zhiwei Bai, Zhangchen Zhou, Jiajie Zhao, Xiaolong Li, Zhiyu Li, Feiyu Xiong, Hongkang Yang, Yaoyu Zhang, and Zhi-Qin John Xu. Adaptive preconditioners trigger loss spikes in Adam.arXiv preprint arXiv:2506.04805, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  34. [34]

    A qualitative study of the dynamic behavior for adaptive gradient algorithms

    Chao Ma, Lei Wu, and Weinan E. A qualitative study of the dynamic behavior for adaptive gradient algorithms. InProceedings of the 2nd Mathematical and Scientific Machine Learning Conference, volume 145 ofProceedings of Machine Learning Research, pages 671–692. PMLR, 2022

    2022

  35. [35]

    Ieee standard for floating-point arithmetic.IEEE Std 754-2019 (Revision of IEEE 754-2008), pages 1–84, 2019

    2019

  36. [36]

    PyTorch: An imperative style, high-performance deep learning library.Advances in neural information processing systems, 32, 2019

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library.Advances in neural information processing systems, 32, 2019

    2019

  37. [37]

    Small-scale proxies for large-scale transformer training instabilities

    Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie E Everett, Alexander A Alemi, Ben Adlam, John D Co- Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies for large-scale transformer training instabilities. InThe Twelfth International C...

    2024

  38. [38]

    Antonio Orvieto and Robert M. Gower. In search of Adam’s secret sauce. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026

    2026

  39. [39]

    Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113, 2023

    2023

  40. [40]

    Batch nor- malization provably avoids ranks collapse for randomly initialised deep networks.Advances in Neural Information Processing Systems, 33:18387–18398, 2020

    Hadi Daneshmand, Jonas Kohler, Francis Bach, Thomas Hofmann, and Aurelien Lucchi. Batch nor- malization provably avoids ranks collapse for randomly initialised deep networks.Advances in Neural Information Processing Systems, 33:18387–18398, 2020

    2020

  41. [41]

    Representation degeneration problem in training natural language generation models

    Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. Representation degeneration problem in training natural language generation models. InInternational Conference on Learning Representations, 2019. 12

    2019

  42. [42]

    Liu Ziyin, Yizhou Xu, and Isaac L. Chuang. Neural thermodynamics: Entropic forces in deep and universal representation learning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  43. [43]

    Neural collapse with unconstrained features.Sampling Theory, Signal Processing, and Data Analysis, 20(2):11, 2022

    Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features.Sampling Theory, Signal Processing, and Data Analysis, 20(2):11, 2022

    2022

  44. [44]

    Implicit bias of adamw: ℓ∞-norm constrained optimization

    Shuo Xie and Zhiyuan Li. Implicit bias of adamw: ℓ∞-norm constrained optimization. InForty-first International Conference on Machine Learning, 2024

    2024

  45. [45]

    On large-batch training for deep learning: Generalization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. InInternational Conference on Learning Representations, 2017

    2017

  46. [46]

    Smith and Quoc V

    Samuel L. Smith and Quoc V . Le. A bayesian perspective on generalization and stochastic gradient descent. InInternational Conference on Learning Representations, 2018

    2018

  47. [47]

    Susskind

    Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua M. Susskind. The slingshot effect: A late-stage optimization anomaly in adaptive gradient methods.Transactions on Machine Learning Research, 2024. ISSN 2835-8856

    2024

  48. [48]

    Output embedding centering for stable llm pretraining.arXiv preprint arXiv:2601.02031, 2026

    Felix Stollenwerk, Anna Lokrantz, and Niclas Hertzberg. Output embedding centering for stable llm pretraining.arXiv preprint arXiv:2601.02031, 2026

    work page arXiv 2026

  49. [49]

    Traces of class/cross-class structure pervade deep learning spectra.Journal of Machine Learning Research, 21(252):1–64, 2020a

    Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra.Journal of Machine Learning Research, 21(252):1–64, 2020a

  50. [50]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  51. [51]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  52. [52]

    Very deep convolutional networks for large-scale image recogni- tion

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In Yoshua Bengio and Yann LeCun, editors,3rd International Conference on Learning Representa- tions, 2015

    2015

  53. [53]

    nearest" solutions are frequently “bad

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In9th International Conference on Learning Representations, 2021. 13 App...

    2021

  54. [54]

    Forλ 1: vW −αv µ = (1 + p αβ)vW =⇒ −αv µ = p αβvW =⇒W G =− 1√ K µG (43)

  55. [55]

    Forλ 2,W G = 1√ K µG. The state vector can be expressed as a linear combination of the eigenvectors: u(t) =c 1λt 1 − 1√ K µG µG +c 2λt 2 1√ K µG µG (44) Ast→ ∞, the term withλ 1 dominates sinceλ 1 >1andλ 2 <1: lim t→∞ u(t) ∝ − 1√ K µG µG (45) This leads to the asymptotic relationship: lim t→∞ W (t) G ≈ − 1√ K µ(t) G (46) Therefore, the vectors asymptotica...

  56. [56]

    The gradient of the loss with respect to logits is the prediction error: ∇zL= ˆy−y

    = 0(58) Using the sub-multiplicativity of the matrix norm, the norm of the GGN term is bounded by: ∥G(θ)∥2 =∥J T HzJ∥ 2 ≤ ∥J∥ 2 2∥Hz∥2 (59) Assuming the Jacobian J is bounded in the local convergence region (i.e., ∃M1 >0 such that ∥J∥ 2 ≤M 1), it follows that: lim ˆy→y ∥G(θ)∥2 = 0(60) C.4.2 Analysis of the Residual TermE(θ). The gradient of the loss with ...

    2048