Pith. sign in

REVIEW 2 major objections 42 references

An exact identity for Frobenius-norm imbalance in deep nonlinear networks reduces saddle escape to a scalar ODE whose time scale depends on bottleneck layer count r rather than total depth L.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-01 00:34 UTC pith:3VKFXRQM

load-bearing objection Exact identity on Frobenius norm imbalance is new and general, but the r-2 escape scaling rests on an unquantified approximate balance law. the 2 major comments →

arxiv 2605.01288 v3 pith:3VKFXRQM submitted 2026-05-02 cs.LG cond-mat.dis-nnstat.ML

A Theory of Saddle Escape in Deep Nonlinear Networks

Divit Rawal , Michael R. DeWeese This is my paper
classification cs.LG cond-mat.dis-nnstat.ML
keywords deep nonlinear networkssaddle escapeFrobenius norm imbalanceuniversality classespermutation-symmetric submanifoldcritical depthscalar ODE reductionactivation functions
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper derives an exact identity for the imbalance of Frobenius norms of layer weight matrices that holds for any smooth activation and any differentiable loss. This identity partitions activation functions into four universality classes. On the permutation-symmetric submanifold the identity combines with an approximate balance law to collapse the full matrix flow to a scalar ordinary differential equation. Solving the ODE produces an escape-time law from saddle points that scales as epsilon to the power of minus (r minus 2), where r is the number of layers at the bottleneck scale. The same exponent appears under He-normal initialization when the r bottleneck layers are rescaled by epsilon, and the predictions match numerical simulations.

Core claim

The authors establish an exact identity for the imbalance of Frobenius norms of the layer weight matrices that is valid for any smooth activation function and any differentiable loss function. When restricted to the permutation-symmetric submanifold and combined with an approximate balance law, this identity reduces the high-dimensional matrix flow to a one-dimensional ODE. The resulting critical-depth escape time is governed by the exponent r-2, where r counts the layers at the bottleneck scale rather than the total depth L. The same scaling is recovered under He-normal initialization with r bottleneck layers rescaled by epsilon, and the predictions agree closely with numerical simulations.

What carries the argument

Exact identity for the imbalance of Frobenius norms of layer weight matrices; it classifies activations into four universality classes and, together with the approximate balance law, reduces the matrix flow to a scalar ODE on the permutation-symmetric submanifold.

Load-bearing premise

The approximate balance law on the permutation-symmetric submanifold must hold in order to reduce the full matrix flow to the scalar ODE.

What would settle it

Numerical experiments that vary the number r of bottleneck layers while holding total depth L fixed and measure an escape-time scaling that deviates from epsilon to the power of minus (r minus 2).

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Escape time from saddles is independent of total depth L and is controlled only by the bottleneck layer count r.
  • Activation functions are grouped into four universality classes according to the form taken by the norm-imbalance identity.
  • The r-2 exponent is recovered under He-normal initialization once the r bottleneck layers are rescaled by epsilon.
  • The scalar-ODE reduction produces predictions that match numerical simulations of the training dynamics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the approximate balance law holds beyond the symmetric submanifold, the scalar reduction could simplify analysis of training phases that begin from asymmetric initializations.
  • The four universality classes suggest that activation choice could be used to adjust the escape exponent without altering network depth or width.
  • The critical-depth result implies that optimization speed near saddles is set by the narrowest scale rather than overall network size.
  • The same reduction technique might be applied to other phases of gradient flow once an analogous balance law is identified.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper derives an exact identity for the imbalance of Frobenius norms of layer weight matrices that holds for arbitrary smooth activations and differentiable losses, using it to classify activations into four universality classes. On the permutation-symmetric submanifold this identity is combined with an approximate balance law to reduce the matrix dynamics to a scalar ODE, producing the escape-time scaling τ★ = Θ(ε^{-(r-2)}) controlled by bottleneck depth r rather than total depth L. The same exponent is recovered under He-normal initialization with r rescaled bottleneck layers (where the symmetry manifold is preserved but not attracting), and the predictions are reported to agree closely with simulations.

Significance. If the approximate balance law can be shown to hold with controlled error in the relevant small-ε regime, the exact identity and resulting critical-depth scaling would constitute a substantive advance in the analysis of saddle escape for deep nonlinear networks, moving beyond the well-understood linear and shallow cases. The parameter-free character of the identity itself is a clear strength.

major comments (2)
  1. [abstract] Abstract (paragraph on reduction to scalar ODE): the headline scaling τ★ = Θ(ε^{-(r-2)}) is obtained only after invoking an unspecified 'approximate balance law' on the permutation-symmetric submanifold; no error bound, scaling regime, or estimate of the neglected terms is supplied. If those terms are O(ε^α) with α < r-2 they can dominate the leading balance and change both the exponent and the claim that only r (not L) governs escape.
  2. [abstract] Abstract (He-normal initialization paragraph): the statement that the r-2 exponent is recovered when the symmetry manifold is preserved but not attracting does not address whether the same approximate balance law remains valid or whether the neglected terms again affect the leading-order scaling; this is load-bearing for the universality of the critical-depth law.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the careful reading and for identifying the exact identity as a strength. We respond point-by-point to the major comments on the approximate balance law.

read point-by-point responses

  1. Referee: [abstract] Abstract (paragraph on reduction to scalar ODE): the headline scaling τ★ = Θ(ε^{-(r-2)}) is obtained only after invoking an unspecified 'approximate balance law' on the permutation-symmetric submanifold; no error bound, scaling regime, or estimate of the neglected terms is supplied. If those terms are O(ε^α) with α < r-2 they can dominate the leading balance and change both the exponent and the claim that only r (not L) governs escape.

    Authors: We agree that the manuscript invokes the approximate balance law without supplying a rigorous error bound or explicit scaling regime for the neglected terms. The law follows from combining the exact imbalance identity with the observation that, on the permutation-symmetric submanifold under small initialization, the layer norms remain close; the resulting scalar ODE is therefore a leading-order reduction. Simulations across multiple activations and depths show that the neglected terms remain subdominant and do not alter the r-2 exponent or the independence from total depth L. In revision we will add a paragraph after the reduction derivation that states the working assumptions, reports the observed numerical error scaling, and clarifies that the claim concerns the leading-order escape time. This is a partial revision because a fully rigorous a-priori bound is not supplied. revision: partial

  2. Referee: [abstract] Abstract (He-normal initialization paragraph): the statement that the r-2 exponent is recovered when the symmetry manifold is preserved but not attracting does not address whether the same approximate balance law remains valid or whether the neglected terms again affect the leading-order scaling; this is load-bearing for the universality of the critical-depth law.

    Authors: Under He-normal initialization with r rescaled bottleneck layers the flow exactly preserves the symmetry manifold, so the same exact identity applies and the identical reduction to the scalar ODE is used. Additional simulations (to be added) confirm that the approximate balance law continues to hold with error of the same order as in the attracting case, yielding the same r-2 scaling. The revision will expand the relevant paragraph and abstract sentence to note this numerical verification explicitly, thereby supporting the universality statement. Again this is partial because the error control remains numerical rather than analytic. revision: partial

standing simulated objections not resolved
  • A rigorous, a-priori error bound establishing that the neglected terms in the approximate balance law are o(ε^{r-2}) throughout the small-ε regime (required for a fully controlled proof of the leading-order scaling).

Circularity Check

0 steps flagged

No significant circularity; derivation chain is self-contained

full rationale

The paper states an exact identity for Frobenius-norm imbalance that holds for arbitrary smooth activations and differentiable losses, then invokes a separate approximate balance law on the permutation-symmetric submanifold to obtain the scalar ODE and the τ★ = Θ(ε^{-(r-2)}) scaling. No step reduces the claimed result to its own inputs by construction, no fitted parameter is relabeled as a prediction, and no load-bearing premise rests on a self-citation chain. The approximate law is an additional assumption whose error is not quantified in the provided text, but this is a question of rigor rather than circularity; the central identity and reduction steps remain independent of the final scaling law.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Review performed on abstract only; the primary additional premise is the approximate balance law used for the reduction. No free parameters or invented entities are mentioned in the abstract.

axioms (1)
  • domain assumption Approximate balance law on the permutation-symmetric submanifold
    Combined with the exact identity to reduce matrix flow to scalar ODE, as stated in the abstract.

pith-pipeline@v0.9.1-grok · 5712 in / 1344 out tokens · 46766 ms · 2026-07-01T00:34:32.783373+00:00 · methodology

0 comments
read the original abstract

In deep networks with small initialization, training exhibits long plateaus separated by sharp feature-acquisition transitions. Whereas shallow nonlinear networks and deep linear networks are well studied, extending these analyses to deep nonlinear networks remains challenging. We derive an exact identity for the imbalance of Frobenius norms of layer weight matrices that holds for any smooth activation and any differentiable loss and use this to classify activation functions into four universality classes. On the permutation-symmetric submanifold, the identity combines with an approximate balance law to reduce the full matrix flow to a scalar ODE, giving a critical-depth escape time law $\tau_\star = \Theta(\varepsilon^{-(r-2)})$ governed by the number $r$ of layers at the bottleneck scale rather than the total depth $L$. We find that this same $r-2$ exponent is recovered under He-normal initialization with $r$ bottleneck layers rescaled by $\varepsilon$, where the symmetry manifold is preserved by the flow but not attracting. We find close agreement between our theory and numerical simulations.

Figures

Figures reproduced from arXiv: 2605.01288 by Divit Rawal, Michael R. DeWeese.

Figure 1
Figure 1. Figure 1: Empirical confirmation of ansatz reduction. (a) Loss vs. time: reduced ODE (solid line) overlaid with empirical NL-parameter gradient descent (circles). (b) Layer scales Xℓ versus time t: scalar ODE trajectories match the full-parameter dynamics. Because the ansatz is flow-invariant (Section B.1), the population gradient flow on W descends to a flow on the L scalars (X1, . . . , XL). The reduction below is… view at source ↗
Figure 2
Figure 2. Figure 2: Escape time obeys critical-depth law on the manifold. (a) Escape time tesc vs initializa￾tion scale ε for balanced init: closed form (solid) and reduced ODE (diamonds), polynomial scaling ε −(L−2) steepens with depth. (b) Same at fixed L = 6 with r layers at bottleneck scale: diamonds track the ε −(r−2) law of Theorem 6. Theoretical prediction and experiment diverge at large ε. Proof sketch. Separability o… view at source ↗
Figure 3
Figure 3. Figure 3: Universality across activations. (a) Raw escape time tesc vs ε for three Class B activations (tanh, erf, sin; solid line) and two Class C (GELU, Swish; dashed). (b) After rescaling by K(σ) : Class B curves collapse onto the master curve of Corollary 7; Class C deviates by O(γC ε) per Section C. Proof sketch. Partition [s 2 1 , 1] into shells [s 2 j , s2 j+1]. Under the strict-hierarchy assumption the shell… view at source ↗
Figure 4
Figure 4. Figure 4: Off-manifold critical-depth exponent. tesc vs ε for L = 8 tanh with r ∈ {3, 5, 8} bot￾tleneck layers, He-normal init, SGD: slopes track the ε −(r−2) law of Theorem 11. Theorem 11 makes the single-mode expo￾nent intrinsic to M and γ rather than to any ansatz. The product structure ∥Gℓ∥F ≍ ∥Bℓ∥2∥Aℓ−1∥2 is the rank-one Frobenius iden￾tity applied to the filtered-composition expan￾sion of Section B.3, and the … view at source ↗
Figure 5
Figure 5. Figure 5: Three-mode tanh cascade and escape-time decomposition. Black: training loss; blue: mode-1 alignment ∥W1v1∥2/ √ N. Light purple: leading-order single-mode prediction of Theo￾rem 5. Dark purple: homotopy identity T(1) = T(0)+R 1 0 A(ν)dν on the homotopy from decoupled single-mode (ν = 0) to augmented block-mean (ν = 1). Proposition 25 (Schur–Perron positive eigenvalue). Let A, D ∈ R n×n be diagonal matrices … view at source ↗

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 33 canonical work pages · 10 internal anchors

  1. [1]

    Sgd learning on neural net- works: leap complexity and saddle-to-saddle dynamics.ArXiv, abs/2302.11055, 2023

    Emmanuel Abbe, Enric Boix-Adserà, and Theodor Misiakiewicz. Sgd learning on neural net- works: leap complexity and saddle-to-saddle dynamics.ArXiv, abs/2302.11055, 2023. URL https://api.semanticscholar.org/CorpusID:257078637

    work page arXiv 2023

  2. [2]

    The merged-staircase prop- erty: a necessary and nearly sufficient condition for sgd learning of sparse functions on two- layer neural networks, 2024

    Emmanuel Abbe, Enric Boix-Adsera, and Theodor Misiakiewicz. The merged-staircase prop- erty: a necessary and nearly sufficient condition for sgd learning of sparse functions on two- layer neural networks, 2024. URLhttps://arxiv.org/abs/2202.08658

    work page arXiv 2024

  3. [3]

    High-dimensional dynamics of generalization error in neural networks

    Madhu S. Advani and Andrew M. Saxe. High-dimensional dynamics of generalization error in neural networks, 2017. URLhttps://arxiv.org/abs/1710.03667

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  4. [4]

    Escaping mediocrity: how two-layer networks learn hard generalized linear models with sgd, 2024

    Luca Arnaboldi, Florent Krzakala, Bruno Loureiro, and Ludovic Stephan. Escaping medi- ocrity: how two-layer networks learn hard generalized linear models with sgd, 2024. URL https://arxiv.org/abs/2305.18502

    work page arXiv 2024

  5. [5]

    On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization

    Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization, 2018. URLhttps://arxiv.org/abs/1802.06509

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  6. [6]

    Why gradient clipping accel- erates training: A theoretical justification for adaptivity

    Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization, 2019. URLhttps://arxiv.org/abs/1905.13655

    work page arXiv 2019

  7. [7]

    Online stochastic gradient descent on non-convex losses from high-dimensional inference, 2021

    Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference, 2021. URLhttps://arxiv.org/ abs/2003.10409

    work page arXiv 2021

  8. [8]

    High-dimensional limit theorems for sgd: Effective dynamics and critical scaling, 2023

    Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. High-dimensional limit theorems for sgd: Effective dynamics and critical scaling, 2023. URLhttps://arxiv.org/abs/ 2206.04030

    work page arXiv 2023

  9. [9]

    Simon, and Cengiz Pehlevan

    Alexander Atanasov, Alexandru Meterez, James B. Simon, and Cengiz Pehlevan. The opti- mization landscape of sgd across the feature learning strength, 2025. URLhttps://arxiv. org/abs/2410.04642

    work page arXiv 2025

  10. [10]

    Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escape

    Ioannis Bantzis, James B. Simon, and Arthur Jacot. Saddle-to-saddle dynamics in deep relu networks: Low-rank bias in the first saddle escape, 2026. URLhttps://arxiv.org/abs/ 2505.21722

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  11. [11]

    Learning by on-line gradient descent.Journal of Physics A: Mathematical and General, 28:643–656, 02 1995

    Michael Biehl and H Schwarze. Learning by on-line gradient descent.Journal of Physics A: Mathematical and General, 28:643–656, 02 1995. doi: 10.1088/0305-4470/28/3/018

    work page doi:10.1088/0305-4470/28/3/018 1995

  12. [12]

    Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape

    Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape, 2019. URLhttps://arxiv.org/abs/1907.02911

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  13. [13]

    Lipschitz Flow-box Theorem

    Craig Calcaterra and Axel Boldt. Lipschitz flow-box theorem, 2006. URLhttps://arxiv. org/abs/math/0305207

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  14. [14]

    Chizat, E

    Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- ming, 2020. URLhttps://arxiv.org/abs/1812.07956

    work page arXiv 2020

  15. [15]

    Du, Wei Hu, and Jason D

    Simon S. Du, Wei Hu, and Jason D. Lee. Algorithmic regularization in learning deep homoge- neous models: Layers are automatically balanced.Advances in Neural Information Processing Systems, 2018-December:384–395, 2018. ISSN 1049-5258. Publisher Copyright: © 2018 Cur- ran Associates Inc..All rights reserved.; 32nd Conference on Neural Information Processin...

    2018

  16. [16]

    Effect of batch learning in multilayer neural networks

    Kenji Fukumizu. Effect of batch learning in multilayer neural networks. InInternational Con- ference on Neural Information Processing, 1998. URLhttps://api.semanticscholar. org/CorpusID:605683. 10

    1998

  17. [17]

    Sebastian Goldt, Madhu S Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup*.Journal of Statistical Mechanics: Theory and Experiment, 2020(12):124010, December

    2020

  18. [18]

    doi: 10.1088/1742-5468/abc61e

    ISSN 1742-5468. doi: 10.1088/1742-5468/abc61e. URLhttp://dx.doi.org/10. 1088/1742-5468/abc61e

    work page doi:10.1088/1742-5468/abc61e

  19. [19]

    Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Sur- passing human-level performance on imagenet classification, 2015. URLhttps://arxiv. org/abs/1502.01852

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  20. [20]

    A. C. Hindmarsh and L. R. Petzold. LSODA, ordinary differential equation solver for stiff or non-stiff system. Nuclear Energy Agency of the OECD (NEA), Sep 2005. URLhttps: //www.osti.gov/etdeweb/biblio/21352532

    work page arXiv 2005

  21. [21]

    Horn and Charles R

    Roger A. Horn and Charles R. Johnson.Matrix Analysis. Cambridge University Press, 1990. ISBN 0521386322. URLhttp://www.amazon.com/Matrix-Analysis-Roger-Horn/ dp/0521386322%3FSubscriptionId%3D192BW6DQ43CK9FN0ZGG2%26tag%3Dws% 26linkCode%3Dxm2%26camp%3D2025%26creative%3D165953%26creativeASIN% 3D0521386322

    work page arXiv 1990

  22. [22]

    Saddle-to- saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity,

    Arthur Jacot, François Ged, Berfin ¸ Sim¸ sek, Clément Hongler, and Franck Gabriel. Saddle-to- saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity,

  23. [23]

    URLhttps://arxiv.org/abs/2106.15933

    work page arXiv

  24. [24]

    Gradient descent aligns the layers of deep linear networks

    Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks, 2019. URLhttps://arxiv.org/abs/1810.02032

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  25. [25]

    How to Escape Saddle Points Efficiently

    Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, and Michael I. Jordan. How to escape saddle points efficiently, 2017. URLhttps://arxiv.org/abs/1703.00887

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  26. [26]

    Directional convergence near small initializations and saddles in two-homogeneous neural networks, 2024

    Akshay Kumar and Jarvis Haupt. Directional convergence near small initializations and saddles in two-homogeneous neural networks, 2024. URLhttps://arxiv.org/abs/2402.09226

    work page arXiv 2024

  27. [27]

    Daniel Kunin, Javier Sagastuy-Brena, Surya Ganguli, Daniel L. K. Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dy- namics, 2021. URLhttps://arxiv.org/abs/2012.04728

    work page arXiv 2021

  28. [28]

    Get rich quick: exact solutions reveal how unbalanced initializations pro- mote rapid feature learning

    Daniel Kunin, Allan Raventós, Clémentine Dominé, Feng Chen, David Klindt, Andrew Saxe, and Surya Ganguli. Get rich quick: exact solutions reveal how unbalanced initializations pro- mote rapid feature learning. InProceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY , USA, 2024. Curran Associates ...

    2024

  29. [29]

    arXiv preprint arXiv:2506.06489 , year=

    Daniel Kunin, Giovanni Luca Marchetti, Feng Chen, Dhruva Karkada, James B. Simon, Michael R. DeWeese, Surya Ganguli, and Nina Miolane. Alternating gradient flows: A the- ory of feature learning in two-layer neural networks, 2025. URLhttps://arxiv.org/abs/ 2506.06489

    work page arXiv 2025

  30. [30]

    A mean field view of the landscape of two-layer neural networks

    Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks.Proceedings of the National Academy of Sciences, 115(33), 2018. ISSN 1091-6490. doi: 10.1073/pnas.1806579115. URLhttp://dx.doi.org/10.1073/ pnas.1806579115

    work page doi:10.1073/pnas.1806579115 2018

  31. [31]

    Convergence and implicit bias of gradient flow on overparametrized linear networks, 2022

    Hancheng Min, Salma Tarmoun, René Vidal, and Enrique Mallada. Convergence and implicit bias of gradient flow on overparametrized linear networks, 2022. URLhttps://arxiv.org/ abs/2105.06351

    work page arXiv 2022

  32. [32]

    Saddle-to-saddle dynamics in diagonal linear networks,

    Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks,

  33. [33]

    URLhttps://arxiv.org/abs/2304.00488

    work page arXiv

  34. [34]

    On the Spectral Bias of Neural Networks

    Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks, 2019. URL https://arxiv.org/abs/1806.08734. 11

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  35. [35]

    Dynamics of on-line gradient descent learning for multilayer neural networks

    David Saad and Sara Solla. Dynamics of on-line gradient descent learning for multilayer neural networks. 04 1999

    1999

  36. [36]

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

    Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, 2014. URLhttps://arxiv.org/abs/ 1312.6120

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  37. [37]

    Saxe, Shagun Sodhani, and Sam Lewallen

    Andrew M. Saxe, Shagun Sodhani, and Sam Lewallen. The neural race reduction: Dynamics of abstraction in gated networks, 2022. URLhttps://arxiv.org/abs/2207.10430

    work page arXiv 2022

  38. [38]

    Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J

    James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning, 2023. URLhttps://arxiv. org/abs/2303.15438

    work page arXiv 2023

  39. [39]

    Geometry of the Loss Landscape in Overparameterized Neural Networks: Sym- metries and Invariances

    Berfin ¸ Sim¸ sek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural net- works: Symmetries and invariances, 2021. URLhttps://arxiv.org/abs/2105.12221

    work page arXiv 2021

  40. [40]

    Tanaka and D

    Hidenori Tanaka and Daniel Kunin. Noether’s learning dynamics: Role of symmetry breaking in neural networks, 2021. URLhttps://arxiv.org/abs/2105.02716. 12 A Proof and Extension of Theorem 1 We give the derivation of Theorem 1, state the matrix-valued refinement, and record the almost- everywhere version for ReLU. A.1 Full Derivation of Theorem 1 Fix a lay...

    work page arXiv 2021

  41. [41]

    strict hierarchy

    =O(∥X∥ L+q−1)sharply; the sharpness argument of (24) (withh (q) σ ̸= 0 for every Class B activation used in this paper) shows the bound is attained, not merely an upper bound. Proposition 13 is the nonlinear analog of deep-linear balance: the Class B bound holds to order ∥X∥ L+2, the Class C bound to∥X∥ L+1. The Class B exponent is sharper than pointwise ...

  42. [42]

    At each quadrature node, the forward trajectory ˙xν =f ν(xν)and backward adjoint˙p ν =−(∂ xfν)⊤pν are integrated jointly. The Jacobian∂ xfν is constructed analytically in closed form from the reduced-variable equations of motion (layer scales, off-block amplitudes, cross-block rotations): each Jacobian entry is a polynomial in the reduced state with Gauss...