REVIEW 6 cited by
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
A solvable high-dimensional model where nonlinear autoencoders learn structure invisible to PCA while test loss misaligns with generalization
read the original abstract
Many real-world datasets contain hidden structure that cannot be detected by simple linear correlations between input features. For example, latent factors may influence the data in a coordinated way, even though their effect is invisible to covariance-based methods such as PCA. In practice, nonlinear neural networks often succeed in extracting such hidden structure in unsupervised and self-supervised learning. However, constructing a minimal high-dimensional model where this advantage can be rigorously analyzed has remained an open theoretical challenge. We introduce a tractable high-dimensional spiked model with two latent factors: one visible to covariance, and one statistically dependent yet uncorrelated, appearing only in higher-order moments. PCA and linear autoencoders fail to recover the latter, while a minimal nonlinear autoencoder provably extracts both. We analyze both the population risk, and empirical risk minimization. Our model also provides a tractable example where self-supervised test loss is poorly aligned with representation quality: nonlinear autoencoders recover latent structure that linear methods miss, even though their reconstruction loss is higher.
Forward citations
Cited by 6 Pith papers
-
High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model
In a solvable attention model, pre-training followed by rank-one LoRA admits sharp asymptotic predictions for test errors and representation alignment via an effective noise term.
-
The Interplay of Data Structure and Imbalance in the Learning Dynamics of Diffusion Models
Higher-variance classes are learned first in diffusion models; strong class imbalance reverses the order and imposes distinct delayed learning times on minority classes.
-
Signal-to-Noise Ratio and Sample Size Govern Representational Alignment in Neural Networks
Representational alignment varies monotonically with SNR and non-monotonically with sample size (minimized near interpolation threshold) across linear and nonlinear networks, and is decoupled from generalization error.
-
Memorisation, convergence and generalisation in generative models
Linear generative models memorize at small data loads but converge continuously once samples scale linearly with dimension; this convergence is insensitive to sharp recovery of principal latent factors.
-
A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights
Neural networks prioritize amplitude over phase in Fourier space during training on translation-invariant data; power-law spectra accelerate phase learning despite not aiding classification.
-
A theory of learning data statistics in diffusion models, from easy to hard
Diffusion models exhibit a distributional simplicity bias, learning pairwise input statistics at linear sample complexity while fourth-order cumulants require cubic complexity unless sharing correlated latent structure.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.