REVIEW 15 cited by
Deep Neural Networks as Gaussian Processes
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Deep Neural Networks as Gaussian Processes
read the original abstract
It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network. In this work, we derive the exact equivalence between infinitely wide deep networks and GPs. We further develop a computationally efficient pipeline to compute the covariance function for these GPs. We then use the resulting GPs to perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10. We observe that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite-width networks. Finally we connect the performance of these GPs to the recent theory of signal propagation in random neural networks.
Forward citations
Cited by 15 Pith papers
-
Spectral Anatomy of Quantum Gaussian Process Kernels
Normalized spectral entropy S(K)/log n of the kernel Gram matrix governs both dequantization in QGP regression and posterior pathologies, supported by proved bounds, a variance identity, and hardware transfer experiments.
-
The Benefits of Temporal Correlations: SGD Learns k-Juntas from Random Walks Efficiently
Temporal correlations from lazy random walks enable efficient SGD learning of k-juntas via temporal-difference loss on ReLU networks, achieving linear sample complexity in d.
-
Phase Transitions in the Fluctuations of Functionals of Random Neural Networks
Functionals of infinite-width random neural networks on the sphere exhibit phase transitions in fluctuations as depth grows, converging to a limiting Gaussian field functional, a Gaussian, or a Qth Wiener chaos distri...
-
Gauge-covariant stochastic neural fields: Stability and finite-width effects
A gauge-covariant stochastic neural field theory is introduced that derives the maximal Lyapunov exponent and amplification factor, showing finite-width effects as perturbative corrections to dressed kernels that leav...
-
Spectral phase transitions and trainability in neural network learning dynamics
SGD on neural network weights induces a BBP phase transition that detaches signal eigenvalues from the random bulk, yielding an analytically solvable phase diagram for trainability in a linear teacher-student model.
-
Radial Basis Function Networks as Projection Heads in Self-Supervised Learning
RBFN projection heads serve as competitive replacements for MLP heads in SSL and enable SNS, a label-free metric from RBF parameters that correlates strongly with logistic regression evaluation.
-
Spontaneous symmetry breaking and Goldstone modes for deep information propagation
Equivariant neural networks support Goldstone-like modes enabling coherent information propagation across depth and recurrent iterations.
-
On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference
An attack aligns differently shuffled intermediate activations from secure Transformer inference queries to recover model weights with low error using roughly one dollar of queries.
-
Optimal Architecture and Fundamental Bounds in Neural Network Field Theory
α=0 architecture in NNFT minimizes finite-width variance, removes IR corrections, and sets a fundamental SNR bound for correlation functions in scalar field theory.
-
Bayesian Neural Network Surrogates for Bayesian Optimization of Carbon Capture and Storage Operations
Comparative evaluation of Bayesian Neural Network surrogates versus Gaussian Processes in Bayesian Optimization applied to Carbon Capture and Storage operations, presented as the first such application in reservoir en...
-
'In-Between' Uncertainty in Bayesian Neural Networks
MFVI in BNNs underestimates uncertainty between data regions, leading to overconfident OOD predictions, while linearised Laplace approximation performs better.
-
Discrete signaling mediates chaotic regularization in recurrent neural networks
Chaotic dynamics in RNNs induce local roughness but preserve global smoothness in representations, acting as an intrinsic regularizer and generating power-law spectral signatures.
-
Lectures on Semiclassical Methods for Composite Operators
Lecture notes develop semiclassical methods to compute large-n scaling dimensions of composite operators in CFTs, recovering known results in free theory and deriving one-loop corrections at the Wilson-Fisher fixed point.
-
SoK: A Comprehensive Analysis of the Current Status of Neural Tangent Generalization Attacks with Research Directions
NTGA is the first clean-label generalization attack under black-box settings but is vulnerable to adversarial training and image transformations, with newer attacks outperforming it.
-
Deep learning applied to computational mechanics: A comprehensive review, state of the art, and the classics
A comprehensive review of deep learning techniques for computational mechanics, including LSTM for constitutive modeling, PINNs for PDE solving, optimizers, and kernel methods.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.