REVIEW 6 cited by
On weight initialization in deep neural networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
On weight initialization in deep neural networks
read the original abstract
A proper initialization of the weights in a neural network is critical to its convergence. Current insights into weight initialization come primarily from linear activation functions. In this paper, I develop a theory for weight initializations with non-linear activations. First, I derive a general weight initialization strategy for any neural network using activation functions differentiable at 0. Next, I derive the weight initialization strategy for the Rectified Linear Unit (RELU), and provide theoretical insights into why the Xavier initialization is a poor choice with RELU activations. My analysis provides a clear demonstration of the role of non-linearities in determining the proper weight initializations.
Forward citations
Cited by 6 Pith papers
-
$\text{DT}^2$: Decision-Targeted Digital Twins
DT² trains digital twins to preserve pairwise policy rankings from fitted Q-evaluation on offline data rather than minimizing one-step transition errors, improving policy ranking and reducing decision regret.
-
Efficient Unlearning through Maximizing Relearning Convergence Delay
The Influence Eliminating Unlearning framework maximizes relearning convergence delay via weight decay and noise injection to remove the influence of a forgetting set while preserving accuracy on retained data.
-
An Extensible and Lightweight Unified Architecture for Demosaicing Pixel-bin Image Sensors
Proposes a modular unified lightweight neural architecture for demosaicing multiple pixel-bin CFAs together with a learning-free CFA identification module.
-
Critical evaluation of PINN for FWD inverse analysis and differentiable FEM as an alternative
Standard PINNs fail to recover layer moduli in multilayer pavement systems due to domain discontinuities; XPINN improves but remains sensitive to weighting and noise; DiffFEM yields superior accuracy, stability, and e...
-
Trainability of IQP Quantum Circuit Born Machines Under Gaussian Initialization
Derives lower bound on gradient variance and probabilistic concentration bounds for Gaussian-initialized IQP QCBMs trained via MMD loss.
-
Embedding Non-Distortive Cancelable Face Template Generation
Presents a non-distortive cancelable face template method via targeted image distortion that maintains identity signals for neural embedding models on MNIST and LFW data.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.