pith. sign in

arxiv: 1606.08415 · v5 · pith:VL3WXSYJnew · submitted 2016-06-27 · 💻 cs.LG

Gaussian Error Linear Units (GELUs)

Dan Hendrycks , Kevin Gimpel This is my paper

Pith reviewed 2026-05-10 12:12 UTC · model grok-4.3

classification 💻 cs.LG
keywords activation functionGELUReLUELUneural networkdeep learningcomputer visionnatural language processing
0
0 comments X

The pith

The GELU activation xΦ(x) outperforms ReLU and ELU on computer vision, natural language processing, and speech tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes the Gaussian Error Linear Unit (GELU) as a new activation function for neural networks. GELU is defined as xΦ(x), where Φ(x) is the cumulative distribution function of the standard normal distribution, which has the effect of weighting inputs by their value rather than gating them by sign like the ReLU. The authors conduct experiments comparing GELU to ReLU and ELU across multiple domains and report consistent performance improvements with GELU. A sympathetic reader would care because activation functions are a basic building block of deep networks, and small changes here can affect overall model quality without requiring architectural redesign.

Core claim

The GELU nonlinearity, given by xΦ(x) with Φ the standard Gaussian CDF, weights inputs by their value and yields better empirical performance than ReLU or ELU on the considered computer vision, natural language processing, and speech tasks.

What carries the argument

The GELU function xΦ(x), which multiplies each input by the probability that a standard normal random variable is less than or equal to that input.

If this is right

  • GELU can be used as a drop-in replacement for ReLU or ELU in existing neural network models.
  • Performance gains are expected in vision, language, and speech applications when using GELU.
  • Training may converge to better solutions because inputs are scaled continuously rather than thresholded.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The weighting mechanism may provide smoother gradient flow during backpropagation compared to hard gating.
  • Similar activation functions could be derived using other probability distributions beyond the Gaussian.
  • Adoption of GELU might reduce the need for careful initialization or normalization techniques in some models.

Load-bearing premise

The performance improvements seen on the specific tasks and models tested will continue to appear on other architectures, datasets, and training setups.

What would settle it

Running the same models with GELU on a new task or dataset and observing no improvement or degradation relative to ReLU.

read the original abstract

We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is $x\Phi(x)$, where $\Phi(x)$ the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs ($x\mathbf{1}_{x>0}$). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 3 minor

Summary. The paper introduces the Gaussian Error Linear Unit (GELU) activation function defined as xΦ(x), with Φ denoting the standard Gaussian cumulative distribution function. It highlights that GELU weights inputs according to their value, in contrast to ReLU which gates by sign. An empirical evaluation is performed comparing GELU to ReLU and ELU, with reported performance gains across computer vision, natural language processing, and speech tasks.

Significance. If the results are reliable, GELU offers a high-performing, parameter-free activation function with a probabilistic interpretation. This could lead to better neural network models in various fields. The direct definition from the Gaussian CDF and the broad empirical testing are positive aspects of the work.

major comments (1)
  1. The abstract and corresponding experimental sections report consistent improvements but omit key details such as the number of runs, statistical significance tests, hyperparameter search methodology, and precise baseline configurations. These omissions make it difficult to fully evaluate the strength of the empirical claims.
minor comments (3)
  1. The definition of Φ(x) as the Gaussian CDF should be stated explicitly in the introduction or methods section for readers unfamiliar with the notation.
  2. Consider adding a figure illustrating the GELU function alongside ReLU and ELU to visually support the textual description.
  3. Ensure that all acronyms are defined at first use, such as NLP if used in the text.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive review and for recognizing the potential significance of the GELU activation function. We address the major comment below and will revise the manuscript to strengthen the experimental reporting.

read point-by-point responses

  1. Referee: The abstract and corresponding experimental sections report consistent improvements but omit key details such as the number of runs, statistical significance tests, hyperparameter search methodology, and precise baseline configurations. These omissions make it difficult to fully evaluate the strength of the empirical claims.

    Authors: We agree that additional experimental details would improve the manuscript's clarity and allow readers to better assess the reliability of the reported gains. In the revised version, we will expand the relevant sections to specify: the number of runs (noting that computational constraints led to single runs for most large-scale experiments, consistent with practices in the field at the time of submission); that formal statistical significance tests were not conducted but improvements were consistent across diverse tasks; a description of the hyperparameter search process (including search ranges and selection criteria for learning rates, regularization, and other settings applied uniformly to GELU, ReLU, and ELU); and more precise baseline configurations, such as exact network architectures, initialization schemes, and training protocols. These additions will support rather than change the abstract. We believe this directly addresses the concern without overstating the original experiments. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The GELU is defined directly as xΦ(x) using the standard Gaussian CDF with no fitted parameters, self-referential equations, or load-bearing self-citations. The central claim is an empirical observation of performance gains on specific tasks, supported by side-by-side experimental results rather than any internal derivation that reduces to its own inputs by construction. No steps match the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The central claim rests only on the standard definition of the Gaussian CDF and the empirical results; no free parameters are introduced, no new axioms are stated, and no new entities are postulated.

pith-pipeline@v0.9.0 · 5378 in / 987 out tokens · 41939 ms · 2026-05-10T12:12:07.972046+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 8.0

    Introduces the first large-scale multimodal benchmark MedLayXPlain-122K showing medical VLMs suffer significant lay-register degradation while general VLMs lack clinical precision.

  2. Efficient AI-Inspired Reduction of Feynman Integrals via Tube Seeding

    hep-ph 2026-06 unverdicted novelty 8.0

    Machine learning discovers a tube-seeding strategy for IBP reduction of Feynman integrals that scales linearly with numerator power, demonstrated on rank-20 2-loop 5-point integrals.

  3. Towards Verifiable Transformers: Solver-Checkable Circuit Explanations

    cs.LG 2026-05 unverdicted novelty 8.0

    Presents a solver-verifiable framework for Transformer circuits, with exhaustive checks on small symbolic tasks and surrogate methods for larger models.

  4. Stability and Generalization in Looped Transformers

    cs.LG 2026-04 unverdicted novelty 8.0

    Looped transformers with recall and outer normalization produce reachable, input-dependent fixed points with stable gradients, enabling generalization, while those without recall cannot; a new internal recall variant ...

  5. Autoregressive Synthesis of Sparse and Semi-Structured Mixed-Type Data

    cs.LG 2026-03 conditional novelty 8.0

    ORiGAMi synthesizes sparse semi-structured mixed-type JSON data using path-encoded autoregressive tokenization and schema constraints, outperforming flattened tabular baselines on 17 of 18 fidelity, detection, and uti...

  6. The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence

    cs.LG 2026-01 unverdicted novelty 8.0

    Contrastive learning evolves population measures on a fixed manifold into either a unique convex Gibbs equilibrium or a cross-coupled multimodal landscape containing a persistent negative symmetric divergence.

  7. A document is worth a structured record: Principled inductive bias design for document recognition

    cs.CV 2025-07 unverdicted novelty 8.0

    Introduces a method to design structure-specific relational inductive biases for a base transformer architecture, enabling end-to-end transcription of documents with intrinsic structures, demonstrated on sheet music, ...

  8. Learning to (Learn at Test Time): RNNs with Expressive Hidden States

    cs.LG 2024-07 conditional novelty 8.0

    TTT layers treat the hidden state as a trainable model updated at test time, allowing linear-complexity sequence models to scale perplexity reduction with context length unlike Mamba.

  9. Mamba: Linear-Time Sequence Modeling with Selective State Spaces

    cs.LG 2023-12 unverdicted novelty 8.0

    Mamba is a linear-time sequence model using input-dependent selective SSMs that achieves SOTA results across modalities and matches twice-larger Transformers on language modeling with 5x higher inference throughput.

  10. What learning algorithm is in-context learning? Investigations with linear models

    cs.LG 2022-11 accept novelty 8.0

    Transformers performing in-context learning implicitly implement gradient descent, ridge regression, and least-squares predictors for linear models, with behavior shifting based on model depth, width, and data noise.

  11. Low-dimensional topology of deep neural networks

    cs.LG 2026-06 unverdicted novelty 7.0

    Restricting layers to width 3 and using linking numbers shows ResNets and transformers match in topological power, exceed monotonic feedforward nets which exceed flows, but nonmonotonic activations match the top class.

  12. FlexiSLM: A Dynamic and Controllable Frame Rate Spoken Language Model

    cs.SD 2026-06 unverdicted novelty 7.0

    FlexiSLM is the first spoken language model supporting dynamic and controllable frame rates on speech input and output, outperforming fixed-rate 7B models at high quality and enabling faster inference at lower rates l...

  13. Probing Memorization of Tabular In-Context Learning

    cs.LG 2026-06 unverdicted novelty 7.0

    A new probing framework detects moderate parametric memorization signals in tabular in-context learning models under single-task fine-tuning, strongest on low-cardinality tasks, but signals largely disappear under rea...

  14. McMg: A Learned Phase-Space Multi-channel Multigrid Preconditioner for Helmholtz Equation

    math.NA 2026-06 unverdicted novelty 7.0

    McMg is a phase-space multi-channel multigrid preconditioner that maps residuals to corrections while retaining unresolved wave information in extra channels, showing fewer iterations and lower runtime than classical ...

  15. McMg: A Learned Phase-Space Multi-channel Multigrid Preconditioner for Helmholtz Equation

    math.NA 2026-06 unverdicted novelty 7.0

    McMg is a learned phase-space multi-channel multigrid preconditioner that maps residuals to corrections for heterogeneous Helmholtz equations and shows fewer iterations than classical and neural baselines in tests.

  16. Estimating Free Energy Differences with Virtually Escorted Trajectories

    cond-mat.stat-mech 2026-06 unverdicted novelty 7.0

    Introduces virtually escorted trajectories with a virtual control field to generate multiple optimized work quantities satisfying the fluctuation theorem for improved free energy estimation convergence.

  17. Lie Group Diffusion Models for Hardware-Aware Quantum Circuit Synthesis

    quant-ph 2026-06 unverdicted novelty 7.0

    Lie group diffusion models combine a discrete circuit skeleton selector with continuous diffusion on SU(2) ≃ S³ to synthesize hardware-aware quantum circuits, outperforming baselines on three-qubit Hamiltonian simulat...

  18. Neural Texture Compression using Hypernetworks

    cs.GR 2026-06 unverdicted novelty 7.0

    A single hypernetwork generates per-material latent features and MLP decoder parameters for neural texture compression, matching the quality of per-material optimization methods while enabling extensions like super-re...

  19. Dark Matter in Draco and Bo\"otes I: Hints of a Core in an Ultra-Faint Dwarf from Simulation-Based Inference

    astro-ph.GA 2026-06 unverdicted novelty 7.0

    GraphNPE recovers a significantly lower central density for Boötes I consistent with a core while Draco remains marginally cuspy, and demonstrates that higher-order velocity moments reduce bias in dynamical modeling.

  20. FunPiQ: A New Benchmark for Pixel-Level Quality Assessment in Fundus Images

    cs.CV 2026-06 unverdicted novelty 7.0

    FunPiQ supplies the first pixel-level FIQA benchmark and EFIQA-CP uses anatomical visibility pseudo-labels with NNPU learning to deliver superior explainable quality assessment over classification and anomaly detectio...

  21. Real vs. Complex Spectral Bases for Neural Operators: The Role of Green's Function Alignment

    cs.LG 2026-06 unverdicted novelty 7.0

    HNO matches FNO performance on elliptic PDEs but lags on time-dependent ones, supporting a Green's function symmetry rule for choosing real versus complex spectral bases in neural operators.

  22. Real vs. Complex Spectral Bases for Neural Operators: The Role of Green's Function Alignment

    cs.LG 2026-06 unverdicted novelty 7.0

    HNO using real Hartley transform outperforms FNO on elliptic PDEs while FNO wins on time-dependent PDEs, yielding a rule to match spectral basis to operator symmetry via Green's functions.

  23. MATCH: Flow Matching for Multi-View Anomaly Detection

    cs.CV 2026-06 unverdicted novelty 7.0

    MATCH is the first flow matching method for multi-view anomaly detection, reporting SOTA results on Real-IAD and the first comprehensive evaluation on MANTA-Tiny while enabling real-time use by omitting the divergence term.

  24. Fabric Image Demoir\'eing Benchmark from Synthesis to Restoration

    cs.CV 2026-06 unverdicted novelty 7.0

    The paper introduces a synthesis framework, a 16,050-image paired dataset, and a baseline model for fabric image demoiréing.

  25. Shifting-based Optimizable Linear Relaxations for General Activation Functions

    cs.LG 2026-06 unverdicted novelty 7.0

    SLiR parameterizes linear relaxations by slope and uses shifting to compute sound bounds for general activation functions, enabling up to 7.8x more verified properties than prior methods.

  26. ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence

    cs.AI 2026-06 unverdicted novelty 7.0

    ITNet unifies convolution, self-attention, and autoregressive recurrence as special cases of a single learnable integral transform implemented via an MLP kernel and matches specialized models on ImageNet, GLUE, ModelN...

  27. Non-negative Elastic Net Decoding for Information Retrieval

    cs.IR 2026-06 unverdicted novelty 7.0

    NNN decoding selects documents via non-negative elastic net reconstruction of the query embedding, with a theorem showing it strictly dominates dense retrieval on correlated corpora and experiments showing gains over ...

  28. FoundCause: Causal Discovery with Latent Confounders from Observational Data

    cs.LG 2026-06 unverdicted novelty 7.0

    FoundCause is a transformer-based amortized model for causal graph discovery that explicitly models latent confounders via learnable tokens and reports better performance than prior methods on 15 real-world datasets.

  29. Continuum Neural Momentum Eigenstate for Variationally Solving Quasiparticles

    cond-mat.quant-gas 2026-06 unverdicted novelty 7.0

    EVE is a neural quantum state that enforces exact momentum eigenstates by construction, allowing VMC to variationally solve quasiparticle states across multiple phases in 2D interacting bosons.

  30. Spectrally Regularized Latent Flow Matching for Turbulence Generation

    cs.LG 2026-06 unverdicted novelty 7.0

    Spectrally regularized compression in latent flow matching raises retained deep-dissipation spectral power from 20% to 79% in generated turbulence on a 256^2 DNS dataset at Re_f ≈ 2250.

  31. Limitations of Learning Tanh Neural Networks with Finite Precision

    cs.LG 2026-06 unverdicted novelty 7.0

    In finite precision, no adaptive algorithm learns tanh networks faster than Monte Carlo rate O(m^{-1/p}) in L^p without exponential sample growth in network size.

  32. GNSS-FM: A Self-Supervised Foundation Model for Daily GNSS Displacement Time Series

    physics.geo-ph 2026-06 unverdicted novelty 7.0

    GNSS-FM is a self-supervised foundation model for GNSS displacement time series that outperforms task-specific baselines on 90-day forecasting and seismic step localization after pretraining on global station data.

  33. WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing

    cs.LG 2026-06 unverdicted novelty 7.0

    WhiFlash introduces token-level cross-paradigm routing between autoregressive and diffusion drafting models, with cache optimizations, to raise acceptance lengths and deliver up to 69.6% throughput gains over EAGLE-3.

  34. Adaptive Oscillatory-State Alignment for Time Series Forecasting

    cs.LG 2026-06 unverdicted novelty 7.0

    AOSNET is a Hilbert-guided framework that performs adaptive oscillatory-state alignment to a learnable prior for improved long-term forecasting under non-stationary conditions.

  35. Biomazon: A Multimodal Dataset for 3D Forest Structure and Biomass Modeling in the Amazon Basin

    cs.CV 2026-06 unverdicted novelty 7.0

    Biomazon is a multimodal benchmark dataset pairing GEDI RH profiles and AGBD targets with Sentinel, ALOS, DEM, and other predictors for joint 3D forest structure and biomass modeling in the Amazon.

  36. Dual Advantage Fields

    cs.LG 2026-06 conditional novelty 7.0

    Dual Advantage Fields converts bilinear dual value models into local advantage scores via learned action-effect models, equaling the goal-conditioned Bellman advantage under realizability and improving aggregate metri...

  37. The Dynamic-Probabilistic Consistency Gap in Chaotic Surrogate Modeling

    cs.LG 2026-05 unverdicted novelty 7.0

    Exposes a dynamic-probabilistic consistency gap in chaotic dynamical systems reconstruction and introduces the KAFFEE differentiable extended Kalman filter training framework to address it.

  38. Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequences

    cs.LG 2026-05 unverdicted novelty 7.0

    Introduces Chess-World-Model benchmark from 10M chess games showing recurrent models (SLiCE, Mamba-3, Gated DeltaNet) outperform Transformers on exact state tracking, with random-play split remaining hard at larger scales.

  39. Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations

    cs.LG 2026-05 unverdicted novelty 7.0

    Floating-point neural networks achieve universal representability for practical activations like ReLU, sigmoid, and tanh under arbitrary reduction orders and bounded ulp errors in activations via a new distinguishabil...

  40. HumanFlow -- Diffusion-Driven MAV Navigation Among Humans via Tightly-Coupled Motion Tracking, Forecasting, and Control

    cs.RO 2026-05 unverdicted novelty 7.0

    HumanFlow is a latent diffusion model for unified human motion tracking and forecasting in 3D scenes, tightly coupled via flow-matching MPC to an approximate policy for MAV social navigation that outperforms prior met...

  41. Word Class Representations Spontaneously Emerge from Successor Representations Trained on Natural Language

    cs.CL 2026-05 unverdicted novelty 7.0

    Successor representation training on natural language causes part-of-speech categories to emerge spontaneously in the learned embeddings, with structure varying by predictive horizon.

  42. Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork

    cs.AI 2026-05 unverdicted novelty 7.0

    New benchmark ICRL4AHT reveals that history-conditioned ICRL methods fail to show robust adaptation in multi-agent Overcooked-V2, underperforming random baselines on unseen teammates and layouts.

  43. Generative Representation Learning on Hyper-relational Knowledge Graphs via Masked Discrete Diffusion

    cs.LG 2026-05 unverdicted novelty 7.0

    KREPE applies masked discrete diffusion to unify link prediction and fact generation on hyper-relational knowledge graphs, claiming SOTA results and superiority over LLM baselines.

  44. Classical State Preparation for Variational Quantum Algorithms via Reinforcement Learning

    quant-ph 2026-05 unverdicted novelty 7.0

    CRiSP uses neural-guided MCTS and curriculum learning to insert Clifford prefixes before parameterized rotations in VQAs, yielding mean 3.17x and max 45x gains in energy accuracy on 22-qubit QAOA benchmarks versus pri...

  45. Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learning

    cs.LG 2026-05 unverdicted novelty 7.0

    Proposes latent analogies and analogy transduction to enable compositional generalization to unseen goal-context pairs in offline GCRL, outperforming trajectory-stitching baselines on manipulation tasks.

  46. Nested-GPT for variable-multiplicity parton showers: A case study in the resummation of non-global logarithms

    hep-ph 2026-05 unverdicted novelty 7.0

    Nested-GPT is an autoregressive Transformer surrogate that generates variable-multiplicity parton showers while enforcing ordered Markovian branching and matches reference Monte Carlo results for leading-log non-globa...

  47. Nested-GPT for variable-multiplicity parton showers: A case study in the resummation of non-global logarithms

    hep-ph 2026-05 unverdicted novelty 7.0

    Nested-GPT is an autoregressive Transformer that dynamically generates variable-multiplicity parton showers matching Monte Carlo references for non-global logarithm resummation in the large-Nc limit.

  48. Bug or Feature$^2$: Weight Drift, Activation Sparsity and Spikes

    cs.LG 2026-05 accept novelty 7.0

    Standard losses induce negative weight drift with positive-biased activations, producing up to 90% sparsity in GPT-nano and an accuracy cliff above ~70% sparsity; clipped ReLU² and GELU² improve the tradeoff.

  49. Bug or Feature$^2$: Weight Drift, Activation Sparsity and Spikes

    cs.LG 2026-05 accept novelty 7.0

    The paper proves negative weight drift at initialization under MSE or cross-entropy with asymmetric activations, links it to up to 90% sparsity in GPT-nano, maps the sparsity-accuracy cliff across 79 configurations, a...

  50. GQA-{\mu}P: The maximal parameterization update for grouped query attention

    cs.LG 2026-05 unverdicted novelty 7.0

    Derives μP scalings for GQA via promoted spectral-norm definition of feature learning and a modified norm preserving scaling laws for non-full-rank matrices, with experiments showing learning-rate transfer.

  51. A Mutual Information Lower Bound for Multimodal Regression Active Learning

    cs.LG 2026-05 unverdicted novelty 7.0

    Derives MI-LB acquisition function from mutual information in a two-index epistemic-aleatoric framework and shows it outperforms baselines on multimodal regression benchmarks.

  52. Training-Free Generative Sampling via Moment-Matched Score Smoothing

    stat.ML 2026-05 unverdicted novelty 7.0

    MM-SOLD is a training-free particle sampler whose large-particle limit converges to a moment-matched Gibbs distribution obtained by exponentially tilting a score-smoothed target.

  53. Parallel Scan Recurrent Neural Quantum States for Scalable Variational Monte Carlo

    cond-mat.str-el 2026-05 conditional novelty 7.0

    PSR-NQS makes recurrent neural quantum states scalable for variational Monte Carlo by using parallel scan recurrence, reaching accurate results on 52x52 two-dimensional lattices.

  54. Graph Neural Networks with Triangle-Based Messages for the Multicut Problem

    cs.LG 2026-05 unverdicted novelty 7.0

    A triangle-message GNN for multicut outperforms heuristics in solution quality on graphs up to 200 nodes and finds optimal solutions faster than exact solvers for some cases.

  55. Pretraining Strategies and Scaling for ECG Foundation Models: A Systematic Study

    eess.SP 2026-05 unverdicted novelty 7.0

    Contrastive predictive coding pretraining combined with structured state space models yields the strongest ECG foundation models, with continued gains from scaling data to 11 million samples.

  56. Disentangled Sparse Representations for Concept-Separated Diffusion Unlearning

    cs.LG 2026-05 unverdicted novelty 7.0

    SAEParate disentangles sparse representations in diffusion models via contrastive clustering and nonlinear encoding to enable more precise concept unlearning with reduced side effects.

  57. Neural Quantum Spectral Operator Learning for Solving Partial Differential Equations

    quant-ph 2026-05 unverdicted novelty 7.0

    NVQLS introduces the first hybrid quantum-classical unsupervised operator learning method for parametric PDEs via Legendre-Galerkin weak form, sign ambiguity resolution, and neural embedding.

  58. Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining

    cs.CL 2026-05 unverdicted novelty 7.0

    Temporarily reducing the learning rate on upper-layer query and key projections during early GPT pretraining prevents premature attention specialization and improves model performance.

  59. From Holo Pockets to Electron Density: GPT-style Drug Design with Density

    cs.AI 2026-05 unverdicted novelty 7.0

    EDMolGPT generates drug-like molecules from low-resolution electron density point clouds of holo binding pockets and shows effectiveness across 101 biological targets.

  60. Zero-Shot Neural Network Evaluation with Sample-Wise Activation Patterns

    cs.LG 2026-05 unverdicted novelty 7.0

    SWAP-Score evaluates neural networks without training by quantifying sample-wise activation patterns, achieving high correlation with true performance on CIFAR-10 for CNNs and GLUE for Transformers while enabling fast NAS.

Reference graph

Works this paper leans on

50 extracted references · 50 canonical work pages · cited by 363 Pith papers

  1. [1]

    Adaptive dropout for training deep neural networks , year =

    Jimmy Ba and Brendan Frey , booktitle =. Adaptive dropout for training deep neural networks , year =

    work page

  2. [2]

    Learning with pseudo-ensembles , year =

    Philip Bachman and Ouais Alsharif and Doina Precup , booktitle =. Learning with pseudo-ensembles , year =

    work page

  3. [3]

    International Conference on Learning Representations , title =

    Djork. International Conference on Learning Representations , title =

    work page

  4. [4]

    A Simple Approximation to the Area Under Standard Normal Curve , year =

    Amit Choudhury , booktitle =. A Simple Approximation to the Area Under Standard Normal Curve , year =

    work page

  5. [5]

    Natural Neural Networks , year =

    Guillaume Desjardins and Karen Simonyan and Razvan Pascanu and Koray Kavukcuoglu , booktitle =. Natural Neural Networks , year =

    work page

  6. [6]

    Smith , publisher =

    Kevin Gimpel and Nathan Schneider and Brendan O ' Connor and Dipanjan Das and Daniel Mills and Jacob Eisenstein and Michael Heilman and Dani Yogatama and Jeffrey Flanigan and Noah A. Smith , publisher =. Part-of-Speech Tagging for

    work page

  7. [7]

    Adjusting for Dropout Variance in Batch Normalization and Weight Initialization , year =

    Dan Hendrycks and Kevin Gimpel , booktitle =. Adjusting for Dropout Variance in Batch Normalization and Weight Initialization , year =

    work page

  8. [8]

    Neural networks and physical systems with emergent collective computational abilities , year =

    John Hopfield , booktitle =. Neural networks and physical systems with emergent collective computational abilities , year =

    work page

  9. [9]

    Bayesian Active Learning for Classification and Preference Learning , year =

    Neil Houlsby and Ferenc Husz\'ar and Zoubin Ghahramani and M\'at\'e Lengyel , booktitle =. Bayesian Active Learning for Classification and Preference Learning , year =

    work page

  10. [10]

    Adam: A Method for Stochastic Optimization , year =

    Diederik Kingma and Jimmy Ba , publisher =. Adam: A Method for Stochastic Optimization , year =

    work page

  11. [11]

    Alex Krizhevsky , publisher =

    work page

  12. [12]

    Zoneout: Regularizing

    David Krueger and Tegan Maharaj and János Kram\'ar and Mohammad Pezeshki and Nicolas Ballas and Nan Rosemary Ke1 and Anirudh Goyal and Yoshua Bengio and Hugo Larochelle and Aaron Courville and Chris Pal , booktitle =. Zoneout: Regularizing

    work page

  13. [13]

    Ilya Loshchilov and Frank Hutter , journal =

    work page

  14. [14]

    Maas and Awni Y

    Andrew L. Maas and Awni Y. Hannun and and Andrew Y. Ng , booktitle =. Rectifier nonlinearities improve neural network acoustic models , year =

    work page

  15. [15]

    McCulloch and Walter Pitts , booktitle =

    Warren S. McCulloch and Walter Pitts , booktitle =. A logical calculus of the ideas immanent in nervous activity , year =

    work page

  16. [16]

    All You Need Is a Good Init , year =

    Dmytro Mishkin and Jiri Matas , booktitle =. All You Need Is a Good Init , year =

    work page

  17. [17]

    Dahl and Geoffrey E

    Abdelrahman Mohamed and George E. Dahl and Geoffrey E. Hinton , booktitle =. Acoustic Modeling Using Deep Belief Networks , year =

    work page

  18. [18]

    Smith , title =

    Olutobi Owoputi and Brendan O'Connor and Chris Dyer and Kevin Gimpel and Nathan Schneider and Noah A. Smith , title =. North American Chapter of the Association for Computational Linguistics (NAACL) , year =

    work page

  19. [19]

    Hinton , booktitle =

    Vinod Nair and Geoffrey E. Hinton , booktitle =. Rectified Linear Units Improve Restricted Boltzmann Machines , year =

    work page

  20. [20]

    Kingma , booktitle =

    Tim Salimans and Diederik P. Kingma , booktitle =. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks , year =

    work page

  21. [21]

    Saxe and James L

    Andrew M. Saxe and James L. McClelland and Surya Ganguli , booktitle =. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks , year =

    work page

  22. [22]

    Deep Residual Networks with Exponential Linear Unit , year =

    Anish Shah and Sameer Shinde and Eashan Kadam and Hena Shah and Sandip Shingade , booktitle =. Deep Residual Networks with Exponential Linear Unit , year =

    work page

  23. [23]

    Hinton and Alex Krizhevsky and Ilya Sutskever and Ruslan Salakhutdinov , booktitle =

    Nitish Srivastava and Geoffrey E. Hinton and Alex Krizhevsky and Ilya Sutskever and Ruslan Salakhutdinov , booktitle =. Dropout: A Simple Way to Prevent Neural Networks from Overfitting , year =

    work page

  24. [24]

    Improving Neural Networks with Dropout , year =

    Nitish Srivastava , booktitle =. Improving Neural Networks with Dropout , year =

    work page

  25. [25]

    Wang and Christopher D

    Sida I. Wang and Christopher D. Manning , booktitle =. Fast dropout training , year =

    work page

  26. [26]

    Wide Residual Networks , year =

    Sergey Zagoruyko and Nikos Komodakis , journal =. Wide Residual Networks , year =

    work page

  27. [27]

    Adaptive dropout for training deep neural networks

    Jimmy Ba and Brendan Frey. Adaptive dropout for training deep neural networks. In Neural Information Processing Systems, 2013

    work page 2013

  28. [28]

    Learning with pseudo-ensembles

    Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Neural Information Processing Systems, 2014

    work page 2014

  29. [29]

    A simple approximation to the area under standard normal curve

    Amit Choudhury. A simple approximation to the area under standard normal curve. In Mathematics and Statistics, 2014

    work page 2014

  30. [30]

    Fast and accurate deep network learning by exponential linear units ( ELUs )

    Djork - Arn \' e Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units ( ELUs ). In International Conference on Learning Representations, 2016

    work page 2016

  31. [31]

    Natural neural networks

    Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, and Koray Kavukcuoglu. Natural neural networks. In arXiv, 2015

    work page 2015

  32. [32]

    Kevin Gimpel, Nathan Schneider, Brendan O ' Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. Part-of-Speech Tagging for T witter: Annotation, Features, and Experiments . Association for Computational Linguistics (ACL), 2011

    work page 2011

  33. [33]

    Adjusting for dropout variance in batch normalization and weight initialization

    Dan Hendrycks and Kevin Gimpel. Adjusting for dropout variance in batch normalization and weight initialization. In arXiv, 2016

    work page 2016

  34. [34]

    Neural networks and physical systems with emergent collective computational abilities

    John Hopfield. Neural networks and physical systems with emergent collective computational abilities. In Proceedings of the National Academy of Sciences of the USA, 1982

    work page 1982

  35. [35]

    Adam: A Method for Stochastic Optimization

    Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. International Conference for Learning Representations, 2015

    work page 2015

  36. [36]

    Learning Multiple Layers of Features from Tiny Images, 2009

    Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images, 2009

    work page 2009

  37. [37]

    Zoneout: Regularizing RNNs by randomly preserving hidden activations

    David Krueger, Tegan Maharaj, János Kram\'ar, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke1, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, Aaron Courville, and Chris Pal. Zoneout: Regularizing RNNs by randomly preserving hidden activations. In Neural Information Processing Systems, 2016

    work page 2016

  38. [38]

    SGDR : Stochastic gradient descent with restarts

    Ilya Loshchilov and Frank Hutter. SGDR : Stochastic gradient descent with restarts. arXiv, 2016

    work page 2016

  39. [39]

    Maas, Awni Y

    Andrew L. Maas, Awni Y. Hannun, , and Andrew Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In International Conference on Machine Learning, 2013

    work page 2013

  40. [40]

    McCulloch and Walter Pitts

    Warren S. McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. In Bulletin of Mathematical Biophysics, 1943

    work page 1943

  41. [41]

    All you need is a good init

    Dmytro Mishkin and Jiri Matas. All you need is a good init. In International Conference on Learning Representations, 2016

    work page 2016

  42. [42]

    Dahl, and Geoffrey E

    Abdelrahman Mohamed, George E. Dahl, and Geoffrey E. Hinton. Acoustic modeling using deep belief networks. In IEEE Transactions on Audio, Speech, and Language Processing, 2012

    work page 2012

  43. [43]

    Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning, 2010

    work page 2010

  44. [44]

    Olutobi Owoputi, Brendan O'Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A. Smith. Improved part-of-speech tagging for online conversational text with word clusters. In North American Chapter of the Association for Computational Linguistics (NAACL), 2013

    work page 2013

  45. [45]

    Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Neural Information Processing Systems, 2016

    work page 2016

  46. [46]

    Saxe, James L

    Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations, 2014

    work page 2014

  47. [47]

    Deep residual networks with exponential linear unit

    Anish Shah, Sameer Shinde, Eashan Kadam, Hena Shah, and Sandip Shingade. Deep residual networks with exponential linear unit. In Vision Net, 2016

    work page 2016

  48. [48]

    Improving neural networks with dropout

    Nitish Srivastava. Improving neural networks with dropout. In University of Toronto, 2013

    work page 2013

  49. [49]

    Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov

    Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. In Journal of Machine Learning Research, 2014

    work page 2014

  50. [50]

    Wide residual networks

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. British Machine Vision Conference, 2016

    work page 2016