Pith. sign in

REVIEW 4 cited by

Learning Activation Functions to Improve Deep Neural Networks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1412.6830 v3 pith:5BGH6R37 submitted 2014-12-21 cs.NE cs.CVcs.LGstat.ML

Learning Activation Functions to Improve Deep Neural Networks

Forest Agostinelli , Matthew Hoffman , Peter Sadowski , Pierre Baldi This is my paper
classification cs.NE cs.CVcs.LGstat.ML
keywords activationfunctionneuraldeepimprovelinearnetworksneuron
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural network architectures composed of static rectified linear units, achieving state-of-the-art performance on CIFAR-10 (7.51%), CIFAR-100 (30.83%), and a benchmark from high-energy physics involving Higgs boson decay modes.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Searching for Activation Functions

    cs.NE 2017-10 conditional novelty 7.0

    Automated search discovers Swish activation f(x) = x * sigmoid(βx) that improves top-1 ImageNet accuracy over ReLU by 0.9% on Mobile NASNet-A and 0.6% on Inception-ResNet-v2.

  2. More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations

    cs.LG 2026-05 unverdicted novelty 6.0

    Mixture of Activations mixes activation functions token-adaptively in FFNs via lightweight gates, strictly more expressive than fixed or learnable activations, and yields lower pretraining loss from 0.12B to 2B models.

  3. Competing nonlinearities, criticality, and order-to-chaos transition in deep networks

    cond-mat.dis-nn 2026-05 unverdicted novelty 6.0

    A statistical mixture of Tanh and Swish activations with critical mixing fraction p_c induces a continuous phase transition to scale-invariant signal propagation in deep networks while preserving smoothness.

  4. Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning

    cs.LG 2019-07 unverdicted novelty 5.0

    Graph Laplacian interpolating activation replaces softmax in DNNs and improves natural accuracy, robust accuracy, and data efficiency.