An overview of gradient descent optimization algorithms
read the original abstract
Gradient descent optimization algorithms, while increasingly popular, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by. This article aims to provide the reader with intuitions with regard to the behaviour of different algorithms that will allow her to put them to use. In the course of this overview, we look at different variants of gradient descent, summarize challenges, introduce the most common optimization algorithms, review architectures in a parallel and distributed setting, and investigate additional strategies for optimizing gradient descent.
This paper has not been read by Pith yet.
Forward citations
Cited by 60 Pith papers
-
Coherent-State Propagation: A Computational Framework for Simulating Bosonic Quantum Systems
Coherent-state propagation enables quasi-polynomial classical simulation of bosonic circuits with logarithmically many Kerr gates at exponentially small trace-distance error, with polynomial runtime in the weak-nonlin...
-
MAOAM: Unified Object and Material Selection with Vision-Language Models
MAOAM unifies object and material selection via a VLM with segmentation head, supporting text and click interactions through multi-task training on VLM-generated material data.
-
Unified High-Probability Analysis of Stochastic Variance-Reduced Estimation
A unified recursion framework for stochastic variance-reduced estimation yields high-probability bounds and the first Õ(ε^{-3}) oracle complexity for stochastic optimization with expectation constraints.
-
Revisiting Shadow Detection from a Vision-Language Perspective
SVL uses language embeddings aligned with global image representations via shadow ratio regression and global-to-local coupling to improve shadow detection robustness in ambiguous cases.
-
Locally Near Optimal Piecewise Linear Regression in High Dimensions via Difference of Max-Affine Functions
ABGD parametrizes piecewise linear functions as difference of max-affine functions and converges linearly to an epsilon-accurate solution with O(d max(sigma/epsilon,1)^2) samples under sub-Gaussian noise, which is min...
-
Efficient classical training of model-free quantum photonic reservoir
Classical light training of photonic quantum reservoirs enables accurate model-free estimation of single-qubit observables and two-qubit entanglement witnesses on unseen quantum states.
-
Large Spikes in Stochastic Gradient Descent: A Large-Deviations View
Large loss spikes in SGD are polynomially likely and serve as the dominant mechanism for escaping sharp minima toward flatter solutions in the NTK regime.
-
Debate-Enhanced Pseudo Labeling and Frequency-Aware Progressive Debiasing for Weakly-Supervised Camouflaged Object Detection with Scribble Annotations
D³ETOR combines debate-enhanced pseudo labeling from SAM with frequency-aware progressive debiasing in FADeNet to achieve state-of-the-art weakly-supervised camouflaged object detection using scribbles.
-
AMIGO: a Data-Driven Calibration of the JWST Interferometer
AMIGO is an end-to-end differentiable forward model of JWST AMI that corrects detector systematics to recover high-precision astrometry and detect close high-contrast companions.
-
An Operational Framework for Nonclassicality in Quantum Communication Networks
A variational optimization framework computes linear classical bounds on network input/output probabilities whose violation certifies nonclassicality, finding entanglement necessary for nonclassicality in single-sende...
-
Residual-Guided Expert Specialization for Incomplete Multimodal Learning
MARS is a residual-guided MoE framework for incomplete multimodal learning that uses training-time residuals to train a deployable feature router and adds discrepancy-aware noise regularization.
-
Supervised Hebbian learning in Deep Counterstream Associative Networks
Supervised counterstream Hebbian learning in deep associative networks reaches high accuracy on binarized MNIST by propagating opposing activity waves linked via local rules.
-
Anti-Collapse Dynamics and the Emergence of Multi-Time-Scale Learning in Recurrent Neural Networks
RNNs can sustain power-law forgetting and multi-time-scale learning when heavy-tailed fluctuations in SGD balance the collapse tendency toward short time scales, governed by a spectral exponent β.
-
Stochastic Gradient Optimization with Model-Assisted Sampling
Model-assisted sampling applies survey sampling variance reduction with auxiliary predictors to stochastic gradients, yielding empirical gains on benchmarks and faster generalization with AdamW.
-
A welding penetration prediction model for laser welding process based on self-supervised learning using physics-informed neural networks
SimPhysNet achieves 96.06% accuracy classifying laser welding penetration states using self-supervised contrastive learning with a physics-informed neural network and prototypical networks on only 200 labeled images.
-
Quantifying the Agreement Between Data-Influence and Data-Similarity to Understand LLM Behavior
Data-similarity and data-influence produce significantly overlapping rankings of training documents for LLM outputs, with asymmetry allowing a favorable cost-accuracy trade-off.
-
Stationary Robust Mean-Field Games under Model Mismatches
Develops infinite-horizon stationary robust mean-field games incorporating distributional uncertainty, proves equilibrium existence via fixed-point on contractive Bellman operator, gives convergent algorithm, and deri...
-
Optical Implementation of Equilibrium Propagation Using Spatial Photonic Ising Machines
Hybrid optical implementation of equilibrium propagation via spatial photonic Ising machine demonstrated on Wine classification with numerical MNIST evaluation.
-
A stochastic gradient algorithm for non-separable optimization with convergence guarantee
Presents a stochastic gradient algorithm for non-separable optimization with local convergence guarantees under smoothness assumptions.
-
Integrable Elasticity via Neural Demand Potentials
ICDN is a neural network that models log-demand from log-prices so elasticities can be derived exactly by differentiation, showing better out-of-sample performance than log-log benchmarks on beer sales data.
-
No-regret optimization of time-varying bilevel problems
W-SparQ-BL models time-varying lower-level responses with multi-output GPs and sparse approximations to achieve sublinear dynamic regret in bilevel optimization under noise.
-
Multi-scale Coarse-to-fine Modeling for Test-time Human Motion Control
MSCoT uses multi-scale hierarchical token prediction, multi-scale guidance, and a token refiner to deliver SOTA text-to-motion control with 48% FID gain, 61% lower error, and 10x faster inference on HumanML3D.
-
Revisiting Shadow Detection from a Vision-Language Perspective
SVL uses vision-language alignment via scene-level shadow ratio regression and global-to-local coupling on a frozen DINOv3 encoder to disambiguate shadows from dark surfaces in dense prediction.
-
UniISP: A Unified ISP Framework for Both Human and Machine Vision
UniISP unifies ISP processing with a Hybrid Attention Module and Feature Adapter to produce images that are both visually pleasing for humans and informative for computer vision models.
-
Hyperspectral Anomaly Detection Using Einstein Fuzzy Computing and Quantum Neural Network
HyFuHAD fuses classical Einstein fuzzy detection from multiple membership functions with quantum fuzzy detection to achieve claimed state-of-the-art performance in unsupervised hyperspectral anomaly detection.
-
BOOOM: Loss-Function-Agnostic Black-Box Optimization over Orthonormal Manifolds for Machine Learning and Statistical Inference
BOOOM parametrizes Stiefel manifold optimization into Euclidean angle space using global Givens rotations and solves it with recursive modified pattern search for loss-agnostic black-box problems.
-
Efficient optimisation of multi-parameter quantum control protocols for strongly-coupled systems
Gradient-based optimization of SUPER and FTPE pulse protocols via auto-differentiation and uniTEMPO yields higher preparation fidelities than resonant pi-pulses or standard two-photon excitation, with the advantage in...
-
Co-evolving Agent Architectures and Interpretable Reasoning for Automated Optimization
EvoOR-Agent co-evolves agent architectures as AOE-style networks with graph-mediated recombination and knowledge-base-assisted mutation to outperform fixed LLM pipelines on OR benchmarks.
-
Material-Agnostic Zero-Shot Thermal Inference for Metal Additive Manufacturing via a Parametric PINN Framework
A decoupled parametric PINN with conditional modulation and Rosenthal-derived output scaling achieves zero-shot thermal inference across arbitrary metal alloys in laser powder bed fusion.
-
Learning by training: emergent return-point memory from cyclically tuning disordered sphere packings
Cyclic inverse design on athermal disordered sphere packings produces an emergent marginally absorbing manifold that encodes return-point memory of the training range through gradient discontinuities.
-
Time-Scale Coupling Between States and Parameters in Recurrent Neural Networks
Gating in RNNs couples state time-scales with parameter gradients to produce lag- and direction-dependent effective learning rates, shown via exact Jacobians and first-order expansion.
-
Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps
Diffusion models improve generation quality via inference-time search over noise candidates guided by verifiers and algorithms, yielding gains beyond denoising step scaling on class- and text-conditioned benchmarks.
-
Secret Leak Detection in Software Issue Reports using LLMs: A Comprehensive Evaluation
Creates a 54k-instance benchmark of GitHub issue secrets and shows fine-tuned LLMs reach 94.49% F1 with 81.6% on 178 real repositories.
-
Personalization as Inverse Planning: Learning Latent Design Intents for Agentic Slide Generation via Structural Denoising
SPIRE approximates page-level slide personalization by training agents to denoise corrupted slide structures via collaborative RL, claiming a proof of consistency as a surrogate for inverse planning.
-
HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression
HARD-KV bridges dynamic head-adaptive KV cache compression with static inference engine constraints via Cascade Cache and Logits Calibration, reporting up to 2x throughput gains on long-context math benchmarks.
-
Exploring Dualistic Meta-Learning to Enhance Domain Generalization in Open Set Scenarios
MEDIC uses dualistic meta-learning with joint domain-class matching to balance decision boundaries in open set domain generalization.
-
Gradient-Descent Steps to Success over Mean Accuracy: A Paradigm Shift for ML
The paper defines computational effort as the number of gradient descent steps to reach target accuracy with high probability, shows large learning rates minimize this effort across models, and identifies phase transi...
-
Central limit theorem for the averaged Adam optimizer
Establishes a central limit theorem for averaged Adam with n^{-1/2} convergence rate to an attracting zero and covariance determined by the algorithm at the attractor.
-
Emotional regulation improves deep learning-based image classification
Emotional Regulation pre-trains ResNet and ViT on four emotional datasets to improve performance on CIFAR-10 and CIFAR-100, claiming new state-of-the-art results in emotion-augmented deep learning.
-
Trace-Mediated Peak Bias: Bridging Temporal Credit Assignment and Cognitive Heuristics in Deep Reinforcement Learning
Eligibility traces in deep RL create a peak bias by amplifying distal TD errors into gradient shocks that fixed-step SGD cannot normalize, leading to overestimation of peak-reward trajectories and a mechanistic accoun...
-
Text-attributed Graph Condensation via Text Selection and Attribute Matching
TAGSAM is a graph condensation method for text-attributed graphs that uses subgraph text selection and attribute similarity matching, claiming 4.9% average accuracy gain over baselines at fixed size and competitive pe...
-
A Residual-Subspace Constraint Framework for Fourier Ptychographic Microscopy
RSCF decouples low-rank systematic residuals from noise via subspace constraints to enable high-fidelity phase and amplitude recovery in Fourier ptychographic microscopy under aberrations and misalignments.
-
Getting large-scale quantum neural networks ready for quantum hardware
Physics-informed quantum neural networks trained on noisy measurements can construct nontrivial decision boundaries to classify quantum states via order parameters and are suited for NISQ hardware due to links with Ma...
-
Micro-DualNet: Dual-Path Spatio-Temporal Network for Micro-Action Recognition
Micro-DualNet employs dual ST and TS pathways with entity-level adaptive routing and Mutual Action Consistency loss to achieve competitive results on MA-52 and state-of-the-art on iMiGUE for micro-action recognition.
-
Enhancing Circuit Fidelity in Transmon Qubit Rings via Operation Duration Tuning under Strong Connectivity Noise
Tuning gate durations in noisy transmon qubit rings raises fidelity, with local maxima appearing under strong noise and an ML model predicting optima for new hardware.
-
Two-Stage Framework for Efficient UAV-Based Wildfire Video Analysis with Adaptive Compression and Fire Source Detection
A two-stage UAV framework prunes redundant wildfire video clips via a policy network with station point mechanism and detects fire sources in real time using an improved YOLOv8 model.
-
A Physics-Inspired Optimizer: Velocity Regularized Adam
VRAdam hybridizes Adam's per-parameter adaptation with a physics-inspired velocity regularizer to stabilize training at the edge of stability, delivering better empirical performance than AdamW and O(ln(N)/sqrt(N)) co...
-
Cloudless-Training: A Framework to Improve Efficiency of Geo-Distributed ML Training
Cloudless-Training proposes a two-layer serverless framework with elastic scheduling and two new synchronization strategies (ASGD-GA and inter-PS model averaging) that reports 9.2-24% cost reduction and up to 1.7x spe...
-
On Using Machine Learning to Identify Knowledge in API Reference Documentation
Supervised classifiers (SVM, k-NN, deep learning) trained on 5,574 annotated Java/.NET API documents achieve up to 87% AUPRC for single knowledge types and 79% MacroAUC for multi-label classification, with partial gen...
-
signADAM: Learning Confidences for Deep Neural Networks
signADAM and signADAM++ are new first-order optimizers that incorporate sign operations and a confidence-based sparsity mechanism, with claimed empirical superiority and theoretical convergence over ADAM and sign-base...
-
Deployable probabilistic programming
Design guidelines and a Go library (Infergo) for deploying probabilistic programming in production systems, with benchmark comparisons.
-
Accurate, Efficient, and Explainable Deep Learning Approaches for Environmental Science Problems
The work introduces WaLeF/FIDLAr for flood forecasting, CoDiCast for probabilistic weather, and Hypercube-RAG for explainable environmental QA, claiming superior accuracy, efficiency, and interpretability over baselines.
-
Accelerated Gradient Descent for Faster Convergence with Minimal Overhead
CT-AGD accelerates first-order optimization in deep learning by using finite-difference curvature estimates and noise-mitigation heuristics, achieving equivalent accuracy with 33% fewer training epochs and overhead co...
-
Beyond Bell Teleportation: Machine-Learned Adaptive Protocols
Machine learning optimizes adaptive quantum teleportation protocols to achieve higher fidelity than standard Bell-state methods in several noise models.
-
Advancing Direct Training for Spiking Neural Networks with Circulate-Firing Neurons and Learnable Gradients
Introduces circulate-firing neurons, time-step-wise learnable surrogate gradients, and balanced loss for direct SNN training, reporting competitive results on datasets and Transformers.
-
Convergence rates for gradient descent in the training of overparameterized artificial neural networks with piecewise affine activation
Batch gradient descent achieves linear convergence to zero MSE with high probability for sufficiently wide shallow NNs with non-affine piecewise affine activations and distinct inputs.
-
Incremental Concept Learning via Online Generative Memory Recall
Pseudo-rehearsal method with cGAN-generated old-concept samples, balanced online recall, and concept contrastive loss for class-incremental learning on MNIST, Fashion-MNIST and SVHN.
-
A CNN-Based Super-Resolution Technique for Active Fire Detection on Sentinel-2 Data
CNN super-resolution enhances Sentinel-2 SWIR bands to 10m, yielding better active fire detection than prior methods on Vesuvius test data.
-
To each route its own ETA: A generative modeling framework for ETA prediction
A route-specific deep generative model learns the probability distribution of bus trip ETAs from historical data alone and conditions updates on real-time trip progress.
-
Explaining Machine Learning and Memorization with Statistical Mechanics
Thesis uses statistical mechanics to study DAM and RBM models for understanding memorization, low-dimensional learning, and adversarial robustness in neural networks.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.