REVIEW 41 cited by
ADADELTA: An Adaptive Learning Rate Method
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
ADADELTA: An Adaptive Learning Rate Method
read the original abstract
We present a novel per-dimension learning rate method for gradient descent called ADADELTA. The method dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent. The method requires no manual tuning of a learning rate and appears robust to noisy gradient information, different model architecture choices, various data modalities and selection of hyperparameters. We show promising results compared to other methods on the MNIST digit classification task using a single machine and on a large scale voice dataset in a distributed cluster environment.
Forward citations
Cited by 41 Pith papers
-
Neural Machine Translation of Rare Words with Subword Units
Subword segmentation via byte pair encoding enables open-vocabulary neural machine translation and improves BLEU scores by 1.1 on English-German and 1.3 on English-Russian WMT 2015 tasks over dictionary back-off baselines.
-
Neural Machine Translation by Jointly Learning to Align and Translate
An attention-based encoder-decoder model achieves English-to-French translation performance comparable to phrase-based systems by automatically learning soft alignments.
-
Adam: A Method for Stochastic Optimization
A first-order stochastic optimizer that maintains bias-corrected exponential moving averages of the gradient and its square, dividing the former by the square root of the latter to set per-parameter step sizes.
-
Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters
Spectral clipping of leading singular values in gradient matrices stabilizes SGD for non-convex problems with heavy-tailed noise and achieves the optimal convergence rate O(K^{(2-2α)/(3α-2)}).
-
When Descent Is Too Stable: Event-Triggered Hamiltonian Learning to Optimize
SHAPE lifts gradient descent to an augmented phase space with a learned Hamiltonian vector field and event-triggered port updates to balance descent, exploitation, and exploration, improving best-so-far performance ov...
-
Universal Adaptive Proximal Gradient Methods via Gradient Mapping Accumulation
A universal adaptive proximal gradient method converges at rates matching standard proximal gradient methods up to logarithmic factors for three problem classes without requiring knowledge of problem parameters.
-
On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization
Momentum SGD incurs a provable drift-amplification penalty in nonstationary stochastic optimization that makes it worse than vanilla SGD in drift-dominated regimes, confirmed by finite-time upper bounds and minimax lo...
-
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
RNN Encoder-Decoder learns semantically meaningful phrase representations whose conditional probabilities improve statistical machine translation when added to log-linear models.
-
bAdag: an adaptive block coordinate gradient method for smooth nonconvex functions
Introduces bAdag, an AdaGrad-based block coordinate gradient method with ergodic sublinear convergence proofs for smooth nonconvex objectives under block Lipschitz gradient assumptions, covering cyclic, uniform random...
-
A stochastic gradient algorithm for non-separable optimization with convergence guarantee
Presents a stochastic gradient algorithm for non-separable optimization with local convergence guarantees under smoothness assumptions.
-
Voronoi-Elitism Genetic Algorithm: A Generic Derivative-Free Routine With Theory and Implementation for Statistical Optimization
VEGA is a derivative-free genetic algorithm that constructs Voronoi neighborhoods around retained elite candidates to balance exploitation and exploration in high-dimensional statistical optimization.
-
Global Convergence of Sampling-Based Nonconvex Optimization through Diffusion-Style Smoothing
Recasts sampling-based nonconvex optimization as smoothed gradient descent to obtain non-asymptotic convergence guarantees and introduces the DIDA annealed algorithm that converges to the global optimum.
-
Auto-Conditioned Frank-Wolfe Algorithms
The paper proposes an auto-conditioned framework for Frank-Wolfe algorithms that replaces global smoothness constants with local estimators computed from first-order information, achieving convergence to stationary po...
-
\mathsf{VISTA}: Decentralized Machine Learning in Adversary Dominated Environments
VISTA adaptively tunes consistency thresholds in decentralized SGD so that the system converges asymptotically like standard SGD even when adversaries dominate the worker pool.
-
Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less
Full finetuning with the pretraining optimizer reduces forgetting compared to other optimizers or LoRA while achieving comparable new-task performance.
-
Cross-Attention End-to-End ASR for Two-Party Conversations
End-to-end ASR model with speaker-specific cross-attention for two-party conversations outperforms standard models on the Switchboard corpus.
-
Gated Embeddings in End-to-End Speech Recognition for Conversational-Context Fusion
Gated fusion of fastText and BERT embeddings into an end-to-end ASR model captures multi-sentence conversational context and lowers word error rate on the Switchboard corpus.
-
SGDR: Stochastic Gradient Descent with Warm Restarts
SGDR uses periodic warm restarts of the learning rate in SGD to reach new state-of-the-art error rates of 3.14% on CIFAR-10 and 16.21% on CIFAR-100.
-
PAPA: Online Personalized Active Preference Alignment
PAPA directly optimizes diffusion models via real-time user feedback for personalized preference alignment, drawing from variational inference, with an efficiency-enhanced variant EPAPA.
-
Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning
AdaNAGED combines zeroth-order gradient-free training, automatic parameter adaptation, and LMO-based non-Euclidean geometry with claimed convergence guarantees, demonstrated on OPT-1.3B fine-tuning.
-
Pareto-Guided Teacher Alignment for Fair Personalized Text Generation
Fairness mitigation in personalized text generation is objective-dependent with methods occupying different regions of the fairness-personalization Pareto frontier rather than any single strategy dominating all objectives.
-
Auto-Conditioned Frank-Wolfe Algorithms
Auto-conditioned Frank-Wolfe methods use local Lipschitz estimators from first-order information to achieve convergence guarantees in convex and nonconvex settings without prior global smoothness knowledge.
-
Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
Pion is an optimizer that preserves the singular values of weight matrices in LLM training by applying orthogonal equivalence transformations.
-
NeuroPlastic: A Plasticity-Modulated Optimizer for Biologically Inspired Learning Dynamics
NeuroPlastic is a gradient-based optimizer augmented with a multi-signal plasticity modulation mechanism that improves performance over standard updates on image classification tasks, especially in low-data regimes.
-
Harmonizing MR Images Across 100+ Scanners: Multi-site Validation with Traveling Subjects and Real-world Protocols
HACA3^+ improves upon HACA3 with better artifact encoding, attention mechanisms, and training on 100+ scanners, validated via traveling subjects for better downstream performance.
-
Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training
Proposes low-rank orthogonalization and derives low-rank Muon and MSGD variants that outperform standard Muon on GPT-2 and LLaMA pretraining while providing iteration complexity bounds.
-
signADAM: Learning Confidences for Deep Neural Networks
signADAM and signADAM++ are new first-order optimizers that incorporate sign operations and a confidence-based sparsity mechanism, with claimed empirical superiority and theoretical convergence over ADAM and sign-base...
-
Quality-aware skill translation models for expert finding on StackOverflow
Translation models and quality-aware scoring improve expert finding on StackOverflow by bridging term gaps between recruiters and users, yielding up to 46% MAP gain over prior methods.
-
TrackNet: A Deep Learning Network for Tracking High-speed and Tiny Objects in Sports Applications
TrackNet, a heatmap-based CNN trained on single or multiple frames, tracks tennis balls in videos achieving 99.7% precision on one match and 95.3% in 10-fold cross-validation.
-
De-risking solutions to optimization problems
A cutting-plane methodology adjusts optimization solutions to lower risk exposure with minimal nominal cost increase or proves impossibility, borrowing from first-order optimization techniques.
-
Accelerated Gradient Descent for Faster Convergence with Minimal Overhead
CT-AGD accelerates first-order optimization in deep learning by using finite-difference curvature estimates and noise-mitigation heuristics, achieving equivalent accuracy with 33% fewer training epochs and overhead co...
-
Many could be better than all: A novel instance-oriented algorithm for Multi-modal Multi-label problem
MCC selects partial modalities per instance for MMML problems and experiments on a herbs dataset plus two public ones indicate using many but not all modalities can outperform using all.
-
Single-bit-per-weight deep convolutional neural networks without batch-normalization layers for embedded systems
Experiments show that shifted-ReLU layers can replace batch-normalization in single-bit-weight wide residual networks on CIFAR-10/100 and ImageNet without consistent accuracy penalty.
-
End-to-End Speech Recognition with High-Frame-Rate Features Extraction
High-frame-rate feature extraction at 200-400 fps improves end-to-end ASR word error rates on WSJ and CHiME-5, with relative reductions up to 24.1% when combined with speed perturbation.
-
Dealing with Topological Information within a Fully Convolutional Neural Network
A geodesic operator pre-processing step is introduced to let FCNs exploit topological information for segmenting histological images of pigmented reconstructed epidermis.
-
End-to-End ASR for Code-switched Hindi-English Speech
End-to-end ASR for code-switched Hindi-English with <50 hours of data shows gains from multi-task learning and corpus balancing but underperforms cascaded baselines.
-
A Theoretical and Experimental Study of a Novel Adaptive Learning Algorithm
Introduces C-Adam optimizer variant with claimed convergence proof and real-life numerical experiments.
-
Deep learning applied to computational mechanics: A comprehensive review, state of the art, and the classics
A comprehensive review of deep learning techniques for computational mechanics, including LSTM for constitutive modeling, PINNs for PDE solving, optimizers, and kernel methods.
-
Improving Malaria Parasite Detection from Red Blood Cell using Deep Convolutional Neural Networks
CNNs applied to the NIH malaria dataset reach 97.77% accuracy for parasite detection in segmented RBC patches using 5-fold cross-validation and holdout testing.
-
Novel Suboptimal approaches for Hyperparameter Tuning of Deep Neural Network [under the shelf of Optical Communication]
Introduces marginal and alternating suboptimal grid search methods for DNN hyperparameter tuning in fiber OC and FSO systems, claiming reduced computation load with favorable performance.
-
Introduction to stochastic gradient methods
Lecture notes on convergence theory for deterministic gradient descent and stochastic gradient methods under standard assumptions.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.