Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
read the original abstract
Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with a prohibitive computational cost. In this paper we develop a new theoretical framework casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes. A direct result of this theory gives us tools to model uncertainty with dropout NNs -- extracting information from existing models that has been thrown away so far. This mitigates the problem of representing uncertainty in deep learning without sacrificing either computational complexity or test accuracy. We perform an extensive study of the properties of dropout's uncertainty. Various network architectures and non-linearities are assessed on tasks of regression and classification, using MNIST as an example. We show a considerable improvement in predictive log-likelihood and RMSE compared to existing state-of-the-art methods, and finish by using dropout's uncertainty in deep reinforcement learning.
This paper has not been read by Pith yet.
Forward citations
Cited by 30 Pith papers
-
ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policies
ActProbe is an action-space detector that uses temporal consistency error and action chunk magnitude from policy outputs, mapped via LSTM-MLP, to predict failures earlier than baselines across policies and real-robot tasks.
-
Precision constraints on stellar physics from main sequence detached eclipsing binaries
Bayesian framework with active-learning surrogate for MESA models constrains ages and α_MLT from 38 main-sequence DEBs, recovering some α_MLT values below the solar calibration.
-
Inducing Artificial Uncertainty in Language Models
Inducing artificial uncertainty on trivial tasks allows training probes that achieve higher calibration on hard data than standard approaches while retaining performance on easy data.
-
Mixed-Precision Information Bottlenecks for On-Device Trait-State Disentanglement in Bipolar Agitation Detection
MP-IB uses an 8x information asymmetry via FP16 trait heads and INT4 state heads to disentangle speaker identity from agitation in voice biomarkers, outperforming larger models on edge devices with low latency and sup...
-
Homogeneous Stellar Parameters from Heterogeneous Spectra with Deep Learning
A single end-to-end Transformer model unifies stellar labels from heterogeneous spectroscopic surveys into a self-consistent scale without post-hoc recalibration.
-
Causal Diffusion Models for Counterfactual Outcome Distributions in Longitudinal Data
Causal Diffusion Model is the first diffusion-based method to produce full probabilistic counterfactual outcome distributions for sequential interventions in longitudinal data, showing 15-30% better distributional acc...
-
Deep Learning for CMB Foreground Removal and Beam Deconvolution: A U-Net GAN Approach
A U-Net GAN reconstructs CMB T and E maps from Planck-like simulations with foregrounds and systematics, achieving under 1% error outside the Galactic region and demonstrating first-time correction for non-circular be...
-
Concrete Problems in AI Safety
The paper categorizes five concrete AI safety problems arising from flawed objectives, costly evaluation, and learning dynamics.
-
Target-Aware Bandit Allocation for Scalable Surrogate Optimization in Chemical Space
Introduces BOBa, a multi-armed bandit method for scalable surrogate optimization that adaptively allocates inference and evaluations to promising partitions of ultra-large chemical libraries.
-
MorphStrata: Layer-Specific Perturbations for Generating Morphence Students in Time-Series Moving Target Defense
MorphStrata generates heterogeneous student models via layer-specific perturbations in a Transformer-based Morphence MTD setup, reporting RMSE gains up to 24% and 98% on AEP data under FGSM and BIM attacks with under ...
-
Active Perception for Radio Map Reconstruction in Uncharted 3D Air-Ground Environments
3D-URAM decouples radio map reconstruction into Bayesian UNet recovery with uncertainty and transformer-based PPO waypoint selection, reporting over 50% error reduction in simulations and real 300x200x100m field tests.
-
Learning the Universe with cosmological rescaling of merger trees and semi-analytic galaxy formation models
Rescaling merger trees with a halo-profile correction enables cheap generation of galaxy summary statistics across cosmologies using semi-analytic models, matching dedicated simulation accuracy with far fewer base runs.
-
Empirical Bayes Conformal Prediction for Vision and Language Models
Empirical Bayes conformal prediction converts score variability into r-value nonconformity scores that preserve target coverage while reducing inclusion of high-variance false candidates in image classification, CLIP ...
-
When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning
Position-Weighted On-Policy Self-Distillation (PW-OPSD) weights later tokens more heavily after a diagnostic shows position predicts teacher reliability better than entropy, yielding +1.0 and +1.1 Avg@12 gains on AIME...
-
Extracting redshifts from 2D slitless spectroscopic images using deep learning for the CSST galaxy survey
A Bayesian CNN maps 2D slitless spectral images to redshift estimates with NMAD precision 0.0104 for SNR_GI >=1 and better for brighter sources, while remaining robust to wavelength calibration errors via spatial augm...
-
A Physics-Aware Variational Graph Autoencoder for Joint Modal Identification with Uncertainty Quantification
A physics-informed graph variational autoencoder jointly predicts modal frequencies, damping, and shapes from PSD data of trusses with uncertainty quantification and orthogonality constraints.
-
U-Cast: A Surprisingly Simple and Efficient Frontier Probabilistic AI Weather Forecaster
A standard U-Net with MAE pre-training followed by short CRPS fine-tuning via Monte Carlo Dropout matches or exceeds GenCast and IFS ENS probabilistic skill at 1.5° resolution while cutting training compute and infere...
-
Ensemble-Based Dirichlet Modeling for Predictive Uncertainty and Selective Classification
Ensemble-based method of moments on softmax outputs produces stable Dirichlet predictive distributions that improve uncertainty-guided tasks like selective classification over evidential deep learning.
-
Probabilistic Precipitation Nowcasting with Rectified Flow Transformers
FREUD applies rectified flow transformers with frame-wise encoding and a unified decoder to achieve state-of-the-art probabilistic precipitation nowcasting on the SEVIR benchmark.
-
Bayesian Inference with Shaped Deep Non-linear MLPs
In the LP/N = Θ(1) regime, Bayesian predictive posteriors for deep MLPs equal those of data-dependent kernels to first order, with a criterion identifying data processes that benefit from larger effective depth.
-
Testing the Assumptions of Active Learning for Translation Tasks with Few Samples
Informativeness and diversity of samples selected by active learning show no correlation with test performance on translation tasks using few samples; ordering and pre-training effects dominate instead.
-
MedFormer-UR: Uncertainty-Routed Transformer for Medical Image Classification
MedFormer-UR integrates evidential uncertainty from Dirichlet distributions and class-specific prototypes into a transformer to improve calibration and selective prediction on medical images across four modalities.
-
ASTRAFier: A Novel and Scalable Transformer-based Stellar Variability Classifier
ASTRAFier is a Transformer-BiLSTM-CNN model that classifies stellar variability from light curves, reporting 94.26% accuracy on Kepler data and 88.22% on TESS, then applied to 2.8 million TESS curves to release a catalog.
-
Uncertainty-Calibrated Explainable Artificial Intelligence for Fetal Ultrasound Plane Classification: A Systematic Review
PRISMA 2020 systematic review of 78 studies on fetal ultrasound plane classification paired with explainability or uncertainty, introducing the CALIB-XFUS reporting framework across six domains.
-
Neural network-based deconvolution for GeV-Scale Gamma-Ray Spectroscopy
A denoising autoencoder followed by a U-Net reconstructs incident gamma spectra from measured positron spectra in a Monte Carlo-optimized spectrometer for multi-MeV to GeV energies.
-
VR-DAgger: Immersive VR for Dexterous Data Collection and Uncertainty-Guided On-Policy Correction
VR-DAgger is a VR-centered human-in-the-loop framework that applies MC dropout uncertainty to select and correct failure segments in diffusion policy rollouts, yielding up to 23 percentage point gains over behavioral ...
-
Portfolio Optimization Proxies under Label Scarcity and Regime Shifts via Bayesian and Deterministic Students under Semi-Supervised Sandwich Training
A semi-supervised teacher-student framework enables neural networks to proxy CVaR portfolio optimization using synthetic data augmentation for scarce labels and regime shifts.
-
Modelling magnetic material properties with uncertainty-aware neural networks
Uncertainty-aware neural networks using Gaussian negative log-likelihood and dropout are applied to predict intrinsic magnetic properties and coercivity via graph neural networks in permanent magnet research.
-
PolyGraphPy: A unified Python framework for atomistic simulation and machine learning-driven polymer design
PolyGraphPy automates DFTB calculations for datasets of monomers and copolymers, uses Bayesian GNNs for property prediction with uncertainty quantification, and applies SELFIES-GPT and BRICS-based GA for de novo polym...
-
Deep Learning for Sequential Decision Making under Uncertainty: Foundations, Frameworks, and Frontiers
A tutorial framing deep learning as a complement to optimization for sequential decision-making under uncertainty, with applications in supply chains, healthcare, and energy.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.