MADE: Masked Autoencoder for Distribution Estimation
read the original abstract
There has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder's parameters to respect autoregressive constraints: each input is reconstructed only from previous inputs in a given ordering. Constrained this way, the autoencoder outputs can be interpreted as a set of conditional probabilities, and their product, the full joint probability. We can also train a single network that can decompose the joint probability in multiple different orderings. Our simple framework can be applied to multiple architectures, including deep ones. Vectorized implementations, such as on GPUs, are simple and fast. Experiments demonstrate that this approach is competitive with state-of-the-art tractable distribution estimators. At test time, the method is significantly faster and scales better than other autoregressive estimators.
This paper has not been read by Pith yet.
Forward citations
Cited by 11 Pith papers
-
Density estimation using Real NVP
Real NVP uses affine coupling layers to create invertible transformations that support exact density estimation, sampling, and latent inference without approximations.
-
Dark Matter in Draco and Bo\"otes I: Hints of a Core in an Ultra-Faint Dwarf from Simulation-Based Inference
GraphNPE recovers a significantly lower central density for Boötes I consistent with a core while Draco remains marginally cuspy, and demonstrates that higher-order velocity moments reduce bias in dynamical modeling.
-
Testing machine-learned distributions against Monte Carlo data for the QCD chiral phase transition
Conditional MAFs interpolate QCD chiral phase structure across coupling, mass, and volume, reproducing reweighting while cutting required ensembles despite bias near transitions.
-
Sampling two-dimensional spin systems with transformers
Transformer networks sample up to 180x180 2D Ising systems and 64x64 Edwards-Anderson systems by generating spin groups with probability approximations, yielding ~20x higher effective sample size than prior neural sam...
-
Geometry-Induced Long-Range Correlations in Recurrent Neural Network Quantum States
Dilated RNN wave functions induce power-law correlations for the critical 1D transverse-field Ising model and the Cluster state, unlike the exponential decay of conventional RNN ansatze.
-
EXPO: Stable Reinforcement Learning with Expressive Policies
EXPO stabilizes online RL for expressive policies by training a base policy with imitation and using a lightweight Gaussian edit policy to select higher-value actions on the fly for sampling and TD backups.
-
Estimation of the reduced density matrix and entanglement entropies using autoregressive networks
Autoregressive networks trained once on fixed lattice volume and discretization estimate reduced density matrix elements for Ising chains, yielding von Neumann and Rényi entanglement entropies for intervals of up to f...
-
SURF: Separation via Unsupervised Remixing Flow
SURF uses teacher-student remixing within a flow-matching framework to achieve unsupervised source separation and reports new state-of-the-art results on audio and image benchmarks.
-
Variational Autoregressive Networks with probability priors
Incorporating probability priors into variational autoregressive networks reduces training burden and enables larger system sizes for sampling in the Ising and Edwards-Anderson models.
-
Simulation-based inference for rapid Bayesian parameter estimation in epidemiological models: a comparison with MCMC
SBI matches MCMC posterior accuracy on a SECIR model but runs 15-120 times faster on GPU for 31-day and 201-day inference windows.
-
Comparative Analysis of EMCEE, Gaussian Process, and Masked Autoregressive Flow in Constraining the Hubble Constant Using Cosmic Chronometers Dataset
EMCEE outperforms GP and MAF in recovering true H0 from mock cosmic chronometer datasets, with GP most sensitive to data points via delete-d jackknife analysis.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.