Recent Advances in Autoencoder-Based Representation Learning
read the original abstract
Learning useful representations with little or no supervision is a key challenge in artificial intelligence. We provide an in-depth review of recent advances in representation learning with a focus on autoencoder-based models. To organize these results we make use of meta-priors believed useful for downstream tasks, such as disentanglement and hierarchical organization of features. In particular, we uncover three main mechanisms to enforce such properties, namely (i) regularizing the (approximate or aggregate) posterior distribution, (ii) factorizing the encoding and decoding distribution, or (iii) introducing a structured prior distribution. While there are some promising results, implicit or explicit supervision remains a key enabler and all current methods use strong inductive biases and modeling assumptions. Finally, we provide an analysis of autoencoder-based representation learning through the lens of rate-distortion theory and identify a clear tradeoff between the amount of prior knowledge available about the downstream tasks, and how useful the representation is for this task.
This paper has not been read by Pith yet.
Forward citations
Cited by 8 Pith papers
-
Constrained Variable Projection for Structured Problems
Extends variable projection to constrained separable nonlinear least-squares via bilevel collapse, yielding exact reduced gradients and a convergent conditional-gradient algorithm.
-
MEDAL: Manifold Embedding Distillation via Autoencoder Learning
MEDAL distills manifold embeddings into autoencoders to enable out-of-sample extension and held-out validation of dimension reduction methods.
-
The Rate-Distortion-Polysemanticity Tradeoff in SAEs
SAEs exhibit a rate-distortion-polysemanticity tradeoff where monosemanticity increases rate and distortion, with optimal polysemanticity set by feature co-occurrence probabilities in the data.
-
Interests Burn-down Diffusion Process for Personalized Collaborative Filtering
A new interests burn-down diffusion process models decaying user interests for personalized collaborative filtering and outperforms prior generative methods in the StageCF implementation.
-
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
VTAB is a 19-task benchmark that measures representation quality by few-shot adaptation performance across diverse vision domains, with a controlled large-scale comparison of popular pretraining methods.
-
Disentangling Influence: Using Disentangled Representations to Audit Model Predictions
Disentangled representations enable a new auditing procedure to identify proxy features and quantify their influence on model outcomes more effectively than prior methods.
-
Cloning Deterministic Worlds: The Critical Role of Latent Geometry in Long-Horizon World Models
GRWM uses temporal contrastive learning to geometrically regularize latent spaces in world models for high-fidelity cloning of deterministic 3D worlds.
-
Variational Autoencoder Layer
Proposes VAE as a neural network layer with new training strategy and performance analysis.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.