Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
read the original abstract
The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change only once per epoch, Temporal Ensembling becomes unwieldy when learning large datasets. To overcome this problem, we propose Mean Teacher, a method that averages model weights instead of label predictions. As an additional benefit, Mean Teacher improves test accuracy and enables training with fewer labels than Temporal Ensembling. Without changing the network architecture, Mean Teacher achieves an error rate of 4.35% on SVHN with 250 labels, outperforming Temporal Ensembling trained with 1000 labels. We also show that a good network architecture is crucial to performance. Combining Mean Teacher and Residual Networks, we improve the state of the art on CIFAR-10 with 4000 labels from 10.55% to 6.28%, and on ImageNet 2012 with 10% of the labels from 35.24% to 9.11%.
This paper has not been read by Pith yet.
Forward citations
Cited by 15 Pith papers
-
Emerging Properties in Self-Supervised Vision Transformers
Self-supervised ViTs show emergent semantic segmentation and 78.3% k-NN accuracy on ImageNet; DINO reaches 80.1% linear evaluation with ViT-Base.
-
VISTA: Variance-Gated Inter-Sequence Test-Time Adaptation for Multi-Sequence MRI Segmentation
VISTA is a source-free TTA framework for multi-sequence MRI segmentation that uses inter-sequence spectral/patch interventions and cross-view variance gating to handle modality-interaction shifts, reporting Dice gains...
-
Identify Then Project: Contrastive Learning of Latent Dynamics from Partial Observations with Port-Hamiltonian Structure
A two-stage contrastive teacher-student framework learns and then projects latent dynamics onto port-Hamiltonian submanifolds from partial observations.
-
TILT: Target-induced loss tilting under covariate shift
TILT adds a target-data penalty on an auxiliary predictor component to induce effective importance weighting for unsupervised domain adaptation under covariate shift.
-
Learned Neighbor Trust for Collaborative Deployment in Model-Agnostic Decentralized Learning
LNTrust has nodes learn compact trust functions from validation evidence that both guide training distillation and define deployment ensembles, yielding higher accuracy with less communication than prior output-only b...
-
Unsupervised Source-Free Ranking of Biomedical Segmentation Models Under Distribution Shift
Presents the first unsupervised source-free framework for ranking semantic and instance segmentation models via prediction consistency under perturbations, with rankings correlating to target-domain performance across...
-
SPADE: Split-and-Delay Embeddings for Autoregressive High-Granularity Calorimeter Simulation
SPADE is a split-and-delay embedding technique for multi-feature autoregressive transformers that achieves competitive performance on high-granularity calorimeter shower simulation.
-
Continuous Reasoning for Vision-Language-Action
Continuous Reasoning for VLA introduces a shared Gaussian latent for continuous thoughts, trained with self-verification to improve action prediction on LIBERO-PRO and real robots.
-
VISTA: Variance-Gated Inter-Sequence Test-Time Adaptation for Multi-Sequence MRI Segmentation
VISTA is a test-time adaptation framework for multi-sequence MRI that uses inter-sequence intervention probes and cross-view disagreement variance to gate self-training, yielding Dice gains of +1.89% on low-field Afri...
-
Unsupervised domain adaptation for radioisotope identification in gamma spectroscopy
Unsupervised domain adaptation via feature alignment raises radioisotope identification accuracy on real LaBr3 gamma spectra from 0.754 to 0.904 for models trained only on synthetic data.
-
Revisiting Feature Prediction for Learning Visual Representations from Video
V-JEPA models trained only on feature prediction from 2 million public videos achieve 81.9% on Kinetics-400, 72.2% on Something-Something-v2, and 77.9% on ImageNet-1K using frozen ViT-H/16 backbones.
-
Vision Transformers Need Registers
Adding register tokens to Vision Transformers eliminates high-norm background artifacts and raises state-of-the-art performance on dense visual prediction tasks.
-
$\mu$Match: Foundation Models for Semi-supervised Learning and Domain Adaptation in EM
μMatch applies student-teacher semi-supervised methods with foundation models to improve segmentation of mitochondria, nuclei, and neurites in EM images over strong baselines.
-
TopoPult-SSL: Gland-Mask-Free Cross-Device Meibomian Gland Segmentation via Self-Distilled Weak Clinical Priors
A two-stage framework adapts source models for cross-device meibomian gland segmentation using weak clinical priors and self-distillation, reaching Dice 0.716 on a 1000-to-100 image benchmark while enabling mask-free ...
-
ZScribbleSeg: A comprehensive segmentation framework with modeling of efficient annotation and maximization of scribble supervision
ZScribbleSeg maximizes scribble supervision with efficient annotation forms, spatial regularization, and EM-estimated class ratios to deliver competitive performance on six medical segmentation tasks without full labels.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.