CoChat
Sign Up
PreprintCited by 4,275Machine Learning and AlgorithmsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

Open access
Tarvainen, Antti, Valpola, Harri·2017·arXiv
DOI 10.48550/arxiv.1703.01780arXiv 1703.01780
Open this paper in CoChat — free accountOpen PDF

Abstract

The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change only once per epoch, Temporal Ensembling becomes unwieldy when learning large datasets. To overcome this problem, we propose Mean Teacher, a method that averages model weights instead of label predictions. As an additional benefit, Mean Teacher improves test accuracy and enables training with fewer labels than Temporal Ensembling. Without changing the network architecture, Mean Teacher achieves an error rate of 4.35% on SVHN with 250 labels, outperforming Temporal Ensembling trained with 1000 labels. We also show that a good network architecture is crucial to performance. Combining Mean Teacher and Residual Networks, we improve the state of the art on CIFAR-10 with 4000 labels from 10.55% to 6.28%, and on ImageNet 2012 with 10% of the labels from 35.24% to 9.11%.

Integrity signals

Registry

DataCite · 10.48550/arxiv.1703.01780

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Weight averaging usually produces a better model than the individual models from each iteration.

    “Indeed, as motivated in Tarvainen et al. [65], weight averaging usually produces a better model than the individual models from each iteration [51].”

    From Emerging Properties in Self-Supervised Vision Transformers (2021) · cites this paper as [65]

Cited by

Sorted by citations

Authors
Tarvainen, A., & Valpola, H.
Venue
arXiv
Publisher
arXiv
Published
2017
Volume
30
Pages
1195-1204
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1703.01780
arXiv
1703.01780
OpenAlex
W2592691248
Cited by
4,275
References
0
Citations by year
2026: 31 · 2025: 514 · 2024: 922 · 2023: 874 · 2022: 619 · 2021: 702 · 2020: 389 · 2019: 176 · 2018: 40 · 2017: 6 · 2016: 1 · 2012: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
  • International Conference on Learning Representations
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
mean teacher, semi-supervised learning, temporal ensembling, weight averaging, consistency regularization, ResNet, label efficiency, ImageNet, CIFAR-10, SVHN

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review