CoChat
Sign Up
PreprintCited by 81Domain Adaptation and Few-Shot LearningAdversarial Robustness in Machine LearningMachine Learning and Data Classification

Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization

Open access
Miller, John, Taori, Rohan, Raghunathan, Aditi, Sagawa, Shiori, Koh, Pang Wei, Shankar, Vaishaal, Liang, Percy, Carmon, Yair +1 more·2021·arXiv
DOI 10.48550/arxiv.2107.04649arXiv 2107.04649
Open this paper in CoChat — free accountOpen PDF

Abstract

For machine learning systems to be reliable, we must understand their performance in unseen, out-of-distribution environments. In this paper, we empirically show that out-of-distribution performance is strongly correlated with in-distribution performance for a wide range of models and distribution shifts. Specifically, we demonstrate strong correlations between in-distribution and out-of-distribution performance on variants of CIFAR-10 & ImageNet, a synthetic pose estimation task derived from YCB objects, satellite imagery classification in FMoW-WILDS, and wildlife classification in iWildCam-WILDS. The strong correlations hold across model architectures, hyperparameters, training set size, and training duration, and are more precise than what is expected from existing domain adaptation theory. To complete the picture, we also investigate cases where the correlation is weaker, for instance some synthetic distribution shifts from CIFAR-10-C and the tissue classification dataset Camelyon17-WILDS. Finally, we provide a candidate theory based on a Gaussian data model that shows how changes in the data covariance arising from distribution shift can affect the observed correlations.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2107.04649

Cites

91 of 91 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior work has shown that for many tasks, ID and OOD performance can be highly correlated across different model architectures and hyperparameters.

    “Prior work has shown that for many tasks, ID and OOD performance can be highly correlated across different model architectures and hyperparameters (Taori et al., 2020; Liu et al., 2021b; Miller et al., 2021).”

    From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Miller et al., 2021

  • These specialized methods are reasonably effective at mitigating stain variation, at least in some contexts.

    “These methods are reasonably effective at mitigating stain variation, at least in some contexts (Tellez et al., 2019; Miller et al., 2021)”

    From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Miller et al., 2021

  • Our findings mirror Accuracy on the Line and Agreement on the Line at a much more granular scale, i.e., data instances rather than data sets.

    “Our findings presented in this work mirror these phenomena at a much more granular scale, i.e., data instances rather than data sets.”

    From Anchor Points: Benchmarking Models with Much Fewer Examples (2024) · cites this paper as Miller et al., 2021

Cites

Flagged first, then by citations

8 of the 99 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Miller, J., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., & Schmidt, L.
Affiliations
  • University of California, Berkeley (US)
  • Stanford University (US)
  • Tel Aviv University (IL)
  • Toyota Research Institute (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2107.04649
arXiv
2107.04649
OpenAlex
W3169976301
Cited by
74
References
99
Citations by year
2025: 15 · 2024: 18 · 2023: 19 · 2022: 16 · 2021: 5 · 2020: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
out-of-distribution generalization, distribution shift, generalization gap, domain adaptation, pose estimation, satellite image classification, tissue classification, model reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review