Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index; 14 of the 30 sources its claims cite were checked for support.

Unrepresentative big surveys significantly overestimated US vaccine uptake

Open access

Published version of a preprint

Abstract

Surveys are a crucial tool for understanding public opinion and behaviour, and their accuracy depends on maintaining statistical representativeness of their target populations by minimizing biases from all sources. Increasing data size shrinks confidence intervals but magnifies the effect of survey bias: an instance of the Big Data Paradox 1. Here we demonstrate this paradox in estimates of first-dose COVID-19 vaccine uptake in US adults from 9 January to 19 May 2021 from two large surveys: Delphi-Facebook 2,3 (about 250,000 responses per week) and Census Household Pulse 4 (about 75,000 every two weeks). In May 2021, Delphi-Facebook overestimated uptake by 17 percentage points (14-20 percentage points with 5% benchmark imprecision) and Census Household Pulse by 14 (11-17 percentage points with 5% benchmark imprecision), compared to a retroactively updated benchmark the Centers for Disease Control and Prevention published on 26 May 2021. Moreover, their large sample sizes led to miniscule margins of error on the incorrect estimates. By contrast, an Axios-Ipsos online panel 5 with about 1,000 responses per week following survey research best practices 6 provided reliable estimates and uncertainty quantification. We decompose observed error using a recent analytic framework 1 to explain the inaccuracy in the three surveys. We then analyse the implications for vaccine hesitancy and willingness. We show how a survey of 250,000 respondents can produce an estimate of the population mean that is no more accurate than an estimate from a simple random sample of size 10. Our central message is that data quality matters more than data quantity, and that compensating the former with the latter is a mathematically provable losing proposition.

Overview

CoChat Summary

TL;DR

Two large US COVID-19 vaccine surveys, Delphi-Facebook and Census Household Pulse, substantially overestimated first-dose vaccine uptake relative to a CDC benchmark, while a much smaller Axios-Ipsos panel tracked the benchmark well. The paper uses the Big Data Paradox framework to show that larger sample sizes magnified bias and produced misleadingly narrow confidence intervals.

Key findings

  • In May 2021, Delphi-Facebook overestimated first-dose vaccine uptake by 17 percentage points (14–20 with 5% benchmark imprecision) and Census Household Pulse by 14 percentage points (11–17 with 5% benchmark imprecision), compared with the CDC benchmark published on 26 May 2021; Axios-Ipsos’ error was 4.2 percentage points (1–7 with 5% benchmark imprecision).
  • Despite having far larger samples—Delphi-Facebook n = 4,525,633 across 19 waves, Census Household Pulse n = 606,615 across 8 waves, Axios-Ipsos n = 11,421 across 11 waves—the large surveys had miniscule margins of error on incorrect estimates.
  • Delphi-Facebook’s bias-adjusted effective sample size was less than 10 in April 2021, a 99.99% reduction from its raw average weekly sample size of 250,000; Census Household Pulse also had over 99% reductions in effective sample size by May 2021.
  • Axios-Ipsos’ 95% confidence intervals contained the CDC benchmark in 10 out of 11 surveys (91% empirical coverage), and the error decomposition suggested increasing error in the large surveys was primarily driven by increasing data defect correlation.

Methods

  • Comparative analysis of three online surveys targeting US adults from 9 January to 19 May 2021: Delphi-Facebook (about 250,000 responses per week), Census Household Pulse (about 75,000 every two weeks), and Axios-Ipsos (about 1,000 per week), benchmarked against CDC first-dose vaccination counts as of 26 May 2021.
  • Used a previously formulated statistical framework to decompose total error into data defect correlation, data scarcity, and inherent problem difficulty, and to calculate bias-adjusted effective sample size; sensitivity analyses used ±5% and ±10% benchmark imprecision.
  • Compared survey designs, recruitment, weighting, and respondent composition, with additional small-to-medium online surveys analysed in Extended Data.

Limitations

  • Stated in the paper: the vaccine uptake analysis only examines data defect correlation for an outcome with an available benchmark, first-dose vaccine uptake; absence of evidence of bias for other outcomes is not evidence of its absence. The analysis relies on the accuracy of CDC estimates, and if CDC selection bias were significant enough to alter results, that would itself be another Big Data Paradox. Axios-Ipsos’ question wording differs from the other surveys and may contribute to observed differences. Small surveys can also be wrong, and whether small-area methods outperform large biased surveys is an open question.
  • Evident from the text: the ±5% and ±10% benchmark imprecision scenarios may not capture latent systemic issues affecting CDC vaccination reporting; CDC age-coded counts were missing for some jurisdictions, reaching about 10% of total US vaccinations at its peak, and the authors assume the adult fraction among missing-age vaccinations equals the adult fraction among reported-age vaccinations. The study is an observational comparison of existing surveys, not a randomized design.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41586-021-04198-4

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

13 Valid · 1 Not valid · 11 Not checked · 98 Own results and interpretations

46 more claims

Create a free account to see all 123 claims and to verify the 2 references still open.

See all claims — free account
  • Probably not a substantial claim. 38
  • It may state two separate claims as one. 42

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 47 references

Research works printed but not linked: 7

  1. Kreuter, F. et al. Partnering with Facebook on a university-based rapid turn-around global survey. Surv. Res. Methods 14, 159–163 (2020).
  2. Kish, L. Survey Sampling (Wiley, 1965).
  3. Groves, R. M. et al. Survey Methodology Vol. 561 (Wiley, 2011).
  4. Groen, J. Sources of error in survey and administrative data: the importance of reporting procedures. J. Off. Stat. 28, 173–198 (2012).
  5. Associated Press–NORC Center for Public Affairs Research. The June 2021 AP–NORC center poll (2021).
  6. Bethlehem, J. in Survey Nonresponse (eds Groves, R. M. et al.) 275–288 (Wiley, 2002).
  7. Meng, X.-L. in Past, Present, and Future of Statistical Science (eds Lin, X. et al.) 537–562 (CRC Press, 2014).

Cited by

See all 252

Metadata

Authors
Bradley, V. C., Kuriwaki, S., Isakov, M., Sejdinovic, D., Meng, X.-L., & Flaxman, S.
Affiliations
  • University of Oxford (GB)
  • Stanford University (US)
  • Harvard University (US)
Corresponding author
Flaxman, Seth
Venue
Nature
Published
2021-12-08
Volume
600
Issue
7890
Pages
695-700
ISSN
  • 0028-0836 (ISSN-L)
  • 1476-4687
OpenAlex
W4200003648
Cited by
248
References
47
Citations by year
2026: 27 · 2025: 28 · 2024: 58 · 2023: 73 · 2022: 53 · 2021: 8
FWCI
66.81
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature (publishedVersion)
  • medRxiv (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • Digital Access to Scholarship at Harvard (DASH) (Harvard University) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation
  • Carnegie Mellon University
  • University of Oxford
  • Centers for Disease Control and Prevention
  • Medical Research Council
  • Engineering and Physical Sciences Research Council (EP/V002910/1, EP/V002910/2, EP/L016710/1)
  • Clarendon Fund
Topics
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
  • Social Sciences › Health › Vaccine Coverage and Hesitancy
Keywords
survey bias, survey representativeness, COVID-19 vaccine uptake, online panel survey, Facebook survey, vaccine hesitancy, vaccine willingness
MeSH terms
Big Data, COVID-19, COVID-19 Vaccines, Vaccination Hesitancy, Centers for Disease Control and Prevention, U.S., Female, Humans, Male, Research Design, United States, Vaccination, Bias, Sample Size, Health Care Surveys, Benchmarking, Social Media, Datasets as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Unrepresentative big surveys significantly overestimated US vaccine uptake. https://app.cochat.ai/papers/10.1038/s41586-021-04198-4
The paper
Bradley, V. C., Kuriwaki, S., Isakov, M., Sejdinovic, D., Meng, X.-L., & Flaxman, S. (2021). Unrepresentative big surveys significantly overestimated US vaccine uptake. Nature. https://doi.org/10.1038/s41586-021-04198-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.