The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index; 14 of the 30 sources its claims cite were checked for support.
Unrepresentative big surveys significantly overestimated US vaccine uptake
Open accessPublished version of a preprint
Abstract
Surveys are a crucial tool for understanding public opinion and behaviour, and their accuracy depends on maintaining statistical representativeness of their target populations by minimizing biases from all sources. Increasing data size shrinks confidence intervals but magnifies the effect of survey bias: an instance of the Big Data Paradox 1. Here we demonstrate this paradox in estimates of first-dose COVID-19 vaccine uptake in US adults from 9 January to 19 May 2021 from two large surveys: Delphi-Facebook 2,3 (about 250,000 responses per week) and Census Household Pulse 4 (about 75,000 every two weeks). In May 2021, Delphi-Facebook overestimated uptake by 17 percentage points (14-20 percentage points with 5% benchmark imprecision) and Census Household Pulse by 14 (11-17 percentage points with 5% benchmark imprecision), compared to a retroactively updated benchmark the Centers for Disease Control and Prevention published on 26 May 2021. Moreover, their large sample sizes led to miniscule margins of error on the incorrect estimates. By contrast, an Axios-Ipsos online panel 5 with about 1,000 responses per week following survey research best practices 6 provided reliable estimates and uncertainty quantification. We decompose observed error using a recent analytic framework 1 to explain the inaccuracy in the three surveys. We then analyse the implications for vaccine hesitancy and willingness. We show how a survey of 250,000 respondents can produce an estimate of the population mean that is no more accurate than an estimate from a simple random sample of size 10. Our central message is that data quality matters more than data quantity, and that compensating the former with the latter is a mathematically provable losing proposition.
Overview
CoChat Summary
TL;DR
Two large US COVID-19 vaccine surveys, Delphi-Facebook and Census Household Pulse, substantially overestimated first-dose vaccine uptake relative to a CDC benchmark, while a much smaller Axios-Ipsos panel tracked the benchmark well. The paper uses the Big Data Paradox framework to show that larger sample sizes magnified bias and produced misleadingly narrow confidence intervals.
Key findings
- In May 2021, Delphi-Facebook overestimated first-dose vaccine uptake by 17 percentage points (14–20 with 5% benchmark imprecision) and Census Household Pulse by 14 percentage points (11–17 with 5% benchmark imprecision), compared with the CDC benchmark published on 26 May 2021; Axios-Ipsos’ error was 4.2 percentage points (1–7 with 5% benchmark imprecision).
- Despite having far larger samples—Delphi-Facebook n = 4,525,633 across 19 waves, Census Household Pulse n = 606,615 across 8 waves, Axios-Ipsos n = 11,421 across 11 waves—the large surveys had miniscule margins of error on incorrect estimates.
- Delphi-Facebook’s bias-adjusted effective sample size was less than 10 in April 2021, a 99.99% reduction from its raw average weekly sample size of 250,000; Census Household Pulse also had over 99% reductions in effective sample size by May 2021.
- Axios-Ipsos’ 95% confidence intervals contained the CDC benchmark in 10 out of 11 surveys (91% empirical coverage), and the error decomposition suggested increasing error in the large surveys was primarily driven by increasing data defect correlation.
Methods
- Comparative analysis of three online surveys targeting US adults from 9 January to 19 May 2021: Delphi-Facebook (about 250,000 responses per week), Census Household Pulse (about 75,000 every two weeks), and Axios-Ipsos (about 1,000 per week), benchmarked against CDC first-dose vaccination counts as of 26 May 2021.
- Used a previously formulated statistical framework to decompose total error into data defect correlation, data scarcity, and inherent problem difficulty, and to calculate bias-adjusted effective sample size; sensitivity analyses used ±5% and ±10% benchmark imprecision.
- Compared survey designs, recruitment, weighting, and respondent composition, with additional small-to-medium online surveys analysed in Extended Data.
Limitations
- Stated in the paper: the vaccine uptake analysis only examines data defect correlation for an outcome with an available benchmark, first-dose vaccine uptake; absence of evidence of bias for other outcomes is not evidence of its absence. The analysis relies on the accuracy of CDC estimates, and if CDC selection bias were significant enough to alter results, that would itself be another Big Data Paradox. Axios-Ipsos’ question wording differs from the other surveys and may contribute to observed differences. Small surveys can also be wrong, and whether small-area methods outperform large biased surveys is an open question.
- Evident from the text: the ±5% and ±10% benchmark imprecision scenarios may not capture latent systemic issues affecting CDC vaccination reporting; CDC age-coded counts were missing for some jurisdictions, reaching about 10% of total US vaccinations at its peak, and the authors assume the adult fraction among missing-age vaccinations equals the adult fraction among reported-age vaccinations. The study is an observational comparison of existing surveys, not a randomized design.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
13 Valid · 1 Not valid · 11 Not checked · 98 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, once this control is lost, small increases in ddc beyond what is expected in simple random samples can result in marked reductions of effective sample sizes for large populations 1 .”
What the sources say
- Supportedcites [1] Meng, X.-L. Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election. Ann. Appl. Stat. 12, 685–726 (2018). source
Section 3 (pp. 14–16) says probabilistic sampling keeps the data defect index under control, while even a small departure can cause a dramatic reduction in effective sample size as population size grows. The paper also describes a seemingly trivial data defect correlation as substantially reducing effective sample size (p. 21). (LLM, conf=0.93)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“small-area methods that borrow information across subgroups 48 can perform better with higher-quality-albeit few-data”
What the sources say
- Partly supportedcites [48] Park, D. K., Gelman, A. & Bafumi, J. Bayesian multilevel estimation with poststratification: State-level estimates from national polls. Polit. Anal. 12, 375–385 (2004). sourceExperimental — accuracy not yet measured
The Overview (p. 2) says that many demographic-by-state categories have few or no observations, yet the multilevel method performs well. The Discussion (p. 9) says partial pooling outperforms complete and no pooling, but the paper does not establish that the data were higher-quality or that data quality explains the improvement. (LLM, conf=0.86)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, their large sample sizes led to miniscule margins of error on the incorrect estimates.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The total error of each survey's estimate of vaccine uptake (Fig. 1b) increases over time for all studies, most markedly for Delphi-Facebook.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Inherent problem difficulty is a population quantity common to all three surveys that peaks when the benchmark vaccination rate approaches 50% in April 2021 (Fig. 1e).”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
46 more claims
Create a free account to see all 123 claims and to verify the 2 references still open.
- Probably not a substantial claim. 38
- It may state two separate claims as one. 42
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Extended Data Table 2 | Contribution of offline recruitment and weighting schemes to discrepancies between surveys · p. 17
“In the Axios-Ipsos 22 March 2020 wave, the offline panellists ( n = 21) were 24 percentage points less likely to be vaccinated than online panellists ( n = 974).”
§ Explanations for error · p. 4
“dropping these offline respondents ( n = 21, or 1% of the sample) in their 22 March 2021 wave”
Citations
Cites
Sample Selection Bias as a Specification Error
Heckman, J. J. (1979) · Econometrica · doi:10.2307/1912352 · Cited by 28,779
Fry, A., Littlejohns, T. J., Sudlow, C., Doherty, N., Adamska, L., Sprosen, T., Collins, R., & Allen, N. E. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx246 · Cited by 3,979
Interval Estimation for a Binomial Proportion
Brown, L. D., Cai, T. T., & DasGupta, A. (2001) · Statistical Science · doi:10.1214/ss/1009213286 · Cited by 3,538
The Parable of Google Flu: Traps in Big Data Analysis
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014) · Science · doi:10.1126/science.1248506 · Cited by 2,523
Nonresponse rates and nonresponse bias in household surveys
Groves, R. M. (2006) · Public Opinion Quarterly · doi:10.1093/poq/nfl033 · Cited by 2,420
Big data: a revolution that will transform how we live, work, and think
Armstrong, K. (2014) · Information, Communication & Society · doi:10.1080/1369118x.2014.923482 · Cited by 1,629
Haas, E. J., Angulo, F. J., McLaughlin, J. M., Anis, E., Singer, S. R., Khan, F., Brooks, N., Smaja, M., Mircus, G., Pan, K., et al. (2021) · The Lancet · doi:10.1016/s0140-6736(21)00947-8 · Cited by 1,564
Neyman, J. (1934) · Journal of the Royal Statistical Society · doi:10.2307/2342192 · Cited by 1,086
Computer and Internet Use in the United States: 2003
Day, J. C., Janus, A. L., & Davis, J. T. (2005) · Cited by 1,029
Allcott, H., Boxell, L., Conway, J., Gentzkow, M., Thaler, M., & Yang, D. (2020) · Journal of Public Economics · doi:10.1016/j.jpubeco.2020.104254 · Cited by 999
Research works printed but not linked: 7
- Kreuter, F. et al. Partnering with Facebook on a university-based rapid turn-around global survey. Surv. Res. Methods 14, 159–163 (2020).
- Kish, L. Survey Sampling (Wiley, 1965).
- Groves, R. M. et al. Survey Methodology Vol. 561 (Wiley, 2011).
- Groen, J. Sources of error in survey and administrative data: the importance of reporting procedures. J. Off. Stat. 28, 173–198 (2012).
- Associated Press–NORC Center for Public Affairs Research. The June 2021 AP–NORC center poll (2021).
- Bethlehem, J. in Survey Nonresponse (eds Groves, R. M. et al.) 275–288 (Wiley, 2002).
- Meng, X.-L. in Past, Present, and Future of Statistical Science (eds Lin, X. et al.) 537–562 (CRC Press, 2014).
Cited by
Advances in Artificial Intelligence for Infectious-Disease Surveillance
Brownstein, J. S., Rader, B., Astley, C. M., & Tian, H. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2119215 · Cited by 167
REFORMS: Consensus-based Recommendations for Machine-learning-based Science
Kapoor, S., Cantrell, E. M., Peng, K., Pham, T. H., Bail, C. A., Gundersen, O. E., Hofman, J. M., Hullman, J. R., Lones, M. A., Malik, M. M., et al. (2024) · Science Advances · doi:10.1126/sciadv.adk3452 · Cited by 303
“When the sampling frame is unrepresentative, increasing the sample size can shrink confidence intervals without decreasing bias in estimates, thus giving false confidence ( 123 ).”
Communicating doctors’ consensus persistently increases COVID-19 vaccinations
Bartoš, V., Bauer, M., Cahlíková, J., & Chytilová, J. (2022) · Nature · doi:10.1038/s41586-022-04805-y · Cited by 128
We need to talk about nonprobability samples
Boyd, R. J., Powney, G. D., & Pescott, O. L. (2023) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2023.01.001 · Cited by 144
“The formula shows that, for a given selection bias, effective sample size is lower in the presence of measurement error [8].”
BackgroundThe Epidemiology of Long Coronavirus Disease in US Adults
Robertson, M. M., Qasmieh, S. A., Kulkarni, S., Teasdale, C. A., Jones, H. E., McNairy, M. L., Borrell, L. N., & Nash, D. (2022) · Clinical Infectious Diseases · doi:10.1093/cid/ciac961 · Cited by 106
Metadata
- Authors
- Bradley, V. C., Kuriwaki, S., Isakov, M., Sejdinovic, D., Meng, X.-L., & Flaxman, S.
- Affiliations
- University of Oxford (GB)
- Stanford University (US)
- Harvard University (US)
- Corresponding author
- Flaxman, Seth
- Venue
- Nature
- Published
- 2021-12-08
- Volume
- 600
- Issue
- 7890
- Pages
- 695-700
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC8653636
- OpenAlex
- W4200003648
- Cited by
- 248
- References
- 47
- Citations by year
- 2026: 27 · 2025: 28 · 2024: 58 · 2023: 73 · 2022: 53 · 2021: 8
- FWCI
- 66.81
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Nature (publishedVersion)
- medRxiv (acceptedVersion, cc-by)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation
- Carnegie Mellon University
- University of Oxford
- Centers for Disease Control and Prevention
- Medical Research Council
- Engineering and Physical Sciences Research Council (EP/V002910/1, EP/V002910/2, EP/L016710/1)
- Clarendon Fund
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
- Social Sciences › Health › Vaccine Coverage and Hesitancy
- Keywords
- survey bias, survey representativeness, COVID-19 vaccine uptake, online panel survey, Facebook survey, vaccine hesitancy, vaccine willingness
- MeSH terms
- Big Data, COVID-19, COVID-19 Vaccines, Vaccination Hesitancy, Centers for Disease Control and Prevention, U.S., Female, Humans, Male, Research Design, United States, Vaccination, Bias, Sample Size, Health Care Surveys, Benchmarking, Social Media, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Unrepresentative big surveys significantly overestimated US vaccine uptake. https://app.cochat.ai/papers/10.1038/s41586-021-04198-4
- The paper
- Bradley, V. C., Kuriwaki, S., Isakov, M., Sejdinovic, D., Meng, X.-L., & Flaxman, S. (2021). Unrepresentative big surveys significantly overestimated US vaccine uptake. Nature. https://doi.org/10.1038/s41586-021-04198-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.