Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index; 5 of the 26 sources its claims cite were checked for support.

Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election

Open access
Meng, Xiao-Li · 2018 · The Annals of Applied Statistics

Abstract

Statisticians are increasingly posed with thought-provoking and even paradoxical questions, challenging our qualifications for entering the statistical paradises created by Big Data. By developing measures for data quality, this article suggests a framework to address such a question: “Which one should I trust more: a 1% survey with 60% response rate or a self-reported administrative dataset covering 80% of the population?” A 5-element Euler-formula-like identity shows that for any dataset of size $n$, probabilistic or not, the difference between the sample average $\overline{X}_{n}$ and the population average $\overline{X}_{N}$ is the product of three terms: (1) a data quality measure, $\rho_{{R,X}}$, the correlation between $X_{j}$ and the response/recording indicator $R_{j}$; (2) a data quantity measure, $\sqrt{(N-n)/n}$, where $N$ is the population size; and (3) a problem difficulty measure, $\sigma_{X}$, the standard deviation of $X$. This decomposition provides multiple insights: (I) Probabilistic sampling ensures high data quality by controlling $\rho_{{R,X}}$ at the level of $N^{-1/2}$; (II) When we lose this control, the impact of $N$ is no longer canceled by $\rho_{{R,X}}$, leading to a Law of Large Populations (LLP), that is, our estimation error, relative to the benchmarking rate $1/\sqrt{n}$, increases with $\sqrt{N}$; and (III) the “bigness” of such Big Data (for population inferences) should be measured by the relative size $f=n/N$, not the absolute size $n$; (IV) When combining data sources for population inferences, those relatively tiny but higher quality ones should be given far more weights than suggested by their sizes. Estimates obtained from the Cooperative Congressional Election Study (CCES) of the 2016 US presidential election suggest a $\rho_{{R,X}}\approx-0.005$ for self-reporting to vote for Donald Trump. Because of LLP, this seemingly minuscule data defect correlation implies that the simple sample proportion of the self-reported voting preference for Trump from $1\%$ of the US eligible voters, that is, $n\approx2\mbox{,}300\mbox{,}000$, has the same mean squared error as the corresponding sample proportion from a genuine simple random sample of size $n\approx400$, a $99.98\%$ reduction of sample size (and hence our confidence). The CCES data demonstrate LLP vividly: on average, the larger the state’s voter populations, the further away the actual Trump vote shares from the usual $95\%$ confidence intervals based on the sample proportions. This should remind us that, without taking data quality into account, population inferences with Big Data are subject to a Big Data Paradox: the more the data, the surer we fool ourselves.

Overview

CoChat Summary

TL;DR

Meng derives an exact identity showing that sample-mean error depends on a data defect correlation, relative sample size, and population standard deviation, yielding a Law of Large Populations and a Big Data Paradox. Applying this to 2016 US election CCES data, a tiny Trump self-reporting correlation of about −0.005 implies that data from 1% of eligible voters can have the effective SRS size of only about 400.

Key findings

  • The fundamental identity decomposes the difference between sample average and population average into three terms: data quality \(\rho_{R,X}\), data quantity \(\sqrt{(N-n)/n}\), and problem difficulty \(\sigma_X\).
  • For the 2016 US presidential election, CCES data suggest \(\rho_{R,X}\approx -0.005\) for self-reporting to vote for Donald Trump; because of the Law of Large Populations, a sample proportion from \(n\approx 2{,}300{,}000\) (1% of US eligible voters) has the same MSE as an SRS of \(n\approx 400\), a 99.98% reduction in sample size.
  • The CCES data show that, on average, larger state voter populations had actual Trump vote shares farther from the usual 95% confidence intervals based on sample proportions; the estimated slope for Trump supporters was 0.44 (SE 0.09, \(p\approx 10^{-6}\)), versus 0.15 (SE 0.16) for Clinton supporters.
  • The paper labels the resulting phenomenon the Big Data Paradox: “the more the data, the surer we fool ourselves.”

Methods

  • Theoretical derivation of an exact finite-population identity and MSE decomposition, defining the data defect correlation \(\rho_{R,G}\), data defect index \(DI=E_R[\rho_{R,G}^2]\), dropout odds \(DO=(1-f)/f\), and degree of uncertainty \(DU=\sigma_G^2\); Theorem 1 equates finite design effect, \(n^{-1}\) MSE rate, and \(DI=O(N^{-1})\).
  • Empirical application to the 2016 US presidential election using the Cooperative Congressional Election Study (CCES), a national stratified-sample online survey; state-level analyses compare actual vote shares with CCES estimates using raw sample averages (\(n=64{,}600\)), turnout-adjusted estimates (\(\hat n=48{,}106\)), and validated voters (\(n=35{,}829\)).
  • Binary-outcome analysis uses Hoeffding–Fréchet bounds, estimates state-level \(\hat\rho_N\), and regresses \(\log|Z_{n,N}|\) on \(\log N\); extensions cover weighted estimators, stratified sampling, Monte Carlo/Quasi-Monte Carlo, and privacy-enhancing adjustments.

Limitations

  • The paper states that \(\rho_{R,G}\) cannot be estimated from the sample itself; it requires post-election or external information, and the empirical estimates rely on retrospective CCES data and validated-voter subsamples.
  • The paper states that response bias is assumed negligible, that CCES stratification specifications were unavailable, that validation via matching is subject to errors, and that weighting is “a mess”; it also notes that defining d.d.i. for estimands other than sample/population averages remains open.
  • Evident from the text: the empirical application is limited to one election and one survey source, so using the estimated \(\rho_N\) for future elections requires borrowing information or prior assumptions, as the paper itself proposes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1214/18-aoas1161sf

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

4 Valid · 7 Not valid · 17 Not checked · 95 Own results and interpretations

55 more claims

Create a free account to see all 123 claims and to verify the 3 references still open.

See all claims — free account
  • Probably not a substantial claim. 22
  • It may state two separate claims as one. 45
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 46 references

Research works printed but not linked: 7

  1. Chen, C., Duan, N., Meng, X.-L. and Alegria, M. (2006). Power-shrinkage and trimming: Two ways to mitigate excessive weights. In Proceedings of the Survey Research Methods Section of the American Statistical Association 2839–2846.
  2. Cohn, N. (2017). Election review: Why crucial state polls turned out to be wrong. The New York Times, June 1 st.
  3. Fienberg, S. E. (1996). Applying statistical concepts and approaches in academic administration. In Education in a Research University 65–82. Stanford Univ. Press, Stanford.
  4. Fuller, W. A. (2011). Sampling Statistics Wiley, New York.
  5. Kish, L. (1965). Survey Sampling. Wiley, New York.
  6. Mehrhoff, J. (2016). Executive summary: Meng, X.-L. (2014), “A trio of inference problems that could win you a Nobel prize in statistics (if you help fund it)”. Conference handout.
  7. Meng, X.-L. (2018). Statistical paradises and paradoxes in big data (II): Multi-resolution inference, Simpson’s paradox, and individualized treatments. Preprint.

Cited by

See all 320

Metadata

Authors
Meng, X.-L.
Affiliations
Harvard University (US)
Corresponding author
Meng, Xiao-Li
Venue
The Annals of Applied Statistics
Published
2018-06-01
Volume
12
Issue
2
Pages
685-726
ISSN
  • 1932-6157 (ISSN-L)
  • 1941-7330
OpenAlex
W2883811944
Cited by
316
References
46
Citations by year
2026: 29 · 2025: 49 · 2024: 44 · 2023: 58 · 2022: 46 · 2021: 40 · 2020: 29 · 2019: 17 · 2018: 4
FWCI
18.88
Type
journal article
Language
en
Open access
Locations
  • The Annals of Applied Statistics (publishedVersion)
  • Project Euclid (Cornell University) (submittedVersion)
Funders
  • National Science Foundation
  • John Templeton Foundation
  • University of Chicago
  • Arizona State University
  • Rice University
  • University of Pittsburgh
  • University of Cincinnati
  • McGill University
  • Colby College
  • University of Cambridge
  • University of Oxford
  • Institut Carnot Santé Animale
  • Harvard Data Science Initiative, Harvard University
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
  • Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
  • Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
Keywords
big data, data quality, population inference, data combination, self-reported data, 2016 US presidential election, Cooperative Congressional Election Study, mean squared error, survey response rates, administrative data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election. https://app.cochat.ai/papers/10.1214/18-aoas1161sf
The paper
Meng, X.-L. (2018). Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election. The Annals of Applied Statistics. https://doi.org/10.1214/18-aoas1161sf

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.