Issues with data and analyses: Errors, underlying themes, and potential solutions
Open accessThe DOI matches its Crossref record; 95 of the 95 references it cites were found in a registry or an index; 22 of the 71 sources its claims cite were checked for support.
Abstract
Some aspects of science, taken at the broadest level, are universal in empirical research. These include collecting, analyzing, and reporting data. In each of these aspects, errors can and do occur. In this work, we first discuss the importance of focusing on statistical and data errors to continually improve the practice of science. We then describe underlying themes of the types of errors and postulate contributing factors. To do so, we describe a case series of relatively severe data and statistical errors coupled with surveys of some types of errors to better characterize the magnitude, frequency, and trends. Having examined these errors, we then discuss the consequences of specific errors or classes of errors. Finally, given the extracted themes, we discuss methodological, cultural, and system-level approaches to reducing the frequency of commonly observed errors. These approaches will plausibly contribute to the self-critical, self-correcting, ever-evolving practice of science, and ultimately to furthering knowledge.
Overview
CoChat Summary
TL;DR
This paper reviews data and statistical errors in empirical research, categorizes their themes and contributing factors, and proposes methodological, cultural, and system-level remedies. It argues that many invalidating errors are common, consequential, and often correctable, but that current correction processes are inconsistent and slow.
Key findings
- The authors define “invalidating errors” as factual mistakes or substantial deviations from accepted procedures that, if corrected, might alter a paper’s conclusions, and they group errors into themes including bad data, data management, statistical analysis, logic, and communication.
- Cited prevalence examples include: only 28% of 149 sampled medical journal studies were considered “acceptable”; 76% of conclusions or abstracts in 196 rheumatoid arthritis drug trials contained “doubtful or invalid statements”; 55% of psychology articles had errors, with 18% having gross errors; and “statcheck” found one in eight psychology articles had errors that could affect conclusions.
- Specific statistical problems discussed include misanalysis of cluster-randomized trials, incorrect effect-size calculations in meta-analyses (errors found in 17 of 27 meta-analyses), and the “differences in nominal significance” error; consequences include retractions and an estimated $28 billion spent on irreproducible preclinical research in the United States in 2015, about half of all preclinical research.
- Proposed responses include commenting on studies rather than authors, respecting due process, developing uniform and timely error-correction procedures, and using education, gatekeeper functions, incentives, resources, and cultural shifts to normalize error correction.
Methods
- Design: a review/commentary that presents a case series of relatively severe data and statistical errors coupled with surveys of some error types to characterize magnitude, frequency, and trends.
- Data or sample: a nonsystematic evaluation of the literature and selected published examples, drawing on cited surveys and meta-research across medicine, psychology, nutrition, and obesity research.
- Main analysis: narrative thematic synthesis of error types, contributing factors, consequences, and potential solutions, rather than a systematic review or pooled quantitative analysis.
Limitations
- The paper states that its provisional themes represent the authors’ opinions from their own vantage point and that there is no mathematical proof these are the best themes; the evaluation of the literature is nonsystematic.
- The paper states that categorizing errors requires a degree of subjectivity and that others have struggled with it; it also explicitly does not specifically address fabrication and falsification because they are defined by intentionality and egregiousness.
- Evident from the text: the prevalence figures are drawn from heterogeneous cited studies with different methods and fields, so they are not presented as pooled or systematically estimated rates.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
16 Valid · 6 Not valid · 36 Not checked · 70 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The measurement errors are sufficiently large and nonrandom that they have led to consistent and statistically significant correlations in the opposite direction from the true correlation for some relationships.”
What the sources say
- Partly supportedcites [19] sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The method, compared with objective measurements of actual energy intake, turns out to be invalid (19), not just “limited” or “imperfect.””
What the sources say
- Supportedcites [19] source
The 'Self-reported energy intake' section (p. 2-3) states that against the objective doubly labeled water measure, self-reported EI systematically underestimates intake by hundreds of kcal/d (up to ~800 kcal/d in NHANES) and that subjective estimates explained less than 10% of the variance of true EI. The Abstract (p. 1) explicitly argues for moving beyond the view that self-reports are merely 'imperfect' to treating them as 'wholly unacceptable for scientific research,' matching the citing sentence's 'invalid, not just limited or imperfect.' (LLM, conf=0.92)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“while 19 of 83 clustered cross-over studies were unclear as to whether both clustering and cross-over effects were included in the analysis (65).”
What the sources say
- Supportedcites [65] source
The Abstract and the Results section ('Quality of reporting in CRXO trials...', p. n/a) both state that of the 83 included CRXO trials, in 19 trials (23%) it could not be determined how or whether the analysis accounted for the cluster randomisation and multiple period aspects of the design. The Table 1 item 'Was it possible to determine the method for accounting for both the cluster randomisation and multiple period aspects? 64 (77%)' confirms the complement of 19/83 unclear, matching the cited figure and denominator exactly. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Associations across epidemiological studies are extrapolated to public health action (43).”
What the sources say
- Supportedcites [43] source
The Introduction states that dietary recommendations are built on epidemiological associations: 'The recommendation to eat a certain amount of fruit and vegetables is based on evaluation of short-term randomized controlled trials ... and of long-term prospective observational studies from which the majority report that a higher fruit and vegetable intake is associated with a reduced risk of all-cause mortality, cardiovascular disease, diabetes, hypertension and several cancer types.' The Discussion (p. Discussion) likewise describes WHO/FAO and Dietary Guidelines population goals resting on 'convincing'/'moderate' evidence from such studies, and the paper's own Conclusion carries its pooled cohort associations straight to action: 'the present findings might have public health relevance and support initiatives to increase the intake of fruit and vegetables.' The source makes this point specifically for fruit/vegetable intake and anthropometric/adiposity outcomes rather than as a universal methodological claim, and it critiques the strength of that extrapolation (GRADE-rated low-quality, inconsistent effects), but it clearly treats the topic the way the citing sentence does. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, misanalysis of cluster-randomized trials (27) may inappropriately and implicitly assume independence of observations.”
What the sources say
- Partly supportedcites [27] sourceExperimental — accuracy not yet measured
Our copy of Li et al. (2015) does address the statistical substance behind the claim in the sample-size paragraph: it notes that 'the impact of clustering, measured by the intraclass correlation coefficient, on the design effect can be profound', that power in a cRCT depends on df = c × (g − 1), that in the cRCT they illustrate 'inference ... was based not on df ≈ 454, but df = 12, as demanded by statistical theory', and that the criticized article 'does not adequately recognize the hierarchical nature of cRCTs'. But nowhere does the paper state that misanalysis of cRCTs inappropriately and implicitly assumes independence of observations; its criticisms concern the Sichieri & Cunha article's claims about baseline covariate adjustment, its meta-analysis, and its sample-size/df reasoning, not an independence assumption in the analysis of cRCT data. The general methodological assertion in the citing sentence is therefore only partly borne out by this source. (LLM, conf=0.65)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
34 more claims · 7 with their cited sources checked
Create a free account to see all 128 claims and to verify the 49 references still open.
- Probably not a substantial claim. 71
- It may state two separate claims as one. 41
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Published errors occur in the context of both genuine and fraudulent or fabricated research, where most studies have focussed on the detection and importance of honest errors (41, 42, 43, 44).”
From Identification of human gene research articles with wrongly identified nucleotide sequences (2022) · cites this paper as [41]
Experimental — accuracy not yet measured
Citations
Cites
2 retracted
Genetic Signatures of Exceptional Longevity in Humans
Sebastiani, P., Solovieff, N., Puca, A., Hartley, S. W., Melista, E., Andersen, S., Dworkis, D. A., Wilk, J. B., Myers, R. H., Steinberg, M. H., et al. (2010) · Science · doi:10.1126/science.1190532 · Cited by 141 · cited 1×
“Other common examples of bad data include confounding batch effects with study variables of interest (21) and cell-line misidentification or contamination (22).”
BackgroundRetracted- Retracted
The Market for "Lemons": Quality Uncertainty and the Market Mechanism
Akerlof, G. A. (1970) · The Quarterly Journal of Economics · doi:10.2307/1879431 · Cited by 23,016 · cited 1×
“An imbalance in trust between the purveyor and consumer of knowledge has been termed a “ lemon market ” (72), which is an idiomatic expression used to describe a decrease in quality that occurs from information asymmetry (73).”
An Inquiry into the Nature and Causes of the Wealth of Nations
Smith, A., Cannan, E., & Stigler, G. J. (1977) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226763750.001.0001 · Cited by 6,806
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
A Meta-Analysis of Personality in Scientific and Artistic Creativity
Feist, G. J. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0204_5 · Cited by 2,180
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407 · cited 1×
“Third, it is inappropriate to compare the nominal significance of two independent statistical tests as a means of drawing a conclusion about differential effects (30).”
BackgroundMicha, R., Peñalvo, J. L., Cudhea, F., Imamura, F., Rehm, C. D., & Mozaffarian, D. (2017) · JAMA · doi:10.1001/jama.2017.0947 · Cited by 1,218 · cited 1×
“In some cases, papers are careful to appropriately describe associations rather than statements of causation—like, “Dietary factors were estimated to be associated with a substantial proportion of deaths from heart disease, stroke, and type 2 diabetes,” (36).”
BackgroundThe Checklist Manifesto: How to Get Things Right
Gawande, A. (2011) · Journal of Nursing Regulation · doi:10.1016/s2155-8256(15)30310-0 · Cited by 1,151 · cited 1×
“25).”
Research works printed but not linked: 4
- ; American Council on Science and Health From Mice to Men: The Benefits and Limitations of Animal Testing in Predicting Human Cancer Risk (American Council on Science and Health, 3rd Ed, New York, 1991).
- F Benford, The law of anomalous numbers. Proc Am Philos Soc 78, 551–572 (1938).
- DS Chawla “We should err on the side of protecting people’s reputation:” Management journal changes policy to avoid fraud—interview with Patrick Wright. Retraction Watch . Available at retractionwatch.com/2016/07/13/management-journal-changes-its-policy-to-avoid-fraud-qa-with-editor-patrick-wright/. Accessed June 22 2016. (2016).
- M Scudellari “Strange. Very strange:” Retracted nutrition study reappears in new journal. Retraction Watch . Available at retractionwatch.com/2017/03/28/strange-strange-retracted-nutrition-study-reappears-new-journal/. Accessed September 22 2017. (2017).
Cited by
Synaptic degeneration in Alzheimer disease
Tzioras, M., McGeachan, R. I., Durrant, C. S., & Spires-Jones, T. L. (2023) · Nature Reviews Neurology · doi:10.1038/s41582-022-00749-z · Cited by 504
Lappan, S. N., Brown, A. W., & Hendricks, P. S. (2020) · Addiction · doi:10.1111/add.14793 · Cited by 370
Self-citationFactors to consider when choosing data collection methods
Mwita, K. (2022) · International Journal of Research in Business and Social Science (2147-4478) · doi:10.20525/ijrbs.v11i5.1842 · Cited by 73
“Novice and experienced researchers may encounter numerous methodological challenges specifically on what data should be collected, from whom to collect and by which means to collect them in order to achieve research objectives (Mwita, 2022; Brown et al., 2018; Barrett & Twycross, 2018).”
BackgroundHow to Write Statistical Analysis Section in Medical Research
Dwivedi, A. K. (2022) · Journal of Investigative Medicine · doi:10.1136/jim-2022-002479 · Cited by 81
“Some assumptions, executions, and errors made at the beginning of the study data collection may not be fixable 13 ; however, additional information collected during the study and data processing, including data distribution obtained at the end of the study, may facilitate additional considerations…”
White, N. M., Balasubramaniam, T., Nayak, R., & Barnett, A. G. (2022) · PLoS ONE · doi:10.1371/journal.pone.0264360 · Cited by 76
“However, using the wrong statistical methods can cause real harm [6, 8] and bad statistical practices are being to used abet weak science [2].”
Method
Metadata
- Authors
- Brown, A. W., Kaiser, K. A., & Allison, D. B.
- Affiliations
- University of Alabama at Birmingham (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2018-03-12
- Volume
- 115
- Issue
- 11
- Pages
- 2563-2570
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC5856502
- OpenAlex
- W2790042388
- Cited by
- 157
- References
- 95
- Citations by year
- 2026: 7 · 2025: 17 · 2024: 15 · 2023: 24 · 2022: 34 · 2021: 26 · 2020: 18 · 2019: 11 · 2018: 4
- FWCI
- 23.80
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- IUScholarWorks Open (Indiana University) (submittedVersion)
- Funders
- HHS | NIH | National Institute of Diabetes and Digestive and Kidney Diseases (R25DK099080, P30DK056336)
- HHS | NIH | National Institute of General Medical Sciences (R25GM116167)
- HHS | NIH | National Heart, Lung, and Blood Institute (R25HL124208)
- HHS | NIH | National Institute on Aging (P30AG050886)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Academic integrity and plagiarism
- Keywords
- data errors, statistical errors, research errors, error reduction, data collection, data analysis, scientific rigor, research culture
- MeSH terms
- Scientific Experimental Error, Data Collection, Humans, Quality Control, Research Design, Science, Statistics as Topic, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Issues with data and analyses: Errors, underlying themes, and potential solutions. https://app.cochat.ai/papers/10.1073/pnas.1708279115
- The paper
- Brown, A. W., Kaiser, K. A., & Allison, D. B. (2018). Issues with data and analyses: Errors, underlying themes, and potential solutions. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.1708279115
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.