Issues with data and analyses: Errors, underlying themes, and potential solutions
Open accessThe DOI matches its Crossref record; 95 of the 95 references it cites were found in a registry or an index; 22 of the 71 sources its claims cite were checked for support.
Abstract
Some aspects of science, taken at the broadest level, are universal in empirical research. These include collecting, analyzing, and reporting data. In each of these aspects, errors can and do occur. In this work, we first discuss the importance of focusing on statistical and data errors to continually improve the practice of science. We then describe underlying themes of the types of errors and postulate contributing factors. To do so, we describe a case series of relatively severe data and statistical errors coupled with surveys of some types of errors to better characterize the magnitude, frequency, and trends. Having examined these errors, we then discuss the consequences of specific errors or classes of errors. Finally, given the extracted themes, we discuss methodological, cultural, and system-level approaches to reducing the frequency of commonly observed errors. These approaches will plausibly contribute to the self-critical, self-correcting, ever-evolving practice of science, and ultimately to furthering knowledge.
Overview
CoChat Summary
TL;DR
This paper reviews data and statistical errors in empirical research, categorizes their themes and contributing factors, and proposes methodological, cultural, and system-level remedies. It argues that many invalidating errors are common, consequential, and often correctable, but that current correction processes are inconsistent and slow.
Key findings
- The authors define “invalidating errors” as factual mistakes or substantial deviations from accepted procedures that, if corrected, might alter a paper’s conclusions, and they group errors into themes including bad data, data management, statistical analysis, logic, and communication.
- Cited prevalence examples include: only 28% of 149 sampled medical journal studies were considered “acceptable”; 76% of conclusions or abstracts in 196 rheumatoid arthritis drug trials contained “doubtful or invalid statements”; 55% of psychology articles had errors, with 18% having gross errors; and “statcheck” found one in eight psychology articles had errors that could affect conclusions.
- Specific statistical problems discussed include misanalysis of cluster-randomized trials, incorrect effect-size calculations in meta-analyses (errors found in 17 of 27 meta-analyses), and the “differences in nominal significance” error; consequences include retractions and an estimated $28 billion spent on irreproducible preclinical research in the United States in 2015, about half of all preclinical research.
- Proposed responses include commenting on studies rather than authors, respecting due process, developing uniform and timely error-correction procedures, and using education, gatekeeper functions, incentives, resources, and cultural shifts to normalize error correction.
Methods
- Design: a review/commentary that presents a case series of relatively severe data and statistical errors coupled with surveys of some error types to characterize magnitude, frequency, and trends.
- Data or sample: a nonsystematic evaluation of the literature and selected published examples, drawing on cited surveys and meta-research across medicine, psychology, nutrition, and obesity research.
- Main analysis: narrative thematic synthesis of error types, contributing factors, consequences, and potential solutions, rather than a systematic review or pooled quantitative analysis.
Limitations
- The paper states that its provisional themes represent the authors’ opinions from their own vantage point and that there is no mathematical proof these are the best themes; the evaluation of the literature is nonsystematic.
- The paper states that categorizing errors requires a degree of subjectivity and that others have struggled with it; it also explicitly does not specifically address fabrication and falsification because they are defined by intentionality and egregiousness.
- Evident from the text: the prevalence figures are drawn from heterogeneous cited studies with different methods and fields, so they are not presented as pooled or systematically estimated rates.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
16 Valid · 6 Not valid · 36 Not checked · 70 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The measurement errors are sufficiently large and nonrandom that they have led to consistent and statistically significant correlations in the opposite direction from the true correlation for some relationships.”
What the sources say
- Partly supportedcites [19] sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The method, compared with objective measurements of actual energy intake, turns out to be invalid (19), not just “limited” or “imperfect.””
What the sources say
- Supportedcites [19] source
The 'Self-reported energy intake' section (p. 2-3) states that against the objective doubly labeled water measure, self-reported EI systematically underestimates intake by hundreds of kcal/d (up to ~800 kcal/d in NHANES) and that subjective estimates explained less than 10% of the variance of true EI. The Abstract (p. 1) explicitly argues for moving beyond the view that self-reports are merely 'imperfect' to treating them as 'wholly unacceptable for scientific research,' matching the citing sentence's 'invalid, not just limited or imperfect.' (LLM, conf=0.92)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“while 19 of 83 clustered cross-over studies were unclear as to whether both clustering and cross-over effects were included in the analysis (65).”
What the sources say
- Supportedcites [65] source
The Abstract and the Results section ('Quality of reporting in CRXO trials...', p. n/a) both state that of the 83 included CRXO trials, in 19 trials (23%) it could not be determined how or whether the analysis accounted for the cluster randomisation and multiple period aspects of the design. The Table 1 item 'Was it possible to determine the method for accounting for both the cluster randomisation and multiple period aspects? 64 (77%)' confirms the complement of 19/83 unclear, matching the cited figure and denominator exactly. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Associations across epidemiological studies are extrapolated to public health action (43).”
What the sources say
- Supportedcites [43] source
The Introduction states that dietary recommendations are built on epidemiological associations: 'The recommendation to eat a certain amount of fruit and vegetables is based on evaluation of short-term randomized controlled trials ... and of long-term prospective observational studies from which the majority report that a higher fruit and vegetable intake is associated with a reduced risk of all-cause mortality, cardiovascular disease, diabetes, hypertension and several cancer types.' The Discussion (p. Discussion) likewise describes WHO/FAO and Dietary Guidelines population goals resting on 'convincing'/'moderate' evidence from such studies, and the paper's own Conclusion carries its pooled cohort associations straight to action: 'the present findings might have public health relevance and support initiatives to increase the intake of fruit and vegetables.' The source makes this point specifically for fruit/vegetable intake and anthropometric/adiposity outcomes rather than as a universal methodological claim, and it critiques the strength of that extrapolation (GRADE-rated low-quality, inconsistent effects), but it clearly treats the topic the way the citing sentence does. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, misanalysis of cluster-randomized trials (27) may inappropriately and implicitly assume independence of observations.”
What the sources say
- Partly supportedcites [27] sourceExperimental — accuracy not yet measured
Our copy of Li et al. (2015) does address the statistical substance behind the claim in the sample-size paragraph: it notes that 'the impact of clustering, measured by the intraclass correlation coefficient, on the design effect can be profound', that power in a cRCT depends on df = c × (g − 1), that in the cRCT they illustrate 'inference ... was based not on df ≈ 454, but df = 12, as demanded by statistical theory', and that the criticized article 'does not adequately recognize the hierarchical nature of cRCTs'. But nowhere does the paper state that misanalysis of cRCTs inappropriately and implicitly assumes independence of observations; its criticisms concern the Sichieri & Cunha article's claims about baseline covariate adjustment, its meta-analysis, and its sample-size/df reasoning, not an independence assumption in the analysis of cRCT data. The general methodological assertion in the citing sentence is therefore only partly borne out by this source. (LLM, conf=0.65)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
34 more claims · 7 with their cited sources checked
Create a free account to see all 128 claims and to verify the 49 references still open.
- Probably not a substantial claim. 71
- It may state two separate claims as one. 41
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Published errors occur in the context of both genuine and fraudulent or fabricated research, where most studies have focussed on the detection and importance of honest errors (41, 42, 43, 44).”
From Identification of human gene research articles with wrongly identified nucleotide sequences (2022) · cites this paper as [41]
Experimental — accuracy not yet measured
Citations
Cites
2 retracted
Genetic Signatures of Exceptional Longevity in Humans
Sebastiani, P., Solovieff, N., Puca, A., Hartley, S. W., Melista, E., Andersen, S., Dworkis, D. A., Wilk, J. B., Myers, R. H., Steinberg, M. H., et al. (2010) · Science · doi:10.1126/science.1190532 · Cited by 141 · cited 1×
“Other common examples of bad data include confounding batch effects with study variables of interest (21) and cell-line misidentification or contamination (22).”
BackgroundRetracted- Retracted
The Market for "Lemons": Quality Uncertainty and the Market Mechanism
Akerlof, G. A. (1970) · The Quarterly Journal of Economics · doi:10.2307/1879431 · Cited by 23,016 · cited 1×
“An imbalance in trust between the purveyor and consumer of knowledge has been termed a “ lemon market ” (72), which is an idiomatic expression used to describe a decrease in quality that occurs from information asymmetry (73).”
An Inquiry into the Nature and Causes of the Wealth of Nations
Smith, A., Cannan, E., & Stigler, G. J. (1977) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226763750.001.0001 · Cited by 6,806
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
A Meta-Analysis of Personality in Scientific and Artistic Creativity
Feist, G. J. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0204_5 · Cited by 2,180
HARKing: Hypothesizing after the results are known
Kerr, N. L. (1998) · Personality and Social Psychology Review · doi:10.1207/s15327957pspr0203_4 · Cited by 1,989
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Gelman, A., & Stern, H. (2006) · The American Statistician · doi:10.1198/000313006x152649 · Cited by 1,407 · cited 1×
“Third, it is inappropriate to compare the nominal significance of two independent statistical tests as a means of drawing a conclusion about differential effects (30).”
BackgroundMicha, R., Peñalvo, J. L., Cudhea, F., Imamura, F., Rehm, C. D., & Mozaffarian, D. (2017) · JAMA · doi:10.1001/jama.2017.0947 · Cited by 1,218 · cited 1×
“In some cases, papers are careful to appropriately describe associations rather than statements of causation—like, “Dietary factors were estimated to be associated with a substantial proportion of deaths from heart disease, stroke, and type 2 diabetes,” (36).”
BackgroundThe Checklist Manifesto: How to Get Things Right
Gawande, A. (2011) · Journal of Nursing Regulation · doi:10.1016/s2155-8256(15)30310-0 · Cited by 1,151 · cited 1×
“25).”
The economics of reproducibility in preclinical research
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002165 · Cited by 1,055 · cited 1×
“One estimate calculated this waste to be half of all preclinical research, amounting to an estimated $28 billion going to irreproducible research in the United States alone in 2015 (79).”
BackgroundPeer Review: A Flawed Process at the Heart of Science and Journals
Smith, R. (2006) · Journal of the Royal Society of Medicine · doi:10.1177/014107680609900414 · Cited by 880 · cited 1×
“Furthermore, universal operational definitions and standards of peer review remain elusive (70), although new models of open review are evolving with some journals and publishers.”
BackgroundEnergy balance measurement: when something is not better than nothing
Dhurandhar, N. V., Schoeller, D., Brown, A. W., Heymsfield, S. B., Thomas, D., Sørensen, T. I. A., Speakman, J. R., Jeansonne, M., Allison, D. B., & the Energy Balance Measurement Working Group (2015) · International Journal of Obesity · doi:10.1038/ijo.2014.199 · Cited by 567
Self-citationWhy Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563 · cited 1×
“The corollary is that if we do not identify and correct errors, science cannot claim to be self-correcting, a concept that has been a source of critical discussion (2).”
BackgroundThe prevalence of statistical reporting errors in psychology (1985–2013)
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0664-2 · Cited by 510 · cited 2×
“Similar errors continue today, are sometimes severe enough to call entire studies into question (11), and may occur with nontrivial frequency (12 – 14).”
BackgroundMethodMyths, Presumptions, and Facts about Obesity
Casazza, K., Fontaine, K. R., Astrup, A., Birch, L. L., Brown, A. W., Bohan Brown, M. M., Durant, N., Dutton, G., Foster, E. M., Heymsfield, S. B., et al. (2013) · New England Journal of Medicine · doi:10.1056/nejmsa1208051 · Cited by 442
Self-citationThe Costs of using Unauthenticated, Over-Passaged Cell Lines: How Much more Data do we Need?
Hughes, P., Marshall, D., Reid, Y., Parkes, H., & Gelber, C. (2007) · BioTechniques · doi:10.2144/000112598 · Cited by 434 · cited 1×
“Other common examples of bad data include confounding batch effects with study variables of interest (21) and cell-line misidentification or contamination (22).”
BackgroundLikelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Kaplan, R. M., & Irvin, V. L. (2015) · PLOS ONE · doi:10.1371/journal.pone.0132382 · Cited by 356
The (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356 · cited 1×
“Bakker and Wicherts surveyed reporting of statistical analyses in psychology journals and noted that 55% of articles had errors, with 18% having gross errors (67).”
BackgroundAn introduction to statistical finance
Bouchaud, J.-P. (2002) · Physica A: Statistical Mechanics and its Applications · doi:10.1016/s0378-4371(02)01039-7 · Cited by 348
Gøtzsche, P. C. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90017-2 · Cited by 339 · cited 1×
“More than 20 y later, an analysis of 196 drug trials to treat rheumatoid arthritis concluded that 76% of the conclusions or abstracts contained “doubtful or invalid statements” (63).”
BackgroundThe Checklist Manifesto. How to get things right: Book review
Newbold, D. (2010) · Journal of Nursing Management · doi:10.1111/j.1365-2834.2010.01123.x · Cited by 338
Yavchitz, A., Boutron, I., Bafeta, A., Marroun, I., Charles, P., Mantz, J., & Ravaud, P. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001308 · Cited by 293 · cited 1×
“Although lay media have contributed to sensationalism and misrepresentation of science, many of these mistaken messages originate from the scientific community itself (78).”
BackgroundLord Kelvin and the Age of the Earth
Burchfield, J. D. (1975) · doi:10.1007/978-1-349-02565-7 · Cited by 271
Correlation not Causation: The Relationship between Personality Traits and Political Ideologies
Verhulst, B., Eaves, L. J., & Hatemi, P. K. (2012) · American Journal of Political Science · doi:10.1111/j.1540-5907.2011.00568.x · Cited by 262
Reproducibility: A tragedy of errors
Allison, D. B., Brown, A. W., George, B. J., & Kaiser, K. A. (2016) · Nature · doi:10.1038/530027a · Cited by 255
Self-citationData Extraction Errors in Meta-analyses That Use Standardized Mean Differences
Gøtzsche, P. C., Hróbjartsson, A., Marić, K., & Tendal, B. (2007) · JAMA · doi:10.1001/jama.298.4.430 · Cited by 238 · cited 1×
“Data extraction for effect-size calculation was similarly problematic, with errors found in 17 of 27 metaanalyses (66).”
Carlisle, J. B. (2017) · Anaesthesia · doi:10.1111/anae.13938 · Cited by 229 · cited 1×
“Similar errors continue today, are sometimes severe enough to call entire studies into question (11), and may occur with nontrivial frequency (12 – 14).”
BackgroundSchwingshackl, L., Hoffmann, G., Kalle-Uhlmann, T., Arregui, M., Buijsse, B., & Boeing, H. (2015) · PLOS ONE · doi:10.1371/journal.pone.0140846 · Cited by 218 · cited 1×
“Associations across epidemiological studies are extrapolated to public health action (43).”
BackgroundBouter, L. M., Tijdink, J., Axelsen, N., Martinson, B. C., & ter Riet, G. (2016) · Research Integrity and Peer Review · doi:10.1186/s41073-016-0024-5 · Cited by 211 · cited 3×
“(16), for instance, classified “research misbehaviors” in four domains: reporting, collaboration, data collection, and study design.”
BackgroundStatistical Evaluation of Medical Journal Manuscripts
Schor, S. (1966) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1966.03100130097026 · Cited by 207 · cited 1×
“Systematic sampling of 149 studies from popular medical journals concluded that only 28% of the sample was considered “acceptable” (62).”
BackgroundThe Papers of Alexander Hamilton
Syrett, H. C., & Cooke, J. E. (1957) · The Historian · doi:10.1111/j.1540-6563.1957.tb01801.x · Cited by 170
Weighing the Evidence of Common Beliefs in Obesity Research
Casazza, K., Brown, A., Astrup, A., Bertz, F., Baum, C., Brown, M. B., Dawson, J., Durant, N., Dutton, G., Fields, D. A., et al. (2015) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2014.922044 · Cited by 169
Self-citationHow to defend society against science
Feyerabend, P. K. (1975) · Radical philosophy · Cited by 167
The Relative Safety of Ephedra Compared with Other Herbal Products
Bent, S., Tiedt, T. N., Odden, M. C., & Shlipak, M. G. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-6-200303180-00010 · Cited by 165
Brown, A. W., Bohan Brown, M. M., & Allison, D. B. (2013) · The American Journal of Clinical Nutrition · doi:10.3945/ajcn.113.064410 · Cited by 155 · cited 2×
“Assuming causation from observational evidence is common (34, 35).”
BackgroundSelf-citationConflict of interest. The new McCarthyism in science
Rothman, K. J. (1993) · JAMA · doi:10.1001/jama.269.21.2782 · Cited by 154
(2013) · Choice Reviews Online · doi:10.5860/choice.51-1458 · Cited by 144
Arrison, T., & Nerem, R. M. (2017) · Special Publications · doi:10.1002/9781119067825.ch2 · Cited by 135 · cited 2×
“Furthermore, we fully recognize that categorizing errors requires a degree of subjectivity and is something that others have struggled with (15, 16).”
BackgroundAn Introduction To Statistical Methods
Gupta, C. B. (1982) · Andalas University Repository (Andalas University) · Cited by 134
McGough, G. (2010) · Nursing Management · doi:10.7748/nm.17.1.8.s10 · Cited by 130
Failure: why science is so successful
(2016) · Choice Reviews Online · doi:10.5860/choice.194830 · Cited by 128
Brown, N. J. L., & Heathers, J. A. J. (2017) · Social Psychological and Personality Science · doi:10.1177/1948550616673876 · Cited by 127
Cope, M. B., & Allison, D. B. (2010) · International Journal of Obesity · doi:10.1038/ijo.2009.239 · Cited by 126 · cited 1×
“In the interest of protecting children from obesity, researchers may succumb to White Hat Bias, which involves “distortion of information in the service of what may be perceived to be righteous ends” (56).”
BackgroundSelf-citationConflict of Interest: The New McCarthyism in Science
Rothman, K. J. (1993) · JAMA · doi:10.1001/jama.1993.03500210082037 · Cited by 113
Food Intakes of Obese and Non-Obese Women
Beaudoin, R., & Mayer, J. (1953) · Journal of the American Dietetic Association · doi:10.1016/s0002-8223(21)31635-2 · Cited by 111 · cited 1×
“Concerns about this method were raised decades ago (20), and yet its use is continued.”
MethodCope, M. B., & Allison, D. B. (2008) · Obesity Reviews · doi:10.1111/j.1467-789x.2008.00504.x · Cited by 110 · cited 1×
“In turn, future research may parrot the righteous stance regardless of true effects, such as when the potential for publication bias was ignored in a World Health Organization report of breastfeeding and obesity (57).”
BackgroundSelf-citationBrown, A. W., Li, P., Brown, M. M. B., Kaiser, K. A., Keith, S. W., Oakes, J. M., & Allison, D. B. (2015) · American Journal of Clinical Nutrition · doi:10.3945/ajcn.114.105072 · Cited by 104 · cited 1×
“While we are aware of no formal, long-standing catalog of errors, either in frequency or category, efforts have critiqued common errors in focused areas of the literature (31, 58), aiming to educate the particular community where these are observed.”
BackgroundSelf-citationHopewell, S., Collins, G. S., Boutron, I., Yu, L.-M., Cook, J., Shanyinde, M., Wharton, R., Shamseer, L., & Altman, D. G. (2014) · BMJ · doi:10.1136/bmj.g4145 · Cited by 103 · cited 1×
“with the existence of guidelines and journal mandates, authors do not always report information specified in guidelines, nor do peer reviewers demand that the information be reported (69).”
BackgroundThe Lanarkshire Milk Experiment
Student (1931) · Biometrika · doi:10.2307/2332424 · Cited by 100
Methodological Errors in Medical Research
Andersen, B. R., & Wurzinger, L. J. (1991) · Blood Coagulation & Fibrinolysis · doi:10.1097/00001721-199102000-00035 · Cited by 99
Common scientific and statistical errors in obesity research
George, B. J., Beasley, T. M., Brown, A. W., Dawson, J., Dimova, R., Divers, J., Goldsby, T. U., Heo, M., Kaiser, K. A., Keith, S. W., et al. (2016) · Obesity · doi:10.1002/oby.21449 · Cited by 97 · cited 2×
“While we are aware of no formal, long-standing catalog of errors, either in frequency or category, efforts have critiqued common errors in focused areas of the literature (31, 58), aiming to educate the particular community where these are observed.”
BackgroundSelf-citationA Large-Scale Analysis of Impact Factor Biased Journal Self-Citations
Chorus, C., & Waltman, L. (2016) · PLOS ONE · doi:10.1371/journal.pone.0161021 · Cited by 92
A systematic appraisal of allegiance effect in randomized controlled trials of psychotherapy
Dragioti, E., Dimoliatis, I., Fountoulakis, K. N., & Evangelou, E. (2015) · Annals of General Psychiatry · doi:10.1186/s12991-015-0063-1 · Cited by 91 · cited 1×
“When individuals believe in an idea so fully or have built an entire career and image upon an idea, publishing something to the contrary would be to conflict with an entrenched ideology (54, 55).”
BackgroundHow Not to Be Wrong: The Power of Mathematical Thinking
Wiseman, J. (2015) · Math Horizons · doi:10.4169/mathhorizons.22.3.24 · Cited by 85
The average contribution of each several ancestor to the total heritage of the offspring
Galton, F. (1897) · Proceedings of the Royal Society of London · doi:10.1098/rspl.1897.0052 · Cited by 82
Learned caloric adjustment of human intake
Louis-Sylvestre, J., Tournier, A., Verger, P., Chabert, M., Delorme, B., & Hossenlopp, J. (1989) · Appetite · doi:10.1016/0195-6663(89)90099-8 · Cited by 79 · cited 1×
“40).”
Use of causal language in observational studies of obesity and nutrition
Cofield, S. S., Corona, R. V., & Allison, D. B. (2010) · Obesity Facts · doi:10.1159/000322940 · Cited by 74 · cited 1×
“Assuming causation from observational evidence is common (34, 35).”
BackgroundSelf-citationThomas, D. M., Martin, C. K., Lettieri, S., Bredlau, C., Kaiser, K., Church, T., Bouchard, C., & Heymsfield, S. B. (2013) · International Journal of Obesity · doi:10.1038/ijo.2013.51 · Cited by 70 · cited 1×
“Effects on outcomes like energy intake are linearly extrapolated to weight change (41), despite energy balance being a dynamic, nonlinear system (42).”
BackgroundSelf-citationHow not to be wrong: the power of mathematical thinking
Ellenberg, J. S. (2014) · Choice Reviews Online · doi:10.5860/choice.185307 · Cited by 66 · cited 2×
“A striking example is the publication of apparently highly statistically significant results in the “Bible Code” that were later debunked as a variant of p-hacking (9).”
BackgroundGadbury, G. L., & Allison, D. B. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046363 · Cited by 61 · cited 1×
“, researcher degrees of freedom (45) and p-fiddling (46), among other names] represent a form of selective reporting and may also be considered errors of statistical analysis.”
BackgroundSelf-citationThe Registration of Observational Studies—When Metaphors Go Bad
(2010) · Epidemiology · doi:10.1097/ede.0b013e3181eafbcf · Cited by 56 · cited 1×
“Gatekeeping functions may increase this burden, and have been criticized as the bureaucrati-zation of science (107).”
BackgroundLord Kelvin and the Age of the Earth
Burchfield, J. D. (1990) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226080260.001.0001 · Cited by 56
Unscientific Beliefs about Scientific Topics in Nutrition
Brown, A. W., Ioannidis, J. P. A., Cope, M. B., Bier, D. M., & Allison, D. B. (2014) · Advances in Nutrition · doi:10.3945/an.114.006577 · Cited by 54 · cited 1×
“…our fields of nutrition and obesity, the constant vacillation of headlines and studies pur-porting that a food is good and then perhaps bad from week to week has contributed to a decreased trust in nutrition science, with many unscientific ideas being advanced through higher exposure (34, 74 – 77).”
BackgroundSelf-citationThe Papers of Alexander Hamilton, by Harold C. Syrett, Jacob E. Cooke
Brown, R. E. (1962) · Political Science Quarterly · doi:10.2307/2146324 · Cited by 50
Amending published articles: time to rethink retractions and corrections?
Barbour, V., Bloom, T., Lin, J., & Moylan, E. C. (2017) · F1000Research · doi:10.12688/f1000research.13060.1 · Cited by 46
Goals in Nutrition Science 2015–2020
Allison, D. B., Bassaganya-Riera, J., Burlingame, B., Brown, A. W., le Coutre, J., Dickson, S. L., van Eden, W., Garssen, J., Hontecillas, R., Khoo, C. S. H., et al. (2015) · Frontiers in Nutrition · doi:10.3389/fnut.2015.00026 · Cited by 45 · cited 1×
“Thus, we are not considering the missteps in thinking or procedures necessary for progress in new ideas and theories (17).”
BackgroundSelf-citationA Statistical Model to Explain the Mendel–Fisher Controversy
Pires, A. M., & Branco, J. A. (2010) · Statistical Science · doi:10.1214/10-sts342 · Cited by 29 · cited 1×
“In some respects, Gregor Mendel’s classic pea genetics studies are consistent with this latter model, with data likely too perfect to have arisen naturally (49).”
BackgroundAn Introduction to Statistical Methods
Working, H., Secrist, H. A., & Jerome, H. (1926) · The Journal of Land & Public Utility Economics · doi:10.2307/3138984 · Cited by 26
The FDP Faculty Burden Survey.
Rockwell, S. (2009) · PubMed · Cited by 25
Tokolahi, E., Hocking, C. S., Kersten, P., & Vandal, A. C. (2015) · OTJR Occupational Therapy Journal of Research · doi:10.1177/1539449215618625 · Cited by 22 · cited 1×
“For cluster randomized trials in occupational therapy interventions, 7 of 10 identified studies included clustering in the analysis (64), while 19 of 83 clustered cross-over studies were unclear as to whether both clustering and cross-over effects were included in the analysis (65).”
Arnup, S. J., Forbes, A. B., Kahan, B. C., Morgan, K. E., & McKenzie, J. E. (2016) · Trials · doi:10.1186/s13063-016-1685-6 · Cited by 20
To Err is Human: Can American Medicine Learn from Past Mistakes?
Ritterman, J. B. (2017) · The Permanente Journal · doi:10.7812/tpp/16-181 · Cited by 14 · cited 1×
“Poor reasoning and data collection resulted in radiation treatment of children to prevent sudden infant death syndrome in the early 1900s, resulting in >10,000 babies dying of thyroid cancer (71).”
BackgroundZalewski, B., Chmielewska, A., Szajewska, H., Keithley, J. K., Li, P., Goldsby, T. U., & Allison, D. B. (2015) · Nutrition · doi:10.1016/j.nut.2015.02.008 · Cited by 14 · cited 1×
“In turn, the metaanalytic estimates from these effect-size calculations may be incorrect, and have sometimes required correction (29).”
BackgroundSelf-citationNeed for new review of article on ketogenic dietary regimes for cancer patients
Klement, R. J., Feinman, R. D., Gross, E. C., Champ, C. E., D’Agostino, D. P., Fine, E. J., Kämmerer, U., Poff, A. M., Rho, J. M., Seyfried, T. N., et al. (2017) · Medical Oncology · doi:10.1007/s12032-017-0968-4 · Cited by 14 · cited 1×
“However, in trying to counter the points of some authors or studies, some individuals resort to ad hominem arguments, often trying to undermine the credibility of arguments by attacking a person based on perceived expertise (85) or presumed motives, focusing especially on funding sources (86).”
BackgroundNutritional epidemiology in practice: learning from data or promulgating beliefs?
Bohan Brown, M. M., Brown, A. W., & Allison, D. B. (2013) · The American Journal of Clinical Nutrition · doi:10.3945/ajcn.112.052472 · Cited by 11 · cited 1×
“…our fields of nutrition and obesity, the constant vacillation of headlines and studies pur-porting that a food is good and then perhaps bad from week to week has contributed to a decreased trust in nutrition science, with many unscientific ideas being advanced through higher exposure (34, 74 – 77).”
BackgroundSelf-citationCorrecting the scientific literature: retraction and republication
Lancet, T. (2015) · The Lancet · doi:10.1016/s0140-6736(15)60137-4 · Cited by 10 · cited 1×
““Retraction and republication” could be used to both maintain the historical record and correct the scientific literature (110).”
BackgroundLi, P., Brown, A. W., Dawson, J. A., Kaiser, K. A., Bohan Brown, M. M., Keith, S. W., Oakes, J. M., & Allison, D. B. (2015) · Obesity Facts · doi:10.1159/000381434 · Cited by 8 · cited 1×
“First, misanalysis of cluster-randomized trials (27) may in-appropriately and implicitly assume independence of observations.”
BackgroundSelf-citationErrors in statistical analysis and questionable randomization lead to unreliable conclusions.
George, B. J., Brown, A. W., & Allison, D. B. (2015) · Archives of Advances in Biosciences · doi:10.22037/jps.v6i3.10273 · Cited by 8 · cited 1×
“We asked authors once to share data from a publication with implausible baseline imbalances and other potential statistical anomalies; they were unable to produce the data, and the journal retracted the paper (26).”
BackgroundSelf-citationVerhulst, B., Eaves, L. J., & Hatemi, P. (2015) · American Journal of Political Science · doi:10.1111/ajps.12216 · Cited by 8
Salsburg, D. S. (2017) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315379081 · Cited by 7
Bohan Brown, M. M., Brown, A. W., & Allison, D. B. (2015) · Biological Psychiatry · doi:10.1016/j.biopsych.2014.10.028 · Cited by 6 · cited 1×
“Effects on outcomes like energy intake are linearly extrapolated to weight change (41), despite energy balance being a dynamic, nonlinear system (42).”
BackgroundSelf-citationScientific integrity and the market for lemons
Cottrell, R. C. (2014) · Research Ethics · doi:10.1177/1747016113494651 · Cited by 6 · cited 1×
“If scientists cannot produce reliable, reproducible experiments, why should the public trust the scientific enterprise? An imbalance in trust between the purveyor and consumer of knowledge has been termed a “lemon market” (72), which is an idiomatic expression used to describe a decrease in quality that occurs from in…”
BackgroundExamination of published articles with respect to statistical errors in veterinary sciences
Ercan, I., Kaya, M. O., Uzabaci, E., Mankir, S., Can, F. E., & Bashir Albishir, M. (2017) · Acta Veterinaria · doi:10.1515/acve-2017-0004 · Cited by 6
Allison, D. B., Williams, M. S., Hand, G. A., Jakicic, J. M., & Fontaine, K. R. (2015) · Disability and Rehabilitation · doi:10.3109/09638288.2014.1002580 · Cited by 5 · cited 1×
“32 and 33).”
Self-citationU.S. report calls for research integrity board
Mervis, J. D. (2017) · Science · doi:10.1126/science.356.6334.123 · Cited by 4 · cited 1×
“The proposed Research Integrity Advisory Board (103), combined with the Committee on Publication Ethics and the International Committee of Medical Journal Editors may be guiding bodies through which this could be accomplished.”
BackgroundHauner, H. (2015) · Obesity Facts · doi:10.1159/000381481 · Cited by 3 · cited 1×
“Editors may abrogate their responsibility for resolving claims of invalidating errors (100), leaving it to teams of authors to make cases in opposing letters (e.”
BackgroundSerlie, M. J. M., ter Horst, K. W., & Brown, A. W. (2016) · Hepatology · doi:10.1002/hep.28588 · Cited by 3 · cited 1×
“32 and 33).”
Self-citationEnserink, M. (2017) · Science · doi:10.1126/science.356.6342.998 · Cited by 3 · cited 1×
“Some have suggested retiring or replacing the allencompassing word “retraction” (which conjures “misconduct”) with more meaningful descriptions of correction (109).”
BackgroundU.S. report calls for research integrity board
Mervis, J. D. (2017) · Science · doi:10.1126/science.aal1030 · Cited by 2
Cuspidi, C., Rescaldani, M., Tadic, M., Sala, C., & Grassi, G. (2016) · American Journal of Hypertension · doi:10.1093/ajh/hpw038 · Cited by 1
Kaiser, K. A., Parman, M. A., Kim, E., George, B. J., & Allison, D. B. (2016) · American Journal of Hypertension · doi:10.1093/ajh/hpw027 · Cited by 1
Self-citationTrasande, L., Lind, P., Lampa, E., & Lind, L. (2017) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2017-208901 · Cited by 1 · cited 2×
“These attacks (38) are not new (87), and remain distractions from the science itself.”
BackgroundResultRobert, C. P. (2018) · CHANCE · doi:10.1080/09332480.2018.1438715 · Cited by 1
Amending published articles: time to rethink retractions and corrections?
Barbour, V., Bloom, T., Lin, J., & Moylan, E. (2017) · doi:10.1101/118356 · cited 1×
“Experience indicates that the handling of such errors (both purported and confirmed) is haphazard, unduly slow, inconsistent, and often markedly socially inappropriate (11, 80, 81).”
Background
Research works printed but not linked: 5
- ; National Academies of Sciences Engineering, & Medicine Fostering Integrity in Research (The National Academies Press, Washington, DC, 2017).
- ; American Council on Science and Health From Mice to Men: The Benefits and Limitations of Animal Testing in Predicting Human Cancer Risk (American Council on Science and Health, 3rd Ed, New York, 1991).
- F Benford, The law of anomalous numbers. Proc Am Philos Soc 78, 551–572 (1938).
- DS Chawla “We should err on the side of protecting people’s reputation:” Management journal changes policy to avoid fraud—interview with Patrick Wright. Retraction Watch . Available at retractionwatch.com/2016/07/13/management-journal-changes-its-policy-to-avoid-fraud-qa-with-editor-patrick-wright/. Accessed June 22 2016. (2016).
- M Scudellari “Strange. Very strange:” Retracted nutrition study reappears in new journal. Retraction Watch . Available at retractionwatch.com/2017/03/28/strange-strange-retracted-nutrition-study-reappears-new-journal/. Accessed September 22 2017. (2017).
Cited by
Synaptic degeneration in Alzheimer disease
Tzioras, M., McGeachan, R. I., Durrant, C. S., & Spires-Jones, T. L. (2023) · Nature Reviews Neurology · doi:10.1038/s41582-022-00749-z · Cited by 504
Lappan, S. N., Brown, A. W., & Hendricks, P. S. (2020) · Addiction · doi:10.1111/add.14793 · Cited by 370
Self-citationFactors to consider when choosing data collection methods
Mwita, K. (2022) · International Journal of Research in Business and Social Science (2147-4478) · doi:10.20525/ijrbs.v11i5.1842 · Cited by 73
“Novice and experienced researchers may encounter numerous methodological challenges specifically on what data should be collected, from whom to collect and by which means to collect them in order to achieve research objectives (Mwita, 2022; Brown et al., 2018; Barrett & Twycross, 2018).”
BackgroundHow to Write Statistical Analysis Section in Medical Research
Dwivedi, A. K. (2022) · Journal of Investigative Medicine · doi:10.1136/jim-2022-002479 · Cited by 82
“Some assumptions, executions, and errors made at the beginning of the study data collection may not be fixable 13 ; however, additional information collected during the study and data processing, including data distribution obtained at the end of the study, may facilitate additional considerations…”
White, N. M., Balasubramaniam, T., Nayak, R., & Barnett, A. G. (2022) · PLoS ONE · doi:10.1371/journal.pone.0264360 · Cited by 76
“However, using the wrong statistical methods can cause real harm [6, 8] and bad statistical practices are being to used abet weak science [2].”
Method
Metadata
- Authors
- Brown, A. W., Kaiser, K. A., & Allison, D. B.
- Affiliations
- University of Alabama at Birmingham (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2018-03-12
- Volume
- 115
- Issue
- 11
- Pages
- 2563-2570
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC5856502
- OpenAlex
- W2790042388
- Cited by
- 157
- References
- 95
- Citations by year
- 2026: 7 · 2025: 17 · 2024: 15 · 2023: 24 · 2022: 34 · 2021: 26 · 2020: 18 · 2019: 11 · 2018: 4
- FWCI
- 23.80
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- IUScholarWorks Open (Indiana University) (submittedVersion)
- Funders
- HHS | NIH | National Institute of Diabetes and Digestive and Kidney Diseases (R25DK099080, P30DK056336)
- HHS | NIH | National Institute of General Medical Sciences (R25GM116167)
- HHS | NIH | National Heart, Lung, and Blood Institute (R25HL124208)
- HHS | NIH | National Institute on Aging (P30AG050886)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Academic integrity and plagiarism
- Keywords
- data errors, statistical errors, research errors, error reduction, data collection, data analysis, scientific rigor, research culture
- MeSH terms
- Scientific Experimental Error, Data Collection, Humans, Quality Control, Research Design, Science, Statistics as Topic, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.