The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 44 of the 78 sources its claims cite were checked for support.
Is science really facing a reproducibility crisis, and do we need it to?
Open accessAbstract
Efforts to improve the reproducibility and integrity of science are typically justified by a narrative of crisis, according to which most published results are unreliable due to growing problems with research and publication practices. This article provides an overview of recent evidence suggesting that this narrative is mistaken, and argues that a narrative of epochal changes and empowerment of scientists would be more accurate, inspiring, and compelling.
Overview
CoChat Summary
TL;DR
Fanelli argues that the “science in crisis” reproducibility narrative is at least partially misguided: integrity and reproducibility problems are real but do not distort most of the literature, are heterogeneously distributed, and are not growing. He proposes a narrative of epochal change and empowerment of scientists instead.
Key findings
- Anonymous surveys suggest on average 1–2% of scientists admit to having fabricated or falsified data at least once; the affected literature is likely smaller, at least five times smaller according to a survey among psychologists. Random audits found “scientific improprieties” in 0.28% of cancer clinical trials, problems sufficient for “for cause” investigations in 4% of FDA clinical trials from 1977–1988, and visual inspections suggested 1–2% of microbiology papers were manipulated in ways suggesting intentional fabrication.
- P-hacking and false positives appear contained: analyses of P values in the medical literature suggested a false-discovery rate of only 14%; a broader analysis found P-hacking was common but had minor effects on meta-analytic conclusions, with stronger “evidential value” suggesting most published studies measure true effects. Low statistical power is below the recommended 80% in several disciplines but varies widely by subfield; for most findings to be false in psychology and neuroscience, one must assume tested hypotheses are correct much less than 50% of the time, which Fanelli calls unrealistic.
- Large-scale reproducibility assessments are mixed: a “Many labs” project reported 10 of 13 psychological studies replicated consistently; an experimental economics analysis suggested at least 11 of 18 studies replicated; the largest psychological reproducibility initiative suggested reproducibility below 50%, but Fanelli argues this is likely too pessimistic because reproducibility varied by subfield, methodology, and replicator expertise, and reanalyses suggest it may be higher.
- There is no evidence that misconduct and questionable research practices have increased: yearly ORI findings and the proportion of ORI investigations resulting in misconduct findings have not increased; retractions per retracting journal have not increased; errata and corrections have not increased; publication rate has not increased since the 1950s once coauthorship is adjusted for; and pressures to publish have not been convincingly linked to bias or misconduct.
Methods
- This is an opinion/overview article, not a new empirical study; it synthesizes recent metaresearch evidence on misconduct, questionable research practices, P-hacking, statistical power, bias, and reproducibility.
- It includes an illustrative Web of Science analysis (Fig. 1) counting records whose title, abstract, or keywords contain phrases such as “reproducibility crisis,” “replication crisis,” or “science in crisis,” classified by the author as endorsing, questioning, or otherwise discussing crisis narratives; the complete dataset dates back to 1933 and is in Dataset S1.
- The main analysis is narrative argument and synthesis rather than a systematic review or formal meta-analysis.
Limitations
- The paper states that the Nature survey on the reproducibility crisis was nonrandomized and may not accurately represent practicing scientists; direct data estimating misconduct prevalence are scarce; assumptions about true effect sizes, research bias, and prior probabilities are difficult to measure; and how reproducibility ought to be measured is subject to ongoing methodological and philosophical debate.
- The text makes evident that this is an opinion piece with illustrative, non-systematic evidence selection; Fig. 1 is described as “merely illustrative” and excludes many articles that discuss the crisis narrative without using the searched phrases.
- The paper does not report a formal quantitative synthesis, so the strength of its conclusions depends on the cited metaresearch literature rather than new primary analysis.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
27 Valid · 2 Not valid · 15 Not checked · 27 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“First because, once again, such a low level of reproducibility was not ubiquitous but varied depending on subfield, methodology, and expertise of the authors conducting the replication (23 -25).”
What the sources say
- Disputedcites [23] sourceExperimental — accuracy not yet measured
Our copy supports variation by subfield and methodology: Evaluating Reproducibility (p. 11) reports 26% of social-psychology effects vs 53% of cognitive-psychology effects replicating at p<.05, and 46% of main/simple-effect tests vs 22% of interaction-effect tests. However, it directly contradicts the third conjunct: Evaluating Reproducibility (p. 12) states there was 'little evidence that ... experience and expertise of the original or replication teams ... accounted for meaningful variation in reproducibility,' and the Discussion (p. 12) says of such factors that 'the evidence is just that they did not do so here.' Attributing dependence on the expertise of the replication authors to this paper is therefore contrary to what it reports. (LLM, conf=0.75)
Checked by a reasoning model — may be wrong.
- Partly supportedcites [24] sourceExperimental — accuracy not yet measured
Van Bavel et al. (2016) supports only the 'methodology' component: their regressions (Results, model 1/model 2; Fig. 1) show statistical power of the replication, original effect size, surprisingness and replication similarity predicted replication success, and the Discussion states 'the variance explained by contextual sensitivity is surpassed by the statistical power of the replication attempt.' The 'subfield' component runs against this paper: the Results report social psychology studies were rated more contextually sensitive than cognitive ones but 'we found no evidence of an interaction between contextual sensitivity and subdiscipline on replication success (P = 0.877)', with nearly identical odds ratios (0.823 vs 0.892), and the abstract states the association 'did not differ across psychological subdisciplines.' The 'expertise of the authors conducting the replication' part is not examined: the paper's expertise finding (Discussion) is that experts/original researchers can identify factors influencing reproducibility and that original authors' concerns predicted failure — coders were deliberately blind to replicator characteristics, and replication-team expertise was never entered as a predictor. (LLM, conf=0.72)
- Not checked yetcites [25] source
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“such studies showed that power varies widely between subfields or methodologies”
What the sources say
- Partly supportedcites [13] sourceExperimental — accuracy not yet measured
The Results (Statistical power) show substantial differences by methodology: cognitive/behavioural studies were mostly above 80% power, while candidate-gene studies were mostly below 20%. The Conclusion, however, finds no clear evidence that average power differs across the three disease research domains studied, so the subfield part is not supported by this comparison. Page numbers were not supplied. (LLM, conf=0.86)
- Partly supportedcites [14] sourceExperimental — accuracy not yet measured
The Results (Table 1) show substantial differences in estimated power across subfields: median power for medium effects was 0.40 in cognitive neuroscience versus 0.60 in psychology. The Discussion’s limitations say that power may also differ by study design, but the authors did not record study-design information, so variation between methodologies was not assessed. (LLM, conf=0.84)
- Not checked yetcites [15] source
- Partly supportedcites [16] sourceExperimental — accuracy not yet measured
Methods and Results reports different estimated power across social/personality psychology subfields: .43 and .49 for two social-psychology sections of JPSP versus .60 for its personality section. It also reports differences across journals. The passages do not directly compare statistical power by methodology, and the evidence is limited to social/personality psychology. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In several disciplines, the average statistical power of studies was found to be significantly below the recommended 80% level (13 -16).”
What the sources say
- Supportedcites [13] source
The Discussion reports similar average statistical power across three biomedical research domains and says that roughly half of studies had power in the 0–20% range, well below the conventional 80% minimum. The Results section identifies the domains as psychiatry, neurology and somatic disease. (LLM, conf=0.91)
- Partly supportedcites [14] sourceExperimental — accuracy not yet measured
Table 1 in the Results reports mean power below 0.80 for small, medium, and large effects in cognitive neuroscience, psychology, and the medically oriented journals surveyed. The estimates depend on the effect size being detected; the paper does not identify 80% as the recommended level or report a statistical test showing that the means are significantly below it. (LLM, conf=0.82)
- Not checked yetcites [15] source
- Partly supportedcites [16] sourceExperimental — accuracy not yet measured
In Methods and Results (page unavailable), the paper reports that typical studies in several social/personality psychology journals had estimated power well below Cohen’s 80% benchmark for detecting an effect of r = .20. It does not establish this finding across several disciplines, nor does the cited passage report a statistical test showing that the shortfall was significant. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Visual inspections of microbiology papers suggested that between 1% and 2% of papers had been manipulated in ways that suggested intentional fabrication (6, 7).”
What the sources say
- Not checked yetcites [6] source
- Partly supportedcites [7] sourceExperimental — accuracy not yet measured
Bik et al. (2016) do report a visual screen of published papers (Abstract; 'Selection strategy' and 'Visual screening' sections) in which 3.8% of 20,621 papers contained problematic figures with 'at least half exhibiting features suggestive of deliberate manipulation' — i.e. roughly 1.9%, which is consistent with the claim's 1-2% magnitude. However, the paper never states a 1-2% figure and its sample was not 'microbiology papers' but papers containing the search term 'Western blot' from 40 journals across microbiology and immunology, cancer biology, and general biology (39.5% from PLoS One alone), so the population qualifier is not supported. The claim's 'intentional fabrication' also overstates the source, which frames the finding as image duplication/manipulation and explicitly states that 'researcher intent could not be definitively determined in this study' (Classification of inappropriately duplicated images). (LLM, conf=0.75)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Moreover, the same analysis found a much stronger ' evidential value ' in the literature of all disciplines”
What the sources say
- Partly supportedcites [10] sourceExperimental — accuracy not yet measured
Our copy confirms the descriptive core: 'Pooling p-values across all disciplines, there was strong evidence for "evidential value"' (Assessing the Extent of P-Hacking... Using Text-Mining), with upper-bin (0.025 <= p < 0.05) proportions of 0.257 for Results and 0.262 for Abstracts, and significant evidential value in every discipline (Tables 1-2, Fig. 3). What the source does not support is the comparative 'much stronger': it never ranks the all-discipline literature above anything on evidential value, and Summary and Conclusions states 'both approaches yielded similar results' for evidential value. Moreover, within the paper the sexual-selection meta-analysis datasets gave a lower (i.e. stronger) upper-bin proportion of 0.202 than the all-discipline text-mining (Results/Abstracts passage; The Consequences of P-Hacking for Meta-analyses), so if the comparison is internal to this study the direction is reversed. (LLM, conf=0.72)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
44 more claims · 24 with their cited sources checked
Create a free account to see all 71 claims and to verify the 34 references still open.
- Probably not a substantial claim. 13
- It may state two separate claims as one. 11
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This observation has been followed by a number of additional reports, the summary of which is, in essence, that 77-90% of biomedical experiments are not reproducible [14-17].”
From Guidelines for the use of flow cytometry and cell sorting in immunological studies (third edition) (2021) · cites this paper as [17]
Experimental — accuracy not yet measuredIn the citing paper
“All of these issues amalgamate into a critical problem, which is not inconsequential and which substantially hinders the ability of science to self-correct [25, 26].”
From Media and social media attention to retracted articles according to Altmetric (2021) · cites this paper as [25]
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911
Self-citationThe Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388
Evaluating replicability of laboratory experiments in economics
Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016) · Science · doi:10.1126/science.aaf0918 · Cited by 1,314
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293
Self-citationNegative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Self-citation
Cited by
Data‐Driven Materials Science: Status, Challenges, and Perspectives
Himanen, L., Geurts, A., Foster, A. S., & Rinke, P. (2019) · Advanced Science · doi:10.1002/advs.201900808 · Cited by 808
The Space Physics Environment Data Analysis System (SPEDAS)
Angelopoulos, V., Cruce, P., Drozdov, A., Grimes, E. W., Hatzigeorgiu, N., King, D. A., Larson, D., Lewis, J. W., McTiernan, J. M., Roberts, D. A., et al. (2019) · Space Science Reviews · doi:10.1007/s11214-018-0576-4 · Cited by 540
“The latter, in fact, addresses a wider problem in all the sciences [Baker 2016]; this realization is slowly transforming our approaches to and expectations of published data and methods [Fanelli 2018].”
BackgroundGuidelines for the use of flow cytometry and cell sorting in immunological studies (third edition)
Cossarizza, A., Chang, H., Radbruch, A., Abrignani, S., Addo, R., Akdis, M., Andrä, I., Andreata, F., Annunziato, F., Arranz, E., et al. (2021) · European Journal of Immunology · doi:10.1002/eji.202170126 · Cited by 531
Automation in the Life Science Research Laboratory
Holland, I., & Davies, J. A. (2020) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2020.571777 · Cited by 200
“Reproducibility is a major concern for the research community both now (Begley and Ioannidis, 2015; Baker, 2016) and historically (reviewed by Fanelli, 2018), with associated economic implications (Freedman et al., 2015) and an undermining of public trust in science (Saltelli and Funtowicz, 2017).”
BackgroundA versatile and customizable low-cost 3D-printed open standard for microscopic imaging
Diederich, B., Lachmann, R., Carlstedt, S., Marsikova, B., Wang, H., Uwurukundo, X., Mosig, A. S., & Heintzmann, R. (2020) · Nature Communications · doi:10.1038/s41467-020-19447-9 · Cited by 164
Metadata
- Authors
- Fanelli, D.
- Affiliations
- Indiana University Bloomington (US)
- Corresponding author
- Fanelli, Daniele
- ORCID
- Fanelli, Daniele
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2018-03-12
- Volume
- 115
- Issue
- 11
- Pages
- 2628-2631
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC5856498
- OpenAlex
- W2790858109
- Semantic Scholar
- 5f61ca1180f833e96afebe4ac293b5ac79d13f11
- Cited by
- 416
- References
- 48
- Citations by year
- 2026: 26 · 2025: 41 · 2024: 39 · 2023: 56 · 2022: 51 · 2021: 71 · 2020: 57 · 2019: 54 · 2018: 20
- FWCI
- 71.56
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- London School of Economics and Political Science Research Online (London School of Economics and Political Science) (submittedVersion)
- London School of Economics and Political Science Theses Online (London School of Economics and Political Science) (submittedVersion)
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- replication crisis, reproducibility, research integrity, publication practices, research reliability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Is science really facing a reproducibility crisis, and do we need it to? https://app.cochat.ai/papers/10.1073/pnas.1708272114
- The paper
- Fanelli, D. (2018). Is science really facing a reproducibility crisis, and do we need it to? Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.1708272114
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.