How to Make More Published Research True
Open accessThe DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
Abstract
Currently, many published research findings are false or exaggerated, and an estimated 85% of research resources are wasted.N To make more published research true, practices that have improved credibility and efficiency in specific fields may be transplanted to others which would benefit from them-possibilities include the adoption of large-scale collaborative research; replication culture; registration; sharing; reproducibility practices; better statistical methods; standardization of definitions and analyses; more appropriate (usually more stringent) statistical thresholds; and improvement in study design standards, peer review, reporting and dissemination of research, and training of the scientific workforce. N Selection of interventions to improve research practices requires rigorous examination and experimental testing whenever feasible.N Optimal interventions need to understand and harness the motives of various stakeholders who operate in scientific research and who differ on the extent to which they are interested in promoting publishable, fundable, translatable, or profitable results.N Modifications need to be made in the reward system for science, affecting the exchange rates for currencies (e.g., publications and grants) and purchased academic goods (e.g., promotion and other academic or administrative power) and introducing currencies that are better aligned with translatable and reproducible research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Accordingly, there is growing recognition that data availability is a critical feature of an efficient, progressive, and ultimately self-correcting scientific ecosystem that generates credible findings [1–3].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some journals have heeded the growing calls for adoption of transparent and reproducible research practices and introduced new policies to implement change [1,2].”
From Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition (2018) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, protocol registration is applicable to and recommended for all studies and is supported by various platforms, such as the Open Science Framework [3, 29, 35–37].”
From Assessing transparency practices in dental randomized controlled trials (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Well-planned, well-executed, and well-reported RCTs, which embrace transparent and open science practices, enhance the likelihood that results are accurate, reliable, and applicable [2, 3].”
From Assessing transparency practices in dental randomized controlled trials (2024) · cites this paper as [3]
Experimental — accuracy not yet measuredIn the citing paper
“Consider that the stakeholders who fund and support science may have incentives that are not always aligned with those of active researchers [82].”
From The natural selection of bad science (2016) · cites this paper as [82]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationConfirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267 · cited 2×
“powerful in scientific processes [8,9].”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 2×
“Many new proposed associations and/or effects are false or grossly exaggerated [2,3], and translation of knowledge into useful applications is often slow and potentially inefficient [4].”
BackgroundSelf-citationClinical Practice Guidelines We Can Trust
Committee on Standards for Developing Trustworthy Clinical Practice Guidelines, Board on Health Care Services, & Institute of Medicine (2011) · National Academies Press eBooks · doi:10.17226/13058 · Cited by 3,081
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“Pharmaceutical industry teams have championed reproducibility in preclinical research [56,57] because they depend on pre-clinical academic investigations accurately pinpointing useful drug targets.”
BackgroundBelieve it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“Pharmaceutical industry teams have championed reproducibility in preclinical research [56,57] because they depend on pre-clinical academic investigations accurately pinpointing useful drug targets.”
BackgroundEvidence based medicine: a movement in crisis?
Greenhalgh, T., Howick, J., Maskrey, N., & for the Evidence Based Medicine Renaissance Group (2014) · BMJ · doi:10.1136/bmj.g3725 · Cited by 1,766
Scientific method: Statistical errors
Nuzzo, R. (2014) · Nature · doi:10.1038/506150a · Cited by 1,536
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495 · cited 1×
“Many new proposed associations and/or effects are false or grossly exaggerated [2,3], and translation of knowledge into useful applications is often slow and potentially inefficient [4].”
BackgroundSelf-citation
Research works printed but not linked: 1
- R Al-Shahi Salman, 2014, Increasing value and reducing waste in biomedical research regulation and management, Lancet, 383, 176, 10.1016/s0140-6736(13)62297-7 · DOI · Paper page
Cited by
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
“Equally, these proposals are not an exhaustive list; there are many other nascent and maturing ideas for making research practices more efficient and reliable 73 .”
Self-citationLefaucheur, J.-P., Aleman, A., Baeken, C., Benninger, D. H., Brunelin, J., Di Lazzaro, V., Filipović, S. R., Grefkes, C., Hasan, A., Hummel, F. C., et al. (2020) · Clinical Neurophysiology · doi:10.1016/j.clinph.2019.11.002 · Cited by 2,382
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Self-citationGagnier, J. J., Lai, J., Mokkink, L. B., & Terwee, C. B. (2021) · Quality of Life Research · doi:10.1007/s11136-021-02822-4 · Cited by 881
The natural selection of bad science
Smaldino, P. E., & McElreath, R. (2016) · Royal Society Open Science · doi:10.1098/rsos.160384 · Cited by 860
“ampbell, 1976; Wilson, Hayes, Biglan & Embry, 2014). We are hardly the first to consider a need to alter the incentives for career success in science (Nosek & Lakens, 2014; Nosek et al., 2012; Ioannidis, 2014a; Vankov et al., 2014; Fischer, Ritchie & Hanspach, 2012; Brembs, Button & Munafo, 2013; Begley…”
BackgroundInfluentialThe gender gap in science: How long until women are equally represented?
Holman, L., Stuart-Fox, D., & Hauser, C. E. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2004956 · Cited by 769
Seyhan, A. A. (2019) · Translational Medicine Communications · doi:10.1186/s41231-019-0050-7 · Cited by 713
“[99] and there has to be clinical impact or utility.”
BackgroundReplicability, Robustness, and Reproducibility in Psychological Science
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., et al. (2022) · Annual Review of Psychology · doi:10.1146/annurev-psych-020821-114157 · Cited by 770
Szucs, D., & Ioannidis, J. P. A. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2000797 · Cited by 761
Self-citationWhy Most Clinical Research Is Not Useful
Ioannidis, J. P. A. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1002049 · Cited by 655
“Joint efforts by multiple stakeholders may yield solutions that are more likely to be more widely adopted and thus successful [3].”
BackgroundSelf-citationThe Power of Bias in Economics Research
Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017) · The Economic Journal · doi:10.1111/ecoj.12461 · Cited by 634
“Many options are available to move forward, as summarised by Ioannidis (2014) and Maniadis et al. (2015).”
BackgroundSelf-citationThe Lancet Psychiatry Commission on psychological treatments research in tomorrow's science
Holmes, E. A., Ghaderi, A., Harmer, C. J., Ramchandani, P. G., Cuijpers, P., Morrison, A. P., Roiser, J. P., Bockting, C. L. H., O'Connor, R. C., Shafran, R., et al. (2018) · The Lancet Psychiatry · doi:10.1016/s2215-0366(17)30513-8 · Cited by 624
Vollmer, S., Mateen, B. A., Bohner, G., Király, F. J., Ghani, R., Jonsson, P., Cumbers, S., Jonas, A., McAllister, K. S. L., Myles, P., et al. (2020) · BMJ · doi:10.1136/bmj.l6927 · Cited by 497
“Reproducibility of research has become a growing concern across many scientific domains, 53 54 and in the ML/AI field, access to the underlying code and raw data are central to preventing and mitigating reproducibility concerns.”
BackgroundSelf-citationRadiomics strategies for risk assessment of tumour failure in head-and-neck cancer
Vallières, M., Kay-Rivest, E., Perrin, L. J., Liem, X., Furstoss, C., Aerts, H. J. W. L., Khaouam, N., Nguyen-Tan, P. F., Wang, C.-S., Sultanem, K., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-10371-5 · Cited by 542
Top 10 Replicated Findings From Behavioral Genetics
Plomin, R., DeFries, J. C., Knopik, V. S., & Neiderhiser, J. M. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691615617439 · Cited by 517
“Dozens of papers have considered general reasons for false positive publications in science (Ioannidis, 2005b, 2014) including psychological science (Pashler & Wagenmakers, 2012).”
On doing better science: From thrill of discovery to policy implications
Antonakis, J. (2017) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.01.006 · Cited by 475
“001 (Johnson, Payne, Wang, Asher, & Mandal, in press), which may help lower false positive findings (Ioannidis, 2014).”
BackgroundInfluentialMeta-research: Evaluation and Improvement of Research Methods and Practices
Ioannidis, J. P. A., Fanelli, D., Dunne, D. D., & Goodman, S. N. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002264 · Cited by 495
Self-citationCharacterization of PET/CT images using texture analysis: the past, the present… any future?
Hatt, M., Tixier, F., Pierce, L., Kinahan, P. E., Le Rest, C. C., & Visvikis, D. (2017) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-016-3427-0 · Cited by 475
Increasing value and reducing waste in biomedical research: who's listening?
Moher, D., Glasziou, P., Chalmers, I., Nasser, M., Bossuyt, P. M. M., Korevaar, D. A., Graham, I. D., Ravaud, P., & Boutron, I. (2016) · The Lancet · doi:10.1016/s0140-6736(15)00307-4 · Cited by 462
Assessing scientists for hiring, promotion, and tenure
Moher, D., Naudet, F., Cristea, I. A., Miedema, F., Ioannidis, J. P. A., & Goodman, S. N. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2004089 · Cited by 462
Self-citationHydrogen Bonds: Simple after All?
Herschlag, D., & Pinney, M. M. (2018) · Biochemistry · doi:10.1021/acs.biochem.8b00217 · Cited by 382
Mirowski, P. (2018) · Social Studies of Science · doi:10.1177/0306312718772086 · Cited by 372
“The situation is exacerbated when statistical inference is involved, given the tendency of the analyst to search for significant positive results, and to suppress the much more common insignificant or negative results (Ioannidis, 2005, 2014; Lehrer, 2010; Mirowski, 2004a: ch. 10).”
Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., et al. (2018) · Royal Society Open Science · doi:10.1098/rsos.180448 · Cited by 336
The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research
Amrhein, V., Korner-Nievergelt, F., & Roth, T. (2017) · PeerJ · doi:10.7717/peerj.3544 · Cited by 308
Reproducible Research Practices and Transparency across the Biomedical Literature
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002333 · Cited by 310
Self-citationLoth, E., Charman, T., Mason, L., Tillmann, J., Jones, E. J. H., Wooldridge, C., Ahmad, J., Auyeung, B., Brogna, C., Ambrosino, S., et al. (2017) · Molecular Autism · doi:10.1186/s13229-017-0146-8 · Cited by 299
“This requires collaborative research using a standardised protocol and analysis plan, and stringent statistical approaches [13].”
When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298
“Diverse stakeholders (journals, funders, institutions, and more) may contribute to align incentives with better research practices (Ioannidis 2014).”
BackgroundMethodInfluentialSelf-citationWallach, J. D., Boyack, K. W., & Ioannidis, J. P. A. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2006930 · Cited by 269
Self-citationWhat is wrong with evidence based policy, and how can it be improved?
Saltelli, A., & Giampietro, M. (2017) · Futures · doi:10.1016/j.futures.2016.11.012 · Cited by 264
Statistical Rituals: The Replication Delusion and How We Got There
Gigerenzer, G. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918771329 · Cited by 263
“One drastic proposal to revamp the current system is that only publications whose findings are replicated, not publication per se, should count in the future and that large grants should count not positively but negatively, unless the recipient delivers proportionally high-quality science (Ioannidis, 2014).”
BackgroundInfluentialVelupillai, S., Suominen, H., Liakata, M., Roberts, A., Shah, A. D., Morley, K., Osborn, D., Hayes, J., Stewart, R., Downs, J., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.10.005 · Cited by 247
“For clinical NLP method development to become more integral in clinical outcomes research, there is a need to develop evaluation workbenches that can be used by clinicians to better understand the underlying parts of an NLP system and its impact on outcomes.”
Choosing experiments to accelerate collective discovery
Rzhetsky, A., Foster, J. G., Foster, I. T., & Evans, J. A. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1509757112 · Cited by 249
Contribution of NIH funding to new drug approvals 2010–2016
Galkina Cleary, E., Beierlein, J. M., Khanuja, N. S., McNamee, L. M., & Ledley, F. D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1715368115 · Cited by 248
Vinkers, C. H., Tijdink, J. K., & Otte, W. M. (2015) · BMJ · doi:10.1136/bmj.h6467 · Cited by 237
Modelling science trustworthiness under publish or perish pressure
Grimes, D. R., Bauch, C. T., & Ioannidis, J. P. A. (2018) · Royal Society Open Science · doi:10.1098/rsos.171511 · Cited by 225
“To counteract this, it has been suggested that rewarding diligence might counteract this trend [31,32].”
BackgroundSelf-citationBouter, L. M., Tijdink, J., Axelsen, N., Martinson, B. C., & ter Riet, G. (2016) · Research Integrity and Peer Review · doi:10.1186/s41073-016-0024-5 · Cited by 211
Agile science: creating useful products for behavior change in the real world
Hekler, E. B., Klasnja, P., Riley, W. T., Buman, M. P., Huberty, J., Rivera, D. E., & Martin, C. A. (2016) · Translational Behavioral Medicine · doi:10.1007/s13142-016-0395-7 · Cited by 208
What is science’s crisis really about?
Saltelli, A., & Funtowicz, S. (2017) · Futures · doi:10.1016/j.futures.2017.05.010 · Cited by 187
“Evidence of this crisis can be found in the existence of a series of world conferences on research integrity,1 in critical editorials of prestigious journals such as Lancet (2015) or Nature (2015) and in numerous reflective articles (Begley & Ioannidis 2015; Horton, 2015; Ioannidis, 2005, 2014).”
BackgroundMethodInfluentialGil, Y., David, C. H., Demir, I., Essawy, B. T., Fulweiler, R. W., Goodall, J. L., Karlstrom, L., Lee, H., Mills, H. J., Oh, J., et al. (2016) · Earth and Space Science · doi:10.1002/2015ea000136 · Cited by 180
“Advocates also propose broader changes such as adopting collaborative research practices, creating a replication culture, and training the scientific workforce [Ioannidis 2014].”
BackgroundPsychology’s Replication Crisis and the Grant Culture: Righting the Ship
Lilienfeld, S. O. (2017) · Perspectives on Psychological Science · doi:10.1177/1745691616687745 · Cited by 187
“…from institutional incentives for prioritizing scholarly quality and replicabil ity over financial success to more radical proposals, such as penalizing scholars who have a lengthy track record of grant funding without a commensurate record of high quality published research (Ioannidis, 2014).”
BackgroundScientific Utopia III: Crowdsourcing Science
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., McCarthy, R. J., Riegelman, A., Silberzahn, R., & Nosek, B. A. (2019) · Perspectives on Psychological Science · doi:10.1177/1745691619850561 · Cited by 174
The oxidative stress theory of disease: levels of evidence and epistemological aspects
Ghezzi, P., Jaquet, V., Marcucci, F., & Schmidt, H. H. H. W. (2017) · British Journal of Pharmacology · doi:10.1111/bph.13544 · Cited by 180
“This is not specific for studies on OS, and studies have shown that a very large proportion of scientific BJP P Ghezzi et al. 1792 British Journal of Pharmacology (2017) 174 1784–1796 studies are not reproducible (Ioannidis, 2014).”
ResultStandardizing end points in perioperative trials: towards a core and extended outcome set
Myles, P. S., Grocott, M. P. W., Boney, O., & Moonesinghe, S. R. (2016) · British Journal of Anaesthesia · doi:10.1093/bja/aew066 · Cited by 178
ENIGMA and the individual: Predicting factors that affect the brain in 35 countries worldwide
Thompson, P. M., Andreassen, O. A., Arias-Vasquez, A., Bearden, C. E., Boedhoe, P. S., Brouwer, R. M., Buckner, R. L., Buitelaar, J. K., Bulayeva, K. B., Cannon, D. M., et al. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2015.11.057 · Cited by 181
Ioannidis, J. P. A. (2015) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2014.11.008 · Cited by 165
Self-citationPoor performance of clinical prediction models: the harm of commonly applied methods
Steyerberg, E. W., Uno, H., Ioannidis, J. P. A., van Calster, B., Ukaegbu, C., Dhingra, T., Syngal, S., & Kastrinos, F. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.013 · Cited by 168
Self-citationIntegrative analysis of exogenous, endogenous, tumour and immune factors for precision medicine
Ogino, S., Nowak, J. A., Hamada, T., Phipps, A. I., Peters, U., Milner Jr, D. A., Giovannucci, E. L., Nishihara, R., Giannakis, M., Garrett, W. S., et al. (2018) · Gut · doi:10.1136/gutjnl-2017-315537 · Cited by 169
Johnson, S. C., Cavallaro, F. L., & Leon, D. A. (2017) · Social Science & Medicine · doi:10.1016/j.socscimed.2017.09.025 · Cited by 180
“Ionnidis and colleagues have 281 suggested that 85% of research resources are wasted, in part due to the lack of 282 standardized definitions and reproducibility in research.(44) Developing a standardized, valid 283 method for operationalizing AL in population studies is critical in order to ensure findings 284 from f…”
BackgroundNavigating the garden of forking paths for data exclusions in fear conditioning research
Lonsdorf, T. B., Klingelhöfer-Jens, M., Andreatta, M., Beckers, T., Chalkia, A., Gerlicher, A., Jentsch, V. L., Meir Drexler, S., Mertens, G., Richter, J., et al. (2019) · eLife · doi:10.7554/elife.52465 · Cited by 164
“…and Ioannidis, 2018), publicly available data including those from excluded participants and pre-registration (Munafò et al., 2017) of definitions and analysis pipelines (Ioannidis, 2014), as well as openly accessible lab-specific standard operational protocols (SOPs), may also be helpful.”
BackgroundMultiple Citation Indicators and Their Composite across Scientific Disciplines
Ioannidis, J. P. A., Klavans, R., & Boyack, K. W. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002501 · Cited by 186
Self-citationTermenón, M., Jaillard, A. S., Delon‐Martin, C., & Achard, S. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.05.062 · Cited by 164
“Indeed, the poor reproducibility of scientific works reported by statisticians (Button et al., 2013; Ioannidis, 2014) would be, at least for a part, Please cite this article as: Termenon, M., et al., Reliability of graph anal Connectome Project, NeuroImage (2016),…”
BackgroundLarson, M. J., & Carbine, K. A. (2016) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2016.06.015 · Cited by 162
Prognostic role of tumor PIK3CA mutation in colorectal cancer: a systematic review and meta-analysis
Mei, Z., Duan, C., Li, C. B., Cui, L., & Ogino, S. (2016) · Annals of Oncology · doi:10.1093/annonc/mdw264 · Cited by 165
Editorial: preclinical data reproducibility for R&D - the challenge for neuroscience
Steckler, T. (2015) · SpringerPlus · doi:10.1186/2193-1801-4-1 · Cited by 140
Interventions for orthodontically induced white spot lesions: a systematic review and meta-analysis
Höchli, D., Hersberger-Zurfluh, M., Papageorgiou, S. N., & Eliades, T. (2016) · The European Journal of Orthodontics · doi:10.1093/ejo/cjw065 · Cited by 136
“Finally, given the complex character of the research question and the multiple treatment groups, outcomes, time points, and the effect clustering can have on a trial’s conlusions, orthodontic researchers and journal editors are encouraged to support both the registration and the provision of widely publicly available…”
BackgroundHow to survive the medical misinformation mess
Ioannidis, J. P. A., Stuart, M. E., Brownlee, S., & Strite, S. A. (2017) · European Journal of Clinical Investigation · doi:10.1111/eci.12834 · Cited by 133
Self-citationAssessment of transparency indicators across the biomedical literature: How open is open?
Serghiou, S., Contopoulos-Ioannidis, D. G., Boyack, K. W., Riedel, N., Wallach, J. D., & Ioannidis, J. P. A. (2021) · PLOS Biology · doi:10.1371/journal.pbio.3001107 · Cited by 139
Self-citationRamirez, F. D., Motazedian, P., Jung, R. G., Di Santo, P., MacDonald, Z. D., Moreland, R., Simard, T., Clancy, A. A., Russo, J. J., Welch, V. A., et al. (2017) · Circulation Research · doi:10.1161/circresaha.117.310628 · Cited by 138
Averitt, A. J., Weng, C., Ryan, P., & Perotte, A. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0277-8 · Cited by 137
Pearce, T., Maple, M. J., Shakeshaft, A. P., Wayland, S., & McKay, K. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17072229 · Cited by 145
“Fourth, despite the methodological benefits of well-controlled evaluation designs, such as Randomised Controlled Trials (RCTs), and their perceived desirability among researchers and academic journal editors, typically the evidence they generate has high internal validity and low external validity (generalisation), me…”
BackgroundQuestioning the Ecological Footprint
Galli, A., Giampietro, M., Goldfinger, S., Lazarus, E., Lin, D. H., Saltelli, A., Wackernagel, M., & Müller, F. (2016) · Ecological Indicators · doi:10.1016/j.ecolind.2016.04.014 · Cited by 148
Millan, M. J., Goodwin, G. M., Meyer-Lindenberg, A., & Ove Ögren, S. (2015) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2015.01.016 · Cited by 132
Brain effective connectivity during motor-imagery and execution following stroke and rehabilitation
Bajaj, S., Butler, A. J., Drake, D., & Dhamala, M. (2015) · NeuroImage: Clinical · doi:10.1016/j.nicl.2015.06.006 · Cited by 124
Exposure-wide epidemiology: revisiting Bradford Hill: Bradford Hill criteria revisited
Ioannidis, J. P. A. (2016) · Statistics in Medicine · doi:10.1002/sim.6825 · Cited by 122
“I think that we need more empirical meta-research studies [39,40] to continue to monitor evidence on these pivotal issues, because things may change in the future.”
BackgroundSelf-citationThe Harm Done to Reproducibility by the Culture of Null Hypothesis Significance Testing
Lash, T. L. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kwx261 · Cited by 134
The search for human pheromones: The lost decades and the necessity of returning to first principles
Wyatt, T. D. (2015) · Proceedings of the Royal Society B: Biological Sciences · doi:10.1098/rspb.2014.2994 · Cited by 123
Reproducible research and GIScience: an evaluation using AGILE conference papers
Nüst, D., Granell, C., Hofer, B., Konkol, M., Ostermann, F., Šileryté, R., & Cerutti, V. (2018) · PeerJ · doi:10.7717/peerj.5072 · Cited by 137
Coyne, M. D., Cook, B. G., & Therrien, W. J. (2016) · Remedial and Special Education · doi:10.1177/0741932516648463 · Cited by 128
“To counter this perception, a number of reforms may help to promote a replication culture (Ioannidis, 2014) in special education, such as (a) journals adopting policies to encourage submission and publication of replication studies regardless of whether they reproduce the findings of the original…”
BackgroundInfluentialBustin, S., & Nolan, T. (2017) · European Journal of Clinical Investigation · doi:10.1111/eci.12801 · Cited by 113
Garattini, S., Jakobsen, J. C., Wetterslev, J., Bertelé, V., Banzi, R., Rath, A., Neugebauer, E. A. M., Laville, M., Masson, Y., Hivert, V., et al. (2016) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2016.03.020 · Cited by 108
Designing a rigorous microscopy experiment: Validating methods and avoiding bias
Jost, A. P.-T., & Waters, J. C. (2019) · The Journal of Cell Biology · doi:10.1083/jcb.201812109 · Cited by 116
The importance of small samples in medical research
Indrayan, A., & Mishra, A. (2021) · Journal of Postgraduate Medicine · doi:10.4103/jpgm.jpgm_230_21 · Cited by 122
“The case for small n Concerns such as truthful research [44] and the effect of aleatory and epistemic uncertainties on the results [45] do not necessarily require a large sample.”
The Molecular Phenotype of Kidney Transplants: Insights From the MMDx Project
Halloran, P. F., Madill-Thomsen, K. S., & REEVE, J. P. (2023) · Transplantation · doi:10.1097/tp.0000000000004624 · Cited by 114
Millard, L. A. C., Davies, N. M., Timpson, N. J., Tilling, K., Flach, P. A., & Smith, G. D. (2015) · Scientific Reports · doi:10.1038/srep16645 · Cited by 96
Archival Data in Micro-Organizational Research: A Toolkit for Moving to a Broader Set of Topics
Barnes, C. M., Dang, C. T., Leavitt, K., Guarana, C. L. O., & Uhlmann, E. L. (2015) · Journal of Management · doi:10.1177/0149206315604188 · Cited by 112
“However, experiments typically suffer from low statistical power (Ioannidis, 2014) and imperfect generalizability to real organizational situations.”
BackgroundRath, A., Salamon, V., Peixoto, S., Hivert, V., Laville, M., Segrestin, B., Neugebauer, E. A. M., Eikermann, M., Bertelè, V., Garattini, S. K., et al. (2017) · Trials · doi:10.1186/s13063-017-2287-7 · Cited by 111
Hendricks, P. S., Crawford, M. S., Cropsey, K. L., Copes, H., Sweat, N. W., Walsh, Z., & Pavela, G. (2018) · Journal of Psychopharmacology · doi:10.1177/0269881117735685 · Cited by 93
Swindell, W. R., Remmer, H. A., Sarkar, M. K., Xing, X., Barnes, D. H., Wolterink, L., Voorhees, J. J., Nair, R. P., Johnston, A. W. B., Elder, J. T., et al. (2015) · Genome Medicine · doi:10.1186/s13073-015-0208-5 · Cited by 110
Predicting replication outcomes in the Many Labs 2 study
Forsell, E., Viganola, D., Pfeiffer, T., Almenberg, J., Wilson, B., Chen, Y., Nosek, B. A., Johannesson, M., & Dreber, A. (2019) · Journal of Economic Psychology · doi:10.1016/j.joep.2018.10.009 · Cited by 104
Styles, B., & Torgerson, C. J. (2018) · Educational Research · doi:10.1080/00131881.2018.1500194 · Cited by 108
“Aspects of research cited by Ioannidis (2014) as critical, which UK education RCTs have already embraced include large-scale collaborative research, trial registration, standardisation of definitions and analyses, and improvement in study design standards.”
BackgroundCalibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
“Pooling expertise, financial resources, and other resources in collaborative work may improve statistical power, reproducibility, access to unique populations, and results generalizability (Ioannidis 2014, Munafò et al. 2017).”
InfluentialSelf-citationCan we save large carnivores without losing large carnivore science?
Allen, B. L., Allen, L. R., Andrén, H., Ballard, G., Boitani, L., Engeman, R. M., Fleming, P. J. S., Ford, A. T., Haswell, P. M., Kowalczyk, R., et al. (2017) · Food Webs · doi:10.1016/j.fooweb.2017.02.008 · Cited by 112
Specific barriers to the conduct of randomised clinical trials on medical devices
Neugebauer, E. A. M., Rath, A., Antoine, S., Eikermann, M., Seidel, D., Koenen, C., Jacobs, E. M. G., Pieper, D., Laville, M., Pitel, S., et al. (2017) · Trials · doi:10.1186/s13063-017-2168-0 · Cited by 103
Laraway, S., Snycerski, S., Pradhan, S., & Huitema, B. E. (2019) · Perspectives on Behavior Science · doi:10.1007/s40614-019-00193-3 · Cited by 103
“Begley and Ioannidis (2014) stated that the failures of reproducibility are “a consequence of a system that is willing to overlook and ignore lack of scientific rigor and instead reward flashy results that generate scientific buzz or excitement” (p. 118).”
BackgroundInfluentialHaller, G., Bampoe, S., Cook, T. M., Fleisher, L. A., Grocott, M. P. W., Neuman, M. D., Story, D., Myles, P. S., Myles, P. S., Grocott, M. P. W., et al. (2019) · British Journal of Anaesthesia · doi:10.1016/j.bja.2019.04.041 · Cited by 98
Preregistration Is Neither Sufficient nor Necessary for Good Science
Pham, M. T., & Oh, T. T. (2020) · Journal of Consumer Psychology · doi:10.1002/jcpy.1209 · Cited by 104
Franz, F. P., Galito, N., de Toledo, A. C., Thomaz, T. G., Moraes, C., Guerra, T. D. R. B., Leão, M., Migowski, A., da Silva, A. R. A., & Weiß, C. (2015) · Pragmatic and Observational Research · doi:10.2147/por.s89946 · Cited by 91
Bishop, D. V. M., & Thompson, P. A. (2016) · PeerJ · doi:10.7717/peerj.1715 · Cited by 90
“The causes of this situation are complex and the solutions are likely to require changes, both in training of scientists in methods and revision of the incentive structure of science (Ioannidis 2014; Academy of Medical Sciences et al. 2015).”
BackgroundMaking Health Research Matter: A Call to Increase Attention to External Validity
Huebschmann, A. G., Leavitt, I. M., & Glasgow, R. E. (2019) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040218-043945 · Cited by 76
“This article reviews and highlights what others have clearly documented over the last two decades: the absence of reporting on key external validity factors that are necessary both to inform adoption by health system and public health decision makers as well as to allow scientists to replicate each other’s work (14, 2…”
BackgroundResultInfluentialMadrid, J., & Benninger, D. H. (2020) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2020.108957 · Cited by 104
“Therefore, we put strong emphasis on the replication of similar studies across independent laboratories – in qualitative analysis – to improve the reliability of research outcomes (Ioannidis, 2014).”
BackgroundKnight, G. M., Dharan, N. J., Fox, G. J., Stennis, N., Zwerling, A. A., Khurana, R., & Dowdy, D. W. (2015) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2015.10.024 · Cited by 89
Kochunov, P., Hong, L. E., Dennis, E. L., Morey, R. A., Tate, D. F., Wilde, E. A., Logue, M., Kelly, S., Donohoe, G., Favre, P., et al. (2022) · Human Brain Mapping · doi:10.1002/hbm.24998 · Cited by 87
“Meta-analysis has always offered a principled approach to screen studies for false positive findings by overcoming the “chasing of significance” observed in some discovery studies (Ioannidis, 2014).”
BackgroundResearch Portfolio Analysis in Science Policy: Moving from Financial Returns to Societal Benefits
Wallace, M. L., & Ràfols, I. (2015) · Minerva · doi:10.1007/s11024-015-9271-8 · Cited by 99
Tamagusko, T., Correia, M. G., & Ferreira, A. (2024) · Infrastructures · doi:10.3390/infrastructures9120213 · Cited by 99
Lamberink, H. J., Otte, W. M., Geleijns, K. P. W., & Braun, K. P. J. (2015) · Epileptic Disorders · doi:10.1684/epd.2015.0764 · Cited by 86
Empirical evaluation of which trial characteristics are associated with treatment effect estimates
Dechartres, A., Trinquart, L., Faber, T., & Ravaud, P. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.04.005 · Cited by 85
Kosumi, K., Hamada, T., Koh, H., Borowsky, J., Bullman, S., Twombly, T. S., Nevo, D., Masugi, Y., Liu, L., da Silva, A., et al. (2018) · American Journal Of Pathology · doi:10.1016/j.ajpath.2018.08.015 · Cited by 91
Ferraris, G., Monzani, D., Coppini, V., Conti, L., Pizzoli, S. F. M., Grasso, R., & Pravettoni, G. (2023) · Digital Health · doi:10.1177/20552076231210663 · Cited by 96
Data management and sharing in neuroimaging: Practices and perceptions of MRI researchers
Borghi, J., & Gulick, A. E. V. (2018) · PLoS ONE · doi:10.1371/journal.pone.0200562 · Cited by 91
Low replicability can support robust and efficient science
Lewandowsky, S., & Oberauer, K. (2020) · Nature Communications · doi:10.1038/s41467-019-14203-0 · Cited by 79
Metadata
- Authors
- Ioannidis, J. P. A.
- Affiliations
- Stanford Medicine (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-10-21
- Volume
- 11
- Issue
- 10
- Pages
- e1001747
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC4204808
- OpenAlex
- W2077181076
- Cited by
- 760
- References
- 75
- Citations by year
- 2026: 26 · 2025: 31 · 2024: 37 · 2023: 50 · 2022: 51 · 2021: 44 · 2020: 51 · 2019: 68 · 2018: 93 · 2017: 108 · 2016: 111 · 2015: 80 · 2014: 10
- FWCI
- 192.35
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Laura and John Arnold Foundation
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- research integrity, reproducibility, research credibility, research registration, statistical thresholds, peer review, research reporting, research incentives, publication incentives, research waste
- MeSH terms
- Publications, Publishing, Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.