Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
BACKGROUND: We explore whether the number of null results in large National Heart Lung, and Blood Institute (NHLBI) funded trials has increased over time. METHODS: We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease. Trials were included if direct costs >$500,000/year, participants were adult humans, and the primary outcome was cardiovascular risk, disease or death. The 55 trials meeting these criteria were coded for whether they were published prior to or after the year 2000, whether they registered in clinicaltrials.gov prior to publication, used active or placebo comparator, and whether or not the trial had industry co-sponsorship. We tabulated whether the study reported a positive, negative, or null result on the primary outcome variable and for total mortality. RESULTS: 17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 (χ2=12.2,df= 1, p=0.0005). There has been no change in the proportion of trials that compared treatment to placebo versus active comparator. Industry co-sponsorship was unrelated to the probability of reporting a significant benefit. Pre-registration in clinical trials.gov was strongly associated with the trend toward null findings. CONCLUSIONS: The number NHLBI trials reporting positive results declined after the year 2000. Prospective declaration of outcomes in RCTs, and the adoption of transparent reporting standards, as required by clinicaltrials.gov, may have contributed to the trend toward null findings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Registering a protocol before the start of the experiment enables researchers to demonstrate that the hypothesis, approach, and analysis were planned in advance and not shaped by data as they emerged; it enhances scientific rigour and protects the researcher against concerns about selective reporting of results [239,240].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [239]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Preregistration is required for all government-funded medical clinical trials in the United States (Food and Drug Administration, 2007; see also Kaplan & Irvin, 2015).”
From Do growth mindset interventions impact students’ academic achievement? A systematic review and meta-analysis with recommendations for best practices. (2023) · cites this paper as Kaplan & Irvin, 2015
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Effect estimates may vary according to whether or not trials are registered (Kaplan 2015).”
From Probiotics for treating acute infectious diarrhoea (2020) · cites this paper as Kaplan 2015
The cited source says what the claim attributes to it.
In the citing paper
“Registering a protocol before the start of the experiment enables researchers to demonstrate that the hypothesis, approach, and analysis were planned in advance and not shaped by data as they emerged; it enhances scientific rigour and protects the researcher against concerns about selective reporting of results [239,240].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [239]
In the citing paper
“The proportion of large, National Heart, Lung, and Blood Institute studies reporting null findings over time has increased, which is suggestive of decreases in publication bias (68).”
From Issues with data and analyses: Errors, underlying themes, and potential solutions (2018) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1×
“It is widely noted that journals favor publication of statistically significant findings[2].”
BackgroundMoher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545
Alzheimer’s disease drug-development pipeline: Few candidates, frequent failures
Cummings, J. L., Morstorf, T., & Zhong, K. (2014) · Alzheimer's Research & Therapy · doi:10.1186/alzrt269 · Cited by 1,744
The ClinicalTrials.gov Results Database — Update and Key Issues
Zarin, D. A., Tse, T., Williams, R. J., Califf, R. M., & Ide, N. C. (2011) · New England Journal of Medicine · doi:10.1056/nejmsa1012065 · Cited by 838 · cited 1×
“Registered studies must provide: the study’s purpose, recruitment status, design, eligibility criteria, locations and pre-specified primary and secondary outcomes[11].”
BackgroundTrial Registration at ClinicalTrials.gov between May and October 2005
Zarin, D. A., Tse, T., & Ide, N. C. (2005) · New England Journal of Medicine · doi:10.1056/nejmsa053234 · Cited by 291
Cardiovascular Drug Development
Fordyce, C. B., Roe, M. T., Ahmad, T., Libby, P., Borer, J. S., Hiatt, W. R., Bristow, M. R., Packer, M., Wasserman, S. M., Braunstein, N., et al. (2015) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2015.03.016 · Cited by 202 · cited 1×
“Since about 1998, there has been a systematic decline in the number approvals for new cardiovascular drugs[6].”
BackgroundMoving Toward Transparency of Clinical Trials
Zarin, D. A., & Tse, T. (2008) · Science · doi:10.1126/science.1153632 · Cited by 182
Publication of Trials Funded by the National Heart, Lung, and Blood Institute
Gordon, D., Taddei-Peters, W., Mascette, A., Antman, M., Kaufmann, P. G., & Lauer, M. S. (2013) · New England Journal of Medicine · doi:10.1056/nejmsa1300237 · Cited by 120
Fisher, J. A., Cottingham, M. D., & Kalbaugh, C. A. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2014.08.023 · Cited by 63 · cited 1×
“It has been argued that there have been few efficacious drugs in the pipeline[4,5].”
BackgroundRoth, J. A., Etzioni, R., Waters, T. M., Pettinger, M., Rossouw, J. E., Anderson, G. L., Chlebowski, R. T., Manson, J. E., Hlatky, M., Johnson, K. C., et al. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2348 · Cited by 58 · cited 1×
“The economic analysis estimated that there was about $140 returned for each dollar invested in the study[13].”
Background
Research works printed but not linked: 1
- D Gordon, 2013, Publication of trials funded by the National Heart, Lung, and Blood Institute, N Engl J Med, 369, 1226, 10.1056/nejmsa1300237 · DOI · Paper page
Cited by
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998
“For example, Kaplan and Irvin (43) observed a dramatic drop in the rate of positive results following the requirement to preregister primary outcomes in a sample of clinical trials.”
BackgroundDo “brain-training” programs work?
Simons, D. J., Boot, W. R., Charness, N., Gathercole, S. E., Chabris, C. F., Hambrick, D. Z., & Stine-Morrow, E. A. L. (2016) · Psychological Science in the Public Interest · doi:10.1177/1529100616661983 · Cited by 1,183
How Open Science Helps Researchers Succeed
McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., McDougall, D., Nosek, B. A., Ram, K., Soderberg, C. K., et al. (2016) · eLife · doi:10.7554/elife.16800 · Cited by 798
Video Improves Learning in Higher Education: A Systematic Review
Noetel, M., Griffith, S., Delaney, O., Sanders, T., Parker, P., del Pozo Cruz, B., & Lonsdale, C. (2021) · Review of Educational Research · doi:10.3102/0034654321990713 · Cited by 378
“For example, studies that prospectively registered their methods and primary outcome are much less likely to find significant results (Kaplan & Irvin, 2015).”
ResultReporting animal research: Explanation and Elaboration for the ARRIVE guidelines 2019
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/703355 · Cited by 559
“Registering a protocol before the start of the experiment enables researchers to demonstrate that the hypothesis, approach and analysis were planned in advance and not shaped by data as they emerged; it enhances scientific rigour and protects the researcher against concerns about selective reporting of results [222, 2…”
BackgroundOn doing better science: From thrill of discovery to policy implications
Antonakis, J. (2017) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.01.006 · Cited by 475
“As a prime example, registered trials wherein the research design is first reviewed prior to gathering data, substantially increase the prevalence of null results and provides a more accurate assessment of treatment effects (Kaplan & Irvin, 2015).”
BackgroundPre-registration in social psychology—A discussion and suggested template
van 't Veer, A. E., & Giner-Sorolla, R. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2016.03.004 · Cited by 439
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports
Scheel, A. M., Schijen, M. R. M. J., & Lakens, D. (2021) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459211007467 · Cited by 370
“However, preregistration does not prevent file-drawering or reviewer bias and may thus be insufficient to fight publication bias (Goldacre et al., 2016; Rasmussen et al., 2009; but see Kaplan & Irvin, 2015).”
BackgroundMacnamara, B. N., & Burgoyne, A. P. (2023) · Psychological Bulletin · doi:10.1037/bul0000352 · Cited by 415
“Following Detailed Preregistered Hypotheses, Design, and Analysis Plans Preregistration is required for all government-funded medical clinical trials in the United States (Food & Drug Administration, 2007; see also Kaplan & Irvin, 2015).”
BackgroundContextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398
“gov registry in 2000, which forced researchers to preregister their methods and outcome measures, the percentage of large heart-disease clinical trials reporting significant positive results plummeted from 57% to a mere 8% (7).”
MethodInvestigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
Winstein, C. J., Wolf, S. L., Dromerick, A. W., Lane, C. J., Nelsen, M. A., Lewthwaite, R., Cen, S. Y., Azen, S. P., & for the Interdisciplinary Comprehensive Arm Rehabilitation Evaluation (ICARE) Investigative Team (2016) · JAMA · doi:10.1001/jama.2016.0276 · Cited by 355
Causal mapping of human brain function
Siddiqi, S. H., Kording, K. P., Parvizi, J., & Fox, M. D. (2022) · Nature Reviews Neuroscience · doi:10.1038/s41583-022-00583-8 · Cited by 338
Challenges for assessing replicability in preclinical cancer biology
Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.67995 · Cited by 267
When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298
Bikson, M., Brunoni, A. R., Charvet, L. E., Clark, V. P., Cohen, L. G., Deng, Z.-D., Dmochowski, J., Edwards, D. J., Frohlich, F., Kappenman, E. S., et al. (2018) · Brain Stimulation · doi:10.1016/j.brs.2017.12.008 · Cited by 213
“This will enforce preregistration of study details which will encourage the publication of null results [190] and increase reproducibility.”
BackgroundThe Preregistration Revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017) · doi:10.31219/osf.io/2dxu5 · Cited by 168
Blind analysis: Hide results to seek the truth
MacCoun, R., & Perlmutter, S. (2015) · Nature · doi:10.1038/526187a · Cited by 218
Reducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E.-J. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 179
Discontinuation and Nonpublication of Randomized Clinical Trials Conducted in Children
Pica, N., & Bourgeois, F. (2016) · Pediatrics · doi:10.1542/peds.2016-0223 · Cited by 191
“evidence suggesting that it has positively impacted the reporting of clinical trials 23 involving human subjects.(10,11) 24 25”
BackgroundProtecting against researcher bias in secondary data analysis: challenges and potential solutions
Baldwin, J. R., Pingault, J.-B., Schoeler, T., Sallis, H. M., & Munafò, M. R. (2022) · European Journal of Epidemiology · doi:10.1007/s10654-021-00839-0 · Cited by 178
Threats of a replication crisis in empirical computer science
Cockburn, A., Dragicevic, P., Besançon, L., & Gutwin, C. (2020) · Communications of the ACM · doi:10.1145/3360311 · Cited by 155
“Results suggest that preregistration has had a substantial effect on scientific outcomes—for example, an analysis of studies funded by the National Heart, Lung, and Blood Institute between 1970 and 2012 showed the rate at which studies showed statistically significant findings plummeted from 57% before the introductio…”
BackgroundResultSmithers, L. G., Sawyer, A. C. P., Chittleborough, C. R., Davies, N. M., Davey Smith, G., & Lynch, J. W. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0461-x · Cited by 175
Issues with data and analyses: Errors, underlying themes, and potential solutions
Brown, A. W., Kaiser, K. A., & Allison, D. B. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708279115 · Cited by 157
Mapping the universe of registered reports
Hardwicke, T. E., & Ioannidis, J. P. A. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0444-y · Cited by 138
Probiotics for treating acute infectious diarrhoea
Collinson, S., Deans, A., Padua-Zamora, A., Gregorio, G. V., Li, C., Dans, L. F., & Allen, S. J. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003048.pub4 · Cited by 153
Strengthening the Practice of Exercise and Sport-Science Research
Halperin, I., Vigotsky, A. D., Foster, C., & Pyne, D. B. (2018) · International Journal of Sports Physiology and Performance · doi:10.1123/ijspp.2017-0322 · Cited by 130
“47 While this solution is not perfect and does not fi t all types of research questions, we believe that it is a model worth adopting in the fi eld of exercise and sport science.”
Cásedas, L., Pirruccio, V., Vadillo, M. A., & Lupiáñez, J. (2020) · Mindfulness · doi:10.1007/s12671-019-01279-4 · Cited by 145
Abràmoff, M. D., Tobey, D., & Char, D. S. (2020) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2020.02.022 · Cited by 140
“Without preregistration, autonomousAI performance tends to be overestimated, while successful study replication becomes less likely: in fact when comparing trials with and without preregistration, the trial effect sizes are larger when lacking preregistration.(43,47) Other considerations for correct validation related…”
MethodMenzies, R. E., Zuccala, M., Sharpe, L., & Dar-Nimrod, I. (2018) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2018.09.004 · Cited by 140
Ensuring the quality and specificity of preregistrations
Bakker, M., Veldkamp, C. L. S., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., & Wicherts, J. M. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000937 · Cited by 138
Open Science Practices are on the Rise: The State of Social Science (3S) SurveyPreprint
Christensen, G., Wang, Z., Paluck, E. L., Swanson, N., Birke, D. J., Miguel, E. A., & Littman, R. (2019) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/5rksu · Cited by 152
“…increase the visibility of results, improving meta-analysis and reducing the selective reporting of null, unexpected or otherwise unfavorable results (Kaplan and Irvin 2015, de Vries et al 2018), and data sharing could increase later data re-use and article citations (Piwowar and Vision 2013).”
ResultNonalcoholic fatty liver disease: a challenge from mechanisms to therapy.
Tarantino, G., Citro, V., & Capone, D. (2019) · Journal of Clinical Medicine · doi:10.3390/jcm9010015 · Cited by 134
“As collateral findings, a number of large National Heart Lung and Blood Institute-funded trials evaluating drugs or dietary supplements for the treatment or prevention of CV disease found the main complication of obesity-related NAFLD has increased over time, but those trials reporting significant results fell after t…”
Backgroundvan Loon, A. W. G., Creemers, H. E., Beumer, W. Y., Okorn, A., Vogelaar, S., Saab, N., Miers, A. C., Westenberg, P. M., & Asscher, J. J. (2020) · Journal of Youth and Adolescence · doi:10.1007/s10964-020-01201-5 · Cited by 128
Open and reproducible neuroimaging: From study inception to publication
Niso, G., Botvinik‐Nezer, R., Appelhoff, S., de la Vega, A. I., Estéban, O., Etzel, J. A., Finc, K., Ganz, M., Gau, R., Halchenko, Y. O., et al. (2022) · NeuroImage · doi:10.1016/j.neuroimage.2022.119623 · Cited by 126
HARK No More: On the Preregistration of CHI Experiments
Cockburn, A., Gutwin, C., & Dix, A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173715 · Cited by 95
“In particular, Kaplan and Irvin [25] examined all large studies (direct costs > $500,000 per year) funded by the National Heart, Lung, and Blood Institute (NHLIBI) between 1970 and 2012, giving 55 studies in total.”
BackgroundLangley, P. C. (2017) · INNOVATIONS in pharmacy · doi:10.24926/21550417.1346 · Cited by 107
Calibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98
“Other investigators have reported that mandatory registration of primary outcomes was associated with a substantial decline in the number of trials reporting statistically significant findings, perhaps due to registration effectively mitigating selective reporting (Kaplan & Irvin 2015).”
Ferrero, M., Vadillo, M. A., & León, S. P. (2021) · Intelligence · doi:10.1016/j.intell.2021.101566 · Cited by 117
Both Questionable and Open Research Practices Are Prevalent in Education Research
Makel, M. C., Hodges, J., Cook, B. G., & Plucker, J. A. (2021) · Educational Researcher · doi:10.3102/0013189x211001356 · Cited by 110
“This methodological self-reflection has led to changes such as requiring all clinical trials to preregister their goals (Kaplan & Irvin, 2015) and the creation of high-profile, open-access journals such as AERA Open and PLOS, as well as the proliferation of preprint and postprint servers that…”
Foundational Considerations for Artificial Intelligence Using Ophthalmic Images
Abràmoff, M. D., Cunningham, B., Patel, B., Eydelman, M. B., Leng, T., Sakamoto, T., Blodi, B., Grenon, S. M., Wolf, R. M., Manrai, A. K., et al. (2022) · Ophthalmology · doi:10.1016/j.ophtha.2021.08.023 · Cited by 96
Cybulski, L., Mayo-Wilson, E., & Grant, S. (2016) · Journal of Consulting and Clinical Psychology · doi:10.1037/ccp0000115 · Cited by 90
The benefits of preregistration and Registered Reports
Lakens, D., Mesquida, C., Rasti, S., & Ditroilo, M. (2024) · Evidence-Based Toxicology · doi:10.1080/2833373x.2024.2376046 · Cited by 104
da Rosa Orssatto, L. B., de la Rocha Freitas, C., Shield, A. J., Silveira Pinto, R., & Trajano, G. S. (2019) · Experimental Gerontology · doi:10.1016/j.exger.2019.110731 · Cited by 86
“The current requirement for prospective pre-registration of clinical trials has been associated with an increase in the number of clinical trials reporting null results (Kaplan and Irvin, 2015).”
BackgroundOpen Science in Education Sciences
van Dijk, W., Schatschneider, C. W., & Hart, S. A. (2020) · Journal of Learning Disabilities · doi:10.1177/0022219420945267 · Cited by 90
“As a result, the reported number of positive findings has decreased while the number of null results that were published increased (Kaplan & Irvin, 2015), expanding the knowledge of the true impact of new treatments.”
ResultEmpirical evidence of widespread exaggeration bias and selective reporting in ecology
Kimmel, K. A., Avolio, M. L., & Ferraro, P. J. (2023) · Nature Ecology & Evolution · doi:10.1038/s41559-023-02144-3 · Cited by 93
Mitigating Illusory Results through Preregistration in Education
Gehlbach, H., & Robinson, C. D. (2017) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2017.1387950 · Cited by 84
Null Effects and Publication Bias in Special Education Research
Cook, B. G., & Therrien, W. J. (2017) · Behavioral Disorders · doi:10.1177/0198742917709473 · Cited by 82
“The National Institutes of Health now requires preregistration of studies it funds, which has increased the likelihood of funded studies with null findings being published (Kaplan & Irvin, 2015).”
BackgroundStudy preregistration: An evaluation of a method for transparent reporting
Toth, A. A., Banks, G. C., Mellor, D., O’Boyle, E. H., Dickson, A., Davis, D. J., DeHaven, A., Bochantin, J., & Borns, J. (2021) · Journal of Business and Psychology · doi:10.1007/s10869-020-09695-3 · Cited by 73
Bandholm, T., Christensen, R., Thorborg, K., Treweek, S. P., & Henriksen, M. (2017) · British Journal of Sports Medicine · doi:10.1136/bjsports-2017-097527 · Cited by 72
The effect of coloured overlays and lenses on reading: a systematic review of the literature
Griffiths, P. G., Taylor, R. H., Henderson, L. M., & Barrett, B. T. (2016) · Ophthalmic and Physiological Optics · doi:10.1111/opo.12316 · Cited by 74
“This measure alone has been shown to reduce the number of trials for which a positive effect is reported(115).”
Preregistration in Single-Case Design Research
Johnson, A. H., & Cook, B. G. (2019) · Exceptional Children · doi:10.1177/0014402919868529 · Cited by 68
“Evidence suggests that preregistration reduces publication bias in group experimental research (Allen & Mehler, 2018; Kaplan & Irvin, 2015), and preregistration is becoming the norm for many funders and in some fields, with multiple options available for preregistering studies (Nosek et al., 2018).”
BackgroundRefining animal research: The Animal Study Registry
Bert, B., Heinl, C., Chmielewska, J., Schwarz, F., Grune, B., Hensel, A., Greiner, M., & Schönfelder, G. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000463 · Cited by 67
“In fact, in clinical research, in which the registration of clinical trials has been mandatory for now more than 10 years, preregistration effectively increased the publication of negative results [27].”
BackgroundChallenges in Building Usable Knowledge in Education
Hedges, L. V. (2018) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2017.1375583 · Cited by 63
“Bob Kaplan and Veronica Irvin published a paper comparing the proportion of significant results of National Heart, Lung, and Blood Institute trials in the years before and just after 2005 when registration became mandatory (see Kaplan & Irvin, 2015).”
BackgroundMaking ERP research more transparent: Guidelines for preregistration
Paul, M., Govaart, G., & Schettino, A. (2021) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2021.02.016 · Cited by 71
Ferraro, P. J., & Agrawal, A. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2106489118 · Cited by 72
Should the Randomistas (Continue to) Rule?
Ravallion, M. D. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0003 · Cited by 64
Open Communication Science: A Primer on Why and Some Recommendations for How
Lewis, N. A. (2020) · Communication Methods and Measures · doi:10.1080/19312458.2019.1685660 · Cited by 59
“After the year 2000 (after registration was required), that success rate plummeted to 8% (Kaplan & Irvin, 2015).”
BackgroundHéroux, M. E., Taylor, J. L., & Gandevia, S. C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0144151 · Cited by 66
Upreti, V. V., & Venkatakrishnan, K. (2019) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1462 · Cited by 61
“gov was identified in an analysis of the results of major clinical trials of cardiovascular outcomes as being associated with a trend toward null findings.(16) The Consolidated Standards Of Reporting Trials (CONSORT) provides a set of standard reporting recommendations for publications of randomized clinical trials, w…”
MethodThe Effect-Size Benchmark That Matters Most: Education Interventions Often Fail
Kraft, M. A. (2023) · Educational Researcher · doi:10.3102/0013189x231155154 · Cited by 60
“Scholars have found this pattern of smaller average effects among samples of preregistered studies and funded trials relative to published academic articles across multiple disciplines (DellaVigna & Linos, 2022; Kaplan & Irvin, 2015).”
BackgroundBias in science: natural and social
May, J. (2020) · Synthese · doi:10.1007/s11229-020-02937-0 · Cited by 66
Local Immigration and Support for Anti‐Immigration Parties: A Meta‐Analysis
Cools, S., Finseraas, H., & Røgeberg, O. J. (2021) · American Journal of Political Science · doi:10.1111/ajps.12613 · Cited by 74
Can the behavioral sciences self-correct? A social epistemic study
Romero, F. (2016) · Studies in History and Philosophy of Science Part A · doi:10.1016/j.shpsa.2016.10.002 · Cited by 57
Smith, M. M., Clarke, E., & Little, C. B. (2016) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2016.10.016 · Cited by 55
Outcome reporting bias in randomized-controlled trials investigating antipsychotic drugs
Lancee, M., Lemmens, C. M. C., Kahn, R. S., Vinkers, C. H., & Luykx, J. J. (2017) · Translational Psychiatry · doi:10.1038/tp.2017.203 · Cited by 51
Why is Getting Rid of P -Values So Hard? Musings on Science and Statistics
Goodman, S. N. (2019) · The American Statistician · doi:10.1080/00031305.2018.1558111 · Cited by 45
Randomized Control Trials in the Field of Development: A Critical Perspective
Bédécarrats, F., Guérin, I., & Roubaud, F. (2020) · doi:10.1093/oso/9780198865360.001.0001 · Cited by 88
Feature—Is a Replicability Crisis on the Horizon for Environmental and Resource Economics?
Ferraro, P. J., & Shukla, P. (2020) · Review of Environmental Economics and Policy · doi:10.1093/reep/reaa011 · Cited by 57
Toward Open and Reproducible Epidemiology
Mathur, M. B., & Fox, M. P. (2023) · American Journal of Epidemiology · doi:10.1093/aje/kwad007 · Cited by 56
“Although these requirements were minimal, their establishment was associated with a substantial decline in the proportion of published clinical trial results that supported the intervention, from 57% to 8% (60).”
BackgroundRegistered reports: an early example and analysis
Wiseman, R., Watt, C., & Kornbrot, D. (2019) · PeerJ · doi:10.7717/peerj.6232 · Cited by 47
Credibility crisis in agricultural economics
Ferraro, P. J., & Shukla, P. (2022) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13323 · Cited by 59
Orssatto, L. B. R., Bezerra, E. D. S., SHIELD, A. J., & Trajano, G. S. (2020) · Applied Physiology, Nutrition, and Metabolism · doi:10.1139/apnm-2020-0021 · Cited by 50
“…can increase research transparency by reporting all the methods, procedures, and outcomes before the intervention commencement, avoiding selective reporting of outcomes and manipulation of statistical analysis favouring significant effects (Zarin et al. 2011; Kaplan and Irvin 2015).”
BackgroundBest practices for data management and sharing in experimental biomedical research
Cunha‐Oliveira, T., Ioannidis, J. P. A., & Oliveira, P. J. (2024) · Physiological Reviews · doi:10.1152/physrev.00043.2023 · Cited by 57
Heirene, R. M., LaPlante, D. A., Louderback, E. R., Keen, B., Bakker, M., Serafimovska, A., & Gainsbury, S. (2024) · Meta-Psychology · doi:10.15626/mp.2021.2909 · Cited by 44
“Pre-registration of RCTs has aided in the detection of outcome switching (Chen et al., 2019; Vassar et al., 2020) and appears to have resulted in an increase in published null findings, indicating a potential reduction in the likelihood of statistical false-positives (Kaplan & Irvin, 2015).”
ResultThe effects of school‐based interventions on physiological stress in adolescents: A meta‐analysis
van Loon, A. W. G., Creemers, H. E., Okorn, A., Vogelaar, S., Miers, A. C., Saab, N., Westenberg, P. M., & Asscher, J. J. (2021) · Stress and Health · doi:10.1002/smi.3081 · Cited by 51
“Lastly, study characteristics might moderate the effectiveness, for instance publication year, since recent publications are more likely to report null ‐ results (Kaplan & Irvin, 2015).”
Nichol, B., McCready, J. L., Steen, M., Unsworth, J., Simonetti, V., & Tomietto, M. (2023) · PLoS ONE · doi:10.1371/journal.pone.0282525 · Cited by 46
Spitzer, L., & Mueller, S. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281086 · Cited by 47
“, see [17, 18] for examples from clinical trials and epidemiology).”
BackgroundBalachandran, A. T., Vigotsky, A. D., Quiles, N. N., Mokkink, L. B., Belio, M., & Glenn, J. M. (2020) · Experimental Gerontology · doi:10.1016/j.exger.2020.111202 · Cited by 48
A How-to Guide for Open-Science Practices in Special Education Research
Cook, B. G., Fleming, J. I., Hart, S. A., Lane, K. L., Therrien, W. J., van Dijk, W., & Wilson, S. E. (2021) · Remedial and Special Education · doi:10.1177/07419325211019100 · Cited by 45
Watt, C. A., & Kennedy, J. E. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.02030 · Cited by 42
“…most undetected methodological biases or “questionable research practices” that can distort study findings (Ioannidis, 2005; Simmons et al., 2011; John et al., 2012; Kaplan and Irvin, 2015; Franco et al., 2016) and they allow researchers to document verifiably that they used good methodology.”
BackgroundBasic statistical considerations for physiology: The journal Temperature toolbox
Caldwell, A. R., & Cheuvront, S. N. (2019) · Temperature · doi:10.1080/23328940.2019.1624131 · Cited by 40
“We strongly encourage authors to use these preregistration resources considering the current evidence suggests they improve replicability [90], reduce poor statistical reporting [92], and improve the ability of other researchers to detect possible problems or errors [93].”
BackgroundOpen Science and Research Reproducibility
Munafo, M. R. (2016) · ecancermedicalscience · doi:10.3332/ecancer.2016.ed56 · Cited by 37
A Call for Open Science in Giftedness Research
McBee, M. T., Makel, M. C., Peters, S. J., & Matthews, M. S. (2018) · Gifted Child Quarterly · doi:10.1177/0016986218784178 · Cited by 40
“For example, when clinical trials started requiring preregistration of hypotheses, statistically significant findings dropped from 57% to 8% of trials (Kaplan & Irvin, 2015).”
BackgroundReporting Guidelines for Artificial Intelligence in Medical Research
Campbell, J. P., Lee, A., Abràmoff, M. D., Keane, P. A., Ting, D. S. W., Lum, F., & Chiang, M. F. (2020) · Ophthalmology · doi:10.1016/j.ophtha.2020.09.009 · Cited by 37
“8,12 Furthermore, AI interventions often may encode the biases of their human creators.”
BackgroundSelective outcome reporting in randomized clinical trials of dental implants
Sendyk, D. I., Rovai, E. S., Souza, N. V., Deboni, M. C. Z., & Pannuti, C. M. (2019) · Journal of Clinical Periodontology · doi:10.1111/jcpe.13128 · Cited by 35
Wuttke, A. (2018) · Politische Vierteljahresschrift · doi:10.1007/s11615-018-0131-7 · Cited by 46
Improving the transparency and reliability of observational studies through registration
Naudet, F., Patel, C. J., DeVito, N., Goff, G. L., Cristea, I. A., Braillon, A., & Hoffmann, S. (2024) · BMJ · doi:10.1136/bmj-2023-076123 · Cited by 37
“Registeredclinicaltrialsaremorelikelytohave negative results than those that were unregistered, 21 although the difference varies across surveys.”
BackgroundEvans, T. R., Branney, P., Clements, A. J., & Hatton, E. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1969233 · Cited by 39
“For example, mandated preregistration of clinical trial outcomes contributed to a substantive increase in the likelihood of reporting null results (Kaplan and Irvin2015).”
BackgroundPromoting learning from null or negative results in prevention science trials
Axford, N., Berry, V., Lloyd, J., Hobbs, T., & Wyatt, K. (2022) · Prevention Science · doi:10.1007/s11121-020-01140-4 · Cited by 34
Ethics of RCTs: Should Economists Care about Equipoise?
Abramowicz, M., & Szafarz, A. (2019) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0012 · Cited by 56
Open Minded: Searching for Truth about the Unconscious Mind
Newell, B. R., & Shanks, D. R. (2023) · doi:10.7551/mitpress/14922.001.0001 · Cited by 17
Political ideology and trust in scientists in the USA
Gligorić, V., Kleef, G. A. V., & Rutjens, B. T. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02147-z · Cited by 38
“This aligns with findings from other areas of scientific research (for example, large medical trials) showing that the rate of significant trials decreases after trials had to be registered 39 .”
TARG Meta-Research Group & Collaborators (2023) · BMJ Open · doi:10.1136/bmjopen-2023-076264 · Cited by 33
An analysis of the quality of experimental design and reliability of results in tribology research
Watson, M., Christoforou, P., Herrera, P., Preece, D., Carrell, J., Harmon, M., Krier, P., Lewis, S. E., Maiti, R., Skipper, W. A., et al. (2019) · Wear · doi:10.1016/j.wear.2018.12.028 · Cited by 34
“[7] found that this reduced the proportion of positive results in one heart disease journal from 57% to 8%.”
BackgroundAppraising reconsolidation theory and its empirical validation
Schroyens, N., Beckers, T., & Luyten, L. (2022) · Psychonomic Bulletin & Review · doi:10.3758/s13423-022-02173-2 · Cited by 38
“…shown that the use of preregistration and Registered Reports goes hand in hand with a decline in the percentage of published studies reporting statistically significant (“positive”) results, possibly by decreasing the use of suboptimal research practices (Kaplan & Irvin, 2015; Scheel et al., 2021).”
BackgroundEgo Depletion May Disappear by 2020
Vadillo, M. A. (2019) · Social Psychology · doi:10.1027/1864-9335/a000375 · Cited by 33
A guide to open science practices for animal research
Diederich, K., Schmitt, K. D., Schwedhelm, P., Bert, B., & Heinl, C. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001810 · Cited by 35
“published, and the implementation of trial registration has strongly reduced the number of publications reporting significant treatment effects [46].”
BackgroundMeta-Analysis of the Effect of Red on Perceived Attractiveness
Lehmann, G. K., Elliot, A. J., & Calin-Jageman, R. J. (2018) · Evolutionary Psychology · doi:10.1177/1474704918802412 · Cited by 34
“Preregistration is thought to help limit research flexibility that can bias effect size estimates (e.g., Kaplan & Irvin, 2015).”
Background
Metadata
- Authors
- Kaplan, R. M., & Irvin, V. L.
- Affiliations
- Agency for Healthcare Research and Quality (US)
- Oregon State University (US)
- Corresponding author
- Kaplan, Robert M.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-08-05
- Volume
- 10
- Issue
- 8
- Pages
- e0132382
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4526697
- OpenAlex
- W1944584781
- Cited by
- 356
- References
- 11
- Citations by year
- 2026: 13 · 2025: 17 · 2024: 22 · 2023: 44 · 2022: 28 · 2021: 50 · 2020: 58 · 2019: 33 · 2018: 37 · 2017: 24 · 2016: 21 · 2015: 9
- FWCI
- 35.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS ONE (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Oregon State University
- National Institutes of Health
- Agency for Healthcare Research and Quality
- National Heart, Lung, and Blood Institute
- U.S. National Library of Medicine
- Office of Behavioral and Social Sciences Research
- Topics
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- negative results, clinical trial results, cardiovascular disease prevention, reporting guidelines, all-cause mortality, publication bias
- MeSH terms
- Cardiovascular Diseases, Humans, Publishing, Registries, Research Design, United States, Randomized Controlled Trials as Topic, Treatment Outcome, National Heart, Lung, and Blood Institute (U.S.)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.