Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
BACKGROUND: We explore whether the number of null results in large National Heart Lung, and Blood Institute (NHLBI) funded trials has increased over time. METHODS: We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease. Trials were included if direct costs >$500,000/year, participants were adult humans, and the primary outcome was cardiovascular risk, disease or death. The 55 trials meeting these criteria were coded for whether they were published prior to or after the year 2000, whether they registered in clinicaltrials.gov prior to publication, used active or placebo comparator, and whether or not the trial had industry co-sponsorship. We tabulated whether the study reported a positive, negative, or null result on the primary outcome variable and for total mortality. RESULTS: 17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 (χ2=12.2,df= 1, p=0.0005). There has been no change in the proportion of trials that compared treatment to placebo versus active comparator. Industry co-sponsorship was unrelated to the probability of reporting a significant benefit. Pre-registration in clinical trials.gov was strongly associated with the trend toward null findings. CONCLUSIONS: The number NHLBI trials reporting positive results declined after the year 2000. Prospective declaration of outcomes in RCTs, and the adoption of transparent reporting standards, as required by clinicaltrials.gov, may have contributed to the trend toward null findings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Registering a protocol before the start of the experiment enables researchers to demonstrate that the hypothesis, approach, and analysis were planned in advance and not shaped by data as they emerged; it enhances scientific rigour and protects the researcher against concerns about selective reporting of results [239,240].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [239]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Preregistration is required for all government-funded medical clinical trials in the United States (Food and Drug Administration, 2007; see also Kaplan & Irvin, 2015).”
From Do growth mindset interventions impact students’ academic achievement? A systematic review and meta-analysis with recommendations for best practices. (2023) · cites this paper as Kaplan & Irvin, 2015
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Effect estimates may vary according to whether or not trials are registered (Kaplan 2015).”
From Probiotics for treating acute infectious diarrhoea (2020) · cites this paper as Kaplan 2015
The cited source says what the claim attributes to it.
In the citing paper
“Registering a protocol before the start of the experiment enables researchers to demonstrate that the hypothesis, approach, and analysis were planned in advance and not shaped by data as they emerged; it enhances scientific rigour and protects the researcher against concerns about selective reporting of results [239,240].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [239]
In the citing paper
“The proportion of large, National Heart, Lung, and Blood Institute studies reporting null findings over time has increased, which is suggestive of decreases in publication bias (68).”
From Issues with data and analyses: Errors, underlying themes, and potential solutions (2018) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1×
“It is widely noted that journals favor publication of statistically significant findings[2].”
BackgroundMoher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545
Alzheimer’s disease drug-development pipeline: Few candidates, frequent failures
Cummings, J. L., Morstorf, T., & Zhong, K. (2014) · Alzheimer's Research & Therapy · doi:10.1186/alzrt269 · Cited by 1,744
The ClinicalTrials.gov Results Database — Update and Key Issues
Zarin, D. A., Tse, T., Williams, R. J., Califf, R. M., & Ide, N. C. (2011) · New England Journal of Medicine · doi:10.1056/nejmsa1012065 · Cited by 838 · cited 1×
“Registered studies must provide: the study’s purpose, recruitment status, design, eligibility criteria, locations and pre-specified primary and secondary outcomes[11].”
BackgroundTrial Registration at ClinicalTrials.gov between May and October 2005
Zarin, D. A., Tse, T., & Ide, N. C. (2005) · New England Journal of Medicine · doi:10.1056/nejmsa053234 · Cited by 291
Cardiovascular Drug Development
Fordyce, C. B., Roe, M. T., Ahmad, T., Libby, P., Borer, J. S., Hiatt, W. R., Bristow, M. R., Packer, M., Wasserman, S. M., Braunstein, N., et al. (2015) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2015.03.016 · Cited by 202 · cited 1×
“Since about 1998, there has been a systematic decline in the number approvals for new cardiovascular drugs[6].”
BackgroundMoving Toward Transparency of Clinical Trials
Zarin, D. A., & Tse, T. (2008) · Science · doi:10.1126/science.1153632 · Cited by 182
Publication of Trials Funded by the National Heart, Lung, and Blood Institute
Gordon, D., Taddei-Peters, W., Mascette, A., Antman, M., Kaufmann, P. G., & Lauer, M. S. (2013) · New England Journal of Medicine · doi:10.1056/nejmsa1300237 · Cited by 120
Fisher, J. A., Cottingham, M. D., & Kalbaugh, C. A. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2014.08.023 · Cited by 63 · cited 1×
“It has been argued that there have been few efficacious drugs in the pipeline[4,5].”
BackgroundRoth, J. A., Etzioni, R., Waters, T. M., Pettinger, M., Rossouw, J. E., Anderson, G. L., Chlebowski, R. T., Manson, J. E., Hlatky, M., Johnson, K. C., et al. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2348 · Cited by 58 · cited 1×
“The economic analysis estimated that there was about $140 returned for each dollar invested in the study[13].”
Background
Research works printed but not linked: 1
- Gordon D, Taddei-Peters W, Mascette A, Antman M, Kaufmann PG, Lauer MS. Publication of trials funded by the National Heart, Lung, and Blood Institute. N Engl J Med. 2013; 369: 1226– 1234. PMC3928673 24224625 · DOI · Paper page
Cited by
Items 201–300 of 360
McVay, M. A., Cooper, K. B., Seoane, M. C., Donahue, M. L., & Scherer, L. D. (2021) · Health Psychology and Behavioral Medicine · doi:10.1080/21642850.2021.1907186 · Cited by 7
Kowalska, M. E., Pot, S. A., & Hartnack, S. (2024) · Translational Vision Science & Technology · doi:10.1167/tvst.13.7.14 · Cited by 7
“99,100 In conclusion, a quantitative knowledge synthesis of the antimicrobial and tissue stabilizing effects of PACK-CXL, as in a meta-analysis or systematic review, is impossible despite the wide range of PACK-CXL protocol modifications that have been explored in in vitro/ex vivo and in vivo…”
A systematic review of pre-registration in autism research journals
Poole, D., Linden, A. H., Sedgewick, F., Allchin, O., & Hobson, H. (2024) · Autism · doi:10.1177/13623613241308312 · Cited by 8
“2 Following this requirement, a study of cardiovascular interventions found that the rate of positive results reported in studies prior to 2000 (none pre-registered) was 57% dropping to 8% after 2000 (all pre-registered; Kaplan & Irvin, 2015).”
InfluentialTrends of Publication of Negative Trials Over Time
Laviolle, B., Locher, C., Allain, J., Cornu, Q. L., Charpentier, P., Lefebvre, M., Pape, C. L., Leven, C., Palpacuer, C., Pontoizeau, C., et al. (2025) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.3535 · Cited by 8
Reducing Political Bias in Political Science Estimates
Zigerell, L. J. (2017) · PS Political Science & Politics · doi:10.1017/s1049096516002389 · Cited by 7
Open Science: Recommendations for Research on School Bullying
Noret, N. F., Hunter, S. C., Pimenta, S. M., Taylor, R., & Johnson, R. (2022) · International Journal of Bullying Prevention · doi:10.1007/s42380-022-00130-0 · Cited by 7
“In an analysis of randomised control trials examining the role of drugs or supplements for intervening in or treating cardiovascular disease, Kaplan and Irvin (2015) identified a substantial change in the number of significant effects reported once pre-registration was introduced (57% reported…”
Segerstrom, S. C., Diefenbach, M. A., Hamilton, K., O'Connor, D. B., & Tomiyama, A. J. (2023) · Health Psychology · doi:10.1037/hea0001236 · Cited by 9
Measuring transparency in the social sciences: political science and international relations
Scoggins, B., & Robertson, M. P. (2024) · Royal Society Open Science · doi:10.1098/rsos.240313 · Cited by 12
“When reforms are implemented mandating preregistration, by research bodies or formats like registered reports, the number of null results reported rises [44,45].”
BackgroundTrials and Tribulations: The Rise and Fall of the RCT in the WASH Sector
Spears, D., Ban, R., & Cumming, O. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0008 · Cited by 26
Spitzer, L., & Mueller, S. (2021) · PLoS ONE · doi:10.1371/journal.pone.0253950 · Cited by 7
“, see [17,18]), these reservations persist and some researchers remain skeptical.”
BackgroundBasic statistical considerations for physiology: The journal Temperature toolboxPreprint
Caldwell, A. R., & Cheuvront, S. N. (2020) · SportRxiv (OSF Preprints) · doi:10.31236/osf.io/rt4bp · Cited by 7
Preclinicaltrials.eu: prospective registration of animal studies
Menon, J. M. L., van der Naald, M., Chamuleau, S. A. J., & Duncker, D. J. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad623 · Cited by 8
Bertin, P., & Nera, K. (2025) · International Review of Social Psychology · doi:10.5334/irsp.979 · Cited by 7
Assessment of Evidential Value Requires More than a Single Data Point
Imhoff, R. (2020) · Archives of Sexual Behavior · doi:10.1007/s10508-020-01836-2 · Cited by 6
Synergy Between the Credibility Revolution and Human Development in Africa
Adetula, A., Forscher, P. S., Basnight-Brown, D. M., Azouaghe, S., Ouherrou, N., Charyate, A. C. A., Hansen, N., Adetula, G. A., & IJzerman, H. R. (2026) · Collabra Psychology · doi:10.1525/collabra.157529 · Cited by 6
Les pratiques de recherche ouvertes en psychologie
Lantian, A. (2020) · Psychologie Française · doi:10.1016/j.psfr.2020.09.001 · Cited by 6
Rubenson, D. L. (2021) · Politics and the Life Sciences · doi:10.1017/pls.2021.23 · Cited by 6
Introducing Preregistration of Research Design to ArchaeologyPreprint
Ross, S. A., & Ballsun-Stanton, B. (2021) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/sbwcq · Cited by 6
:null (2024) · Waxmann Verlag GmbH eBooks · doi:10.31244/9783830999126 · Cited by 5
Lochman, J. E. (2021) · Prevention Science · doi:10.1007/s11121-021-01219-6 · Cited by 6
“This approach seems to help, as recent analyses of pre-registered studies find a clear rise in publications of null findings and of replication studies (Cook et al., 2018; Kaplan & Irvin, 2015; Hillary & Medaglia, 2019; Allen & Mehler, 2019).”
ResultRoller, R. (2023) · Journal of Cultural Analytics · doi:10.22148/001c.57764 · Cited by 6
Segerstrom, S. C., Diefenbach, M. A., Hamilton, K., O'Connor, D. B., & Tomiyama, A. J. (2023) · Psychosomatic Medicine · doi:10.1097/psy.0000000000001186 · Cited by 5
10. Getting your head around the Universal Basic Income
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.10 · Cited by 6
Morduch, J. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0005 · Cited by 5
Priboi, C., Mayer, B., Vergauwe, E., Elger, B. S., & Würbel, H. (2026) · PLoS Biology · doi:10.1371/journal.pbio.3003511 · Cited by 4
Improving Replication in Endometrial Omics: Understanding the Influence of the Menstrual Cycle
Chung, J., & Rogers, P. A. W. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms26020857 · Cited by 4
“Preregistration has been empirically shown to increase the number of null results, with Kaplan and Irvin [71] observing that the number of positive results from 55 large RCTs decreased from 57% to 8% under the requirement for preregistration and transparency measures.”
Sheeran, P., Pei, Y., & Rothman, A. J. (2025) · Social Science & Medicine · doi:10.1016/j.socscimed.2025.118484 · Cited by 5
Garchitorena, A., Murray, M. B., Hedt‐Gauthier, B., Farmer, P. E., & Bonds, M. H. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0007 · Cited by 5
Qu, Y. L., Rappaport, B. I., Luby, J. L., & Barch, D. M. (2021) · Neuroimage Reports · doi:10.1016/j.ynirp.2021.100036 · Cited by 5
Autonomous Artificial Intelligence Safety and Trust
Abràmoff, M. D. (2021) · Springer eBooks · doi:10.1007/978-3-030-78601-4_4 · Cited by 4
RegCheck: A tool for structured comparisons between study registrations and papersPreprint
Cummins, J., Clarke, B., Hussey, I., & Elson, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.13330 · Cited by 5
“Crucially, registrations enable the reader to distinguish which analytic decisions were made before and after observing the data, which can have implications for the veracity and severity of the reported statistical tests (Kaplan & Irvin, 2015).”
Basarkod, G., Cahill, L. S., Burston, A., Barnett, D. W., Mahoney, J. W., Griffith, S., Swaryandini, G., Bradshaw, E. L., Devine, E. K., Akhtaruzzaman, M., et al. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02456-x · Cited by 5
Flowers, D. W., Swanson, B. T., Shaffer, S. M., Clewley, D. J., Martin, M. T., Russell, N. A., & Riley, S. P. (2025) · Journal of Manual & Manipulative Therapy · doi:10.1080/10669817.2025.2464548 · Cited by 3
“Kaplan and colleagues have shown us that heterogeneity decreases and null effects become normal when prospective registration is required and verified by the grant funder [16].”
Global research trends in the medical therapy of pulmonary arterial hypertension 2000–2014
Schreiber, C., Edlinger, C. R., Eder, S., Ausserwinkler, M., Wernly, B., Pretsch, I., Jung, C., Hoppe, U. C., & Lichtenauer, M. (2016) · Pulmonary Pharmacology & Therapeutics · doi:10.1016/j.pupt.2016.05.004 · Cited by 3
“However, published data for the treatment of cardiovascular disease reports a success rate of 8% for randomized controlled trials and therefore substantially differs from data on PAH treatment [42].”
BackgroundThe case for introducing pre-registered confirmatory pharmacological pre-clinical studies
Kiwanuka, O., Bellander, B., & Hånell, A. (2018) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1177/0271678x18760109 · Cited by 5
2. What we talk about when we talk about biology
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.02 · Cited by 3
Accelerating addiction research via Open Science and Team Science
Dora, J., Piccirillo, M. L., Foster, K. T., & King, K. M. (2023) · The Psychology of learning and motivation/The psychology of learning and motivation · doi:10.1016/bs.plm.2023.06.004 · Cited by 4
Registered clinical trials make positive findings vanish
Woolston, C. (2015) · Nature · doi:10.1038/524269f · Cited by 3
1. How my theory explains everything: and can make you happier, healthier, and wealthier
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.01 · Cited by 3
Cultural Problems Cannot Be Solved with Technical Solutions Alone
Lilburn, S. D., Little, D. R., Osth, A. F., & Smith, P. L. (2019) · Computational Brain & Behavior · doi:10.1007/s42113-019-00036-z · Cited by 3
“Its efficacy in some settings 99 is unquestionable: Kaplan and Irvin (2015) noted a 100 large change in the number of positive results reported 101 in randomized clinical trials from before and after the 102 year 2000 with the introduction of preregistration at 103 ClinicalTrials.”
BackgroundThe autonomous point-of-care diabetic retinopathy examination
Abràmoff, M. D. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-817485-2.00012-2 · Cited by 5
Wuttke, A. (2020) · Politics and the Life Sciences · doi:10.1017/pls.2020.13 · Cited by 3
“…opportunities for specification search and rewarding sound questions and methods independent of the study’s outcome affect the composition of published findings such as the share of hypothesis- refuting results (Kaplan & Irvin, 2015; Kvarven et al., 2019; Murray et al., 2020; Scheel et al., 2020).”
ResultHoebeke, Y., Desmedt, O., Özçimen, B., & Heeren, A. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/sdmfq · Cited by 3
O'Connor, D. B. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/bnvxe · Cited by 3
A FAIR Workflow Guide for Researchers in Human Cognitive NeurosciencePreprint
Zheng, Z., Chen, X., Brown, T., Cousijn, H., & Melloni, L. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/yhj5c · Cited by 3
Spitzer, L., Bošnjak, M., & Mueller, S. (2024) · Meta-Psychology · doi:10.15626/mp.2023.4039 · Cited by 2
“Studies suggest that preregistration can help reduce questionable research practices and the rate of false positive findings (Kaplan and Irvin, 2015; Swaen et al., 2001) and, among other open science techniques, might increase the replication rate of research (Protzko et al., 2020).”
Huamán‐Mendoza, A. A., do Amaral, G. C. L. S., Vilela, N., Chuquimez‐Ventura, C. V., Rovai, E. S., Romito, G. A., Pannuti, C. M., Villar, C. C., & Holzhausen, M. (2025) · Clinical Oral Implants Research · doi:10.1111/clr.14427 · Cited by 2
Riedel, A., Ruth, P., Loeffler, M. W., & Stransky, N. (2025) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1155/jcpt/3796111 · Cited by 2
Integrating Open Science Principles into Quasi-Experimental Social Science Research
Heller, B., & Robinson, C. D. (2025) · Social Sciences · doi:10.3390/socsci14080499 · Cited by 2
“…potential drawbacks of preregistration and preanalysis plans is that researchers are less likely to find statistically significant results through HARKing and p -hacking (Kaplan & Irvin, 2015), which may make it harder to publish a study and contribute to the file drawer problem (Rosenthal, 1979).”
Noninvasive Treatments for Acute, Subacute, and Chronic Low Back Pain
Le, L. H., & Badgett, R. G. (2017) · Annals of Internal Medicine · doi:10.7326/l17-0472 · Cited by 2
Open Science and Special Education Research
Cook, B. G., Beahm, L. A., Myers, A. M., VanUitert, V. J., & Wilson, S. E. (2021) · Emerald Group Publishing Limited eBooks · doi:10.1108/s0735-004x20210000031005 · Cited by 2
Preregistration in Animal Research
Heinl, C., Schönfelder, G., & Bert, B. (2022) · Springer eBooks · doi:10.1007/978-3-030-99680-2_37 · Cited by 2
Lundmark, S., Protzko, J., & Weissenbilder, M. (2022) · International Journal of Public Opinion Research · doi:10.1093/ijpor/edac040 · Cited by 2
Huamán‐Mendoza, A. A., Bommarito, R., Marcos Hideki, H. A. G. Y., Vilela, N., Romano, M. M., Braga, M. M., Pannuti, C. M., & Holzhausen, M. (2024) · Clinical Oral Investigations · doi:10.1007/s00784-024-05785-1 · Cited by 3
“This is because trials registered retrospectively are more prone to selective outcome reporting bias [26, 27], and unregistered studies are more inclined to say significant results than registered ones [28].”
9. The worst thing about poverty is not having enough money
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.09 · Cited by 2
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.07 · Cited by 2
Behavioral Medicine and the Benefits of Healthcare: A Critical Appraisal and the Need for Exnovation
Kaplan, R. M., Chambers, D., & Glasgow, R. E. (2018) · Springer eBooks · doi:10.1007/978-0-387-93826-4_36 · Cited by 3
Self-citationMortality Salience, Fears of Death, and Their Treatment in Anxiety-Related Disorders
Menzies, R. E. (2020) · The Sydney eScholarship Repository (The University of Sydney) · Cited by 3
Holmberg, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.13701 · Cited by 2
Psychology and Law, Meet Open Science
McAuliff, B. D., Fessinger, M. B., Perillo, A. D., & Perillo, J. T. (2023) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780197649138.013.5 · Cited by 1
A contemporary approach to research and statistics in psychology
Somoray, K. (2023) · James Cook University eBooks · doi:10.25120/6xg7-djxk · Cited by 1
Bauer, J., Bayer, S., Daumiller, M., Emslander, V., Fütterer, T., Hahn, S., Hahnel, C., Keller, S., Krammer, G., Kullmann, S., et al. (2026) · Pedocs (German Institute for International Educational Research) · doi:10.25656/01:34935 · Cited by 2
Study preregistration: an early example and analysis
Wiseman, R. J., & Watt, C. A. (2026) · PeerJ · doi:10.7717/peerj.21360 · Cited by 1
“In contrast, Kaplan & Irvin (2015) reported a drop in the effect size of studies evaluating treatments for cardiovascular disease once researchers were required to preregister key aspects of their studies.”
Bauer, J., Bayer, S., Daumiller, M., Emslander, V., Fütterer, T., Hahn, S., Hahnel, C., Keller, S., Krammer, G., Kullmann, S., et al. (2026) · Pedocs (German Institute for International Educational Research) · doi:10.25656/01:35615 · Cited by 1
Registered and Published Outcomes of Randomized Clinical Trials in Ophthalmology
Wang, M. T. M., Wolffsohn, J. S., & Craig, J. P. (2019) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2019.0311 · Cited by 1
Behavioral Political Science and Voter Turnout: Evidence from Field Experiments
Gerber, A. S., & Tucker, P. (2020) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190634131.013.14 · Cited by 1
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.08 · Cited by 1
Biobusiness in Brief:What Ails Clinical Trials?
Saberwal, G. (2018) · Current Science · doi:10.18520/cs/v115/i9/1648-1652 · Cited by 1
Ethische Analyse der klinischen Forschung mit humanen induzierten pluripotenten Stammzellen
Hansen, S. L., Heyder, C., & Wiesemann, C. (2020) · Veröffentlichungen des Instituts für Deutsches, Europäisches und Internationales Medizinrecht, Gesundheitsrecht und Bioethik der Universitäten Heidelberg und Mannheim · doi:10.1007/978-3-662-59052-2_7 · Cited by 1
Kolijn, L., van den Bulk, B. G., Euser, S., Bakermans‐Kranenburg, M. J., van IJzendoorn, M. H., & Huffmeijer, R. (2021) · Brain and Behavior · doi:10.1002/brb3.1972 · Cited by 2
Howard, J. (2018) · Springer eBooks · doi:10.1007/978-3-319-93224-8_30 · Cited by 1
Bausell, R. B. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780197536537.003.0002 · Cited by 1
Introducing Preregistration of Research Design to Archaeology
Ross, S. A., & Ballsun-Stanton, B. (2022) · University Press of Florida eBooks · doi:10.2307/j.ctv2mwg3mj.6 · Cited by 1
Preregistration and Transparency in the Research Process
Mellor, D. J. (2021) · American Educational Research Association Annual Meeting (AERA) · doi:10.3102/1679938 · Cited by 1
Noetel, M., Ciarrochi, J., Conigrave, J. H., & Lonsdale, C. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/8bk7m · Cited by 1
Pre-commitment and Updating BeliefsPreprint
Ebersole, C. R. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/5vsq3 · Cited by 2
Odoms‐Young, A. M., & Thorpe, R. J. (2019) · American Journal of Epidemiology · doi:10.1093/aje/kwz269 · Cited by 1
Souza, N. V. (2021) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.23.2021.tde-10112021-140837 · Cited by 1
Actions for Increasing the Credibility of Educational ResearchPreprint
Carvalho, P. F., Cook, B. G., de Leeuw, J. R., Fyfe, E. R., Gehlbach, H., Goldstone, R. L., Makel, M. C., Mellor, D. T., Montoya, A. K., Motz, B. A., et al. (2019) · EdArXiv (OSF Preprints) · doi:10.35542/osf.io/tv7rn · Cited by 2
Jafari, A., & Jenner, B. (2017) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/vgbw2 · Cited by 1
Starck, T., & Langevin, M. (2023) · medRxiv · doi:10.1101/2023.12.16.23300086 · Cited by 1
“This resulted in a large drop in randomized controlled trials reporting drugs benefit, from 60% to 10% [Kaplan and Irvin, 2015, Dickersin and Rennie, 2012].”
BackgroundReplicability and transparency in physical therapy research: Time to wake upPreprint
Jabouille, F., Farajzadeh, A., Exertier, A. G., Martine, J., Sadek, N., Klinger, L. H., Iliut, C. M., Kapsa, V., & Boisgontier, M. P. (2025) · medRxiv · doi:10.1101/2025.09.23.25336497 · Cited by 2
Griffith, S., Swaryandini, G., McKee, L., Oxnard, K. L., Cahill, L. S., Forbes, H., Rees, K., Davis, J., Sanders, T., Coleman, J. A., et al. (2026) · medRxiv · doi:10.64898/2026.04.27.26351794 · Cited by 1
Dee, L. E., Kimmel, K., Hayden, M. T., Kelleher, S., Ohlert, T., Merchant, T., Thuiller, W., & Schrodt, F. (2026) · EcoEvoRxiv · doi:10.32942/x2sd5s · Cited by 1
Gold Standard Science and Biomedical Research — Principles, Governance, and Scientific Independence
Fineberg, H. V. (2026) · New England Journal of Medicine · doi:10.1056/nejmms2607021 · Cited by 1
Your health: Analog or digital?
Ewing, G. W. (2016) · Digital Medicine · doi:10.4103/2226-8561.194690
Prävention muss effizienter werden
Porzsolt, F., Wambo, G. O. K., Rösch, M.-C., & Weiß, C. (2017) · Frauenheilkunde up2date · doi:10.1055/s-0043-115678
“Ulm: Universität Ulm; 2016 [10] Kaplan RM, Irvin VL. Likelihood of null effects of large NHLBI clinical trials has increased over time.”
BackgroundThe p-value Case, a Review of the Debate: Issues and Plausible Remedies
Pauli, F. (2018) · Springer proceedings in mathematics & statistics · doi:10.1007/978-3-319-73906-9_9
Arlinghaus, N., & Kekecs, Z. (2018) · Lund University Publications Student Papers (Lund University) · doi:10.13140/rg.2.2.34414.02882 · Cited by 1
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.11
6. The mill that grinds young people old
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.06
3. The cultural and the agentic
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.03 · Cited by 1
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.05
14. Morale is high (since I gave up hope)
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.14
12. Waking up and going out to work in the uncanny valley
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.12
From One Environment to Many: The Problem of Reproducibility of Experimental Results
Lin, J. (2018) · K-State Research Exchange (Kansas State University)
Transparency in Reporting Results From Randomized Clinical Trials
Kaplan, R. M., & Irvin, V. L. (2019) · Journal of Evidence-Based Healthcare · doi:10.17267/2675-021xevidence.v1i1.2254
Self-citationCognitive Interventions in Later Life: Introduction to the Special Section
Small, B. J. (2020) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbaa032
Using Priors in Experimental Design: How Much Are We Leaving on the Table?
Vivalt, E. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0013 · Cited by 1
Metadata
- Authors
- Kaplan, R. M., & Irvin, V. L.
- Affiliations
- Agency for Healthcare Research and Quality (US)
- Oregon State University (US)
- Corresponding author
- Kaplan, Robert M.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-08-05
- Volume
- 10
- Issue
- 8
- Pages
- e0132382
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4526697
- OpenAlex
- W1944584781
- Cited by
- 356
- References
- 11
- Citations by year
- 2026: 13 · 2025: 17 · 2024: 22 · 2023: 44 · 2022: 28 · 2021: 50 · 2020: 58 · 2019: 33 · 2018: 37 · 2017: 24 · 2016: 21 · 2015: 9
- FWCI
- 35.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS ONE (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Oregon State University
- National Institutes of Health
- Agency for Healthcare Research and Quality
- National Heart, Lung, and Blood Institute
- U.S. National Library of Medicine
- Office of Behavioral and Social Sciences Research
- Topics
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- negative results, clinical trial results, cardiovascular disease prevention, reporting guidelines, all-cause mortality, publication bias
- MeSH terms
- Cardiovascular Diseases, Humans, Publishing, Registries, Research Design, United States, Randomized Controlled Trials as Topic, Treatment Outcome, National Heart, Lung, and Blood Institute (U.S.)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.