Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
BACKGROUND: We explore whether the number of null results in large National Heart Lung, and Blood Institute (NHLBI) funded trials has increased over time. METHODS: We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease. Trials were included if direct costs >$500,000/year, participants were adult humans, and the primary outcome was cardiovascular risk, disease or death. The 55 trials meeting these criteria were coded for whether they were published prior to or after the year 2000, whether they registered in clinicaltrials.gov prior to publication, used active or placebo comparator, and whether or not the trial had industry co-sponsorship. We tabulated whether the study reported a positive, negative, or null result on the primary outcome variable and for total mortality. RESULTS: 17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 (χ2=12.2,df= 1, p=0.0005). There has been no change in the proportion of trials that compared treatment to placebo versus active comparator. Industry co-sponsorship was unrelated to the probability of reporting a significant benefit. Pre-registration in clinical trials.gov was strongly associated with the trend toward null findings. CONCLUSIONS: The number NHLBI trials reporting positive results declined after the year 2000. Prospective declaration of outcomes in RCTs, and the adoption of transparent reporting standards, as required by clinicaltrials.gov, may have contributed to the trend toward null findings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Registering a protocol before the start of the experiment enables researchers to demonstrate that the hypothesis, approach, and analysis were planned in advance and not shaped by data as they emerged; it enhances scientific rigour and protects the researcher against concerns about selective reporting of results [239,240].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [239]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Preregistration is required for all government-funded medical clinical trials in the United States (Food and Drug Administration, 2007; see also Kaplan & Irvin, 2015).”
From Do growth mindset interventions impact students’ academic achievement? A systematic review and meta-analysis with recommendations for best practices. (2023) · cites this paper as Kaplan & Irvin, 2015
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Effect estimates may vary according to whether or not trials are registered (Kaplan 2015).”
From Probiotics for treating acute infectious diarrhoea (2020) · cites this paper as Kaplan 2015
The cited source says what the claim attributes to it.
In the citing paper
“Registering a protocol before the start of the experiment enables researchers to demonstrate that the hypothesis, approach, and analysis were planned in advance and not shaped by data as they emerged; it enhances scientific rigour and protects the researcher against concerns about selective reporting of results [239,240].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [239]
In the citing paper
“The proportion of large, National Heart, Lung, and Blood Institute studies reporting null findings over time has increased, which is suggestive of decreases in publication bias (68).”
From Issues with data and analyses: Errors, underlying themes, and potential solutions (2018) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1×
“It is widely noted that journals favor publication of statistically significant findings[2].”
BackgroundMoher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545
Alzheimer’s disease drug-development pipeline: Few candidates, frequent failures
Cummings, J. L., Morstorf, T., & Zhong, K. (2014) · Alzheimer's Research & Therapy · doi:10.1186/alzrt269 · Cited by 1,744
The ClinicalTrials.gov Results Database — Update and Key Issues
Zarin, D. A., Tse, T., Williams, R. J., Califf, R. M., & Ide, N. C. (2011) · New England Journal of Medicine · doi:10.1056/nejmsa1012065 · Cited by 838 · cited 1×
“Registered studies must provide: the study’s purpose, recruitment status, design, eligibility criteria, locations and pre-specified primary and secondary outcomes[11].”
BackgroundTrial Registration at ClinicalTrials.gov between May and October 2005
Zarin, D. A., Tse, T., & Ide, N. C. (2005) · New England Journal of Medicine · doi:10.1056/nejmsa053234 · Cited by 291
Cardiovascular Drug Development
Fordyce, C. B., Roe, M. T., Ahmad, T., Libby, P., Borer, J. S., Hiatt, W. R., Bristow, M. R., Packer, M., Wasserman, S. M., Braunstein, N., et al. (2015) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2015.03.016 · Cited by 202 · cited 1×
“Since about 1998, there has been a systematic decline in the number approvals for new cardiovascular drugs[6].”
BackgroundMoving Toward Transparency of Clinical Trials
Zarin, D. A., & Tse, T. (2008) · Science · doi:10.1126/science.1153632 · Cited by 182
Publication of Trials Funded by the National Heart, Lung, and Blood Institute
Gordon, D., Taddei-Peters, W., Mascette, A., Antman, M., Kaufmann, P. G., & Lauer, M. S. (2013) · New England Journal of Medicine · doi:10.1056/nejmsa1300237 · Cited by 120
Fisher, J. A., Cottingham, M. D., & Kalbaugh, C. A. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2014.08.023 · Cited by 63 · cited 1×
“It has been argued that there have been few efficacious drugs in the pipeline[4,5].”
BackgroundRoth, J. A., Etzioni, R., Waters, T. M., Pettinger, M., Rossouw, J. E., Anderson, G. L., Chlebowski, R. T., Manson, J. E., Hlatky, M., Johnson, K. C., et al. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2348 · Cited by 58 · cited 1×
“The economic analysis estimated that there was about $140 returned for each dollar invested in the study[13].”
Background
Research works printed but not linked: 1
- Gordon D, Taddei-Peters W, Mascette A, Antman M, Kaufmann PG, Lauer MS. Publication of trials funded by the National Heart, Lung, and Blood Institute. N Engl J Med. 2013; 369: 1226– 1234. PMC3928673 24224625 · DOI · Paper page
Cited by
Items 101–200 of 360
A Future for Observational Epidemiology: Clarity, Credibility, Transparency
Harper, S. B. (2019) · American Journal of Epidemiology · doi:10.1093/aje/kwy280 · Cited by 33
Prevention of selective outcome reporting: let us start from the beginning
Dal‐Ré, R., & Marušić, A. (2016) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-016-2112-3 · Cited by 34
Pre-Registration in Social Psychology—a Discussion and Suggested TemplatePreprint
Veer, A. V. '., & Giner‐Sorolla, R. (2016) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/4frms · Cited by 32
Randomizing Development: Method or Madness?
Pritchett, L. H. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0004 · Cited by 53
Mapping the Universe of Registered ReportsPreprint
Hardwicke, T. E., & Ioannidis, J. P. A. (2018) · MetArXiv (OSF Preprints) · doi:10.31222/osf.io/fzpcy · Cited by 31
RCTs in Development Economics, Their Critics and Their Evolution
Ogden, T. N. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0006 · Cited by 34
“Kaplan and Irvin (2015) study the results of medical trials using RCTs and find that the number of “no-effect” results increased markedly when researchers had to file a pre-analysis plan documenting exactly how they would assess the data gathered before the experiment was conducted.”
BackgroundBasu, A. P., Pearse, J. E., & Rapley, T. (2017) · Trials · doi:10.1186/s13063-016-1757-7 · Cited by 28
“Trial preregistration, that requires declaration of the primary outcome measure in advance, has probably contributed to a recent increase in reported null effects in clinical trials [25] and could also help reduce duplication of research efforts even without a full published protocol.”
BackgroundThe Research Reproducibility Crisis and Economics of Science
Maniadis, Z., & Tufano, F. (2017) · The Economic Journal · doi:10.1111/ecoj.12526 · Cited by 32
Preregistration in infant research—A primer
Havron, N., Bergmann, C., & Tsuji, S. (2020) · Infancy · doi:10.1111/infa.12353 · Cited by 29
Research integrity requires to be aware of good and questionable research practices
Boisgontier, M. P. (2022) · European Rehabilitation Journal · doi:10.52057/erj.v2i1.24 · Cited by 28
“Moreover, the effect of registration on QRPs is illustrated in a study on 55 randomized clinical trials published before or after the year 2000 [10], when prospective registration in clinicaltrials.gov became required in medicine.”
Ellis, R. J. (2022) · eNeuro · doi:10.1523/eneuro.0017-22.2022 · Cited by 30
“…improvements on primary outcomes, compared with 8% of studies published after 2000, when preregistration of primary outcomes became mandatory for clinical trials after Congress passed the Food and Drug Administration Modernization Act in 1997 (Kaplan and Irvin, 2015; ClinicalTrials.gov, 2021).”
ResultNew developments in research methods
Ledgerwood, A. (2019) · eScholarship (California Digital Library) · Cited by 28
Hoebeke, Y., Desmedt, O., Özçimen, B., & Heeren, A. (2021) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2021.152226 · Cited by 28
On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics
Eble, A., Boone, P., & Elbourne, D. R. (2016) · World Bank, Washington, DC eBooks · doi:10.1596/1813-9450-7746 · Cited by 29
Brænd, A. M., Straand, J., Jakobsen, R. B., & Klovning, A. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010535 · Cited by 25
P ‐hacking in meta‐analyses: A formalization and new meta‐analytic methods
Mathur, M. B. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1701 · Cited by 32
“A minority of these studies (51) were preregistered, meaning that some form of protocol was created and time-stamped prior to data collection, which may limit SWS. 51,52 Of the preregistered studies, 5 were affirmative and 46 were nonaffirmative.”
BackgroundMachine learning for psychiatry: getting doctors at the black box?
Hedderich, D. M., & Eickhoff, S. B. (2020) · Molecular Psychiatry · doi:10.1038/s41380-020-00931-z · Cited by 28
Refinement on the Way Towards Replacement: Are We Doing What We Can?
Herrmann, K. (2019) · BRILL eBooks · doi:10.1163/9789004391192_002 · Cited by 39
Walters, C. G., Harter, Z. J., Wayant, C., Vo, N. H., Warren, M., Chronister, J., Tritz, D., & Vassar, M. B. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-033962 · Cited by 24
Open Science and Single-Case Design Research
Cook, B. G., Johnson, A. H., Maggin, D. M., Therrien, W. J., Barton, E. E., Lloyd, J. W., Reichow, B., Talbott, E., & Travers, J. C. (2021) · Remedial and Special Education · doi:10.1177/0741932521996452 · Cited by 23
“Perhaps by making questionable research practice discoverable, preregistration is associated with smaller— and potentially more credible—study effects (Kaplan & Irvin, 2015; Schäfer & Schwarz, 2019).”
BackgroundSelective outcome reporting in root coverage randomized clinical trials
Sendyk, D. I., Vilela, N., Neto, J. B. C., Tatakis, D. N., & Pannuti, C. M. (2021) · Journal Of Clinical Periodontology · doi:10.1111/jcpe.13451 · Cited by 24
Introduction: Randomization in the Tropics Revisited, a Theme and Eleven Variations
Deaton, A. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0002 · Cited by 29
Editors’ Introduction: Controversies around RCT in Development
Bédécarrats, F., Guérin, I., & Roubaud, F. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0001 · Cited by 22
The Rhetorical Superiority of Poor Economics
Labrousse, A. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0010 · Cited by 21
Is Red Really Romantic? Two Pre-Registered Replications of the Red-Romance Hypothesis
Lehmann, G. K., & Calin-Jageman, R. J. (2017) · Social Psychology · doi:10.1027/1864-9335/a000296 · Cited by 22
“Pre-registered replications limit flexibility in data analysis and may help reduce bias in estimating effect sizes (Kaplan & Irvin, 2015).”
BackgroundHanging on to the Edges: Essays on Science, Society and the Academic Life
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155 · Cited by 24
Tsuji, S., Cristià, A., Frank, M. C., & Bergmann, C. (2020) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000393 · Cited by 21
Phylotocol: Promoting Transparency and Overcoming Bias in Phylogenetics
DeBiasse, M. B., & Ryan, J. F. (2018) · Systematic Biology · doi:10.1093/sysbio/syy090 · Cited by 22
“gov registry (USFDA 1997) led to the widespread adoption of transparent reporting standards in clinical trials, a dramatic drop in the frequency of positive results was observed, suggesting that bias may have been inflating the number positive results (Kaplan and Irvin 2015).”
BackgroundEntertaining without endorsing: The case for the scientific investigation of anomalous cognition.
Schooler, J. W., Baumgart, S., & Franklin, M. S. (2018) · Psychology of Consciousness Theory Research and Practice · doi:10.1037/cns0000151 · Cited by 23
“…in a host of conventional domains including psychology (Open Science Collaboration, 2015), medicine (Ioannidis, 2005; Ioannidis & Trikalinos, 2005; Kaplan & Irvin, 2015) and biology (e.g., Jennions & Møller, 2002) are less robust than originally assumed (Schooler, 2011; Protzko & Schooler, in…”
BackgroundAssessment of Consistency Between Peer-Reviewed Publications and Clinical Trial Registries
Sun, L. W., Lee, D. J., Collins, J. A., Carll, T. C., Ramahi, K., Sandy, S. J., Unteriner, J., & Weinberg, D. V. (2019) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2019.0312 · Cited by 21
Caballero, D. C., Castillo, C. A., Ballesteros‐Yáñez, I., & Juárez, L. M. (2019) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-019-0161-2 · Cited by 20
Optimizing the scientific study of suicide with open and transparent research practices
Carpenter, T. P., & Law, K. C. (2021) · Suicide and Life-Threatening Behavior · doi:10.1111/sltb.12665 · Cited by 24
Are the “Randomistas” Evaluators?
Picciotto, R. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0011 · Cited by 21
Accounting research and the significance test crisis
Johnstone, D. J. (2021) · Critical Perspectives on Accounting · doi:10.1016/j.cpa.2021.102296 · Cited by 22
Improving the reproducibility of findings by updating research methodology
Klein, J. (2021) · Quality & Quantity · doi:10.1007/s11135-021-01196-6 · Cited by 22
Deterding, S., & Cutting, J. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3611065 · Cited by 12
“But we know empirically that introducing preregistration substantially increases the proportion of reported null results in a given body of literature [40].”
BackgroundExamination of CIs in health and medical journals from 1976 to 2019: an observational study
Barnett, A. G., & Wren, J. D. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-032506 · Cited by 18
Suomi, A., Lucas, N., McArthur, M., Humphreys, C., Dobbins, T., & Taplin, S. (2020) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2020.104708 · Cited by 18
Opioid-related genetic polymorphisms do not influence postoperative opioid requirement
Aubrun, F., Zahr, N., Langeron, O., Boccheciampe, N., Cozic, N., Belin, L., Hulot, J., Khiami, F., & Riou, B. (2018) · European Journal of Anaesthesiology · doi:10.1097/eja.0000000000000793 · Cited by 18
A Targeted Review of Open Practices in Special Education Publications
Cook, B. G., van Dijk, W., Vargas, I., Aigotti, S. M., Fleming, J. I., McDonald, S. D., Richmond, C. L., Griendling, L. M., McLucas, A. S., & Johnson, R. M. (2023) · Exceptional Children · doi:10.1177/00144029221145195 · Cited by 19
“Research in medicine and psychology suggests preregistration is associated with smaller study effects, perhaps by discouraging the use of questionable research practices (see Dechartres et al., 2016; Kaplan & Irvin, 2015; Papageorgiou et al., 2018; Schäfer & Schwarz, 2019).”
BackgroundLandrum, A. R., Hilgard, J., Lull, R. B., Akin, H., & Jamieson, K. H. (2018) · Journal of Science Communication · doi:10.22323/2.17020204 · Cited by 19
“Prior to these regulations, 17 out of 30 randomized clinical trials reported significant benefits; after these regulations, only 2 out of 25 reported significant benefits [Kaplan and Irvin, 2015].”
BackgroundO'Connor, D. B. (2021) · Health Psychology Bulletin · doi:10.5334/hpb.30 · Cited by 16
Visual Belief Elicitation Reduces the Incidence of False Discovery
Koonchanok, R., Tawde, G. Y., Narayanasamy, G. R., Walimbe, S., & Reda, K. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580808 · Cited by 21
“Pre-registration is associatedwith increased reporting of null results [27], suggesting that it may alleviate the bias in favor of positive outcomes.”
ResultThe Bias Dynamics Model: Correcting for Meta-biases in Therapeutic Prediction
Erasmus, A. (2023) · Philosophy of Science · doi:10.1017/psa.2023.66 · Cited by 16
“For instance, one study found a reduction in some of these mechanisms, including questionable design and analytic practices, after a requirement that studies be recorded on official trial registries was introduced at the turn of the millennium (Kaplan and Irvin 2015).”
BackgroundModelling Scientific Communities
O’Connor, C. (2023) · Cambridge University Press eBooks · doi:10.1017/9781009359535 · Cited by 28
Sandbank, M. P., Bottema‐Beutel, K., Syu, Y.-C., Caldwell, N., Feldman, J. I., & Woynaroski, T. G. (2024) · Autism · doi:10.1177/13623613241231624 · Cited by 16
Re-Analyzing Phase III Bremelanotide Trials for “Hypoactive Sexual Desire Disorder” in Women
Spielmans, G. I. (2021) · The Journal of Sex Research · doi:10.1080/00224499.2021.1885601 · Cited by 17
“Kaplan et al. (2015) examined whether study preregistration related to reported study outcomes among clinical trials funded by the National Heart Lung, and Blood Institute (NHLBI).”
4. What is cultural evolution like?
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.04 · Cited by 15
Detecting Deception in Our Research Participants: Are Your Participants Who You Think They Are?
McCaul, M. E., & Wand, G. S. (2017) · Alcoholism Clinical and Experimental Research · doi:10.1111/acer.13556 · Cited by 15
“Indeed, one recent review reported a significant decline in the percentage of cardiovascular studies that showed a significant benefit of treatment prior to versus following the year 2000 (Kaplan and Irvin, 2015).”
BackgroundOsborne, C., & Norris, E. (2022) · Cogent Psychology · doi:10.1080/23311908.2022.2066304 · Cited by 15
Munafo, M. R. (2015) · Addiction · doi:10.1111/add.13147 · Cited by 16
Choosing quality problems wisely: identifying improvements worth developing and sustaining
Soong, C., Cho, H. J., & Shojania, K. G. (2020) · BMJ Quality & Safety · doi:10.1136/bmjqs-2020-011054 · Cited by 15
Segerstrom, S. C., Diefenbach, M. A., Hamilton, K., O'Connor, D. B., Tomiyama, A. J., Bacon, S., Bennett, G. G., Brondolo, E. N., Czajkowski, S. M., Davidson, K. W., et al. (2023) · Annals of Behavioral Medicine · doi:10.1093/abm/kaac044 · Cited by 14
Supporting study registration to reduce research waste
Purgar, M., Glasziou, P., Klanjšček, T., Nakagawa, S., & Čulina, A. (2024) · Nature Ecology & Evolution · doi:10.1038/s41559-024-02433-5 · Cited by 17
“…(found in five of five meta-studies 21,24,29–31 on the topic), less often supported the hypothesis (found in four of five meta-studies 26,27,32–34 ) and had lower statistical significance (found in one of one meta-study 35 ) than published studies that were not preregistered (the details are…”
Langley, P. C. (2017) · INNOVATIONS in pharmacy · doi:10.24926/iip.v8i2.514 · Cited by 13
Norris, E., Prescott, A., Noone, C., Green, J., Reynolds, J. P., Grant, S., & Toomey, E. C. (2022) · Psychology and Health · doi:10.1080/08870446.2022.2139830 · Cited by 13
“…cessation randomised controlled trial behaviour change papers published in 2018 and 2019 contrastingly found greater pre-registration at 74% (Norris, He, et al., 2021), contributed to by funder requirements in health science to pre-register clinical trials (Kaplan & Irvin, 2015; Kimmelman, 2021).”
Publication Rates of Heart Failure Clinical Trials Remain Low
Psotka, M. A., Latta, F., Cani, D., Fiuzat, M., Sbolli, M., Barnett, S. D., Metra, M., & O’Connor, C. M. (2020) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2020.04.068 · Cited by 16
Reis, D., & Friese, M. (2022) · Springer eBooks · doi:10.1007/978-3-031-04968-2_5 · Cited by 14
The Myth of the Placebo Response
Jonas, W. B. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00577 · Cited by 13
“The result was that these drugs added ~8% over the spontaneous or placebo healing rates for those diseases (11).”
Resource Sharing to Improve Research Quality
Hamra, G. B., Goldstein, N. D., & Harper, S. B. (2019) · Journal of the American Heart Association · doi:10.1161/jaha.119.012292 · Cited by 12
“org, which directly addresses the file drawer and p-hacking problems in publication.(27) A recent comment suggested that preregistration and eventual sharing of resources (including study data) were vital components of clinical trials to improve transparency and public trust.”
BackgroundNow is the time to assess the effects of open science practiceswith randomized control trials.
Suls, J. M., Rothman, A. J., & Davidson, K. W. (2022) · American Psychologist · doi:10.1037/amp0000871 · Cited by 15
Open Science in Developmental Science
Gennetian, L. A., Frank, M. C., & Tamis‐LeMonda, C. S. (2022) · Annual Review of Developmental Psychology · doi:10.1146/annurev-devpsych-120920-042335 · Cited by 13
ENTRUST-PE: An Integrated Framework for Trustworthy Pain Evidence. White Paper.Preprint
O'Connell, N. E., Richards, G. C., Soliman, N., Ferraro, M. C., Segelcke, D., Eccleston, C., Stewart, G. B., Smart, K. M., Palermo, T. M., Rice, A. S. C., et al. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/e39ys · Cited by 17
Registered reports and associated benefits for agricultural economics
Arpinon, T., & Lefebvre, M. (2024) · Q Open · doi:10.1093/qopen/qoae011 · Cited by 13
Preregistration of Randomized Controlled Trials
Cook, B. G., Wong, V. C., Fleming, J. I., & Solari, E. J. (2022) · Research on Social Work Practice · doi:10.1177/10497315221121117 · Cited by 11
“Kaplan and Irvin (2015) examined the outcomes of all large (i.e., budgets >$500,000/year) RCTs supported by the US National Heart, Lung, and Blood Institute (NHLBI) between 1970 and 2012 that evaluated the effects of drugs or dietary supplements on cardiovascular disease.”
Kavanagh, C. M., & Kapitány, R. (2021) · Journal for the Cognitive Science of Religion · doi:10.1558/jcsr.38713 · Cited by 11
Epilogue: Randomization and Social Policy Evaluation Revisited
Heckman, J. J. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0014 · Cited by 13
“Technology and Human Behavior”
Elson, M., Przybylski, A. K., & Krämer, N. C. (2015) · Journal of Media Psychology Theories Methods and Applications · doi:10.1027/1864-1105/a000170 · Cited by 10
Glöckner, A., Gollwitzer, M., Hahn, L., Lange, J., Sassenberg, K., & Unkelbach, C. (2024) · Social Psychology · doi:10.1027/1864-9335/a000548 · Cited by 11
Toward a Better Understanding of Statistical Significance and p Values in Nursing
Holmberg, C. (2024) · Nursing Forum · doi:10.1155/2024/7263781 · Cited by 11
Neuroethics: Agency in the Age of Brain Science
May, J. (2023) · doi:10.1093/oso/9780197648087.001.0001 · Cited by 14
Practical approaches to improving translatability and reproducibility in preclinical pain research
Soliman, N., & Denk, F. (2023) · Brain Behavior and Immunity · doi:10.1016/j.bbi.2023.09.023 · Cited by 10
“…requirement for clinical trials, it immediately affected the publication landscape: before 2000, when trials were not routinely logged on clinicaltrials.gov, 57 % of studies claimed that their primary outcome was significant; after 2000, this figure dropped down to 8 % (Kaplan and Irvin, 2015).”
BackgroundTranscranial Direct Current Stimulation in Stroke Rehabilitation: Present and Future
Awosika, O. O., & Cohen, L. G. (2019) · Springer eBooks · doi:10.1007/978-3-319-95948-1_17 · Cited by 10
Microfinance RCTs in Development: Miracle or Mirage?
Bédécarrats, F., Guérin, I., & Roubaud, F. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0009 · Cited by 34
Long, J. A. (2021) · Annals of the International Communication Association · doi:10.1080/23808985.2021.1979421 · Cited by 12
Good clinical practice improves rigor and transparency: Lessons from the ACTIVE trial.
Taylor, B., Rebok, G. W., & Marsiske, M. (2022) · Psychology and Aging · doi:10.1037/pag0000653 · Cited by 10
Pang, R. Y., Reinecke, K., & Just, R. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517707 · Cited by 12
“Evidence in clinical trials shows that preregistration reduced the number of studies that rejected null hypotheses from 57% to 8% [42].”
BackgroundWeston, L., Rybczynska‐Bunt, S., Quinn, C., Lennox, C., Maguire, M., Pearson, M., Stirzaker, A., Durcan, G., Stevenson, C., Graham, J., et al. (2022) · PLoS ONE · doi:10.1371/journal.pone.0270691 · Cited by 11
Shedding light on functional near-infrared spectroscopy and open science practices
Kelsey, C. M., Taylor, J., Pirazzoli, L., Lorenzo, R. D., Sullivan, E., & Nelson, C. A. (2023) · Neurophotonics · doi:10.1117/1.nph.10.2.023520 · Cited by 14
Irvin, V. L., & Kaplan, R. M. (2015) · Annals of Behavioral Medicine · doi:10.1007/s12160-015-9739-7 · Cited by 11
“Failure rates were much higher in large-budget randomized controlled trials [9, 13].”
BackgroundResultInfluentialSelf-citationThor, M., Oh, J. H., Apte, A., & Deasy, J. O. (2020) · Frontiers in Oncology · doi:10.3389/fonc.2020.00978 · Cited by 11
“Public pre-registration of SAPs could be thought of as committing to an analytical path but without advancing knowledge of the research outcome (4).”
BackgroundHeinl, C., Chmielewska, J., Olevska, A., Grune, B., Schönfelder, G., & Bert, B. (2020) · The EMBO Reports · doi:10.15252/embr.201949709 · Cited by 10
“A meta-analysis has shown that this measure can indeed increase the number of reported “negative” results [6].”
Waqas, A., Baig, A. A., Khalid, M. A., Aedma, K. K., & Naveed, S. (2019) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2019.05.029 · Cited by 9
Hernán, M. Á., Dickerman, B. A., Swanson, S. A., & Dahabreh, I. J. (2026) · Epidemiology · doi:10.1097/ede.0000000000001951 · Cited by 9
Miniussi, C., & Pellicciari, M. C. (2025) · Journal of Neuropsychology · doi:10.1111/jnp.12425 · Cited by 9
“Sharing FAIR data and running collaborative projects to build consensus-based evaluation and intervention protocols will be a significant step forward to get off the ground and offer the patient the most likely approaches to maximise benefit and reduce harm (Kaplan & Irvin, 2015).”
Make like a glass frog: In support of increased transparency in herpetology
Marshall, B. M. (2021) · Herpetological Journal · doi:10.33256/31.1.3545 · Cited by 10
Cook, B. G. (2016) · Behavioral Disorders · doi:10.17988/0198-7429-41.3.161 · Cited by 9
“Kaplan and Irvin (2015) reported that the proportion of null results reported from large trials funded by the National Heart, Lung, and Blood Institute increased substantially subsequent to the National Institutes of Health requiring funded Behavioral Disorders, 41 (3), 161–172 May 2016 /…”
BackgroundResultJoy and rigor in behavioral science
Collins, H. K., Whillans, A. V., & John, L. K. (2021) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2021.03.002 · Cited by 10
Is the replication crisis a base-rate fallacy?
Autzen, B. (2021) · Metamedicine · doi:10.1007/s11017-022-09561-8 · Cited by 9
On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics
Eble, A., Boone, P., & Elbourne, D. R. (2016) · The World Bank Economic Review · doi:10.1093/wber/lhw034 · Cited by 11
“This increased scrutiny during peer review and after publication has resulted in a reduction, over time, in the presence of the biases described above in medical RCT reports (Plint et al. 2006, Kaplan and Irvin, 2015).”
Estimating the replicability of highly cited clinical research (2004–2018)
da Costa, G. G., Neves, K., & Amaral, O. B. (2024) · PLoS ONE · doi:10.1371/journal.pone.0307145 · Cited by 8
“…research is warranted to examine whether the high replicability of highly cited clinical research is related to particular research practices that are not as widely used in other areas of biomedical science, such as randomization, blinding or prospective protocol registration [14, 16, 99–102].”
BackgroundResultHow open science can support the 3Rs and improve animal research
Janssens, M. R. E., Gaillard, S. D. M., de Haan, J. J., de Leeuw, W., Brooke, M., Burke, M., Flores-Dourojeanni, J. P., Kruijen, I., Menon, J. M. L., Smith, A. J., et al. (2023) · Research Ideas and Outcomes · doi:10.3897/rio.9.e105198 · Cited by 8
How can we optimise learning from trials in child and adolescent mental health?
Axford, N., Berry, V., Lloyd, J., & Wyatt, K. M. (2022) · Evidence-Based Mental Health · doi:10.1136/ebmental-2022-300500 · Cited by 8
Behind the curtains of academic publishing: strategic responses of economists and business scholars
Scheidegger, F., Briviba, A., & Frey, B. S. (2023) · Scientometrics · doi:10.1007/s11192-023-04772-9 · Cited by 8
“Using this approach, particularly for research designs, would reduce bias against null results and again support intrinsically motivated academics (Kaplan & Irvin, 2015).”
MethodReporting Compliance of Stroke Trials: Cross-Sectional Analysis
Malhotra, K., Rayi, A., Khunger, M., Thompson, S., & Liebeskind, D. S. (2017) · Journal of Stroke and Cerebrovascular Diseases · doi:10.1016/j.jstrokecerebrovasdis.2017.03.010 · Cited by 8
Resultados negativos na pesquisa científica: aspectos éticos
Guimarães, R. A. B., Succi, G. D. M., Montalli, V. Â. M., Niederauer, A. J. S., & Succi, R. C. D. M. (2018) · Revista Bioética · doi:10.1590/1983-80422018262245 · Cited by 8
Pre-registration for empirical research and its practices: A tutorial on Open Science Framework
Hasegawa, R., Tada, K., Yonemitsu, F., Ikeda, A., Yamada, Y., Takahashi, K., & Kondo, H. M. (2021) · The Japanese journal of psychology · doi:10.4992/jjpsy.92.20217 · Cited by 13
“人間を対象とするす べての研究は,最初の被験者を募集する前に一般的に アクセス可能なデータベースに登録されなければなら ない」とされており(https://www.med.or.jp/doctor/ international/wma/helsinki.html),プレレジを必須とし ている。Kaplan & Irvin(2015)が行った医学分野の ランダム化比較試験(randomized controlled trial: 以下, RCT研究4とする)についての分析では,プレレジ導 入前の 2000年以前は 57%の研究が介入効果を示した…”
Scientific Methods and the Reporting of Negative Results: Critically Important to Patient Safety
Mastracci, T. M. (2015) · European Journal of Vascular and Endovascular Surgery · doi:10.1016/j.ejvs.2015.08.007 · Cited by 9
Forensic science needs registered reports
Chin, J. M., McFadden, R., & Edmond, G. (2019) · Forensic Science International Synergy · doi:10.1016/j.fsisyn.2019.10.005 · Cited by 8
“Clinical medical researchers learned this lesson decades ago when published studies in their field concluded that an unrealistically high percentage of new drugs were effective [2,3].”
BackgroundMake like a glass frog: In support of increased transparency in herpetologyPreprint
Marshall, B. M., & Strine, C. T. (2020) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/74frd · Cited by 8
“Although moves towards more transparent practices are new, the implemented solutions do indeed appear to be reducing publication bias and associated false positive rates (Allen & Mehler, 2018; Kaplan & Irvin, 2015; Scheel et al., 2020; Toth et al., 2020).”
Background
Metadata
- Authors
- Kaplan, R. M., & Irvin, V. L.
- Affiliations
- Agency for Healthcare Research and Quality (US)
- Oregon State University (US)
- Corresponding author
- Kaplan, Robert M.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-08-05
- Volume
- 10
- Issue
- 8
- Pages
- e0132382
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4526697
- OpenAlex
- W1944584781
- Cited by
- 356
- References
- 11
- Citations by year
- 2026: 13 · 2025: 17 · 2024: 22 · 2023: 44 · 2022: 28 · 2021: 50 · 2020: 58 · 2019: 33 · 2018: 37 · 2017: 24 · 2016: 21 · 2015: 9
- FWCI
- 35.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS ONE (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Oregon State University
- National Institutes of Health
- Agency for Healthcare Research and Quality
- National Heart, Lung, and Blood Institute
- U.S. National Library of Medicine
- Office of Behavioral and Social Sciences Research
- Topics
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- negative results, clinical trial results, cardiovascular disease prevention, reporting guidelines, all-cause mortality, publication bias
- MeSH terms
- Cardiovascular Diseases, Humans, Publishing, Registries, Research Design, United States, Randomized Controlled Trials as Topic, Treatment Outcome, National Heart, Lung, and Blood Institute (U.S.)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.