Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
BACKGROUND: We explore whether the number of null results in large National Heart Lung, and Blood Institute (NHLBI) funded trials has increased over time. METHODS: We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease. Trials were included if direct costs >$500,000/year, participants were adult humans, and the primary outcome was cardiovascular risk, disease or death. The 55 trials meeting these criteria were coded for whether they were published prior to or after the year 2000, whether they registered in clinicaltrials.gov prior to publication, used active or placebo comparator, and whether or not the trial had industry co-sponsorship. We tabulated whether the study reported a positive, negative, or null result on the primary outcome variable and for total mortality. RESULTS: 17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 (χ2=12.2,df= 1, p=0.0005). There has been no change in the proportion of trials that compared treatment to placebo versus active comparator. Industry co-sponsorship was unrelated to the probability of reporting a significant benefit. Pre-registration in clinical trials.gov was strongly associated with the trend toward null findings. CONCLUSIONS: The number NHLBI trials reporting positive results declined after the year 2000. Prospective declaration of outcomes in RCTs, and the adoption of transparent reporting standards, as required by clinicaltrials.gov, may have contributed to the trend toward null findings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Registering a protocol before the start of the experiment enables researchers to demonstrate that the hypothesis, approach, and analysis were planned in advance and not shaped by data as they emerged; it enhances scientific rigour and protects the researcher against concerns about selective reporting of results [239,240].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [239]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Preregistration is required for all government-funded medical clinical trials in the United States (Food and Drug Administration, 2007; see also Kaplan & Irvin, 2015).”
From Do growth mindset interventions impact students’ academic achievement? A systematic review and meta-analysis with recommendations for best practices. (2023) · cites this paper as Kaplan & Irvin, 2015
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Effect estimates may vary according to whether or not trials are registered (Kaplan 2015).”
From Probiotics for treating acute infectious diarrhoea (2020) · cites this paper as Kaplan 2015
The cited source says what the claim attributes to it.
In the citing paper
“Registering a protocol before the start of the experiment enables researchers to demonstrate that the hypothesis, approach, and analysis were planned in advance and not shaped by data as they emerged; it enhances scientific rigour and protects the researcher against concerns about selective reporting of results [239,240].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [239]
In the citing paper
“The proportion of large, National Heart, Lung, and Blood Institute studies reporting null findings over time has increased, which is suggestive of decreases in publication bias (68).”
From Issues with data and analyses: Errors, underlying themes, and potential solutions (2018) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Publication bias in clinical research
Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991) · The Lancet · doi:10.1016/0140-6736(91)90201-y · Cited by 3,109 · cited 1×
“It is widely noted that journals favor publication of statistically significant findings[2].”
BackgroundMoher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545
Alzheimer’s disease drug-development pipeline: Few candidates, frequent failures
Cummings, J. L., Morstorf, T., & Zhong, K. (2014) · Alzheimer's Research & Therapy · doi:10.1186/alzrt269 · Cited by 1,744
The ClinicalTrials.gov Results Database — Update and Key Issues
Zarin, D. A., Tse, T., Williams, R. J., Califf, R. M., & Ide, N. C. (2011) · New England Journal of Medicine · doi:10.1056/nejmsa1012065 · Cited by 838 · cited 1×
“Registered studies must provide: the study’s purpose, recruitment status, design, eligibility criteria, locations and pre-specified primary and secondary outcomes[11].”
BackgroundTrial Registration at ClinicalTrials.gov between May and October 2005
Zarin, D. A., Tse, T., & Ide, N. C. (2005) · New England Journal of Medicine · doi:10.1056/nejmsa053234 · Cited by 291
Cardiovascular Drug Development
Fordyce, C. B., Roe, M. T., Ahmad, T., Libby, P., Borer, J. S., Hiatt, W. R., Bristow, M. R., Packer, M., Wasserman, S. M., Braunstein, N., et al. (2015) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2015.03.016 · Cited by 202 · cited 1×
“Since about 1998, there has been a systematic decline in the number approvals for new cardiovascular drugs[6].”
BackgroundMoving Toward Transparency of Clinical Trials
Zarin, D. A., & Tse, T. (2008) · Science · doi:10.1126/science.1153632 · Cited by 182
Publication of Trials Funded by the National Heart, Lung, and Blood Institute
Gordon, D., Taddei-Peters, W., Mascette, A., Antman, M., Kaufmann, P. G., & Lauer, M. S. (2013) · New England Journal of Medicine · doi:10.1056/nejmsa1300237 · Cited by 120
Fisher, J. A., Cottingham, M. D., & Kalbaugh, C. A. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2014.08.023 · Cited by 63 · cited 1×
“It has been argued that there have been few efficacious drugs in the pipeline[4,5].”
BackgroundRoth, J. A., Etzioni, R., Waters, T. M., Pettinger, M., Rossouw, J. E., Anderson, G. L., Chlebowski, R. T., Manson, J. E., Hlatky, M., Johnson, K. C., et al. (2014) · Annals of Internal Medicine · doi:10.7326/m13-2348 · Cited by 58 · cited 1×
“The economic analysis estimated that there was about $140 returned for each dollar invested in the study[13].”
Background
Research works printed but not linked: 1
- Gordon D, Taddei-Peters W, Mascette A, Antman M, Kaufmann PG, Lauer MS. Publication of trials funded by the National Heart, Lung, and Blood Institute. N Engl J Med. 2013; 369: 1226– 1234. PMC3928673 24224625 · DOI · Paper page
Cited by
Items 301–360 of 360
Wang, Z. (2020) · eScholarship (California Digital Library)
Mimex Montague Healthcare Limited, M. H., & Graham, W. E. (2020) · Journal of Clinical Cases & Reports · doi:10.46619/joccr.2020.3-1068
Transparency and replication in alcohol research
Drax, K., & Munafo, M. R. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-816720-5.00005-0
Preregistration, Data Sharing, and Other Salutary Behaviors
Bausell, R. B. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780197536537.003.0011
A COVID-19 Vaccines Review: What We Have Learned So Far and Where Do We Go from Here?Preprint
Wong, S. C. Y., & Morgenstern, C. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3812637
Python Uygulamalı İstatiksel Veri Bilimi ve Analizi
Ahmet, S. E. L. (2021) · Akademisyen Kitabevi eBooks · doi:10.37609/akya.286
Preferences, Equipoise, and Blinding
Powell, L. H., Kaufmann, P. G., & Freedland, K. E. (2021) · Springer eBooks · doi:10.1007/978-3-030-39330-4_10
Questionable Research Practices and Determinants of Their FrequencyPreprint
Alsalti, T. (2021) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/uqz2e
Questionable Research Practices and Determinants of Their FrequencyPreprint
Alsalti, T. (2021) · Thesis Commons (OSF Preprints) · doi:10.31237/osf.io/mjvwu
(2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.003.0015
de Winter, J., & Dodou, D. (2017) · SpringerBriefs in applied sciences and technology · doi:10.1007/978-3-319-56964-2_4
Can we Trust Science and, if so, Why?Preprint
Wuttke, A. (2020) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/7fmt9
Qu, Y. L., Rappaport, B. I., Luby, J. L., & Barch, D. M. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/7gzkj
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.15
Nettle, D. (2018) · Open Book Publishers · doi:10.11647/obp.0155.13
Haslwanter, T. (2016) · Statisctics and computing/Statistics and computing · doi:10.1007/978-3-319-28316-6_5
Assessment of Evidential Value Requires More Than a Single Data PointPreprint
Imhoff, R. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/cpjr7
(2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.002.0007
(2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.002.0006
(2020) · Oxford University Press eBooks · doi:10.1093/oso/9780198865360.002.0003
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.002.0004
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.002.0006
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.003.0010
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.002.0007
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.003.0004
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.002.0008
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.002.0003
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.003.0001
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.003.0009
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.003.0003
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.003.0002
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.003.0006
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.003.0005
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.003.0008 · Cited by 1
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.003.0007
May, J. (2023) · Oxford University Press eBooks · doi:10.1093/oso/9780197648087.002.0010
(2023) · The MIT Press eBooks · doi:10.7551/mitpress/14922.003.0016
Plummer, M., & Franceschi, S. (2023) · Springer eBooks · doi:10.1007/978-1-4614-6625-3_9-1
How does ClinicalTrials.gov Impact Company Innovation?Preprint
Niu, Y. (2024) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2405.07998
The replication crisis in science and its overlap with kinanthropology: an introduction (Part 2)
Mališ, J., & Baloun, L. (2024) · Tělesná kultura · doi:10.5507/tk.2024.003
“Tyto výsledky poukazují na významné zmírnění publikačního zkreslení (Kaplan & Irvin, 2015).”
Reproduzierbarkeit bei Forschungssynthesen. Herausforderungen und Lösungsansätze
Schneider, J. M., & Heck, T. (2024) · Waxmann Verlag GmbH eBooks · doi:10.31244/9783830999126.08
Merits and Limits of Preregistration for Visualization Research
Besançon, L., Nosek, B. A., Haven, T. L., Meyer, M., Dunne, C., & Ghoniem, M. (2024) · IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV) · doi:10.1109/beliv64461.2024.00015 · Cited by 3
“In practice, preregistration helps alleviate issues around questionable research practices such as outcome switching or HARK-ing [17, 18], selective reporting [1, 16], or p-hacking [31], and is also seen as a way to reduce the amount of duplicated research efforts [8].”
Bias in Randomized Controlled Trials
Schmaling, K. B., & Kaplan, R. M. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009391733.008
Self-citationSchmaling, K. B., & Kaplan, R. M. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009391733.007
Self-citationImplementation: Getting the Right Interventions to Market
Schmaling, K. B., & Kaplan, R. M. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009391733.012
Self-citationSchmaling, K. B., & Kaplan, R. M. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009391733.006
Self-citationAutonomous Artificial Intelligence Safety and Trust
Abràmoff, M. D. (2025) · Springer eBooks · doi:10.1007/978-3-031-83756-2_4
We found only one-third of published psychology research is reliable – now what?
Gilbert, E. A., & Strohminger, N. (2015) · The Conversation · doi:10.64628/aai.mtamexvge
Investigating factors influencing audiences’ integration of scientific evidence.
Schreiner, M. R., Rebholz, T. R., Pütter, J. Q., & Landrum, A. R. (2025) · Journal of Experimental Psychology Applied · doi:10.1037/xap0000552
Optimising methodological practices within psychology: a parapsychology case-study
Pooley, A. L. (2025) · ERA · doi:10.7488/era/6177
Digital tools and services to support research replicability and verifiability
(2018) · JISC Information Environment Repository (Jisc)
RDoFlow: Automatically assessing under-specified statistical analyses in HCI
Grunde-McLaughlin, M., Liu, W., Patil, R., Migineishvili, N., Reif, E., Krishna, R., Weld, D. S., & Heer, J. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789160
“The replication crisis has garnered concern across psychology [17, 107], medicine [47, 50], and the social sciences [13, 71].”
Accroître la généralisabilité de la psychologie en incluant les Africains
Adetula, A. (2024) · theses.fr (ABES) · doi:10.70675/b3891ccaz527az4633z94e7z178af65a3b34
Yi, S., Ruan, J., Wang, Z., Ding, F., Liu, Y., Ao, Q., Ren, Z., Wu, C., Luo, Y., Niu, M., et al. (2026) · BMC Veterinary Research · doi:10.1186/s12917-026-05468-x
“Typically, embargo rules impose a restricted period following submission, during which key registration details—such as objectives, hypotheses, designs, methods, and analysis plans— remain inaccessible to the public [7, 8, 17, 18].”
Heinl, C., Franco, N. H., Bert, B., Siewert, K., Movia, D., Aruçi, E., Ladeira, C., Sanin, A. Y., Neuhaus, W., Kahlert, U. D., et al. (2026) · EMBO Reports · doi:10.1038/s44319-026-00764-x
“…instance, by de fi ning exclusion criteria based on the results, selecting another group as control, changing the statistical test, adding more replicates, transforming the data, and so on — hereby reducing the risk of biased or spurious fi ndings (Kaplan and Irvin, 2015; van den Akker et al, 2024).”
Riley, S. P., & Flowers, D. W. (2026) · Journal of Manual & Manipulative Therapy · doi:10.1080/10669817.2026.2696150
“Both retrospective and prospective studies identify the most likely culprit as high levels of unexplained variability, secondary to a lack of verified research and publication integrity through verified prospective clinical trial registration [5,11,12].”
León, S. P., Lipnevich, A. A., & Garrido, M. G. (2026) · Review of Educational Research · doi:10.3102/00346543261458396
Open Veins of Algorithmic Auditing: Why AI Assessment Lags Behind Its Deployment in the Global SouthPreprint
Clavell, G. G., & Magaard, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.21317
“…tripled within two years, and among large publicly funded cardiovascular trials the share reporting positive outcomes fell from 57 percent before mandatory registration to 8 percent after, a direct measure of the distortion that unmeasured practice had been hiding (Kaplan and Irvin, 2015).”
Die Präregistrierung in der klinischen Psychologie
Mundorf, A., Peterburs, J., & Ocklenburg, S. (2026) · Springer eBooks · doi:10.1007/978-3-662-73259-5_4
Thornton, J., & Henderson, J. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag152
Metadata
- Authors
- Kaplan, R. M., & Irvin, V. L.
- Affiliations
- Agency for Healthcare Research and Quality (US)
- Oregon State University (US)
- Corresponding author
- Kaplan, Robert M.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-08-05
- Volume
- 10
- Issue
- 8
- Pages
- e0132382
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4526697
- OpenAlex
- W1944584781
- Cited by
- 356
- References
- 11
- Citations by year
- 2026: 13 · 2025: 17 · 2024: 22 · 2023: 44 · 2022: 28 · 2021: 50 · 2020: 58 · 2019: 33 · 2018: 37 · 2017: 24 · 2016: 21 · 2015: 9
- FWCI
- 35.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- public-domain
- creativecommons.org/publicdomain/zero/1.0/ (vor)
- Locations
- PLoS ONE (publishedVersion, public-domain)
- Europe PMC (PubMed Central) (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Oregon State University
- National Institutes of Health
- Agency for Healthcare Research and Quality
- National Heart, Lung, and Blood Institute
- U.S. National Library of Medicine
- Office of Behavioral and Social Sciences Research
- Topics
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- negative results, clinical trial results, cardiovascular disease prevention, reporting guidelines, all-cause mortality, publication bias
- MeSH terms
- Cardiovascular Diseases, Humans, Publishing, Registries, Research Design, United States, Randomized Controlled Trials as Topic, Treatment Outcome, National Heart, Lung, and Blood Institute (U.S.)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.