Why Most Clinical Research Is Not Useful
Open accessThe DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
John Ioannidis argues that problem base, context placement, information gain, pragmatism, patient centeredness, value for money, feasibility, and transparency define useful clinical research. He suggests most clinical research is not useful and reform is overdue.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Neglecting the actual development phase can be problematic, as a poor problem definition, insufficient attention to existing evidence and context needs, a missing model underlying the intervention, and an unsound selection of hypothesized change mechanisms can lead to inefficient treatments (30–32).”
From Developing a mechanism-based therapy for acute psychiatric inpatients with psychotic symptoms: an Intervention Mapping approach (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredIn the citing paper
“Systematic review guidelines typically advocate that all relevant studies must be included for the conclusions of that review to be valid 22”
From Software support for environmental evidence synthesis (2018) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“I have previously written about why most published research is false [2] and how to make more of it true Summary Points”
BackgroundSelf-citationTunis (2003) · doi:10.1001/jama.290.12.1624 · Cited by 1,724 · cited 1×
“Pragmatism has long been advocated in clinical research [15], but it is rare.”
BackgroundSurrogate End Points in Clinical Trials: Are We Being Misled?
Fleming, T. R., & DeMets, D. L. (1996) · Annals of Internal Medicine · doi:10.7326/0003-4819-125-7-199610010-00011 · Cited by 1,651 · cited 1×
“Nonvalidated surrogate outcomes lacking clinical insight [13] and composite outcomes that combine outcomes of very different clinical portent [14] are often utilized so that authors can claim that clinical studies are well powered.”
BackgroundIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Self-citationReducing waste from incomplete or unusable reports of biomedical research
Glasziou, P., Altman, D. G., Bossuyt, P., Boutron, I., Clarke, M., Julious, S., Michie, S., Moher, D., & Wager, E. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62228-x · Cited by 1,319
Biomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Self-citationMoynihan, R., G⊘tzsche, P. C., Heath, I., & Henry, D. (2002) · BMJ · doi:10.1136/bmj.324.7342.886 · Cited by 900
Increasing value and reducing waste: addressing inaccessible research
Chan, A.-W., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gøtzsche, P. C., Krumholz, H. M., Ghersi, D., & van der Worp, H. B. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62296-5 · Cited by 887
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760 · cited 1×
“Joint efforts by multiple stakeholders may yield solutions that are more likely to be more widely adopted and thus successful [3].”
BackgroundSelf-citationLaunching PCORnet, a national patient-centered clinical research network
Fleurence, R. L., Curtis, L. H., Califf, R. M., Platt, R., Selby, J. V., & Brown, J. S. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2014-002747 · Cited by 637 · cited 1×
“Partnerships with patient-centered research initiatives [50] and academia can potentially solve some of the challenges of designing and implementing more pragmatic trials [51].”
BackgroundHopewell, S., Dutton, S., Yu, L.-M., Chan, A.-W., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c723 · Cited by 504 · cited 1×
“Across PubMed, the median sample size for published randomized trials in 2006 was 36 per arm [12].”
MethodFerreira-González, I., Permanyer-Miralda, G., Domingo-Salvany, A., Busse, J. W., Heels-Ansdell, D., Montori, V. M., Akl, E. A., Bryant, D. M., Alonso-Coello, P., Alonso, J., et al. (2007) · BMJ · doi:10.1136/bmj.39136.682083.ae · Cited by 473
(2014) · Choice Reviews Online · doi:10.5860/choice.51-3280 · Cited by 449 · cited 1×
“Moreover, disease mongering affects multiple medical specialties [4,7,8].”
Jørgensen, A. W., Hilden, J., & Gøtzsche, P. C. (2006) · BMJ · doi:10.1136/bmj.38973.444699.0b · Cited by 444
Prevalence, Characteristics, and Publication of Discontinued Randomized Trials
Kasenda, B., von Elm, E., You, J., Blümle, A., Tomonaga, Y., Saccilotto, R., Amstutz, A., Bengough, T., Meerpohl, J. J., Stegert, M., et al. (2014) · JAMA · doi:10.1001/jama.2014.1361 · Cited by 415
Bias in published cost effectiveness studies: systematic review
Bell, C. M., Urbach, D. R., Ray, J. G., Bayoumi, A., Rosen, A. B., Greenberg, D., & Neumann, P. J. (2006) · BMJ · doi:10.1136/bmj.38737.607558.80 · Cited by 382
Outcome Reporting in Industry-Sponsored Trials of Gabapentin for Off-Label Use
Vedula, S. S., Bero, L., Scherer, R. W., & Dickersin, K. (2009) · New England Journal of Medicine · doi:10.1056/nejmsa0906126 · Cited by 361 · cited 1×
“Trust has been eroded whenever major subversion of the evidence has been uncovered by legal proceedings [36] or reanalysis [37] with different conclusions (e.”
Discontinuation and non-publication of surgical randomised controlled trials: observational study
Chapman, S. J., Shelton, B., Mahmood, H., Fitzgerald, J. E., Harrison, E. M., & Bhangu, A. (2014) · BMJ · doi:10.1136/bmj.g6870 · Cited by 221 · cited 1×
“Twenty-five percent of the trials approved by six research ethics committees between 2000 and 2003 in Canada, Germany, and Switzerland were discontinued [28], and the discontinuation rate was 43% for a cohort of surgical trials registered between 2008 and 2009 [29].”
BackgroundHemkens, L. G., Contopoulos-Ioannidis, D. G., & Ioannidis, J. P. A. (2016) · BMJ · doi:10.1136/bmj.i493 · Cited by 220 · cited 2×
“Observational studies often add more confusion rather than filling the information deficits [18,19].”
BackgroundSelf-citationShould people at low risk of cardiovascular disease take a statin?
Abramson, J. D., Rosenberg, H. G., Jewell, N., & Wright, J. M. (2013) · BMJ · doi:10.1136/bmj.f6123 · Cited by 208
Attention to Local Health Burden and the Global Disparity of Health Research
Evans, J. A., Shim, J.-M., & Ioannidis, J. P. A. (2014) · PLoS ONE · doi:10.1371/journal.pone.0090147 · Cited by 191 · cited 1×
“Data show only weak or modest correlations between the amount of research done and the burden of various diseases [5,6].”
BackgroundSelf-citationEnsuring the integrity of clinical practice guidelines: a tool for protecting patients
Lenzer, J., Hoffman, J. R., Furberg, C. D., & Ioannidis, J. P. A. (2013) · BMJ · doi:10.1136/bmj.f5535 · Cited by 187
Self-citationRoutinely collected data and comparative effectiveness evidence: promises and limitations
Hemkens, L. G., Contopoulos-Ioannidis, D. G., & Ioannidis, J. P. A. (2016) · Canadian Medical Association Journal · doi:10.1503/cmaj.150653 · Cited by 181
Self-citationSensible approaches for reducing clinical trial costs
Eisenstein, E. L., Collins, R., Cracknell, B. S., Podesta, O., Reid, E. D., Sandercock, P., Shakhov, Y., Terrin, M. L., Sellers, M. A., Califf, R. M., et al. (2008) · Clinical Trials · doi:10.1177/1740774507087551 · Cited by 176 · cited 1×
“Clinical research remains extremely expensive, even though an estimated 90% of the present cost of trials could be safely eliminated [26,27].”
BackgroundClarke, M., & Chalmers, I. (1998) · JAMA · doi:10.1001/jama.280.3.280 · Cited by 169
Chen, Y.-F., Madan, J., Welton, N., Yahaya, I., Aveyard, P., Bauld, L., Wang, D., Fry-Smith, A., & Munafò, M. (2012) · Health Technology Assessment · doi:10.3310/hta16380 · Cited by 165
Estimates of the Continuously Publishing Core in the Scientific Workforce
Ioannidis, J. P. A., Boyack, K. W., & Klavans, R. (2014) · PLoS ONE · doi:10.1371/journal.pone.0101698 · Cited by 160 · cited 1×
“The clinical research workforce is huge: millions of people have coauthored at least one biomedical paper, and most have done so only once [48].”
BackgroundSelf-citationWager, E., Williams, P., & on behalf of the OPEN Project (Overcome failure to Publish nEgative fiNdings) Consortium (2013) · BMJ · doi:10.1136/bmj.f5248 · Cited by 139
How to make more published research true
Ioannidis, J. P. A. (2015) · Acimed · Cited by 134
Self-citationClarke, M., Hopewell, S., & Chalmers, L. (2007) · Journal of the Royal Society of Medicine · doi:10.1177/014107680710011415 · Cited by 129 · cited 1×
“Most new research is not preceded or accompanied by systematic reviews [10,11].”
BackgroundEthics and Regulatory Complexities for Pragmatic Clinical Trials
Sugarman, J., & Califf, R. M. (2014) · JAMA · doi:10.1001/jama.2014.4164 · Cited by 127 · cited 1×
“Partnerships with patient-centered research initiatives [50] and academia can potentially solve some of the challenges of designing and implementing more pragmatic trials [51].”
BackgroundRandomized Clinical Trials — Removing Unnecessary Obstacles
Reith, C., Landray, M., Devereaux, P. J., Bosch, J., Granger, C. B., Baigent, C., Califf, R. M., Collins, R., & Yusuf, S. (2013) · New England Journal of Medicine · doi:10.1056/nejmsb1300760 · Cited by 123 · cited 1×
“Practical applications of value-of-information methods are counted in single digit numbers [24,25].”
MethodAre Clinical Trials a Cost-effective Investment?
Detsky, A. S. (1989) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1989.03430130071037 · Cited by 105 · cited 1×
“However, the benefits of useful clinical research more than offset the cost of performing it [23].”
BackgroundPCORI at 3 Years — Progress, Lessons, and Plans
Selby, J. V., & Lipstein, S. H. (2014) · New England Journal of Medicine · doi:10.1056/nejmp1313061 · Cited by 103 · cited 1×
“There is currently a heightened interest in patient-centered research, as exemplified by the Patient-Centered Outcomes Research Institute (PCORI), which was launched in 2012 in the United States to foster research relevant to patient needs [21].”
BackgroundIoannidis, J. P. A., Cappelleri, J. C., Lau, J., Skolnik, P. R., Melville, B., Chalmers, T. C., & Sacks, H. S. (1995) · Annals of Internal Medicine · doi:10.7326/0003-4819-122-11-199506010-00009 · Cited by 92 · cited 1×
“In one of my first papers, a systematic review of zidovudine monotherapy [42], the question was extremely”
BackgroundSelf-citationSoares, M., Welton, N., Harrison, D., Peura, P., Hari, M., Harvey, S., Madan, J., Ades, A., Palmer, S., & Rowan, K. (2012) · Health Technology Assessment · doi:10.3310/hta16070 · Cited by 91 · cited 1×
“Practical applications of value-of-information methods are counted in single digit numbers [24,25].”
MethodPrasad, V., Jorgenson, J., Ioannidis, J. P. A., & Cifu, A. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.11.005 · Cited by 90 · cited 1×
“Observational studies (performed in the thousands) are often precariously interpreted as able to answer questions about causal treatment effects [17].”
BackgroundSelf-citationPatient-Centeredness in the Design of Clinical Trials
Mullins, C. D., Vandigo, J., Zheng, Z., & Wicks, P. (2014) · Value in Health · doi:10.1016/j.jval.2014.02.012 · Cited by 85 · cited 1×
“Useful research is patient centered [20].”
Trial Registration Numbers Are Underreported in Biomedical Publications
van de Wetering, F. T., Scholten, R. J. P. M., Haring, T., Clarke, M., & Hooft, L. (2012) · PLoS ONE · doi:10.1371/journal.pone.0049599 · Cited by 79 · cited 1×
“Only 61% of trials published in clinical journals in 2010 had been registered [30], and rates are much lower for nonregulated interventions [31] (e.”
BackgroundDisease Mongering Is Now Part of the Global Health Debate
Moynihan, R. N., Doran, E., & Henry, D. A. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050106 · Cited by 76 · cited 2×
“Moreover, disease mongering affects multiple medical specialties [4,7,8].”
BackgroundMilette, K., Roseman, M., & Thombs, B. D. (2010) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2010.09.015 · Cited by 71 · cited 1×
“, 21% and 29% for trials published in psychological or behavioral [32] and physical therapy [33] journals, respectively).”
BackgroundRaw data from clinical trials: within reach?
Doshi, P., Goodman, S. N., & Ioannidis, J. P. A. (2013) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2013.10.006 · Cited by 67 · cited 1×
“Few full protocols are registered, analysis plans are almost never prespecified, and the full study data are rarely available [35].”
BackgroundSelf-citationMultisystem failure: the story of anti-influenza drugs
Jefferson, T., & Doshi, P. (2014) · BMJ · doi:10.1136/bmj.g2263 · Cited by 62 · cited 1×
“Trust has been eroded whenever major subversion of the evidence has been uncovered by legal proceedings [36] or reanalysis [37] with different conclusions (e.”
Use of data from registered clinical trials to identify gaps in health research and development
Viergever, R. F., Terry, R. F., & Karam, G. (2013) · Bulletin of the World Health Organization · doi:10.2471/blt.12.114454 · Cited by 62 · cited 1×
“Data show only weak or modest correlations between the amount of research done and the burden of various diseases [5,6].”
BackgroundMinimal Modeling Approaches to Value of Information Analysis for Health Research
Meltzer, D., Hoomans, T., Chung, J. W., & Basu, A. (2011) · Medical Decision Making · doi:10.1177/0272989x11412975 · Cited by 62 · cited 1×
“Good value for money is an important consideration, especially in an era of limited resources, and this can be assessed with formal modeling (value of information) [22].”
BackgroundClarke, M., Hopewell, S., & Chalmers, L. (2007) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.100.4.187 · Cited by 59
Call to improve transparency of trials of non-regulated interventions
Dal-Re, R., Bracken, M. B., & Ioannidis, J. P. A. (2015) · BMJ · doi:10.1136/bmj.h1323 · Cited by 55 · cited 1×
“Only 61% of trials published in clinical journals in 2010 had been registered [30], and rates are much lower for nonregulated interventions [31] (e.”
BackgroundSelf-citationA Guide to Reading Health Care News Stories
Schwitzer, G. (2014) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2014.1359 · Cited by 54 · cited 1×
“Public media and related commentators of health news [53] may also help by focusing on the need to obtain clinically useful research and not compromise for less.”
BackgroundValue of Information: A Tool to Improve Research Prioritization and Reduce Waste
Minelli, C., & Baio, G. (2015) · PLoS Medicine · doi:10.1371/journal.pmed.1001882 · Cited by 50 · cited 1×
“Clinical research remains extremely expensive, even though an estimated 90% of the present cost of trials could be safely eliminated [26,27].”
BackgroundΕvangelou, Ε., Siontis, K., Pfeiffer, T., & Ioannidis, J. P. A. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.06.009 · Cited by 42 · cited 1×
“Useful clinical research procures a clinically relevant information gain [9]: it adds to what we already know.”
BackgroundSelf-citationIoannidis, J. P. A. (2013) · JAMA · doi:10.1001/jama.2012.168095 · Cited by 42
Self-citationMagnitude of effects in clinical trials published in high-impact general medical journals
Siontis, K. C., Evangelou, E., & Ioannidis, J. P. (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyr095 · Cited by 38 · cited 1×
“However, the small clinical trials published in major general journals actually have more exaggerated results, on average, than equally small studies published elsewhere [47].”
ResultSelf-citationClinical Trial Registration in Physical Therapy Journals: A Cross-Sectional Study
Babu, A. S., Veluswamy, S. K., Rao, P. T., & Maiya, A. G. (2013) · Physical Therapy · doi:10.2522/ptj.20120531 · Cited by 27 · cited 1×
“, 21% and 29% for trials published in psychological or behavioral [32] and physical therapy [33] journals, respectively).”
BackgroundThe role of private industry in pragmatic comparative effectiveness trials
Buesching, D. P., Luce, B. R., & Berger, M. L. (2012) · Journal of Comparative Effectiveness Research · doi:10.2217/cer.12.9 · Cited by 16 · cited 1×
“Only nine industry-funded pragmatic comparative drug effectiveness trials were published between 1996 and 2010 according to a systematic review of the literature [16], while thousands of efficacy trials have been published that explore optimization of testing circumstances.”
BackgroundISIS and the emergence of large, simple trials
Altman, D. G. (2015) · The Lancet · doi:10.1016/s0140-6736(15)61450-7 · Cited by 3 · cited 1×
“, median sample size 3,116 and 3,104, respectively, for papers published in The Lancet and BMJ in September 2007 [46]).”
Background
Cited by
CARE guidelines for case reports: explanation and elaboration document
Riley, D. S., Barber, M. S., Kienle, G. S., Aronson, J. K., von Schoen-Angerer, T., Tugwell, P., Kiene, H., Helfand, M., Altman, D. G., Sox, H., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.04.026 · Cited by 2,287
“The evidence that hypotheses are not false is associatedwith the prior probability of an association between variablesdfor example, a diagnostic single nucleotide polymorphism test and a disease [64,65].”
BackgroundResearch co-design in health: a rapid overview of reviews
Slattery, P., Saeri, A. K., & Bragge, P. (2020) · Health Research Policy and Systems · doi:10.1186/s12961-020-0528-9 · Cited by 1,313
“The resulting gap between research and enduser needs is underscored by Ioannidis, who points out that “practicing doctors and other health care professionals [are] familiar with how little of what they find in medical journals is useful” ([3], p.”
BackgroundInfluentialMachine Learning Approaches for Clinical Psychology and Psychiatry
Dwyer, D. B., Falkai, P., & Koutsouleris, N. (2018) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-032816-045037 · Cited by 966
Seyhan, A. A. (2019) · Translational Medicine Communications · doi:10.1186/s41231-019-0050-7 · Cited by 713
Lederer, D. J., Bell, S. C., Branson, R. D., Chalmers, J. D., Marshall, R., Maslove, D. M., Ost, D. E., Punjabi, N. M., Schatz, M., Smyth, A. R., et al. (2019) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201808-564ps · Cited by 715
Metadata
- Authors
- Ioannidis, J. P. A.
- Affiliations
- Stanford Medicine (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLOS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-06-21
- Volume
- 13
- Issue
- 6
- Pages
- e1002049
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC4915619
- OpenAlex
- W2469474955
- Cited by
- 655
- References
- 56
- Citations by year
- 2026: 68 · 2025: 55 · 2024: 48 · 2023: 60 · 2022: 55 · 2021: 77 · 2020: 81 · 2019: 59 · 2018: 69 · 2017: 65 · 2016: 18
- FWCI
- 69.75
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Patient-Centered Outcomes Research Institute
- Laura and John Arnold Foundation
- National Institutes of Health
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Health and Medical Research Impacts
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Healthcare cost, quality, practices
- Keywords
- research value, research reform, problem-based research, information gain, pragmatism, patient-oriented research, value for money, feasibility, research transparency
- MeSH terms
- Humans, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.