Protect us from poor-quality medical research
Open accessThe DOI matches its Crossref record; 47 of the 48 references it cites were found in a registry or an index.
Abstract
Much of the published medical research is apparently flawed, cannot be replicated and/or has limited or no utility. This article presents an overview of the current landscape of biomedical research, identifies problems associated with common study designs and considers potential solutions. Randomized clinical trials, observational studies, systematic reviews and meta-analyses are discussed in terms of their inherent limitations and potential ways of improving their conduct, analysis and reporting. The current emphasis on statistical significance needs to be replaced by sound design, transparency and willingness to share data with a clear commitment towards improving the quality and utility of clinical research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationChan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004) · JAMA · doi:10.1001/jama.291.20.2457 · Cited by 1,845
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Self-citationReducing waste from incomplete or unusable reports of biomedical research
Glasziou, P., Altman, D. G., Bossuyt, P., Boutron, I., Clarke, M., Julious, S., Michie, S., Moher, D., & Wager, E. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62228-x · Cited by 1,319
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293
Self-citationSavović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049
Self-citationBiomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Self-citationThe scandal of poor medical research.
Altman, D. G. (1994) · BMJ · doi:10.1136/bmj.308.6924.283 · Cited by 897
How to Make More Published Research True
Ioannidis, J. P. A. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001747 · Cited by 760
Self-citationSavović, J., Jones, H., Altman, D., Harris, R., Jűni, P., Pildal, J., Als-Nielsen, B., Balk, E., Gluud, C., Gluud, L., et al. (2012) · Health Technology Assessment · doi:10.3310/hta16350 · Cited by 668
Self-citationWhy Most Clinical Research Is Not Useful
Ioannidis, J. P. A. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1002049 · Cited by 655
Self-citationSimmonds, M. C., Higginsa, J. P. T., Stewartb, L. A., Tierneyb, J. F., Clarke, M. J., & Thompson, S. G. (2005) · Clinical Trials · doi:10.1191/1740774505cn087oa · Cited by 555
Anglemyer, A., Horvath, H. T., & Bero, L. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000034.pub2 · Cited by 534
Increasing value and reducing waste in biomedical research: who's listening?
Moher, D., Glasziou, P., Chalmers, I., Nasser, M., Bossuyt, P. M. M., Korevaar, D. A., Graham, I. D., Ravaud, P., & Boutron, I. (2016) · The Lancet · doi:10.1016/s0140-6736(15)00307-4 · Cited by 462
Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
Chan, A.-W. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1041086 · Cited by 444
Khoury, M. J., & Ioannidis, J. P. A. (2014) · Science · doi:10.1126/science.aaa2709 · Cited by 387
Self-citationEvolution of Reporting P Values in the Biomedical Literature, 1990-2015
Chavalarias, D., Wallach, J. D., Li, A. H. T., & Ioannidis, J. P. A. (2016) · JAMA · doi:10.1001/jama.2016.1952 · Cited by 370
Self-citationEvidence-based medicine has been hijacked: a report to David Sackett
Ioannidis, J. P. A. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.02.012 · Cited by 282
Self-citationSpironolactone Metabolites in TOPCAT — New Insights into Regional Variation
de Denus, S., O’Meara, E., Desai, A. S., Claggett, B., Lewis, E. F., Leclair, G., Jutras, M., Lavoie, J., Solomon, S. D., Pitt, B., et al. (2017) · New England Journal of Medicine · doi:10.1056/nejmc1612601 · Cited by 275
Patel, C. J., Burford, B., & Ioannidis, J. P. A. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.05.029 · Cited by 274
Self-citationEmpirical Evaluation of Very Large Treatment Effects of Medical Interventions
Pereira, T. V., Horwitz, R. I., & Ioannidis, J. P. A. (2012) · JAMA · doi:10.1001/jama.2012.13444 · Cited by 257
Self-citationHemkens, L. G., Contopoulos-Ioannidis, D. G., & Ioannidis, J. P. A. (2016) · BMJ · doi:10.1136/bmj.i493 · Cited by 220
Self-citationvan Wely, M., Kwan, I., Burt, A. L., Thomas, J., Vail, A., Van der Veen, F., & Al-Inany, H. G. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd005354.pub2 · Cited by 210
Mastering variation: Variance components and personalised medicine
Senn, S. (2016) · Statistics in Medicine · doi:10.1002/sim.6739 · Cited by 205
Ensuring the integrity of clinical practice guidelines: a tool for protecting patients
Lenzer, J., Hoffman, J. R., Furberg, C. D., & Ioannidis, J. P. A. (2013) · BMJ · doi:10.1136/bmj.f5535 · Cited by 187
Self-citationFlacco, M. E., Manzoli, L., Boccia, S., Capasso, L., Aleksovska, K., Rosso, A., Scaioli, G., De Vito, C., Siliquini, R., Villari, P., et al. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.12.016 · Cited by 177
Self-citationHow to survive the medical misinformation mess
Ioannidis, J. P. A., Stuart, M. E., Brownlee, S., & Strite, S. A. (2017) · European Journal of Clinical Investigation · doi:10.1111/eci.12834 · Cited by 133
Self-citationEthics and Epistemology in Big Data Research
Lipworth, W. L., Mason, P. H., Kerridge, I., & Ioannidis, J. P. A. (2017) · Journal of Bioethical Inquiry · doi:10.1007/s11673-017-9771-3 · Cited by 117
Self-citationHow to reduce unnecessary duplication: use PROSPERO
Moher, D., Booth, A., & Stewart, L. (2014) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.12657 · Cited by 113
The CROWN Initiative: journal editors invite researchers to develop core outcomes in women's health
Khan, K. (2014) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.12929 · Cited by 102
Panagiotou, O. A., Contopoulos-Ioannidis, D. G., & Ioannidis, J. P. A. (2013) · BMJ · doi:10.1136/bmj.f707 · Cited by 89
Self-citationHolmes, J. H., Sun, J., & Peek, N. (2014) · Yearbook of Medical Informatics · doi:10.15265/iy-2014-0018 · Cited by 78
Tricco, A. C., Cogo, E., Page, M. J., Polisena, J., Booth, A., Dwan, K. M., MacDonald, H. R., Clifford, T., Stewart, L. A., Straus, S. E., et al. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.03.025 · Cited by 75
Considerations for Observational Research Using Large Data Sets in Radiation Oncology
Jagsi, R., Bekelman, J. E., Chen, A., Chen, R. C., Hoffman, K., Tina Shih, Y.-C., Smith, B. D., & Yu, J. B. (2014) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2014.05.013 · Cited by 73
Overlapping network meta-analyses on the same topic: survey of published studies
Naudet, F., Schuit, E., & Ioannidis, J. P. A. (2017) · International Journal of Epidemiology · doi:10.1093/ije/dyx138 · Cited by 68
Self-citationRoutinely collected data as a strategic resource for research: priorities for methods and workforce
Jorm, L. R. (2015) · Public Health Research & Practice · doi:10.17061/phrp2541540 · Cited by 64
Farland, L. V., Correia, K. F. B., Wise, L. A., Williams, P. L., Ginsburg, E. S., & Missmer, S. A. (2016) · Human Reproduction · doi:10.1093/humrep/dew192 · Cited by 59
Data Resource Profile: The Aberdeen Maternity and Neonatal Databank (AMND)
Ayorinde, A. A., Wilde, K., Lemon, J., Campbell, D., & Bhattacharya, S. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyv356 · Cited by 50
Self-citationLea, N., Nicholls, J., Dobbs, C., Sethi, N., Cunningham, J. A., Ainsworth, J. D., Heaven, M. L., Peacock, T., Peacock, A. J., Jones, K. H., et al. (2016) · JMIR Medical Informatics · doi:10.2196/medinform.5571 · Cited by 47
Meta-analyses Can Be Credible and Useful: A New Standard
Ioannidis, J. P. A. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0035 · Cited by 36
Self-citationPrior, M., Hibberd, R., Asemota, N., & Thornton, J. (2017) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.14506 · Cited by 36
Can Observational Data Replace Randomized Trials?
McGale, P., Cutter, D., Darby, S. C., Henson, K. E., Jagsi, R., & Taylor, C. W. (2016) · Journal of Clinical Oncology · doi:10.1200/jco.2016.68.8879 · Cited by 32
Hemkens, L. G., Contopoulos‐Ioannidis, D. G., & Ioannidis, J. P. A. (2016) · CMAJ Open · doi:10.9778/cmajo.20150036 · Cited by 29
Self-citationBraakhekke, M., Mol, F., Mastenbroek, S., Mol, B. W., & van der Veen, F. (2016) · Human Reproduction · doi:10.1093/humrep/dew286 · Cited by 27
Network meta-analyses performed by contracting companies and commissioned by industry
Schuit, E., & Ioannidis, J. P. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0377-3 · Cited by 24
Self-citationThe CROWN Initiative: journal editors invite researchers to develop core outcomes in women's health
Initiative, T. C. O. I. W. H. (. (2014) · Human Reproduction · doi:10.1093/humrep/deu150 · Cited by 9
The Texas sharpshooter fallacy
Evers, J. L. H. (2017) · Human Reproduction · doi:10.1093/humrep/dex103 · Cited by 7
Self-citation
Research works printed but not linked: 1
- Clinical Practice Guidelines We Can Trust, 2011, Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines
Cited by
Preventive psychiatry: a blueprint for improving the mental health of young people
Fusar‐Poli, P., Correll, C. U., Arango, C., Berk, M., Patel, V., & Ioannidis, J. P. A. (2021) · World Psychiatry · doi:10.1002/wps.20869 · Cited by 497
Self-citationGuidance to best tools and practices for systematic reviews
Kolaski, K., Logan, L. R., & Ioannidis, J. P. A. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02255-9 · Cited by 236
“Once a research question is established, searching on registry sites and databases for existing systematic reviews addressing the same or a similar topic is necessary in order to avoid contributing to research waste [131].”
Self-citationWang, R., Li, W., Bordewijk, E. M., Legro, R. S., Zhang, H., Wu, X., Gao, J., Morin-Papunen, L., Homburg, R., König, T. E., et al. (2019) · Human Reproduction Update · doi:10.1093/humupd/dmz029 · Cited by 137
Self-citationKolaski, K., Logan, L. R., Goss, K. D., & Butler, C. (2021) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.14949 · Cited by 47
Rajan, S., Campagnolo, M., Callaghan, B. C., & Gibbons, C. H. (2018) · Clinical Autonomic Research · doi:10.1007/s10286-018-0540-0 · Cited by 47
Metadata
- Authors
- ESHRE Capri Workshop Group, Ioannidis, J. P. A., Bhattacharya, S., Evers, J. L. H., der Veen, F. V., Somigliana, E., Barratt, C. L. R., Bontempi, G., Baird, D. T., Crosignani, P., Devroey, P., Diedrich, K., Farquharson, R. G., Fraser, L. R., Geraedts, J. P. M., Gianaroli, L., La Vecchia, C., Magli, C., Negri, E., Sunde, A., Tapanainen, J. S., Tarlatzis, B. C., Steirteghem, A. V., & Veiga, A.
- Affiliations
- Stanford University (US)
- University of Aberdeen (GB)
- Maastricht University Medical Centre (NL)
- Academic Medical Centre, University of Amsterdam, Reproduction Medicine, Amsterdam, The Netherlands
- University of Dundee (GB)
- Université Libre de Bruxelles (BE)
- Fondazione IRCCS Ca' Granda Ospedale Maggiore Policlinico (IT)
- Vrije Universiteit Brussel (BE)
- Liverpool Women's Hospital (GB)
- King's College London (GB)
- University of Milan (IT)
- Department of Clinical Sciences and Community Health, Università degli Studi di Milano, Milan, Italy
- Reproductive Medicine, Sahlgrenska University Hospital, Gothenburg, Sweden
- University of Helsinki (FI)
- Centre for Reproductive Medicine, Universitair Ziekenhuis Vrije Universiteit Brussel, Belgium
- Reproductive Medicine Service, Dexeus Woemn’s Health, Barcelona, Spain
- ORCID
- Venue
- Human Reproduction
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-03-29
- Volume
- 33
- Issue
- 5
- Pages
- 770-776
- ISSN
- 0268-1161 (ISSN-L)
- 1460-2350
- OpenAlex
- W2794626044
- Cited by
- 59
- References
- 48
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 2 · 2023: 10 · 2022: 6 · 2021: 5 · 2020: 10 · 2019: 6 · 2018: 7
- FWCI
- 5.34
- Type
- journal article
- Language
- en
- Open access
- Licence
- Locations
- Human Reproduction (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- Discovery Research Portal (University of Dundee) (publishedVersion, other-oa)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion)
- Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) (submittedVersion)
- ORCA Online Research @Cardiff (Cardiff University) (submittedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Funders
- European Society of Human Reproduction and Embryology
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Nursing › Research and Theory › Methodologies in Health Research and Practice
- Keywords
- research quality, reproducibility, randomized controlled trials, observational studies, systematic reviews, meta-analysis, statistical significance, research design, transparency, data sharing, clinical research quality
- MeSH terms
- Humans, Research Design, Information Dissemination, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.