The scandal of poor medical research.
Open accessAbstract
We need less research, better research, and research done for the right reasons What should we think about a doctor who uses the wrong treatment, either wilfully or through ignorance, or who uses the right treatment wrongly (such as by giving the wrong dose of a drug)? Most people would agree that such behaviour was unprofessional, arguably unethical, and certainly unacceptable. What, then, should we think about researchers who use the wrong techniques (either wilfully or in ignorance), use the right techniques wrongly, misinterpret their results, report their results selectively, cite the literature selectively, and draw unjustified conclusions? We should be appalled. Yet numerous studies of the medical literature, in both general and specialist journals, have shown that all of the above phenomena are common.1 2 3 4 5 6 7 This is surely a scandal. When I tell friends outside medicine that many papers published in medical journals are misleading because of methodological weaknesses they are rightly shocked. Huge sums of money are spent annually on research that is seriously flawed through the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods …
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Modifications to trial procedures can introduce biases, raising doubts about the validity of the conclusion of a clinical trial.
“Modifications can also introduce biases, raising doubts about the validity of the conclusion of a clinical trial.”
Cites
Packer, M., Gheorghiade, M., Young, J. B., Costantini, P. J., Adams, K. F., Cody, R. J., Smith, L. K., Voorhees, L. V., Gourley, L. A., & Jolly, M. (1993) · New England Journal of Medicine · doi:10.1056/nejm199307013290101 · Cited by 826 · see citing works
Statistical Problems in the Reporting of Clinical Trials
Pocock, S., Hughes, M. D., & Lee, R. J. (1987) · New England Journal of Medicine · doi:10.1056/nejm198708133170706 · Cited by 660 · see citing works
Gøtzsche, P. C. (1989) · Controlled Clinical Trials · doi:10.1016/0197-2456(89)90017-2 · Cited by 339 · see citing works
Statistics in medical journals
Altman, D. G. (1982) · Statistics in Medicine · doi:10.1002/sim.4780010109 · Cited by 227 · see citing works
Self-citationStatistics and ethics in medical research. Misuse of statistics is unethical.
Altman, D. G. (1980) · BMJ · doi:10.1136/bmj.281.6249.1182 · Cited by 132 · see citing works
Self-citationImpact of multiple comparisons in randomized clinical trials
Smith, D. G., Clemens, J. D., Credè, W. B., Harvey, M., & Gracely, E. J. (1987) · The American Journal of Medicine · doi:10.1016/0002-9343(87)90768-6 · Cited by 88 · see citing works
Reflections on the Decline of Science in England (1830)
Babbage, C. (2024) · Routledge eBooks · doi:10.4324/9781003548249-13 · Cited by 76 · see citing works
Inadequate size of 'negative' clinical trials in dermatology
Williams, H. C., & Seed, P. T. (1993) · British Journal of Dermatology · doi:10.1111/j.1365-2133.1993.tb00178.x · Cited by 67 · see citing works
The task of a statistical referee
Murray, G. D. (1988) · British journal of surgery · doi:10.1002/bjs.1800750714 · Cited by 64 · see citing works
Methodological errors in Medical Research
Huxtable, R. J. (1991) · Trends in Pharmacological Sciences · doi:10.1016/0165-6147(91)90504-l · Cited by 39 · see citing works
Screening for congenital hypothyroidism
Fisher, D. A. (1991) · Trends in Endocrinology and Metabolism · doi:10.1016/1043-2760(91)90002-5 · Cited by 36 · see citing works
Methodological Errors in Medical Research. An Incomplete Catalogue
(1991) · Annals of Internal Medicine · doi:10.7326/0003-4819-114-7-608_1 · Cited by 25 · see citing works
Bailar's laws of data analysis
Bailar, J. C. (1976) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt1976201113 · Cited by 19 · see citing works
Statisticians in Medical Schools
Bland, J. M., Altman, D. G., & Royston, J. P. (1990) · Journal of the Royal College of Physicians of London · doi:10.1016/s0035-8819(25)02119-1 · Cited by 12 · see citing works
Self-citationTherapeutics and government: 1776 and 1976
Abrams, W. B. (1976) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt19762011 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · see citing works
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883 · see citing works
“[10-15]”
Self-citationMoher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373 · see citing works
Self-citationThe Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P. C., Lang, T., & for the CONSORT Group (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00012 · Cited by 3,835 · see citing works
Self-citationImproving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127 · see citing works
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755 · see citing works
Self-citationMoher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545 · see citing works
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2012) · International Journal of Surgery · doi:10.1016/j.ijsu.2011.10.001 · Cited by 2,345 · see citing works
Self-citationMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.03.004 · Cited by 1,858 · see citing works
Self-citationIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597 · see citing works
How to increase value and reduce waste when research priorities are set
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Howells, D. W., Ioannidis, J. P. A., & Oliver, S. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62229-1 · Cited by 1,436 · see citing works
Moher, D., Schulz, K. F., Altman, D. G., & for the CONSORT Group* (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00011 · Cited by 1,426 · see citing works
Self-citationProgress in evidence-based medicine: a quarter century on
Djulbegovic, B., & Guyatt, G. H. (2017) · The Lancet · doi:10.1016/s0140-6736(16)31592-6 · Cited by 1,022 · see citing works
The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews
Turner, R. M., Bird, S. M., & Higgins, J. P. T. (2013) · PLoS ONE · doi:10.1371/journal.pone.0059202 · Cited by 672 · see citing works
The problem of pseudoreplication in neuroscientific studies: is it affecting your analysis?
Lazic, S. E. (2010) · BMC Neuroscience · doi:10.1186/1471-2202-11-5 · Cited by 381 · see citing works
Orkin, A. M., Gill, P. J., Ghersi, D., Campbell, L., Sugarman, J., Emsley, R., Steg, P. G., Weijer, C., Simes, J., Rombey, T., et al. (2021) · JAMA · doi:10.1001/jama.2021.9941 · Cited by 299 · see citing works
Cobo, E., Cortes, J., Ribera, J. M., Cardellach, F., Selva-O'Callaghan, A., Kostov, B., Garcia, L., Cirugeda, L., Altman, D. G., Gonzalez, J. A., et al. (2011) · BMJ · doi:10.1136/bmj.d6783 · Cited by 290 · see citing works
Self-citationFalse Alarms and Pseudo-Epidemics: The Limitations of Observational Epidemiology
Grimes, D. A., & Schulz, K. F. (2012) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e31826af61a · Cited by 254 · see citing works
What errors do peer reviewers detect, and does training improve their ability to detect them?
Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., & Smith, R. (2008) · Journal of the Royal Society of Medicine · doi:10.1258/jrsm.2008.080062 · Cited by 226 · see citing works
Are Peer Reviewers Encouraged to Use Reporting Guidelines? A Survey of 116 Health Research Journals
Hirst, A., & Altman, D. G. (2012) · PLoS ONE · doi:10.1371/journal.pone.0035621 · Cited by 200 · see citing works
Self-citationMoher, D., Schulz, K. F., & Altman, D. G. (2001) · Journal of the American Podiatric Medical Association · doi:10.7547/87507315-91-8-437 · Cited by 193 · see citing works
Self-citationIncongruence between test statistics and P values in medical papers
García-Berthou, E., & Alcaraz, C. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-13 · Cited by 181 · see citing works
Methodology over metrics: current scientific standards are a disservice to patients and society
Van Calster, B., Wynants, L., Riley, R. D., van Smeden, M., & Collins, G. S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.018 · Cited by 105 · see citing works
Sources of error in the retracted scientific literature
Casadevall, A., Steen, R. G., & Fang, F. C. (2014) · The FASEB Journal · doi:10.1096/fj.14-256735 · Cited by 89 · see citing works
Marshall University, Gress, T. W., Denvir, J., Marshall University, Shapiro, J. I., & Marshall University (2018) · Marshall Journal of Medicine · doi:10.18590/mjm.2018.vol4.iss2.9 · Cited by 60 · see citing works
Protect us from poor-quality medical research
ESHRE Capri Workshop Group, Ioannidis, J. P. A., Bhattacharya, S., Evers, J. L. H., der Veen, F. V., Somigliana, E., Barratt, C. L. R., Bontempi, G., Baird, D. T., Crosignani, P., et al. (2018) · Human Reproduction · doi:10.1093/humrep/dey056 · Cited by 59 · see citing works
Evidence synthesis for tackling research waste
Grainger, M. J., Bolam, F. C., Stewart, G. B., & Nilsen, E. B. (2020) · Nature Ecology & Evolution · doi:10.1038/s41559-020-1141-6 · Cited by 59 · see citing works
Loss, C. M., Melleu, F. F., Domingues, K., Lino-de-Oliveira, C., & Viola, G. G. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.763428 · Cited by 44 · see citing works
“Case-Control” Confusion: Mislabeled Reports in Obstetrics and Gynecology Journals
Grimes, D. A. (2009) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e3181c03421 · Cited by 33 · see citing works
Hanin, L. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0399-0 · Cited by 32 · see citing works
Evidence appraisal: a scoping review, conceptual framework, and research agenda
Goldstein, A., Venker, E., & Weng, C. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx050 · Cited by 30 · see citing works
Increase value and reduce waste in research on psychological therapies
Cristea, I. A., & Naudet, F. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103479 · Cited by 22 · see citing works
Puljak, L., Riva, N., Parmelli, E., González-Lorenzo, M., Moja, L., & Pieper, D. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.09.003 · Cited by 16 · see citing works
Research transparency in dental research: A programmatic analysis
Raittio, E., Sofi‐Mahmudi, A., & Uribe, S. E. (2023) · European Journal of Oral Sciences · doi:10.1111/eos.12908 · Cited by 15 · see citing works
Bullock, G. S., Ward, P., Impellizzeri, F. M., Kluzek, S., Hughes, T., Hillman, C., Waterman, B. R., Danelson, K., Henry, K., Barr, E., et al. (2023) · Journal of Orthopaedic & Sports Physical Therapy · doi:10.2519/jospt.2023.12016 · Cited by 15 · see citing works
Elagami, R. A., Reis, T. M., Hassan, M. A., Tedesco, T. K., Braga, M. M., Mendes, F. M., Cenci, M. S., Huysmans, M.-C., & Raggio, D. P. (2024) · BMC Oral Health · doi:10.1186/s12903-024-04417-0 · Cited by 14 · see citing works
GPT for RCTs?: Using AI to determine adherence to reporting guidelines
Wrightson, J. G., Blazey, P., Moher, D., Khan, K. M., & Ardern, C. L. (2023) · doi:10.1101/2023.12.14.23299971 · Cited by 24 · see citing works
“Poor reporting of clinical trials negatively impacts care and is unethical [3,4,28,29].”
BackgroundScientific integrity and the market for lemons
Cottrell, R. C. (2014) · Research Ethics · doi:10.1177/1747016113494651 · Cited by 6 · see citing works
- Authors
- Altman, D. G.
- Corresponding author
- Altman, D G
- ORCID
- Altman, D G
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 1994-01-29
- Volume
- 308
- Issue
- 6924
- Pages
- 283-284
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- PMC
- PMC2539276
- OpenAlex
- W2056590595
- Cited by
- 897
- References
- 15
- Citations by year
- 2026: 41 · 2025: 59 · 2024: 57 · 2023: 48 · 2022: 70 · 2021: 48 · 2020: 45 · 2019: 60 · 2018: 20 · 2017: 21 · 2016: 28 · 2015: 27 · 2014: 36 · 2013: 24 · 2012: 15
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- www.bmj.com/company/legal-information/terms-conditions/legal-information/tdm-licencepolicy (tdm)
- Locations
- BMJ (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- DigitalGeorgetown (Georgetown University Library) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- research quality, methodological weaknesses, selective reporting, misinterpretation of results, research ethics, publication bias, study design flaws, small sample size
- MeSH terms
- Decision Making, Ethics, Medical, Publishing, Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review