Why we need observational studies to evaluate the effectiveness of health care
Open accessAbstract
The view is widely held that experimental methods (randomised controlled trials) are the "gold standard" for evaluation and that observational methods (cohort and case control studies) have little or no value. This ignores the limitations of randomised trials, which may prove unnecessary, inappropriate, impossible, or inadequate. Many of the problems of conducting randomised trials could often, in theory, be overcome, but the practical implications for researchers and funding bodies mean that this is often not possible. The false conflict between those who advocate randomised trials in all situations and those who believe observational data provide sufficient evidence needs to be replaced with mutual recognition of the complementary roles of the two approaches. Researchers should be united in their quest for scientific rigour in evaluation, regardless of the method used.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Observational studies have a role in research into the benefits and harms of medical interventions.
“Observational studies also have a role in research into the benefits and harms of medical interventions [2].”
From The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies (2007) · cites this paper as [2]
Many research questions cannot be investigated in randomised controlled trials for ethical or methodological reasons.
“Many research questions cannot be investigated in randomised controlled trials (RCTs) for ethical or methodological reasons [1]”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [1]
Cites
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634 · see citing works
Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)00031-w · Cited by 1,308 · see citing works
Explanatory and Pragmatic Attitudes in Therapeutical Trials
Schwartz, D., & Lellouch, J. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.01.012 · Cited by 1,253 · see citing works
A New Design for Randomized Clinical Trials
Zelen, M. (1979) · New England Journal of Medicine · doi:10.1056/nejm197905313002203 · Cited by 1,057 · see citing works
Ten-year follow-up study of total hip replacement.
STAUFFER, R. N. (1982) · Journal of Bone and Joint Surgery · doi:10.2106/00004623-198264070-00003 · Cited by 640 · see citing works
Wasson, J. H., Reda, D. J., Bruskewitz, R. C., et al. (1995) · New England Journal of Medicine · doi:10.1056/nejm199501123320202 · Cited by 611 · see citing works
Subverting Randomization in Controlled Trials
Schulz, K. F. (1995) · JAMA · doi:10.1001/jama.1995.03530180050029 · Cited by 518 · see citing works
Patient preferences and randomised clinical trials.
Brewin, C. R., & Bradley, C. (1989) · BMJ · doi:10.1136/bmj.299.6694.313 · Cited by 503 · see citing works
Symptom Status and Quality of Life Following Prostatectomy
Fowler, F. J. (1988) · JAMA · doi:10.1001/jama.1988.03720200040030 · Cited by 437 · see citing works
Centralised treatment, entry to trials and survival
Stiller, C. A. (1994) · British Journal of Cancer · doi:10.1038/bjc.1994.306 · Cited by 251 · see citing works
Equipoise and the ethics of randomization.
Lilford, R. J., & Jackson, J. (1995) · PubMed · Cited by 201 · see citing works
Hlatky, M. A., Califf, R. M., Harrell, F. E., Lee, K. L., Mark, D. B., & Pryor, D. B. (1988) · Journal of the American College of Cardiology · doi:10.1016/0735-1097(88)90086-1 · Cited by 180 · see citing works
Maw, R., & BAWDEN, R. (1993) · BMJ · doi:10.1136/bmj.306.6880.756 · Cited by 119 · see citing works
Volume and outcome in coronary artery bypass graft surgery: true association or artefact?
Sowden, A. J., Deeks, J. J., & Sheldon, T. (1995) · BMJ · doi:10.1136/bmj.311.6998.151 · Cited by 108 · see citing works
What do we know about fundholding in general practice?
Dixon, J., & Glennerster, H. (1995) · BMJ · doi:10.1136/bmj.311.7007.727 · Cited by 76 · see citing works
Randomised clinical trials with clinician-preferred treatment
Korn, E. L. (1991) · The Lancet · doi:10.1016/0140-6736(91)90809-4 · Cited by 57 · see citing works
for the British Stomach Cancer Group*, Ward, L., Fielding, J., Dunn, J., & Kelly, K. (1992) · British Journal of Cancer · doi:10.1038/bjc.1992.390 · Cited by 39 · see citing works
Chalmers, I. (1986) · Birth · doi:10.1111/j.1523-536x.1986.tb01037.x · Cited by 38 · see citing works
Volume and Outcome in Hospital Care: Evidence, Explanations and Implications
Black, N. A., & Johnston, A. (1990) · Health Services Management Research · doi:10.1177/095148489000300205 · Cited by 28 · see citing works
Self-citationMaking do without randomised trials
Dorey, F., Grigoris, P., & Amstutz, H. (1994) · Journal of Bone and Joint Surgery - British Volume · doi:10.1302/0301-620x.76b1.8300649 · Cited by 25 · see citing works
The relationship between evaluative research and audit
Black, N. A. (1992) · Journal of Public Health · doi:10.1093/oxfordjournals.pubmed.a042773 · Cited by 20 · see citing works
Self-citationAngioplasty versus bypass surgery
Marchbank, A. J., Pepper, J., & White, H. (1995) · The Lancet · doi:10.1016/s0140-6736(95)91957-0 · Cited by 18 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020 · see citing works
“Observational studies also have a role in research into the benefits and harms of medical interventions [2].”
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61602-x · Cited by 21,320 · see citing works
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329 · see citing works
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00010 · Cited by 17,796 · see citing works
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.008 · Cited by 17,893 · see citing works
Developing and evaluating complex interventions: the new Medical Research Council guidance
Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008) · BMJ · doi:10.1136/bmj.a1655 · Cited by 10,995 · see citing works
“It is important designs preferable (Black, 1996; Glasziou et al., 2007), but”
Elm, E. V., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · BMJ · doi:10.1136/bmj.39335.541782.ad · Cited by 10,766 · see citing works
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2014) · International Journal of Surgery · doi:10.1016/j.ijsu.2014.07.013 · Cited by 9,909 · see citing works
Thematic networks: an analytic tool for qualitative research
Attride-Stirling, J. (2001) · Qualitative Research · doi:10.1177/146879410100100307 · Cited by 5,506 · see citing works
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907 · see citing works
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · Preventive Medicine · doi:10.1016/j.ypmed.2007.08.012 · Cited by 2,698 · see citing works
External validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544 · see citing works
Interpretation of the evidence for the efficacy and safety of statin therapy
Collins, R., Reith, C., Emberson, J., Armitage, J., Baigent, C., Blackwell, L., Blumenthal, R., Danesh, J., Smith, G. D., DeMets, D., et al. (2016) · The Lancet · doi:10.1016/s0140-6736(16)31357-5 · Cited by 1,857 · see citing works
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Bulletin of the World Health Organization · doi:10.2471/blt.07.045120 · Cited by 1,632 · see citing works
Developing and evaluating complex interventions: the new Medical Research Council guidance
Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2013) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2012.09.010 · Cited by 1,658 · see citing works
Zhang, W., Doherty, M., Leeb, B. F., Alekseeva, L., Arden, N. K., Bijlsma, J. W., Dinçer, F., Dziedzic, K., Häuselmann, H. J., Herrero-Beaumont, G., et al. (2007) · Annals of the Rheumatic Diseases · doi:10.1136/ard.2006.062091 · Cited by 1,338 · see citing works
Qualitative research methods in health technology assessment: a review of the literature.
Murphy, Dingwall, Greatbatch, Parker, & Watson (1998) · Health Technology Assessment · doi:10.3310/hta2160 · Cited by 1,107 · see citing works
Evidence-Based Public Health: Moving Beyond Randomized Trials
Victora, C. G., Habicht, J.-P., & Bryce, J. (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.400 · Cited by 1,056 · see citing works
Criteria for evaluating evidence on public health interventions
Rychetnik, L., Frommer, M., Hawe, P., & Shiell, A. (2002) · Journal of Epidemiology and Community Health · doi:10.1136/jech.56.2.119 · Cited by 1,042 · see citing works
<i>Meta-analysis</i> Spurious precision? Meta-analysis of observational studies
Egger, M., Schneider, M., & Smith, G. D. (1998) · BMJ · doi:10.1136/bmj.316.7125.140 · Cited by 997 · see citing works
Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions
Evans, D. (2003) · Journal of Clinical Nursing · doi:10.1046/j.1365-2702.2003.00662.x · Cited by 993 · see citing works
Flynn, M. A. T., McNeil, D. A., Maloff, B., Mutasingwa, D., Wu, M., Ford, C., & Tough, S. C. (2006) · Obesity Reviews · doi:10.1111/j.1467-789x.2006.00242.x · Cited by 925 · see citing works
Evaluating health promotion—progress, problems and solutions
Nutbeam, D. (1998) · Health Promotion International · doi:10.1093/heapro/13.1.27 · Cited by 671 · see citing works
Bias in clinical intervention research.
Gluud, L. L. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj069 · Cited by 636 · see citing works
The double-blind, randomized, placebo-controlled trial
Kaptchuk, T. J. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00347-4 · Cited by 595 · see citing works
Confounding by Indication in Clinical Research
Kyriacou, D. N., & Lewis, R. J. (2016) · JAMA · doi:10.1001/jama.2016.16435 · Cited by 565 · see citing works
Limitations of the Randomized Controlled Trial in Evaluating Population-Based Health Interventions
Sanson-Fisher, R. W., Bonevski, B., Green, L. W., & D’Este, C. (2007) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2007.04.007 · Cited by 514 · see citing works
Study Design, Precision, and Validity in Observational Studies
Carlson, M. D. A., & Morrison, R. S. (2009) · Journal of Palliative Medicine · doi:10.1089/jpm.2008.9690 · Cited by 517 · see citing works
The prevalence and impact of alcohol problems in major depression: A systematic review
Sullivan, L. E., Fiellin, D. A., & O’Connor, P. G. (2005) · The American Journal of Medicine · doi:10.1016/j.amjmed.2005.01.007 · Cited by 489 · see citing works
Randomisation to protect against selection bias in healthcare trials
Odgaard-Jensen, J., Vist, G. E., Timmer, A., Kunz, R., Akl, E. A., Schünemann, H., Briel, M., Nordmann, A. J., Pregno, S., & Oxman, A. D. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000012.pub3 · Cited by 439 · see citing works
Merlin, T., Weston, A., & Tooher, R. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-34 · Cited by 435 · see citing works
Von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2008) · Gaceta Sanitaria · doi:10.1157/13119325 · Cited by 417 · see citing works
Mueller, M., D’Addario, M., Egger, M., Cevallos, M., Dekkers, O., Mugglin, C., & Scott, P. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0495-9 · Cited by 400 · see citing works
Need for expertise based randomised controlled trials
Devereaux, P. J., Bhandari, M., Clarke, M., Montori, V. M., Cook, D. J., Yusuf, S., Sackett, D. L., Cinà, C. S., Walter, S. D., Haynes, B., et al. (2005) · BMJ · doi:10.1136/bmj.330.7482.88 · Cited by 427 · see citing works
Davis, A. C., Voelkel, J. L., Remmers, C. L., Adams, J. L., & McGlynn, E. A. (2023) · The Permanente Journal · doi:10.7812/tpp/22.172 · Cited by 402 · see citing works
The Västerbotten Intervention Programme: background, design and implications
Norberg, M., Wall, S., Boman, K., & Weinehall, L. (2010) · Global Health Action · doi:10.3402/gha.v3i0.4643 · Cited by 373 · see citing works
Natural experiments: an underused tool for public health?
Petticrew, M., Cummins, S., Ferrell, C., Findlay, A., Higgins, C., Hoy, C., Kearns, A., & Sparks, L. (2005) · Public Health · doi:10.1016/j.puhe.2004.11.008 · Cited by 318 · see citing works
MacLehose, Reeves, Harvey, Sheldon, Russell, & Black (2000) · Health Technology Assessment · doi:10.3310/hta4340 · Cited by 311 · see citing works
Foy, R., Eccles, M., Jamtvedt, G., Young, J., Grimshaw, J., & Baker, R. (2005) · BMC Health Services Research · doi:10.1186/1472-6963-5-50 · Cited by 250 · see citing works
“The issues of external validity of randomised controlled trials (and by inference, systematic reviews) have been aired in the context of clinical studies.[15,16].”
BackgroundBonell, C. P., Hargreaves, J., Cousens, S., Ross, D., Hayes, R., Petticrew, M., & Kirkwood, B. R. (2011) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2008.082602 · Cited by 223 · see citing works
Manchikanti, L. (2009) · Pain Physician · doi:10.36076/ppj.2009/12/819 · Cited by 168 · see citing works
Improving the reporting of public health intervention research: advancing TREND and CONSORT
Armstrong, R., Waters, E., Moore, L., Riggs, E., Cuervo, L. G., Lumbiganon, P., & Hawe, P. (2008) · Journal of Public Health · doi:10.1093/pubmed/fdm082 · Cited by 171 · see citing works
Evaluation of Communication Training Programs in Nursing Care: A Review of the Literature
Kruijver, I. P. M., Kerkstra, A., Francke, A. L., Bensing, J. M., & van de Wiel, H. B. M. (2000) · Patient Education and Counseling · doi:10.1016/s0738-3991(99)00096-8 · Cited by 174 · see citing works
The search for evidence of effective health promotion
Speller, V., Learmonth, A., & Harrison, D. (1997) · BMJ · doi:10.1136/bmj.315.7104.361 · Cited by 171 · see citing works
Duggan, C., Parry, G., McMurran, M., Davidson, K., & Dennis, J. (2014) · Trials · doi:10.1186/1745-6215-15-335 · Cited by 165 · see citing works
Pragmatic randomised controlled trials in psychiatry
Hotopf, M., Churchill, R., & Lewis, G. (1999) · British Journal of Psychiatry · doi:10.1192/bjp.175.3.217 · Cited by 165 · see citing works
Differences between clinical trials and postmarketing use
Martin, K., Bégaud, B., Latry, P., Miremont‐Salamé, G., Fourrier, A., & Moore, N. (2004) · British Journal of Clinical Pharmacology · doi:10.1046/j.1365-2125.2003.01953.x · Cited by 157 · see citing works
The value of (pre)school playgrounds for children's physical activity level: a systematic review
Broekhuizen, K., Scholten, A.-M., & de Vries, S. I. (2014) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/1479-5868-11-59 · Cited by 145 · see citing works
Olivares, J. M., Rodriguez-Morales, A., Diels, J., Povey, M., Jacobs, A., Zhao, Z., & Lam, A. (2009) · European Psychiatry · doi:10.1016/j.eurpsy.2008.12.002 · Cited by 141 · see citing works
Hayes, J. F., Marston, L., Walters, K., Geddes, J. R., King, M., & Osborn, D. P. J. (2016) · World Psychiatry · doi:10.1002/wps.20298 · Cited by 115 · see citing works
Pack, A. I., Magalang, U. J., Singh, B., Kuna, S. T., Keenan, B. T., & Maislin, G. (2021) · Sleep · doi:10.1093/sleep/zsaa229 · Cited by 114 · see citing works
Stewart, W. F., Shah, N. R., Selna, M. J., Paulus, R. A., & Walker, J. M. (2007) · Health Affairs · doi:10.1377/hlthaff.26.2.w181 · Cited by 111 · see citing works
A proposed method of bias adjustment for meta-analyses of published observational studies
Thompson, S., Ekelund, U., Jebb, S., Lindroos, A. K., Mander, A., Sharp, S., Turner, R., & Wilks, D. (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyq248 · Cited by 101 · see citing works
Unbiased, relevant, and reliable assessments in health care
Chalmers, I. (1998) · BMJ · doi:10.1136/bmj.317.7167.1167 · Cited by 98 · see citing works
A systematic review of the evidence for telemedicine in burn care: With a UK perspective
Wallace, D. L., Hussain, A., Khan, N., & Wilson, Y. T. (2012) · Burns · doi:10.1016/j.burns.2011.09.024 · Cited by 75 · see citing works
Drukker, M., Weltens, I., van Hooijdonk, C. F. M., Vandenberk, E., & Bak, M. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.675071 · Cited by 75 · see citing works
Zhuo, Z., Wang, A., & Yu, H. (2014) · Journal of Diabetes Research · doi:10.1155/2014/381231 · Cited by 55 · see citing works
Performing an Informatics Consult: Methods and Challenges
Schuler, A., Callahan, A., Jung, K., & Shah, N. H. (2018) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2017.12.023 · Cited by 37 · see citing works
Using theory-based evaluation to build evidence-based health and social care policy and practice
Bonner, L. (2003) · Critical Public Health · doi:10.1080/0958159031000100224 · Cited by 33 · see citing works
Moore, L., Graham, A., & Diamond, I. (2003) · British Educational Research Journal · doi:10.1080/0141192032000133712 · Cited by 32 · see citing works
Generalized synthesis of evidence and the threat of dissemination bias
Sutton, A. J., Abrams, K. R., & Jones, D. R. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(02)00460-2 · Cited by 27 · see citing works
Using Aggregate Patient Data at the Bedside via an On-Demand Consultation Service
Callahan, A., Gombar, S., Cahan, E. M., Jung, K., Steinberg, E., Polony, V., Morse, K., Tibshirani, R., Hastie, T., Harrington, R., et al. (2021) · NEJM Catalyst · doi:10.1056/cat.21.0224 · Cited by 24 · see citing works
Dudley, M., Hadzi-Pavlovic, D., Andrews, D., & Perich, T. (2008) · Australian & New Zealand Journal of Psychiatry · doi:10.1080/00048670802050538 · Cited by 15 · see citing works
(2006) · Systematic Reviews in the Social Sciences · doi:10.1002/9780470754887.ch6 · Cited by 12 · see citing works
Untitled work
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · PLoS Medicine
- Authors
- Black, N.
- Affiliations
- London School of Hygiene & Tropical Medicine (GB)
- Corresponding author
- Black, Nick
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 1996-05-11
- Volume
- 312
- Issue
- 7040
- Pages
- 1215-1218
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- PMC
- PMC2350940
- OpenAlex
- W2088438184
- Cited by
- 1,598
- References
- 22
- Citations by year
- 2026: 10 · 2025: 27 · 2024: 41 · 2023: 46 · 2022: 48 · 2021: 41 · 2020: 55 · 2019: 49 · 2018: 60 · 2017: 41 · 2016: 78 · 2015: 57 · 2014: 73 · 2013: 73 · 2012: 69
- FWCI
- 49.85
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- www.bmj.com/company/legal-information/terms-conditions/legal-information/tdm-licencepolicy (tdm)
- Locations
- BMJ (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- DigitalGeorgetown (Georgetown University Library) (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- observational studies, randomized controlled trials, health service evaluation, effectiveness evaluation, cohort studies, case-control study, evidence-based assessment, research methodology
- MeSH terms
- Data Collection, United Kingdom, Health Services Research, Humans, Politics, Research, Randomized Controlled Trials as Topic, Liability, Legal, Treatment Outcome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
