Do observational studies using propensity score methods agree with randomized trials? a systematic comparison of studies on acute coronary syndromes
Open accessAbstract
AIMS: Randomized controlled trials (RCTs) are the gold standard for assessing the efficacy of therapeutic interventions because randomization protects from biases inherent in observational studies. Propensity score (PS) methods, proposed as a potential solution to confounding of the treatment-outcome association, are widely used in observational studies of therapeutic interventions for acute coronary syndromes (ACS). We aimed to systematically assess agreement between observational studies using PS methods and RCTs on therapeutic interventions for ACS. METHODS AND RESULTS: We searched for observational studies of interventions for ACS that used PS methods to estimate treatment effects on short- or long-term mortality. Using a standardized algorithm, we matched observational studies to RCTs based on patients' characteristics, interventions, and outcomes ('topics'), and we compared estimates of treatment effect between the two designs. When multiple observational studies or RCTs were identified for the same topic, we performed a meta-analysis and used the summary relative risk for comparisons. We matched 21 observational studies investigating 17 distinct clinical topics to 63 RCTs (median = 3 RCTs per observational study) for short-term (7 topics) and long-term (10 topics) mortality. Estimates from PS analyses differed statistically significantly from randomized evidence in two instances; however, observational studies reported more extreme beneficial treatment effects compared with RCTs in 13 of 17 instances (P = 0.049). Sensitivity analyses limited to large RCTs, and using alternative meta-analysis models yielded similar results. CONCLUSION: For the treatment of ACS, observational studies using PS methods produce treatment effect estimates that are of more extreme magnitude compared with those from RCTs, although the differences are rarely statistically significant.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Such methods often yield results similar to those in controlled treatment trials.
“Such methods often yield results similar to those in controlled treatment trials (133, 134).”
From Using Administrative Data to Predict Suicide After Psychiatric Hospitalization in the Veterans Health Administration System (2020) · cites this paper as [133]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Meta-analysis in clinical trials
DerSimonian, R., & Laird, N. (1986) · Controlled Clinical Trials · doi:10.1016/0197-2456(86)90046-2 · Cited by 39,552 · see citing works
The central role of the propensity score in observational studies for causal effects
ROSENBAUM, P. R., & RUBIN, D. B. (1983) · Biometrika · doi:10.1093/biomet/70.1.41 · Cited by 31,241 · see citing works
Heart Disease and Stroke Statistics—2023 Update: A Report From the American Heart Association
Tsao, C. W., Aday, A. W., Almarzooq, Z. I., Anderson, C. A. M., Arora, P., Avery, C. L., Baker-Smith, C. M., Beaton, A. Z., Boehme, A. K., Buxton, A. E., et al. (2023) · Circulation · doi:10.1161/cir.0000000000001123 · Cited by 8,300 · see citing works
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709 · see citing works
Heart Disease and Stroke Statistics—2011 Update: A Report From the American Heart Association
Roger, V. L., Go, A. S., Lloyd-Jones, D. M., Adams, R. J., Berry, J. D., Brown, T. M., Carnethon, M. R., Dai, S., de Simone, G., Ford, E. S., et al. (2011) · Circulation · doi:10.1161/cir.0b013e3182009701 · Cited by 6,149 · see citing works
D, E. B. M., Antman, E. M., Beasley, J. W., Califf, R. M., Cheitlin, M. D., Hochman, J. S., Jones, R. H., Kereiakes, D. J., Kupersmith, J., Levin, T. N., et al. (2000) · Circulation · doi:10.1161/01.cir.102.10.1193 · Cited by 4,474 · see citing works
Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
Concato, J., Shah, N., & Horwitz, R. I. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422507 · Cited by 3,563 · see citing works
A comparison of observational studies and randomized, controlled trials.
Benson, K., & Hartz, A. J. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422506 · Cited by 2,347 · see citing works
What is meant by intention to treat analysis? Survey of published randomised controlled trials
Hollis, S., & Campbell, F. (1999) · BMJ · doi:10.1136/bmj.319.7211.670 · Cited by 2,160 · see citing works
A critical appraisal of propensity‐score matching in the medical literature between 1996 and 2003
Austin, P. C. (2007) · Statistics in Medicine · doi:10.1002/sim.3150 · Cited by 1,249 · see citing works
Schneeweiss, S., Rassen, J. A., Glynn, R. J., Avorn, J., Mogun, H., & Brookhart, M. A. (2009) · Epidemiology · doi:10.1097/ede.0b013e3181a663cc · Cited by 1,128 · see citing works
Self-citationJneid, H. M., Anderson, J. L., Wright, R. S., Adams, C. D., Bridges, C. R., Casey, D. E., Ettinger, S. M., Fesmire, F. M., Ganiats, T. G., Lincoff, A. M., et al. (2012) · Circulation · doi:10.1161/cir.0b013e318256f1e0 · Cited by 1,090 · see citing works
Rubin, D. B. (2007) · Statistics in Medicine · doi:10.1002/sim.2739 · Cited by 1,047 · see citing works
Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies
Ioannidis, J. P. A. (2001) · JAMA · doi:10.1001/jama.286.7.821 · Cited by 919 · see citing works
Cepeda, M. S. (2003) · American Journal of Epidemiology · doi:10.1093/aje/kwg115 · Cited by 859 · see citing works
Combining Propensity Score Matching with Additional Adjustments for Prognostic Covariates
Rubin, D., & Thomas, N. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10474233 · Cited by 796 · see citing works
Analysis of Observational Studies in the Presence of Treatment Selection Bias
Stukel, T. A., Fisher, E. S., Wennberg, D. E., Alter, D. A., Gottlieb, D. J., & Vermeulen, M. J. (2007) · JAMA · doi:10.1001/jama.297.3.278 · Cited by 789 · see citing works
Systematic review of treatments for atopic eczema.
Hoare, Po, A. L. W., & Williams (2000) · Health Technology Assessment · doi:10.3310/hta4370 · Cited by 789 · see citing works
Stürmer, T., Joshi, M., Glynn, R., Avorn, J., Rothman, K., & Schneeweiss, S. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2005.07.004 · Cited by 729 · see citing works
Little, R. J., & Rubin, D. B. (2000) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.21.1.121 · Cited by 721 · cited 1× · see citing works
“Analyses using PS methods attempt to emulate randomized comparisons because they allow contrasts between patient groups that are on average similar on all observed confounders; however, PS methods cannot adjust for unmeasured confounding.(3,12,13) The literature on ACS offers a unique opportunity for assessing agreeme…”
MethodWhy do we need some large, simple randomized trials?
Yusuf, S., Collins, R., & Peto, R. (1984) · Statistics in Medicine · doi:10.1002/sim.4780030421 · Cited by 700 · see citing works
Holm, J., Nowicka, P., Farpour‐Lambert, N. J., O’Malley, G., Hassapidou, M. N., Weiss, R., & Baker, J. L. (2014) · Obesity Facts · doi:10.1159/000365773 · Cited by 596 · see citing works
Randomized Trials or Observational Tribulations?
Pocock, S. J., & Elbourne, D. R. (2000) · New England Journal of Medicine · doi:10.1056/nejm200006223422511 · Cited by 550 · see citing works
Combining Propensity Score Matching with Additional Adjustments for Prognostic Covariates
Thomas, N. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511810725.025 · Cited by 508 · see citing works
Shah, B. R., Laupacis, A., Hux, J. E., & Austin, P. C. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.016 · Cited by 493 · see citing works
An overview of the objectives of and the approaches to propensity score analyses
Heinze, G., & Jüni, P. (2011) · European Heart Journal · doi:10.1093/eurheartj/ehr031 · Cited by 379 · see citing works
2011 Writing Group Members, Wright, R. S., Anderson, J. L., Adams, C. D., Bridges, C. R., Casey, D. E., Ettinger, S. M., Fesmire, F. M., Ganiats, T. G., Jneid, H. M., et al. (2011) · Circulation · doi:10.1161/cir.0b013e31820f2f3e · Cited by 356 · see citing works
MacLehose, Reeves, Harvey, Sheldon, Russell, & Black (2000) · Health Technology Assessment · doi:10.3310/hta4340 · Cited by 311 · see citing works
Austin, P. C., Grootendorst, P. V., Normand, S. T., & Anderson, G. M. (2006) · Statistics in Medicine · doi:10.1002/sim.2618 · Cited by 230 · see citing works
Evidence for Use of Coronary Stents: A Hierarchical Bayesian Meta-Analysis
Brophy, J., Belisle, P., & Joseph, L. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-10-200305200-00005 · Cited by 208 · see citing works
Why Data Bases Should Not Replace Randomized Clinical Trials
Byar, D. P. (1980) · Biometrics · doi:10.2307/2529989 · Cited by 206 · see citing works
Austin, P. C. (2008) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.108.790634 · Cited by 153 · see citing works
Observational Methods in Comparative Effectiveness Research
Concato, J., Lawler, E. V., Lew, R., Gaziano, J. M., Aslan, M., & Huang, G. D. (2010) · The American Journal of Medicine · doi:10.1016/j.amjmed.2010.10.004 · Cited by 124 · see citing works
Any casualties in the clash of randomised and observational evidence?
Ioannidis, J. P. A., Haidich, A., & Lau, J. (2001) · BMJ · doi:10.1136/bmj.322.7291.879 · Cited by 107 · see citing works
Rassen, J. A., Avorn, J., & Schneeweiß, S. (2010) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1867 · Cited by 49 · see citing works
The Washington Manual of medical therapeutics
Cooper, D., Krainik, A. J., Lubner, S. J., & Reno, H. (2007) · Lippincott Williams & Wilkins eBooks · Cited by 47 · see citing works
Ottervanger, J. P., Armstrong, P. W., Barnathan, E. S., Boersma, E. H., COOPER, J., Ohman, E. M., James, S. K., Wallentin, L. C., & Simoons, M. L. (2004) · European Heart Journal · doi:10.1016/j.ehj.2004.07.004 · Cited by 35 · see citing works
The Washington Manual of medical therapeutics
Green, G. B., & Schaiff, R. (2004) · Lippincott Williams & Wilkins eBooks · Cited by 31 · see citing works
Therapeutic Innovations, Diminishing Returns, and Control Rate Preservation
Kent, D. M. (2009) · JAMA · doi:10.1001/jama.2009.1679 · Cited by 30 · see citing works
Flagged first, then by citations
1 of the 40 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- [3]Rosenbaum PR. Observational Studies. 2nd ed. New York, USA: Springer; 2002. [Google Scholar]
- [4]Byar DP. Why databases should not replace randomized clinical trials. Biometrics. 1980;36:337–342. [PubMed] [Google Scholar]
Cited by
Sorted by citations
Liddle, A. D., Judge, A., Pandit, H. G., & Murray, D. W. (2014) · The Lancet · doi:10.1016/s0140-6736(14)60419-0 · Cited by 676 · see citing works
Anglemyer, A., Horvath, H. T., & Bero, L. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000034.pub2 · Cited by 534 · see citing works
Emulation of Randomized Clinical Trials With Nonrandomized Database Analyses
Wang, S. V., Schneeweiß, S., Initiative, R.-D., Franklin, J. M., Desai, R. J., Feldman, W. B., Garry, E. M., Glynn, R. J., Lin, K. J., Paik, J. M., et al. (2023) · JAMA · doi:10.1001/jama.2023.4221 · Cited by 424 · see citing works
When and how can real world data analyses substitute for randomized controlled trials?
Franklin, J. M., & Schneeweiss, S. (2017) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.857 · Cited by 316 · see citing works
Emulating Randomized Clinical Trials With Nonrandomized Real-World Evidence Studies
Franklin, J. M., Patorno, E., Desai, R. J., Glynn, R. J., Martin, D., Quinto, K., Pawar, A. M., Bessette, L. G., Lee, H., Garry, E. M., et al. (2020) · Circulation · doi:10.1161/circulationaha.120.051718 · Cited by 301 · see citing works
Ageno, W., Mantovani, L. G., Haas, S. K., Kreutz, R. H., Monje, D., Schneider, J., van Eickels, M., Gebel, M., Zell, E. R., & Turpie, A. G. G. (2015) · The Lancet Haematology · doi:10.1016/s2352-3026(15)00257-4 · Cited by 241 · see citing works
Lonjon, G., Boutron, I., Trinquart, L., Ahmad, N., Aïm, F., Nizard, R., & Ravaud, P. (2013) · Annals of Surgery · doi:10.1097/sla.0000000000000256 · Cited by 235 · see citing works
Hemkens, L. G., Contopoulos‐Ioannidis, D. G., & Ioannidis, J. P. A. (2016) · BMJ · doi:10.1136/bmj.i493 · Cited by 220 · see citing works
Kory, P., Meduri, G. U., Varon, J., Iglesias, J., & Marik, P. E. (2021) · American Journal of Therapeutics · doi:10.1097/mjt.0000000000001377 · Cited by 203 · see citing works
Adjusted Analyses in Studies Addressing Therapy and Harm
Agoritsas, T., Merglen, A., Shah, N. D., O'Donnell, M. J., & Guyatt, G. H. (2017) · JAMA · doi:10.1001/jama.2016.20029 · Cited by 185 · see citing works
Survival Advantage of Laparoscopic Versus Open Resection For Colorectal Liver Metastases
Syn, N. L. X., Kabir, T., Koh, Y. X., Tan, H. L., Wang, L. Z., Chin, B. Z., Wee, I. J. Y., Teo, J. Y., Tai, B. C., & Goh, B. K. P. (2019) · Annals of Surgery · doi:10.1097/sla.0000000000003672 · Cited by 145 · see citing works
Long-Term Health Outcomes in Women With Silicone Gel Breast Implants
Balk, E. M., Earley, A., Avendano, E. E., & Raman, G. (2015) · Annals of Internal Medicine · doi:10.7326/m15-1169 · Cited by 144 · see citing works
Long-term Outcomes of Laparoscopic Versus Open Surgery for Clinical Stage II/III Gastric Cancer
Kinoshita, T., Uyama, I., Terashima, M., Noshirο, H., Nagai, E., Obama, K., Tamamori, Y., Nabae, T., Honda, M., & Abe, T. (2018) · Annals of Surgery · doi:10.1097/sla.0000000000002768 · Cited by 143 · see citing works
Hettle, R., Corbett, M. S., Hinde, S., Hodgson, R. M., Jones-Diette, J. S., Woolacott, N. F., & Palmer, S. (2017) · Health Technology Assessment · doi:10.3310/hta21070 · Cited by 126 · see citing works
Franklin, J. M., Pawar, A. M., Martin, D., Glynn, R. J., Levenson, M. S., Temple, R., & Schneeweiß, S. (2019) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.1633 · Cited by 122 · see citing works
Use of Health Care Databases to Support Supplemental Indications of Approved Medications
Fralick, M., Kesselheim, A. S., Avorn, J., & Schneeweiß, S. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2017.3919 · Cited by 119 · see citing works
Sarri, G., Patorno, E., Yuan, H., Guo, J., Bennett, D., Wen, X., Zullo, A. R., Largent, J. A., Panaccio, M. P., Gokhale, M., et al. (2020) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2020-111493 · Cited by 120 · see citing works
“Several reviews have found little difference between the evidence from observational studies and RCTs, 48–50 but counterexamples exist.”
Ramos, R., García-Gil, M., Comas‐Cufí, M., Quesada, M., Marrugat, J., Elosúa, R., Sala, J., Grau, M., Martí‐Lluch, R., Ponjoan, A., et al. (2016) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2015.11.052 · Cited by 116 · see citing works
“Researchers have demonstrated that such studies properly replicate the effects of statins observed in RCTs (42); it also has been suggested that observational studies could tend to overestimate the effect size of interventions (43) if several key points are not well addressed.”
Kent, D. M., Dahabreh, I. J., Ruthazer, R., Furlan, A. J., Weimar, C., Serena, J. L., Meier, B., Mattle, H. P., Angelantonio, E. D., Paciaroni, M., et al. (2015) · European Heart Journal · doi:10.1093/eurheartj/ehv252 · Cited by 112 · see citing works
“sometimes disagree with randomized study results.(27,28) Fourth, the quality of study procedures can vary from database to database, with variable patient selection criteria, rates of loss to follow-up, and outcome ascertainment methods across databases.”
BackgroundKitsios, G. D., Mascari, P., Ettunsi, R., & Gray, A. W. (2013) · Journal of Critical Care · doi:10.1016/j.jcrc.2013.10.004 · Cited by 105 · see citing works
“Our review did not consider observational studies that can inform on the pragmatic effectiveness of interventions [24].”
BackgroundAmoah, J., Stuart, E. A., Cosgrove, S. E., Harris, A. D., Han, J., Lautenbach, E., & Tamma, P. D. (2020) · Clinical Infectious Diseases · doi:10.1093/cid/ciaa169 · Cited by 95 · see citing works
Association of Alendronate and Risk of Cardiovascular Events in Patients With Hip Fracture
Sing, C., Wong, A. Y. S., Kiel, D. P., Cheung, E. Y. N., Lam, J. K. Y., Cheung, T. T., Chan, E. W., Kung, A. W. C., Wong, I. C. K., & Cheung, C. (2018) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.3448 · Cited by 94 · see citing works
“Overestimation of treatment effect in PS-based observational studies is commonly reported.((31,32)) One possible reason for the discrepancy is the difference in the study populations.”
BackgroundKitsios, G. D., Dahabreh, I. J., Callahan, S. P., Paulus, J. K., Campagna, A. C., & Dargin, J. (2015) · Critical Care Medicine · doi:10.1097/ccm.0000000000001135 · Cited by 92 · see citing works
“Assessment of the Methodological Quality of PS Studies and RCTs Based on surveys of the methodological features of studies using PS methods (10, 13, 14), we identified a set of items as potentially indicative of the validity of PS-based analyses (supplemental document, Supplemental Digital Content 1, http://”
BackgroundMethodResultInfluentialLong-term Outcomes of Laparoscopic Versus Open Surgery for Clinical Stage I Gastric Cancer
Honda, M., Hiki, N., Kinoshita, T., Yabusaki, H., Abe, T., Nunobe, S., Terada, M., Matsuki, A., Sunagawa, H., Aizawa, M., et al. (2016) · Annals of Surgery · doi:10.1097/sla.0000000000001654 · Cited by 82 · see citing works
Chaplin, D., Cook, T. D., Zurovac, J., Coopersmith, J., Finucane, M. M., Vollmer, L. N., & Morris, R. E. (2018) · Journal of Policy Analysis and Management · doi:10.1002/pam.22051 · Cited by 82 · see citing works
Raman, G., Adam, G. P., Halladay, C. W., Langberg, V. N., Azodo, I. A., & Balk, E. M. (2016) · Annals of Internal Medicine · doi:10.7326/m16-1053 · Cited by 81 · see citing works
Machine learning methods for developing precision treatment rules with observational data
Kessler, R. C., Bossarte, R. M., Luedtke, A. R., Zaslavsky, A. M., & Zubizarreta, J. R. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103412 · Cited by 84 · see citing works
“It has been shown that analyses of balanced databases often yield aggregate results about comparative treatment effects very similar to those obtained in randomized clinical trials (Anglemyer, Horvath, & Bero, 2014; Dahabreh et al., 2012).”
ResultChiow, A. K. H., Fuks, D., Choi, G. H., Syn, N. L. X., Sucandy, I., Marino, M. V., Prieto, M., Chong, C. C. N., Lee, J. H., Efanov, M. G., et al. (2021) · British journal of surgery · doi:10.1093/bjs/znab321 · Cited by 78 · see citing works
Liu, L., Cui, S., Liu, M., Huo, X., Zhang, G., & Wang, N. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.829709 · Cited by 76 · see citing works
“It has been shown that the practical meaning of HR and RR are very similar, so we converted HR directly to RR for merging and the final effect size is represented by RR (16).”
MethodAdmon, A. J., Donnelly, J. P., Casey, J. D., Janz, D. R., Russell, D. W., Joffe, A. M., Vonderhaar, D. J., Dischert, K. M., Stempek, S. B., Dargin, J., et al. (2019) · Annals of the American Thoracic Society · doi:10.1513/annalsats.201903-241oc · Cited by 73 · see citing works
Can the Learning Health Care System Be Educated With Observational Data?
Dahabreh, I. J., & Kent, D. M. (2014) · JAMA · doi:10.1001/jama.2014.4364 · Cited by 71 · see citing works
The codex of science: honesty, precision, and truth--and its violations
Lüscher, T. F. (2013) · European Heart Journal · doi:10.1093/eurheartj/eht063 · Cited by 67 · see citing works
Paulus, J. K., Dahabreh, I. J., Balk, E. M., Avendano, E. E., Lau, J., & Ip, S. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1101 · Cited by 64 · see citing works
“Furthermore, empirical evidence indicates that in many cases, different study designs reach similar conclusions (Britton et al., 1998; Dahabreh et al., 2012; Dalziel et al., 2005; Ioannidis et al., 2001; MacLehose et al., 2000; Vandenbroucke, 2011), offering another rationale for quantitatively…”
ResultTenets for the Proper Conduct and Use of Meta-Analyses: A Practical Guide for Neurosurgeons
Lee, K. S., Zhang, J. J. Y., Nga, V. D. W., Ng, C. H., Tai, B. C., Higgins, J. P. T., & Syn, N. L. X. (2022) · World Neurosurgery · doi:10.1016/j.wneu.2021.09.034 · Cited by 63 · see citing works
Ribeiro, S. C., Figueiredo, A. E., Barretti, P., Pecoits‐Filho, R., & de Moraes, T. P. (2015) · PLoS ONE · doi:10.1371/journal.pone.0127453 · Cited by 61 · see citing works
“To overcome this limitation, we used a propensity score to mimic some of the characteristics of randomized clinical trials (RCT) instead of relying on the use of regression adjustments to account for differences at the baseline [13].”
MethodToews, I., Anglemyer, A., Nyirenda, J. L., Alsaid, D., Balduzzi, S., Grummich, K., Schwingshackl, L., & Bero, L. (2024) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000034.pub3 · Cited by 60 · see citing works
Integration of Pain Score and Morphine Consumption in Analgesic Clinical Studies
Dai, F., Silverman, D. G., Chelly, J. E., Li, J., Belfer, I., & Li, Q. (2013) · Journal of Pain · doi:10.1016/j.jpain.2013.04.004 · Cited by 59 · see citing works
Vahidy, F., Haque, M. E., Rahbar, M. H., Zhu, H., Rowan, P. J., Aisiku, I. P., Lee, D. A., Juneja, H. S., Alderman, S., Barreto, A. D., et al. (2019) · Stem Cells · doi:10.1002/stem.3080 · Cited by 53 · see citing works
Borah, B. J., Moriarty, J. P., Crown, W. H., & Doshi, J. A. (2013) · Journal of Comparative Effectiveness Research · doi:10.2217/cer.13.89 · Cited by 53 · see citing works
Matsui, H., Fushimi, K., Yasunaga, H., & Tagami, T. (2015) · Thrombosis and Haemostasis · doi:10.1160/th15-01-0053 · Cited by 53 · see citing works
“However, when the study participants were restricted to patients with sepsis (40) and with coronary artery syndrome (41), the results were suggested conflicting one.”
ResultKessler, R. C., Bauer, M. S., Bishop, T. M., Demler, O. V., Dobscha, S. K., Gildea, S. M., Goulet, J. L., Karras, E., Kreyenbuhl, J., Landes, S. J., et al. (2020) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2020.00390 · Cited by 52 · see citing works
Risk of dementia or cognitive impairment in COPD patients: A meta-analysis of cohort studies
Wang, J., Li, X., Lei, S., Zhang, D., Zhang, S., Zhang, H., & Li, J. (2022) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2022.962562 · Cited by 51 · see citing works
“In the results section, all relative effects metrics were referred to as RRs, which did not affect our results or their interpretation (Dahabreh et al., 2012).”
ResultKory, P. D., Meduri, G. U., Iglesias, J. I., Varón, J., Berkowitz, K., Kornfeld, H., Vinjevoll, E. H., Scott, M., Wagshul, F. A., & Marik, P. E. (2020) · doi:10.31219/osf.io/wx3zn · Cited by 49 · see citing works
Use of Open Claims vs Closed Claims in Health Outcomes Research
Başer, O., Samayoa, G., Yapar, N., Baser, E., & Mete, F. (2023) · Journal of health economics and outcomes research · doi:10.36469/001c.87538 · Cited by 48 · see citing works
Methodological choices for the clinical development of medical devices
Bernard, A., Vaneau, M., Fournel, I., Galmiche, H., Nony, P., & Dubernard, J. M. (2014) · Medical Devices Evidence and Research · doi:10.2147/mder.s63869 · Cited by 47 · see citing works
“A systematic review that compared the results of RCTs with those of observational studies using propensity score on the various treatments during acute coronary syndromes has been published.(59) Overall, observational studies report a greater amount of effect for long-term mortality than RCTs.”
D’Silva, M., Han, H., Liu, R., Kingham, T. P., Choi, G. H., Syn, N. L. X., Prieto, M., Choi, S. H., Sucandy, I., Chiow, A. K. H., et al. (2022) · British journal of surgery · doi:10.1093/bjs/znac270 · Cited by 46 · see citing works
Forbes, S. P., & Dahabreh, I. J. (2020) · Journal of General Internal Medicine · doi:10.1007/s11606-020-05713-5 · Cited by 45 · see citing works
What type of clinical evidence is needed to assess medical devices?
Tarricone, R., Boscolo, P. R., & Armeni, P. (2016) · European Respiratory Review · doi:10.1183/16000617.0016-2016 · Cited by 43 · see citing works
Mellado, M., Pijoán, J. I., Jiménez, D., Muriel, A., Aujesky, D., Bertoletti, L., Decousus, H., Barrios, D., Clarà, A. B. I., Yusen, R. D., et al. (2016) · JACC: Cardiovascular Interventions · doi:10.1016/j.jcin.2016.08.039 · Cited by 42 · see citing works
“Studies have found good agreement between the treatment effects in the propensity score studies compared with those in the randomized trials (23).”
ResultWu, C., Luedtke, A. R., Sadikova, E., Tsai, H., Liao, S., Liu, C., Gau, S. S., VanderWeele, T. J., & Kessler, R. C. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.21660 · Cited by 42 · see citing works
Zhang, Z., Ni, H., & Xu, X. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.02.018 · Cited by 41 · see citing works
Dahabreh, I. J., Matthews, A., Steingrimsson, J. A., Scharfstein, D. O., & Stuart, E. A. (2023) · Epidemiologic Reviews · doi:10.1093/epirev/mxac011 · Cited by 40 · see citing works
Laborde‐Castérot, H., Agrinier, N., & Thilly, N. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.04.003 · Cited by 38 · see citing works
Collins, P. G. S., & Manach, Y. L. (2012) · European Heart Journal · doi:10.1093/eurheartj/ehs186 · Cited by 37 · see citing works
“observational studies using propensity scores and RCTs evaluating therapeutic interventions for acute coronary syndrome.(5) By”
MethodRubens, F. D., Chen, L., & Bourke, M. (2015) · The Annals of Thoracic Surgery · doi:10.1016/j.athoracsur.2015.10.031 · Cited by 32 · see citing works
Woolacott, N. F., Corbett, M. S., Jones-Diette, J. S., & Hodgson, R. M. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.07.002 · Cited by 32 · see citing works
Cuello‐García, C. A., Morgan, R. L., Brożek, J., Santesso, N. A., Verbeek, J. H. A. M., Thayer, K. A., Guyatt, G. H., & Schünemann, H. J. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.01.010 · Cited by 32 · see citing works
Using observational data for personalized medicine when clinical trial evidence is limited
Gershman, B., Guo, D. P., & Dahabreh, I. J. (2018) · Fertility and Sterility · doi:10.1016/j.fertnstert.2018.04.005 · Cited by 32 · see citing works
Hatano, Y., Aoyama, M., Morita, T., Yamaguchi, T., Maeda, I., Kizawa, Y., Tsuneto, S., Shima, Y., & Miyashita, M. (2017) · Psycho-Oncology · doi:10.1002/pon.4412 · Cited by 30 · see citing works
“Recent systematic reviews show that the estimates of treatment effects from nonrandomized controlled studies with propensity score analysis were consistent with the results of randomized controlled trials.(24,25) Our results would suggest that the effect of the place of death on bereavement outcomes may be more modest…”
ResultAssociations of Bariatric Interventions With Micronutrient and Endocrine Disturbances
Syn, N. L. X., Lee, P. C., Kovalik, J., Tham, K. W., Ong, H. S., Chan, W. H., Tan, C. S., & Lim, C. H. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.5123 · Cited by 30 · see citing works
Adejumo, A., Adegbala, O. M., Adejumo, K. L., & Bukong, T. N. (2018) · Canadian Journal of Gastroenterology and Hepatology · doi:10.1155/2018/9430953 · Cited by 29 · see citing works
Loiseau, N., Trichelair, P., He, M., Andreux, M., Zaslavskiy, M., Wainrib, G., & Blum, M. G. B. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01799-z · Cited by 29 · see citing works
Ageno, W., Mantovani, L. G., Haas, S. K., Kreutz, R. H., Haupt, V., Schneider, J., & Turpie, A. G. (2014) · Thrombosis Journal · doi:10.1186/1477-9560-12-16 · Cited by 27 · see citing works
“The potential for allocation bias in open-label, nonrandomized studies will be minimized by also presenting a propensity score analysis to control the effect of unbalanced covariates [26,27].”
Clinical data mining: challenges, opportunities, and recommendations for translational applications
Qiao, H., Chen, Y., Qian, C., & Guo, Y. (2024) · Journal of Translational Medicine · doi:10.1186/s12967-024-05005-0 · Cited by 27 · see citing works
“Data mining of EHRs has been demonstrated to be able to replicate the findings of clinical trials [118–120], and to be more realistically assessed than in RCTs given their size and multifariousness of patients.”
BackgroundKempfert, J., Meyer, A., Kim, W.-K., Linden, A. V., Arsalan, M., Blumenstein, J., Möllmann, H., & Walther, T. (2014) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezu097 · Cited by 26 · see citing works
“In addition, PS matching has been shown to be a reliable statistical method [18] but in general is inferior to a prospective randomized trial as PS matching cannot adjust for any potential unknown/unmeasured confounder [19].”
MethodAdejumo, A., Kim, D., Iqbal, U., Yoo, E. R., Boursiquot, B. C., Cholankeril, G. T., Wong, R. J., Kwo, P. Y., & Ahmed, A. (2019) · Journal of Palliative Medicine · doi:10.1089/jpm.2019.0100 · Cited by 26 · see citing works
“Propensity matching approximates a pseudorandomized clinical study and produces estimates similar in size to double-blinded clinical studies.(26,27) After matching, we compared between PCC and noPCC using the statistical method for paired comparison such as paired t test, Wilcoxon signed-rank test, and McNemar’s test.”
MethodTranslating cell therapies for neurodegenerative diseases: Huntington’s disease as a model disorder
Rosser, A., Busse, M. E., Gray, W., Badin, R. A., Perrier, A. L., Wheelock, V., Cozzi, E., Perpiñá, U., Salado-Manzano, C., Mills, L. J., et al. (2022) · Brain · doi:10.1093/brain/awac086 · Cited by 26 · see citing works
“the use of historical control data or observational data from disease registries, that methods to account for confounders are also taken into account.(57,58) Given the very early stage of development of these novel therapeutics in a rare disease such asHuntington’s disease, itmay be important to start by focusing on s…”
Naito, Y., Ida, M., Yamamoto, R., Tachibana, K., & Kinouchi, K. (2019) · JA Clinical Reports · doi:10.1186/s40981-019-0260-z · Cited by 26 · see citing works
Tabák, Á. G., Anderson, J. E., Aschner, P. J., Liu, M., Saremi, A., Stella, P., Tinahones, F. J., Wysham, C., & Meier, J. J. (2019) · Diabetes Therapy · doi:10.1007/s13300-019-00735-7 · Cited by 26 · see citing works
Current challenges for clinical trials of cardiovascular medical devices
Zannad, F. M., Stough, W. G., Piña, I. L., Mehran, R., Abraham, W. T., Anker, S. D., Ferrari, G. M. D., Farb, A. A., Geller, N. L., Kieval, R. S., et al. (2014) · International Journal of Cardiology · doi:10.1016/j.ijcard.2014.05.021 · Cited by 24 · see citing works
“For example, clinical outcomes in subjects enrolled in observational studies may appear to be better than those seen in randomized trials [30], or they may differ completely [31].”
BackgroundMüller, G., Pfinder, M., Schmahl, C. G., Bohus, M., & Lyssenko, L. (2019) · BMC Public Health · doi:10.1186/s12889-019-7585-4 · Cited by 25 · see citing works
Li, G., Holbrook, A. M., Jin, Y., Zhang, Y., Levine, M., Mbuagbaw, L. C. E., Witt, D. M., Crowther, M. A., Connolly, S. J., Chai‐Adisaksopha, C., et al. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0178-y · Cited by 24 · see citing works
“Several systematic reviews have compared the treatment effect estimates from PS studies and RCTs in different clinical settings, with a conclusion that PS studies may either overestimate [24, 26], underestimate [27], or be consistent with [25, 28, 29] the estimates from RCTs.”
ResultAssessing Heterogeneity of Treatment Effects in Observational Studies
Robertson, S. E., Leith, A., Schmid, C. H., & Dahabreh, I. J. (2020) · American Journal of Epidemiology · doi:10.1093/aje/kwaa235 · Cited by 23 · see citing works
Impact of the matching algorithm on the treatment effect estimate: A neutral comparison study
Heinz, P., Wendel‐Garcia, P. D., & Held, U. (2022) · Biometrical Journal · doi:10.1002/bimj.202100292 · Cited by 23 · see citing works
“Results based on observational data may reflect clinical reality more closely, but, on the other hand, bias and confounding are common to affect treatment effect estimates, producing potentially conflicting evidence as compared to RCT results (Dahabreh et al., 2012).”
ResultComparing surgical strategies for end-stage anteromedial osteoarthritis
Mikkelsen, M., Wilson, H. A., Gromov, K., Price, A. J., & Troelsen, A. (2022) · Bone & Joint Open · doi:10.1302/2633-1462.35.bjo-2021-0174.r1 · Cited by 23 · see citing works
Zhang, X., Xie, J., Li, G., Chen, Q., & Xu, B. (2014) · PLoS ONE · doi:10.1371/journal.pone.0097934 · Cited by 23 · see citing works
“In a comprehensive meta-analysis comparing safety and efficiency of DES with BMS, DES significantly reduces mortality by 22% in observational studies, but the effect is attenuated in RCTs (RR = 0.97, 95% CI = 0.81–1.15) [21].”
BackgroundInfluentialWhy the Propensity for Propensity Scores?*
Day, A. G. (2015) · Critical Care Medicine · doi:10.1097/ccm.0000000000001175 · Cited by 23 · see citing works
“In 2012, Dahabreh et al (5) concluded that PS studies of acute coronary syndromes tended to overestimate the efficacy of experimental treatments compared with RCTs.”
ResultRicci, C., Ingaldi, C., Alberici, L., Pagano, N., Mosconi, C., Marasco, G., Minni, F., & Casadei, R. (2021) · World Journal of Surgery · doi:10.1007/s00268-021-06039-x · Cited by 22 · see citing works
Hong, Y. D., Jansen, J. P., Guerino, J., Berger, M. L., Crown, W. H., Goettsch, W., Mullins, C. D., Willke, R. J., & Orsini, L. S. (2021) · BMC Medicine · doi:10.1186/s12916-021-02176-1 · Cited by 22 · see citing works
“43) [16] and absence of an extreme difference.”
BackgroundGoh, B. K. P., Syn, N. L. X., Koh, Y. X., Teo, J., Cheow, P. C., Jeyaraj, P. R., Chow, P. K. H., Ooi, L. L., Chung, A. Y. F., & Chan, C. Y. (2021) · Journal of Surgical Oncology · doi:10.1002/jso.26556 · Cited by 22 · see citing works
Carlsson, A. C., Wändell, P. E., Sundquist, K., Johansson, S., & Sundquist, J. (2014) · Hypertension Research · doi:10.1038/hr.2014.32 · Cited by 22 · see citing works
Ndrepepa, G., Kufner, S., Mayer, K., Cassese, S., Xhepa, E., Fusaro, M., Hasimi, E., Schüpke, S., Laugwitz, K., Schunkert, H., et al. (2018) · Cardiovascular revascularization medicine · doi:10.1016/j.carrev.2018.05.010 · Cited by 22 · see citing works
McEvoy, R. D., Sánchez‐de‐la‐Torre, M., Peker, Y., Anderson, C. S., Redline, S. S., & Barbé, F. (2021) · SLEEP · doi:10.1093/sleep/zsab019 · Cited by 21 · see citing works
“Propensity score and other adjusted analyses provide information on the utility of evidence in real-world settings but cannot establish effectiveness, may overestimate CV treatment effects, and even conflict with RCT findings [19].”
BackgroundHamai, S., Nakashima, Y., Mashima, N., Yamamoto, T., Kamada, T., Motomura, G., Imai, H., Fukushi, J., Miura, H., & Iwamoto, Y. (2015) · Journal of the mechanical behavior of biomedical materials/Journal of mechanical behavior of biomedical materials · doi:10.1016/j.jmbbm.2015.12.022 · Cited by 19 · see citing works
Liu, F., Yang, Y., Cheng, W., Ma, J., & Zhu, W. (2021) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2021.757188 · Cited by 18 · see citing works
“An effective method to evaluate interventions’ effectiveness in typical clinical settings can be provided by the propensity score (PS) (14).”
MethodCao, Y., Zhou, M., Guo, H., & Zhu, W. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.840452 · Cited by 18 · see citing works
“Ultimately, we selected RRs and 95% CIs to represent all the effect estimates in the results, which did not influence the final interpretation (14).”
MethodUse of real world data to improve drug coverage decisions in China
Wang, W., Tan, J., Wu, J., Huang, S., Huang, Y., Xie, F., & Sun, X. (2023) · BMJ · doi:10.1136/bmj-2021-068911 · Cited by 18 · see citing works
The past, present and perhaps future of pharmacovigilance: homage to Folke Sjoqvist
Moore, N. D. (2013) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-013-1486-8 · Cited by 18 · see citing works
Mason, C., Sabariego, C., Thắng, Đ. M., & Weber, J. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-022544 · Cited by 17 · see citing works
“A systematic review comparing 21 PSM studies to 63 randomised controlled studies (RCTs) on therapeutic interventions for acute coronary syndromes found that PSM produced more extreme treatment effect estimates when compared with those from RCTs, although these differences were rarely statistically significant.(13) A s…”
ResultKomiyama, K., Hamai, S., Motomura, G., Ikemura, S., Fujii, M., Kawahara, S., & Nakashima, Y. (2021) · Archives of Orthopaedic and Trauma Surgery · doi:10.1007/s00402-021-03817-z · Cited by 16 · see citing works
Shibata, N., Kimura, S. H., Hoshino, T., Takeuchi, M., & Urushihara, H. (2018) · Vaccine · doi:10.1016/j.vaccine.2018.03.082 · Cited by 16 · see citing works
Morfaw, F., Fundoh, M., Pisoh, C. T., Ayaba, B., Mbuagbaw, L. C. E., Anderson, L. N., & Thabane, L. (2019) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-019-2407-3 · Cited by 16 · see citing works
Mei, F., Yao, M., Wang, Y., Huan, J., Ma, Y., Li, G., Zou, K., Li, L., & Sun, X. (2024) · BMC Medicine · doi:10.1186/s12916-024-03778-1 · Cited by 15 · see citing works
“RoRs indicated an “important difference” (< 0.70 or > 1.43) or not (0.7 ≤ RoR ≤ 1.43) [27, 28].”
InfluentialAddressing Multimorbidity in Evidence Integration and Synthesis
Trikalinos, T. A., Segal, J. B., & Boyd, C. M. (2014) · Journal of General Internal Medicine · doi:10.1007/s11606-013-2661-4 · Cited by 15 · see citing works
Alves‐Cabratosa, L., García-Gil, M., Comas‐Cufí, M., Ponjoan, A., Martí‐Lluch, R., Parramon, D., Blanch, J., Elosua-Bayés, M., & Ramos, R. (2017) · PLoS ONE · doi:10.1371/journal.pone.0186972 · Cited by 15 · see citing works
Liver Resection Versus Embolization for Recurrent Hepatocellular Carcinoma
Midorikawa, Y., Takayama, T., Moriguchi, M., Yagi, R., Yamagishi, S., Nakayama, H., Aramaki, O., Yamazaki, S., Tsuji, S., & Higaki, T. (2019) · World Journal of Surgery · doi:10.1007/s00268-019-05225-2 · Cited by 15 · see citing works
Mori, R., Uchino, T., Mizuno, M., Yamaguchi, T., Katagiri, N., & Nemoto, T. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm12081281 · Cited by 17 · see citing works
“For example, using methods such as propensity score matching may be useful [43,44].”
MethodChin, K. M., Linn, Y. L., Cheong, C. K., Koh, Y. X., Teo, J., Chung, A. Y. F., Chan, C. Y., & Goh, B. K. P. (2021) · ANZ Journal of Surgery · doi:10.1111/ans.16719 · Cited by 15 · see citing works
“PSM is thus deemed to be a close approximation to randomization and is one of the most commonly employed methods for evaluation of differing treatment modalities in the current surgical literature.(47,48) While most previous studies have performed PSM based on pre-operative parameters of qualitative hepatic function (…”
MethodWang, S., Ye, G., Wang, J., Xu, S., Ye, Q., & Ye, H. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.939877 · Cited by 14 · see citing works
Janke, K., Salifu, Y., Gavini, S., Preston, N. J., & Gadoud, A. (2024) · BMC Palliative Care · doi:10.1186/s12904-024-01516-1 · Cited by 14 · see citing works
“Observational methods can overestimate beneficial treatment effects compared to RCTs [30], which might explain our more nuanced findings for RCTs.”
- Authors
- Dahabreh, I. J., Sheldrick, R. C., Paulus, J. K., Chung, M., Varvarigou, V., Jafri, H., Rassen, J. A., Trikalinos, T. A., & Kitsios, G. D.
- Affiliations
- Tufts Medical Center (US)
- Tufts University (US)
- Harvard University (US)
- Johns Hopkins University (US)
- Brigham and Women's Hospital (US)
- Lahey Hospital and Medical Center (US)
- Corresponding author
- Dahabreh, Issa J.
- ORCID
- Venue
- European Heart Journal
- Publisher
- Oxford University Press (OUP)
- Published
- 2012-06-17
- Volume
- 33
- Issue
- 15
- Pages
- 1893-1901
- ISSN
- 0195-668X (ISSN-L)
- 1522-9645
- PMC
- PMC3409422
- OpenAlex
- W2162409127
- Cited by
- 204
- References
- 40
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 10 · 2023: 10 · 2022: 23 · 2021: 22 · 2020: 18 · 2019: 14 · 2018: 18 · 2017: 15 · 2016: 12 · 2015: 13 · 2014: 24 · 2013: 10 · 2012: 2
- FWCI
- 14.97
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- European Heart Journal (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Center for Research Resources (UL1RR025752)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- propensity score methods, acute coronary syndrome, observational studies, randomized controlled trials, treatment effect estimation, confounding, mortality
- MeSH terms
- Humans, Regression Analysis, Risk Factors, Reproducibility of Results, Randomized Controlled Trials as Topic, Treatment Outcome, Acute Coronary Syndrome, Propensity Score
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
