Comparing apples and oranges

Abstract

M itral valve repair versus replacement, internal thoracic artery versus saphenous vein graft conduits for coronary bypass, effect of chronic preoperative atrial fibrillation on outcome, gastric versus colon esophageal substitutes, complete versus incomplete off-pump revascularization, surgery in highversus low-volume centers, balloon versus surgical aortic valvotomy. These are but a sample of studies of comparative outcome whose basis was clinical experience rather than a formal clinical trial. Often, a cursory glance at patient characteristics in each group reveals important differences that lead medical and statistical reviewers and readers alike to scoff, “They’re comparing ‘apples and oranges!’” What does it take to convince the skeptic that the difference in outcome attributed to difference in treatment (or patient condition) is real? The answer to this question is not academic; it can affect the way we as physicians learn to treat our patients from studies of clinical experience. When comparison is made in the context of a properly designed, appropriate, ethical, feasible, well-analyzed, generalizable randomized trial, most of us would accept a cause-and-effect linkage between treatment and difference in outcome. In contrast, when the comparison emanates from studies of clinical experience—ubiquitous in surgical experience and reporting—cause-and-effect attribution is considered “speculative” at best. For 3 decades, multivariable risk factor analysis has been the mainstay for identifying and quantifying treatment outcome differences adjusted for patient characteristics. However, Kirklin and Barratt-Boyes1 recommended that these differences be treated as associations with outcomes, not causes. There is no guarantee that risk factor analysis is an effective strategy for discovery of cause-and-effect mechanisms.2,3 During the 1980s, federal support for complex clinical trials in heart disease was abundant. Few of us noticed important advances being made in statistical methods for valid, nonrandomized comparisons. An example of the advances was the seminal 1983 Biometrika article by Paul Rosenbaum at the University of Wisconsin, Madison, and Donald Rubin at the University of Chicago, “The Central Role of the Propensity Score in Observational Studies for Causal Effects.”4 In the 1990s, as the funding climate changed, interest in methods for making nonrandomized comparisons accelerated.5-10 Recently, these methods have been recommended by statistical reviewers for comparative clinical studies and have been adopted by some clinical research groups. The result has been the introduction into our literature of unfamiliar methods with their unfamiliar terminology. Rather than being relieved that at last applesto-apples comparisons can be made with rigor, medical and sometimes statistical reviewers, as well as readers, have become bewildered! From the Departments of Thoracic and Cardiovascular Surgery and Biostatistics and Epidemiology, The Cleveland Clinic Foundation, Cleveland, Ohio.