The Central Role of Receiver Operating Characteristic (ROC) Curves in Evaluating Tests for the Early Detection of Cancer

Abstract

Cancer screening involves early detection coupled with early intervention. For cancer screening to reduce cancer mortality, the cancer must be detected sufficiently early and the early intervention must be effective. For cancer screening to minimize harms, the cancers detected early should, to the extent possible, exclude cancers that would not cause medical problems in a person’s lifetime (a phenomenon known as overdiagnosis). Detection of overdiagnosed cancers leads to unnecessary work-ups, biopsies, and treatment. The gold standard for evaluating benefits and harms of cancer screening is a randomized controlled trial (1). Recently, with enthusiasm over advances in biomarker and imaging technology, early detection of cancer has taken a more prominent role in cancer research. Although the performance evaluation of early detection tests is only one component in evaluating cancer screening (and good performance of an early detection test does not necessarily translate into a reduction in cancer mortality), it is important for cancer researchers to understand how best to evaluate the performance of early detection tests. Researchers need good methods for identifying those early detection tests that are most promising for further investigation in a randomized trial in which the early detection test triggers early intervention (2,3). Researchers also need good methods for identifying factors or making comparisons that could improve the performance of existing early detection tests (4). An important and useful technique for evaluating the performance of diagnostic medical tests is the receiver operating characteristic (ROC) curve (5). However, for evaluating early detection tests of cancer, ROC curves have been either underused or not always used in the best manner. This commentary explains why ROC curves should be the primary method for evaluating the performance of early detection tests of cancer. It also discusses special considerations in applying ROC curves in particular applications. Early detection of cancer fundamentally involves a rule for classifying asymptomatic subjects as likely having cancer (ideally excluding overdiagnosed cancers) or not. There are two principal types of errors involved in tests for the early detection of cancer: classifying cancers as non-cancers and classifying non-cancers as cancers. Consequently, measures of performance should have two components related to the two types of errors. Single measures of performance, such as the odds ratio (in a 2 × 2 table with margins of cancer/no cancer and classifications of positive/negative), do not fully characterize performance and are therefore not appropriate. By convention, the performance of a classification rule is usually summarized by the following two quantities related to the two types of errors: true-positive rate and false-positive rate. In this context, the true-positive rate is the probability that a subject with cancer is correctly classified as having cancer, and the false-positive rate is the probability that a subject without cancer is incorrectly classified as having cancer. (The true-positive rate is also called sensitivity and one minus the false-positive rate is also called specificity.) For an ideal classification rule, the true-positive rate is one and the false-positive rate is zero. For cancer screening, where the target cancer typically has a very low prevalence in the screened population, the false-positive rate needs to be very low; otherwise, an inordinate number of screened healthy subjects will have unnecessary diagnostic work-ups or biopsies. The magnitudes of acceptable false-positive rates and true-positive rates depend on the corresponding costs and perceived benefits. For mammography screening with a positive result stringently defined as a recommendation for biopsy, the false-positive rate was 0.01 and the true-positive rate was 0.80 (6,7). [Some investigators define the false-positive rate on the basis of the decision to work-up any suspicious abnormality, yielding a false-positive rate of 0.065 for mammography (8).] These target values of the false-positive rate and true-positive rate can serve as a benchmark for other cancer screening evaluations (2,6). With cancer screening, a low false-positive rate is necessary for a reasonable positive predictive value, which is the probability of cancer given a positive screen. For example, with a false-positive rate of 0.01, a true-positive rate of 0.80, and a cancer prevalence of 0.001, the positive predictive value would be 0.074. However, for a false-positive rate of 0.05, a true-positive rate of 0.80, and a cancer prevalence of 0.001, the positive predictive value is only 0.016. To put the target values of the false-positive rate and the true-positive rate into perspective, an odds ratio corresponding to a false-positive rate of 0.01 and a true-positive rate of 0.80 would equal 396 (2), which is much higher than in most epidemiologic studies of cancer risk