The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
The use of receiver operating characteristic curves in biomedical informatics
Open accessAbstract
Receiver operating characteristic (ROC) curves are frequently used in biomedical informatics research to evaluate classification and prediction models for decision support, diagnosis, and prognosis. ROC analysis investigates the accuracy of a model's ability to separate positive from negative cases (such as predicting the presence or absence of disease), and the results are independent of the prevalence of positive cases in the study population. It is especially useful in evaluating predictive models or other tests that produce output values over a continuous range, since it captures the trade-off between sensitivity and specificity over that range. There are many ways to conduct an ROC analysis. The best approach depends on the experiment; an inappropriate approach can easily lead to incorrect conclusions. In this article, we review the basic concepts of ROC analysis, illustrate their use with sample calculations, make recommendations drawn from the literature, and list readily available software.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · cited 4×
“AccuROC [19,37] (commercial) Empirical Nonparametric asymptotic, BCa, other bootstrap Nonparametric calculated correlation correction [17] For Microsoft Windows.”
MethodInfluentialThe meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Measuring the Accuracy of Diagnostic Systems
Swets, J. A. (1988) · Science · doi:10.1126/science.3287615 · Cited by 10,058
Hanley, J. A., & McNeil, B. J. (1983) · Radiology · doi:10.1148/radiology.148.3.6878708 · Cited by 7,016
Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine
Zweig, M. H., & Campbell, G. (1993) · Clinical Chemistry · doi:10.1093/clinchem/39.4.561 · Cited by 6,569
Evaluating the Yield of Medical Tests
Harrell, F. E. (1982) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1982.03320430047030 · Cited by 3,490
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems
Hand, D. J., & Till, R. J. (2001) · Machine Learning · doi:10.1023/a:1010920819831 · Cited by 2,276
The Statistical Evaluation of Medical Tests for Classification and Prediction
Pepe, M. S. (2003) · doi:10.1093/oso/9780198509844.001.0001 · Cited by 2,198
The Statistical Evaluation of Medical Tests for Classification and Prediction
(2005) · Technometrics · doi:10.1198/tech.2005.s278 · Cited by 1,899
Statistical Methods in Diagnostic Medicine
Zhou, X., Obuchowski, N. A., & McClish, D. K. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9780470906514 · Cited by 1,784
ROC Methodology in Radiologic Imaging
METZ, C. E. (1986) · Investigative Radiology · doi:10.1097/00004424-198609000-00009 · Cited by 1,642
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,439
Some Practical Issues of Experimental Design and Data Analysis in Radiological ROC Studies
METZ, C. E. (1989) · Investigative Radiology · doi:10.1097/00004424-198903000-00012 · Cited by 859
Receiver Operating Characteristic Curves and Their Use in Radiology
Obuchowski, N. A. (2003) · Radiology · doi:10.1148/radiol.2291010898 · Cited by 843
Sources of Variation and Bias in Studies of Diagnostic Accuracy: A Systematic Review
Whiting, P., Rutjes, A. W. S., Reitsma, J. B., Glas, A. S., Bossuyt, P. M. M., & Kleijnen, J. (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-140-3-200402030-00010 · Cited by 798
Metz, C. E., Herman, B. A., & Shen, J.-H. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980515)17:9<1033::aid-sim784>3.0.co;2-z · Cited by 793
Dorfman, D. D., Berbaum, K. S., & Metz, C. E. (1992) · PubMed · Cited by 724
Analyzing a Portion of the ROC Curve
McClish, D. K. (1989) · Medical Decision Making · doi:10.1177/0272989x8900900307 · Cited by 707
DORFMAN, D. D., BERBAUM, K. S., & METZ, C. E. (1992) · Investigative Radiology · doi:10.1097/00004424-199209000-00015 · Cited by 661
MedCalc: a new computer program for medical statistics
Schoonjans, F., Zalata, A., Depuydt, C. E., & Comhaire, F. H. (1995) · Computer Methods and Programs in Biomedicine · doi:10.1016/0169-2607(95)01703-8 · Cited by 547
Estimation of the area under the ROC curve
Faraggi, D., & Reiser, B. (2002) · Statistics in Medicine · doi:10.1002/sim.1228 · Cited by 432
Nonparametric Analysis of Clustered ROC Curve Data
Obuchowski, N. A. (1997) · Biometrics · doi:10.2307/2533958 · Cited by 429
The interpretation of diagnostic tests
Shapiro, D. E. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800203 · Cited by 383
ROC Curves in Clinical Chemistry: Uses, Misuses, and Possible Solutions
Obuchowski, N. A., Lieber, M. L., & Wians, F. H. (2004) · Clinical Chemistry · doi:10.1373/clinchem.2004.031823 · Cited by 358
A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data
WIEAND, S., GAIL, M. H., JAMES, B. R., & JAMES, K. L. (1989) · Biometrika · doi:10.1093/biomet/76.3.585 · Cited by 355
Statistical Comparison of Two ROC-curve Estimates Obtained from Partially-paired Datasets
Metz, C. E., Herman, B. A., & Roe, C. A. (1998) · Medical Decision Making · doi:10.1177/0272989x9801800118 · Cited by 338
Mossman, D. (1999) · Medical Decision Making · doi:10.1177/0272989x9901900110 · Cited by 302
Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests
Zou, K. H., Hall, W. J., & Shapiro, D. E. (1997) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19971015)16:19<2143::aid-sim655>3.0.co;2-3 · Cited by 274
Self-citationVenkatraman, E. (1996) · Biometrika · doi:10.1093/biomet/83.4.835 · Cited by 219
Data mining techniques for cancer detection using serum proteomic profiling
Li, L., Tang, H., Wu, Z., Gong, J., Gruidl, M., Zou, J., Tockman, M., & Clark, R. A. (2004) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2004.03.006 · Cited by 198
Statistical Methods in Diagnostic Medicine
Zhou, X., Sun, J., Pennello, G. A., Obuchowski, N. A., & McClish, D. K. (2003) · Wiley series in probability and statistics · doi:10.1002/9781394220243.fmatter · Cited by 197
Hanley, J. A., & Hajian-Tilaki, K. O. (1997) · Academic Radiology · doi:10.1016/s1076-6332(97)80161-4 · Cited by 180
Text Categorization Models for High-Quality Article Retrieval in Internal Medicine
Aphinyanaphongs, Y. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1641 · Cited by 177 · cited 1×
“Recent examples of continuous classifiers evaluated using ROC curves include statistical models for retrieving highly useful clinical articles [8], radiology reports that suggest pneumonia [9] and radiology reports suggesting anthrax [10].”
BackgroundKairisto, V., & Poola, A. (1995) · Scandinavian Journal of Clinical and Laboratory Investigation · doi:10.3109/00365519509088450 · Cited by 175
Using Smoothed Receiver Operating Characteristic Curves to Summarize and Compare Diagnostic Systems
Lloyd, C. J. (1998) · Journal of the American Statistical Association · doi:10.1080/01621459.1998.10473797 · Cited by 170
Comparing Three-class Diagnostic Tests by Three-way ROC Analysis
Dreiseitl, S., Lucila Ohno-machado, A., & Binder, M. (2000) · Medical Decision Making · doi:10.1177/0272989x0002000309 · Cited by 164
Hajian‐Tilaki, K., Hanley, J. A., Joseph, L., & Collet, J. (1997) · Medical Decision Making · doi:10.1177/0272989x9701700111 · Cited by 144
Comparison of Eight Computer Programs for Receiver-Operating Characteristic Analysis
Stephan, C., Wesseling, S., Schink, T., & Jung, K. (2003) · Clinical Chemistry · doi:10.1373/49.3.433 · Cited by 140
Zhang, D. D., Zhou, X., Freeman, D. H., & Freeman, J. L. (2002) · Statistics in Medicine · doi:10.1002/sim.1011 · Cited by 137
Chapman, W. W., Fizman, M., Chapman, B. E., & Haug, P. J. (2001) · Journal of Biomedical Informatics · doi:10.1006/jbin.2001.1000 · Cited by 116
Kester, A. D. M., & Buntinx, F. (2000) · Medical Decision Making · doi:10.1177/0272989x0002000407 · Cited by 101
A computer program for non-parametric receiver operating characteristic analysis
Vida, S. (1993) · Computer Methods and Programs in Biomedicine · doi:10.1016/0169-2607(93)90004-5 · Cited by 83
Computing Sample Size for Receiver Operating Characteristic Studies
Obuchowski, N. A. (1994) · Investigative Radiology · doi:10.1097/00004424-199402000-00020 · Cited by 75
Preoperative prediction of malignancy of ovarian tumors using least squares support vector machines
Lü, C., Gestel, T. V., Suykens, J. A. K., Huffel, S. V., Vergote, I. B., & Timmerman, D. (2003) · Artificial Intelligence in Medicine · doi:10.1016/s0933-3657(03)00051-4 · Cited by 75
Chapman, W. W., Cooper, G. F., Hanbury, P., Chapman, B. E., Harrison, L. H., & D, M. M. W. M. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1330 · Cited by 45
Hajian‐Tilaki, K., & Hanley, J. A. (2002) · Academic Radiology · doi:10.1016/s1076-6332(03)80561-5 · Cited by 38
Obuchowski, N. A. (2003) · Academic Radiology · doi:10.1016/s1076-6332(03)00386-6 · Cited by 21
Confidence Bounds When the Estimated ROC Area is 1.0
Obuchowski, N. A., & Lieber, M. L. (2002) · Academic Radiology · doi:10.1016/s1076-6332(03)80329-x · Cited by 18
Evaluation of a Cardiac Diagnostic Program in a Typical Clinical Setting
Fraser, H. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1184 · Cited by 15
Cited by
Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models
Zou, K. H., O’Malley, A. J., & Mauri, L. (2007) · Circulation · doi:10.1161/circulationaha.105.594929 · Cited by 1,473
Self-citationSample size estimation in diagnostic test studies of biomedical informatics
Hajian-Tilaki, K. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.02.013 · Cited by 995
Translational biomarker discovery in clinical metabolomics: an introductory tutorial
Xia, J., Broadhurst, D. I., Wilson, M., & Wishart, D. S. (2013) · Metabolomics · doi:10.1007/s11306-012-0482-9 · Cited by 953
the DIARETDB1 diabetic retinopathy database and evaluation protocol
Kauppi, T., Kalesnykiene, V., Kamarainen, J.-K., Lensu, L., Sorri, I., Raninen, A., Voutilainen, R., Uusitalo, H., Kalviainen, H., & Pietila, J. (2007) · Procedings of the British Machine Vision Conference 2007 · doi:10.5244/c.21.15 · Cited by 854
Deformability-based cell classification and enrichment using inertial microfluidics
Hur, S. C., Henderson-MacLennan, N. K., McCabe, E. R. B., & Di Carlo, D. (2011) · Lab on a Chip · doi:10.1039/c0lc00595a · Cited by 576
Metadata
- Authors
- Lasko, T. A., Bhagwat, J. G., Zou, K. H., & Ohno-Machado, L.
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2005-10
- Volume
- 38
- Issue
- 5
- Pages
- 404-415
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W2029426756
- Cited by
- 823
- References
- 49
- FWCI
- 28.51
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.