The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
The use of receiver operating characteristic curves in biomedical informatics
Open accessAbstract
Receiver operating characteristic (ROC) curves are frequently used in biomedical informatics research to evaluate classification and prediction models for decision support, diagnosis, and prognosis. ROC analysis investigates the accuracy of a model's ability to separate positive from negative cases (such as predicting the presence or absence of disease), and the results are independent of the prevalence of positive cases in the study population. It is especially useful in evaluating predictive models or other tests that produce output values over a continuous range, since it captures the trade-off between sensitivity and specificity over that range. There are many ways to conduct an ROC analysis. The best approach depends on the experiment; an inappropriate approach can easily lead to incorrect conclusions. In this article, we review the basic concepts of ROC analysis, illustrate their use with sample calculations, make recommendations drawn from the literature, and list readily available software.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649 · cited 4×
“AccuROC [19,37] (commercial) Empirical Nonparametric asymptotic, BCa, other bootstrap Nonparametric calculated correlation correction [17] For Microsoft Windows.”
MethodInfluentialThe meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Measuring the Accuracy of Diagnostic Systems
Swets, J. A. (1988) · Science · doi:10.1126/science.3287615 · Cited by 10,058
Hanley, J. A., & McNeil, B. J. (1983) · Radiology · doi:10.1148/radiology.148.3.6878708 · Cited by 7,016
Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine
Zweig, M. H., & Campbell, G. (1993) · Clinical Chemistry · doi:10.1093/clinchem/39.4.561 · Cited by 6,569
Evaluating the Yield of Medical Tests
Harrell, F. E. (1982) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1982.03320430047030 · Cited by 3,490
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems
Hand, D. J., & Till, R. J. (2001) · Machine Learning · doi:10.1023/a:1010920819831 · Cited by 2,276
The Statistical Evaluation of Medical Tests for Classification and Prediction
Pepe, M. S. (2003) · doi:10.1093/oso/9780198509844.001.0001 · Cited by 2,198
The Statistical Evaluation of Medical Tests for Classification and Prediction
(2005) · Technometrics · doi:10.1198/tech.2005.s278 · Cited by 1,899
Statistical Methods in Diagnostic Medicine
Zhou, X., Obuchowski, N. A., & McClish, D. K. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9780470906514 · Cited by 1,784
Cited by
Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models
Zou, K. H., O’Malley, A. J., & Mauri, L. (2007) · Circulation · doi:10.1161/circulationaha.105.594929 · Cited by 1,473
Self-citationSample size estimation in diagnostic test studies of biomedical informatics
Hajian-Tilaki, K. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.02.013 · Cited by 995
Translational biomarker discovery in clinical metabolomics: an introductory tutorial
Xia, J., Broadhurst, D. I., Wilson, M., & Wishart, D. S. (2013) · Metabolomics · doi:10.1007/s11306-012-0482-9 · Cited by 953
the DIARETDB1 diabetic retinopathy database and evaluation protocol
Kauppi, T., Kalesnykiene, V., Kamarainen, J.-K., Lensu, L., Sorri, I., Raninen, A., Voutilainen, R., Uusitalo, H., Kalviainen, H., & Pietila, J. (2007) · Procedings of the British Machine Vision Conference 2007 · doi:10.5244/c.21.15 · Cited by 854
Deformability-based cell classification and enrichment using inertial microfluidics
Hur, S. C., Henderson-MacLennan, N. K., McCabe, E. R. B., & Di Carlo, D. (2011) · Lab on a Chip · doi:10.1039/c0lc00595a · Cited by 576
What's under the ROC? An Introduction to Receiver Operating Characteristics Curves
Streiner, D. L., & Cairney, J. (2007) · The Canadian Journal of Psychiatry · doi:10.1177/070674370705200210 · Cited by 527
Osuchowski, M. F., Welch, K., Siddiqui, J., & Remick, D. G. (2006) · The Journal of Immunology · doi:10.4049/jimmunol.177.3.1967 · Cited by 510
The Immune Epitope Database and Analysis Resource in Epitope Discovery and Synthetic Vaccine Design
Fleri, W., Paul, S., Dhanda, S. K., Mahajan, S., Xu, X., Peters, B., & Sette, A. (2017) · Frontiers in Immunology · doi:10.3389/fimmu.2017.00278 · Cited by 500
Cellular Imaging Predictions of Clinical Drug-Induced Liver Injury
Xu, J. J., Henstock, P. V., Dunn, M. C., Smith, A. R., Chabot, J. R., & de Graaf, D. (2008) · Toxicological Sciences · doi:10.1093/toxsci/kfn109 · Cited by 492
Measuring and comparing the accuracy of species distribution models with presence–absence data
Liu, C., White, M., & Newell, G. (2011) · Ecography · doi:10.1111/j.1600-0587.2010.06354.x · Cited by 456
Forreryd, A., Zeller, K. S., Lindberg, T., Johansson, H., & Lindstedt, M. (2016) · Toxicology in Vitro · doi:10.1016/j.tiv.2016.09.013 · Cited by 438
Handelman, G. S., Kok, H. K., Chandra, R. V., Razavi, A. H., Huang, S., Brooks, M., Lee, M. J., & Asadi, H. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.18.20224 · Cited by 406
Van Den Eeckhaut, M., Vanwalleghem, T., Poesen, J., Govers, G., Verstraeten, G., & Vandekerckhove, L. (2006) · Geomorphology · doi:10.1016/j.geomorph.2005.12.003 · Cited by 403
Mining α-Helix-Forming Molecular Recognition Features with Cross Species Sequence Alignments
Cheng, Y., Oldfield, C. J., Meng, J., Romero, P., Uversky, V. N., & Dunker, A. K. (2007) · Biochemistry · doi:10.1021/bi7012273 · Cited by 323
Lodise, T. P., Patel, N., Kwa, A., Graves, J., Furuno, J. P., Graffunder, E., Lomaestro, B., & McGregor, J. C. (2007) · Antimicrobial Agents and Chemotherapy · doi:10.1128/aac.00338-07 · Cited by 318
Lasko, T. A., Denny, J. C., & Levy, M. A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0066341 · Cited by 313
Self-citationWild, B., Eckl, A., Herzog, W., Niehoff, D., Lechner, S., Maatouk, I., Schellberg, D., Brenner, H., Müller, H., & Löwe, B. (2014) · The American Journal of Geriatric Psychiatry · doi:10.1016/j.jagp.2013.01.076 · Cited by 305
Secondary use of clinical data: The Vanderbilt approach
Danciu, I., Cowan, J. D., Basford, M., Wang, X., Saip, A., Osgood, S., Shirey-Rice, J., Kirby, J., & Harris, P. A. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.02.003 · Cited by 277
Beyond the paradigm: Combining mass spectrometry and nuclear magnetic resonance for metabolomics
Marshall, D. D., & Powers, R. (2017) · Progress in Nuclear Magnetic Resonance Spectroscopy · doi:10.1016/j.pnmrs.2017.01.001 · Cited by 270
A novel feature selection approach for biomedical data classification
Peng, Y., Wu, Z., & Jiang, J. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.07.008 · Cited by 257
Careless responding in questionnaire measures: Detection, impact, and remedies
Goldammer, P., Annen, H., Stöckli, P. L., & Jonas, K. (2020) · The Leadership Quarterly · doi:10.1016/j.leaqua.2020.101384 · Cited by 200
Kim, S., Hwang, J., Xuan, J., Jung, Y. H., Cha, H.-S., & Kim, K. H. (2014) · PLoS ONE · doi:10.1371/journal.pone.0097501 · Cited by 169
Denton, C. A., Barth, A. E., Fletcher, J. M., Wexler, J., Vaughn, S., Cirino, P. T., Romain, M., & Francis, D. J. (2011) · Scientific Studies of Reading · doi:10.1080/10888431003623546 · Cited by 140
Verbakel, J. Y., Steyerberg, E. W., Uno, H., De Cock, B., Wynants, L., Collins, G. S., & Van Calster, B. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.028 · Cited by 138
Comparing the Areas Under Two Correlated ROC Curves: Parametric and Non‐Parametric Approaches
Molodianovitch, K., Faraggi, D., & Reiser, B. (2006) · Biometrical Journal · doi:10.1002/bimj.200610223 · Cited by 131
Mountcastle, S. E., Vyas, N., Villapun, V. M., Cox, S. C., Jabbari, S., Sammons, R. L., Shelton, R. M., Walmsley, A. D., & Kuehne, S. A. (2021) · npj Biofilms and Microbiomes · doi:10.1038/s41522-021-00214-7 · Cited by 130
A spline-based tool to assess and visualize the calibration of multiclass risk predictions
Van Hoorde, K., Van Huffel, S., Timmerman, D., Bourne, T., & Van Calster, B. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.12.016 · Cited by 85
A Survey of Gene Prioritization Tools for Mendelian and Complex Human Diseases
Zolotareva, O., & Kleine, M. (2019) · Journal of Integrative Bioinformatics · doi:10.1515/jib-2018-0069 · Cited by 46
Rojalin, T., Antonio, D., Kulkarni, A., & Carney, R. P. (2022) · Applied Spectroscopy · doi:10.1177/00037028211034543 · Cited by 27
“44 After hyperparameter tuning using the AUC score (walkthrough can be found in our Jupyter notebooks on GitHub), all six models were assessed by training on the training dataset and their performance validated using the validation set.”
MethodBrown, T. E., Brams, M., Gasior, M., Adeyi, B., Babcock, T., Dirks, B., Scheckner, B., & Wigal, T. (2011) · Current Medical Research and Opinion · doi:10.1185/03007995.2011.605441 · Cited by 24
Uchikawa, K., Furui, K., & Ohue, M. (2025) · Biochemistry and Biophysics Reports · doi:10.1016/j.bbrep.2025.102110 · Cited by 1
Metadata
- Authors
- Lasko, T. A., Bhagwat, J. G., Zou, K. H., & Ohno-Machado, L.
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2005-10
- Volume
- 38
- Issue
- 5
- Pages
- 404-415
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W2029426756
- Cited by
- 823
- References
- 49
- FWCI
- 28.51
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.