The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
ROC and AUC with a Binary Predictor: a Potentially Misleading Metric
Open accessAbstract
In analysis of binary outcomes, the receiver operator characteristic (ROC) curve is heavily used to show the performance of a model or algorithm. The ROC curve is informative about the performance over a series of thresholds and can be summarized by the area under the curve (AUC), a single number. When a predictor is categorical, the ROC curve has one less than number of categories as potential thresholds; when the predictor is binary, there is only one threshold. As the AUC may be used in decision-making processes on determining the best model, it important to discuss how it agrees with the intuition from the ROC curve. We discuss how the interpolation of the curve between thresholds with binary predictors can largely change the AUC. Overall, we show using a linear interpolation from the ROC curve with binary predictors corresponds to the estimated AUC, which is most commonly done in software, which we believe can lead to misleading results. We compare R, Python, Stata, and SAS software implementations. We recommend using reporting the interpolation used and discuss the merit of using the step function interpolator, also referred to as the “pessimistic” approach by Fawcett (2006).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Matplotlib: a 2D graphics environment
Hunter, J. D. (2007) · Computing in Science & Engineering · doi:10.1109/mcse.2007.55 · Cited by 39,249
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
pROC: an open-source package for R and S+ to analyze and compare ROC curves
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-77 · Cited by 13,697
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309
ROCR: visualizing classifier performance in R
Sing, T., Sander, O., Beerenwinkel, N., & Lengauer, T. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti623 · Cited by 3,200
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,439
Deep learning improves antimicrobial peptide recognition
Veltri, D., Kamath, U., & Shehu, A. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty179 · Cited by 593
Diagnostic Criteria of Ulcerative Pyoderma Gangrenosum: A Delphi Consensus of International Experts
Maverakis, E., Ma, C., Shinkai, K., Fiorentino, D., Callen, J. P., Wollina, U., Marzano, A. V., Wallach, D., Kim, K., Schadt, C., et al. (2018) · JAMA Dermatology · doi:10.1001/jamadermatol.2017.5980 · Cited by 452
Estimation and Comparison of Receiver Operating Characteristic Curves
Pepe, M. S., Longton, G., & Janes, H. (2009) · The Stata Journal: Promoting communications on statistics and Stata · doi:10.1177/1536867x0900900101 · Cited by 306
Mwipatayi, B. P., Sharma, S., Daneshmand, A., Thomas, S. D., Vijayan, V., Altaf, N., Garbowski, M., Jackson, M., Benveniste, G., Denton, M., et al. (2016) · Journal of Vascular Surgery · doi:10.1016/j.jvs.2016.02.064 · Cited by 250
Kushnir, V. A., Darmon, S. K., Barad, D. H., & Gleicher, N. (2018) · Reproductive Biology and Endocrinology · doi:10.1186/s12958-018-0322-5 · Cited by 58
Xiong, X., Li, Q., Yang, W., Xiao, W., Hu, X., Wang, X.-C., Zhu, D., Li, R., Cao, D., & Xie, P. (2018) · Medical Science Monitor · doi:10.12659/msm.906708 · Cited by 42
A Revised Approach for the Detection of Sight-Threatening Diabetic Macular Edema
Litvin, T. V., Bresnick, G. H., Cuadros, J. A., Selvin, S., Kanai, K., & Ozawa, G. Y. (2016) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2016.4772 · Cited by 19
Factors associated with significant MRI findings in medical walk-in patients with acute headache
Budweg, J., Sprenger, T., Vere-Tyndall, A. D., Hagenkord, A., Stippich, C., & Berger, C. T. (2016) · Swiss Medical Weekly · doi:10.4414/smw.2016.14349 · Cited by 10
Cited by
Chicco, D., & Jurman, G. (2023) · BioData Mining · doi:10.1186/s13040-023-00322-4 · Cited by 597
Understanding metric-related pitfalls in image analysis validation
Reinke, A., Tizabi, M. D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A. E., Rädsch, T., Sudre, C. H., Acion, L., Antonelli, M., et al. (2024) · Nature Methods · doi:10.1038/s41592-023-02150-0 · Cited by 241
We need more randomized clinical trials of AI.
Ouyang, D., & Hogan, J. (2024) · NEJM AI · doi:10.1056/aie2400881 · Cited by 24
Metadata
- Authors
- Muschelli, J.
- Venue
- Journal of Classification
- Published
- 2019-12-23
- Volume
- 37
- Issue
- 3
- Pages
- 696-708
- ISSN
- 0176-4268 (ISSN-L)
- 1432-1343
- OpenAlex
- W2921539374
- Cited by
- 293
- References
- 17
- FWCI
- 6.56
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.