The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Deep ROC Analysis and AUC as Balanced Average Accuracy, for Improved Classifier Selection, Audit and Explanation
Open accessAbstract
Optimal performance is desired for decision-making in any field with binary classifiers and diagnostic tests, however common performance measures lack depth in information. The area under the receiver operating characteristic curve (AUC) and the area under the precision recall curve are too general because they evaluate all decision thresholds including unrealistic ones. Conversely, accuracy, sensitivity, specificity, positive predictive value and the F1 score are too specific-they are measured at a single threshold that is optimal for some instances, but not others, which is not equitable. In between both approaches, we propose deep ROC analysis to measure performance in multiple groups of predicted risk (like calibration), or groups of true positive rate or false positive rate. In each group, we measure the group AUC (properly), normalized group AUC, and averages of: sensitivity, specificity, positive and negative predictive value, and likelihood ratio positive and negative. The measurements can be compared between groups, to whole measures, to point measures and between models. We also provide a new interpretation of AUC in whole or part, as balanced average accuracy, relevant to individuals instead of pairs. We evaluate models in three case studies using our method and Python toolkit and confirm its utility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
A systematic analysis of performance measures for classification tasks
Sokolova, M., & Lapalme, G. (2009) · Information Processing & Management · doi:10.1016/j.ipm.2009.03.002 · Cited by 7,515
The use of the area under the ROC curve in the evaluation of machine learning algorithms
Bradley, A. P. (1997) · Pattern Recognition · doi:10.1016/s0031-3203(96)00142-2 · Cited by 7,217
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Basic principles of ROC analysis
Metz, C. E. (1978) · Seminars in Nuclear Medicine · doi:10.1016/s0001-2998(78)80014-2 · Cited by 6,026
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
AUC: A misleading measure of the performance of predictive distribution models
Lobo, J. M., Jiménez‐Valverde, A., & Real, R. (2008) · Global Ecology and Biogeography · doi:10.1111/j.1466-8238.2007.00358.x · Cited by 3,499
STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration
Cohen, J. F., Korevaar, D. A., Altman, D. G., Bruns, D. E., Gatsonis, C. A., Hooft, L., Irwig, L., Levine, D., Reitsma, J. B., de Vet, H. C. W., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-012799 · Cited by 2,570
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems
Hand, D. J., & Till, R. J. (2001) · Machine Learning · doi:10.1023/a:1010920819831 · Cited by 2,276
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation
Sokolova, M., Japkowicz, N., & Szpakowicz, S. (2006) · Lecture Notes in Computer Science · doi:10.1007/11941439_114 · Cited by 1,974
Towards better clinical prediction models: seven steps for development and an ABCD for validation
Steyerberg, E. W., & Vergouwe, Y. (2014) · European Heart Journal · doi:10.1093/eurheartj/ehu207 · Cited by 1,928
Causability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885 · cited 2×
“ROC analysis is one of multiple tools we use to understand and explain models and their test results, as a goal of explainable artificial intelligence (xAI) [25], [26].”
MethodSelf-citationPredicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884
Diagnostic tests 4: likelihood ratios
Deeks, J. J., & Altman, D. G. (2004) · BMJ · doi:10.1136/bmj.329.7458.168 · Cited by 1,600
Uno, H., Cai, T., Pencina, M. J., D'Agostino, R. B., & Wei, L. J. (2011) · Statistics in Medicine · doi:10.1002/sim.4154 · Cited by 1,535 · cited 1×
“The C statistic for binary outcomes [2], [58] we refer to should not be confused with Harrell or Uno’s C statistics for continuous outcomes [59], [60], [61].”
MethodSteyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491
Statistical Methods in Diagnostic Medicine
Zhou, X., Obuchowski, N. A., & McClish, D. K. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9780470317082 · Cited by 1,165
A note on Platt’s probabilistic outputs for support vector machines
Lin, H.-T., Lin, C.-J., & Weng, R. C. (2007) · Machine Learning · doi:10.1007/s10994-007-5018-6 · Cited by 867
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Circulation · doi:10.1161/circulationaha.114.014508 · Cited by 793
Analyzing a Portion of the ROC Curve
McClish, D. K. (1989) · Medical Decision Making · doi:10.1177/0272989x8900900307 · Cited by 707
Statistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC Curve
Cook, N. R. (2008) · Clinical Chemistry · doi:10.1373/clinchem.2007.096529 · Cited by 691
Obuchowski, N. A., & Bullen, J. A. (2018) · Physics in Medicine & Biology · doi:10.1088/1361-6560/aab4b1 · Cited by 542
Holzinger, A., Malle, B., Saranti, A., & Pfeifer, B. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.01.008 · Cited by 352
Self-citationPartial AUC Estimation and Regression
Dodd, L. E., & Pepe, M. S. (2003) · Biometrics · doi:10.1111/1541-0420.00071 · Cited by 346
Making Machine Learning Models Clinically Useful
Shah, N. H., Milstein, A., & Bagley, P. (2019) · JAMA · doi:10.1001/jama.2019.10306 · Cited by 294
A receiver operating characteristic partial area index for highly sensitive diagnostic tests.
Jiang, Y., Metz, C. E., & Nishikawa, R. M. (1996) · Radiology · doi:10.1148/radiology.201.3.8939225 · Cited by 275
Austin, P. C., & Steyerberg, E. W. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-82 · Cited by 270
Interpreting diagnostic accuracy studies for patient care
Mallett, S., Halligan, S., Thompson, M., Collins, G. S., & Altman, D. G. (2012) · BMJ · doi:10.1136/bmj.e3999 · Cited by 248
Schumacher, M., Bastert, G., Bojar, H., Hübner, K., Olschewski, M., Sauerbrei, W., Schmoor, C., Beyerle, C., Neumann, R. L., & Rauschecker, H. F. (1994) · Journal of Clinical Oncology · doi:10.1200/jco.1994.12.10.2086 · Cited by 242
On the statistical analysis of ROC curves
Thompson, M. L., & Zucchini, W. (1989) · Statistics in Medicine · doi:10.1002/sim.4780081011 · Cited by 225
Index of Balanced Accuracy: A Performance Measure for Skewed Class Distributions
García, V., Mollineda, R. A., & Sánchez, J. S. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02172-5_57 · Cited by 214
Analysis of sampling techniques for imbalanced data: An n = 648 ADNI study
Dubey, R., Zhou, J., Wang, Y., Thompson, P. M., & Ye, J. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.10.005 · Cited by 204
On the performance of Matthews correlation coefficient (MCC) for imbalanced dataset
Zhu, Q. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.03.030 · Cited by 202
Holzinger, A., Kieseberg, P., Weippl, E., & Tjoa, A. M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-99740-7_1 · Cited by 201
Self-citationThe Area under the ROC Curve and Its Competitors
Hilden, J. (1991) · Medical Decision Making · doi:10.1177/0272989x9101100204 · Cited by 185
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179
A Small Sphere and Large Margin Approach for Novelty Detection Using Training Data with Outliers
Wu, M., & Ye, J. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.24 · Cited by 176
Statistical significance tests for binormal ROC curves
Metz, C. E., & Kronman, H. B. (1980) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(80)90020-6 · Cited by 175
Performance Evaluation in Machine Learning: The Good, the Bad, the Ugly, and the Way Forward
Flach, P. A. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019808 · Cited by 175
Carrington, A., Fieguth, P., Qazi, H. A., Holzinger, A., Chen, H., Mayr, F., & Manuel, D. G. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-1014-6 · Cited by 174
Self-citationDealing with the evaluation of supervised classification algorithms
Santafé, G., Inza, I., & Lozano, J. A. (2015) · Artificial Intelligence Review · doi:10.1007/s10462-015-9433-y · Cited by 171
An Algorithm for Isotonic Regression for Two or More Independent Variables
Dykstra, R. L., & Robertson, T. (1982) · The Annals of Statistics · doi:10.1214/aos/1176345866 · Cited by 138
Pairwise Fairness for Ranking and Regression
Narasimhan, H., Cotter, A., Gupta, M. R., & Wang, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5970 · Cited by 137
Efficient Multiclass ROC Approximation by Decomposition via Confusion Matrix Perturbation Analysis
Landgrebe, T. C. W., & Duin, R. P. W. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.70740 · Cited by 124
Rosset, S. (2004) · International Conference on Machine Learning · doi:10.1145/1015330.1015400 · Cited by 119
Everything you always wanted to know about evaluating prediction models (but were too afraid to ask)
Vickers, A. J., & Cronin, A. M. (2010) · Urology · doi:10.1016/j.urology.2010.06.019 · Cited by 112
Why Does Rebalancing Class-Unbalanced Data Improve AUC for Linear Discriminant Analysis?
Xue, J., & Hall, P. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2359660 · Cited by 78
Defining ranges for certainty ratings of diagnostic accuracy: a GRADE concept paper
Hultcrantz, M., Mustafa, R. A., Leeflang, M., Lavergne, V., Estrada-Orozco, K., Ansari, M., Izcovich, A., Singh, J. A., Chong, L. Y., Rutjes, A. W., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.05.002 · Cited by 38
nsROC: An R package for Non-Standard ROC Curve Analysis
Pérez‐Fernández, S., Martínez‐Camblor, P., Peter Filzmoser, A., & Corral, N. (2019) · The R Journal · doi:10.32614/rj-2018-043 · Cited by 30
Measures of Model Interpretability for Model Selection
Carrington, A., Fieguth, P., & Chen, H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99740-7_24 · Cited by 18
Self-citationVivo, J.-M., Franco, M., & Vicari, D. (2017) · Advances in Data Analysis and Classification · doi:10.1007/s11634-017-0295-9 · Cited by 13
Half-AUC for the evaluation of sensitive or specific classifiers
Bradley, A. P. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2013.11.015 · Cited by 9
Cited by
Yang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717
Albahri, A. S., Duhaim, A. M., Fadhel, M. A., Alnoor, A., Baqer, N. S., Alzubaidi, L., Albahri, O. S., Alamoodi, A. H., Bai, J., Salhi, A., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.03.008 · Cited by 716
AI for life: Trends in artificial intelligence for biotechnology
Holzinger, A., Keiblinger, K., Holub, P., Zatloukal, K., & Müller, H. (2023) · New Biotechnology · doi:10.1016/j.nbt.2023.02.001 · Cited by 438
Self-citationA review on the recent applications of deep learning in predictive drug toxicological studies
Sinha, K., Ghosh, N., & Sil, P. C. (2023) · Chemical Research in Toxicology · doi:10.1021/acs.chemrestox.2c00375 · Cited by 64
Marchesi, R., Micheletti, N., I-Hsien Kuo, N., Barbieri, S., Jurman, G., & Osmani, V. (2025) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1013080 · Cited by 29
Self-citation
Metadata
- Authors
- Carrington, A. M., Manuel, D. G., Fieguth, P. W., Ramsay, T., Osmani, V., Wernly, B., Bennett, C., Hawken, S., Magwood, O., Sheikh, Y., McInnes, M., & Holzinger, A.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2023-01-01
- Volume
- 45
- Issue
- 1
- Pages
- 329-341
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4210494925
- Cited by
- 339
- References
- 55
- FWCI
- 21.41
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.