A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

Hand, David J.; Till, Robert J.

doi:10.1023/a:1010920819831

Cited by 2,276Imbalanced Data Classification TechniquesMachine Learning and Data ClassificationData Mining and Machine Learning Applications

The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.

A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

Open access

Hand, David J., Till, Robert J.·2001·Machine Learning

DOI 10.1023/a:1010920819831

Open this paper in CoChat — free account Open PDF

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1023/a:1010920819831

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

The time complexity for computing the AUC is with respect to the Hand and Till AUC model and according to Provost and Domingo's AUC model.
“In addition, the time complexity for computing the AUC is with respect to the Hand and Till AUC model [341] and according to Provost and Domingo’s AUC model [336].”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [341]
Section 3 (“Multiple class extensions”; page unavailable) describes the Hand–Till method for computing a multiclass AUC measure. The cited paper does not give a time-complexity analysis for computing it, so it supports the identification of the model but not the attributed complexity point. (LLM, conf=0.78)
Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,858
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 21,030
Classification And Regression Trees
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (2017) · doi:10.1201/9781315139470 · Cited by 11,417
UCI Repository of machine learning databases
Blake, C. L. (1998) · Medical Entomology and Zoology · Cited by 10,582
The use of the area under the ROC curve in the evaluation of machine learning algorithms
Bradley, A. P. (1997) · Pattern Recognition · doi:10.1016/s0031-3203(96)00142-2 · Cited by 7,217
Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine
ZWEIG, M. H., & Campbell, G. (1993) · Clinical Chemistry · doi:10.1093/clinchem/39.4.561 · Cited by 6,584
Modern Applied Statistics with S-Plus.
Venables, W. N., & Ripley, B. D. (1996) · Biometrics · doi:10.2307/2532871 · Cited by 3,012
The Case against Accuracy Estimation for Comparing Induction Algorithms
Provost, F. J., Fawcett, T. E., & Kohavi, R. (1998) · Cited by 1,031
Modern Applied Statistics with S-Plus
Venables, W. N., & Ripley, B. D. (1994) · Statisctics and computing/Statistics and computing · doi:10.1007/978-1-4899-2819-1 · Cited by 802
Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions
Provost, F. J., & Fawcett, T. E. (1997) · Cited by 723
Construction and Assessment of Classification Rules
Wade, M. R., & Hand, D. J. (1999) · Technometrics · doi:10.2307/1270579 · Cited by 621
Self-citation
Construction and Assessment of Classification Rules.
Ringrose, T. J., & Hand, D. J. (1997) · Biometrics · doi:10.2307/2533581 · Cited by 490
Self-citation
On fully automatic feature measurement for banded chromosome classification
Piper, J., & Granum, E. (1989) · Cytometry · doi:10.1002/cyto.990100303 · Cited by 189
Comparing classifiers when the misallocation costs are uncertain
Adams, N. M., & Hand, D. J. (1999) · Pattern Recognition · doi:10.1016/s0031-3203(98)00154-x · Cited by 171
Self-citation
Robust classification systems for imprecise environments
Provost, F. J., & Fawcett, T. E. (1998) · Cited by 121
Improving the Practice of Classifier Performance Assessment
Adams, N. M., & Hand, D. J. (2000) · Neural Computation · doi:10.1162/089976600300015808 · Cited by 46
Self-citation
Measuring Diagnostic Accuracy of Statistical Prediction Rules
Hand, D. J. (2001) · Statistica Neerlandica · doi:10.1111/1467-9574.00153 · Cited by 42
Self-citation
An evaluation of four serum tests for pregnancy.
Ryder, K. W., Munsick, R. A., Oei, T. O., et al. (1983) · Clinical Chemistry · doi:10.1093/clinchem/29.3.561 · Cited by 10
Finding 'superclassifications' with an acceptable misclassification rate
Taylor, P. C., & Hand, D. J. (1999) · Journal of Applied Statistics · doi:10.1080/02664769922241 · Cited by 5
Self-citation

Flagged first, then by citations

Research works printed but not linked: 3

Efron, B.& Tibshirani, R. J. (1995). Cross-validation and the bootstrap: Estimating the error rate of a prediction rule. Technical Report 176. Stanford, CA: Stanford University, Department of Statistics.
Provost, F. J., Fawcett, T.,& Kohavi, R. (1998). The case against accuracy estimation for comparing classifiers. In Proceedings of the Fifteenth International Conference on Machine Learning (pp. 445–453). Madison, WI: Morgan Kaufmann.
Scott, M. (1999). Parcel: Feature selection in variable cost domains. Doctoral Dissertation, Engineering Department, Cambridge University, UK.

Sorted by citations

An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
Learning from Imbalanced Data
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Fake News Detection on Social Media: A Data Mining Perspective
Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, H. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3137597.3137600 · Cited by 3,304
Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network
Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H., Bourn, C., Turakhia, M. P., & Ng, A. Y. (2019) · Nature Medicine · doi:10.1038/s41591-018-0268-3 · Cited by 3,148
Using AUC and accuracy in evaluating learning algorithms
Huang, J., & Ling, C. X. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.50 · Cited by 2,148
Locality Sensitive Deep Learning for Detection and Classification of Nuclei in Routine Colon Cancer Histology Images
Sirinukunwattana, K., Raza, S. E. A., Tsang, Y.-W., Snead, D. R. J., Cree, I. A., & Rajpoot, N. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2525803 · Cited by 1,224
Interpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687
Calibrating Probability with Undersampling for Unbalanced Classification
Pozzolo, A. D., Caelen, O., Johnson, R. A., & Bontempi, G. (2015) · 2015 IEEE Symposium Series on Computational Intelligence · doi:10.1109/ssci.2015.33 · Cited by 667
Tree Induction for Probability-Based Ranking
Provost, F., & Domingos, P. (2003) · Machine Learning · doi:10.1023/a:1024099825458 · Cited by 541
An up-to-date comparison of state-of-the-art classification algorithms
Zhang, C., Liu, C., Zhang, X., & Almpanidis, G. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.04.003 · Cited by 440
A Comparison of MCC and CEN Error Measures in Multi-Class Prediction
Jurman, G., Riccadonna, S., & Furlanello, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041882 · Cited by 406
Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: the CADDementia challenge
Bron, E. E., Smits, M., van der Flier, W. M., Vrenken, H., Barkhof, F., Scheltens, P., Papma, J. M., Steketee, R. M. E., Méndez Orellana, C., Meijboom, R., et al. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.01.048 · Cited by 352
Small-sample precision of ROC-related estimates
Hanczar, B., Hua, J., Sima, C., Weinstein, J., Bittner, M., & Dougherty, E. R. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq037 · Cited by 323
Psychological distress associated with COVID-19 quarantine: latent profile analysis, outcome prediction and mediation analysis
Fernández, R. S., Crivelli, L., Guimet, N. M., Allegri, R. F., & Pedreira, M. E. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.07.133 · Cited by 245
Using the Census Bureau’s surname list to improve estimates of race/ethnicity and associated disparities
Elliott, M. N., Morrison, P. A., Fremont, A., McCaffrey, D. F., Pantoja, P., & Lurie, N. (2009) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-009-0047-1 · Cited by 224
Are white storks addicted to junk food? Impacts of landfill use on the movement and behaviour of resident white storks (Ciconia ciconia) from a partially migratory population
Gilbert, N. I., Correia, R. A., Silva, J. P., Pacheco, C., Catry, I., Atkinson, P. W., Gill, J. A., & Franco, A. M. A. (2016) · Movement Ecology · doi:10.1186/s40462-016-0070-0 · Cited by 217
Explainable decision forest: Transforming a decision forest into an interpretable tree
Sagi, O., & Rokach, L. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.03.013 · Cited by 189
A Comparison of Bacterial Composition in Diabetic Ulcers and Contralateral Intact Skin
Gontcharova, V., Youn, E., Sun, Y., Wolcott, R. D., & Dowd, S. E. (2010) · The Open Microbiology Journal · doi:10.2174/1874285801004010008 · Cited by 156
Characterisation of liver fat in the UK Biobank cohort
Wilman, H. R., Kelly, M., Garratt, S., Matthews, P. M., Milanesi, M., Herlihy, A., Gyngell, M., Neubauer, S., Bell, J. D., Banerjee, R., et al. (2017) · PLOS ONE · doi:10.1371/journal.pone.0172921 · Cited by 159
The successful discrimination of depression from EEG could be attributed to proper feature extraction and not to a particular classification method
Čukić, M., Stokić, M., Simić, S., & Pokrajac, D. (2020) · Cognitive Neurodynamics · doi:10.1007/s11571-020-09581-x · Cited by 130
Support vector machine for diagnosis Cancer disease: a comparative study
Sweilam, N. H., Tharwat, A. A., & Abdel Moniem, N. K. (2010) · Egyptian Informatics Journal · doi:10.1016/j.eij.2010.10.005 · Cited by 131
Clinical text classification research trends: Systematic literature review and open issues
Mujtaba, G., Shuib, L., Idris, N., Hoo, W. L., Raj, R. G., Khowaja, K., Shaikh, K., & Nweke, H. F. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.09.034 · Cited by 123
Query-by-committee improvement with diversity and density in batch active learning
Kee, S., del Castillo, E., & Runger, G. (2018) · Information Sciences · doi:10.1016/j.ins.2018.05.014 · Cited by 98
A spline-based tool to assess and visualize the calibration of multiclass risk predictions
Van Hoorde, K., Van Huffel, S., Timmerman, D., Bourne, T., & Van Calster, B. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.12.016 · Cited by 85
Happiness Recognition from Mobile Phone Data
Bogomolov, A., Lepri, B., & Pianesi, F. (2013) · 2013 International Conference on Social Computing · doi:10.1109/socialcom.2013.118 · Cited by 67
Bayesian classifiers based on probability density estimation and their applications to simultaneous fault diagnosis
He, Y.-L., Wang, R., Kwong, S., & Wang, X.-Z. (2014) · Information Sciences · doi:10.1016/j.ins.2013.09.003 · Cited by 64
Deep learning enabled classification of Mayo endoscopic subscore in patients with ulcerative colitis
Bhambhvani, H. P., & Zamora, A. (2021) · European Journal of Gastroenterology & Hepatology · doi:10.1097/meg.0000000000001952 · Cited by 59
Five-class differential diagnostics of neurodegenerative diseases using random undersampling boosting
Tong, T., Ledig, C., Guerrero, R., Schuh, A., Koikkalainen, J., Tolonen, A., Rhodius, H., Barkhof, F., Tijms, B., Lemstra, A. W., et al. (2017) · NeuroImage: Clinical · doi:10.1016/j.nicl.2017.06.012 · Cited by 54
Temporal feature aggregation with attention for insider threat detection from activity logs
Pal, P., Chattopadhyay, P., & Swarnkar, M. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119925 · Cited by 52
On the effect of calibration in classifier combination
Bella, A., Ferri, C., Hernández-Orallo, J., & Ramírez-Quintana, M. J. (2013) · Applied Intelligence · doi:10.1007/s10489-012-0388-2 · Cited by 45
Unsupervised Feature Selection for Outlier Detection by Modelling Hierarchical Value-Feature Couplings
Pang, G., Cao, L., Chen, L., & Liu, H. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0052 · Cited by 43
Identifying progression-specific Alzheimer’s subtypes using multimodal transformer
Machado Reyes, D., Chao, H., Hahn, J., Shen, L., Yan, P., & Initiative, F. T. A. D. N. (2024) · Journal of Personalized Medicine · doi:10.3390/jpm14040421 · Cited by 17
Chronological Age Assessment in Young Individuals Using Bone Age Assessment Staging and Nonradiological Aspects: Machine Learning Multifactorial Approach
Dallora, A. L., Kvist, O., Berglund, J. S., Ruiz, S. D., Boldt, M., Flodmark, C.-E., & Anderberg, P. (2020) · JMIR Medical Informatics · doi:10.2196/18846 · Cited by 13
Forecasting the progression of Alzheimer's disease using neural networks and a novel preprocessing algorithm
Albright, J., & Initiative, A. D. N. (2019) · Alzheimer's & Dementia: Translational Research & Clinical Interventions · doi:10.1016/j.trci.2019.07.001 · Cited by 104

Authors: Hand, D. J., & Till, R. J.
Affiliations: Imperial College London (GB)
ORCID: Hand, David J.
Venue: Machine Learning
Publisher: Springer Science and Business Media LLC
Published: 2001-11
Volume: 45
Issue: 2
Pages: 171-186
ISSN: 0885-6125 (ISSN-L)
1573-0565
DOI: 10.1023/a:1010920819831
OpenAlex: W1912982817
Cited by: 2,276
References: 22
Citations by year: 2026: 153 · 2025: 148 · 2024: 142 · 2023: 157 · 2022: 174 · 2021: 188 · 2020: 141 · 2019: 134 · 2018: 121 · 2017: 90 · 2016: 87 · 2015: 87 · 2014: 88 · 2013: 81 · 2012: 84
FWCI: 16.01
Type: journal article
Language: en
Open access: bronze
doi.org
Licence: www.springernature.com/gp/researchers/text-and-data-mining (tdm)
www.springernature.com/gp/researchers/text-and-data-mining (vor)
Locations: Machine Learning (publishedVersion)
Funders: Royal Society
Engineering and Physical Sciences Research Council
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
Keywords: area under the ROC curve, Gini coefficient, multi-class classification, pairwise comparison, classification performance measures

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks