A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The time complexity for computing the AUC is with respect to the Hand and Till AUC model and according to Provost and Domingo's AUC model.
“In addition, the time complexity for computing the AUC is with respect to the Hand and Till AUC model [341] and according to Provost and Domingo’s AUC model [336].”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [341]
Section 3 (“Multiple class extensions”; page unavailable) describes the Hand–Till method for computing a multiclass AUC measure. The cited paper does not give a time-complexity analysis for computing it, so it supports the identification of the model but not the attributed complexity point. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500 · see citing works
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,834 · see citing works
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · see citing works
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230 · see citing works
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 21,028 · see citing works
Classification And Regression Trees
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (2017) · doi:10.1201/9781315139470 · Cited by 11,417 · see citing works
UCI Repository of machine learning databases
Blake, C. L. (1998) · Medical Entomology and Zoology · Cited by 10,491 · see citing works
The use of the area under the ROC curve in the evaluation of machine learning algorithms
Bradley, A. P. (1997) · Pattern Recognition · doi:10.1016/s0031-3203(96)00142-2 · Cited by 7,217 · see citing works
Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine
ZWEIG, M. H., & Campbell, G. (1993) · Clinical Chemistry · doi:10.1093/clinchem/39.4.561 · Cited by 6,528 · see citing works
Modern Applied Statistics with S-Plus.
Venables, W. N., & Ripley, B. D. (1996) · Biometrics · doi:10.2307/2532871 · Cited by 3,012 · see citing works
The Case against Accuracy Estimation for Comparing Induction Algorithms
Provost, F. J., Fawcett, T. E., & Kohavi, R. (1998) · Cited by 1,018 · see citing works
Modern Applied Statistics with S-Plus
Venables, W. N., & Ripley, B. D. (1994) · Statisctics and computing/Statistics and computing · doi:10.1007/978-1-4899-2819-1 · Cited by 802 · see citing works
Provost, F. J., & Fawcett, T. E. (1997) · Cited by 723 · see citing works
Construction and Assessment of Classification Rules
Wade, M. R., & Hand, D. J. (1999) · Technometrics · doi:10.2307/1270579 · Cited by 621 · see citing works
Self-citationConstruction and Assessment of Classification Rules.
Ringrose, T. J., & Hand, D. J. (1997) · Biometrics · doi:10.2307/2533581 · Cited by 490 · see citing works
Self-citationOn fully automatic feature measurement for banded chromosome classification
Piper, J., & Granum, E. (1989) · Cytometry · doi:10.1002/cyto.990100303 · Cited by 189 · see citing works
Comparing classifiers when the misallocation costs are uncertain
Adams, N. M., & Hand, D. J. (1999) · Pattern Recognition · doi:10.1016/s0031-3203(98)00154-x · Cited by 171 · see citing works
Self-citationRobust classification systems for imprecise environments
Provost, F. J., & Fawcett, T. E. (1998) · Cited by 121 · see citing works
Improving the Practice of Classifier Performance Assessment
Adams, N. M., & Hand, D. J. (2000) · Neural Computation · doi:10.1162/089976600300015808 · Cited by 46 · see citing works
Self-citationMeasuring Diagnostic Accuracy of Statistical Prediction Rules
Hand, D. J. (2001) · Statistica Neerlandica · doi:10.1111/1467-9574.00153 · Cited by 42 · see citing works
Self-citationAn evaluation of four serum tests for pregnancy.
Ryder, K. W., Munsick, R. A., Oei, T. O., et al. (1983) · Clinical Chemistry · doi:10.1093/clinchem/29.3.561 · Cited by 10 · see citing works
Finding 'superclassifications' with an acceptable misclassification rate
Taylor, P. C., & Hand, D. J. (1999) · Journal of Applied Statistics · doi:10.1080/02664769922241 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- Provost, F. J., Fawcett, T., & Kohavi, R. (1998). The case against accuracy estimation for comparing classifiers. In Proceedings of the Fifteenth International Conference on Machine Learning (pp. 445-453). Madison, WI: Morgan Kaufmann.
- Scott, M. (1999). Parcel: Feature selection in variable cost domains. Doctoral Dissertation, Engineering Depart- ment, Cambridge University, UK.
- Taylor, P. C. & Hand, D. J. (1999). Finding superclassifications with acceptable misclassification rates. Journal of Applied Statistics, 26, 579-590.
Cited by
Sorted by citations
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879 · see citing works
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359 · see citing works
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906 · see citing works
Fake News Detection on Social Media: A Data Mining Perspective
Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, H. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3137597.3137600 · Cited by 3,304 · see citing works
Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H., Bourn, C., Turakhia, M. P., & Ng, A. Y. (2019) · Nature Medicine · doi:10.1038/s41591-018-0268-3 · Cited by 3,148 · see citing works
Using AUC and accuracy in evaluating learning algorithms
Huang, J., & Ling, C. X. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.50 · Cited by 2,148 · see citing works
Sirinukunwattana, K., Raza, S. E. A., Tsang, Y.-W., Snead, D. R. J., Cree, I. A., & Rajpoot, N. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2525803 · Cited by 1,224 · see citing works
Interpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687 · see citing works
Calibrating Probability with Undersampling for Unbalanced Classification
Pozzolo, A. D., Caelen, O., Johnson, R. A., & Bontempi, G. (2015) · 2015 IEEE Symposium Series on Computational Intelligence · doi:10.1109/ssci.2015.33 · Cited by 667 · see citing works
Tree Induction for Probability-Based Ranking
Provost, F., & Domingos, P. (2003) · Machine Learning · doi:10.1023/a:1024099825458 · Cited by 541 · see citing works
An up-to-date comparison of state-of-the-art classification algorithms
Zhang, C., Liu, C., Zhang, X., & Almpanidis, G. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.04.003 · Cited by 440 · see citing works
A Comparison of MCC and CEN Error Measures in Multi-Class Prediction
Jurman, G., Riccadonna, S., & Furlanello, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041882 · Cited by 406 · see citing works
Bron, E. E., Smits, M., van der Flier, W. M., Vrenken, H., Barkhof, F., Scheltens, P., Papma, J. M., Steketee, R. M. E., Méndez Orellana, C., Meijboom, R., et al. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.01.048 · Cited by 352 · see citing works
Small-sample precision of ROC-related estimates
Hanczar, B., Hua, J., Sima, C., Weinstein, J., Bittner, M., & Dougherty, E. R. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq037 · Cited by 323 · see citing works
Fernández, R. S., Crivelli, L., Guimet, N. M., Allegri, R. F., & Pedreira, M. E. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.07.133 · Cited by 245 · see citing works
Elliott, M. N., Morrison, P. A., Fremont, A., McCaffrey, D. F., Pantoja, P., & Lurie, N. (2009) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-009-0047-1 · Cited by 224 · see citing works
Gilbert, N. I., Correia, R. A., Silva, J. P., Pacheco, C., Catry, I., Atkinson, P. W., Gill, J. A., & Franco, A. M. A. (2016) · Movement Ecology · doi:10.1186/s40462-016-0070-0 · Cited by 217 · see citing works
Explainable decision forest: Transforming a decision forest into an interpretable tree
Sagi, O., & Rokach, L. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.03.013 · Cited by 189 · see citing works
Characterisation of liver fat in the UK Biobank cohort
Wilman, H. R., Kelly, M., Garratt, S., Matthews, P. M., Milanesi, M., Herlihy, A., Gyngell, M., Neubauer, S., Bell, J. D., Banerjee, R., et al. (2017) · PLOS ONE · doi:10.1371/journal.pone.0172921 · Cited by 159 · see citing works
A Comparison of Bacterial Composition in Diabetic Ulcers and Contralateral Intact Skin
Gontcharova, V., Youn, E., Sun, Y., Wolcott, R. D., & Dowd, S. E. (2010) · The Open Microbiology Journal · doi:10.2174/1874285801004010008 · Cited by 156 · see citing works
Čukić, M., Stokić, M., Simić, S., & Pokrajac, D. (2020) · Cognitive Neurodynamics · doi:10.1007/s11571-020-09581-x · Cited by 130 · see citing works
Support vector machine for diagnosis Cancer disease: a comparative study
Sweilam, N. H., Tharwat, A. A., & Abdel Moniem, N. K. (2010) · Egyptian Informatics Journal · doi:10.1016/j.eij.2010.10.005 · Cited by 131 · see citing works
Clinical text classification research trends: Systematic literature review and open issues
Mujtaba, G., Shuib, L., Idris, N., Hoo, W. L., Raj, R. G., Khowaja, K., Shaikh, K., & Nweke, H. F. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.09.034 · Cited by 123 · see citing works
Query-by-committee improvement with diversity and density in batch active learning
Kee, S., del Castillo, E., & Runger, G. (2018) · Information Sciences · doi:10.1016/j.ins.2018.05.014 · Cited by 98 · see citing works
A spline-based tool to assess and visualize the calibration of multiclass risk predictions
Van Hoorde, K., Van Huffel, S., Timmerman, D., Bourne, T., & Van Calster, B. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.12.016 · Cited by 85 · see citing works
Happiness Recognition from Mobile Phone Data
Bogomolov, A., Lepri, B., & Pianesi, F. (2013) · 2013 International Conference on Social Computing · doi:10.1109/socialcom.2013.118 · Cited by 67 · see citing works
He, Y.-L., Wang, R., Kwong, S., & Wang, X.-Z. (2014) · Information Sciences · doi:10.1016/j.ins.2013.09.003 · Cited by 64 · see citing works
Deep learning enabled classification of Mayo endoscopic subscore in patients with ulcerative colitis
Bhambhvani, H. P., & Zamora, A. (2021) · European Journal of Gastroenterology & Hepatology · doi:10.1097/meg.0000000000001952 · Cited by 59 · see citing works
Tong, T., Ledig, C., Guerrero, R., Schuh, A., Koikkalainen, J., Tolonen, A., Rhodius, H., Barkhof, F., Tijms, B., Lemstra, A. W., et al. (2017) · NeuroImage: Clinical · doi:10.1016/j.nicl.2017.06.012 · Cited by 54 · see citing works
Temporal feature aggregation with attention for insider threat detection from activity logs
Pal, P., Chattopadhyay, P., & Swarnkar, M. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119925 · Cited by 52 · see citing works
On the effect of calibration in classifier combination
Bella, A., Ferri, C., Hernández-Orallo, J., & Ramírez-Quintana, M. J. (2013) · Applied Intelligence · doi:10.1007/s10489-012-0388-2 · Cited by 45 · see citing works
Pang, G., Cao, L., Chen, L., & Liu, H. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0052 · Cited by 43 · see citing works
Identifying progression-specific Alzheimer’s subtypes using multimodal transformer
Machado Reyes, D., Chao, H., Hahn, J., Shen, L., Yan, P., & Initiative, F. T. A. D. N. (2024) · Journal of Personalized Medicine · doi:10.3390/jpm14040421 · Cited by 17 · see citing works
Dallora, A. L., Kvist, O., Berglund, J. S., Ruiz, S. D., Boldt, M., Flodmark, C.-E., & Anderberg, P. (2020) · JMIR Medical Informatics · doi:10.2196/18846 · Cited by 13 · see citing works
Albright, J., & Initiative, A. D. N. (2019) · Alzheimer's & Dementia: Translational Research & Clinical Interventions · doi:10.1016/j.trci.2019.07.001 · Cited by 104 · see citing works
- Authors
- Hand, D. J., & Till, R. J.
- Affiliations
- Imperial College London (GB)
- ORCID
- Hand, David J.
- Venue
- Machine Learning
- Published
- 2001-11
- Volume
- 45
- Issue
- 2
- Pages
- 171-186
- ISSN
- 0885-6125 (ISSN-L)
- 1573-0565
- OpenAlex
- W1912982817
- Cited by
- 2,276
- References
- 22
- Citations by year
- 2026: 153 · 2025: 148 · 2024: 142 · 2023: 157 · 2022: 174 · 2021: 188 · 2020: 141 · 2019: 134 · 2018: 121 · 2017: 90 · 2016: 87 · 2015: 87 · 2014: 88 · 2013: 81 · 2012: 84
- FWCI
- 16.01
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Machine Learning (publishedVersion)
- Funders
- Royal Society
- Engineering and Physical Sciences Research Council
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Information Systems › Data Mining and Machine Learning Applications
- Keywords
- area under the ROC curve, Gini coefficient, multi-class classification, pairwise comparison, classification performance measures
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review