Small-sample precision of ROC-related estimates
Open accessAbstract
Motivation: The receiver operator characteristic (ROC) curves are commonly used in biomedical applications to judge the performance of a discriminant across varying decision thresholds. The estimated ROC curve depends on the true positive rate (TPR) and false positive rate (FPR), with the key metric being the area under the curve (AUC). With small samples these rates need to be estimated from the training data, so a natural question arises: How well do the estimates of the AUC, TPR and FPR compare with the true metrics? Results: Through a simulation study using data models and analysis of real microarray data, we show that (i) for small samples the root mean square differences of the estimated and true metrics are considerable; (ii) even for large samples, there is only weak correlation between the true and estimated metrics; and (iii) generally, there is weak regression of the true metric on the estimated metric. For classification rules, we consider linear discriminant analysis, linear support vector machine (SVM) and radial basis function SVM. For error estimation, we consider resubstitution, three kinds of cross-validation and bootstrap. Using resampling, we show the unreliability of some published ROC results. Availability: Companion web site at http://compbio.tgen.org/paper_supp/ROC/roc.html Contact: edward@mail.ece.tamu.edu
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ROC AUC presents several flaws and is sensitive to class imbalance.
“However, ROC AUC presents several flaws [92], and it is sensitive to class imbalance [93].”
From The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation (2020) · cites this paper as [93]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
Mann, H. B., & Whitney, D. R. (1947) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730491 · Cited by 14,320 · see citing works
A Gene-Expression Signature as a Predictor of Survival in Breast Cancer
van de Vijver, M. J., He, Y. D., Veer, L. V. ‘., Dai, H. A., Hart, A. A. M., Voskuil, D. W., Schreiber, G. J., Peterse, J. L., Roberts, C., Marton, M. J., et al. (2002) · New England Journal of Medicine · doi:10.1056/nejmoa021967 · Cited by 6,411 · see citing works
A Probabilistic Theory of Pattern Recognition
Devroye, L., Györfi, L., & Lugosi, G. (1996) · Stochastic Modelling and Applied Probability · doi:10.1007/978-1-4612-0711-5 · Cited by 3,274 · see citing works
Bhattacharjee, A., Richards, W. G., Staunton, J. E., Li, C., Monti, S., Vasa, P. P., Ladd, C., Beheshti, J., Bueno, R., Gillette, M. A., et al. (2001) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.191502998 · Cited by 2,540 · see citing works
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems
Hand, D. J., & Till, R. J. (2001) · Machine Learning · doi:10.1023/a:1010920819831 · Cited by 2,276 · see citing works
Pepe, M. S. (2004) · American Journal of Epidemiology · doi:10.1093/aje/kwh101 · Cited by 1,276 · see citing works
Is cross-validation valid for small-sample microarray classification?
Braga-Neto, U., & Dougherty, E. R. (2004) · Bioinformatics · doi:10.1093/bioinformatics/btg419 · Cited by 599 · see citing works
Self-citationSaal, L. H., Johansson, P. A., Holm, K., Gruvberger-Saal, S. K., She, Q., Maurer, M. S., Koujak, S., Ferrando, A. A., Malmström, P., Memeo, L., et al. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0702507104 · Cited by 514 · see citing works
Molecular markers of early Parkinson's disease based on gene expression in blood
Scherzer, C. R., Eklund, A. C., Morse, L. J., Liao, Z., Locascio, J. J., Fefer, D., Schwarzschild, M. A., Schlossmacher, M. G., Hauser, M. A., Vance, J. M., et al. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0610204104 · Cited by 511 · see citing works
SIGNAL DETECTION THEORY: VALUABLE TOOLS FOR EVALUATING INDUCTIVE LEARNING
Spackman, K. A. (1989) · Elsevier eBooks · doi:10.1016/b978-1-55860-036-2.50047-3 · Cited by 344 · see citing works
Additive estimators for probabilities of correct classification
Glick, N. (1978) · Pattern Recognition · doi:10.1016/0031-3203(78)90029-8 · Cited by 146 · see citing works
What should be expected from feature selection in small-sample settings
Sima, C., & Dougherty, E. R. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl407 · Cited by 96 · see citing works
Self-citationExact performance of error estimators for discrete classifiers
Braga-Neto, U., & Dougherty, E. R. (2005) · Pattern Recognition · doi:10.1016/j.patcog.2005.02.013 · Cited by 56 · see citing works
Self-citationDecorrelation of the True and Estimated Classifier Errors in High-Dimensional Settings
Hanczar, B., Jianping Hua, A., & Dougherty, E. R. (2007) · EURASIP Journal on Bioinformatics and Systems Biology · doi:10.1155/2007/38473 · Cited by 53 · see citing works
Self-citationZollanvari, A., Braga-Neto, U., & Dougherty, E. R. (2009) · Pattern Recognition · doi:10.1016/j.patcog.2009.05.003 · Cited by 34 · see citing works
Self-citationRelation Between Permutation-Test P Values and Classifier Error Estimates
Hsing, T., Attoor, S. N., & Dougherty, E. R. (2003) · Machine Learning · doi:10.1023/a:1023985022691 · Cited by 28 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Chicco, D., & Jurman, G. (2020) · BMC Genomics · doi:10.1186/s12864-019-6413-7 · Cited by 6,143 · see citing works
Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them)
Berrar, D., & Flach, P. (2012) · Briefings in Bioinformatics · doi:10.1093/bib/bbr008 · Cited by 136 · see citing works
Circulating Cytokines as Predictors of Depression in Patients with Breast Cancer
Ho, H.-Y., Chin-Hung Chen, V., Tzang, B.-S., Hsieh, C.-C., Wang, W.-K., Weng, Y.-P., Hsu, Y.-T., Hsaio, H.-P., Weng, J.-C., & Chen, Y.-L. (2021) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2021.02.037 · Cited by 20 · see citing works
- Authors
- Hanczar, B., Hua, J., Sima, C., Weinstein, J., Bittner, M., & Dougherty, E. R.
- Affiliations
- Translational Genomics Research Institute (US)
- 1 LIPADE, University Paris Descartes, 45 rue des Saint-Peres, Paris, France, 2 Computational Biology Division, Translational Genomics Research Institute, Phoenix, AZ, 3 Department of Bioinformatics and Computation Biology, MD Anderson Cancer Center, Houston and 4 Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA
- Corresponding author
- Dougherty, Edward R.
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2010-02-03
- Volume
- 26
- Issue
- 6
- Pages
- 822-830
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- OpenAlex
- W2122814955
- Cited by
- 323
- References
- 16
- Citations by year
- 2026: 18 · 2025: 23 · 2024: 22 · 2023: 14 · 2022: 24 · 2021: 25 · 2020: 17 · 2019: 22 · 2018: 15 · 2017: 14 · 2016: 18 · 2015: 24 · 2014: 14 · 2013: 15 · 2012: 34
- FWCI
- 24.67
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Bioinformatics (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (0634794)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- receiver operating characteristic, area under the curve, small sample estimation, diagnostic sensitivity, false positive rate, linear discriminant analysis, support vector machine, Gaussian kernel, cross-validation, bootstrapping, resubstitution, microarray data, estimation bias
- MeSH terms
- Algorithms, False Positive Reactions, Pattern Recognition, Automated, ROC Curve, Oligonucleotide Array Sequence Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
