Probabilistic classifiers with high-dimensional data
Open accessAbstract
For medical classification problems, it is often desirable to have a probability associated with each class. Probabilistic classifiers have received relatively little attention for small n large p classification problems despite of their importance in medical decision making. In this paper, we introduce 2 criteria for assessment of probabilistic classifiers: well-calibratedness and refinement and develop corresponding evaluation measures. We evaluated several published high-dimensional probabilistic classifiers and developed 2 extensions of the Bayesian compound covariate classifier. Based on simulation studies and analysis of gene expression microarray data, we found that proper probabilistic classification is more difficult than deterministic classification. It is important to ensure that a probabilistic classifier is well calibrated or at least not "anticonservative" using the methods developed here. We provide this evaluation for several probabilistic classifiers and also evaluate their refinement as a function of sample size under weak and strong signal conditions. We also present a cross-validation method for evaluating the calibration and refinement of any probabilistic classifier on any data set.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Calibration is a pivotal aspect of model performance.
“Calibration is a pivotal aspect of model performance [2][3][4]”
From A calibration hierarchy for risk models was defined: from utopia to empirical data (2016) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013) · doi:10.1201/b16018 · Cited by 27,276 · see citing works
Regularization Paths for Generalized Linear Models via Coordinate Descent
Friedman, J., Hastie, T., & Tibshirani, R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i01 · Cited by 19,927 · see citing works
Modern Applied Statistics with S
Venables, W., & Ripley, B. (2010) · Statisctics and computing/Statistics and computing · doi:10.1007/978-0-387-21706-2 · Cited by 16,188 · see citing works
Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004) · The Annals of Statistics · doi:10.1214/009053604000000067 · Cited by 9,357 · see citing works
Diagnosis of multiple cancer types by shrunken centroids of gene expression
Tibshirani, R. J., Hastie, T. J., Narasimhan, B., & Chu, G. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.082099299 · Cited by 2,938 · see citing works
Comparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data
Dudoit, S., Fridlyand, J., & Speed, T. P. (2002) · Journal of the American Statistical Association · doi:10.1198/016214502753479248 · Cited by 2,673 · see citing works
Schäfer, J., & Strimmer, K. (2005) · Statistical Applications in Genetics and Molecular Biology · doi:10.2202/1544-6115.1175 · Cited by 1,556 · see citing works
Prediction error estimation: a comparison of resampling methods
Molinaro, A. M., Simon, R. M., & Pfeiffer, R. M. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti499 · Cited by 1,343 · see citing works
Self-citationWright, G. W., Tan, B. K., Rosenwald, A., Hurt, E. H., Wiestner, A. U., & Staudt, L. M. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1732008100 · Cited by 1,039 · see citing works
The Comparison and Evaluation of Forecasters
DeGroot, M. H., & Fienberg, S. E. (1983) · The Statistician · doi:10.2307/2987588 · Cited by 604 · see citing works
BagBoosting for tumor classification with gene expression data
Dettling, M. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bth447 · Cited by 540 · see citing works
A protocol for building and evaluating predictors of disease state based on microarray data
Wessels, L. F. A., Reinders, M. J. T., Hart, A. A. M., Veenman, C. J., Dai, H., He, Y. D., & Veer, L. V. ‘. (2005) · Computer applications in the biosciences · doi:10.1093/bioinformatics/bti429 · Cited by 146 · see citing works
Lai, C., Reinders, M. J. T., Veer, L. J. V., & Wessels, L. F. A. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-235 · Cited by 128 · see citing works
pamr: Pam: Prediction Analysis for Microarrays
Hastie, T. J., Tibshirani, R., Narasimhan, B., et al. (2003) · doi:10.32614/cran.package.pamr · Cited by 30 · see citing works
An Example of Slow Convergence of the Bootstrap in High Dimensions
Troendle, J. F., Korn, E. L., & McShane, L. M. (2004) · The American Statistician · doi:10.1198/0003130042845 · Cited by 25 · see citing works
Distribution modeling and simulation of gene expression data
Parrish, R. S., Spencer, H. J., & Xu, P. (2008) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2008.03.023 · Cited by 24 · see citing works
Dawid, A. P. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess2064.pub2 · Cited by 20 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · see citing works
- Authors
- Kim, K. I., & Simon, R.
- Affiliations
- National Cancer Institute (US)
- ORCID
- Simon, Richard
- Venue
- Biostatistics
- Publisher
- Oxford University Press (OUP)
- Published
- 2010-11-17
- Volume
- 12
- Issue
- 3
- Pages
- 399-412
- ISSN
- 1465-4644 (ISSN-L)
- 1468-4357
- PMC
- PMC3138069
- OpenAlex
- W2128129551
- Cited by
- 23
- References
- 17
- Citations by year
- 2026: 3 · 2021: 2 · 2020: 1 · 2018: 2 · 2017: 3 · 2016: 4 · 2015: 1 · 2014: 4 · 2013: 2
- FWCI
- 0.39
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Biostatistics (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Keywords
- probabilistic classifiers, high-dimensional classification, large p small n, gene expression microarrays, medical classification, cross-validation, evaluation metrics
- MeSH terms
- Bayes Theorem, Colonic Neoplasms, Computer Simulation, Humans, Male, Probability, Prostatic Neoplasms, Models, Statistical, Oligonucleotide Array Sequence Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
