Toward Uncertainty Quantification for Supervised Classification
Open accessAbstract
Our goal is to develop a general theoretical basis for quantifying uncertainty in supervised machine learning models. Current machine learning accuracy-based validation metrics indicate how well a classifier performs on a given data set as a whole. However, these metrics do not tell us a model's efficacy in predicting particular samples. We quantify uncertainty by constructing probability distributions of the predictions made by an ensemble of classifiers. This report details our initial investigations into uncertainty quantification for supervised machine learning. We apply an uncertainty analysis to the problem of malicious website detection. Machine learning models can be trained to find suspicious characteristics in the text of a website's Uniform Resource Locator (URL). However, given the vast numbers of URLs and the ever changing tactics of malicious actors, it will always be possible to find sets of websites which are outliers with respect to a model's hypothesis. Therefore, we seek to understand a model's per-sample reliability when classifying URL data. Acknowledgements This work was funded by the Sandia National Laboratories Laboratory Directed Research and Development (LDRD) program.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The form of the distribution determines the range of a system's behavior.
“Ultimately, the distribution form characterizes the range of a system's behavior [69].”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [69]
Cited by
Sorted by citations
Borys, K., Schmitt, Y. A., Nauta, M., Seifert, C., Krämer, N., Friedrich, C. M., & Nensa, F. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.110786 · Cited by 233 · see citing works
Jorgensen, S., Holodnak, J. T., Dempsey, J., de Souza, K., Raghunath, A., Rivet, V., DeMoes, N., Alejos, A., & Wollaber, A. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3244168 · Cited by 72 · see citing works
“While there has been some limited mention of uncertainty quantification in cyber security applications of machine learning in general [17]–[20], the problem has not received the same attention in cyber security as it has in other domains, such as computer”
BackgroundComparison of machine learning methods for the classification of cardiovascular disease
Hagan, R., Gillan, C. J., & Mallett, F. (2021) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2021.100606 · Cited by 53 · see citing works
“This because some of the data in the validation sample may still lie inside the boundaries of the subset that was used for training [27].”
BackgroundIncorporating Measurement Error in Astronomical Object Classification
Shy, S., Tak, H., Feigelson, E. D., Timlin, J. D., III, & Babu, G. J. (2022) · The Astronomical Journal · doi:10.3847/1538-3881/ac6e64 · Cited by 18 · see citing works
“Simulation-based approaches, such as bootstrapping (Efron 1992; Efron & Tibshirani 1994), subsampling (Von Luxburg et al. 2010), and random projections on a lower dimensional space (Achlioptas 2003) are widely used in statistical machine learning. For example, (i) Yu et al. (2013) and Sun (2015) review various ways to…”
BackgroundMethodBitencourt, J., Osho, J., Wooley, A., & Harris, G. A. (2025) · International Journal of Production Research · doi:10.1080/00207543.2025.2524516 · Cited by 6 · see citing works
Energy-Efficient Data Transfer Optimization via Decision-Tree Based Uncertainty Reduction
Jamil, H., Rodolph, L., Goldverg, J., et al. (2022) · Proceedings/Proceedings - International Conference on Computer Communications and Networks · doi:10.1109/icccn54977.2022.9868866 · Cited by 2 · see citing works
“, decision search trees in our case) to improve on the search result accuracy offered by any one of its members [19] [20].”
BackgroundRigorous Viability Assessment of Machine Learning Projects
Zipperling, D., Ott, L., Vössing, M., & Kühl, N. (2026) · Business & Information Systems Engineering · doi:10.1007/s12599-026-00986-2 · Cited by 1 · see citing works
Sandia National Laboratories (SNL-CA), L., Chen, M., Administration, U. A. S. F. H. R. A., Sandia National Laboratories (SNL-NM), A., Darling, M., & Stracuzzi, D. J. (2018) · doi:10.2172/1475256
Self-citationHerasymuk, D., Poliakov, A., & Fedorchenko, V. M. (2025) · Innovative technologies and scientific solutions for industries · doi:10.30837/2522-9818.2025.1.005
- Authors
- Sandia National Laboratories (SNL-NM), A., Darling, M., (NNSA), U. N. N. S. A., & Stracuzzi, D.
- Affiliations
- Sandia National Laboratories California (US)
- Published
- 2018-01-01
- OpenAlex
- W2955157344
- Cited by
- 9
- References
- 0
- Citations by year
- 2026: 1 · 2025: 2 · 2023: 2 · 2022: 2 · 2021: 1 · 2018: 1
- Type
- report
- Language
- en
- Open access
- green
- www.osti.gov
- Locations
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
- Funders
- USDOE National Nuclear Security Administration (NNSA)
- DOE (NA0003525, AC04-94AL85000)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
- Keywords
- uncertainty quantification, supervised classification, ensemble classifiers, malicious website detection, URL analysis, model validation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
