The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Evaluation of LLM Reasoning Under Uncertainty: An Atomic Comparison to Normative Approaches
Abstract
Evaluating uncertainty in large language model (LLM) reasoning is challenging due to their vast parameter space, abstract knowledge representation, and limited transparency regarding training data. While normative formalisms, such as deductive logic, clearly define sound reasoning in the absence of uncertainty, reasoning under uncertainty admits multiple approaches, including probabilistic reasoning (e.g. Bayesian), belief function reasoning (e.g. Dempster-Shafer), or fuzzy logic, to name a few. This paper examines how LLMs handle uncertainty by analyzing outcomes based on an atomic fusion and reasoning problem. We establish a point of reference using the simplest of fusion topologies to facilitate transparency and understanding of how LLMs align with established theories. The reasoning approaches of different LLMs with varying complexities are compared to established normative frameworks, providing insights into which formalism best aligns with LLM reasoning and assessing its soundness and consistency. A deviation function for assessment is developed, and the results indicate that the tested LLMs' reasoning under uncertainty does not consistently align with established theories, even for the simplest information fusion topologies. These preliminary results form the basis for further investigations and LLM refinements.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
A Mathematical Theory of Evidence
Shafer, G. R. (1976) · Princeton University Press eBooks · doi:10.1515/9780691214696 · Cited by 9,246
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Smets, P., & Kennes, R. (1994) · Artificial Intelligence · doi:10.1016/0004-3702(94)90026-4 · Cited by 2,234
Bovens, L., & Hartmann, S. (2004) · doi:10.1093/0199269750.001.0001 · Cited by 647
Distances in evidence theory: Comprehensive survey and generalizations
Jousselme, A.-L., & Maupin, P. (2011) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2011.07.006 · Cited by 267
Self-citationChain-of-Thought Reasoning Without Prompting
Wang, X., & Zhou, D. (2024) · neural information processing systems · doi:10.52202/079017-2123 · Cited by 209
Analyzing Uncertainty in Neural Machine Translation
Ott, M., Auli, M., Grangier, D., & Ranzato, M. (2018) · arXiv · doi:10.48550/arxiv.1803.00047 · Cited by 140
Relevance and truthfulness in information correction and fusion
Pichon, F., Didier Dubois, A., & Denœux, T. (2011) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2011.02.006 · Cited by 96
Lee, S., Sim, W., Shin, D., Seo, W., Park, J., Lee, S. C., Hwang, S., Kim, S., & Kim, S. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3712701 · Cited by 65
Learning, Bayesian Probability, Graphical Models, and Abduction
Poole, D. (2000) · Applied logic series · doi:10.1007/978-94-017-0606-3_10 · Cited by 22
Wang, L., & Shen, Y. (2024) · Electronics · doi:10.3390/electronics13234584 · Cited by 22
Uncertain about ChatGPT: enabling the uncertainty evaluation of large language models
Jousselme, A.-L., de Villiers, J. P., de Freitas, A., Blasch, E., Dragos, V., Pavlin, G., Costa, P., Laskey, K. B., & Laudy, C. (2023) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion52260.2023.10224086 · Cited by 16
Self-citationQuality of Information Sources in Information Fusion
Pichon, F., Didier Dubois, A., & Denœux, T. (2019) · Information fusion and data science · doi:10.1007/978-3-030-03643-0_2 · Cited by 15
Towards a formal comparison of uncertainty handling
Flores, C. R., Jousselme, A.-L., & Costa, P. (2020) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion45008.2020.9190168 · Cited by 3
Self-citationA Survey of RAG and Reasoning in Large Language ModelsPreprint
Liu, L. (2026) · doi:10.36227/techrxiv.176784429.92121712/v1 · Cited by 2
Flagged first, then by citations
Research works printed but not linked: 10
- Lu, A survey of deep learning for mathematical reasoning, 14605
- Costa, 2018, URREF: Uncertainty representation and reasoning evaluation framework for information fusion, Journal of Advances in Information Fusion, 13, 137
- Bakman, MARS: Meaning-aware response scoring for uncertainty estimation in generative LLMs, 7752
- Wan, LogicAsker: Evaluating and improving the logical reasoning ability of large language models, 2124
- Chollet, 2024, Abstraction and reasoning corpus for artificial general intelligence (arc-agi)
- Paruchuri, What are the odds? language models are capable of probabilistic reasoning, 11 712
- Shenoy, 1992, Valuation-Based Systems: A framework for managing uncertainty in expert systems, 83
- 2025, Gemini 2.0 Flash
- 2025, OpenAI o3-mini
- 2025, Sonar Reasoning
Cited by
Sorted by citations
Secure AIoT Swarming in MCP-Orchestrated Agent Systems through Physical Anchoring
Pham, V. T., Lakadawala, H., Chen, Y., & Blasch, E. (2026) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion67272.2026.11726669
Self-citation
Metadata
- Authors
- de Villiers, J. P., de Freitas, A., Jousselme, A.-L., Kaplan, L., Blasch, E., Laudy, C., & Costa, P. C.
- Affiliations
- University of Pretoria,Pretoria,South Africa
- CS Research Lab,La Garde,France
- DEVCOM Army Research Laboratory (US)
- Rome Laboratory (US)
- CortAIxLabs/Thales,Palaiseau,France
- George Mason University (US)
- Venue
- 2025 28th International Conference on Information Fusion (FUSION)
- Publisher
- IEEE
- Published
- 2025-07-07
- Pages
- 1-8
- OpenAlex
- W4413679792
- Cited by
- 1
- References
- 13
- Citations by year
- 2026: 1
- FWCI
- 0.96
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Information Fusion (FUSION) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Keywords
- large language models, uncertainty reasoning, Dempster-Shafer theory, belief functions, data fusion, normative reasoning, Bayesian reasoning, fuzzy logic, deviation function, LLM evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
