A Comprehensive Evaluation of Cognitive Biases in LLMs
Open accessAbstract
We present a large-scale evaluation of 30 cognitive biases in 20 state-of-the-art large language models (LLMs) under various decision-making scenarios. Our contributions include a novel general-purpose test framework for reliable and large-scale generation of tests for LLMs, a benchmark dataset with 30,000 tests for detecting cognitive biases in LLMs, and a comprehensive assessment of the biases found in the 20 evaluated LLMs. Our work confirms and broadens previous findings suggesting the presence of cognitive biases in LLMs by reporting evidence of all 30 tested biases in at least some of the 20 LLMs. We publish our framework code to encourage future research on biases in LLMs: https://github.com/simonmalberg/cognitive-biases-in-llms
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The marketing and computer science literatures predominantly treat heuristic susceptibility in LLMs as a static, immutable defect hardwired during training.
“The marketing and computer science literatures predominantly treat heuristic susceptibility in LLMs as a static, immutable defect hardwired during training (Echterhoff et al., 2024;Knipper et al., 2025;Malberg et al., 2025).”
From Shopping by algorithm: How agentic AI deploys human heuristics as a surrogate consumer (2026) · cites this paper as Malberg et al., 2025
The Introduction says human biases likely enter LLMs through training data and procedures. But the paper does not say that marketing and computer science literatures predominantly regard heuristic susceptibility as static or immutable; Section 4.6 instead hypothesizes that susceptibility varies with model size, architecture, and training procedure. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe rationality paradox is not resolved simply by increasing computational capability of the LLMs.
“recent large-scale benchmarks evaluating cognitive biases demonstrate that this paradox is not resolved simply by increasing computational capability of the LLMs (Echterhoff et al., 2024;Knipper et al., 2025;Malberg et al., 2025).”
From Shopping by algorithm: How agentic AI deploys human heuristics as a surrogate consumer (2026) · cites this paper as Malberg et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Part of this paper’s reference list could not be split into single entries.
Research works printed but not linked: 4
- [725]725. Stephen Pilli. 2023. Exploring conversational agents as an effective tool for measuring cognitive biases in decision-making. In 2023 10th International Confer- ence on Behavioural and Social Computing (BESC), pages 1–5. IEEE. Sivan Portal, Russell Abratt, and Michael Bendixen. 2018. Building a human brand: Brand anthropo- morphism unravelled. Business Horizons, 61(3):367–
- [118]118. Rüdiger Schmitt-Beck. 2015. Bandwagon Effect. John Wiley & Sons, Ltd. Hamish GW Seaward and Simon Kemp. 2000. Opti- mism bias and student debt. New Zealand journal of psychology, 29(1):17–19. Sakib Shahriar, Brady D Lund, Nishith Reddy Man- nuru, Muhammad Arbab Arshad, Kadhim Hayawi, Ravi Varma Kumar Bevara, Aashrith Mannuru, and Laiba Batool. 2024. Putting gpt-4o to the sword: A comprehensive evaluation of language, vision, speech, and multimodal proficiency. Applied Sci- ences, 14(17):7782. Jonathan Shalev. 2000. Loss aversion equilibrium. In- ternational Journal of Game Theory, 29:269–287. Hersh Shefrin and Meir Statman. 1985. The dispo- sition to sell winners too early and ride losers too long: Theory and evidence. The Journal of finance, 40(3):777–790. James Shepperd, Wendi Malone, and Kate Sweeny. 2008. Exploring causes of the self-serving bias. So- cial and Personality Psychology Compass, 2(2):895–
- [908]908. Emmanuel Marques Silva, Rafael de Lacerda Moreira, and Patricia Maria Bortolon. 2023. Mental account- ing and decision making: a systematic literature re- view. Journal of Behavioral and Experimental Eco- nomics, 107:102092. Herbert A Simon. 1990. Bounded rationality. Utility and probability, pages 15–18. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mah- davi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172–180. Dustin Sleesman, Anna Lennard, Gerry Mcnamara, and Donald Conlon. 2018. Putting escalation of commitment in context: A multi-level review and analysis. Academy of Management Annals, 12:an- nals.2016.0046. Mark Snyder and William Swann. 1978. Hypothesis testing in social judgment. Journal of Personality and Social Psychology, 36:1202–1212. Melvin Snyder and Arthur Frankel. 1976. Observer bias: A stringent test of behavior engulfing the field. Jour- nal of Personality and Social Psychology, 34:857–
- [864]864. Melvin L Snyder, Walter G Stephan, and David Rosen- field. 1976. Egotism and attribution. Journal of Personality and Social Psychology, 33(4):435. Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2024. The good, the bad, and the greedy: Eval- uation of llms should not ignore non-determinism. arXiv preprint arXiv:2407.10457. Barry M. Staw. 1976. Knee-deep in the big muddy: a study of escalating commitment to a chosen course of action. Organizational Behavior and Human Per- formance, 16(1):27–44. Barry M. Staw. 1981. The escalation of commitment to a course of action. The Academy of Management Review, 6(4):577–587. Christina Steindl, Eva Jonas, Sandra Sittenthaler, Eva Traut-Mattausch, and Jeff Greenberg. 2015. Under- standing psychological reactance. Zeitschrift für Psy- chologie. Joachim Stöber. 2001. The social desirability scale-17 (sds-17): Convergent validity, discriminant validity, and relationship with age. European Journal of Psy- chological Assessment, 17(3):222. Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Large language models are in- consistent and biased evaluators. arXiv preprint arXiv:2405.01724. Alaina N Talboy and Elizabeth Fuller. 2023. Challeng- ing the appearance of machine intelligence: Cog- nitive bias in llms and best practices for adoption. arXiv preprint arXiv:2304.01358. Zhen Tan, Alimohammad Beigi, Song Wang, Ruocheng Guo, Amrita Bhattacharjee, Bohan Jiang, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation: A survey. arXiv preprint arXiv:2402.13446. Richard Thaler. 1980. Toward a positive theory of con- sumer choice. Journal of economic behavior & orga- nization, 1(1):39–60. Richard Thaler. 1985. Mental accounting and consumer choice. Marketing Science, 4(3):199–214. Richard H Thaler. 1999. Mental accounting matters. Journal of Behavioral decision making, 12(3):183– · arXiv
Cited by
Sorted by citations
Confirmation and Specificity Biases in Large Language Models: An Explorative Study
O’Leary, D. E. (2025) · IEEE Intelligent Systems · doi:10.1109/mis.2024.3513992 · Cited by 17 · see citing works
Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendations
Filandrianos, G., Dimitriou, A., Lymperaiou, M., Thomas, K., & Stamou, G. B. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1140 · Cited by 6 · see citing works
Dietrich, N. T., Patel, D., Bellissimo, J., Loh, C. T., & Tyrrell, P. N. (2026) · Radiology Artificial Intelligence · doi:10.1148/ryai.250585 · Cited by 3 · see citing works
Confirmation Bias in Large Language Models: Risks for Clinical Decision Support Systems
Jardim, R. R.-A. J., Gonçalves, G., Lopes, L. S., Dantas, T., Paulino, D., Netto, A. T. C., Guimarães, D., Teixeira, O., Rocha, A., Vivacqua, A. S., et al. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11581664 · Cited by 1 · see citing works
Cognitive Biases in Large Language Model based Decision Making: Insights and Mitigation Strategies
Chen, S. (2025) · Applied and Computational Engineering · doi:10.54254/2755-2721/2025.21389 · Cited by 1 · see citing works
Shopping by algorithm: How agentic AI deploys human heuristics as a surrogate consumer
Wadi, D., & Ma, Y. (2026) · arXiv · doi:10.48550/arxiv.2609.28372
Holst, C., Wiegräbe, F., Redecker, C., & Lohweg, V. (2025) · Springer eBooks · doi:10.1007/978-3-658-48573-3_11
Vulnerability of LLM Outputs to Heuristics-Inducing Prompt Structures
Kuramoto, T., Kamei, R., & Suzuki, J. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789108
AI Through the Human Lens: Investigating Cognitive Theories in Machine Psychology
Kundu, A., & Goswami, R. (2025) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2025.ijcnlp-srw.14
Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework
Şenol, A., Agrawal, G., & Liu, H. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10090300
Возможности интеграции поведенческой и социокультурной экономик
Nikishina, E. (2026) · Journal of the New Economic Association · doi:10.31737/22212264_2026_2_287-305
Automated bias blind spots: Examining stereotype associations in VLM-as-a-judge paradigms
Hong, R. S., Ovalle, A., Ung, M., Spiliopoulou, E., Sagun, L., Williams, A., & Ross, C. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806459
Chen, Z., Jiao, W.-C., Tsang, Y. P., Wu, C. H., & Wang, X.-J. (2026) · Group Decision and Negotiation · doi:10.1007/s10726-026-10017-w
Almeida, L. M., Cadete, A. B., & Queiroz, V. X. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729360
Unveiling Latent Cognitive Bias Constructs in Large Language Models
Tilaye, B. L., Day, H., McWilliams, C., & Filho, T. M. S. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.1750
- Authors
- Malberg, S., Poletukhin, R., Schuster, C., & Groh, G. G.
- Venue
- Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities
- Published
- 2025
- Pages
- 578-613
- OpenAlex
- W4411119351
- Cited by
- 14
- References
- 0
- Citations by year
- 2026: 9 · 2025: 5
- FWCI
- 10.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- International Conference on Natural Language Processing for Digital Humanities (NLP4DH) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, cognitive biases, bias assessment, algorithmic fairness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
