Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulations
Open accessAbstract
There is an increasing interest in using language models (LMs) for automated decision-making, with multiple countries actively testing LMs to aid in military crisis decision-making. To scrutinize relying on LM decision-making in high-stakes settings, we examine the inconsistency of responses in a crisis simulation ("wargame"), similar to reported tests conducted by the US military. Prior work illustrated escalatory tendencies and varying levels of aggression among LMs but were constrained to simulations with pre-defined actions. This was due to the challenges associated with quantitatively measuring semantic differences and evaluating natural language decision-making without relying on pre-defined actions. In this work, we query LMs for free form responses and use a metric based on BERTScore to measure response inconsistency quantitatively. Leveraging the benefits of BERTScore, we show that the inconsistency metric is robust to linguistic variations that preserve semantic meaning in a question-answering setting across text lengths. We show that all five tested LMs exhibit levels of inconsistency that indicate semantic differences, even when adjusting the wargame setting, anonymizing involved conflict countries, or adjusting the sampling temperature parameter $T$. Further qualitative evaluation shows that models recommend courses of action that share few to no similarities. We also study the impact of different prompt sensitivity variations on inconsistency at temperature $T = 0$. We find that inconsistency due to semantically equivalent prompt variations can exceed response inconsistency from temperature sampling for most studied models across different levels of ablations. Given the high-stakes nature of military deployment, we recommend further consideration be taken before using LMs to inform military decisions or other cases of high-stakes decision-making.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs often display more aggressive responses than humans do in military simulations and troubling inconsistencies in crisis decision-making.
“These findings mirror other evidence showing that LLMs often display more aggressive responses than humans do in military simulations and troubling inconsistencies in crisis decision-making (Lamparth et al., 2024;Shrivastava et al., 2024).”
From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Shrivastava et al., 2024
The Conclusion reports that five language models gave inconsistent responses in a military crisis wargame, supporting the decision-making inconsistency claim. The paper’s Computers in Strategic Decision-Making section describes prior work as finding different levels of aggression relative to human experts, but it does not establish that LLMs are often more aggressive than humans. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThese findings mirror other evidence showing that LLMs often display more aggressive responses than humans in military simulations and troubling inconsistencies in crisis decision-making.
“These findings mirror other evidence showing that LLMs often display more aggressive responses than humans in military simulations and troubling inconsistencies in crisis decision-making (Lamparth et al., 2024; Shrivastava et al., 2024).”
From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Shrivastava et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Cited by 2
Research works printed but not linked: 32
- Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
- Garry D Brewer and Bruce G Blair. War games and national security with a grain of SALT. Bulletin of the Atomic Scientists, 35(6):18-26, 1979.
- William N Caballero and Phillip R Jenkins. On Large Language Models in National Security Ap- plications. arXiv preprint arXiv:2407.03453, 2024. · arXiv
- Brian J Connolly. Battlefield information and tactics engine (BITE): a multimodal large language model approach for battlespace management. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications VI, volume 13051, pp. 1305103. SPIE, 2024.
- Mary L Cummings. Automation bias in intelligent time critical decision support systems. In Deci- sion making in aviation, pp. 289-294. Routledge, 2017.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. · arXiv
- Eva Dou, Nitasha Tiku, and Gerrit De Vynck. Pentagon explores military uses of large lan- guage models, 2024. URL https://www.washingtonpost.com/technology/2024/ 02/20/pentagon-ai-llm-conference/.
- John R Emery. Moral Choices Without Moral Language: 1950s Political-Military Wargaming at the RAND Corporation (Fall 2021). Texas National Security Review, 2021. EUCOM History Office. This Week in EUCOM History: January 23-29, 1995, 2012. URL https://web.archive.org/web/20160105033448/http://www.eucom.mil/ media-library/article/23042/this-week-in-eucom-history-january- 23-29-1995.
- FAIR, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science, 378(6624):1067-1074, 2022.
- Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625-630, 2024.
- Geoffrey Forden, Pavel Podvig, and Theodore A Postol. False alarm, nuclear danger. IEEE Spec- trum, 37(3):31-39, 2000.
- Kanishk Gandhi, Dorsa Sadigh, and Noah D Goodman. Strategic reasoning with language models. arXiv preprint arXiv:2305.19165, 2023. · arXiv
- Declan Grabb, Max Lamparth, and Nina Vasan. Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation. In First Conference on Language Modeling, 2024.
- Michael Hanna and Ondřej Bojar. "A Fine-Grained Analysis of BERTScore". In Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Tom Kocmi, Andre Martins, Makoto Morishita, and Christof Monz (eds.), Proceedings of the Sixth Conference on Machine Translation, pp. 507-517, Online, November 2021. Association for Computational Linguistics.
- Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. In International Conference on Learning Representations, 2021.
- Daniel P Hogan and Andrea Brennen. Open-Ended Wargames with Large Language Models. arXiv preprint arXiv:2404.11446, 2024. · arXiv
- Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. War and peace (waragent): Large language model-based multi-agent simulation of world wars. arXiv preprint arXiv:2311.17227, 2023. · arXiv
- Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 1938.
- Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023.
- Max Lamparth, Anthony Corso, Jacob Ganz, Oriana Skylar Mastro, Jacquelyn Schneider, and Harold Trinkunas. Human vs. machine: Behavioral differences between expert humans and lan- guage models in wargame simulations. arXiv preprint arXiv:2403.03407, 2024. · arXiv
- Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252. Association for Computational Linguistics, 2022.
- Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023. · arXiv
- Nunzio Lorè and Babak Heydari. Strategic behavior of large language models and the role of game structure versus contextual framing. Scientific Reports, 14(1):18490, 2024.
- Potsawee Manakul, Adian Liusie, and Mark Gales. MQAG: Multiple-choice question answering and generation for assessing information consistency in summarization. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 39-53. Association for Computational Linguistics, 2023a.
- Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: Zero-resource black-box hallu- cination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9004-9017. Association for Compu- tational Linguistics, 2023b.
- Jared Moore, Tanvi Deshpande, and Diyi Yang. Are large language models consistent over value- laden questions? arXiv preprint arXiv:2407.02996, 2024. · arXiv
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
- Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of op- tions in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2006-2017, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
- Anka Reuel, Ben Bucknall, Stephen Casper, Tim Fist, Lisa Soder, Onni Aarne, Lewis Hammond, Lujain Ibrahim, Alan Chan, Peter Wills, et al. Open problems in technical ai governance. arXiv preprint arXiv:2407.14981, 2024. · arXiv
- Juan-Pablo Rivera, Gabriel Mukobi, Anka Reuel, Max Lamparth, Chandler Smith, and Jacquelyn Schneider. Escalation risks from language models in military and diplomatic decision-making. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 836-898, 2024.
- Yash Saxena, Sarthak Chopra, and Arunendra Mani Tripathi. Evaluating Consistency and Reasoning Capabilities of Large Language Models. arXiv preprint arXiv:2404.16478, 2024. · arXiv
- Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. Evaluating the moral beliefs encoded in llms. Advances in Neural Information Processing Systems, 36, 2024.
- Authors
- Shrivastava, A., Hullman, J., & Lamparth, M.
- Affiliations
- University of Chicago (US)
- Northwestern University (US)
- Stanford University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2410.13204
- OpenAlex
- W4403579811
- Cited by
- 2
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- University of Chicago
- Center for International Security and Cooperation, Stanford University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- language models, war games, BERTScore, semantic similarity, prompt sensitivity, sampling temperature, automated decision-making, high-stakes decision making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
