CoChat
Sign Up
PreprintCited by 17Ethics and Social Impacts of AIMilitary Strategy and TechnologyDefense, Military, and Policy Studies

The DOI matches its DataCite record.

AI-Powered Autonomous Weapons Risk Geopolitical Instability and Threaten AI Research

Open access
Simmons-Edler, Riley, Badman, Ryan, Longpre, Shayne, Rajan, Kanaka·2024·arXiv
DOI 10.48550/arxiv.2405.01859arXiv 2405.01859
Open this paper in CoChat — free accountOpen PDF

Abstract

The recent embrace of machine learning (ML) in the development of autonomous weapons systems (AWS) creates serious risks to geopolitical stability and the free exchange of ideas in AI research. This topic has received comparatively little attention of late compared to risks stemming from superintelligent artificial general intelligence (AGI), but requires fewer assumptions about the course of technological development and is thus a nearer-future issue. ML is already enabling the substitution of AWS for human soldiers in many battlefield roles, reducing the upfront human cost, and thus political cost, of waging offensive war. In the case of peer adversaries, this increases the likelihood of "low intensity" conflicts which risk escalation to broader warfare. In the case of non-peer adversaries, it reduces the domestic blowback to wars of aggression. This effect can occur regardless of other ethical issues around the use of military AI such as the risk of civilian casualties, and does not require any superhuman AI capabilities. Further, the military value of AWS raises the specter of an AI-powered arms race and the misguided imposition of national security restrictions on AI research. Our goal in this paper is to raise awareness among the public and ML researchers on the near-future risks posed by full or near-full autonomy in military technology, and we provide regulatory suggestions to mitigate these risks. We call upon AI policy experts and the defense AI community in particular to embrace transparency and caution in their development and deployment of AWS to avoid the negative effects on global stability and AI research that we highlight here.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2405.01859

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 62

  1. Amoroso, D. and Tamburrini, G. Autonomous weapons systems and meaningful human control: Ethical and legal issues. Current Robotics Reports, 1(4):187-194, Decem- ber 2020.
  2. Ams, S. Blurred lines: the convergence of military and civilian uses of AI & data use and its impact on liberal democracy. International Politics, 60(4):879-896, Au- gust 2023.
  3. Andrychowicz, O. M., Baker, B., Chociej, M., Józefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., Schneider, J., Sidor, S., Tobin, J., Welinder, P., Weng, L., and Zaremba, W. Learning dexter- ous in-hand manipulation. Int. J. Rob. Res., 39(1):3-20, January 2020. Article
  4. Bachmann, S.-D. D. and Grant, R. V. The need for an Australian regulatory code for the use of artificial intelli- gence (AI) in military application. American University National Security Law Brief, 13(2):2, 2023.
  5. Benvenisti, E. The US and the use of force: Double-edged hegemony and the management of global emergencies. Eur. J. Int. Law, 15(4):677-700, September 2004.
  6. Bero, L. Industry influence on research: A cycle of bias. The Commercial Determinants of Health, pp. 185-C19, October 2022.
  7. Bihl, T. J. and Talbert, M. Analytics for Autonomous C4ISR within e-Government: a research agenda. In HICSS, pp. 1-10, 2020.
  8. Blanchard, A. and Taddeo, M. Jus in bello necessity, the requirement of minimal force, and autonomous weapons systems. Journal of Military Ethics, 21(3-4):286-303, October 2022.
  9. Bode, I., Huelss, H., Nadibaidze, A., Qiao-Franco, G., and Watts, T. F. A. Prospects for the global governance of autonomous weapons: comparing Chinese, Russian, and US practices. Ethics Inf. Technol., 25(1):5, February 2023.
  10. Boys, J. D. Predator's progress: the bureaucratic challenges to the Clinton administration's development and deploy- ment of unmanned aerial vehicles (1993-2001). Intell. Natl. Sec., 38(4):538-557, June 2023.
  11. Brown, A. Ethics, autonomy, and killer drones: Can ma- chines do right? Comparative Strategy, 42(6):731-746, November 2023.
  12. Can, M. and Vieira, A. The Chinese Military-Civil Fusion strategy: A state action theory perspective. Int. Spect., 57 (3):85-102, July 2022.
  13. Chen, J. and Ran, X. Deep learning with edge computing: A review. Proc. IEEE, 107(8):1655-1674, August 2019.
  14. Chu, M.-C. M. China's defence semiconductor industrial base in an age of globalisation: Cross-strait dynamics and regional security implications. Journal of Strategic Studies, pp. 1-26, 2023.
  15. Crofts, P. and van Rijswijk, H. Negotiating 'evil': Google, Project Maven and the corporate form. Law, Technology and Humans, 2(1):75-90, January 2020.
  16. Daniels, M. Controlling knowledge, controlling people: Travel restrictions of U.S. scientists and national security. Diplomatic History, 43(1):57-82, January 2019.
  17. del Cerro, J., Cruz Ulloa, C., Barrientos, A., and de León Ri- vas, J. Unmanned aerial vehicles in agriculture: A survey. Agronomy, 11(2):203, January 2021.
  18. Department of Commerce. Implementation of additional ex- port controls: Certain advanced computing items. Federal Register, 88(205), 2023.
  19. Drezner, D. W. Military primacy doesn't pay (nearly as much as you think). Int. Secur., 38(1):52-79, July 2013.
  20. Fabbri, A., Lai, A., Grundy, Q., and Bero, L. A. The influ- ence of industry sponsorship on the research agenda: A scoping review. Am. J. Public Health, 108(11):e9-e16, November 2018.
  21. Fan, T., Long, P., Liu, W., and Pan, J. Distributed multi- robot collision avoidance via deep reinforcement learning for navigation in complex scenarios. Int. J. Rob. Res., 39 (7):856-892, June 2020.
  22. Farrell, H. and Newman, A. L. Weak links in finance and supply chains are easily weaponized. http:// dx.doi.org/10.1038/d41586-022-01254-5, May 2022. Accessed: 2023-12-16. · DOI
  23. Geist, E. M. It's already too late to stop the AI arms race- we must manage it instead. Bull. At. Sci., 72(5):318-321, September 2016.
  24. Gómez de Ágreda, Á. Ethics of autonomous weapons sys- tems and its applicability to any AI systems. Telecomm. Policy, 44(6):101953, July 2020.
  25. Gruszczak, A. and Kaempf, S. (eds.). Routledge Handbook of the Future of Warfare. Routledge, 1 edition, September 2023.
  26. Hegre, H. Democracy and armed conflict. J. Peace Res., 51 (2):159-172, March 2014.
  27. Hunter, C. and Bowen, B. E. We'll never have a model of an AI major-general: Artificial intelligence, command decisions, and kitsch visions of war. Journal of Strategic Studies, pp. 1-31, 2022.
  28. Johnson, J. Artificial intelligence, drone swarming and escalation risks in future warfare. The RUSI Journal, 165 (2):26-36, February 2020.
  29. Klare, M. T. The coming of automated warfare. Curr. Hist., 119(813):9-14, January 2020.
  30. Klare, M. T. Pentagon plans mass autonomous weapons deployment. Arms Control Today, 53(8):36-37, October 2023.
  31. Kwik, J. Mitigating the risk of autonomous weapon misuse by insurgent groups. Laws, 12(1):5, December 2022.
  32. Larrick, B. M., Dwyer, J. T., Erdman, J. W., D'Aloisio, R. F., and Jones, W. An updated framework for indus- try funding of food and nutrition research: Managing financial conflicts and scientific integrity. J. Nutr., 152(8): 1812-1818, August 2022.
  33. Longpre, S., Storm, M., and Shah, R. Lethal autonomous weapons systems & artificial intelligence: Trends, chal- lenges, and policies. MIT Science Policy Review, 3:47-56, August 2022.
  34. Lyu, C. and Zhan, R. Global analysis of active defense tech- nologies for unmanned aerial vehicle. IEEE Aerospace and Electronic Systems Magazine, 37(1):6-31, January 2022.
  35. Mathiassen, K., Baksaas, M., Græe, S. A., Mentzoni, E. A., and Nielsen, N. H. Making the Milrem Themis UGV ready for autonomous operations. In Unmanned Systems Technology XXIII, volume 11758, pp. 221-240. SPIE, April 2021.
  36. Matthew Robert, L. The rise of killer robots: The impact on Autonomous Weapon Systems (AWS) on the United States' Nuclear Deterrence. PhD thesis, Stellenbosch : Stellenbosch University, February 2021.
  37. Michel, A. H. AI-Assisted warfare. MIT Technology Review, 126(5):46+, 2023.
  38. Mikhailov, D. Artificial intelligence integration as a strategic imperative for national security. ACM Technology Policy Council, March 2023.
  39. Nagabandi, A., Konolige, K., Levine, S., and Kumar, V. Deep dynamics models for learning dexterous manipu- lation. In Kaelbling, L. P., Kragic, D., and Sugiura, K. (eds.), Proceedings of the Conference on Robot Learn- ing, volume 100 of Proceedings of Machine Learning Research, pp. 1101-1112. PMLR, 2020.
  40. Pfaff, C. A., Lowrance, C. J., Washburn, B. M., and Carey, B. A. Trusting AI: Integrating Artificial Intelligence Into the Army's Professional Expert Knowledge. US Army War College (USAWC) Press, 2023.
  41. Qiao-Franco, G. and Bode, I. Weaponised artificial intel- ligence and Chinese practices of Human-Machine inter- action. Chinese Journal of International Politics, 16(1): 106-128, March 2023.
  42. Reynolds, I. and Cetin, O. A. War is messy. AI can't handle it. Bulletin of the Atomic Scientists, 2023. Accessed: 2024-1-12.
  43. Rickli, J.-M. and Ienca, M. The security and military im- plications of neurotechnology and artificial intelligence. In Friedrich, O., Wolkenstein, A., Bublitz, C., Jox, R. J., and Racine, E. (eds.), Clinical Neurotechnology meets Artificial Intelligence: Philosophical, Ethical, Legal and Social Implications, pp. 197-214. Springer International Publishing, Cham, 2021.
  44. Riesen, E. The moral case for the development and use of autonomous weapon systems. Journal of Military Ethics, 21(2):132-150, April 2022.
  45. Rivera, J.-P., Mukobi, G., Reuel, A., Lamparth, M., Smith, C., and Schneider, J. Escalation risks from language models in military and diplomatic decision-making. arXiv:2401.03408, January 2024. · arXiv
  46. Robertson, C. The money blind: How to stop industry bias in biomedical science, without violating the first amendment. American Journal of Law and Medicine, 358, 2011.
  47. Russell, S. AI weapons: Russia's war in Ukraine shows why the world must enact a ban. http://dx.doi.org/ 10.1038/d41586-023-00511-5, February 2023. Accessed: 2023-11-27. · DOI
  48. Santora, M. and Triebert, C. Ukraine hits a distant Russian ship, showing reach of naval drones. The New York Times (Digital Edition), pp. NA, August 2023.
  49. Saylam, S. and Gündogdu, F. K. Decision for the next unmanned aerial vehicle considering regional operational requirements based on Z-Fuzzy Best-Worst method. In 2023 10th International Conference on Recent Advances in Air and Space Technologies (RAST), pp. 1-6. IEEE, June 2023.
  50. Schneider, J. The capability/vulnerability paradox and mili- tary revolutions: Implications for computing, cyber, and the onset of war. Journal of Strategic Studies, 42(6): 841-863, September 2019.
  51. Seraphin, A. and Miles, W. Toward a balanced approach: Bridging the military, policy, and technical communities. Ethics Int. Aff., 37(3):272-286, March 2023.
  52. Taddeo, M. and Blanchard, A. A comparative analysis of the definitions of autonomous weapons systems. Sci. Eng. Ethics, 28(5):37, August 2022.
  53. Tamburrini, G. Nuclear weapons and the militarization of AI. In Nuclear Risks and Arms Control -Problems and Progresses in the Time of Pandemics and War, pp. 147-158. Springer International Publishing, 2023.
  54. Templeman, K. A. and Mastro, O. Deepening US-Taiwan cooperation through semiconductors. Book chapter in Silicon Triangle, edited by Diamond, L., Ellis J.O. Jr., and Schell, O., July 2023.
  55. Thornton, R. and Miron, M. Towards the 'third revolution in military affairs'. The RUSI Journal, 165(3):12-21, April 2020.
  56. Trapp, K. N. Boots (on the ground). In Hohmann, J. and Joyce, D. (eds.), International Law's Objects. Oxford University Press, December 2018.
  57. Triolo, P. Technology crossroads: innovation in China's telecommunications and high-performance computer sec- tors threatened by US stranglehold on semiconductors. Asian Security, 19(2):143-158, May 2023.
  58. van Esch, P., Northey, G., Striluk, M., and Wilson, H. Au- tonomous weapon systems: Is a space warfare manual required? Computer Law & Security Review, 33(3):382- 389, June 2017.
  59. Wang, H., Cheng, H., and Hao, H. The use of unmanned aerial vehicle in military operations. In Man-Machine- Environment System Engineering, pp. 939-945. Springer Singapore, 2020.
  60. Warren, A. and Hillas, A. Friend or frenemy? The role of trust in human-machine teaming and lethal autonomous weapons systems. Small Wars & Insurgencies, 31(4): 822-850, May 2020.
  61. Warren, A. and Hillas, A. Red robots: Chinese military mod- ernization and perceptions on lethal autonomous weapons system. CIAR, 15(1):9-56, 2021.
  62. Warren, A. and Hillas, A. 'Xi Jinping Thought': Lethal autonomous weapons systems and military modernization with Chinese characteristics. The Journal of International Relations, Peace Studies, and Development, 7(1):6, 2022.

Cited by

Sorted by citations

Metadata

Authors
Simmons-Edler, R., Badman, R., Longpre, S., & Rajan, K.
Affiliations
  • Harvard University (US)
  • Massachusetts Institute of Technology (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2405.01859
arXiv
2405.01859
OpenAlex
W4396779577
Cited by
17
References
0
Citations by year
2026: 5 · 2025: 9 · 2024: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Engineering › Control and Systems Engineering › Military Strategy and Technology
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Defense, Military, and Policy Studies
Keywords
autonomous weapons, artificial intelligence in warfare, geopolitical stability, AI arms race, low-intensity conflict, war escalation, civilian casualties, AI regulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review