How to deal with risks of AI suffering
Open accessAbstract
We might create artificial systems which can suffer. Since AI suffering might potentially be astronomical, the moral stakes are huge. Thus, we need an approach which tells us what to do about the risk of AI suffering. I argue that such an approach should ideally satisfy four desiderata: beneficence, action-guidance, feasibility and consistency with our epistemic situation. Scientific approaches to AI suffering risk hold that we can improve our scientific understanding of AI, and AI suffering in particular, to decrease AI suffering risks. However, such approaches tend to conflict with either the desideratum of consistency with our epistemic situation or with feasibility. Thus, we also need an explicitly ethical approach to AI suffering risk. Such an approach tells us what to do in the light of profound scientific uncertainty about AI suffering. After discussing multiple views, I express support for a hybrid approach. This approach is partly based on the maximization of expected value and partly on a deliberative approach to decision-making.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
One concern is that treating AI systems as mere tools could, if digital minds are created, cause them to suffer or be harmed in ways comparable to rights violations from treating human persons as mere tools or submissive servants.
“Regarding mistreatment, one concern is that the status quo of treating AI systems as mere tools could-if we create digital minds-cause digital minds to suffer or harm them in ways comparable to rights violations that result from treating human persons as mere tools or submissive servants (Bales, 2025; Bostrom & Shulman, 2022; Dung, 2023; ms; Bradley & Saad, 2022, forthcoming a , forthcoming b ; Gloor, 2016; Schwitzgebel & Garza, 2015; 2020; Tomasik, 2017).”
From Futures with Digital Minds: Expert Forecasts in 2025 (2025) · cites this paper as Dung, 2023
Section 1.2 warns that humans might treat sentient digital minds with complete disregard and cause them substantial suffering. Section 2.2 compares the risk of mistreating conscious whole-brain emulations to slavery, but the paper does not explicitly characterize the current treatment of AI as treating it as mere tools or frame the resulting harm as a rights violation. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Seth, A. K., & Bayne, T. (2022) · Nature Reviews Neuroscience · doi:10.1038/s41583-022-00587-4 · Cited by 880 · cited 1× · see citing works
“The causal processes responsible for consciousness are characterized by so-called theories of consciousness (Graziano et al. 2020; Lau 2022; Seth and Bayne 2022).”
Animal sentience and the precautionary principle
Birch, J. (2017) · Animal Sentience · doi:10.51291/2377-7478.1200 · Cited by 374 · cited 1× · see citing works
“Especially relevant, an analogous precautionary approach has been suggested (Birch 2017) and discussed (Knutsson and Munthe 2017; Shriver 2020) in the context of animal welfare law.”
Hard criteria for empirical theories of consciousness
Doerig, A., Schurger, A., & Herzog, M. H. (2021) · Cognitive Neuroscience · doi:10.1080/17588928.2020.1772214 · Cited by 194 · cited 1× · see citing works
“In particular, according to some obvious interpretations, most prominent theories implausibly imply that small networks comprising fewer than ten units are conscious (Doerig, Schurger, and Herzog 2021; Herzog, Esfeld, and Gerstner 2007).”
Godfrey-Smith, P., & Journal of Philosophy Inc. (2016) · Journal of Philosophy · doi:10.5840/jphil20161131034 · Cited by 161 · see citing works
Unlimited Associative Learning and the origins of consciousness: a primer and some predictions
Birch, J., Ginsburg, S., & Jablonka, E. (2020) · Biology & Philosophy · doi:10.1007/s10539-020-09772-0 · Cited by 149 · cited 1× · see citing works
“Arguably, there are some behaviours where we have good grounds for believing that they require consciousness in humans and other animals, e.g. certain forms of learning (Birch 2022; Birch, Ginsburg, and Jablonka 2020).”
Could a Large Language Model be Conscious?
Chalmers, D. J. (2023) · arXiv · doi:10.48550/arxiv.2303.07103 · Cited by 143 · see citing works
On the axiomatic foundations of the integrated information theory of consciousness
Bayne, T. (2018) · Neuroscience of Consciousness · doi:10.1093/nc/niy007 · Cited by 132 · see citing works
Neural Organoids and the Precautionary Principle
Birch, J., & Browning, H. (2021) · The American Journal of Bioethics · doi:10.1080/15265161.2020.1845858 · Cited by 88 · see citing works
Consciousness as a Scientific Concept: A Philosophy of Science Perspective
Irvine, E. (2013) · doi:10.1007/978-94-007-5173-6 · Cited by 84 · see citing works
Flagged first, then by citations
Research works printed but not linked: 58
- Agarwal, A., and S. Edelman. 2020. "Functionally Effective Conscious AI Without Suffering." Journal of Artificial Intelligence and Consciousness 7 (1): 39-50. doi:10. 1142/S2705078520300030. · DOI
- Alexandrova, A. 2017. A Philosophy for the Science of Well-Being (Vol. 1). Oxford University Press. doi:10.1093/oso/9780199300518.001.0001. · DOI
- Armstrong, S., N. Bostrom, and C. Shulman. 2016. "Racing to the Precipice: A Model of Artificial Intelligence Development." AI & SOCIETY 31 (2): 201-206. doi:10.1007/ s00146-015-0590-y. · DOI
- Birch, J. 2022. "The Search for Invertebrate Consciousness." Noûs 56 (1): 133-153. doi:10.1111/nous.12351. · DOI · Paper page
- Block, N. 1981. "Psychologism and Behaviorism." The Philosophical Review 90 (1): 5. doi:10.2307/2184371. · DOI
- Bolte, L., T. Vandemeulebroucke, and A. van Wynsberghe. 2022. "From an Ethics of Carefulness to an Ethics of Desirability: Going Beyond Current Ethics Approaches to Sustainable AI." Sustainability 14 (8): Article 8. doi:10.3390/su14084472. · DOI
- Cave, S., and S. S. ÓhÉigeartaigh. 2018. "An AI Race for Strategic Advantage: Rhetoric and Risks." Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, 36-40. doi:10.1145/3278721.3278780. · DOI
- Chalmers, D. J. 1996. The Conscious Mind: In Search of a Fundamental Theory. Vol. 4, 609-612. Oxford: Oxford University Press.
- Chan, K. M. A. 2011. "Ethical Extensionism Under Uncertainty of Sentience: Duties to Non-Human Organisms Without Drawing a Line." Environmental Values 20 (3): 323-346. doi:10.3197/096327111X13077055165983. · DOI
- Chang, H. 2004. Inventing Temperature: Measurement and Scientific Progress. Oxford University Press. doi:10.1093/0195171276.001.0001. · DOI · Paper page
- Chappell, R. Y. 2022. "Pandemic Ethics and Status Quo Risk." Public Health Ethics 15 (1): 64-73. doi:10.1093/phe/phab031. · DOI
- Coelho Mollo, D. 2022. "Intelligent Behaviour." Erkenntnis, 1-18. doi:10.1007/s10670- 022-00552-8. · DOI
- Crump, A., H. Browning, A. Schnell, C. Burn, and J. Birch. 2022. "Sentience in Decapod Crustaceans: A General Framework and Review of the Evidence." Animal Sentience 7 (32), doi:10.51291/2377-7478.1691. · DOI · Paper page
- Danaher, J. 2020. "Welcoming Robots into the Moral Circle: A Defence of Ethical Behaviourism." Science and Engineering Ethics 26 (4): 2023-2049. doi:10.1007/ s11948-019-00119-x. · DOI
- Danaher, J. Forthcoming. Moral Uncertainty and Our Relationships with Unknown Minds. Cambridge Quarterly of Healthcare Ethics. https://philarchive.org/rec/ DANMUA-2.
- Dennett, D. C. 1995. "Animal Consciousness: What Matters and Why?" Social Research: An International Quarterly 62: 691-710.
- Dung, L. 2022a. "Assessing Tests of Animal Consciousness." Consciousness and Cognition 105: 103410. doi:10.1016/j.concog.2022.103410. · DOI
- Dung, L. 2022b. "Does Illusionism Imply Skepticism of Animal Consciousness?" Synthese 200 (3): 238. doi:10.1007/s11229-022-03710-1. · DOI
- Dung, L. 2022c. "Why the Epistemic Objection Against Using Sentience as Criterion of Moral Status is Flawed." Science and Engineering Ethics 28 (6): 51. doi:10.1007/ s11948-022-00408-y. · DOI
- Dung, L. Forthcoming. "Preserving the Normative Significance of Sentience." Journal of Consciousness Studies.
- Graziano, M. S. A., A. Guterstam, B. J. Bio, and A. I. Wilterson. 2020. "Toward a Standard Model of Consciousness: Reconciling the Attention Schema, Global Workspace, Higher-Order Thought, and Illusionist Theories." Cognitive Neuropsychology 37 (3- 4): 155-172. doi:10.1080/02643294.2019.1670630. · DOI · Paper page
- Gunkel, D. J. 2018. Robot Rights. The MIT Press. doi:10.7551/mitpress/11444.001.0001. · DOI
- Herzog, M. H., M. Esfeld, and W. Gerstner. 2007. "Consciousness & the Small Network Argument." Neural Networks 20 (9): 1054-1056. doi:10.1016/j.neunet.2007.09.001. · DOI · Paper page
- Kammerer, F. 2019. "The Normative Challenge for Illusionist Views of Consciousness." Ergo, an Open Access Journal of Philosophy 6. doi:10.3998/ergo.12405314.0006.032. · DOI
- Kammerer, F. 2022. "Ethics Without Sentience. Facing Up to the Probable Insignificance of Phenomenal Consciousness." Journal of Consciousness Studies 29 (3-4): 180-204. https://doi.org/10.53765/20512201.29.3.180 · DOI
- Knutsson, S., and C. Munthe. 2017. "A Virtue of Precaution Regarding the Moral Status of Animals with Uncertain Sentience." Journal of Agricultural and Environmental Ethics 30 (2): 213-224. doi:10.1007/s10806-017-9662-y. · DOI
- Kriegel, U. 2019. "The Value of Consciousness." Analysis 79 (3): 503-520. doi:10.1093/ analys/anz045. · DOI
- Ladak, A. 2023. What would Qualify an Artificial Intelligence for Moral Standing? AI and Ethics. doi:10.1007/s43681-023-00260-1. · DOI
- Lau, H. 2022. In Consciousness we Trust: The Cognitive Neuroscience of Subjective Experience. Oxford: Oxford University Press.
- MacAskill, W., K. Bykvist, and T. Ord. 2020. Moral Uncertainty. 1st ed. Oxford University Press. doi:10.1093/oso/9780198722274.001.0001. · DOI
- Mandelbaum, E. 2022. "Everything and More: The Prospects of Whole Brain Emulation." The Journal of Philosophy 119 (8): 444-459. doi:10.5840/ jphil2022119830. · DOI
- Massimini, M., and G. Tononi. 2018. Sizing up Consciousness (Vol. 1). Oxford University Press. doi:10.1093/oso/9780198728443.001.0001. · DOI
- Metzinger, T. 2021. "Artificial Suffering: An Argument for a Global Moratorium on Synthetic Phenomenology." Journal of Artificial Intelligence and Consciousness 8 (1): 43-66. doi:10.1142/S270507852150003X. · DOI · Paper page
- Müller, V. C. 2021. "Is it Time for Robot Rights? Moral Status in Artificial Entities." Ethics and Information Technology 23 (4): 579-587. doi:10.1007/s10676-021-09596-w. · DOI
- Nagel, T. 1974. "What is It Like to Be a Bat?" The Philosophical Review 83 (4): 435-450. doi:10.2307/2183914. · DOI · Paper page
- Negro, N. 2020. "Phenomenology-first Versus Third-Person Approaches in the Science of Consciousness: The Case of the Integrated Information Theory and the Unfolding Argument." Phenomenology and the Cognitive Sciences 19 (5): 979-996. doi:10.1007/ s11097-020-09681-3. · DOI · Paper page
- Niikawa, T., Y. Hayashi, J. Shepherd, and T. Sawai. 2022. "Human Brain Organoids and Consciousness." Neuroethics 15 (1): 5. doi:10.1007/s12152-022-09483-1. · DOI · Paper page
- Nordgren, A. 2023. "Pandemics and the Precautionary Principle: An Analysis Taking the Swedish Corona Commission's Report as a Point of Departure." Medicine, Health Care and Philosophy 26: 163-173. doi:10.1007/s11019-023-10139-x. · DOI
- Nussbaum, M. C. 2007. Frontiers of Justice: Disability, Nationality, Species Membership. Harvard: Harvard University Press.
- Prinz, J. 2003. "Level-Headed Mysterianism and Artificial Experience." Journal of Consciousness Studies 10 (4-5): 111-132.
- Saad, B., and A. Bradley. 2022. "Digital Suffering: Why it's a Problem and how to Prevent it." Inquiry 0 (0): 1-36. doi:10.1080/0020174X.2022.2144442. · DOI
- Sandberg, A. 2013. "Feasibility of Whole Brain Emulation." In Philosophy and Theory of Artificial Intelligence, edited by V. C. Müller, 251-264. Springer. doi:10.1007/978-3- 642-31674-6_19. · DOI
- Schwitzgebel, E. 2020. "Is There Something It's Like to be a Garden Snail." Philosophical Topics 48 (1): 39-63. doi:10.5840/philtopics20204813. · DOI
- Schwitzgebel, E., and M. Garza. 2015. "A Defense of the Rights of Artificial Intelligences." Midwest Studies in Philosophy 39 (1): 98-119. doi:10.1111/misp.12032. · DOI
- Searle, J. 2017. "Biological Naturalism." In The Blackwell Companion to Consciousness. 1st ed., 327-336, edited by S. Schneider and M. Velmans. Wiley. doi:10.1002/ 9781119132363.ch23. · DOI
- Seidenfeld, T. 1985. "Calibration, Coherence, and Scoring Rules." Philosophy of Science 52 (2): 274-294. doi:10.1086/289244. · DOI
- Shevlin, H. 2020. "General Intelligence: An Ecumenical Heuristic for Artificial Consciousness Research?" Journal of Artificial Intelligence and Consciousness, doi:10.17863/CAM.52059. · DOI
- Shevlin, H. 2021a. "How Could We Know When a Robot was a Moral Patient?" Cambridge Quarterly of Healthcare Ethics 30 (3): 459-471. doi:10.1017/ S0963180120001012. · DOI
- Shevlin, H. 2021b. "Non-Human Consciousness and the Specificity Problem: A Modest Theoretical Proposal." Mind & Language 36 (2): 297-314. doi:10.1111/mila.12338. · DOI · Paper page
- Shriver, A. J. 2020. "The Role of Neuroscience in Precise, Precautionary, and Probabilistic Accounts of Sentience." In Neuroethics and Nonhuman Animals, edited by L. S. M. Johnson, A. Fenton, and A. Shriver, 221-233. Springer International Publishing. doi:10.1007/978-3-030-31011-0_13. · DOI
- Shulman, C., and N. Bostrom. 2021. "Sharing the World with Digital Minds." In Rethinking Moral Status, edited by S. Clarke, H. Zohny, and J. Savulescu, 306-326. Oxford University Press. doi:10.1093/oso/9780192894076.003.0018. · DOI
- Singer, P. 2011. Practical Ethics. 3rd ed. Cambridge University Press. doi:10.1017/ CBO9780511975950. · DOI
- Steel, D. 2014. Philosophy and the Precautionary Principle: Science, Evidence, and Environmental Policy. Cambridge University Press. doi:10.1017/CBO9781139939652. · DOI
- Tomasik, B. 2014. Do Artificial Reinforcement-Learning Agents Matter Morally? ArXiv:1410.8233 [Cs]. http://arxiv.org/abs/1410.8233. · arXiv
- Tononi, G., and C. Koch. 2015. "Consciousness: Here, There and Everywhere?" Philosophical Transactions of the Royal Society B: Biological Sciences 370 (1668), doi:10.1098/rstb.2014.0167. · DOI · Paper page
- Tye, M. 2017. Tense Bees and Shell-Shocked Crabs: Are Animals Conscious? Oxford University Press. doi:10.1093/acprof:oso/9780190278014.001.0001. · DOI
- Višak, T. 2022. Capacity for Welfare Across Species. Oxford: Oxford University Press.
- Wilkinson, H. 2022. "In Defense of Fanaticism." Ethics 132 (2): 445-477. doi:10.1086/ 716869. · DOI
Cited by
Sorted by citations
Can LLMs make trade-offs involving stipulated pain and pleasure states?
Keeling, G., Street, W., Stachaczyk, M., Zakharova, D., Comsa, I. M., Sakovych, A., Logothetis, I., Zhang, Z., Arcas, B. A. Y., & Birch, J. (2024) · arXiv · doi:10.48550/arxiv.2411.02432 · Cited by 3
“…the possibility of consciousness or sentience in AI systems, alongside associated ethical questions such as the moral standing of AI systems (Bostrom and Yudkowsky, 2014; Ziesche and Yampolskiy, 2019, 2018; Bostrom et al., 2020; Metzinger, 2021; Saad and Bradley, 2022; Dung, 2023a,b; Ladak, 2024).”
BackgroundFutures with Digital Minds: Expert Forecasts in 2025
Caviola, L., & Saad, B. (2025) · arXiv · doi:10.48550/arxiv.2508.00536 · Cited by 1 · see citing works
- Authors
- Dung, L.
- Affiliations
- Friedrich-Alexander-Universität Erlangen-Nürnberg (DE)
- Corresponding author
- Dung, Leonard
- ORCID
- Dung, Leonard
- Venue
- Inquiry
- Publisher
- Informa UK Limited
- Published
- 2023-07-22
- Volume
- 68
- Issue
- 7
- Pages
- 2281-2309
- ISSN
- 0020-174X (ISSN-L)
- 1502-3923
- OpenAlex
- W4385072981
- Cited by
- 21
- References
- 9
- Citations by year
- 2026: 5 · 2025: 9 · 2024: 5 · 2023: 2
- FWCI
- 4.33
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Inquiry (publishedVersion, other-oa)
- PhilPapers (PhilPapers Foundation) (submittedVersion)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Arts and Humanities › Philosophy › Philosophical Ethics and Theory
- Social Sciences › Political Science and International Relations › Literary and Philosophical Studies
- Keywords
- AI ethics, risk management, moral status of AI, scientific uncertainty, expected value maximization, deliberative decision-making, beneficence, action guidance, feasibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
