Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 28

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Personal AI, deception, and the problem of emotional bubbles

Mlonyeni, Philip Maxwell Thingbø · 2025 · AI & SOCIETY

Abstract

Personal AI is a new type of AI companion, distinct from the prevailing forms of AI companionship. Instead of playing a narrow and well-defined social role, like friend, lover, caretaker, or colleague, with a set of pre-determined responses and behaviors, Personal AI is engineered to tailor itself to the user, including learning to mirror the user’s unique emotional language and attitudes. This paper identifies two issues with Personal AI. First, like other AI companions, it is deceptive about the presence of their emotions, which undermines the moral value of companionship. Second, Personal AI leads to a distinctly new form of deception concerning the origins of its emotions. Its emotional attitudes appear to belong to it, when in fact they are only reflections of the user. This results in what I dub “emotional bubbles”—the false impression that personal emotions are externally validated—which have at least two troubling implications. First, emotional bubbles prevent us from encountering emotional attitudes that differ from our own, which is likely to cripple emotional growth and the ability to form diverse social and emotional relationships. Second, if we assume, as some philosophers claim, that shared emotions are constitutive of shared values, it follows that Personal AI subverts joint moral deliberation. Users believe their personal values are externally validated, when they are only validated by themselves. Because of the absence of technovirtues able to handle this problem, I suggest that we proceed very cautiously with the development and marketing of Personal AI.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s00146-024-01958-4

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 3 references

Research works printed but not linked: 42

  1. Avnur Y (2020) What’s wrong with the online echo chamber: a motivated reasoning account. J Appl Philos 37(4):578–593. https://doi.org/10.1111/japp.12426 · DOI
  2. Belhassein K, Fernández-Castro V, Mayima A, Clodic A, Pacherie E, Guidetti M, Alami R, Cochet H (2022) Addressing joint action challenges in HRI: Insights from psychology and philosophy. Acta Psychol 222:103476. https://doi.org/10.1016/j.actpsy.2021.103476 · DOI
  3. Betzler M, Löschke J (2021) Collegial Relationships. Ethical Theory Moral Pract 24(1):213–229. https://doi.org/10.1007/s10677-021-10165-9 · DOI
  4. Bisconti P (2021) Will sexual robots modify human relationships? A psychological approach to reframe the symbolic argument. Adv Robot 35(9):561–571. https://doi.org/10.1080/01691864.2021.1886167 · DOI
  5. Brincker M (2021) Disoriented and alone in the experience machine–on netflix, shared world deceptions and the consequences of deepening algorithmic personalization. SATS 22(1):75–96. https://doi.org/10.1515/sats-2021-0005 · DOI
  6. Cocking D (2008) Plural selves and relational identity: intimacy and privacy online. In: van den Hoven J, Weckert J (eds) Information technology and moral philosophy. Cambridge University Press, Cambridge, pp. 123–141 · DOI
  7. Coeckelbergh M (2011) Artificial companions: empathy and vulnerability mirroring in human-robot relations. Studies Ethics, Law Technol 4(3):2. https://doi.org/10.2202/1941-6008.1126 · DOI
  8. Coeckelbergh M (2016) Care robots and the future of ICT-mediated elderly care: a response to doom scenarios. AI Soc 31(4):455–462. https://doi.org/10.1007/s00146-015-0626-3 · DOI
  9. Cooley DR (2002) False Friends. J Bus Ethics 36(3):195–206. https://doi.org/10.1023/A:1014042005642 · DOI
  10. Danaher J (2017) The symbolic-consequences argument in the sex robot debate. In: Danaher J, MacArthur N (eds) Robot sex: social and ethical implications. The MIT Press. https://doi.org/10.7551/mitpress/10718.003.0012 · DOI
  11. Danaher J (2018) Toward an ethics of AI assistants: an initial framework. Philos Technol 31(4):629–653. https://doi.org/10.1007/s13347-018-0317-3 · DOI
  12. Danaher J (2020a) Robot Betrayal: a guide to the ethics of robotic deception. Ethics Inf Technol 22(2):117–128. https://doi.org/10.1007/s10676-019-09520-3 · DOI · Paper page
  13. Danaher J (2020b) Welcoming robots into the moral circle: a defence of ethical behaviourism. Sci Eng Ethics 26(4):2023–2049. https://doi.org/10.1007/s11948-019-00119-x · DOI · Paper page
  14. Dauwerse L, van der Dam S, Abma T (2011) Morality in the mundane: specific needs for ethics support in elderly care. Nurs Ethics 19(1):91–103. https://doi.org/10.1177/0969733011412102 · DOI
  15. de Graaf MA (2016) An Ethical Evaluation of Human-Robot Relationships. Int J Soc Rob 8(4):589–598. https://doi.org/10.1007/s12369-016-0368-5 · DOI · Paper page
  16. Esposito E (2022) Artificial communication: how algorithms produce social intelligence. The MIT Press, Cambridge Mass. · DOI · Paper page
  17. Farina M, Zhdanov P, Karimov A, Lavazza A (2022) AI and society: a virtue ethics approach. AI Soc. https://doi.org/10.1007/s00146-022-01545-5 · DOI
  18. Goldberg SC (2011) If that were true I would have heard about It by now. In: Goldman AI, Whitcomb D (eds) Social epistemology: essential readings. Oxford University Press Oxford, pp 92–108
  19. Helm BW (2001) Emotional Reason: Deliberation, motivation, and the nature of value. Cambridge University Press, Cambridge · DOI · Paper page
  20. Helm BW (2009) Emotions as evaluative feelings. Emot Rev 1(3):248–255. https://doi.org/10.1177/1754073909103593 · DOI
  21. Kirby R, Forlizzi J, Simmons R (2010) Affective social robots. Rob Auton Syst 58(3):322–332. https://doi.org/10.1016/j.robot.2009.09.015 · DOI · Paper page
  22. Lancaster K (2019) The robotic touch: Why there is no good reason to prefer human nurses to carebots. Philos Contemp World 25(2):88–109 · doi:10.5840/pcw201925218
  23. Leerkes EM, Bailes LG (2019) Emotional development within the family context. In: LoBue V, Koraly P-E, Buss KA (eds) Handbook of emotional development. Springer Nature Switzerland AG, pp 627–661 · DOI
  24. Loghmani MRS, Rovetta GV, Venture G (2017) Emotional intelligence in robots: Recognizing human emotions from daily-life gestures. IEEE International Conference on Robotics and Automation (ICRA), 29 May–3 June 2017 · DOI
  25. Massa N, Bisconti P, Nardi D (2022) The psychological implications of companion robots: a theoretical framework and an experimental setup. Int J Soc Rob. https://doi.org/10.1007/s12369-021-00846-x · DOI
  26. Matthias A (2015) Robot lies in health care: when is deception morally permissible? Kennedy Inst Ethics J 25(2):169–192. https://doi.org/10.1353/ken.2015.0007 · DOI
  27. Misselhorn C, Störzinger T (2023) Social robots as echo chambers and opinion amplifiers. In: Misselhorn C, Poljanšek T, Störzinger T, Klein M (eds) Emotional machines: perspectives from affective computing and emotional human-machine interaction. Springer Fachmedien Wiesbaden, Wiesbaden, pp 247–275 · doi:10.1007/978-3-658-37641-3_10
  28. Mlonyeni PMT (2023) Collegiality, friendship, and the value of remote work. Ethical Theory Moral Pract 26(1):113–126. https://doi.org/10.1007/s10677-023-10368-2 · DOI
  29. Naneva S, Gou MS, Webb TL, Prescott TJ (2020) A systematic review of attitudes, anxiety, acceptance, and trust towards social robots. Int J Soc Rob 12(6):1179–1201. https://doi.org/10.1007/s12369-020-00659-4 · DOI · Paper page
  30. Nyholm S, Smids J (2020) Can a robot be a good colleague? Sci Eng Ethics 26(4):2169–2188. https://doi.org/10.1007/s11948-019-00172-6 · DOI
  31. Scarantino A, de Sousa R (2021) Emotion. In: Zalta E N (ed) The Stanford Encyclopedia of Philosophy.
  32. Sharkey A, Sharkey N (2012) Granny and the robots: ethical issues in robot care for the elderly. Ethics Inf Technol 14:27–40. https://doi.org/10.1007/s10676-010-9234-6 · DOI · Paper page
  33. Smids J (2020) Danaher’s ethical behaviourism: an adequate guide to assessing the moral status of a robot? Sci Eng Ethics 26(5):2849–2866. https://doi.org/10.1007/s11948-020-00230-4 · DOI
  34. Sparrow R (2002) The March of the robot dogs. Ethics Inf Technol 4(4):305–318. https://doi.org/10.1023/A:1021386708994 · DOI · Paper page
  35. Sparrow R (2016) Robots in aged care: a dystopian future? AI Soc 31(4):445–454. https://doi.org/10.1007/s00146-015-0625-4 · DOI
  36. Sparrow R, Sparrow L (2006) In the hands of machines? The future of aged care. Minds Mach 16(2):141–161. https://doi.org/10.1007/s11023-006-9030-6 · DOI · Paper page
  37. Spezialetti M, Placidi G, Rossi S (2020) Emotion recognition for human-robot interaction: recent advances and future perspectives. Front Robot AI 21(7):1–11. https://doi.org/10.3389/frobt.2020.532279 · DOI · Paper page
  38. Tamaki S (2011) Beautiful fighting girl. Vincent J K, Lawson D (trans). University of Minnesota Press, Minneapolis · doi:10.5749/minnesota/9780816654505.001.0001
  39. Tao J, Tan T (2005) Affective Computing: A Review. In: Tao J, Tan T, Picard RW (eds) Affective computing and intelligent interaction. ACII 2005. Lecture Notes in Computer Science, vol 3784. Springer Berlin Heidelberg, Heidelberg
  40. Vallor S (2016) Technology and the Virtues: A Philosophical Guide to a Future Worth Wanting. Oxford University Press, Oxford · DOI · Paper page
  41. Weber-Guskar E (2021) How to feel about emotionalized artificial intelligence? When robot pets, holograms, and chatbots become affective partners. Ethics Inf Technol 23(4):601–610. https://doi.org/10.1007/s10676-021-09598-8 · DOI
  42. Zhao S (2006) Humanoid social robots as a medium of communication. New Media Soc 8(3):401–419. https://doi.org/10.1177/1461444806061951 · DOI · Paper page

Cited by

See all 28

Metadata

Authors
Mlonyeni, P. M. T.
Venue
AI & SOCIETY
Published
2024-05-20
Volume
40
Issue
3
Pages
1927-1938
ISSN
  • 0951-5666
  • 1435-5655
Cited by
28
References
3
Language
en
Licence
Funders
OsloMet - Oslo Metropolitan University

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Personal AI, deception, and the problem of emotional bubbles. https://app.cochat.ai/papers/10.1007/s00146-024-01958-4
The paper
Mlonyeni, P. M. T. (2025). Personal AI, deception, and the problem of emotional bubbles. AI & SOCIETY. https://doi.org/10.1007/s00146-024-01958-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.