The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Personal AI, deception, and the problem of emotional bubbles
Abstract
Personal AI is a new type of AI companion, distinct from the prevailing forms of AI companionship. Instead of playing a narrow and well-defined social role, like friend, lover, caretaker, or colleague, with a set of pre-determined responses and behaviors, Personal AI is engineered to tailor itself to the user, including learning to mirror the user’s unique emotional language and attitudes. This paper identifies two issues with Personal AI. First, like other AI companions, it is deceptive about the presence of their emotions, which undermines the moral value of companionship. Second, Personal AI leads to a distinctly new form of deception concerning the origins of its emotions. Its emotional attitudes appear to belong to it, when in fact they are only reflections of the user. This results in what I dub “emotional bubbles”—the false impression that personal emotions are externally validated—which have at least two troubling implications. First, emotional bubbles prevent us from encountering emotional attitudes that differ from our own, which is likely to cripple emotional growth and the ability to form diverse social and emotional relationships. Second, if we assume, as some philosophers claim, that shared emotions are constitutive of shared values, it follows that Personal AI subverts joint moral deliberation. Users believe their personal values are externally validated, when they are only validated by themselves. Because of the absence of technovirtues able to handle this problem, I suggest that we proceed very cautiously with the development and marketing of Personal AI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The result is a more immersive form of emotional bonding in which the user perceives reciprocity, even though none truly exists (Mlonyeni, 2025).”
From Emotional AI and the rise of pseudo-intimacy: are we trading authenticity for algorithmic affection? (2025) · cites this paper as Mlonyeni, 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ECHO CHAMBERS AND EPISTEMIC BUBBLES
Nguyen, C. T. (2020) · Episteme · doi:10.1017/epi.2018.32 · Cited by 1,115
Should we worry about filter bubbles?
Zuiderveen Borgesius, F. J., Trilling, D., Möller, J., Bodó, B., de Vreese, C. H., & Helberger, N. (2016) · Internet Policy Review · doi:10.14763/2016.1.401 · Cited by 541
We need to talk about deception in social robotics!
Sharkey, A., & Sharkey, N. (2021) · Ethics and Information Technology · doi:10.1007/s10676-020-09573-9 · Cited by 179
Research works printed but not linked: 42
- Avnur Y (2020) What’s wrong with the online echo chamber: a motivated reasoning account. J Appl Philos 37(4):578–593. https://doi.org/10.1111/japp.12426 · DOI
- Belhassein K, Fernández-Castro V, Mayima A, Clodic A, Pacherie E, Guidetti M, Alami R, Cochet H (2022) Addressing joint action challenges in HRI: Insights from psychology and philosophy. Acta Psychol 222:103476. https://doi.org/10.1016/j.actpsy.2021.103476 · DOI
- Betzler M, Löschke J (2021) Collegial Relationships. Ethical Theory Moral Pract 24(1):213–229. https://doi.org/10.1007/s10677-021-10165-9 · DOI
- Bisconti P (2021) Will sexual robots modify human relationships? A psychological approach to reframe the symbolic argument. Adv Robot 35(9):561–571. https://doi.org/10.1080/01691864.2021.1886167 · DOI
- Brincker M (2021) Disoriented and alone in the experience machine–on netflix, shared world deceptions and the consequences of deepening algorithmic personalization. SATS 22(1):75–96. https://doi.org/10.1515/sats-2021-0005 · DOI
- Cocking D (2008) Plural selves and relational identity: intimacy and privacy online. In: van den Hoven J, Weckert J (eds) Information technology and moral philosophy. Cambridge University Press, Cambridge, pp. 123–141 · DOI
- Coeckelbergh M (2011) Artificial companions: empathy and vulnerability mirroring in human-robot relations. Studies Ethics, Law Technol 4(3):2. https://doi.org/10.2202/1941-6008.1126 · DOI
- Coeckelbergh M (2016) Care robots and the future of ICT-mediated elderly care: a response to doom scenarios. AI Soc 31(4):455–462. https://doi.org/10.1007/s00146-015-0626-3 · DOI
- Cooley DR (2002) False Friends. J Bus Ethics 36(3):195–206. https://doi.org/10.1023/A:1014042005642 · DOI
- Danaher J (2017) The symbolic-consequences argument in the sex robot debate. In: Danaher J, MacArthur N (eds) Robot sex: social and ethical implications. The MIT Press. https://doi.org/10.7551/mitpress/10718.003.0012 · DOI
- Danaher J (2018) Toward an ethics of AI assistants: an initial framework. Philos Technol 31(4):629–653. https://doi.org/10.1007/s13347-018-0317-3 · DOI
- Danaher J (2020a) Robot Betrayal: a guide to the ethics of robotic deception. Ethics Inf Technol 22(2):117–128. https://doi.org/10.1007/s10676-019-09520-3 · DOI · Paper page
- Danaher J (2020b) Welcoming robots into the moral circle: a defence of ethical behaviourism. Sci Eng Ethics 26(4):2023–2049. https://doi.org/10.1007/s11948-019-00119-x · DOI · Paper page
- Dauwerse L, van der Dam S, Abma T (2011) Morality in the mundane: specific needs for ethics support in elderly care. Nurs Ethics 19(1):91–103. https://doi.org/10.1177/0969733011412102 · DOI
- de Graaf MA (2016) An Ethical Evaluation of Human-Robot Relationships. Int J Soc Rob 8(4):589–598. https://doi.org/10.1007/s12369-016-0368-5 · DOI · Paper page
- Esposito E (2022) Artificial communication: how algorithms produce social intelligence. The MIT Press, Cambridge Mass. · DOI · Paper page
- Farina M, Zhdanov P, Karimov A, Lavazza A (2022) AI and society: a virtue ethics approach. AI Soc. https://doi.org/10.1007/s00146-022-01545-5 · DOI
- Goldberg SC (2011) If that were true I would have heard about It by now. In: Goldman AI, Whitcomb D (eds) Social epistemology: essential readings. Oxford University Press Oxford, pp 92–108
- Helm BW (2001) Emotional Reason: Deliberation, motivation, and the nature of value. Cambridge University Press, Cambridge · DOI · Paper page
- Helm BW (2009) Emotions as evaluative feelings. Emot Rev 1(3):248–255. https://doi.org/10.1177/1754073909103593 · DOI
- Kirby R, Forlizzi J, Simmons R (2010) Affective social robots. Rob Auton Syst 58(3):322–332. https://doi.org/10.1016/j.robot.2009.09.015 · DOI · Paper page
- Lancaster K (2019) The robotic touch: Why there is no good reason to prefer human nurses to carebots. Philos Contemp World 25(2):88–109 · doi:10.5840/pcw201925218
- Leerkes EM, Bailes LG (2019) Emotional development within the family context. In: LoBue V, Koraly P-E, Buss KA (eds) Handbook of emotional development. Springer Nature Switzerland AG, pp 627–661 · DOI
- Loghmani MRS, Rovetta GV, Venture G (2017) Emotional intelligence in robots: Recognizing human emotions from daily-life gestures. IEEE International Conference on Robotics and Automation (ICRA), 29 May–3 June 2017 · DOI
- Massa N, Bisconti P, Nardi D (2022) The psychological implications of companion robots: a theoretical framework and an experimental setup. Int J Soc Rob. https://doi.org/10.1007/s12369-021-00846-x · DOI
- Matthias A (2015) Robot lies in health care: when is deception morally permissible? Kennedy Inst Ethics J 25(2):169–192. https://doi.org/10.1353/ken.2015.0007 · DOI
- Misselhorn C, Störzinger T (2023) Social robots as echo chambers and opinion amplifiers. In: Misselhorn C, Poljanšek T, Störzinger T, Klein M (eds) Emotional machines: perspectives from affective computing and emotional human-machine interaction. Springer Fachmedien Wiesbaden, Wiesbaden, pp 247–275 · doi:10.1007/978-3-658-37641-3_10
- Mlonyeni PMT (2023) Collegiality, friendship, and the value of remote work. Ethical Theory Moral Pract 26(1):113–126. https://doi.org/10.1007/s10677-023-10368-2 · DOI
- Naneva S, Gou MS, Webb TL, Prescott TJ (2020) A systematic review of attitudes, anxiety, acceptance, and trust towards social robots. Int J Soc Rob 12(6):1179–1201. https://doi.org/10.1007/s12369-020-00659-4 · DOI · Paper page
- Nyholm S, Smids J (2020) Can a robot be a good colleague? Sci Eng Ethics 26(4):2169–2188. https://doi.org/10.1007/s11948-019-00172-6 · DOI
- Scarantino A, de Sousa R (2021) Emotion. In: Zalta E N (ed) The Stanford Encyclopedia of Philosophy.
- Sharkey A, Sharkey N (2012) Granny and the robots: ethical issues in robot care for the elderly. Ethics Inf Technol 14:27–40. https://doi.org/10.1007/s10676-010-9234-6 · DOI · Paper page
- Smids J (2020) Danaher’s ethical behaviourism: an adequate guide to assessing the moral status of a robot? Sci Eng Ethics 26(5):2849–2866. https://doi.org/10.1007/s11948-020-00230-4 · DOI
- Sparrow R (2002) The March of the robot dogs. Ethics Inf Technol 4(4):305–318. https://doi.org/10.1023/A:1021386708994 · DOI · Paper page
- Sparrow R (2016) Robots in aged care: a dystopian future? AI Soc 31(4):445–454. https://doi.org/10.1007/s00146-015-0625-4 · DOI
- Sparrow R, Sparrow L (2006) In the hands of machines? The future of aged care. Minds Mach 16(2):141–161. https://doi.org/10.1007/s11023-006-9030-6 · DOI · Paper page
- Spezialetti M, Placidi G, Rossi S (2020) Emotion recognition for human-robot interaction: recent advances and future perspectives. Front Robot AI 21(7):1–11. https://doi.org/10.3389/frobt.2020.532279 · DOI · Paper page
- Tamaki S (2011) Beautiful fighting girl. Vincent J K, Lawson D (trans). University of Minnesota Press, Minneapolis · doi:10.5749/minnesota/9780816654505.001.0001
- Tao J, Tan T (2005) Affective Computing: A Review. In: Tao J, Tan T, Picard RW (eds) Affective computing and intelligent interaction. ACII 2005. Lecture Notes in Computer Science, vol 3784. Springer Berlin Heidelberg, Heidelberg
- Vallor S (2016) Technology and the Virtues: A Philosophical Guide to a Future Worth Wanting. Oxford University Press, Oxford · DOI · Paper page
- Weber-Guskar E (2021) How to feel about emotionalized artificial intelligence? When robot pets, holograms, and chatbots become affective partners. Ethics Inf Technol 23(4):601–610. https://doi.org/10.1007/s10676-021-09598-8 · DOI
- Zhao S (2006) Humanoid social robots as a medium of communication. New Media Soc 8(3):401–419. https://doi.org/10.1177/1461444806061951 · DOI · Paper page
Cited by
Emotional AI and the rise of pseudo-intimacy: are we trading authenticity for algorithmic affection?
Babu, J., Joseph, D., Kumar, R. M., Alexander, E., Sasi, R., & Joseph, J. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1679324 · Cited by 76
Liu, T., Lo, T.-Y., Wen, K.-H., Sun, Y., & Wei, Z.-Q. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1687686 · Cited by 13
Bhat, M. (2025) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v8i1.36560 · Cited by 9
A scoping review on the mental health harms of LLM-based chatbots
Diel, A., Torous, J., Cuijpers, P., Kleesiek, J., Nensa, F., Weber, N., Faust, F., Lalgi, T. J., Mellis, F. S., Teufel, M., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-03054-x · Cited by 4
“Syco-phantic responses may validate or reinforce delusional thoughts, harmful behavior, or maladaptive thinking 37,42,43,49 by creating the impression that expressed beliefs are externally socially validated when they are actually mirrored by the LLM-CB 57 .”
"Eternal Youth": the Ethics of an Ideological Alliance Between Youth and Artificial Intelligence
Artamonov, D. S., Nevvazhay, I. D., Suslov, I. V., Syusyukin, V. A., & Tikhonova, S. V. (2025) · 2025 IEEE Ural-Siberian Conference on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT) · doi:10.1109/usbereit65494.2025.11054193 · Cited by 1
Metadata
- Authors
- Mlonyeni, P. M. T.
- Venue
- AI & SOCIETY
- Published
- 2024-05-20
- Volume
- 40
- Issue
- 3
- Pages
- 1927-1938
- ISSN
- 0951-5666
- 1435-5655
- Cited by
- 28
- References
- 3
- Language
- en
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Funders
- OsloMet - Oslo Metropolitan University
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Personal AI, deception, and the problem of emotional bubbles. https://app.cochat.ai/papers/10.1007/s00146-024-01958-4
- The paper
- Mlonyeni, P. M. T. (2025). Personal AI, deception, and the problem of emotional bubbles. AI & SOCIETY. https://doi.org/10.1007/s00146-024-01958-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.