The DOI matches its Crossref record; 153 of the 153 references it cites were found in a registry or an index; 70 of the 135 sources its claims cite were checked for support.
The benefits and dangers of anthropomorphic conversational agents
Open accessAbstract
A growing body of research suggests that the recent generation of large language model (LLMs) excel, and in many cases outpace humans, at writing persuasively and empathetically, at inferring user traits from text, and at mimicking human-like conversation believably and effectively—without possessing any true empathy or social understanding. We refer to these systems as “anthropomorphic conversational agents” to aptly conceptualize the ability of LLM-based systems to mimic human communication so convincingly that they become increasingly indistinguishable from human interlocutors. This ability challenges the many efforts that caution against “anthropomorphizing” LLMs, attaching human-like qualities to nonhuman entities. When the systems themselves exhibit human-like qualities, calls to resist anthropomorphism will increasingly fall flat. While the AI industry directs much effort into improving the reasoning abilities of LLMs—with mixed results—the progress in communicative abilities remains underappreciated. In this perspective, we aim to raise awareness for both the benefits and dangers of anthropomorphic agents. We ask: should we lean into the human-like abilities, or should we aim to dehumanize LLM-based systems, given concerns over anthropomorphic seduction? When users cannot tell the difference between human interlocutors and AI systems, threats emerge of deception, manipulation, and disinformation at scale. We suggest that we must engage with anthropomorphic agents across design and development, deployment and use, and regulation and policy-making. We outline in detail implications and associated research questions.
Overview
CoChat Summary
TL;DR
The paper argues that recent LLMs have become “anthropomorphic conversational agents” that convincingly mimic human communication—persuasive and empathetic writing, trait inference, and role-play—often matching or exceeding average human performance. It warns that these abilities create both accessible conversational interfaces and risks of “anthropomorphic seduction,” deception, manipulation, and disinformation at scale, requiring attention in design, deployment, and regulation.
Key findings
- LLMs excel in three communicative areas: persuasive/empathetic writing, inference of emotion/sentiment/user traits, and believable interaction/role-play. Examples include GPT-4 “significantly outperform[ing] human participants across every topic and demographic” in persuasiveness; a 1,000-participant study finding “statistically significant superiority of the empathetic responding capability of LLMs over humans,” with GPT-4 “most empathetic”; a fine-tuned LLM detecting emotions in social media posts with 84% accuracy; and another finding over 95% precision for sentiment across multiple languages.
- Because the systems themselves exhibit human-like communicative qualities, the paper argues anthropomorphism can no longer be located only in users’ minds; calls to resist anthropomorphizing will “increasingly fall flat” when users cannot distinguish AI from humans, as Turing test evidence suggests.
- Benefits include conversational UX, making dense information more accessible (e.g., HR, medical, analytics), tutoring/coaching/mentoring (e.g., Khanmigo), and health behavior engagement; dangers include “anthropomorphic seduction,” trust exploitation, privacy risks, deceptive/manipulative writing, and manipulation at scale, especially with open-source or jailbroken models.
- The paper calls for benchmarks of anthropomorphic qualities across writing, inference, and interaction; context-dependent deployment decisions; transparency/labeling and possible safety ratings; and urgent engagement because business models around anthropomorphic agents will likely solidify in the next two years.
Methods
- Design: perspective/conceptual article, not an empirical study; the authors synthesize emerging research and outline implications and research questions for design/development, deployment/use, and regulation/policy.
- Data or sample: Not stated in the paper (no primary dataset or sample collected); it cites prior studies, including a 1,000-participant between-subject study and other experiments.
- Main analysis: narrative literature synthesis and conceptual argument; no formal meta-analysis or statistical analysis reported.
Limitations
- The paper states it is “too early to give clear directives” because the technology is progressing fast and the effects of anthropomorphic agents are not yet fully understood; early research efforts are “disconnected.”
- The paper states regulation is still emerging and the effectiveness of warnings/labels remains uncertain, given mixed past results with labeling.
- Evident from the text: as a perspective, it presents no new empirical data, systematic review protocol, or causal tests; many cited findings are preprints or secondary sources, and the paper itself notes mixed results (e.g., humans can write more empathetically when explicitly told to do so).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
42 Valid · 2 Not valid · 28 Not checked · 71 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“they consistently pass the Turing test (20) as a result (14, 21, 22)”
- Also stated in: § Anthropomorphic Conversational Agents · boosted
“they now pass the Turing test reliably (14, 21, 22)”
§ The benefits and dangers of anthropomorphic conversational agents > Abstract
What the sources say
- Not checked — no full text of the sourcecites [20] A. Turing, Computing Machinery and Intelligence (Mind LIX, 1950), pp. 433– 460. source
- Not checked — no full text of the sourcecites [14] C. Biever, ChatGPT broke the Turing test—The race is on for new ways to assess AI. Nature 619, 686– 689 (2023). 37491395 source
- Disputedcites [21] C. R. Jones, B. K. Bergen, People cannot distinguish GPT-4 from a human in a Turing test. arXiv [Preprint] (2024). 10.48550/arXiv.2405.08007 (Accessed 29 April 2025). sourceExperimental — accuracy not yet measured
Jones & Bergen (2024) do report that a suitably prompted GPT-4/GPT-4o passes a two-party Turing test (Introduction, contribution bullets: 50% pass rate in the main study, 77% in the replication vs a 71% human rate; §5.1 'On the basis of this definition, GPT-4 passes this version of the Turing test'). But their copy directly contradicts the 'consistently' qualifier: §5.1 states 'While GPT-4 may not consistently match human performance...', §4.2/Table 1 show pass rates ranging from 0.00% (Andrew) and 5.26% (Emily) through 25.00% (minimal prompt, 'significantly below chance' and indistinguishable from ELIZA) to 86.67% (qinn), and §5.3 concludes that 'careful prompting plays a crucial role', with the parity result flagged as unplanned and needing confirmation. The Introduction also notes that before their work there had been 'no controlled experimental demonstrations that any machine has passed the test'. (LLM, conf=0.70)
Checked by a reasoning model — may be wrong.
- Partly supportedcites [22] C. R. Jones, B. K. Bergen, Large language models pass the Turing test. arXiv [Preprint] (2025). https://arxiv.org/abs/2503.23674 (Accessed 29 April 2025). sourceExperimental — accuracy not yet measured
Jones & Bergen (2025) do report that LLMs pass the original three-party Turing test, replicated across two populations: Section 3.1 (p. 8) states 'both GPT-4.5 and LLaMa-3.1-405B pass the Turing test when they are given prompts to adopt a humanlike persona,' and GPT-4.5-PERSONA's win rate was significantly above chance in both the Undergraduate and Prolific studies (4.6 Statistics, p. 16). However, the paper does not support 'consistently': the same systems 'did not robustly pass without the PERSONA prompt' (3.1, p. 8), LLaMa-PERSONA was not above chance in the Undergraduate study (4.6, p. 16), and the NO-PERSONA witnesses were significantly below chance for undergraduates (2.1 Win Rates, p. 5; 4.6, p. 15). The result is also bounded to two evaluated models plus GPT-4o/ELIZA baselines, not LLMs in general (1.1, p. 2; 3.2, p. 9 calls for longer tests, other populations and expert interrogators as open questions). (LLM, conf=0.78)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Anthropomorphic Conversational Agents · boosted
Every source the paper cites for this claim was checked and supports it.
In the paper
“At the same time, advanced LLMs also possess deceptive potential that results from their convincing role-play abilities (27).”
§ The benefits and dangers of anthropomorphic conversational agents > Emerging Challenges: Anthropomorphic Seduction
What the sources say
- Supportedcites [27] M. Shanahan, K. McDonell, L. Reynolds, Role play with large language models. Nature 623, 493– 498 (2023). 37938776 source
The Role-playing deception section (p. 5) says a dialogue agent can role-play a deceptive character and provide false information in a way comparable to deliberate human deception. The Conclusion (p. 6) explains that such role-play can have real consequences, including a user being deceived into sending money. The paper distinguishes this potential from literal deceptive intent. (LLM, conf=0.92)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In the absence of concrete regulation, and as regulation in this space is still emerging, we would need to rely on the awareness and voluntary restraint of developers and system providers to dehumanize their systems, given how readily LLMs can be instructed to evoke and mimic humanness (61).”
§ The benefits and dangers of anthropomorphic conversational agents > Implications > Policy and Regulation.
What the sources say
- Supportedcites [61] B. Kleinberg, Trying to be human: Linguistic traces of stochastic empathy in language models. arXiv [Preprint] (2024). https://arxiv.org/abs/2410.01675 (Accessed 29 April 2025). source
In “LLMs pretending to be humans” (page not provided), the paper reports that instructions to appear human increased humanness judgments for LLM-generated text across three experiments. The “Stochastic empathy” section says a simple experimental manipulation invoked human-like writing in the LLM. (LLM, conf=0.92)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In addition to persuasive writing, LLMs are now capable of reliably inferring emotion and sentiment from human-written text. In one study, a fine-tuned LLM was able to detect emotions in social media posts with 84% accuracy (55). Another study found precision levels of over 95% for LLMs assessing sentiment across multiple languages, with the ability to identify nuanced feelings such as nostalgia”
§ The benefits and dangers of anthropomorphic conversational agents > Anthropomorphic Conversational Agents
What the sources say
- Not checked — no full text of the sourcecites [55] S. J. Lee, L. Paas, H. S. Ahn, The power of specific emotion analysis in predicting donations: A comparative empirical study between sentiment and specific emotion analysis in social media. Int. J. Mark. Res. 66, 610– 630 (2024). source
- Partly supportedcites [54] J. O. Krugmann, J. Hartmann, Sentiment analysis in the age of generative AI. Cust. Needs Solut. 11, 3 (2024). sourceExperimental — accuracy not yet measured
Section 3.3 reports strong zero-shot LLM performance in sentiment classification, while noting lower accuracy on Twitter datasets and cases where a specialized model performs better. The paper assesses positive, negative, and neutral sentiment; it does not establish that LLMs reliably identify specific emotions in human-written text. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Similarly, LLMs have been shown to outperform humans in reframing written scenarios to reduce negative emotion, a skill known in humans as cognitive reappraisal (56).”
§ The benefits and dangers of anthropomorphic conversational agents > Anthropomorphic Conversational Agents
What the sources say
- Supportedcites [56] J. Z. Li, A. Herderich, A. Goldenberg, Skill but not effort drive GPT overperformance over humans in cognitive reframing of negative scenarios. OSF Preprint (2024). https://osf.io/preprints/psyarxiv/fzvd8_v2. Accessed 29 April 2025. source
The Abstract reports that GPT-4 and humans reframed negative vignettes as a cognitive-reappraisal task, and that GPT-4 outperformed humans on three of four performance metrics. It describes the task as reframing negative situations to reduce negative emotions. (LLM, conf=0.94)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
53 more claims · 16 with their cited sources checked
Create a free account to see all 143 claims and to verify the 13 references still open.
- It may state two separate claims as one. 49
- Probably not a substantial claim. 51
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent reports suggest that interactions with large language model (LLM) chatbots can shape delusional content, amplify conviction, and entrench maladaptive safety behaviors [9,10].”
From Delusional Experiences Emerging From AI Chatbot Interactions or “AI Psychosis” (2025) · cites this paper as [10]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“simulated empathy can make persuasive outputs feel caring [23, 81, 83].”
From Caring Without Feeling: Affective Dynamics as the Control Layer of Human-AI Agent Collaboration (2026) · cites this paper as [81]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290
On seeing human: A three-factor theory of anthropomorphism
Epley, N., Waytz, A., & Cacioppo, J. T. (2007) · Psychological Review · doi:10.1037/0033-295x.114.4.864 · Cited by 4,069
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550
Building Machines That Learn and Think Like People
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2016) · arXiv · doi:10.48550/arxiv.1604.00289 · Cited by 2,550
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516
Age of Surveillance Capitalism – The Fight for a Human Future at the New Frontier of Power.
Kendell, D. (2020) · Brock Education Journal · doi:10.26522/brocked.v29i2.849 · Cited by 2,135
The echo chamber effect on social media
Cinelli, M., De Francisci Morales, G., Galeazzi, A., Quattrociocchi, W., & Starnini, M. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2023301118 · Cited by 2,092
Research works printed but not linked: 7
- T. B. Lee, S. Trott, A Jargon-Free Explanation of How AI Large Language Models Work (ARSTechnica, 2023).
- W. A. Sahlman A. M. Ciechanover E. Grandjean Khanmigo: Revolutionizing Learning with GenAI (HBS Case Collection) (Harvard Business School Case 2024) pp. 824–859.
- J. Manyika, K. Sneader, AI, Automation, and the Future of Work: Ten Things to Solve for (McKinsey & Company, 2018).
- S. Gibbons, T. Mugunthan, J. Nielsen, The 4 Degrees of Anthropomorphism of Generative AI (Nielsen Norman Group, 2023).
- J. Letzing, To Fully Appreciate AI Expectations, Look to the Trillions Being Invested (World Economic Forum, 2024).
- R. Mahari P. Pataranutaporn We need to prepare for ‘addictive intelligence’. MIT Technology Review 5 August 2024.
- D. Tuggener “Role-playing LLMs in professional communication training: The case of investigative interviews with children” in Proceedings of the 20th Conference on Natural Language Processing (KONVENS 2024) (Association for Computational Linguistics Vienna Austria 2024) pp. 249–263.
Cited by
Delusional Experiences Emerging From AI Chatbot Interactions or “AI Psychosis”
Hudon, A., & Stip, E. (2025) · JMIR Mental Health · doi:10.2196/85799 · Cited by 88
“Recent reports suggest that interactions with large language model (LLM) chatbots can shape delusional content, amplify conviction, and entrench maladaptive safety behaviors [9,10].”
Technological folie à deux: feedback loops between AI chatbots and mental health
Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2026) · Nature Mental Health · doi:10.1038/s44220-026-00595-8 · Cited by 51
“With regard to human factors, the potential relates to a capacity for anthropomorphism : the attribution of human-like qualities such as agency, intentionality, emotional states, and consciousness to non-human systems 3,5,27,39,40 .”
InfluentialFrom language to action: a review of large language models as autonomous agents and tool users
Chowa, S. S., Alvi, R., Rahman, S. S., Rahman, M. A., Raiaan, M. A. K., Islam, M. R., Hussain, M., & Azam, S. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-025-11471-9 · Cited by 50
“Their humanlike interactions create deception and encourage vulnerable populations, such as children or the elderly, to disclose sensitive information [215].”
Inventing with Machines: Generative AI and the Evolving Landscape of IS Research
Gopal, R. D., Li, J., Riemer, K., Sarker, S., Singh, P. V., Susarla, A., Bichler, M., & Thatcher, J. B. (2025) · Information Systems Research · doi:10.1287/isre.2025.editorial.v36.n4 · Cited by 55
Self-citationMetaphors of AI indicate that people increasingly perceive AI as warm and human-like
Cheng, M., Lee, A. Y., Rapuano, K. M., Niederhoffer, K. G., Liebscher, A., & Hancock, J. T. (2026) · Communications Psychology · doi:10.1038/s44271-025-00376-6 · Cited by 75
“However, this increase in trust may increase individuals ’ willingness to disclose sensitive personal information 114 and their susceptibility to persuasion or deception 57 .”
Metadata
- Authors
- Peter, S., Riemer, K., & West, J. D.
- Affiliations
- The University of Sydney (AU)
- University of Washington (US)
- Corresponding author
- Peter, Sandra
- Riemer, Kai
- West, Jevin D.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2025-05-16
- Volume
- 122
- Issue
- 22
- Pages
- e2415898122-e2415898122
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC12146756
- OpenAlex
- W4410431873
- Semantic Scholar
- af4eb2e344fe0564b975e58c5263c41342cea7f1
- Cited by
- 110
- References
- 104
- Citations by year
- 2026: 89 · 2025: 21
- FWCI
- 113.21
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (2120496)
- John S. and James L. Knight Foundation (UWCIP)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- large language models, anthropomorphism, AI deception, manipulation, disinformation, argumentative writing, empathic AI, human-AI communication, AI regulation, AI policy
- MeSH terms
- Artificial Intelligence, Communication, Humans, Language
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The benefits and dangers of anthropomorphic conversational agents. https://app.cochat.ai/papers/10.1073/pnas.2415898122
- The paper
- Peter, S., Riemer, K., & West, J. D. (2025). The benefits and dangers of anthropomorphic conversational agents. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2415898122
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.