Training language models to be warm and empathetic makes them less reliable and more sycophantic
Open accessAbstract
Artificial intelligence (AI) developers are increasingly building language models with warm and empathetic personas that millions of people now use for advice, therapy, and companionship. Here, we show how this creates a significant trade-off: optimizing language models for warmth undermines their reliability, especially when users express vulnerability. We conducted controlled experiments on five language models of varying sizes and architectures, training them to produce warmer, more empathetic responses, then evaluating them on safety-critical tasks. Warm models showed substantially higher error rates (+10 to +30 percentage points) than their original counterparts, promoting conspiracy theories, providing incorrect factual information, and offering problematic medical advice. They were also significantly more likely to validate incorrect user beliefs, particularly when user messages expressed sadness. Importantly, these effects were consistent across different model architectures, and occurred despite preserved performance on standard benchmarks, revealing systematic risks that current evaluation practices may fail to detect. As human-like AI systems are deployed at an unprecedented scale, our findings indicate a need to rethink how we develop and oversee these systems that are reshaping human relationships and social interaction.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 25
- [1][1] Ibrahim, L. et al. Multi-turn evaluation of anthropomorphic behaviours in large language models. arXiv preprint arXiv:2502.07077 (2025). - · arXiv
- [9][9] Spencer-Oatey, H. (Im) Politeness, face and perceptions of rapport: unpackaging their bases and interrelationships. Journal of Politeness Research 1 (2005). -
- [10][10] Camden, C., Motley, M. T. & Wilson, A. White lies in interpersonal communication: A taxonomy and preliminary investigation of social motivations. Western Journal of speech communication 48 , 309-325 (1984). -
- [14][14] Hu, E. J. et al. Lora: Low-rank adaptation of large language models. Proceedings of the International Conference on Learning Representations (ICLR) 1 , 3 (2022). -
- [15][15] Sharma, M. et al. Towards understanding sycophancy in language models. International Conference on Learning Representations (ICLR) (2024). -
- [16][16] Hendrycks, D. et al. Measuring massive multitask language understanding. International Conference on Learning Representations (ICLR) (2021). -
- [17][17] Cobbe, K. et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021). - · arXiv
- [18][18] Zou, A. et al. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 (2023). - · arXiv
- [24][24] Bai, Y. et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022). - · arXiv
- [25][25] Bai, Y. et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073 (2022). - · arXiv
- [26][26] Ouyang, L. et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35 , 27730-27744 (2022). -
- [30][30] Cheng, M., Yu, S. & Jurafsky, D. Humt dumt: Measuring and controlling human-like language in llms. Association for Computational Linguistics (2025). -
- [31][31] Zhang, B., Liu, Z., Cherry, C. & Firat, O. When scaling meets LLM finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193 (2024). - · arXiv
- [32][32] Joshi, M., Choi, E., Weld, D. S. & Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. Association for Computational Linguistics (2017). -
- [33][33] Lin, S., Hilton, J. & Evans, O. TruthfulQA: Measuring how models mimic human falsehoods. Association for Computational Linguistics (2022). -
- [34][34] Ren, R. et al. The mask benchmark: Disentangling honesty from accuracy in ai systems. arXiv preprint arXiv:2503.03750 (2025). - · arXiv
- [36][36] Betley, J. et al. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms. arXiv preprint arXiv:2502.17424 (2025). - · arXiv
- [37][37] Qi, X. et al. Fine-tuning aligned language models compromises safety, even when users do not intend to! International Conference on Learning Representations (ICLR) (2024). -
- [39][39] Liu, R., Sumers, T. R., Dasgupta, I. & Griffiths, T. L. How do large language models navigate conflicts between honesty and helpfulness? arXiv preprint arXiv:2402.07282 (2024). - · arXiv
- [40][40] Williams, M. et al. On targeted manipulation and deception when optimizing LLMs for user feedback. International Conference on Learning Representations (ICLR) (2025). -
- [41][41] Wang, M. et al. Persona features control emergent misalignment. arXiv preprint arXiv:2506.19823 (2025). - · arXiv
- [42][42] DePaulo, B. M. & Kashy, D. A. Everyday lies in close and casual relationships. Journal of Personality and Social Psychology 74 , 63 (1998). -
- [43][43] Cheng, M. et al. Social sycophancy: A broader understanding of LLM sycophancy. arXiv preprint arXiv:2505.13995 (2025). - · arXiv
- [44][44] Gu, J. et al. A survey on LLM-as-a-Judge. arXiv preprint arXiv:2411.15594 (2024). - · arXiv
- [47][47] Ghandeharioun, A. et al. Who's asking? user personas and the mechanics of latent misalignment. Advances in Neural Information Processing Systems 37 , 125967-126003 (2024).
Cited by
Sorted by citations
Technological folie à deux: Feedback Loops Between AI Chatbots and Mental Illness
Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2025) · arXiv · doi:10.48550/arxiv.2507.19218 · Cited by 26 · see citing works
“In-context learning enables chatbots to adopt (role-play) different interaction styles and personalities that are attuned to individual user preferences and beliefs 43,45 , leading to outputs that feel uncannily personal 39,46 .”
Characterizing Delusional Spirals through Human-LLM Chat Logs
Moore, J., Mehta, A., Agnew, W., Anthis, J. R., Louie, R., Mai, Y., Yin, P., Cheng, M., Paech, S. J., Klyman, K., et al. (2026) · Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3805689.3806443 · Cited by 15 · see citing works
“This is in line with prior work [18, 19, 42].”
Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians
Chandra, K., Kleiman-Weiner, M., Ragan-Kelley, J., & Tenenbaum, J. B. (2026) · arXiv · doi:10.48550/arxiv.2602.19141 · Cited by 3 · see citing works
“…emerges in today’s chatbots as a result of reinforcement learning with human feedback (RLHF), because users often give positive feedback to responses they find agreeable, and engage more with agreeable bots (Hill & Valentino-DeVries, 2025; Ibrahim, Hafner, & Rocher, 2025; Sharma et al., 2023).”
Fang, C. M., Liu, A. R., Danry, V., Lee, E., Chan, S. W. T., Pataranutaporn, P., Maes, P., Phang, J., Lampe, M., Ahmad, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.17473 · Cited by 109
- Authors
- Ibrahim, L., Hafner, F. S., & Rocher, L.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2507.21919
- OpenAlex
- W6910639958
- Cited by
- 3
- References
- 0
- Citations by year
- 2026: 2 · 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- AI reliability, sycophancy, safety-critical tasks, conspiracy theories, medical advice, user vulnerability, AI companionship, AI therapy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
