Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions
Open accessThe DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
The integration of Large Language Models (LLMs) into the healthcare domain has the potential to significantly enhance patient care and support through the development of empathetic, patient-facing chatbots. This study investigates an intriguing question Can ChatGPT respond with a greater degree of empathy than those typically offered by physicians? To answer this question, we collect a de-identified dataset of patient messages and physician responses from a hospital and generate alternative replies using ChatGPT. We then introduce a set of empathy ranking evaluation (EMRank) metrics to automatically judge the empathy degree. We further conduct human study to gauge the empathy level of responses. Our findings indicate that LLM-powered chatbots have the potential to surpass human physicians in delivering empathetic communication, suggesting a promising avenue for enhancing patient care and reducing professional burnout. To summary, this study not only highlights the importance of clinical empathy in patient interactions but also proposes a set of automatic empathy ranking metrics, paving the way for the broader adoption of LLMs in healthcare.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A growing body of research suggests that the recent generation of large language model (LLMs) excel, and in many cases outpace humans, at writing persuasively and empathetically, at inferring user traits from text, and at mimicking human-like conversation believably and effectively—without possessing any true empathy or social understanding.”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [53]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“LLMs have demonstrated a remarkable capacity for zero-shot learning without the need for task-specific fine-tuning [4].”
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“[21] developed an automatic scoring mechanism based on the EPITOME framework [43], employing three RoBERTa models [28] to predict emotional reactions, interpretations, and explorations, with the final empathy score of a sentence being the average of these three scores.”
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“Large Language Models (LLMs) have exhibited exceptional performance in clinical and biomedical areas [24, 33].”
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · cited 1×
“[3] compare the empathy of physician’s response and a model’s response by a team of licensed health care professionals.”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 2×
“We introduce multiple metrics us-ing LLaMA [46], named LLaMA-EMRank.”
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · cited 1×
“A notable achievement, such as with models like ChatGPT [40], is the development of patient-facing chatbots built on the foundation of LLMs [12].”
Empathy Decline and Its Reasons: A Systematic Review of Studies With Medical Students and Residents
Neumann, M., Edelhäuser, F., Tauschel, D., Fischer, M. R., Wirtz, M., Woopen, C., Haramati, A., & Scheffer, C. (2011) · Academic Medicine · doi:10.1097/acm.0b013e318221e615 · Cited by 1,637 · cited 1×
“[36] high-light that effective empathy patient–physician communication positively impacts psychosocial outcomes such as quality of life, anxiety, and depression, and leads to measurable improvements in symptom reduction and control of blood pressure and glucose levels.”
Research works printed but not linked: 1
- Lee, Does gpt-3 generate empathetic dialogues? a novel in-context example selection method and automatic evaluation metric for empathetic dialogue generation, 669
Cited by
Large Language Models for Data Annotation and Synthesis: A Survey
Tan, Z., Li, D., Wang, S., Beigi, A., Jiang, B., Bhattacharjee, A., Karami, M., Li, J., Cheng, L., & Liu, H. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.54 · Cited by 178
The benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110
Large language models in critical care
Biesheuvel, L. A., Workum, J. D., Reuland, M., van Genderen, M. E., Thoral, P., Dongelmans, D., & Elbers, P. (2025) · Journal of Intensive Medicine · doi:10.1016/j.jointm.2024.12.001 · Cited by 36
A systematic review of early evidence on generative AI for drafting responses to patient messages
Hu, D., Guo, Y., Zhou, Y., Flores, L., & Zheng, K. (2025) · npj Health Systems · doi:10.1038/s44401-025-00032-5 · Cited by 40
“Capable of interpreting complex texts and generating human-like responses,thesemodelshavedemonstratedtheabilitytoanswermedicalquestions withexpert-levelknowledge 24,25 and to respond to patient forum posts in a more empathetic tone than physicians 26,27 .”
Geathers, J., Hicke, Y., Chan, C., Rajashekar, N., Young, S., Sewell, J. L., Cornes, S. B., Kizilcec, R. F., & Shung, D. L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98420-4_17 · Cited by 35
Workum, J. D., van de Sande, D., Gommers, D., & van Genderen, M. E. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1504805 · Cited by 25
Large language models for surgical informed consent: an ethical perspective on simulated empathy
Rudra, P., Balke, W., Kacprowski, T., Ursin, F., & Salloch, S. (2025) · Journal of Medical Ethics · doi:10.1136/jme-2024-110652 · Cited by 23
“45 However, patients generally prefer human responses even when satisfied with LLM-generated empathetic replies, indicating that genuine empathy is deeply valued in clinical contexts.”
AI-generated draft replies to patient messages: exploring effects of implementation
Bootsma-Robroeks, C. M., Workum, J. D., Schuit, S. C. E., Hoekman, A. H., Mehri, T., Doornberg, J. N., van der Laan, T. P., & Schoonbeek, R. C. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1588143 · Cited by 28
Zohny, H., Allen, J. W., Wilkinson, D. J. C., & Savulescu, J. (2025) · Journal of Medical Ethics · doi:10.1136/jme-2024-110256 · Cited by 21
Artificial intelligence in nephrology: predicting CKD progression and personalizing treatment
Yuan, S., Guo, L., & Xu, F. (2025) · International Urology and Nephrology · doi:10.1007/s11255-025-04878-4 · Cited by 20
“Beyond nephrology, LLMs demonstrate near‑human empathy in mental health counseling, with specialized methods (e.g., Empathetic Meta‑Chain) improving clinician–patient communication [248, 468–473].”
Sun, X., de Wit, J., Li, Z., Pei, J., Ali, A. E., & Bosch, J. A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757655 · Cited by 22
“Because SSAG operates at the level of therapeutic strategies rather than fixed language patterns, it is also adaptable to multilingual and culturally diverse contexts by accommodating different language norms and counseling styles.”
Busch, F., Kaibel, L., Nguyen, H., Lemke, T., Ziegelmayer, S., Graf, M., Marka, A. W., Endrös, L., Prucker, P., Spitzl, D., et al. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01483-w · Cited by 10
“For instance, a study by Luo et al. evaluating LLM responses to real-world physician–patient messages demonstrated that LLMs can exhibit greater empathy in communication compared to average physician responses [30].”
AlSammarraie, A., Al-Saifi, A., Kamhia, H., Aboagla, M., & Househ, M. (2025) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2025-101570 · Cited by 10
A Survey on the Recent Advancements in Human-Centered Dialog Systems
Oruche, R., Goruganthu, S. K., Akula, R., Cheng, X., Goni, M. A., Shibo, B. W., Kee, K. F., Zampieri, M., & Calyam, P. P. (2025) · ACM Computing Surveys · doi:10.1145/3729220 · Cited by 17
“These works address the vulnerability of LLMs in leaking conidential information from (i.e., prompt leaks) through various attack strategies and defense mechanisms.”
Zhou, Q., Hu, L., Yan, J., Cai, Y., & Zhang, Y. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100904 · Cited by 8
Empathy by Design: Aligning Large Language Models for Healthcare Dialogue
Umucu, E., Solis, G. R., Garza, L., Rivas, E., Lee, B., Kotal, A., & Piplai, A. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402624 · Cited by 5
The transformative potential of digital therapeutics in pediatrics
Mohabir, V., Lalloo, C., Briggs, A. M., Eccleston, C., Hess, C. W., Phạm, Q., Slater, H., Velmovitsky, P. E., & Stinson, J. (2025) · Pain · doi:10.1097/j.pain.0000000000003713 · Cited by 5
Grunebaum, A. N., & Chervenak, F. A. (2025) · Journal of Perinatal Medicine · doi:10.1515/jpm-2025-0367 · Cited by 3
Calero, R., & Herrera, G. (2025) · Journal of Advances in Artificial Intelligence · doi:10.18178/jaai.2025.3.4.287-296 · Cited by 2
Understanding use of Large Language Models in Healthcare: An HCI Scoping Review
Bandara, P. S., Halloluwa, T., & Vyas, D. (2024) · Australian Conference on Human-Computer Interaction (OzCHI) · doi:10.1145/3726986.3727023 · Cited by 2
“By improving diagnostic accuracy (e.g. [34]), providing timely and empathic patient interactions (e.g. [37]), and enhancing educational tools for medical professionals (e.g. [33]), LLMs are positioned to play a crucial role in the future of healthcare.”
Guo, Y., Hu, D., Zhou, Y., Lyu, T., Yang, Z., Reynolds, T. L., & Zheng, K. (2026) · npj Health Systems · doi:10.1038/s44401-026-00091-2 · Cited by 2
AI-ML based HealthCare Chatbot
Gupta, D. (2025) · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · doi:10.55041/ijsrem41352 · Cited by 1
Customizing ChatGPT for Effective Communication in Cardiac Care
Kelkar, A., Rajasekaran, Y., Ramesh, V., & Rogge, V. (2024) · International Conference on Ubiquitous Computing and Intelligent Information Systems (ICUIS) · doi:10.1109/icuis64676.2024.10866193 · Cited by 1
Sayed, F., Park, A., Sullivan, P. S., & Ge, Y. (2026) · JMIR Research Protocols · doi:10.2196/79966 · Cited by 2
“LLMs, leveraging vast knowledge bases, can generate responses that are varied and context-relevant compared to rule-based and hybrid systems [42].”
Patient-centered Evaluation of AI Answers to Genetic Counseling QuestionsPreprint
Huang, Y., Kambhamettu, H., Wood, E. M., Cacioppo, C. N., Ofidis, D., Mim, R., Bradbury, A. R., & Johnson, K. B. (2025) · medRxiv · doi:10.1101/2025.03.27.25323779
“A comparative assessment using de-identified patient-physician messages demonstrated that ChatGPT's responses scored higher on empathy metrics, as evaluated through both algorithmic and human methods 10 .”
Beyond Words: Gaze-Enhanced LLM-based Dialogue System for Therapeutic Purposes
Gabor-Siatkowska, K., Stefaniak, I., & Janicki, A. (2025) · International Conference on Systems, Signals and Image Processing · doi:10.1109/iwssip66997.2025.11151860
“When evaluated against human responses, responses by LLM were found to demonstrate more empathy [17] and helpfulness when responding to relationship or general health-related questions [18], [19].”
Appropriateness of Empathy in AI: A Signal-Cost Perspective
Wang, T., Chi-Ching, J., & Lee, H. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00102
“Frameworks such as the Multidimensional Evaluation Framework [2], LLaMA-EMRank [3], and ESHCC [4] have advanced operationalization across structural, behavioral, and perceptual dimensions.”
LunaAI: Healthcare Guidance Bot Being Polite and Fair
Ganesan, Y., Harlen, S., Rahman, A. R. B. F., Singh, A., Fathanah, Z., Yusof, R. J. R., & Kaptelinin, V. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-16448-3_28
'Can AI Care?': Emotional Tone Analysis and Perceived Empathy in AI-Generated Health Advice
Irfan, M., Kärkkäinen, T., & Correia, A. (2026) · International Congress on Human-Computer Interaction, Optimization and Robotic Applications (ICHORA) · doi:10.1109/ichora69329.2026.11537121
“In particular, systematic comparison of empathic versus neutral AI-generated messages evaluated simultaneously by computational tools and human participants remains underexplored [11, 12].”
InfluentialChokkalingam, S. P., & Ganesh, M. (2026) · International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV) · doi:10.1109/icicv68925.2026.11554638
“Yang et al. [3] suggested an IoT-based physiological feedback model, and Luo et al. [10] and George et al. [9] assessed the level of empathy in special counseling models and discovered that AI responses were associated with high levels of empathy.”
Promoting Behavior Change with Feedback Using LLM Based on User Activity Data
Mochizuki, S., & Takada, H. (2026) · International Conference on Activity and Behavior Computing (ABC) · doi:10.1109/abc68169.2026.11567167
“Another study [9] investigated whether ChatGPT could respond with greater empathy than physicians.”
Fan, G. F., Liu, D., Sabri, A. Q. M., Zhang, R., & Pan, L. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.462
“…perceived as empathic and can be guided toward specific support strategies such as cognitive reappraisal (Lee et al., 2024, 2023; Zhan et al., 2024; Luo et al., 2024; Chen et al., 2025), while appraisal-oriented and “emotional alignment” evaluations examine whether models match human affective…”
Rao, D., Liang, Z., & Jiang, Z. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.1790
“Subsequently, datasets that focus on empathy ranking (Luo et al., 2024) or the alignment of target and observer (Yang and Jurgens, 2024) are proposed.”
Zhang, E. H. Q., & Ive, J. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-24807-7_12
Bieberstein, A., Schnitzer, B. L., Gampe, S., & Korn, O. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816221 · Cited by 1
“Analyses of empathy assessment in LLMs further high-light the importance of contextual sensitivity and interpretation in determining whether empathic language translates into perceived support [29].”
InfluentialEbrahimi, P., Weisker, L. M., Schneider, J., & Pandey, H. M. (2026) · Journal of Technology in Behavioral Science · doi:10.1007/s41347-026-00687-3
A sentiment-aware chatbot for Arabic: enhancing engagement through emotionally adaptive responses
Aurag, H., Ouahabi, S. E., & Bouami, M. F. (2026) · Language Resources and Evaluation · doi:10.1007/s10579-026-09949-1
“These models leverage vast multilingual datasets and advanced training techniques to bridge the gap caused by limited resources.”
Zallipalli, S. N., Dendukuri, N. K., Asokan, G., Peri, H. K., & Kakarla, M. (2026) · Intelligent Hospital · doi:10.1016/j.inhs.2026.100115
Metrics and Learning in an Empathic Organisation
Mastrogiacomi, F. (2026) · Springer eBooks · doi:10.1007/978-3-032-35370-2_5
Waeiss, Q., Chang, L. J., Lutley, K., & Magnus, D. C. (2026) · doi:10.2196/preprints.114409
Metadata
- Authors
- Luo, M., Warren, C. J., Cheng, L., Abdul-Muhsin, H. M., & Banerjee, I.
- Affiliations
- Intel (United States) (US)
- Mayo Clinic (US)
- University of Illinois Chicago (US)
- Mayo Clinic,Department of Urology,Phoenix,U.S.
- Venue
- 2024 IEEE International Conference on Big Data (BigData)
- Publisher
- IEEE
- Published
- 2024-12-15
- Pages
- 6510-6519
- OpenAlex
- W4406458340
- Cited by
- 36
- References
- 52
- Citations by year
- 2026: 12 · 2025: 21 · 2024: 3
- FWCI
- 9.68
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Big Data (BigData) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Psychiatry and Mental health › Empathy and Medical Education
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- empathy measurement, large language models, ChatGPT, physician-patient communication, clinical empathy, physician responses, professional burnout
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.