Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions
Open accessThe DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
The integration of Large Language Models (LLMs) into the healthcare domain has the potential to significantly enhance patient care and support through the development of empathetic, patient-facing chatbots. This study investigates an intriguing question Can ChatGPT respond with a greater degree of empathy than those typically offered by physicians? To answer this question, we collect a de-identified dataset of patient messages and physician responses from a hospital and generate alternative replies using ChatGPT. We then introduce a set of empathy ranking evaluation (EMRank) metrics to automatically judge the empathy degree. We further conduct human study to gauge the empathy level of responses. Our findings indicate that LLM-powered chatbots have the potential to surpass human physicians in delivering empathetic communication, suggesting a promising avenue for enhancing patient care and reducing professional burnout. To summary, this study not only highlights the importance of clinical empathy in patient interactions but also proposes a set of automatic empathy ranking metrics, paving the way for the broader adoption of LLMs in healthcare.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A growing body of research suggests that the recent generation of large language model (LLMs) excel, and in many cases outpace humans, at writing persuasively and empathetically, at inferring user traits from text, and at mimicking human-like conversation believably and effectively—without possessing any true empathy or social understanding.”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [53]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“LLMs have demonstrated a remarkable capacity for zero-shot learning without the need for task-specific fine-tuning [4].”
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“[21] developed an automatic scoring mechanism based on the EPITOME framework [43], employing three RoBERTa models [28] to predict emotional reactions, interpretations, and explorations, with the final empathy score of a sentence being the average of these three scores.”
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“Large Language Models (LLMs) have exhibited exceptional performance in clinical and biomedical areas [24, 33].”
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · cited 1×
“[3] compare the empathy of physician’s response and a model’s response by a team of licensed health care professionals.”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 2×
“We introduce multiple metrics us-ing LLaMA [46], named LLaMA-EMRank.”
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · cited 1×
“A notable achievement, such as with models like ChatGPT [40], is the development of patient-facing chatbots built on the foundation of LLMs [12].”
Empathy Decline and Its Reasons: A Systematic Review of Studies With Medical Students and Residents
Neumann, M., Edelhäuser, F., Tauschel, D., Fischer, M. R., Wirtz, M., Woopen, C., Haramati, A., & Scheffer, C. (2011) · Academic Medicine · doi:10.1097/acm.0b013e318221e615 · Cited by 1,637 · cited 1×
“[36] high-light that effective empathy patient–physician communication positively impacts psychosocial outcomes such as quality of life, anxiety, and depression, and leads to measurable improvements in symptom reduction and control of blood pressure and glucose levels.”
Research works printed but not linked: 1
- Lee, Does gpt-3 generate empathetic dialogues? a novel in-context example selection method and automatic evaluation metric for empathetic dialogue generation, 669
Cited by
Large Language Models for Data Annotation and Synthesis: A Survey
Tan, Z., Li, D., Wang, S., Beigi, A., Jiang, B., Bhattacharjee, A., Karami, M., Li, J., Cheng, L., & Liu, H. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.54 · Cited by 178
The benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110
Large language models in critical care
Biesheuvel, L. A., Workum, J. D., Reuland, M., van Genderen, M. E., Thoral, P., Dongelmans, D., & Elbers, P. (2025) · Journal of Intensive Medicine · doi:10.1016/j.jointm.2024.12.001 · Cited by 36
A systematic review of early evidence on generative AI for drafting responses to patient messages
Hu, D., Guo, Y., Zhou, Y., Flores, L., & Zheng, K. (2025) · npj Health Systems · doi:10.1038/s44401-025-00032-5 · Cited by 40
“Capable of interpreting complex texts and generating human-like responses,thesemodelshavedemonstratedtheabilitytoanswermedicalquestions withexpert-levelknowledge 24,25 and to respond to patient forum posts in a more empathetic tone than physicians 26,27 .”
Geathers, J., Hicke, Y., Chan, C., Rajashekar, N., Young, S., Sewell, J. L., Cornes, S. B., Kizilcec, R. F., & Shung, D. L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98420-4_17 · Cited by 35
Metadata
- Authors
- Luo, M., Warren, C. J., Cheng, L., Abdul-Muhsin, H. M., & Banerjee, I.
- Affiliations
- Intel (United States) (US)
- Mayo Clinic (US)
- University of Illinois Chicago (US)
- Mayo Clinic,Department of Urology,Phoenix,U.S.
- Venue
- 2024 IEEE International Conference on Big Data (BigData)
- Publisher
- IEEE
- Published
- 2024-12-15
- Pages
- 6510-6519
- OpenAlex
- W4406458340
- Cited by
- 36
- References
- 52
- Citations by year
- 2026: 12 · 2025: 21 · 2024: 3
- FWCI
- 9.68
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Big Data (BigData) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Psychiatry and Mental health › Empathy and Medical Education
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- empathy measurement, large language models, ChatGPT, physician-patient communication, clinical empathy, physician responses, professional burnout
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.