CoChat

Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions

Open access
Luo, Man, Warren, Christopher J., Cheng, Lu, Abdul-Muhsin, Haidar M., Banerjee, Imon · 2024 · 2024 IEEE International Conference on Big Data (BigData)

The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.

Abstract

The integration of Large Language Models (LLMs) into the healthcare domain has the potential to significantly enhance patient care and support through the development of empathetic, patient-facing chatbots. This study investigates an intriguing question Can ChatGPT respond with a greater degree of empathy than those typically offered by physicians? To answer this question, we collect a de-identified dataset of patient messages and physician responses from a hospital and generate alternative replies using ChatGPT. We then introduce a set of empathy ranking evaluation (EMRank) metrics to automatically judge the empathy degree. We further conduct human study to gauge the empathy level of responses. Our findings indicate that LLM-powered chatbots have the potential to surpass human physicians in delivering empathetic communication, suggesting a promising avenue for enhancing patient care and reducing professional burnout. To summary, this study not only highlights the importance of clinical empathy in patient interactions but also proposes a set of automatic empathy ranking metrics, paving the way for the broader adoption of LLMs in healthcare.

Overview

Integrity signals

Registry

Crossref · 10.1109/bigdata62323.2024.10825307

Cites

52 of 52 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 52 references

Research works printed but not linked: 1

  1. Lee, Does gpt-3 generate empathetic dialogues? a novel in-context example selection method and automatic evaluation metric for empathetic dialogue generation, 669

Cited by

Metadata

Authors
Luo, M., Warren, C. J., Cheng, L., Abdul-Muhsin, H. M., & Banerjee, I.
Affiliations
  • Intel (United States) (US)
  • Mayo Clinic (US)
  • University of Illinois Chicago (US)
  • Mayo Clinic,Department of Urology,Phoenix,U.S.
Venue
2024 IEEE International Conference on Big Data (BigData)
Publisher
IEEE
Published
2024-12-15
Pages
6510-6519
OpenAlex
W4406458340
Cited by
36
References
52
Citations by year
2026: 12 · 2025: 21 · 2024: 3
FWCI
9.68
Type
conference paper
Language
en
Open access
Locations
  • IEEE International Conference on Big Data (BigData) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Psychiatry and Mental health › Empathy and Medical Education
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
Keywords
empathy measurement, large language models, ChatGPT, physician-patient communication, clinical empathy, physician responses, professional burnout

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.