Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.

Outpatient reception via collaboration between nurses and a large language model: a randomized controlled trial

Abstract

Reception is an essential process for patients seeking medical care and a critical component influencing the healthcare experience. However, current communication systems rely mainly on human efforts, which are both labor and knowledge intensive. A promising alternative is to leverage the capabilities of large language models (LLMs) to assist the communication in medical center reception sites. Here we curated a unique dataset comprising 35,418 cases of real-world conversation audio corpus between outpatients and receptionist nurses from 10 reception sites across two medical centers, to develop a site-specific prompt engineering chatbot (SSPEC). The SSPEC efficiently resolved patient queries, with a higher proportion of queries addressed in fewer rounds of queries and responses (Q&Rs; 68.0% ≤2 rounds) compared with nurse-led sessions (50.5% ≤2 rounds) (P = 0.009) across administrative, triaging and primary care concerns. We then established a nurse-SSPEC collaboration model, overseeing the uncertainties encountered during the real-world deployment. In a single-center randomized controlled trial involving 2,164 participants, the primary endpoint indicated that the nurse-SSPEC collaboration model received higher satisfaction feedback from patients (3.91 ± 0.90 versus 3.39 ± 1.15 in the nurse group, P < 0.001). Key secondary outcomes indicated reduced rate of repeated Q&R (3.2% versus 14.4% in the nurse group, P < 0.001) and reduced negative emotions during visits (2.4% versus 7.8% in the nurse group, P < 0.001) and enhanced response quality in terms of integrity (4.37 ± 0.95 versus 3.42 ± 1.22 in the nurse group, P < 0.001), empathy (4.14 ± 0.98 versus 3.27 ± 1.22 in the nurse group, P < 0.001) and readability (3.86 ± 0.95 versus 3.71 ± 1.07 in the nurse group, P = 0.006). Overall, our study supports the feasibility of integrating LLMs into the daily hospital workflow and introduces a paradigm for improving communication that benefits both patients and nurses. Chinese Clinical Trial Registry identifier: ChiCTR2300077245 .

Overview

Integrity signals

Registry

Crossref · 10.1038/s41591-024-03148-7

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 31 references

Research works printed but not linked: 6

  1. Guo, S., Yang, T. & Dong, S. Research advances on cost-efficiency measurement and evaluation of public hospitals in China. Chin. J. Health Policy 13, 45–51 (2020).
  2. Zeng, D. The comparison and implication on the cost management of public hospitals between China and the US. Chin. J. Health Policy 12, 13–17 (2012).
  3. Loh, S. B. & Raamkumar, A. S. Harnessing large language models’ empathetic response generation capabilities for online mental health counselling support. Preprint at https://arxiv.org/abs/2310.08017 (2023). · arXiv
  4. Martin-Maroto, F. & de Polavieja, G. G. Semantic embeddings in semilattices. Preprint at https://arxiv.org/abs/2205.12618 (2022). · arXiv
  5. Mao, R., Chen, G., Zhang, X., Guerin, F. & Cambria, E. GPTEval: a survey on assessments of ChatGPT and GPT-4. In Proc. 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (eds Calzolari, N. et al.) 7844–7866 (ELRA and ICCL, 2024).
  6. Es, S., James, J., Espinosa-Anke, L. & Schockaert, S. RAGAS: Automated Evaluation of Retrieval Augmented Generation. In Proc. 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (eds Aletras, N. & De Clercq, O.) 150–158 (Association for Computational Linguistics, 2024).

Cited by

See all 105

Metadata

Authors
Wan, P., Huang, Z., Tang, W., Nie, Y., Pei, D., Deng, S., Chen, J., Zhou, Y., Duan, H., Chen, Q., & Long, E.
Affiliations
  • National Institutes of Health (US)
  • Chinese Academy of Medical Sciences & Peking Union Medical College (CN)
  • Southern University of Science and Technology (CN)
  • Renmin Hospital of Wuhan University (CN)
  • Yale University (US)
Corresponding author
  • Chen, Qingyu
  • Long, Erping
Venue
Nature Medicine
Published
2024-07-15
Volume
30
Issue
10
Pages
2878-2885
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
OpenAlex
W4400657362
Cited by
97
References
31
Citations by year
2026: 44 · 2025: 47 · 2024: 6
FWCI
9.07
Type
journal article
Language
en
Open access
closed
Licence
Locations
  • Nature Medicine (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
large language models, hospital reception, chatbots, prompt engineering, patient reception, patient satisfaction, nurse-patient communication, triage, response quality, empathy, negative emotions, hospital workflow
MeSH terms
Adult, Communication, Cooperative Behavior, Female, Humans, Language, Male, Middle Aged, Nurses, Outpatients, Patient Satisfaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Outpatient reception via collaboration between nurses and a large language model: a randomized controlled trial. https://app.cochat.ai/papers/10.1038/s41591-024-03148-7
The paper
Wan, P., Huang, Z., Tang, W., Nie, Y., Pei, D., Deng, S., Chen, J., Zhou, Y., Duan, H., Chen, Q., & Long, E. (2024). Outpatient reception via collaboration between nurses and a large language model: a randomized controlled trial. Nature Medicine. https://doi.org/10.1038/s41591-024-03148-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.