Outpatient reception via collaboration between nurses and a large language model: a randomized controlled trial
The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Abstract
Reception is an essential process for patients seeking medical care and a critical component influencing the healthcare experience. However, current communication systems rely mainly on human efforts, which are both labor and knowledge intensive. A promising alternative is to leverage the capabilities of large language models (LLMs) to assist the communication in medical center reception sites. Here we curated a unique dataset comprising 35,418 cases of real-world conversation audio corpus between outpatients and receptionist nurses from 10 reception sites across two medical centers, to develop a site-specific prompt engineering chatbot (SSPEC). The SSPEC efficiently resolved patient queries, with a higher proportion of queries addressed in fewer rounds of queries and responses (Q&Rs; 68.0% ≤2 rounds) compared with nurse-led sessions (50.5% ≤2 rounds) (P = 0.009) across administrative, triaging and primary care concerns. We then established a nurse-SSPEC collaboration model, overseeing the uncertainties encountered during the real-world deployment. In a single-center randomized controlled trial involving 2,164 participants, the primary endpoint indicated that the nurse-SSPEC collaboration model received higher satisfaction feedback from patients (3.91 ± 0.90 versus 3.39 ± 1.15 in the nurse group, P < 0.001). Key secondary outcomes indicated reduced rate of repeated Q&R (3.2% versus 14.4% in the nurse group, P < 0.001) and reduced negative emotions during visits (2.4% versus 7.8% in the nurse group, P < 0.001) and enhanced response quality in terms of integrity (4.37 ± 0.95 versus 3.42 ± 1.22 in the nurse group, P < 0.001), empathy (4.14 ± 0.98 versus 3.27 ± 1.22 in the nurse group, P < 0.001) and readability (3.86 ± 0.95 versus 3.71 ± 1.07 in the nurse group, P = 0.006). Overall, our study supports the feasibility of integrating LLMs into the daily hospital workflow and introduces a paradigm for improving communication that benefits both patients and nurses. Chinese Clinical Trial Registry identifier: ChiCTR2300077245 .
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our system employs local language models to preserve data privacy, which comes at some cost to performance – the local model may not match that of commercial models58,59.”
From A large language model digital patient system enhances ophthalmology history taking skills (2025) · cites this paper as [58]
In the citing paper
“Our system employs local language models to preserve data privacy, which comes at some cost to performance – the local model may not match that of commercial models58,59.”
From A large language model digital patient system enhances ophthalmology history taking skills (2025) · cites this paper as [58]
In the citing paper
“in the largest randomised controlled trial identified in this review, published in Nature Medicine and involving 2,164 patients-to significantly enhance patient satisfaction when integrated into nurse-led communication workflows (Wan et al., 2024).”
From When the algorithm enters the classroom: A critical integrative review of large language models, nursing education structural gaps, and the reconstitution of professional identity (2026) · cites this paper as Wan et al., 2024
In the citing paper
“suggesting LLMs may function effectively as cognitive offloading tools for highly routinised communication tasks, potentially preserving nurses' attentional resources for higher-order clinical judgement (Wan et al., 2024).”
From When the algorithm enters the classroom: A critical integrative review of large language models, nursing education structural gaps, and the reconstitution of professional identity (2026) · cites this paper as Wan et al., 2024
In the citing paper
“Of the participating nurses, 95% reported reduced workload and 90% reported reduced occupational stress”
From When the algorithm enters the classroom: A critical integrative review of large language models, nursing education structural gaps, and the reconstitution of professional identity (2026) · cites this paper as Wan et al. (2024)
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · cited 1×
“Although close-to-human performance of LLMs has been achieved in answering United States Medical Licensing Examination (USMLE)-style questions and patient questions on a public forum 11,12 , several issues have constrained the implementation of LLMs 13 .”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“Although close-to-human performance of LLMs has been achieved in answering United States Medical Licensing Examination (USMLE)-style questions and patient questions on a public forum 11,12 , several issues have constrained the implementation of LLMs 13 .”
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · cited 12×
“Second, current LLMs suffer from fabricated facts and other errors, inhibiting the confidence of their application in healthcare conditions.”
InfluentialRajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212 · cited 1×
“In one aspect, because of the high sensitivity of the alert system, that small proportion of uncertain responses could Article https://doi.org/10.1038/s41591-024-03148-7 be alerted and amended by nurses 23 , whereas most patient queries could be addressed by the SSPEC without involving nurses.”
InfluentialOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · cited 1×
“LLMs are trained on datasets consisting of literature, natural language tasks and online healthcare contexts 16 , which underrepresent real-world chaos.”
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · cited 1×
“Although close-to-human performance of LLMs has been achieved in answering United States Medical Licensing Examination (USMLE)-style questions and patient questions on a public forum 11,12 , several issues have constrained the implementation of LLMs 13 .”
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · cited 1×
“First, existing LLM models are predominantly trained on knowledge from online materials and scientific literature 14 , resulting in a lack of context-specific knowledge that is necessary for addressing queries in real-world scenarios 15 .”
Kwame, A., & Petrucka, P. M. (2021) · BMC Nursing · doi:10.1186/s12912-021-00684-2 · Cited by 1,067 · cited 1×
“Their communication skills profoundly influence patient satisfaction, medical adherence and, ultimately, health outcomes 4,5 .”
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847 · cited 1×
“(4) We further applied a de-identification to remove protected health information (PHI) 29,30 from text manually.”
Active Retrieval Augmented Generation
Jiang, Z., Xu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.495 · Cited by 497
Wang, J., Wu, X., Lai, W., Long, E., Zhang, X., Li, W., Zhu, Y., Chen, C., Zhong, X., Liu, Z., et al. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-017173 · Cited by 486 · cited 2×
“Studies indicate a prevalence of 27% depressive symptoms among outpatients, substantially higher than that in healthy individuals 6 , highlighting the importance of empathetic support during outpatient visits.”
InfluentialSelf-citationSharkiya, S. H. (2023) · BMC Health Services Research · doi:10.1186/s12913-023-09869-8 · Cited by 416 · cited 1×
“Their communication skills profoundly influence patient satisfaction, medical adherence and, ultimately, health outcomes 4,5 .”
Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey
Agrawal, G., Kumarage, T., Alghamdi, Z., & Liu, H. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.219 · Cited by 335 · cited 1×
“First, existing LLM models are predominantly trained on knowledge from online materials and scientific literature 14 , resulting in a lack of context-specific knowledge that is necessary for addressing queries in real-world scenarios 15 .”
Burnout and Workload Among Health Care Workers: The Moderating Role of Job Control
Portoghese, I., Galletta, M., Coppola, R. C., Finco, G., & Campagna, M. (2014) · Safety and Health at Work · doi:10.1016/j.shaw.2014.05.004 · Cited by 330 · cited 1×
“Consequently, communication may become inefficient, disagreements may arise and negative emotions may be triggered 9 .”
Artificial intelligence technology, public trust, and effective governance
Robles, P., & Mallinson, D. J. (2023) · Review of Policy Research · doi:10.1111/ropr.12555 · Cited by 235 · cited 1×
“This relatively high satisfaction could be attributed to the supervision of receptionist nurses and the substantial recognition and public trust of AI technology 24 .”
A Comparison Of Hospital Administrative Costs In Eight Nations: US Costs Exceed All Others By Far
Himmelstein, D. U., Jun, M., Busse, R., Chevreul, K., Geissler, A., Jeurissen, P., Thomson, S., Vinet, M.-A., & Woolhandler, S. (2014) · Health Affairs · doi:10.1377/hlthaff.2013.1327 · Cited by 218 · cited 1×
“Administrative costs in Western countries are notably high, accounting for 11.6–25.3% of total hospital expenses 1 .”
InfluentialCompound facial expressions of emotion: from basic research to clinical applications
Du, S., & Martinez, A. M. (2015) · Dialogues in Clinical Neuroscience · doi:10.31887/dcns.2015.17.4/sdu · Cited by 93 · cited 2×
“Specifically, our assessment of compromised experiences within the internal validation focused solely on textual content, without taking into account other metrics, such as patients’ facial expressions or tone of speech 21,22 .”
Fang, H., Zhao, X., Yang, H., Sun, P., Li, Y., Jiang, K., Li, P., Jiao, M., Liu, M., Qiao, H. Y., et al. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019514 · Cited by 76 · cited 1×
“However, nurses frequently find themselves over-whelmed, resulting in burnout and depression rates ranging from 34.2% to 57.2% 7,8 .”
InfluentialOn Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.244 · Cited by 76
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2022) · arXiv · doi:10.48550/arxiv.2212.08061 · Cited by 76
A study of deep learning methods for de-identification of clinical notes in cross-institute settings
Yang, X., Lyu, T., Li, Q., Lee, C.-Y., Bian, J., Hogan, W. R., & Wu, Y. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0935-4 · Cited by 73 · cited 2×
“(4) We further applied a de-identification to remove protected health information (PHI) 29,30 from text manually.”
Ito, N., Kadomatsu, S., Fujisawa, M., Fukaguchi, K., Ishizawa, R., Kanda, N., Kasugai, D., Nakajima, M., Goto, T., & Tsugawa, Y. (2023) · JMIR Medical Education · doi:10.2196/47532 · Cited by 63 · cited 1×
“Moreover, considerations of fairness and equity are critical in healthcare 25 , where ensuring equal and unbiased access for all patients, regardless of their background, is a fundamental ethical concern 26,27 .”
Understanding context specificity: the effect of contextual factors on clinical reasoning
Konopasky, A., Artino, A. R., Battista, A., Ohmer, M., Hemmer, P. A., Torre, D., Ramani, D., van Merrienboer, J., Teunissen, P. W., McBee, E., et al. (2020) · Diagnosis · doi:10.1515/dx-2020-0016 · Cited by 58 · cited 1×
“Second, our results underscore the value of site-specific knowledge, as the nature of queries can vary greatly depending on the surrounding environment 17,18 .”
The care work of general practice receptionists
Neuwelt, P. M., KEARNS, R. A., & Cairns, I. (2016) · Journal of Primary Health Care · doi:10.1071/hc15059 · Cited by 42
Gao, K., He, S., He, Z., Lin, J., Pei, Q., Shao, J., & Zhang, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.14149 · Cited by 32 · cited 1×
“0) 33,34 on a scale of 1 to 10.”
Yuan, J., Bao, P., Chen, Z., Yuan, M., Zhao, J., Pan, J., Xie, Y., Cao, Y., Wang, Y., Wang, Z., et al. (2023) · The Innovation Medicine · doi:10.59717/j.xinn-med.2023.100019 · Cited by 27
Centering health equity in large language model deployment
Singh, N., Lawrence, K., Richardson, S., & Mann, D. M. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000367 · Cited by 25
A Study of Deep Learning Methods for De-identification of Clinical Notes at Cross Institute Settings
Yang, X., Lyu, T., Lee, C.-Y., Bian, J., Hogan, W. R., & Wu, Y. (2019) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi.2019.8904544 · Cited by 21
He, A. J., & Qian, J. (2013) · The China Quarterly · doi:10.1017/s0305741013001112 · Cited by 19 · cited 1×
“Hence, a promising alternative is to leverage the capability of large language models (LLMs) to assist nurses, allowing them to focus on more complex tasks and reducing the administrative burden in the healthcare system 10 .”
Active Retrieval Augmented Generation
Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023) · arXiv · doi:10.48550/arxiv.2305.06983 · Cited by 18 · cited 1×
“Automatic Evaluation: The pre-trained RAG model 35 was used to assess faithfulness and response relevance 36 , minimizing the risk of hallucinations on a scale of 0 to 1.”
Interpreting tone of voice: Musical pitch relationships convey agreement in dyadic conversation
Okada, B. M., Lachs, L., & Boone, B. (2012) · The Journal of the Acoustical Society of America · doi:10.1121/1.4742316 · Cited by 17 · cited 2×
“Specifically, our assessment of compromised experiences within the internal validation focused solely on textual content, without taking into account other metrics, such as patients’ facial expressions or tone of speech 21,22 .”
Improvement in Medical Consultation Responses with a Structured Request Form
Geist, S.-M. R. Y., & Geist, J. R. (2008) · Journal of Dental Education · doi:10.1002/j.0022-0337.2008.72.5.tb04519.x · Cited by 15 · cited 2×
“For each site, the model was fine-tuned based on the unprocessed dialogue, including those general to all sites or centers or specific to each site, and structured in a ‘step-by-step’ manner to address logical questions (examples in Supplementary Table 20).”
The Limits of Language-Thought Influences Can Be Set by the Constraints of Embodiment
Mondal, P. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.593137 · Cited by 6 · cited 1×
“Second, our results underscore the value of site-specific knowledge, as the nature of queries can vary greatly depending on the surrounding environment 17,18 .”
Survey on Mental Health Status of the Medical Staff
Jin, K. (2007) · Nursing Journal of Chinese People's Liberation Army · Cited by 3
Research works printed but not linked: 6
- Guo, S., Yang, T. & Dong, S. Research advances on cost-efficiency measurement and evaluation of public hospitals in China. Chin. J. Health Policy 13, 45–51 (2020).
- Zeng, D. The comparison and implication on the cost management of public hospitals between China and the US. Chin. J. Health Policy 12, 13–17 (2012).
- Loh, S. B. & Raamkumar, A. S. Harnessing large language models’ empathetic response generation capabilities for online mental health counselling support. Preprint at https://arxiv.org/abs/2310.08017 (2023). · arXiv
- Martin-Maroto, F. & de Polavieja, G. G. Semantic embeddings in semilattices. Preprint at https://arxiv.org/abs/2205.12618 (2022). · arXiv
- Mao, R., Chen, G., Zhang, X., Guerin, F. & Cambria, E. GPTEval: a survey on assessments of ChatGPT and GPT-4. In Proc. 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (eds Calzolari, N. et al.) 7844–7866 (ELRA and ICCL, 2024).
- Es, S., James, J., Espinosa-Anke, L. & Schockaert, S. RAGAS: Automated Evaluation of Retrieval Augmented Generation. In Proc. 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (eds Aletras, N. & De Clercq, O.) 150–158 (Association for Computational Linguistics, 2024).
Cited by
Artsi, Y., Sorin, V., Glicksberg, B. S., Korfiatis, P. D., Nadkarni, G. N., & Klang, E. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1659134 · Cited by 152
“Wan et al. (13) randomized 2,164 outpatients across two medical centers to nurse-only vs. nurse–SSPEC workflows.”
Howcroft, A., Bennett‐Weston, A., Khan, A. H., Griffiths, J., Gay, S. P., & Howick, J. (2025) · British Medical Bulletin · doi:10.1093/bmb/ldaf017 · Cited by 133
“The ROBINS-I tool [19] was used to assess the RoB across the 15 studies, given that 14 were non-randomized cross-sectional comparative analyses and one was a randomized controlled trial (RCT) [20].”
InfluentialEvaluating large language models and agents in healthcare: key challenges in clinical applications
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.03.002 · Cited by 68
The Roles of Artificial Intelligence in Teaching Anatomy: A Systematic Review
Joseph, T. S., Gowrie, S., Montalbano, M. J., Bandelow, S., Clunes, M. T., Dumont, A. S., Iwanaga, J., Tubbs, R. S., & Loukas, M. (2025) · Clinical Anatomy · doi:10.1002/ca.24272 · Cited by 73
“Wan et al. (2024) highlighted how a hybrid model combining AI and human over-sight in outpatient care improved communication outcomes, empathy, and efficiency without diminishing human agency.”
Collaborative, T. C. (2025) · BMJ · doi:10.1136/bmj-2024-083305 · Cited by 73
“Future expansions may address other types of generative AI applications in healthcare, 82 or hybrid approaches combining generative AI with other AI technologies.”
Metadata
- Authors
- Wan, P., Huang, Z., Tang, W., Nie, Y., Pei, D., Deng, S., Chen, J., Zhou, Y., Duan, H., Chen, Q., & Long, E.
- Affiliations
- National Institutes of Health (US)
- Chinese Academy of Medical Sciences & Peking Union Medical College (CN)
- Southern University of Science and Technology (CN)
- Renmin Hospital of Wuhan University (CN)
- Yale University (US)
- Corresponding author
- Chen, Qingyu
- Long, Erping
- Venue
- Nature Medicine
- Published
- 2024-07-15
- Volume
- 30
- Issue
- 10
- Pages
- 2878-2885
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- OpenAlex
- W4400657362
- Cited by
- 97
- References
- 31
- Citations by year
- 2026: 44 · 2025: 47 · 2024: 6
- FWCI
- 9.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- large language models, hospital reception, chatbots, prompt engineering, patient reception, patient satisfaction, nurse-patient communication, triage, response quality, empathy, negative emotions, hospital workflow
- MeSH terms
- Adult, Communication, Cooperative Behavior, Female, Humans, Language, Male, Middle Aged, Nurses, Outpatients, Patient Satisfaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.