Detecting emergencies in patient portal messages using large language models and knowledge graph-based retrieval-augmented generation
Open accessAbstract
Objectives: This study aims to develop and evaluate an approach using large language models (LLMs) and a knowledge graph to triage patient messages that need emergency care. The goal is to notify patients when their messages indicate an emergency, guiding them to seek immediate help rather than using the patient portal, to improve patient safety. Materials and Methods: We selected 1020 messages sent to Vanderbilt University Medical Center providers between January 1, 2022 and March 7, 2023. We developed four models to triage these messages for emergencies: (1) Prompt-Only: the patient message was input with a prompt directly into the LLM; (2) Naïve Retrieval Augmented Generation (RAG): provided retrieved information as context to the LLM; (3) RAG from Knowledge Graph with Local Search: a knowledge graph was used to retrieve locally relevant information based on semantic similarities; (4) RAG from Knowledge Graph with Global Search: a knowledge graph was used to retrieve globally relevant information through hierarchical community detection. The knowledge base was a triage book covering 225 protocols. Results: The RAG from Knowledge Graph model with global search outperformed other models, achieving an accuracy of 0.99, a sensitivity of 0.98, and a specificity of 0.99. It demonstrated significant improvements in triaging emergency messages compared to LLM without RAG and naïve RAG. Discussion: The traditional LLM without any retrieval mechanism underperformed compared to models with RAG, which aligns with the expected benefits of augmenting LLMs with domain-specific knowledge sources. Our results suggest that providing external knowledge, especially in a structured manner and in community summaries, can improve LLM performance in triaging patient portal messages. Conclusion: LLMs can effectively assist in triaging emergency patient messages after integrating with a knowledge graph about a nurse triage book. Future research should focus on expanding the knowledge graph and deploying the system to evaluate its impact on patient outcomes.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
[18] demonstrated accuracy 0.99, sensitivity 0.98 and specificity 0.99, which significantly exceed clinician-coded rules and classifiers in patient-portal triage.
“[18] Demonstrated excellent performance accuracy 0.99, sensitivity 0.98 and specificity 0.99 which significantly exceed clinician-coded rules and classifiers found in patient-portal triage [33][34].”
From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [18]
Table 2 in the Results reports accuracy 0.99, sensitivity 0.98, and specificity 0.99 for the knowledge-graph RAG model with global search. The Results report statistically significant differences among the four models tested, but those comparisons do not include clinician-coded rules or classifiers; the claimed superiority over those approaches is not established by this paper. (LLM, conf=0.89)
Experimental — accuracy not yet measuredKG-RAG systems enhance trust by centering triage decisions in biomedical ontologies with almost perfect sensitivity and specificity.
“KG-RAG systems [18] enhanced trust even further, by centering triage decisions in biomedical ontologies with almost perfect sensitivity and specificity, something that the clinical community finds appealing during emergencies.”
From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [18]
The Results report that the knowledge graph–RAG model with global search achieved 0.98 sensitivity and 0.99 specificity in triaging emergency patient messages. The Materials and methods describe a graph built from nursing triage protocols, not biomedical ontologies. The paper does not report that the system enhanced trust or that clinicians found it appealing during emergencies. (LLM, conf=0.88)
Experimental — accuracy not yet measuredEmergency triage using KG-RAG achieved near-perfect performance (Accuracy 0.99; Sensitivity 0.98; Specificity 0.99), far exceeding keyword-based triage systems.
“Emergency triage using KG-RAG achieved near-perfect performance (Accuracy 0.99; Sensitivity 0.98; Specificity 0.99), far exceeding keywordbased triage systems [18].”
From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [18]
Table 2 in the Results reports accuracy 0.99, sensitivity 0.98, and specificity 0.99 for KG-RAG with global search. Its comparators were a prompt-only model, naïve RAG, and KG-RAG with local search—not keyword-based triage systems—so the claimed comparison with keyword-based systems is unsupported. (LLM, conf=0.94)
Experimental — accuracy not yet measuredAbsence of knowledge-graph links or lack of multilingual coverage might create dangers such as misclassification or unsafe recommendations.
“or in [18][21] the absence of knowledge-graph links or lack of multilingual coverage might create danger such as misclassification or unsafe recommendations.”
From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [18]
The Results (page unavailable) describe an emergency message misclassified as non-emergency, likely because specific drug-interaction information was absent from the knowledge graph. The Limitations section (page unavailable) also notes that the graph may not cover all emergency scenarios. The paper does not address multilingual coverage or link these gaps to unsafe recommendations; it discusses missing resources rather than specifically missing graph links. (LLM, conf=0.86)
Experimental — accuracy not yet measuredKnowledge-graph-based RAG models necessitated constant graph maintenance, ontology updates, and reliable storage systems, generally absent in rural and low-income healthcare settings.
“Similarly, knowledge-graph-based RAG models [18] also necessitated constant graph maintenance in addition to ontology updates and reliable storage systems, such as the kind that are generally absent in rural and low-income healthcare settings.”
From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [18]
The Materials and methods section describes constructing a knowledge graph, and the Discussion proposes expanding its knowledge base in future research. Neither passage says the model requires constant graph maintenance, ontology updates, or reliable storage systems, or addresses whether those systems are absent in rural and low-income healthcare settings. Searches for maintenance, updates, storage, and infrastructure found no passage establishing those claims; page numbers were unavailable in our copy. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · cited 2× · see citing works
“27 Incorporating a knowledge graph as part of the RAG approach provided a structured way to represent relationships between medical concepts, which in turn enhanced the LLM’s inference capabilities.”
Unifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376 · see citing works
Arndt, B. G., Beasley, J. W., Watkinson, M. D., Temte, J. L., Tuan, W.-J., Sinsky, C. A., & Gilchrist, V. J. (2017) · The Annals of Family Medicine · doi:10.1370/afm.2121 · Cited by 972 · cited 1× · see citing works
“42,43 Limitations One limitation of our study is the reliance on a nursing telephone triage book as a single source for knowledge graph development, which may not cover all possible emergency scenarios.”
Liu, S., McCoy, A. B., & Wright, A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf008 · Cited by 200 · cited 1× · see citing works
“20,21 24 Integrating external knowledge, for example, a nurse triage book with over 200 protocols for common concerns and symptoms, might further support the emergency tri-age task.”
Self-citationHallucination is Inevitable: An Innate Limitation of Large Language Models
Xu, Z., Jain, S., & Kankanhalli, M. (2024) · arXiv · doi:10.48550/arxiv.2401.11817 · Cited by 180 · see citing works
Flagged first, then by citations
Research works printed but not linked: 36
- [1]Huang M, Fan J, Prigge J, et al Characterizing patient-clinician communication in secure medical messages: retrospective study. J Med Internet Res. 2022; 24: e17273. 10.2196/17273 35014964 PMC8790696 · DOI
- [2]North F, Luhman KE, Mallmann EA, et al A retrospective analysis of provider-to-patient secure messages: how much are they increasing, who is doing the work, and is the work happening after hours? JMIR Med Inform. 2020; 8: e16521. 10.2196/16521 32673238 PMC7381047 · DOI
- [3]Sinsky CA, Shanafelt TD, Ripp JA. The electronic health record inbox: recommendations for relief. J Gen Intern Med. 2022; 37: 4002- 4003. 10.1007/s11606-022-07766-0 36036837 PMC9640509 · DOI
- [4]Holmgren AJ, Downing NL, Tang M, et al Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use. J Am Med Inf Assoc. 2022; 29: 453- 460. 10.1093/jamia/ocab268 PMC8689796 34888680 · DOI
- [5]Shimada SL, Petrakis BA, Rothendler JA, et al An analysis of patient-provider secure messaging at two Veterans Health Administration medical centers: message content and resolution through secure messaging. J Am Med Inf Assoc. 2017; 24: 942- 949. 10.1093/jamia/ocx021 PMC7651909 28371896 · DOI
- [6]Brown TB, Mann B, Ryder N, et al Language models are few-shot learners. Adv Neural Inf Process Syst. 2020; 33: 1877- 1901. 10.48550/arXiv.2005.14165 · DOI · Paper page
- [8]Alapati R, Campbell D, Molin N, et al Evaluating insomnia queries from an artificial intelligence chatbot for patient education. J Clin Sleep Med. 2024; 20: 583- 594. 10.5664/jcsm.10948 38217478 PMC10985291 · DOI
- [9]Liu S, Wright AP, Patterson BL, et al Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Inf Assoc. 2023; 30: 1237- 1245. 10.1093/jamia/ocad072 PMC10280357 37087108 · DOI · Paper page
- [10]Liu S, McCoy AB, Wright AP, et al Why do users override alerts? Utilizing large language model to summarize comments and optimize clinical decision support. J Am Med Inf Assoc. 2024; 31: 1388- 1396. 10.1093/jamia/ocae041 PMC11105133 38452289 · DOI
- [11]Liu S, Wright AP, Mccoy AB, et al Using large language model to guide patients to create efficient and comprehensive clinical care message. J Am Med Inf Assoc. 2024; 31: 1665- 1670. 10.1093/JAMIA/OCAE142 PMC11258400 38917441 · DOI
- [12]Liu S, McCoy AB, Wright AP, et al Leveraging large language models for generating responses to patient messages—a subjective analysis. J Am Med Inf Assoc. 2024; 31: 1367- 1379. 10.1093/jamia/ocae052 PMC11105129 38497958 · DOI
- [13]Levine DM, Tuwani R, Kompa B, et al The diagnostic and triage accuracy of the GPT-3 artificial intelligence model: an observational study. Lancet Digit Health. 2024; 6: e555-61. e561. 10.1016/S2589-7500(24)00097-9 39059888 · DOI · Paper page
- [15]Zhang Y, Li Y, Cui L, et al Siren’s song in the AI ocean: a survey on hallucination in large language models. 2023, preprint: not peer reviewed. https://https://arxiv.org/abs/2309.01219
- [16]Huang L, Yu W, Ma W, et al a survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. 2023, preprint: not peer reviewed. https://arxiv.org/abs/2311.05232 · arXiv
- [17]Rotmensch M, Halpern Y, Tlimat A, et al Learning a health knowledge graph from electronic medical records. Sci Rep. 2017; 7: 5994. 10.1038/s41598-017-05778-z 28729710 PMC5519723 · DOI · Paper page
- [18]Lo P-C, Tsai Y-H, Lim E-P, et al On exploring the reasoning capability of large language models with knowledge graphs. 2023, preprint: not peer reviewed. https://arxiv.org/abs/2312.00353 · arXiv
- [19]Tang W, Qiao X, Zhao X, et al HW-TSC at TextGraphs-17 shared task: enhancing inference capabilities of LLMs with knowledge graphs. In: Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing. Bangkok, Thailand: Association for Computational Linguistics; 2024: 131- 136.
- [20]Li X, Zhao R, Chia YK, et al Chain-of-knowledge: grounding large language models via dynamic knowledge adapting over heterogeneous sources. In: 12th International Conference on Learning Representations. Vienna, Austria: International Conference on Learning Representations, ICLR; 2024.
- [21]Peng B, Galley M, He P, et al Check your facts and try again: improving large language models with external knowledge and automated feedback. 2023, preprint: not peer reviewed. https://arxiv.org/abs/2302.12813 · arXiv
- [22]Xiong G, Jin Q, Wang X, et al Improving retrieval-augmented generation in medicine with iterative follow-up questions. 2024, preprint: not peer reviewed. https://arxiv.org/abs/2408.00727 PMC11997844 39670371 · arXiv
- [23]Huang H, Wang S, Liu H, et al Benchmarking large language models on communicative medical coaching: a dataset and a novel system. In: Findings of the Association for Computational Linguistics: ACL 2024. Bangkok, Thailand: Association for Computational Linguistics; 2024:1624-1637. 10.18653/V1/2024.FINDINGS-ACL.94 · DOI
- [25]Briggs JK. Telephone Triage Protocols for Nurses. Wolters Kluwer Health; 2021.
- [26]Gao Y, Xiong Y, Gao X, et al Retrieval-augmented generation for large language models: a survey. 2023, preprint: not peer reviewed. https://arxiv.org/abs/2312.10997 · arXiv
- [28]Xu P, Ping W, Wu X, et al Retrieval meets long context large language models. In: 12th International Conference on Learning Representations. Vienna, Austria: International Conference on Learning Representations, ICLR; 2024.
- [30]Sansford H, Richardson N, Maretic HP, et al GraphEval: a knowledge-graph based LLM hallucination evaluation framework. 2024, preprint: not peer reviewed. https://arxiv.org/abs/2407.10793 · arXiv
- [31]Chambon PJ, Wu C, Steinkamp JM, et al Automated deidentification of radiology reports combining transformer and “hide in plain sight” rule-based methods. J Am Med Inf Assoc. 2023; 30: 318- 328. 10.1093/jamia/ocac219 PMC9846681 36416419 · DOI
- [32]Traag VA, Waltman L, van Eck NJ. From Louvain to Leiden: guaranteeing well-connected communities. Sci Rep. 2019; 9: 5233. 10.1038/s41598-019-41695-z 30914743 PMC6435756 · DOI · Paper page
- [33]Edge D, Trinh H, Cheng N, et al From local to global: a graph rag approach to query-focused summarization. 2024, preprint: not peer reviewed. https://arxiv.org/abs/2404.16130 · arXiv
- [35]Rainio O, Teuho J, Klén R. Evaluation metrics and statistical tests for machine learning. Sci Rep. 2024; 14: 6086. 10.1038/s41598-024-56706-x 38480847 PMC10937649 · DOI · Paper page
- [36]Demsar J. Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res. 2006; 7: 1- 30.
- [38]Franc JM, Cheng L, Hart A, et al Repeatability, reproducibility, and diagnostic accuracy of a commercial large language model (ChatGPT) to perform emergency department triage using the Canadian triage and acuity scale. Can J Emerg Med. 2024; 26: 40- 46. 10.1007/s43678-023-00616-w 38206515 · DOI
- [39]Yazaki M, Maki S, Furuya T, et al Emergency patient triage improvement through a retrieval-augmented generation enhanced large-scale language model. Prehospital Emerg Care. 2025; 29: 203- 209. 10.1080/10903127.2024.2374400 38950135 · DOI · Paper page
- [40]Masanneck L, Schmidt L, Seifert A, et al Triage performance across large language models, ChatGPT, and untrained doctors in emergency medicine: comparative study. J Med Internet Res. 2024; 26: e53297. 10.2196/53297 38875696 PMC11214027 · DOI
- [41]Liu J, Wang C, Liu S. Utility of ChatGPT in clinical practice. J Med Internet Res. 2023; 25: e48568. 10.2196/48568 37379067 PMC10365580 · DOI · Paper page
- [42]Akbar F, Mark G, Warton EM, et al Physicians’ electronic inbox work patterns and factors associated with high inbox work duration. J Am Med Inf Assoc. 2021; 28: 923- 930. 10.1093/jamia/ocaa229 PMC8068414 33063087 · DOI
- [44]Jin Q, Chen F, Zhou Y, et al Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine. NPJ Digit Med. 2024; 7: 190. 10.1038/s41746-024-01185-7 39043988 PMC11266508 · DOI · Paper page
Cited by
Sorted by citations
Mugambiwa, T., & Ndlovu, B. (2026) · Journal of Applied Informatics and Computing · doi:10.30871/jaic.v10i1.11900 · Cited by 4 · see citing works
“KG-RAG systems [18] enhanced trust even further, by centering triage decisions in biomedical ontologies with almost perfect sensitivity and specificity, something that the clinical community finds appealing during emergencies.”
InfluentialDeobhakta, A., Chang, J., Chen, R. W. S., Jaipalli, S., Pinhas, A., Tania Tai, T. Y., & Rosen, R. B. (2026) · doi:10.64898/2026.10.09.26365153
- Authors
- Liu, S., Wright, A. P., McCoy, A. B., Huang, S. S., Steitz, B., & Wright, A.
- Affiliations
- Vanderbilt University (US)
- Vanderbilt University Medical Center (US)
- Corresponding author
- Liu, Siru
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2025-04-12
- Volume
- 32
- Issue
- 6
- Pages
- 1032-1039
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC12089757
- OpenAlex
- W4409386825
- Cited by
- 44
- References
- 6
- Citations by year
- 2026: 32 · 2025: 12
- FWCI
- 17.65
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- NIH (R00LM014097-02, R01LM013995-01)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- patient portal messages, emergency department triage, large language models, retrieval-augmented generation, knowledge graphs, community detection, patient safety, clinical decision support
- MeSH terms
- Patient Portals, Large Language Models, Emergencies, Humans, Natural Language Processing, Triage, Information Storage and Retrieval, Knowledge Bases
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
