Unsupervised Machine Learning of Topics Documented by Nurses about Hospitalized Patients Prior to a Rapid-Response Event
Open accessAbstract
Background In the hospital setting, it is crucial to identify patients at risk for deterioration before it fully develops, so providers can respond rapidly to reverse the deterioration. Rapid response (RR) activation criteria include a subjective component (“worried about the patient”) that is often documented in nurses' notes and is hard to capture and quantify, hindering active screening for deteriorating patients. Objectives We used unsupervised machine learning to automatically discover RR event risk/protective factors from unstructured nursing notes. Methods In this retrospective cohort study, we obtained nursing notes of hospitalized, nonintensive care unit patients, documented from 2015 through 2018 from Partners HealthCare databases. We applied topic modeling to those notes to reveal topics (clusters of associated words) documented by nurses. Two nursing experts named each topic with a representative Systematized Nomenclature of Medicine–Clinical Terms (SNOMED CT) concept. We used the concepts along with vital signs and demographics in a time-dependent covariates extended Cox model to identify risk/protective factors for RR event risk. Results From a total of 776,849 notes of 45,299 patients, we generated 95 stable topics, of which 80 were mapped to 72 distinct SNOMED CT concepts. Compared with a model containing only demographics and vital signs, the latent topics improved the model's predictive ability from a concordance index of 0.657 to 0.720. Thirty topics were found significantly associated with RR event risk at a 0.05 level, and 11 remained significant after Bonferroni correction of the significance level to 6.94E-04, including physical examination (hazard ratio [HR] = 1.07, 95% confidence interval [CI], 1.03–1.12), informing doctor (HR = 1.05, 95% CI, 1.03–1.08), and seizure precautions (HR = 1.08, 95% CI, 1.04–1.12). Conclusion Unsupervised machine learning methods can automatically reveal interpretable and informative signals from free-text and may support early identification of patients at risk for RR events.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI is able to interpret nursing documentation to recognize patient types and predict clinical deterioration.
“AI is able to interpret nursing documentation to recognize patient types and predict clinical deterioration.43–47”
From Potential uses of AI for perioperative nursing handoffs: a qualitative study (2023) · cites this paper as [43]
The Outcome Calculation section says unsupervised machine learning extracted clinical concepts from nursing notes for a model of rapid-response events. The Discussion says those topics improved prediction of rapid-response-event hazard beyond vital signs alone. The paper identifies concepts in notes, not patient types, and studies rapid-response events as the outcome rather than clinical deterioration broadly. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,649 · see citing works
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,171 · see citing works
Peduzzi, P., Concato, J., Feinstein, A. R., & Holford, T. R. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00048-8 · Cited by 2,154 · see citing works
Introduction of the medical emergency team (MET) system: a cluster-randomised controlled trial
‡ (2005) · The Lancet · doi:10.1016/s0140-6736(05)66733-5 · Cited by 1,291 · see citing works
Concato, J., Peduzzi, P. N., Holford, T. R., & Feinstein, A. R. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00510-2 · Cited by 797 · see citing works
Bellomo, R., Goldsmith, D., Uchino, S., Buckmaster, J., Hart, G. K., Opdam, H. I., Silvester, W., Doolan, L., & Gutteridge, G. A. (2004) · Critical Care Medicine · doi:10.1097/01.ccm.0000119428.02968.9e · Cited by 507 · see citing works
Rapid-Response Systems as a Patient Safety Strategy: A Systematic Review
Winters, B. D., Weaver, S. J., Pfoh, E. R., Yang, T., Pham, J. C., & Dy, S. M. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-5-201303051-00009 · Cited by 438 · see citing works
Hodgetts, T. J., Kenward, G., Vlackonikolis, I., Payne, S., Nicolas Castle, B., Crouch, R., Ineson, N., & Shaikh, L. A. H. A. (2002) · Resuscitation · doi:10.1016/s0300-9572(02)00098-9 · Cited by 413 · see citing works
Statistical topic models for multi-label document classification
Rubin, T. N., Chambers, A., Smyth, P., & Steyvers, M. (2011) · Machine Learning · doi:10.1007/s10994-011-5272-5 · Cited by 295 · see citing works
Solomon, R. S., Corwin, G. S., Barclay, D. C., Quddusi, S. F., & Dannenberg, M. D. (2016) · Journal of Hospital Medicine · doi:10.1002/jhm.2554 · Cited by 225 · see citing works
Parr, M. J., Hadfield, J., Flabouris, A., Bishop, G., & Hillman, K. (2001) · Resuscitation · doi:10.1016/s0300-9572(01)00323-9 · Cited by 157 · see citing works
Relationship Between Nursing Documentation and Patients' Mortality
Collins, S. A., Cato, K., Albers, D., Scott, K., Stetson, P. D., Bakken, S., & Vawdrey, D. K. (2013) · American Journal of Critical Care · doi:10.4037/ajcc2013426 · Cited by 157 · see citing works
Self-citationPerotte, A., Ranganath, R., Hirsch, J. S., Blei, D., & Elhadad, N. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv024 · Cited by 128 · see citing works
Wang, L., Sha, L., Lakin, J. R., Bynum, J., Bates, D. W., Hong, P., & Zhou, L. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.6972 · Cited by 105 · see citing works
Self-citationKollef, M. H., Chen, Y., Heard, K. M., LaRossa, G. N., Lu, C., Martin, N. R., Martin, N., Micek, S. T., & Bailey, T. C. (2014) · Journal of Hospital Medicine · doi:10.1002/jhm.2193 · Cited by 91 · see citing works
Activation of a Medical Emergency Team Using an Electronic Medical Recording–Based Screening System*
Huh, J. W., Lim, C., Koh, Y., Lee, J., Jung, Y.-K., Seo, H.-S., & Hong, S. (2013) · Critical Care Medicine · doi:10.1097/ccm.0000000000000031 · Cited by 68 · see citing works
Repeated assessment of risk factors in survival analysis
Andersen, P. K. (1992) · Statistical Methods in Medical Research · doi:10.1177/096228029200100305 · Cited by 57 · see citing works
Shao, Y., Mohanty, A. F., Ahmed, A., Weir, C. R., Bray, B. E., Shah, R. U., Redd, D., & Zeng‐Treitler, Q. (2016) · PubMed Central · Cited by 36 · see citing works
(2018) · Europe PMC (PubMed Central) · Cited by 34 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Use of unstructured text in prognostic clinical prediction models: a systematic review
Seinen, T. M., Fridgeirsson, E. A., Ioannou, S., Jeannetot, D. Y. B., John, L. H., Kors, J. A., Markus, A. F., Pera, V., Rekkas, A., Williams, R. D., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac058 · Cited by 80 · see citing works
Machine learning applied to electronic health record data in home healthcare: A scoping review
Hobensack, M., Song, J., Scharp, D., Bowles, K. H., & Topaz, M. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104978 · Cited by 44 · see citing works
Rossetti, S. C., Knaplund, C., Albers, D., Dykes, P. C., Kang, M. J., Korach, T. Z., Zhou, L., Schnock, K., Garcia, J., Schwartz, J., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab006 · Cited by 42 · see citing works
Self-citationHome Healthcare Clinical Notes Predict Patient Hospitalization and Emergency Department Visits
Topaz, M., Woo, K., Ryvicker, M. B., Zolnoori, M., & Cato, K. (2020) · Nursing Research · doi:10.1097/nnr.0000000000000470 · Cited by 41 · see citing works
Self-citationHuang, K., Gray, T. F., Brufau, S. R., Tulsky, J. A., & Lindvall, C. J. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab051 · Cited by 33 · see citing works
Potential uses of AI for perioperative nursing handoffs: a qualitative study
King, C. R., Shambe, A., & Abraham, J. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad015 · Cited by 27 · see citing works
Principles for Real-World Implementation of Bedside Predictive Analytics Monitoring
Moorman, L. P. (2021) · Applied Clinical Informatics · doi:10.1055/s-0041-1735183 · Cited by 26 · see citing works
Chung, J., & Jung, H. (2024) · BMC Nursing · doi:10.1186/s12912-024-02531-6 · Cited by 25 · see citing works
Dagli, M. M., Rajesh, A., Asaad, M., & Butler, C. E. (2021) · The American Surgeon · doi:10.1177/00031348211065101 · Cited by 22 · see citing works
Artificial intelligence and clinical deterioration
Malycha, J., Bacchi, S., & Redfern, O. (2022) · Current Opinion in Critical Care · doi:10.1097/mcc.0000000000000945 · Cited by 20 · see citing works
Chopannejad, S., Sadoughi, F., Bagherzadeh, R., & Shekarchi, S. (2022) · Applied Clinical Informatics · doi:10.1055/a-1863-1589 · Cited by 19 · see citing works
Identifying High-Need Primary Care Patients Using Nursing Knowledge and Machine Learning Methods
Hewner, S. J., Smith, E., & Sullivan, S. S. (2023) · Applied Clinical Informatics · doi:10.1055/a-2048-7343 · Cited by 16 · see citing works
Kim, M., Park, S., Kim, C., & Choi, M. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2022.104411 · Cited by 15 · see citing works
Williams, C. Y. K., Li, R. X., Luo, M. Y., & Bance, M. (2023) · Clinical Otolaryngology · doi:10.1111/coa.14037 · Cited by 12 · see citing works
Detecting modeling inconsistencies in SNOMED CT using a machine learning technique
Agrawal, A., & Qazi, K. (2020) · Methods · doi:10.1016/j.ymeth.2020.05.019 · Cited by 13 · see citing works
Hoekstra, O., Hurst, W., & Tummers, J. (2022) · Healthcare Analytics · doi:10.1016/j.health.2022.100107 · Cited by 12 · see citing works
Tiase, V. L., Sward, K. A., & Facelli, J. C. (2024) · JMIR Nursing · doi:10.2196/55793 · Cited by 7 · see citing works
Butler, J. M., Doubleday, A., Sattar, U., Nies, M. A., Jeppesen, A., Wright, M. C., Reese, T., Kawamoto, K., Fiol, G. D., & Madaras‐Kelly, K. (2025) · Applied Clinical Informatics · doi:10.1055/a-2505-7743 · Cited by 5 · see citing works
Celebrating Clinical Informatics as a Specialty Practice
Yu, F., & O’Brien, A. (2020) · Applied Clinical Informatics · doi:10.1055/s-0039-3401812 · Cited by 4 · see citing works
Combining simulation and machine learning in healthcare: a literature review
Zhao, Q.-X., Blake, J., Taghavi, M., & Vanberkel, P. (2025) · Journal of Simulation · doi:10.1080/17477778.2025.2549092 · Cited by 2 · see citing works
Li, M.-Y., Chen, D., Xiao, Q., He, Z.-J., Zhang, Y.-Y., Zhong, J., Luo, Y.-W., & Ma, H. (2026) · Journal of Medical Internet Research · doi:10.2196/89850 · Cited by 2 · see citing works
Reunamo, A., Moen, H., Salanterä, S., & Lähteenmäki, P. M. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1585309 · Cited by 1 · see citing works
Chen, Z., Liu, Y., Prosperi, M., Vaddiparti, K., Cook, R. L., Bian, J., Guo, Y. M., & Wu, Y. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106269 · Cited by 1 · see citing works
Applied Informatics Research in Nursing for eHealth
Saranto, K., & Kinnunen, U. (2021) · Health informatics · doi:10.1007/978-3-030-58740-6_13 · Cited by 1 · see citing works
Kondratieff, K., Brown, J. T., Barron, M., Warner, J. L., & Yin, Z. (2022) · PubMed Central · Cited by 1 · see citing works
Identification of Throat Cancer by Machine Learning : A Surey
Akshara, E. A. R. (2021) · Türk bilgisayar ve matematik eğitimi dergisi · doi:10.17762/turcomat.v12i11.5939
Analysis of COVID-19 Database for Defining the Most Important Symptoms
Salih, M. A., Hamad, M. M., & Jasim, W. M. (2023) · Journal Healthcare Treatment Development · doi:10.55529/jhtd.33.17.26
Li, L., Zhang, X. S., & Pan, P. (2024) · International Conference on Big Data Economy and Information Management (BDEIM) · doi:10.1145/3724154.3724356
Chen, Z., Liu, Y., Prosperi, M., Vaddiparti, K., Cook, R. L., Bian, J., Guo, Y., & Wu, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5334140
Byrd, T. F., Mattson, M., Polt, M., Pint, K., Dimitrov, K., Willis, A., Lister, J., Beacom, E., & Tignanelli, C. (2025) · Applied Clinical Informatics · doi:10.1055/a-2648-4817 · Cited by 1 · see citing works
Physicians Report Benefit from Guided Critical Care Algorithms During Inpatient Rapid Responses
Senter-Zapata, M. J., Baugh, C. W., Onorato, S. E., Tiako, M. J. N., Hare, A. J., Aribeana, C. N., Isselbacher, E. M., & Conley, J. J. (2025) · Applied Clinical Informatics · doi:10.1055/a-2734-1754
Nurses’ Role in Identifying Clinical Deterioration Without Vital Sign Abnormalities
Alhaddad, A. M., AlSomali, N. A. Y., Alruwaili, A. M. A., Alruwaili, A. N. H., Mohammed Matar, J., Alanazi, A. K. J., Alanazi, A. K. J., Alanazi, E. M. F., al., H. S. J. A. E., Alenzi, F. F. A., et al. (2024) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.4722
- Authors
- Korach, Z. T., Cato, K. D., Collins, S. A., Kang, M. J., Knaplund, C., Dykes, P. C., Wang, L., Schnock, K. O., Garcia, J. P., Jia, H., Chang, F., Schwartz, J. M., & Zhou, L.
- Affiliations
- Brigham and Women's Hospital (US)
- Columbia University (US)
- Corresponding author
- Korach, Zfania Tom
- ORCID
- Venue
- Applied Clinical Informatics
- Publisher
- Georg Thieme Verlag KG
- Published
- 2019-10
- Volume
- 10
- Issue
- 05
- Pages
- 952-963
- ISSN
- 1869-0327 (ISSN-L)
- OpenAlex
- W2996337964
- Cited by
- 32
- References
- 20
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 4 · 2023: 4 · 2022: 7 · 2021: 6 · 2020: 3
- FWCI
- 1.82
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Applied Clinical Informatics (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institute of Nursing Research (P30NR016587, T32NR007969, R01NR016941)
- National Institute on Minority Health and Health Disparities
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Nursing › Issues, ethics and legal aspects › Nursing Diagnosis and Documentation
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- clinical deterioration, nursing notes, topic modeling, unsupervised learning, risk factors, protective factors, SNOMED CT, early warning, vital signs
- MeSH terms
- Unsupervised Machine Learning, Documentation, Hospitalization, Humans, Nurses, Survival Analysis, Risk Assessment, Hospital Rapid Response Team, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
