Mixed-methods evaluation of three natural language processing modeling approaches for measuring documented goals-of-care discussions in the electronic health record
Open accessAbstract
Context Documented goals-of-care discussions are an important quality metric for patients with serious illness. Natural language processing (NLP) is a promising approach for identifying goals-of-care discussions in the electronic health record (EHR). Objectives To compare three NLP modeling approaches for identifying EHR documentation of goals-of-care discussions and generate hypotheses about differences in performance. Methods We conducted a mixed-methods study to evaluate performance and misclassification for three NLP featurization approaches modeled with regularized logistic regression: bag-of-words (BOW), rule-based, and a hybrid approach. From a prospective cohort of 150 patients hospitalized with serious illness over 2018 to 2020, we collected 4391 inpatient EHR notes; 99 (2.3%) contained documented goals-of-care discussions. We used leave-one-out cross-validation to estimate performance by comparing pooled NLP predictions to human abstractors with receiver-operating-characteristic (ROC) and precision-recall (PR) analyses. We qualitatively examined a purposive sample of 70 NLP-misclassified notes using content analysis to identify linguistic features that allowed us to generate hypotheses underpinning misclassification. Results All three modeling approaches discriminated between notes with and without goals-of-care discussions (AUC ROC: BOW, 0.907; rule-based, 0.948; hybrid, 0.965). Precision and recall were only moderate (precision at 70% recall: BOW, 16.2%; rule-based, 50.4%; hybrid, 49.3%; AUC PR: BOW, 0.505; rule-based, 0.579; hybrid, 0.599). Qualitative analysis revealed patterns underlying performance differences between BOW and rule-based approaches. Conclusion NLP holds promise for identifying EHR-documented goals-of-care discussions. However, the rarity of goals-of-care content in EHR data limits performance. Our findings highlight opportunities to optimize NLP modeling approaches, and support further exploration of different NLP approaches to identify goals-of-care discussions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cost barrier of task-specific model development poses significant challenges for researchers interested in measuring multidimensional constructs whose definitions appropriately differ from study to study.
“This cost barrier poses significant challenges for researchers interested in measuring multidimensional constructs for which definitions are likely to differ appropriately from study to study.37,38,35”
From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [35]
The Discussion (page unavailable) says training data for documented goals-of-care discussions are sparse and costly to acquire, and describes those discussions as multifaceted. The Methods (Data Source; page unavailable) says there is no consensus definition for identifying them in inpatient records and explains the study’s pragmatic definition. The paper does not explicitly extend this cost barrier to researchers measuring multidimensional constructs generally or say that definitions appropriately differ from study to study. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Three Approaches to Qualitative Content Analysis
Hsieh, H.-F., & Shannon, S. E. (2005) · Qualitative Health Research · doi:10.1177/1049732305276687 · Cited by 44,601 · see citing works
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559 · see citing works
A statistical interpretation of term specificity and its application in retrieval
Jones, K. S. (2021) · J. Documentation · doi:10.1108/eb026526 · Cited by 5,541 · see citing works
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309 · see citing works
Artificial intelligence in healthcare: past, present and future
Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., & Wang, Y. (2017) · Stroke and Vascular Neurology · doi:10.1136/svn-2017-000101 · Cited by 4,837 · see citing works
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921 · see citing works
Wright, A. A. (2008) · JAMA · doi:10.1001/jama.300.14.1665 · Cited by 2,788 · see citing works
Proceedings of the 23rd international conference on Machine learning
Cohen, W. W., & Moore, A. (2006) · Cited by 2,647 · see citing works
Charles, C. A., Gafni, A., & Whelan, T. J. (1999) · Social Science & Medicine · doi:10.1016/s0277-9536(99)00145-8 · Cited by 2,336 · see citing works
Detering, K. M., Hancock, A. D., Reade, M. C., & Silvester, W. (2010) · BMJ · doi:10.1136/bmj.c1345 · Cited by 2,332 · see citing works
Natural language processing: an introduction
Nadkarni, P. M., Lucila Ohno-machado, A., & Chapman, W. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000464 · Cited by 1,984 · see citing works
Sudore, R. L., Lum, H. D., You, J. J., Hanson, L. C., Meier, D. E., Pantilat, S. Z., Matlock, D. D., Rietjens, J. A. C., Korfage, I. J., Ritchie, C. S., et al. (2017) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2016.12.331 · Cited by 1,726 · see citing works
Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation
Wong, T.-T. (2015) · Pattern Recognition · doi:10.1016/j.patcog.2015.03.009 · Cited by 1,562 · see citing works
Communication About Serious Illness Care Goals: A Review and Synthesis of Best Practices
Bernacki, R. E., & Block, S. D. (2014) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2014.5271 · Cited by 1,422 · see citing works
Clinical information extraction applications: A literature review
Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.011 · Cited by 910 · see citing works
Health Care Costs in the Last Week of Life: Associations With End-of-Life Conversations
Zhang, B., Wright, A. A., Huskamp, H. A., Nilsson, M. E., Maciejewski, M. L., Earle, C. C., Block, S. D., Maciejewski, P. K., & Prigerson, H. G. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2008.587 · Cited by 898 · see citing works
Redefining the “Planning” in Advance Care Planning: Preparing for End-of-Life Decision Making
Sudore, R. L., & Fried, T. R. (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-153-4-201008170-00008 · Cited by 854 · see citing works
FERRELL, B. R., Twaddle, M. L., Melnick, A., & Meier, D. E. (2018) · Journal of Palliative Medicine · doi:10.1089/jpm.2018.0431 · Cited by 775 · see citing works
Kreimeyer, K., Foster, M., Pandey, A., Arya, N., Halford, G., Jones, S. F., Forshee, R., Walderhaug, M., & Botsis, T. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.012 · Cited by 616 · see citing works
The retrospective chart review: important methodological considerations
Vassar, M. B., & Matthew, H. C. G. (2013) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2013.10.12 · Cited by 603 · see citing works
Area under the Precision-Recall Curve: Point Estimates and Confidence Intervals
Boyd, K., Eng, K., & Page, D. (2013) · ECML/PKDD · doi:10.1007/978-3-642-40994-3_29 · Cited by 549 · see citing works
Deep learning in clinical natural language processing: a methodical review
Wu, S., Roberts, K. E., Datta, S., Du, J., Ji, Z., Si, Y., Soni, S., Wang, Q., Wei, Q., Xiang, Y., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz200 · Cited by 495 · see citing works
Patient Preferences for Communication with Physicians about End-of-Life Decisions
Hofmann, J. C., Wenger, N. S., Davis, R. B., Teno, J., Connors, A. F., Desbiens, N., Lynn, J., & Phillips, R. S. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-1-199707010-00001 · Cited by 449 · see citing works
What’s Wrong With Advance Care Planning?
Morrison, R. S., Meier, D. E., & Arnold, R. M. (2021) · JAMA · doi:10.1001/jama.2021.16430 · Cited by 447 · see citing works
Curtis, J. R., Treece, P. D., Nielsen, E. L., Gold, J., Ciechanowski, P. S., Shannon, S. E., Khandelwal, N., Young, J. P., & Engelberg, R. A. (2016) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201505-0900oc · Cited by 398 · see citing works
Self-citationOzenne, B., Subtil, F., & Maucort-Boulch, D. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.02.010 · Cited by 379 · see citing works
Investigator Triangulation: A Collaborative Strategy With Potential for Mixed Methods Research
Archibald, M. M. (2016) · Journal of Mixed Methods Research · doi:10.1177/1558689815570092 · Cited by 355 · see citing works
White, D. B. (2007) · Archives of Internal Medicine · doi:10.1001/archinte.167.5.461 · Cited by 345 · see citing works
REMAP: A Framework for Goals of Care Conversations
Childers, J. W., Back, A. L., Tulsky, J. A., & Arnold, R. M. (2017) · Journal of Oncology Practice · doi:10.1200/jop.2016.018796 · Cited by 327 · see citing works
Sanders, J. J., Curtis, J. R., & Tulsky, J. A. (2017) · Journal of Palliative Medicine · doi:10.1089/jpm.2017.0459 · Cited by 318 · see citing works
Self-citationCurtis, J. R., Downey, L., Back, A. L., Nielsen, E. L., Paul, S. R., Lahdya, A. Z., Treece, P. D., Armstrong, P., Peck, R., & Engelberg, R. A. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.2317 · Cited by 265 · see citing works
Self-citationBarriers and Facilitators to End-of-Life Care Communication for Patients with COPD
Knauft, E., Nielsen, E. L., Engelberg, R. A., Patrick, D. L., & Curtis, J. R. (2005) · CHEST Journal · doi:10.1378/chest.127.6.2188 · Cited by 265 · see citing works
Self-citationTulsky, J. A., Beach, M. C., Butow, P. N., Hickman, S. E., Mack, J. W., Morrison, R. S., Street, R. L., Sudore, R. L., White, D. B., & Pollak, K. I. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2017.2005 · Cited by 246 · see citing works
Sinuff, T., Dodek, P., You, J. J., Barwich, D., Tayler, C., Downar, J., Hartwick, M., Frank, C., Stelfox, H. T., & Heyland, D. K. (2015) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2014.12.007 · Cited by 241 · see citing works
When to consult precision-recall curves
Cook, J. A., & Ramadas, V. (2020) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x20909693 · Cited by 137 · see citing works
Secunda, K. E., Wirpsa, M. J., Neely, K. J., Szmuilowicz, E., Wood, G., Panozzo, E., McGrath, J., Levenson, A. R., Peterson, J., Gordon, E. J., et al. (2019) · Journal of General Internal Medicine · doi:10.1007/s11606-019-05446-0 · Cited by 134 · see citing works
The Quality of Care Provided to Hospitalized Patients at the End of Life
Walling, A. M. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.175 · Cited by 117 · see citing works
Scheunemann, L. P., Arnold, R. M., & White, D. B. (2012) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201204-0710cp · Cited by 115 · see citing works
Utilizing the Electronic Health Record to Improve Advance Care Planning: A Systematic Review
Huber, M. T., Highland, J., Krishnamoorthi, V. R., & Tang, J. W. (2017) · American Journal of Hospice and Palliative Medicine® · doi:10.1177/1049909117715217 · Cited by 112 · see citing works
Lamas, D. J., Panariello, N., Henrich, N. J., Hammes, B. J., Hanson, L. C., Meier, D. E., Guinn, N., Corrigan, J. M., Hubber, S., Luetke-Stahlman, H., et al. (2018) · Journal of Palliative Medicine · doi:10.1089/jpm.2017.0451 · Cited by 101 · see citing works
Lindvall, C. J., Lilley, E. J., Zupanc, S. N., Chien, I., Udelsman, B. V., Walling, A. M., Cooper, Z. R., & Tulsky, J. A. (2018) · Journal of Palliative Medicine · doi:10.1089/jpm.2018.0326 · Cited by 100 · see citing works
Natural Language Processing in Biomedicine: A Unified System Architecture Overview
Doan, S., Conway, M., Phuong, T. M., & Lucila Ohno-machado, A. (2014) · Methods in molecular biology · doi:10.1007/978-1-4939-0847-9_16 · Cited by 89 · see citing works
Wilson, C. J., Newman, J., Tapper, S., Lai, S., Cheng, P. H., Wu, F. M., & Tai-Seale, M. (2013) · Journal of Palliative Medicine · doi:10.1089/jpm.2012.0472 · Cited by 86 · see citing works
Methods to Achieve High Interrater Reliability in Data Collection From Primary Care Medical Records
Liddy, C. E., Wiens, M., & Hogg, W. E. (2011) · The Annals of Family Medicine · doi:10.1370/afm.1195 · Cited by 85 · see citing works
Curtis, J. R., Sathitratanacheewin, S., Starks, H. E., Lee, R. Y., Kross, E. K., Downey, L., Sibley, J., Lober, W. B., Loggers, E. T., Fausto, J. A., et al. (2017) · Journal of Palliative Medicine · doi:10.1089/jpm.2017.0542 · Cited by 85 · see citing works
Self-citationLee, R. Y., Brumback, L. C., Lober, W. B., Sibley, J., Nielsen, E. L., Treece, P. D., Kross, E. K., Loggers, E. T., Fausto, J. A., Lindvall, C. J., et al. (2020) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2020.08.024 · Cited by 81 · see citing works
Self-citationTowards Linguistically Generalizable NLP Systems: A Workshop and Shared Task
Ettinger, A., Rao, S., Daumé, H., & Bender, E. M. (2017) · doi:10.18653/v1/w17-5401 · Cited by 78 · see citing works
Stratification bias in low signal microarray studies
Parker, B. J., Günter, S., & Bedő, J. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-326 · Cited by 68 · see citing works
Chan, A., Chien, I., Moseley, E., Salman, S., Kaminer Bourland, S., Lamas, D., Walling, A. M., Tulsky, J. A., & Lindvall, C. (2019) · Palliative Medicine · doi:10.1177/0269216318810421 · Cited by 56 · see citing works
Code Status Orders and Goals of Care in the Medical ICU
Gehlbach, T. G., Shinkunas, L. A., Forman-Hoffman, V. L., Thomas, K. W., Schmidt, G. A., & Kaldjian, L. C. (2011) · Chest · doi:10.1378/chest.10-1798 · Cited by 51 · see citing works
Modes, M. E., Engelberg, R. A., Downey, L., Nielsen, E. L., Curtis, J. R., & Kross, E. K. (2018) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2018.10.507 · Cited by 50 · see citing works
Self-citationValidation of quality indicators for end-of-life communication: results of a multicentre survey
Heyland, D. K., Dodek, P., You, J. J., Sinuff, T., Hiebert, T., Tayler, C., Jiang, X., Simon, J. E., & Downar, J. (2017) · Canadian Medical Association Journal · doi:10.1503/cmaj.160515 · Cited by 44 · see citing works
Ahluwalia, S. C., Tisnado, D. M., Walling, A. M., Dy, S. M., Asch, S. M., Ettner, S. L., Kim, B. B., Pantoja, P. M., Schreibeis-Baum, H. C., & Lorenz, K. (2015) · Journal of Palliative Medicine · doi:10.1089/jpm.2014.0431 · Cited by 43 · see citing works
Orford, N. R., Milnes, S. L., Lambert, N., Berkeley, L., Lane, S. E., Simpson, N., Elderkin, T., Bone, A., Martin, P., Corke, C. F., et al. (2016) · Critical Care and Resuscitation · doi:10.1016/s1441-2772(23)00944-4 · Cited by 43 · see citing works
Three Stories About the Value of Advance Care Planning
Curtis, J. R. (2021) · JAMA · doi:10.1001/jama.2021.21075 · Cited by 43 · see citing works
Self-citationDocumentation Quality of Inpatient Code Status Discussions
Thurston, A. L., Wayne, D. B., Feinglass, J. M., & Sharma, R. K. (2014) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2013.11.014 · Cited by 34 · see citing works
A Method of Short Text Representation Based on the Feature Probability Embedded Vector
Zhou, W., Wang, H., Sun, H., & Sun, T. (2019) · Sensors · doi:10.3390/s19173728 · Cited by 30 · see citing works
Deep Natural Language Processing Identifies Variation in Care Preference Documentation
Udelsman, B. V., Moseley, E. T., Sudore, R. L., Keating, N. L., & Lindvall, C. J. (2020) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2019.12.374 · Cited by 27 · see citing works
Koopman, B., Zuccon, G., Nguyen, A. N., Bergheim, A., & Grayson, N. (2018) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2018.04.011 · Cited by 24 · see citing works
Silfen, E. Z. (2014) · Methods in molecular biology · doi:10.1007/978-1-4939-0847-9 · Cited by 18 · see citing works
Exploiting Technology to Popularize Goals-of-Care Conversations and Advance Care Planning
Esfahani, S., Yi, C., Madani, C., Davidson, J. E., Edmonds, K. P., & Wynn, S. (2020) · Critical Care Nurse · doi:10.4037/ccn2020576 · Cited by 14 · see citing works
Chien, I., Shi, A., Chan, A., & Lindvall, C. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-12738-1_15 · Cited by 9 · see citing works
Howell, K., Barnes, M., Curtis, J. R., Engelberg, R. A., Lee, R. Y., Lober, W. B., Sibley, J., & Cohen, T. (2020) · Studies in computational intelligence · doi:10.1007/978-3-030-53352-6_25 · Cited by 6 · see citing works
Self-citationSteiner, J. M., Morse, C., Lee, R. Y., Curtis, J. R., & Engelberg, R. A. (2020) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2020.06.018 · Cited by 5 · see citing works
Self-citationPrognostic information, goals of care, and code status decision‐making among older patients
van Dyck, L. I., & Fried, T. R. (2021) · Journal of the American Geriatrics Society · doi:10.1111/jgs.17080 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [50]Rothman KJ, Greenland S, Lash TL. Modern Epidemiology. Philadelphia: Lippincott Williams & Wilkins; 2008.
Cited by
Sorted by citations
Curtis, J. R., Lee, R. Y., Brumback, L. C., Kross, E. K., Downey, L., Torrence, J., LeDuc, N., Mallon Andrews, K., Im, J., Heywood, J., et al. (2023) · JAMA · doi:10.1001/jama.2023.8812 · Cited by 91 · see citing works
Self-citationLee, R. Y., Kross, E. K., Torrence, J., Li, K. S., Sibley, J., Cohen, T., Lober, W. B., Engelberg, R. A., & Curtis, J. R. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.1204 · Cited by 74 · see citing works
Self-citationPredictors of Documented Goals-of-Care Discussion for Hospitalized Patients With Chronic Illness
Uyeda, A. M., Lee, R. Y., Pollack, L. R., Paul, S. R., Downey, L., Brumback, L. C., Engelberg, R. A., Sibley, J., Lober, W. B., Cohen, T., et al. (2022) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2022.11.012 · Cited by 45 · see citing works
Self-citationThe use of natural language processing in palliative care research: A scoping review
Sarmet, M., Kabani, A., Coelho, L. P., Reis, S. S., Zeredo, J. L. L., & Mehta, A. K. (2022) · Palliative Medicine · doi:10.1177/02692163221141969 · Cited by 34 · see citing works
Classification of neurologic outcomes from medical notes using natural language processing
Fernandes, M. P., Valizadeh, N., Alabsi, H. S., Quadri, S. A., Tesh, R. A., Bucklin, A. A., Sun, H., Jain, A., Brenner, L. N., Ye, E. M., et al. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119171 · Cited by 25 · see citing works
Bozkurt, S., Fereydooni, S., Kar, İ., Chalmers, C. D., Leslie, S., Pathak, R., Walling, A. M., Lindvall, C. J., Lorenz, K., Parikh, R. B., et al. (2025) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2025.08.009 · Cited by 16 · see citing works
Curtis, J. R., Lee, R. Y., Brumback, L. C., Kross, E. K., Downey, L., Torrence, J., Heywood, J., LeDuc, N., Mallon Andrews, K., Im, J., et al. (2022) · Contemporary Clinical Trials · doi:10.1016/j.cct.2022.106879 · Cited by 12 · see citing works
Self-citationDurieux, B. N., Zverev, S. R., Tarbi, E. C., Kwok, A., Sciacca, K. R., Pollak, K. I., Tulsky, J. A., & Lindvall, C. J. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad009 · Cited by 9 · see citing works
Detecting goals of care conversations in clinical notes with active learning
Weissenbacher, D., Courtright, K. R., Rawal, S., Crane‐Droesch, A., O’Connor, K., Kuhl, N., Merlino, C., Foxwell, A. M., Haines, L., Puhl, J., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104618 · Cited by 9 · see citing works
Sharma, S., Dahne, J., Gallicchio, L. M., Mollica, M. A., Tonorezos, E. S., Petrillo, L. A., Nasso, S. F., Ross, B., & Graboyes, E. M. (2025) · JCO Oncology Practice · doi:10.1200/op-25-00872 · Cited by 5 · see citing works
Xu, T., Liu, C., Li, L., Song, D., Rose, G. M., & Zhu, S. (2026) · Journal of Palliative Medicine · doi:10.1177/10966218261418542 · Cited by 7 · see citing works
Xu, T., & Rose, G. M. (2025) · Journal of Hospice and Palliative Nursing · doi:10.1097/njh.0000000000001120 · Cited by 4 · see citing works
Lee, R. Y., Uyeda, A. M., Li, K. S., Sibley, J., Cohen, T., Lober, W. B., Dotolo, D. G., Engelberg, R. A., & Kross, E. K. (2025) · Journal of Palliative Medicine · doi:10.1089/jpm.2024.0496 · Cited by 3 · see citing works
Self-citationAssessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions
Lee, R. Y., Li, K. S., Sibley, J., Cohen, T., Lober, W. B., Dotolo, D. G., & Kross, E. K. (2026) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2025.09.025 · Cited by 5 · see citing works
Self-citationLeveraging artificial intelligence in advance care planning: A scoping review
Tan, M., Tang, S., Ni, Z., Kan, S., Macharia, P. M., Zhang, H., Yi, H., Li, G., & Ding, J. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103315 · Cited by 1 · see citing works
Cohen, T., Pakhomov, S., Paullada, A., & Yetişgen, M. (2024) · Cognitive informatics in biomedicine and healthcare · doi:10.1007/978-3-031-55865-8_7 · Cited by 1 · see citing works
Self-citationEvaluating NLP Models for Mental Health
Menon, V. A., Kumar, T. K. S., Thomas, L. P., & Thomas, J. (2025) · IGI Global eBooks · doi:10.4018/979-8-3693-6910-4.ch004
Lee, R. Y., Li, K. S., Sibley, J., Cohen, T., Lober, W. B., O’Brien, J., LeDuc, N., Andrews, K. M., Ungar, A., Walsh, J., et al. (2026) · Clinical Trials · doi:10.1177/17407745251405386
Self-citationGazquez-Garcia, J., Martínez-Lentisco, M. D. M., Martinez-Castro, S., Sánchez-Laguna, F. J., & Sánchez-Bocanegra, C. L. (2026) · doi:10.2196/preprints.114843
- Authors
- Uyeda, A. M., Curtis, J. R., Engelberg, R. A., Brumback, L. C., Guo, Y., Sibley, J., Lober, W. B., Cohen, T., Torrence, J., Heywood, J., Paul, S. R., Kross, E. K., & Lee, R. Y.
- Affiliations
- University of Washington (US)
- Corresponding author
- Curtis, J. Randall
- ORCID
- Venue
- Journal of Pain and Symptom Management
- Publisher
- Elsevier BV
- Published
- 2022-06
- Volume
- 63
- Issue
- 6
- Pages
- e713-e723
- ISSN
- 0885-3924 (ISSN-L)
- PMC
- PMC9124686
- OpenAlex
- W4212982930
- Cited by
- 19
- References
- 66
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 2 · 2023: 3 · 2022: 4
- FWCI
- 1.32
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Pain and Symptom Management (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Palliative Care Research Center
- National Institutes of Health (R01 AG062441, K12 HL137940)
- Institute of Translational Health Sciences, University of Washington
- Institute of Translational Health Sciences
- Cambia Health Foundation
- National Center for Advancing Translational Sciences (UL1 TR002319)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Palliative Care and End-of-Life Issues
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- goals of care discussions, natural language processing, electronic health records, serious illness, bag-of-words, quality metrics, misclassification analysis
- MeSH terms
- Goals, Humans, Natural Language Processing, Prospective Studies, Cohort Studies, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
