Performance of predefined search patterns for identifying documented goals-of-care discussions in inpatient electronic Health records
Open accessAbstract
Objective:To characterize the sensitivity and predictive value of predefined search terms for identifying documented goals-of-care discussions in health records of hospitalized patients with serious illness. Methods:We evaluated the performance of 30 previously published and investigator-defined search terms codified into regular expressions (a type of pattern-based text search) in detecting goals-of-care documentation in a 2974-note corpus of electronic health record notes belonging to 159 inpatients enrolled in a U.S. clinical trial over 2020–2021. Results:Compared to conventional chart abstraction, search terms for “goals of care” and synonyms such as “GOC” had poor sensitivity (range: 29.5–38.3%) and modest positive predictive value (PPV; range: 48.3–61.7%) for identifying notes with goals-of-care documentation. Combinations of search terms demonstrated modest performance (sensitivity 62.0%, PPV 59.4%, F1 0.61) but fell short of more complex natural language processing models. Conclusion:In certain contexts, predefined regular-expression-based search terms may have suboptimal sensitivity and predictive value for identifying documented goals-of-care discussions.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Goals-of-care discussions are difficult to measure using rule-based methods.
“This is especially important for linguistically complex outcomes such as goals-of-care discussions, which are difficult to measure using rule-based methods.12”
From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [12]
Goals-of-care discussions are frequently documented in unstructured text such as progress notes or other inconspicuous locations, making historical discussions difficult for clinicians to unearth during clinical care.
“Additionally, because goals-of-care discussions are frequently documented in unstructured text such as progress notes or other inconspicuous locations, documentation of historical discussions can be difficult for clinicians to unearth during clinical care.12–14”
From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [12]
Minor differences in operational definitions (e.g., the inclusion vs. exclusion of routine code status discussions) can lead to important differences in the choice of best measurement tools.
“Differences in the performance of rule-based methods across these studies12,20 illustrates how minor differences in operational definitions (e.g., the inclusion vs. exclusion of routine code status discussions) can lead to important differences in the choice of best measurement tools.”
From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [12]
Cites
Explainability for artificial intelligence in healthcare: a multidisciplinary perspective
the Precise4Q consortium, Amann, J., Blasimme, A., Vayena, E., Frey, D., & Madai, V. I. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01332-6 · Cited by 2,203 · see citing works
Can we open the black box of AI?
Castelvecchi, D. (2016) · Nature · doi:10.1038/538020a · Cited by 1,773 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · see citing works
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552 · see citing works
Communication About Serious Illness Care Goals: A Review and Synthesis of Best Practices
Bernacki, R. E., & Block, S. D. (2014) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2014.5271 · Cited by 1,422 · see citing works
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · see citing works
Savova, G. K., Kipper-Schuler, K. C., Hurdle, J. F., & Meystre, S. M. (2008) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638592 · Cited by 895 · see citing works
The unreasonable effectiveness of deep learning in artificial intelligence
Sejnowski, T. J. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907373117 · Cited by 352 · see citing works
Curtis, J. R., Downey, L., Back, A. L., Nielsen, E. L., Paul, S. R., Lahdya, A. Z., Treece, P. D., Armstrong, P., Peck, R., & Engelberg, R. A. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.2317 · Cited by 265 · see citing works
Self-citationWhen to consult precision-recall curves
Cook, J. A., & Ramadas, V. (2020) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x20909693 · Cited by 137 · see citing works
Secunda, K. E., Wirpsa, M. J., Neely, K. J., Szmuilowicz, E., Wood, G., Panozzo, E., McGrath, J., Levenson, A. R., Peterson, J., Gordon, E. J., et al. (2019) · Journal of General Internal Medicine · doi:10.1007/s11606-019-05446-0 · Cited by 134 · see citing works
Lindvall, C. J., Lilley, E. J., Zupanc, S. N., Chien, I., Udelsman, B. V., Walling, A. M., Cooper, Z. R., & Tulsky, J. A. (2018) · Journal of Palliative Medicine · doi:10.1089/jpm.2018.0326 · Cited by 100 · see citing works
Turchin, A., Kolatkar, N. S., Grant, R. W., Makhni, E. C., Pendergrass, M. L., & Einbinder, J. S. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2078 · Cited by 98 · see citing works
Lindvall, C., Deng, C.-Y., Moseley, E., Agaronnik, N., El-Jawahri, A., Paasche-Orlow, M. K., Lakin, J. R., Volandes, A., & Tulsky, T. A.-P. I. (2022) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2021.06.025 · Cited by 91 · see citing works
Curtis, J. R., Lee, R. Y., Brumback, L. C., Kross, E. K., Downey, L., Torrence, J., LeDuc, N., Mallon Andrews, K., Im, J., Heywood, J., et al. (2023) · JAMA · doi:10.1001/jama.2023.8812 · Cited by 91 · see citing works
Self-citationLee, R. Y., Brumback, L. C., Lober, W. B., Sibley, J., Nielsen, E. L., Treece, P. D., Kross, E. K., Loggers, E. T., Fausto, J. A., Lindvall, C. J., et al. (2020) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2020.08.024 · Cited by 81 · see citing works
Self-citationLee, R. Y., Kross, E. K., Torrence, J., Li, K. S., Sibley, J., Cohen, T., Lober, W. B., Engelberg, R. A., & Curtis, J. R. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.1204 · Cited by 74 · see citing works
Self-citationMeasuring Processes of Care in Palliative Surgery
Lilley, E. J., Lindvall, C. J., Lillemoe, K. D., Tulsky, J. A., Wiener, D. C., & Cooper, Z. R. (2017) · Annals of Surgery · doi:10.1097/sla.0000000000002579 · Cited by 54 · see citing works
Lee, R. Y., Kross, E. K., Downey, L., Paul, S. R., Heywood, J., Nielsen, E. L., Okimoto, K., Brumback, L. C., Merel, S. E., Engelberg, R. A., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.5088 · Cited by 44 · see citing works
Self-citationUyeda, A. M., Curtis, J. R., Engelberg, R. A., Brumback, L. C., Guo, Y., Sibley, J., Lober, W. B., Cohen, T., Torrence, J., Heywood, J., et al. (2022) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2022.02.006 · Cited by 19 · see citing works
Self-citationAlbashayreh, A., Bandyopadhyay, A., Zeinali, N., Zhang, M., Fan, W., & Gilbertson‐White, S. (2024) · JCO Clinical Cancer Informatics · doi:10.1200/cci.23.00235 · Cited by 18 · see citing works
Curtis, J. R., Lee, R. Y., Brumback, L. C., Kross, E. K., Downey, L., Torrence, J., Heywood, J., LeDuc, N., Mallon Andrews, K., Im, J., et al. (2022) · Contemporary Clinical Trials · doi:10.1016/j.cct.2022.106879 · Cited by 12 · see citing works
Self-citationForms or Free-Text? Measuring Advance Care Planning Activity Using Electronic Health Records
Zupanc, S. N., Lakin, J. R., Volandes, A. E., Paasche‐Orlow, M. K., Moseley, E. T., Gundersen, D. A., Das, S., Penumarthy, A., Martins-Welch, D., Burns, E. A., et al. (2023) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2023.07.016 · Cited by 12 · see citing works
Xu, W. V., Gu, B., Lotter, W. E., & Kehl, K. L. (2024) · JCO Clinical Cancer Informatics · doi:10.1200/cci.23.00269 · Cited by 10 · see citing works
Chien, I., Shi, A., Chan, A., & Lindvall, C. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-12738-1_15 · Cited by 9 · see citing works
Chatham, A. H., Bradley, E. D., Schirle, L. M., Sanchez‐Roige, S., Samuels, D. C., & Jeffery, A. D. (2023) · medRxiv · doi:10.1101/2023.06.08.23290894 · Cited by 5 · see citing works
Limsomwong, P., Ingviya, T., & Fumaneeshoat, O. (2024) · BMC Palliative Care · doi:10.1186/s12904-024-01419-1 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions
Lee, R. Y., Li, K. S., Sibley, J., Cohen, T., Lober, W. B., Dotolo, D. G., & Kross, E. K. (2026) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2025.09.025 · Cited by 5 · see citing works
Self-citationLee, R. Y., Li, K. S., Sibley, J., Cohen, T., Lober, W. B., O’Brien, J., LeDuc, N., Andrews, K. M., Ungar, A., Walsh, J., et al. (2026) · Clinical Trials · doi:10.1177/17407745251405386
Self-citationLaidley, J., Zverev, S. R., Kwok, A., Sciacca, K. R., Ljungberg, B. F., Samineni, S., Coombs, L. A., Tulsky, J. A., Lindvall, C. J., & Gray, T. F. (2026) · Journal of Palliative Medicine · doi:10.1177/10966218261446776
- Authors
- Lee, R. Y., Uyeda, A. M., Li, K. S., Sibley, J., Cohen, T., Lober, W. B., Dotolo, D. G., Engelberg, R. A., & Kross, E. K.
- Affiliations
- University of Washington (US)
- Corresponding author
- Lee, Robert Y.
- ORCID
- Venue
- Journal of Palliative Medicine
- Publisher
- SAGE Publications
- Published
- 2025-07-01
- Volume
- 28
- Issue
- 7
- Pages
- 952-959
- ISSN
- 1096-6218 (ISSN-L)
- 1557-7740
- OpenAlex
- W4409171387
- Cited by
- 3
- References
- 26
- Citations by year
- 2026: 2 · 2025: 1
- FWCI
- 2.72
- Type
- journal article
- Language
- en
- Open access
- Locations
- Journal of Palliative Medicine (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion)
- Funders
- National Heart, Lung, and Blood Institute (T32HL125195, K23HL161503)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Palliative Care and End-of-Life Issues
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Nursing › Issues, ethics and legal aspects › Nursing Diagnosis and Documentation
- Keywords
- goals of care discussions, electronic health records, text mining, medical records review, hospitalized patients, serious illness, sensitivity and predictive value, palliative care
- MeSH terms
- Adult, Aged, Aged, 80 and over, Documentation, Female, Humans, Inpatients, Male, Middle Aged, Natural Language Processing, Patient Care Planning, United States, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review