CoChat
Sign Up
Cited by 74Machine Learning in HealthcareElectronic Health Records SystemsNursing Diagnosis and Documentation

Assessment of natural language processing of electronic health records to measure goals-of-care discussions as a clinical trial outcome

Open access
Lee, Robert Y., Kross, Erin K., Torrence, Janaki, Li, Kevin S., Sibley, James, Cohen, Trevor, Lober, William B., Engelberg, Ruth A. +1 more·2023·JAMA Network Open
DOI 10.1001/jamanetworkopen.2023.1204PMC PMC9982698
Open this paper in CoChat — free accountOpen PDF

Abstract

Importance: Many clinical trial outcomes are documented in free-text electronic health records (EHRs), making manual data collection costly and infeasible at scale. Natural language processing (NLP) is a promising approach for measuring such outcomes efficiently, but ignoring NLP-related misclassification may lead to underpowered studies. Objective: To evaluate the performance, feasibility, and power implications of using NLP to measure the primary outcome of EHR-documented goals-of-care discussions in a pragmatic randomized clinical trial of a communication intervention. Design, Setting, and Participants: This diagnostic study compared the performance, feasibility, and power implications of measuring EHR-documented goals-of-care discussions using 3 approaches: (1) deep-learning NLP, (2) NLP-screened human abstraction (manual verification of NLP-positive records), and (3) conventional manual abstraction. The study included hospitalized patients aged 55 years or older with serious illness enrolled between April 23, 2020, and March 26, 2021, in a pragmatic randomized clinical trial of a communication intervention in a multihospital US academic health system. Main Outcomes and Measures: Main outcomes were natural language processing performance characteristics, human abstractor-hours, and misclassification-adjusted statistical power of methods of measuring clinician-documented goals-of-care discussions. Performance of NLP was evaluated with receiver operating characteristic (ROC) curves and precision-recall (PR) analyses and examined the effects of misclassification on power using mathematical substitution and Monte Carlo simulation. Results: A total of 2512 trial participants (mean [SD] age, 71.7 [10.8] years; 1456 [58%] female) amassed 44 324 clinical notes during 30-day follow-up. In a validation sample of 159 participants, deep-learning NLP trained on a separate training data set identified patients with documented goals-of-care discussions with moderate accuracy (maximal F1 score, 0.82; area under the ROC curve, 0.924; area under the PR curve, 0.879). Manual abstraction of the outcome from the trial data set would require an estimated 2000 abstractor-hours and would power the trial to detect a risk difference of 5.4% (assuming 33.5% control-arm prevalence, 80% power, and 2-sided α = .05). Measuring the outcome by NLP alone would power the trial to detect a risk difference of 7.6%. Measuring the outcome by NLP-screened human abstraction would require 34.3 abstractor-hours to achieve estimated sensitivity of 92.6% and would power the trial to detect a risk difference of 5.7%. Monte Carlo simulations corroborated misclassification-adjusted power calculations. Conclusions and Relevance: In this diagnostic study, deep-learning NLP and NLP-screened human abstraction had favorable characteristics for measuring an EHR outcome at scale. Adjusted power calculations accurately quantified power loss from NLP-related misclassification, suggesting that incorporation of this approach into the design of studies using NLP would be beneficial.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1001/jamanetworkopen.2023.1204

Cites

62 of 62 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The training data for the referent BERT model was acquired over 287 human abstractor-hours across a 4-month period, a ~$20,000 investment to measure a single linguistic construct, a cost that must be repeated for any related construct or different operational definition of the same construct.

    “The training data alone for the referent BERT model was acquired over the course of 287 human abstractor-hours over a 4-month period—a ~$20,000 investment to measure a single linguistic construct that must be repeated for any related constructs of interest, or even for different operational definitions of the same construct.19”

    From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [19]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Evaluation of the selected BERT screening threshold in the Trial 1 test set yielded 99.3% note-level sensitivity (95%CI 97.6%, 99.9%) and 100% patient-level sensitivity (one-sided 97.5%CI 97.7%, 100%).

    “Evaluation of the selected BERT screening threshold in the Trial 1 test set yielded 99.3% note-level sensitivity (95%CI 97.6%, 99.9%) and 100% patient-level sensitivity (one-sided 97.5%CI 97.7%, 100%).19”

    From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [19]

    The Results (Comparison of Outcome Measurement Strategies; page unavailable) confirm that the authors selected a BERT screening threshold, but report an estimated patient-level sensitivity of 92.6% for it. The Methods and validation Results describe a Trial 1 validation sample, not the claimed test-set evaluation; our copy does not report the stated 99.3% and 100% sensitivities or their confidence intervals. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • To date, use of NLP to identify goals-of-care discussions has facilitated at least two clinical trials.

    “To date, the use of NLP to identify this linguistically complex outcome has facilitated at least two clinical trials;19–22 and, while current clinical EHRs rely on either search-based approaches or the use of semi-structured documentation,12,23 real-time ability for clinicians to efficiently locate goals-of-care documentation seems likely to improve delivery of goal-concordant care.24”

    From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [19]

    The Introduction and Conclusions (pages unavailable in our copy) say NLP was used to measure documented goals-of-care discussions as the primary outcome of one pragmatic clinical trial. The Methods describe a previous pilot trial as the source of manually reviewed notes used to train the NLP model, not as a second trial facilitated by NLP. The Discussion mentions NLP use for similar palliative-care outcomes but does not establish a second trial identifying goals-of-care discussions. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Acquiring task-specific training and validation data for the previous BERT model required hundreds of human abstractor-hours.

    “However, acquiring the task-specific training and validation data for this model required hundreds of human abstractor-hours,19 limiting our ability to adapt this model toward measuring other constructs or conducting larger trials.”

    From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [19]

  • Ground truth for BERT passage- and note-level presence or absence of EHR-documented goals-of-care discussions was determined for all notes in the pilot set and Trial 1 test set using manual human chart abstraction.

    “Ground-truth for BERT passage- and note-level presence or absence of EHR-documented goals-of-care discussions was determined for all notes in the pilot set and Trial 1 test set using manual human chart abstraction, as previously described.19”

    From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [19]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. [7]Chien I, Shi A, Chan A, Lindvall C. Identification of serious illness conversations in unstructured clinical notes using deep neural networks. In: Koch F, Koster A, Bichindaritz I, et al, eds. Artificial Intelligence in Health: First International Workshop, AIH 2018, Stockholm, Sweden, July 13-14, 2018, Revised Selected Papers. Springer Nature Switzerland; 2019:199-212.
  2. [37]Wennberg JE, Fisher ES, Goodman DC, Skinner JS. Tracking the Care of Patients With Severe Chronic Illness: The Dartmouth Atlas of Health Care 2008. The Dartmouth Institute for Health Policy and Clinical Practice; 2008. 36378804
  3. [38]Goodman DC, Esty AR, Fisher ES, Chang CH. Trends and Variation in End-of-life Care for Medicare Beneficiaries With Severe Chronic Illness: A Report of the Dartmouth Atlas Project. The Dartmouth Institute for Health Policy and Clinical Practice; April 12, 2011. 36516285

Cited by

Sorted by citations

Related

Authors
Lee, R. Y., Kross, E. K., Torrence, J., Li, K. S., Sibley, J., Cohen, T., Lober, W. B., Engelberg, R. A., & Curtis, J. R.
Affiliations
University of Washington (US)
Corresponding author
Lee, Robert Y.
ORCID
Venue
JAMA Network Open
Publisher
American Medical Association (AMA)
Published
2023-03-02
Volume
6
Issue
3
Pages
e231204
ISSN
2574-3805 (ISSN-L)
DOI
10.1001/jamanetworkopen.2023.1204
PMC
PMC9982698
OpenAlex
W4322757661
Cited by
74
References
62
Citations by year
2026: 19 · 2025: 26 · 2024: 27 · 2023: 2
FWCI
6.96
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JAMA Network Open (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • University of Washington (UL1TR002319.)
  • Institute of Translational Health Sciences (UL1TR002319)
  • National Palliative Care Research Center
  • Cambia Health Foundation
  • National Institutes of Health (UL1TR002319)
  • National Institute on Aging (R01AG062441)
  • National Heart, Lung, and Blood Institute (K12HL137940, K23HL161503)
  • National Center for Advancing Translational Sciences
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Nursing › Issues, ethics and legal aspects › Nursing Diagnosis and Documentation
Keywords
goals of care discussions, natural language processing, electronic health records, statistical power, misclassification, deep learning, communication intervention, serious illness, hospitalized older adults, Monte Carlo simulation, trial feasibility, clinician documentation, shared decision-making, palliative care communication
MeSH terms
Deep Learning, Aged, Clinical Trials as Topic, Computer Simulation, Data Collection, Feasibility Studies, Female, Hospitalization, Humans, Male, Middle Aged, Natural Language Processing, Patient Care Planning, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review