Natural language processing to identify advance care planning documentation in a multisite pragmatic clinical trial
Open accessAbstract
Context Large multisite clinical trials studying decision-making when facing serious illness require an efficient method for abstraction of advance care planning (ACP) documentation from clinical text documents. However, the current gold standard method of manual chart review is time-consuming and unreliable.Objectives To evaluate the ability to use natural language processing (NLP) to identify ACP documention in clinical notes from patients participating in a multisite trial.Methods Patients with advanced cancer followed in three disease-focused oncology clinics at Duke Health, Mayo Clinic, and Northwell Health were identified using administrative data. All outpatient and inpatient notes from patients meeting inclusion criteria were extracted from electronic health records (EHRs) between March 2018 and March 2019. NLP text identification software with semi-automated chart review was applied to identify documentation of four ACP domains: (1) conversations about goals of care, (2) limitation of life-sustaining treatment, (3) involvement of palliative care, and (4) discussion of hospice. The performance of NLP was compared to gold standard manual chart review.Results 435 unique patients with 79,797 notes were included in the study. In our validation data set, NLP achieved F1 scores ranging from 0.84 to 0.97 across domains compared to gold standard manual chart review. NLP identified ACP documentation in a fraction of the time required by manual chart review of EHRs (1-5 minutes per patient for NLP, vs. 30-120 minutes for manual abstraction).Conclusion NLP is more efficient and as accurate as manual chart review for identifying ACP documentation in studies with large patient cohorts.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Other operational definitions of the goals-of-care construct are more amenable to measurement using rule-based and other methods.
“Even within the community of investigators interested in measuring goals-of-care content, other operational definitions of the same construct are more amenable to measurement using rule-based and other methods.15,16,20”
From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [20]
The Methods (page unavailable) describe a goals-of-care conversation domain identified with a keyword library and semi-automated chart review, supporting the claim that an operational definition can be measured this way. The paper does not establish that this definition is *more* amenable to measurement than the other definitions implied by the citing sentence. (LLM, conf=0.82)
Experimental — accuracy not yet measuredTo date, use of NLP to identify goals-of-care discussions has facilitated at least two clinical trials.
“To date, the use of NLP to identify this linguistically complex outcome has facilitated at least two clinical trials;19–22 and, while current clinical EHRs rely on either search-based approaches or the use of semi-structured documentation,12,23 real-time ability for clinicians to efficiently locate goals-of-care documentation seems likely to improve delivery of goal-concordant care.24”
From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [20]
The Abstract Methods and Introduction describe NLP identifying goals-of-care documentation among patients in a multisite pragmatic clinical trial. This paper supports one trial as an example, but our copy does not establish the claim’s count of at least two clinical trials. (LLM, conf=0.85)
Experimental — accuracy not yet measuredMinor differences in operational definitions (e.g., the inclusion vs. exclusion of routine code status discussions) can lead to important differences in the choice of best measurement tools.
“Differences in the performance of rule-based methods across these studies12,20 illustrates how minor differences in operational definitions (e.g., the inclusion vs. exclusion of routine code status discussions) can lead to important differences in the choice of best measurement tools.”
From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 35 · Cited by 91
- Authors
- Lindvall, C., Deng, C.-Y., Moseley, E., Agaronnik, N., El-Jawahri, A., Paasche-Orlow, M. K., Lakin, J. R., Volandes, A., & Tulsky, T. A.-P. I.
- Affiliations
- Brigham and Women's Hospital (US)
- Dana-Farber Cancer Institute (US)
- Harvard University (US)
- Corresponding author
- Lindvall, Charlotta
- ORCID
- Venue
- Journal of Pain and Symptom Management
- Publisher
- Elsevier BV
- Published
- 2022-01
- Volume
- 63
- Issue
- 1
- Pages
- e29-e36
- ISSN
- 0885-3924 (ISSN-L)
- PMC
- PMC9124370
- OpenAlex
- W3179277507
- Cited by
- 91
- References
- 35
- Citations by year
- 2026: 19 · 2025: 21 · 2024: 23 · 2023: 14 · 2022: 13 · 2021: 1
- FWCI
- 11.05
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Pain and Symptom Management (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute on Aging (U24AT009676)
- National Institutes of Health (UH3AG060626)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Palliative Care and End-of-Life Issues
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Keywords
- advance care planning, natural language processing, electronic health records, manual chart review, goals of care, life-sustaining treatment, palliative care, hospice, advanced cancer, end-of-life decisions, pragmatic trials
- MeSH terms
- Documentation, Humans, Natural Language Processing, Palliative Care, Advance Care Planning, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
