Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.

Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models

Open access

Published version of a preprint

Abstract

Many areas of medicine would benefit from deeper, more accurate phenotyping, but there are limited approaches for phenotyping using clinical notes without substantial annotated data. Large language models (LLMs) have demonstrated immense potential to adapt to novel tasks with no additional training by specifying task-specific instructions. Here we report the performance of a publicly available LLM, Flan-T5, in phenotyping patients with postpartum hemorrhage (PPH) using discharge notes from electronic health records ( n = 271,081). The language model achieves strong performance in extracting 24 granular concepts associated with PPH. Identifying these granular concepts accurately allows the development of interpretable, complex phenotypes and subtypes. The Flan-T5 model achieves high fidelity in phenotyping PPH (positive predictive value of 0.95), identifying 47% more patients with this complication compared to the current standard of using claims codes. This LLM pipeline can be used reliably for subtyping PPH and outperforms a claims-based approach on the three most common PPH subtypes associated with uterine atony, abnormal placentation, and obstetric trauma. The advantage of this approach to subtyping is its interpretability, as each concept contributing to the subtype determination can be evaluated. Moreover, as definitions may change over time due to new guidelines, using granular concepts to create complex phenotypes enables prompt and efficient updating of the algorithm. Using this language modelling approach enables rapid phenotyping without the need for any manually annotated training data across multiple clinical use cases.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-023-00957-x

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Obstetric delivery cohort. a Inclusion criteria for identifying delivery-related discharge summaries. The final cohort consists of 271,081 discharge notes for 131,284 women with an obstetric encounter at Mass General Brigham hospitals from 1998–2015. b–e Characteristics of the cohort based on (b) delivery hospital and (c) patient age at delivery, (d) patient race, and (e) patient ethnicity. ER emergency room, MGH Massachusetts General Hospital, BWH Brigham and Women’s Hospital, NSM North Shore Medical Center, NWH Newton Wellesley Hospital.
Fig. 1 Obstetric delivery cohort. a Inclusion criteria for identifying delivery-related discharge summaries. The final cohort consists of 271,081 discharge notes for 131,284 women with an obstetric encounter at Mass General Brigham hospitals from 1998–2015. b–e Characteristics of the cohort based on (b) delivery hospital and (c) patient age at delivery, (d) patient race, and (e) patient ethnicity. ER emergency room, MGH Massachusetts General Hospital, BWH Brigham and Women’s Hospital, NSM North Shore Medical Center, NWH Newton Wellesley Hospital.CC BY · PubMed Central
Fig. 2 Overview of zero-shot NLP pipeline for accurate and interpretable postpartum hemorrhage (PPH) phenotyping. a Zero-shot extraction of PPH concepts using Flan-T5. We constructed either yes/no or extraction prompts for each PPH concept, concatenated the discharge summary of interest, and fed the combined input into Flan-T5. This process enabled the rapid extraction of PPH-related information from notes with no training data. b We leveraged the extracted concepts to perform interpretable phenotyping of PPH defined as cesarean deliveries with at least 1L or vaginal deliveries with at least 500mL of estimated blood loss. c We used the extracted concepts to perform interpretable subtyping of PPH based on the underlying etiology. We assigned a delivery note to the “tone", “tissue", “trauma", or “thrombin" subtype if any of the concepts associated with the subtype are present. PPH postpartum hemorrhage, EBL estimated blood loss, image: Flaticon.com.
Fig. 2 Overview of zero-shot NLP pipeline for accurate and interpretable postpartum hemorrhage (PPH) phenotyping. a Zero-shot extraction of PPH concepts using Flan-T5. We constructed either yes/no or extraction prompts for each PPH concept, concatenated the discharge summary of interest, and fed the combined input into Flan-T5. This process enabled the rapid extraction of PPH-related information from notes with no training data. b We leveraged the extracted concepts to perform interpretable phenotyping of PPH defined as cesarean deliveries with at least 1L or vaginal deliveries with at least 500mL of estimated blood loss. c We used the extracted concepts to perform interpretable subtyping of PPH based on the underlying etiology. We assigned a delivery note to the “tone", “tissue", “trauma", or “thrombin" subtype if any of the concepts associated with the subtype are present. PPH postpartum hemorrhage, EBL estimated blood loss, image: Flaticon.com.CC BY · PubMed Central
Fig. 3 Comparison of the Flan-T5 language model and regular expression approaches for zero-shot extraction of postpartum hemorrhage (PPH)-related concepts. The prevalence of each concept in the annotated test set is reported and compared to the model performance according to binary F1 score. The stars (*) denote that there is a significant difference (p < 0.05, McNemar test) between the regex and language model performance. PPH postpartum hemorrhage.
Fig. 3 Comparison of the Flan-T5 language model and regular expression approaches for zero-shot extraction of postpartum hemorrhage (PPH)-related concepts. The prevalence of each concept in the annotated test set is reported and compared to the model performance according to binary F1 score. The stars (*) denote that there is a significant difference (p < 0.05, McNemar test) between the regex and language model performance. PPH postpartum hemorrhage.CC BY · PubMed Central
Fig. 4 Prevalence of postpartum hemorrhage (PPH) international classification of diseases (ICD) codes in notes with varying estimated blood loss (EBL). The y-axis depicts the range of estimated blood loss (EBL) values extracted from the delivery note by Flan-T5 (if any), and the x-axis denotes the proportion of notes in each EBL category with a postpartum hemorrhage ICD diagnostic code. PPH ICD codes are defined according to the definition in Zheutlin et al.8. Refer to Supplementary Fig. 2 for a similar plot using the PPH ICD definition from Butwick et al.7. PPH postpartum hemorrhage, ICD international classification of diseases, EBL estimated blood loss.
Fig. 4 Prevalence of postpartum hemorrhage (PPH) international classification of diseases (ICD) codes in notes with varying estimated blood loss (EBL). The y-axis depicts the range of estimated blood loss (EBL) values extracted from the delivery note by Flan-T5 (if any), and the x-axis denotes the proportion of notes in each EBL category with a postpartum hemorrhage ICD diagnostic code. PPH ICD codes are defined according to the definition in Zheutlin et al.8. Refer to Supplementary Fig. 2 for a similar plot using the PPH ICD definition from Butwick et al.7. PPH postpartum hemorrhage, ICD international classification of diseases, EBL estimated blood loss.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 35 references

Research works printed but not linked: 2

  1. Aramaki, E., Imai, T., Miyo, K. & Ohe, K. Patient status classification by using rule based sentence extraction and bm25 knn-based classifier. In i2b2 Workshop on Challenges in Natural Language Processing for Clinical Data (2006).
  2. Lehman, E. et al. Do we still need clinical language models? In: Proceedings of the Conference on Health, Inference, and Learning, PMLR 209, 578–597, (2023).

Cited by

See all 70

Metadata

Authors
Alsentzer, E., Rasmussen, M. J., Fontoura, R., Cull, A. L., Beaulieu-Jones, B., Gray, K. J., Bates, D. W., & Kovacheva, V. P.
Affiliations
Brigham and Women's Hospital (US)
Corresponding author
Kovacheva, Vesela P.
Venue
npj Digital Medicine
Published
2023-11-30
Volume
6
Issue
1
Pages
212-212
ISSN
2398-6352 (ISSN-L)
OpenAlex
W4389205282
Cited by
69
References
35
Citations by year
2026: 16 · 2025: 28 · 2024: 24 · 2023: 1
FWCI
6.50
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • medRxiv (acceptedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion)
  • Knowledge@UChicago (University of Chicago) (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • University of Chicago (cc-by)
Funders
  • Anesthesia Patient Safety Foundation
  • Brigham and Women’s Hospital (IGNITE)
  • U.S. Department of Health & Human Services | NIH | National Heart, Lung, and Blood Institute (K08HL161326-01A1, K08 HL146963-02, K08 HL146963-02S1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
postpartum hemorrhage, phenotyping, large language models, Flan-T5, discharge notes, electronic health records, uterine atony, abnormal placentation, obstetric trauma, zero-shot learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models. https://app.cochat.ai/papers/10.1038/s41746-023-00957-x
The paper
Alsentzer, E., Rasmussen, M. J., Fontoura, R., Cull, A. L., Beaulieu-Jones, B., Gray, K. J., Bates, D. W., & Kovacheva, V. P. (2023). Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models. npj Digital Medicine. https://doi.org/10.1038/s41746-023-00957-x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.