Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index; 38 of the 67 sources its claims cite were checked for support.

Large language models to identify social determinants of health in electronic health records

Open access

Abstract

Social determinants of health (SDoH) play a critical role in patient outcomes, yet their documentation is often missing or incomplete in the structured data of electronic health records (EHRs). Large language models (LLMs) could enable high-throughput extraction of SDoH from the EHR to support research and clinical care. However, class imbalance and data limitations present challenges for this sparsely documented yet critical information. Here, we investigated the optimal methods for using LLMs to extract six SDoH categories from narrative text in the EHR: employment, housing, transportation, parental status, relationship, and social support. The best-performing models were fine-tuned Flan-T5 XL for any SDoH mentions (macro-F1 0.71), and Flan-T5 XXL for adverse SDoH mentions (macro-F1 0.70). Adding LLM-generated synthetic data to training varied across models and architecture, but improved the performance of smaller Flan-T5 models (delta F1 + 0.12 to +0.23). Our best-fine-tuned models outperformed zero- and few-shot performance of ChatGPT-family models in the zero- and few-shot setting, except GPT4 with 10-shot prompting for adverse SDoH. Fine-tuned models were less likely than ChatGPT to change their prediction when race/ethnicity and gender descriptors were added to the text, suggesting less algorithmic bias ( p < 0.05). Our models identified 93.8% of patients with adverse SDoH, while ICD-10 codes captured 2.0%. These results demonstrate the potential of LLMs in improving real-world evidence on SDoH and assisting in identifying patients who could benefit from resource support.

Overview

CoChat Summary

TL;DR

This study evaluated large language models for extracting six social determinants of health (SDoH) categories from EHR narrative text, including fine-tuned Flan-T5/BERT models, synthetic data augmentation, and ChatGPT-family zero/few-shot prompting. Fine-tuned models achieved macro-F1 around 0.70 on in-domain tasks, generally outperformed ChatGPT-family models, were less sensitive to demographic descriptors, and identified far more patients with adverse SDoH than structured ICD-10 Z-codes.

Key findings

  • Best in-domain performance was around macro-F1 0.70: the abstract reports Flan-T5 XL for any SDoH mentions (macro-F1 0.71) and Flan-T5 XXL for adverse SDoH mentions (macro-F1 0.70); the Results text/table report Flan-T5 XXL with synthetic data for any SDoH (macro-F1 0.70) and Flan-T5 XL without synthetic data for adverse SDoH (macro-F1 0.70).
  • Adding LLM-generated synthetic data improved smaller Flan-T5 models (abstract: delta F1 +0.12 to +0.23); ablation studies showed performance was maintained until ~50% of gold data were removed when synthetic data were included, versus drops after ~10–20% without synthetic data.
  • Fine-tuned models outperformed zero- and few-shot ChatGPT-family models for most SDoH classes, except GPT4 with 10-shot prompting for adverse SDoH (Table 3: fine-tuned Flan-T5 XL 0.86 vs GPT4 10-shot 0.90); on any SDoH, fine-tuned Flan-T5 XXL scored 0.92 vs GPT4 10-shot 0.88.
  • Fine-tuned models changed predictions less often than ChatGPT when race/ethnicity and gender descriptors were injected: 14.3% vs 21.5% of sentence pairs for any SDoH (P = 0.007) and 9.9% vs 18.2% for adverse SDoH (P = 0.005). Models identified 93.8% (45/48) of patients with adverse SDoH, while ICD-10 Z-codes captured 2.0% (1/48).

Methods

  • Sentence-level multilabel classification for six SDoH categories (employment, housing, transportation, parental status, relationship, social support) and two tasks (any SDoH mentions, adverse SDoH mentions). Primary radiotherapy corpus: 800 clinic notes from 770 patients with cancer receiving radiotherapy at Brigham and Women’s Hospital/Dana-Farber Cancer Institute, 2015–2022; out-of-domain datasets: immunotherapy (200 notes, 170 patients) and MIMIC-III (200 notes, 183 patients). Annotation used MAE; 300/800 notes were dual annotated, with Krippendorff’s alpha 0.86 for any SDoH and 0.76 for adverse SDoH.
  • Models included BERT-base baseline and Flan-T5 base/large/XL/XXL, with LoRA for XL/XXL; synthetic data were generated via GPT3.5 prompting rounds. ChatGPT-family models were evaluated zero- and few-shot on a manually validated synthetic test set; bias was assessed with demographic-injected synthetic sentence pairs. Analysis used bootstrap macro-F1 confidence intervals, Mann–Whitney U tests, and chi-squared/2-proportion z tests.

Limitations

  • Stated: training and out-of-domain datasets came from a predominantly white population treated at Boston hospitals, limiting generalizability; synthetic generation methods were not exhaustively assessed; ChatGPT evaluations were limited to manually verified synthetic sentences rather than protected health information, so performance may not reflect real clinical text; models are only as good as the annotated corpus, and SDoH annotation is challenging, especially for Support, and may be subject to annotator bias.
  • Evident from the text: severe class imbalance (only 3% of training sentences contained any SDoH mention; housing had 20 instances in the training set) and sparse labels likely constrain performance. The ChatGPT comparison used synthetic rather than real clinical text, and the text contains minor inconsistencies between the abstract and Results/Table 1 for best model/task and macro-F1 values.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-023-00970-0

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Ablation studies. Performance in Macro-F1 of Flan-T5 XL models fine-tuned using gold data only (orange line) and gold and synthetic data (green line), as gold-labeled sentences are gradually reduced by undersample value from the training dataset for the a adverse social determinant of health (SDoH) mention task and b any SDoH mention task. The full gold-labeled training set is comprised of 29,869 sentences, augmented with 1800 synthetic SDoH sentences, and tested on the in-domain RT test dataset. SDoH Social determinants of health.
Fig. 1 Ablation studies. Performance in Macro-F1 of Flan-T5 XL models fine-tuned using gold data only (orange line) and gold and synthetic data (green line), as gold-labeled sentences are gradually reduced by undersample value from the training dataset for the a adverse social determinant of health (SDoH) mention task and b any SDoH mention task. The full gold-labeled training set is comprised of 29,869 sentences, augmented with 1800 synthetic SDoH sentences, and tested on the in-domain RT test dataset. SDoH Social determinants of health.CC BY · PubMed Central
Fig. 2 Fine-tuned LLMs versus ChatGPT-family models. Comparison of model performance between our fine-tuned Flan-T5 models against zero- and 10-shot GPT. Macro-F1 was measured using our manually validated synthetic dataset. The GPT-turbo-0613 version of GPT3.5 and the GPT4–0613 version of GPT4 were used. Error bars indicate the 95% confidence intervals. LLM large language model.
Fig. 2 Fine-tuned LLMs versus ChatGPT-family models. Comparison of model performance between our fine-tuned Flan-T5 models against zero- and 10-shot GPT. Macro-F1 was measured using our manually validated synthetic dataset. The GPT-turbo-0613 version of GPT3.5 and the GPT4–0613 version of GPT4 were used. Error bars indicate the 95% confidence intervals. LLM large language model.CC BY · PubMed Central
Fig. 3 LLM bias evaluation. The proportion of synthetic sentence pairs with and without demographics injected led to a classification mismatch, meaning that the model predicted a different SDoH label for each sentence in the pair. Results are shown across race/ethnicity and gender for a any SDoH mention task and b adverse SDoH mention task. Asterisks indicate statistical significance (P ≤ 0.05) chi-squared tests for multi-class comparisons and 2-proportion z tests for binary comparisons. LLM large language model, SDoH Social determinants of health.
Fig. 3 LLM bias evaluation. The proportion of synthetic sentence pairs with and without demographics injected led to a classification mismatch, meaning that the model predicted a different SDoH label for each sentence in the pair. Results are shown across race/ethnicity and gender for a any SDoH mention task and b adverse SDoH mention task. Asterisks indicate statistical significance (P ≤ 0.05) chi-squared tests for multi-class comparisons and 2-proportion z tests for binary comparisons. LLM large language model, SDoH Social determinants of health.CC BY · PubMed Central
Fig. 4 Prompting methods. Example of prompt templates used in the SKLLM package for GPT-turbo-0301 (GPT3.5) and GPT4 with temperature 0 to classify our labeled synthetic data. {labels} and {training_data} were sampled from a separate synthetic dataset, which was not human-annotated. The final label output is highlighted in green.
Fig. 4 Prompting methods. Example of prompt templates used in the SKLLM package for GPT-turbo-0301 (GPT3.5) and GPT4 with temperature 0 to classify our labeled synthetic data. {labels} and {training_data} were sampled from a separate synthetic dataset, which was not human-annotated. The final label output is highlighted in green.CC BY · PubMed Central
Fig. 5 Demographic-injected SDoH language development. Illustration of generating and comparing synthetic demographic-injected SDoH language pairs to assess how adding race/ethnicity and gender information into a sentence may impact model performance. FT fine-tuned, SDoH Social determinants of health.
Fig. 5 Demographic-injected SDoH language development. Illustration of generating and comparing synthetic demographic-injected SDoH language pairs to assess how adding race/ethnicity and gender information into a sentence may impact model performance. FT fine-tuned, SDoH Social determinants of health.CC BY · PubMed Central

Verification

Claims the paper makes

19 Valid · 0 Not valid · 4 Not checked · 66 Own results and interpretations

35 more claims · 3 with their cited sources checked

Create a free account to see all 89 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 26
  • It may state two separate claims as one. 27
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

7 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 6 · the rest aren’t loaded

1 more finding

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Show all 58 references

Research works printed but not linked: 1

  1. Hu, E. J. et al. LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations (2022).

Cited by

See all 278

Metadata

Authors
Guevara, M., Chen, S., Thomas, S., Chaunzwa, T. L., Franco, I., Kann, B. H., Moningi, S., Qian, J. M., Goldstein, M., Harper, S., Aerts, H. J. W. L., Catalano, P. J., Savova, G. K., Mak, R. H., & Bitterman, D. S.
Affiliations
  • Brigham and Women's Hospital (US)
  • Dana-Farber Cancer Institute (US)
  • Boston Children's Hospital (US)
Corresponding author
Bitterman, Danielle S.
Venue
npj Digital Medicine
Published
2024-01-11
Volume
7
Issue
1
Pages
6-6
ISSN
2398-6352 (ISSN-L)
OpenAlex
W4390745503
Cited by
278
References
58
Citations by year
2026: 90 · 2025: 127 · 2024: 60
FWCI
40.71
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • U.S. Department of Health & Human Services | NIH | National Cancer Institute (U54CA274516-01A1, R35CA22052)
  • Radiological Society of North America
  • Conquer Cancer Foundation
  • U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine (R01LM013486)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
social determinants of health, large language models, electronic health records, natural language processing, synthetic data augmentation, Flan-T5, algorithmic bias, race and ethnicity, gender, ICD-10 codes, real-world evidence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-15, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-15 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Large language models to identify social determinants of health in electronic health records. https://app.cochat.ai/papers/10.1038/s41746-023-00970-0
The paper
Guevara, M., Chen, S., Thomas, S., Chaunzwa, T. L., Franco, I., Kann, B. H., Moningi, S., Qian, J. M., Goldstein, M., Harper, S., Aerts, H. J. W. L., Catalano, P. J., Savova, G. K., Mak, R. H., & Bitterman, D. S. (2024). Large language models to identify social determinants of health in electronic health records. npj Digital Medicine. https://doi.org/10.1038/s41746-023-00970-0

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.