The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index; 38 of the 67 sources its claims cite were checked for support.
Large language models to identify social determinants of health in electronic health records
Open accessAbstract
Social determinants of health (SDoH) play a critical role in patient outcomes, yet their documentation is often missing or incomplete in the structured data of electronic health records (EHRs). Large language models (LLMs) could enable high-throughput extraction of SDoH from the EHR to support research and clinical care. However, class imbalance and data limitations present challenges for this sparsely documented yet critical information. Here, we investigated the optimal methods for using LLMs to extract six SDoH categories from narrative text in the EHR: employment, housing, transportation, parental status, relationship, and social support. The best-performing models were fine-tuned Flan-T5 XL for any SDoH mentions (macro-F1 0.71), and Flan-T5 XXL for adverse SDoH mentions (macro-F1 0.70). Adding LLM-generated synthetic data to training varied across models and architecture, but improved the performance of smaller Flan-T5 models (delta F1 + 0.12 to +0.23). Our best-fine-tuned models outperformed zero- and few-shot performance of ChatGPT-family models in the zero- and few-shot setting, except GPT4 with 10-shot prompting for adverse SDoH. Fine-tuned models were less likely than ChatGPT to change their prediction when race/ethnicity and gender descriptors were added to the text, suggesting less algorithmic bias ( p < 0.05). Our models identified 93.8% of patients with adverse SDoH, while ICD-10 codes captured 2.0%. These results demonstrate the potential of LLMs in improving real-world evidence on SDoH and assisting in identifying patients who could benefit from resource support.
Overview
CoChat Summary
TL;DR
This study evaluated large language models for extracting six social determinants of health (SDoH) categories from EHR narrative text, including fine-tuned Flan-T5/BERT models, synthetic data augmentation, and ChatGPT-family zero/few-shot prompting. Fine-tuned models achieved macro-F1 around 0.70 on in-domain tasks, generally outperformed ChatGPT-family models, were less sensitive to demographic descriptors, and identified far more patients with adverse SDoH than structured ICD-10 Z-codes.
Key findings
- Best in-domain performance was around macro-F1 0.70: the abstract reports Flan-T5 XL for any SDoH mentions (macro-F1 0.71) and Flan-T5 XXL for adverse SDoH mentions (macro-F1 0.70); the Results text/table report Flan-T5 XXL with synthetic data for any SDoH (macro-F1 0.70) and Flan-T5 XL without synthetic data for adverse SDoH (macro-F1 0.70).
- Adding LLM-generated synthetic data improved smaller Flan-T5 models (abstract: delta F1 +0.12 to +0.23); ablation studies showed performance was maintained until ~50% of gold data were removed when synthetic data were included, versus drops after ~10–20% without synthetic data.
- Fine-tuned models outperformed zero- and few-shot ChatGPT-family models for most SDoH classes, except GPT4 with 10-shot prompting for adverse SDoH (Table 3: fine-tuned Flan-T5 XL 0.86 vs GPT4 10-shot 0.90); on any SDoH, fine-tuned Flan-T5 XXL scored 0.92 vs GPT4 10-shot 0.88.
- Fine-tuned models changed predictions less often than ChatGPT when race/ethnicity and gender descriptors were injected: 14.3% vs 21.5% of sentence pairs for any SDoH (P = 0.007) and 9.9% vs 18.2% for adverse SDoH (P = 0.005). Models identified 93.8% (45/48) of patients with adverse SDoH, while ICD-10 Z-codes captured 2.0% (1/48).
Methods
- Sentence-level multilabel classification for six SDoH categories (employment, housing, transportation, parental status, relationship, social support) and two tasks (any SDoH mentions, adverse SDoH mentions). Primary radiotherapy corpus: 800 clinic notes from 770 patients with cancer receiving radiotherapy at Brigham and Women’s Hospital/Dana-Farber Cancer Institute, 2015–2022; out-of-domain datasets: immunotherapy (200 notes, 170 patients) and MIMIC-III (200 notes, 183 patients). Annotation used MAE; 300/800 notes were dual annotated, with Krippendorff’s alpha 0.86 for any SDoH and 0.76 for adverse SDoH.
- Models included BERT-base baseline and Flan-T5 base/large/XL/XXL, with LoRA for XL/XXL; synthetic data were generated via GPT3.5 prompting rounds. ChatGPT-family models were evaluated zero- and few-shot on a manually validated synthetic test set; bias was assessed with demographic-injected synthetic sentence pairs. Analysis used bootstrap macro-F1 confidence intervals, Mann–Whitney U tests, and chi-squared/2-proportion z tests.
Limitations
- Stated: training and out-of-domain datasets came from a predominantly white population treated at Boston hospitals, limiting generalizability; synthetic generation methods were not exhaustively assessed; ChatGPT evaluations were limited to manually verified synthetic sentences rather than protected health information, so performance may not reflect real clinical text; models are only as good as the annotated corpus, and SDoH annotation is challenging, especially for Support, and may be subject to annotator bias.
- Evident from the text: severe class imbalance (only 3% of training sentences contained any SDoH mention; housing had 20 instances in the training set) and sparse labels likely constrain performance. The ChatGPT comparison used synthetic rather than real clinical text, and the text contains minor inconsistencies between the abstract and Results/Table 1 for best model/task and macro-F1 values.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
19 Valid · 0 Not valid · 4 Not checked · 66 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“SDoH are rarely documented comprehensively in structured data in the electronic health records (EHRs)10–12, creating an obstacle to research and clinical care.”
§ Large language models to identify social determinants of health in electronic health records > Introduction
What the sources say
- Supportedcites [10] Truong, H. P. et al. Utilization of social determinants of health ICD-10 Z-codes among hospitalized patients in the United States, 2016-2017. Med. Care 58, 1037–1043 (2020). source
The Discussion (page not supplied) says EHR social-determinants data are typically unstructured and their documentation is usually non-standardized. It also reports that structured SDOH Z-codes are vastly underutilized, supporting the claim that comprehensive structured documentation is rare. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [11] Heidari, E., Zalmai, R., Richards, K., Sakthisivabalan, L. & Brown, C. Z-code documentation to identify social determinants of health among medicaid beneficiaries. Res. Soc. Adm. Pharm. 19, 180–183 (2023). source
- Supportedcites [12] Wang, M., Pantell, M. S., Gottlieb, L. M. & Adler-Milstein, J. Documentation and review of social determinants of health data in the EHR: measures and associated insights. J. Am. Med. Inform. Assoc. 28, 2608–2616 (2021). source
The Discussion reports almost negligibly low documentation in the structured problem list and patient social history questions, and says other structured fields were used at low levels. It also notes an exception—a housing-status nursing question documented for nearly 90% of encounters—so the finding is not that every structured field is rarely used. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Prior studies have demonstrated the feasibility of NLP for extracting a range of SDoH13–23.”
§ Large language models to identify social determinants of health in electronic health records > Introduction
What the sources say
- Supportedcites [13] Conway, M. et al. Moonstone: a novel natural language processing system for inferring social risk from clinical narratives. J. Biomed. Semant. 10, 1–10 (2019). source
The Background identifies Moonstone as an NLP system for extracting three social determinants of health from clinical notes: housing situation, living alone, and social support. The Conclusions report that it achieved excellent results on this task, supporting its use as one of the prior feasibility studies cited. Our copy provides section labels but no page numbers. (LLM, conf=0.92)
- Supportedcites [14] Bejan, C. A. et al. Mining 100 million notes to find homelessness and adverse childhood experiences: 2 case studies of rare and severe social determinants of health in electronic health records. J. Am. Med. Inform. Assoc. 25, 61–71 (2017). source
The Conclusion reports that the system extracted two social determinants of health—homelessness and adverse childhood experiences—from clinical notes with high precision. The Discussion likewise describes the study as demonstrating the feasibility of extracting low-prevalence determinants from millions of notes. This paper supports the citing sentence as one of several studies showing feasibility; it does not claim to have tested additional determinants. (LLM, conf=0.90)
- Not matchedcites [15] Topaz, M., Murga, L., Bar-Bachar, O., Cato, K. & Collins, S. Extracting alcohol and substance abuse status from clinical notes: the added value of nursing data. Stud. Health Technol. Inform. 264, 1056–1060 (2019).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [16] Gundlapalli, A. V. et al. Using natural language processing on the free text of clinical documents to screen for evidence of homelessness among US veterans. AMIA Annu. Symp. Proc. 2013, 537–546 (2013).
- Not checked — no full text of the sourcecites [17] Hammond, K. W., Ben-Ari, A. Y., Laundry, R. J., Boyko, E. J. & Samore, M. H. The feasibility of using large-scale text mining to detect adverse childhood experiences in a VA-treated population. J. Trauma. Stress 28, 505–514 (2015). source
- Not checked — no full text of the sourcecites [18] Han, S. et al. Classifying social determinants of health from unstructured electronic health records using deep learning-based natural language processing. J. Biomed. Inform. 127, 103984 (2022). source
- Not checked — no full text of the sourcecites [19] Rouillard, C. J., Nasser, M. A., Hu, H. & Roblin, D. W. Evaluation of a natural language processing approach to identify social determinants of health in electronic health records in a diverse community cohort. Med. Care 60, 248–255 (2022). source
- Not checked — no full text of the sourcecites [20] Feller, D. J. et al. Detecting social and behavioral determinants of health with structured and free-text clinical data. Appl. Clin. Inform. 11, 172–181 (2020). source
- Supportedcites [21] Yu, Z. et al. A study of social and behavioral determinants of health in lung cancer patients using transformers-based natural language processing models. AMIA Annu. Symp. Proc. 2021, 1225–1233 (2021). source
The Introduction (p. 2) describes prior NLP studies extracting 5, 11, and 12 categories of social and behavioral determinants of health from clinical text. The paper also reports successful extraction in its own study (Discussion and conclusion, p. 6). (LLM, conf=0.93)
- Supportedcites [22] Lybarger, K. et al. Leveraging natural language processing to augment structured social determinants of health data in the electronic health record. J. Am. Med. Inform. Assoc. 30, 1389–1397 (2023). source
The Background and Significance section (pp. 3–4) describes prior NLP studies extracting varied SDoH from clinical narratives, including housing insecurity, substance use, and social support. It also describes several extraction methods, supporting the claim that prior work demonstrated feasibility across a range of SDoH. (LLM, conf=0.92)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We found that fine-tuning LMs that are orders of magnitude smaller than ChatGPT-family models, even with our relatively small dataset, generally out-performed zero-shot and few-shot learning with ChatGPT-family models, consistent with prior work evaluating large LMs for clinical uses32–34.”
§ Large language models to identify social determinants of health in electronic health records > Discussion
What the sources say
- Supportedcites [32] Chen, S. et al. Evaluation of ChatGPT family of models for biomedical reasoning and classification. Preprint at https://arxiv.org/abs/2304.02496 (2023). source
The Results section (‘Effect of amount of training data on model performance’) reports that BioBERT fine-tuned on 20% of the development set surpassed the best GPT settings on three of four datasets, and fine-tuning on 50% surpassed them on all four. The Discussion identifies BioBERT as a 110-million-parameter model and says it consistently outperformed the much larger GPT models on the studied biomedical tasks. This supports the cited prior-work comparison, while ‘our relatively small dataset’ describes the citing authors’ own study. (LLM, conf=0.91)
- Partly supportedcites [33] Lehman, E. et al. Do we still need clinical language models? arXiv https://arxiv.org/abs/2302.08091 (2023). sourceExperimental — accuracy not yet measured
Section 5 reports that smaller fine-tuned clinical models generally outperformed in-context learning with GPT-3 and Flan-T5-XXL, including in limited-data settings. But the paper tested prompts with one to three examples, not zero-shot prompting, and did not directly evaluate ChatGPT. The citing authors’ reference to their own dataset is their finding, not one attributed to this source. (LLM, conf=0.84)
- Partly supportedcites [34] Ramachandran, G. K. et al. Prompt-based extraction of social determinants of health using few-shot learning. In: Proceedings of the 5th Clinical Natural Language Processing Workshop, 385–393 (Association for Computational Linguistics, 2023). sourceExperimental — accuracy not yet measured
The Results and Discussion (Table 1; page unavailable) report that fine-tuned models outperformed GPT-4 on social-determinants-of-health extraction. But the paper tested GPT-4 only with one-shot prompting, not zero-shot or broader few-shot approaches; its stronger fine-tuned systems used the entire training set. It does not establish the claim about performance with a relatively small fine-tuning dataset or quantify the models’ size difference. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“There have been several prior studies developing NLP methods to extract SDoH from the EHR13–21,40.”
§ Large language models to identify social determinants of health in electronic health records > Discussion
What the sources say
- Supportedcites [13] Conway, M. et al. Moonstone: a novel natural language processing system for inferring social risk from clinical narratives. J. Biomed. Semant. 10, 1–10 (2019). source
The Background states that Moonstone is an NLP system developed to extract social determinants of health—housing situation, living alone, and social support—from VA clinical notes. It is therefore an example of the prior studies identified by the citing sentence. (LLM, conf=0.93)
- Supportedcites [14] Bejan, C. A. et al. Mining 100 million notes to find homelessness and adverse childhood experiences: 2 case studies of rare and severe social determinants of health in electronic health records. J. Am. Med. Inform. Assoc. 25, 61–71 (2017). source
The Introduction describes this study’s goal as developing and testing a text-mining approach to extract homelessness and adverse childhood experiences from EHR clinical notes. The Conclusion reports that the method extracted both determinants from those notes. This paper therefore supports its role as one example among the cited prior studies. (LLM, conf=0.94)
- Not matchedcites [15] Topaz, M., Murga, L., Bar-Bachar, O., Cato, K. & Collins, S. Extracting alcohol and substance abuse status from clinical notes: the added value of nursing data. Stud. Health Technol. Inform. 264, 1056–1060 (2019).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [16] Gundlapalli, A. V. et al. Using natural language processing on the free text of clinical documents to screen for evidence of homelessness among US veterans. AMIA Annu. Symp. Proc. 2013, 537–546 (2013).
- Not checked — no full text of the sourcecites [17] Hammond, K. W., Ben-Ari, A. Y., Laundry, R. J., Boyko, E. J. & Samore, M. H. The feasibility of using large-scale text mining to detect adverse childhood experiences in a VA-treated population. J. Trauma. Stress 28, 505–514 (2015). source
- Not checked — no full text of the sourcecites [18] Han, S. et al. Classifying social determinants of health from unstructured electronic health records using deep learning-based natural language processing. J. Biomed. Inform. 127, 103984 (2022). source
- Not checked — no full text of the sourcecites [19] Rouillard, C. J., Nasser, M. A., Hu, H. & Roblin, D. W. Evaluation of a natural language processing approach to identify social determinants of health in electronic health records in a diverse community cohort. Med. Care 60, 248–255 (2022). source
- Not checked — no full text of the sourcecites [20] Feller, D. J. et al. Detecting social and behavioral determinants of health with structured and free-text clinical data. Appl. Clin. Inform. 11, 172–181 (2020). source
- Supportedcites [21] Yu, Z. et al. A study of social and behavioral determinants of health in lung cancer patients using transformers-based natural language processing models. AMIA Annu. Symp. Proc. 2021, 1225–1233 (2021). source
The Introduction (p. 2) describes NLP studies by Feller, Stemerman, and Lybarger that extracted social and behavioral determinants of health from clinical documents. It also states that this paper applies transformer-based NLP to extract those concepts from clinical narratives, supporting its inclusion among the cited studies. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites [40] Derton, A. et al. Natural language processing methods to empirically explore social contexts and needs in cancer patient notes. JCO Clin. Cancer Inf. 7, e2200196 (2023). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“SDoH are notoriously under-documented in existing EHR structured data10–12,39.”
§ Large language models to identify social determinants of health in electronic health records > Discussion
What the sources say
- Supportedcites [10] Truong, H. P. et al. Utilization of social determinants of health ICD-10 Z-codes among hospitalized patients in the United States, 2016-2017. Med. Care 58, 1037–1043 (2020). source
The Discussion says SDoH information in EHRs is typically unstructured and its documentation is often non-standardized. It also reports that structured SDoH Z-codes are vastly underutilized, supporting the claim about under-documentation in structured data. (LLM, conf=0.93)
- Not checked — no full text of the sourcecites [11] Heidari, E., Zalmai, R., Richards, K., Sakthisivabalan, L. & Brown, C. Z-code documentation to identify social determinants of health among medicaid beneficiaries. Res. Soc. Adm. Pharm. 19, 180–183 (2023). source
- Supportedcites [12] Wang, M., Pantell, M. S., Gottlieb, L. M. & Adler-Milstein, J. Documentation and review of social determinants of health data in the EHR: measures and associated insights. J. Am. Med. Inform. Assoc. 28, 2608–2616 (2021). source
The Introduction reports that poor social support was recorded for 0.4% of patients in structured EHR data versus 16.0% in clinical notes. The Discussion also says that structured social-history fields and Z-codes had low documentation levels, while noting that a nursing question built into standard workflow was a notable exception. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [39] Kharrazi, H. et al. The value of unstructured electronic health record data in geriatric syndrome case identification. J. Am. Geriatr. Soc. 66, 1499–1507 (2018). source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
35 more claims · 3 with their cited sources checked
Create a free account to see all 89 claims and every source verdict.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 27
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, ML and AI have been employed to investigate the social determinants of health and identify populations at high risk of adverse health outcomes [23].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, large language models (LLMs) can be integrated into this principle to analyze social media data and patient-generated content, providing insights into the social and environmental factors influencing health outcomes across communities and networks [23, 48–52].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Many of the studies on LLM have been conducted using large anonymised US datasets [5–7].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models (LLM) such as ChatGPT, GPT-4 API and Llama2 can interrogate and summarise large volumes of medical notes to create succinct summaries of radiology reports, internal medical progress notes and patient dialogue [5–8].”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Alongside recent advancements in LLMs, prompt engineering has emerged as an effective, low-cost method of enhancing the quality of LLM outputs for specific tasks.”
From A pilot feasibility study comparing large language models in extracting key information from ICU patient text records from an Irish population (2024) · cites this paper as [6]
Experimental — accuracy not yet measured
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Large language models to identify social determinants of health in electronic health records > Abstract
“The best-performing models were fine-tuned Flan-T5 XL for any SDoH mentions (macro-F1 0.71), and Flan-T5 XXL for adverse SDoH mentions (macro-F1 0.70).”
§ Large language models to identify social determinants of health in electronic health records > Results > Model performance
“The best-performing model for any SDoH mention task was Flan-T5 XXL (3 out of 6 categories) using synthetic data (Macro-F1 0.71). The best-performing model for the adverse SDoH mention task was Flan-T5 XL without synthetic data (Macro-F1 0.70).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Asian | 21 (2.7%) | 11 (2.4%) | 6 (3.9%) | 4 (2.6%) | 9 (5.3%) | 5 (2.7%) | N/A | 106 (21.6%) |”
§ Large language models to identify social determinants of health in electronic health records > Methods > Data
Reported 21.6 · recomputed 25.298329
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Hispanic | 11 (1.4%) | 8 (1.7%) | 2 (1.3%) | 1 (0.6%) | 20 (5.9%) | 11 (6.0%) | N/A | 97 (23.2%) |”
§ Large language models to identify social determinants of health in electronic health records > Methods > Data
Reported 5.9 · recomputed 11.764706
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“(21.0% (9/29)), respectively”
§ Large language models to identify social determinants of health in electronic health records > Results > Language model bias evaluation
Reported 21 · recomputed 31.034483
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Large language models to identify social determinants of health in electronic health records > Methods > Task definition and data labeling
“In total, ten thousand one hundred clinical notes were annotated line-by-line using the annotation software Multi-document Annotation Environment (MAE v2.2.13)59.”
§ Large language models to identify social determinants of health in electronic health records > Methods > Data
“Our primary dataset consisted of a corpus of 800 clinic notes from 770 patients with cancer who received radiotherapy (RT)”
1 more finding
Create a free account to see every finding of this check.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
The Lifelong Effects of Early Childhood Adversity and Toxic Stress
Shonkoff, J. P., Garner, A. S., THE COMMITTEE ON PSYCHOSOCIAL ASPECTS OF CHILD AND FAMILY HEALTH, C. O. E. C., Siegel, B. S., Dobbins, M. I., Earls, M. F., Garner, A. S., McGuinn, L., Pascoe, J., & Wood, D. L. (2012) · Pediatrics · doi:10.1542/peds.2011-2663 · Cited by 5,164
Caregiving as a Risk Factor for Mortality: The Caregiver Health Effects Study
Schulz, R., & Beach, S. R. (1999) · JAMA · doi:10.1001/jama.282.23.2215 · Cited by 3,340
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
The Association Between Income and Life Expectancy in the United States, 2001–2014
Chetty, R., Stepner, M., Abraham, S., Lin, S., Scuderi, B., Turner, N., Bergeron, A., & Cutler, D. (2016) · JAMA · doi:10.1001/jama.2016.4226 · Cited by 2,812
Automated Hate Speech Detection and the Problem of Offensive Language
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v11i1.14955 · Cited by 2,610
Manuscripts Accepted for Publication
(2005) · Child Development · doi:10.1111/j.1467-8624.2005.00876.x · Cited by 2,047
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
Research works printed but not linked: 1
- Hu, E. J. et al. LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations (2022).
Cited by
The TRIPOD-LLM reporting guideline for studies using large language models
Gallifant, J., Afshar, M., Ameen, S., Aphinyanaphongs, Y., Chen, S., Cacciamani, G., Demner-Fushman, D., Dligach, D., Daneshjou, R., Fernandes, C., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03425-5 · Cited by 615
Self-citationBioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains
Labrak, Y., Bazoge, A., Morin, E., Gourraud, P.-A., Rouvier, M., & Dufour, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.348 · Cited by 269
Mapping the regulatory landscape for artificial intelligence in health within the European Union
Schmidt, J., Schutte, N. M., Buttigieg, S., Novillo-Ortiz, D., Sutherland, E., Anderson, M., de Witte, B., Peolsson, M., Unim, B., Pavlova, M., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01221-6 · Cited by 187
Ong, J. C. L., Seng, B. J. J., Law, J. Z. F., Low, L. L., Kwa, A. L. H., Giacomini, K. M., & Ting, D. S. W. (2024) · Cell Reports Medicine · doi:10.1016/j.xcrm.2023.101356 · Cited by 111
Ng, K. K. Y., Matsuba, I., & Zhang, P. C. (2025) · NEJM AI · doi:10.1056/aira2400380 · Cited by 107
Metadata
- Authors
- Guevara, M., Chen, S., Thomas, S., Chaunzwa, T. L., Franco, I., Kann, B. H., Moningi, S., Qian, J. M., Goldstein, M., Harper, S., Aerts, H. J. W. L., Catalano, P. J., Savova, G. K., Mak, R. H., & Bitterman, D. S.
- Affiliations
- Brigham and Women's Hospital (US)
- Dana-Farber Cancer Institute (US)
- Boston Children's Hospital (US)
- Corresponding author
- Bitterman, Danielle S.
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2024-01-11
- Volume
- 7
- Issue
- 1
- Pages
- 6-6
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC10781957
- OpenAlex
- W4390745503
- Cited by
- 278
- References
- 58
- Citations by year
- 2026: 90 · 2025: 127 · 2024: 60
- FWCI
- 40.71
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. Department of Health & Human Services | NIH | National Cancer Institute (U54CA274516-01A1, R35CA22052)
- Radiological Society of North America
- Conquer Cancer Foundation
- U.S. Department of Health & Human Services | NIH | U.S. National Library of Medicine (R01LM013486)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- social determinants of health, large language models, electronic health records, natural language processing, synthetic data augmentation, Flan-T5, algorithmic bias, race and ethnicity, gender, ICD-10 codes, real-world evidence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-15, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-15 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large language models to identify social determinants of health in electronic health records. https://app.cochat.ai/papers/10.1038/s41746-023-00970-0
- The paper
- Guevara, M., Chen, S., Thomas, S., Chaunzwa, T. L., Franco, I., Kann, B. H., Moningi, S., Qian, J. M., Goldstein, M., Harper, S., Aerts, H. J. W. L., Catalano, P. J., Savova, G. K., Mak, R. H., & Bitterman, D. S. (2024). Large language models to identify social determinants of health in electronic health records. npj Digital Medicine. https://doi.org/10.1038/s41746-023-00970-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.