CoChat
Sign Up
Cited by 24Machine Learning in HealthcareBiomedical Text Mining and OntologiesTopic Modeling

Social determinants of health extraction from clinical notes across institutions using large language models

Open access
Keloth, Vipina K., Selek, Salih, Chen, Qingyu, Gilman, Christopher, Fu, Sunyang, Dang, Yifang, Chen, Xinghan, Hu, Xinyue +8 more·2025·npj Digital Medicine
DOI 10.1038/s41746-025-01645-8PMC PMC12084648
Open this paper in CoChat — free accountOpen PDF

Abstract

Detailed social determinants of health (SDoH) is often buried within clinical text in EHRs. Most current NLP efforts for SDoH have limitations, investigating limited factors, deriving data from a single institution, using specific patient cohorts/note types, with reduced focus on generalizability. We aim to address these issues by creating cross-institutional corpora and developing and evaluating the generalizability of classification models, including large language models (LLMs), for detecting SDoH factors using data from four institutions. Clinical notes were annotated with 21 SDoH factors at two levels: level 1 (SDoH factors only) and level 2 (SDoH factors and associated values). Compared to other models, instruction tuned LLM achieved top performance with micro-averaged F1 over 0.9 on level 1 corpora and over 0.84 on level 2 corpora. While models performed well when trained and tested on individual datasets, cross-dataset generalization highlighted remaining obstacles. Access to trained models will be made available at https://github.com/BIDS-Xu-Lab/LLMs4SDoH .

AI snapshot

Written from the paper’s full text.

TL;DR

This study created multi-institution annotated corpora for 21 social determinants of health (SDoH) factors at two annotation levels and evaluated classification models, including instruction-tuned LLaMA, for extracting SDoH from clinical notes. Instruction-tuned LLaMA achieved top performance in most in-domain and cross-dataset settings, but cross-institution generalization remained challenging.

Key findings

  • Instruction-tuned LLaMA achieved micro-averaged F1 over 0.9 on level 1 corpora and over 0.84 on level 2 corpora; per-dataset level 1 F1 was 0.927 (HCPC), 0.984 (UTP), 0.883 (MIMIC-III), and 0.945 (Mayo), while level 2 F1 was 0.886 (HCPC), 0.956 (UTP), 0.837 (MIMIC-III), and 0.918 (Mayo).
  • ClinicalBERT outperformed LLaMA on MIMIC-III level 1 (F1 0.892 vs. 0.883) and was second-best for level 1 corpora; LLaMA performed best on all other datasets.
  • In cross-dataset evaluation, LLaMA generally generalized better: trained on HCPC, it achieved F1 0.91 on both UTP and Mayo and 0.82 on MIMIC-III; the lowest level 2 cross-dataset F1 was 0.40 for LLaMA versus 0.08 for ClinicalBERT, 0.06 for SBERT, 0.04 for TextCNN, and 0.09 for XGBoost.
  • On five SDoH factors present in all four datasets with at least 25 training examples, LLaMA achieved top performance in 41 of 60 experiments, ClinicalBERT in 13, XGBoost in 5, and TextCNN and SBERT in 1 each; in-context learning was poor, with the best micro-F1 only 0.518 on HCPC level 1 using the 13B-chat model.

Methods

  • Four corpora were annotated: HCPC (2000 notes, 1529 patients), UTP (1000 notes, 698 patients), MIMIC-III (500 social work notes, 264 patients), and Mayo (527 notes, 62 patients). Notes were annotated for 21 SDoH factors at level 1 (factors only) and level 2 (factors plus values/attributes), with duplicate sentences removed; inter-annotator agreement after training was Cohen’s Kappa 0.73–0.90.
  • Models included XGBoost, TextCNN, SBERT, ClinicalBERT, and instruction-tuned LLaMA 2 7B. Non-LLaMA models used multi-label binary relevance; LLaMA was instruction fine-tuned. Evaluation used fivefold cross-validation, cross-dataset train/test splits (7:1:2), and combined-dataset training, with micro- and macro-averaged precision, recall, and F1.

Limitations

  • The paper states that differing dataset construction and preprocessing across sites may have introduced biases hindering cross-dataset generalization; the Mayo dataset came from a chronic pain prescreening cohort, likely not representative of Mayo’s general clinical population, and was small with high class imbalance.
  • The paper states that the study used US-based medical centers, limiting geographic and cultural generalizability; it was not designed for comprehensive bias evaluation, and limited structured data prevented full cohort characterization for bias assessment.
  • The paper notes computational resource demands for LLM training, fine-tuning, and inference; the text also makes evident that cross-dataset performance dropped substantially, especially on level 2 and macro-averaged evaluations, indicating remaining generalizability obstacles.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-025-01645-8

Cites

85 of 85 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

47 of 60 cited sources checked · 21 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Among 82 papers examined in a comprehensive review, only three SDoH factors were commonly addressed: smoking status, substance use, and homelessness.

    “A comprehensive review6 revealed that among the 82 papers examined, only three SDoH factors were commonly addressed: smoking status (27), substance use (21), and homelessness (20).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [6] Patra, B. G. et al. Extracting social determinants of health from electronic health records using natural language processing: A systematic review. J. Am. Med Inf. Assoc. 28, 2716–2727 (2021). source

      The Abstract (Results) gives the cited counts for smoking status (27), substance use (21), and homelessness (20), and the Results (SDoH identification/extraction) refers to 82 publications. But the abstract also lists alcohol use (15) among the most frequently studied categories, so the source does not support saying only three were commonly addressed. (LLM, conf=0.90)

      Experimental — accuracy not yet measured
  • Premisecitation

    Our extensive prior work comparing zero/few-shot and fine-tuned LLaMA models have demonstrated the superior performance of fine-tuned models.

    “Our extensive prior work comparing zero/few-shot and fine-tuned LLaMA models have demonstrated the superior performance of fine-tuned models51–54.”
    Quote in the paperSubstantial claim4 of 4 sources checked
    • cites [51] Keloth, V. K. et al. Advancing entity recognition in biomedicine via instruction tuning of large language models. Bioinformatics 40, btae163 (2024). source

      The Results (§4.1, Table 1; page unavailable) compare zero- and five-shot GPT models with instruction-tuned BioNER-LLaMA and report lower performance for the GPT models. This supports superior performance for the tuned LLaMA models in that comparison, but the paper does not compare zero- or few-shot LLaMA models with fine-tuned LLaMA models. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [52] Xie, Q. et al. Medical foundation large language models for comprehensive text analysis and beyond. npj Digital Medicine 8, 141 (2025). source

      The Results section (“Overall performance of medical text analysis”; page unavailable) compares zero-shot and task-specifically fine-tuned Me-LLaMA models and reports stronger results for the fine-tuned models against ChatGPT and GPT-4. The Methods section (“Evaluation settings”; page unavailable) identifies zero-shot and supervised learning as the two settings evaluated. Our copy does not report a few-shot comparison, so it does not support that part of the claim. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [53] Hu, Y. et al. Improving large language models for clinical named entity recognition via prompt engineering. J. Am. Med. Inform. Assoc. 31, 1812–1820 (2024). source

      The Results section reports that fine-tuned BioClinicalBERT outperformed few-shot GPT models on two clinical NER datasets, supporting the general performance comparison. But the Discussion says the study was limited to GPT models and identifies LLaMA as future work; it does not demonstrate superior performance for fine-tuned LLaMA models. Page numbers are unavailable in our copy. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [54] Chen, Q. et al. Benchmarking large language models for biomedical natural language processing applications and recommendations.Nature Communications. 16, 3280 (2025). source
  • Premisecitation

    Many studies, even recent ones, have focused on a limited set of SDoH factors (up to 8 factors).

    “Firstly, many studies, even the recent ones19–25 have focused on a limited set of SDoH factors (up to 8 factors).”
    Quote in the paperSubstantial claim7 of 7 sources checked
    • cites [19] Ong, J. C. L. et al. Artificial intelligence, ChatGPT, and other large language models for social determinants of health: Current state and future directions. Cell Rep. Med. 5, (2024). source

      The SDOH section describes gaps in research on complex determinants, and the ‘Research on SDOH and health outcomes’ section names determinants emphasized in several reviews. Neither says that many studies, including recent ones, examined no more than eight SDoH factors. Searches for a limited number of factors and the eight-factor threshold found no passage establishing that claim. (LLM, conf=0.70)

      Experimental — accuracy not yet measured
    • cites [20] Lybarger, K. et al. Leveraging natural language processing to augment structured social determinants of health data in the electronic health record. J. Am. Med Inf. Assoc. 30, 1389–1397 (2023). source
    • cites [21] Richie, R., Ruiz, V. M., Han, S., Shi, L. & Tsui, F. Extracting social determinants of health events with transformer-based multitask, multilabel named entity recognition. J. Am. Med. Inf. Assoc. 30, 1379–1388 (2023). source
    • cites [22] Romanowski, B., Ben Abacha, A. & Fan, Y. Extracting social determinants of health from clinical note text with classification and sequence-to-sequence approaches. J. Am. Med. Inf. Assoc. 30, 1448–1455 (2023). source
    • cites [23] Yu, Z. et al. Identifying social determinants of health from clinical narratives: A study of performance, documentation ratio, and potential bias. J. Biomed. Inf. 153, 104642 (2024). source

      The Introduction (p. 6) describes several earlier studies examining a single SDoH category or five or six categories, supporting the point that some studies had a limited scope. But it also describes recent studies examining 12 and 13 categories, so it does not support an eight-factor ceiling for recent studies generally. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [24] Fu, Y. et al. Extracting social determinants of health from pediatric patient notes using large language models: novel corpus and methods. arXiv preprint arXiv:2404.00826 (2024). source

      Section 2.1, “SDoH Corpora” (page unavailable), says many studies of adult populations focused on a limited number of SDoH factors and includes a 2022 example. It does not establish the specific ceiling of eight factors or extend that characterization beyond the adult-population studies discussed. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [25] Allen, K. S. et al. Natural language processing-driven state machines to extract social factors from unstructured clinical documentation. JAMIA Open 6, ooad024 (2023). source
  • Premisecitation

    Most methods for SDoH extraction were developed and evaluated within specific healthcare systems.

    “Thirdly, most methods were developed and evaluated within specific healthcare systems, such as a Veterans Health Administration center35, a Medical Center36, and a Trauma Center37.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [35] Wray, C. M. et al. Examining the interfacility variation of social determinants of health in the Veterans Health Administration. Fed. Pract. 38, 15 (2021). source

      The Methods section says the study applied an SDoH-extraction NLP program to VHA records, and the Results report its use across VHA facilities. But Methods calls the program “previously developed”; this paper does not establish that the method was developed at a VHA center. The citation supports the VHA evaluation example, not the development claim. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [36] Feller, D. J. et al. Towards the inference of social and behavioral determinants of sexual health: development of a gold-standard corpus with semi-supervised learning. AMIA Annu Symp. Proc. 2018, 422 (2018).
    • cites [37] Afshar, M. et al. Natural language processing and machine learning to identify alcohol misuse from the electronic health record in trauma patients: development and internal validation. J. Am. Med. Inf. Assoc. 26, 254–261 (2019). source
  • Premisecitation

    The study by Guevera et al. was limited in scope to identifying the presence or absence of only six SDoH factors, without exploring temporality or nuanced assertions beyond a broad adverse categorization.

    “However, the study was limited in scope to identifying the presence or absence of only six SDoH factors, without exploring temporality or nuanced assertions beyond a broad adverse categorization.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [45] Guevara, M. et al. Large language models to identify social determinants of health in electronic health records. NPJ Digital Med. 7, 6 (2024). source
  • Classified as probably not a substantial claim. 49
  • May combine two claims. 53
  • The quote only partly states this claim. 2

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Existing research has explored integrating patient SDoH data with geospatial datasets like census demographics and social vulnerability index (SVI) to estimate SDoH factors at a community or neighborhood level.

    “Existing research has also explored integrating patient SDoH data with geospatial datasets like census demographics and social vulnerability index (SVI) to estimate SDoH factors at a community or neighborhood level41–44.”
    Quote in the paperNot a substantial claim2 of 4 sources checked
    • cites [41] Comer, K. F., Grannis, S., Dixon, B. E., Bodenhamer, D. J. & Wiehe, S. E. Incorporating geospatial capacity within clinical data systems to address social determinants of health. Public Health Rep. 126, 54–61 (2011). source
    • cites [42] Brignone, E., LeJeune, K., Mihalko, A. E., Shannon, A. L. & Sinoway, L. I. Self-reported social determinants of health and area-level social vulnerability. JAMA Netw. open 7, e2412109–e2412109 (2024). source

      The Methods (Measures; page not provided) say patient addresses were geocoded to census tracts and joined to SVI data, and the Discussion reports associations between self-reported needs and community vulnerability. The paper does not describe using the combined data to estimate SDoH factors at the community or neighborhood level; that purpose goes beyond what it reports. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [43] Maroko, A. R. et al. Integrating social determinants of health with treatment and prevention: a new tool to assess local area deprivation.Prev. Chronic Dis. 13, E128 (2016). source

      The Results describe a local area deprivation index built from 17 census variables, and the Methods compare it with hospitalization data aggregated by ZIP code. This supports estimating neighborhood-level deprivation, but the paper does not report integrating patient SDoH data with geospatial datasets or using a social vulnerability index (SVI). (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [44] Brown, E. M. et al. Assessing area-level deprivation as a proxy for individual-level social risks. Am. J. Prev. Med 65, 1163–1171 (2023). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    One study analyzed the association between self-reported SDoH needs assessment by a health system and census tract-level social vulnerability measured using the SVI.

    “One such study analyzed to what extend the self-reported SDoH needs assessment by a health system are associated with census tract–level social vulnerability measured using the SVI42.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [42] Brignone, E., LeJeune, K., Mihalko, A. E., Shannon, A. L. & Sinoway, L. I. Self-reported social determinants of health and area-level social vulnerability. JAMA Netw. open 7, e2412109–e2412109 (2024). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Guevera et al. conducted a study using encoder-decoder language models, specifically fine-tuned Flan-T5 XL and Flan-T5 XXL, to classify mentions of SDoH categories from narrative text in EHRs.

    “Guevera et al. conducted a study using encoder-decoder language models, specifically fine-tuned Flan-T5 XL and Flan-T5 XXL, to classify mentions of SDoH categories from narrative text in EHRs45.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [45] Guevara, M. et al. Large language models to identify social determinants of health in electronic health records. NPJ Digital Med. 7, 6 (2024). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent studies have utilized large language models (LLMs) for the identification of SDoH factors.

    “Recent studies have utilized the use of large language models (LLMs) for the identification of SDoH factors19,24,45,46.”
    Quote in the paperNot a substantial claim4 of 4 sources checked
    • cites [19] Ong, J. C. L. et al. Artificial intelligence, ChatGPT, and other large language models for social determinants of health: Current state and future directions. Cell Rep. Med. 5, (2024). source
    • cites [24] Fu, Y. et al. Extracting social determinants of health from pediatric patient notes using large language models: novel corpus and methods. arXiv preprint arXiv:2404.00826 (2024). source
    • cites [45] Guevara, M. et al. Large language models to identify social determinants of health in electronic health records. NPJ Digital Med. 7, 6 (2024). source
    • cites [46] Gabriel, R. A. et al. On the development and validation of large language model-based classifiers for identifying social determinants of health. Proc. Natl. Acad. Sci. 121, e2320716121 (2024). source

  • May combine two claims.

    Premisecitation

    Most current studies do not use the full spectrum of EHR data, and more efforts are being devoted to including unstructured textual data in EHRs for real world evidence generation.

    “However, most current studies do not use the full spectrum of EHR data, and more efforts are being devoted to including unstructured textual data in EHRs for real world evidence generation14.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [14] Keloth, V. K. et al. Representing and utilizing clinical textual data for real world studies: An OHDSI approach. J. Biomed. Inf. 142, 104343 (2023). source

      The Introduction (page not provided) describes efforts to incorporate clinical text into observational studies, and the Conclusion (page not provided) describes a framework for using textual data to generate real-world evidence. The paper does not establish the claim that most current studies fail to use the full spectrum of EHR data. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • Rests on an assumption: Clinical narratives contain more detailed characterization of several SDoH factors (e.g., living status, financial instability, social support) beyond the structured representation.
    • Rests on an assumption: The unstructured text data often contains detailed information to describe how multiple socio-economic, behavioral factors, and situational variables interact with each other.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    Further characteristics such as age, sex, race, etc. could also explain differences in SDoH extracted from each dataset, which might also result in bias in model performance.

    “we note that further characteristics such as age, sex, race, etc. could also explain differences in SDoH extracted from each dataset which might also result in bias in the model performance56.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [56] Dong, X., Wang, Y., Yu, P. S. & Caverlee, J. Disclosure and mitigation of gender bias in llms. arXiv preprint arXiv:2402.11190 (2024). source

      Section 4.2 discusses how different input topics and probing strategies affect measured gender bias in LLMs, but it does not address SDoH extraction or demographic differences between clinical datasets. Searches for SDoH and for age, sex, and race as dataset characteristics found no passage supporting the cited claim. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    These factors might be less prevalent in clinical notes written by physicians from other specialties, and SDoH documentation depends also on the patient population.

    “These factors might be less prevalent in clinical notes written by physicians from other specialties47,48 and hence SDoH documentation depends also on the patient population.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [47] Wang, M., Pantell, M. S., Gottlieb, L. M. & Adler-Milstein, J. Documentation and review of social determinants of health data in the EHR: measures and associated insights. J. Am. Med. Inf. Assoc. 28, 2608–2616 (2021). source

      Materials and Methods (page unavailable) says some types of clinical notes are enriched for SDoH information, particularly social work notes. That supports the possibility that note content varies by author, but the paper does not compare physicians across specialties or establish that the differences depend on patient population. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [48] Vest, J. R., Grannis, S. J., Haut, D. P., Halverson, P. K. & Menachemi, N. Using structured and unstructured data to identify patients’ need for services that address the social determinants of health. Int J. Med. Inf. 107, 101–106 (2017). source

      The Limitations section (p. 13) says the findings may vary with patient population and documentation practices across providers. It does not compare physicians’ specialties or establish that the factors would be less prevalent in notes from other specialties. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The corpus from study34 characterized chronic pain episodes in adult patients receiving treatment at Mayo Clinic and the Olmsted Medical Center, comprising 62 adults with noncancer chronic pain.

    “For our fourth dataset we utilized a subset of the corpus from a study34 that characterized chronic pain episodes in adult patients receiving treatment at Mayo Clinic and the Olmsted Medical Center. An annotated corpus of 62 adults experiencing noncancer chronic pain was created using chronic pain ICD code for the anchor diagnosis and retrieving all the patient’s clinical notes 6 months before and”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [34] Carlson, L. A. et al. Characterizing chronic pain episodes in clinical text at two health care systems: Comprehensive annotation and corpus analysis. JMIR Med. Inf. 8, e18659 (2020). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Several studies have shown an association of SDoH factors such as education level, isolation, and loneliness with the onset of Alzheimer's disease and related dementias (ADRD) in older adults.

    “Several studies have shown the association of SDoH factors such as education level, isolation, and loneliness with the onset of Alzheimer’s disease and related dementias (ADRD) in older adults84–86.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [84] Shafighi, K. et al. Social isolation is linked to classical risk factors of Alzheimer’s disease-related dementias. PLoS One 18, e0280471 (2023). source
    • cites [85] Majoka, M. A. & Schimming, C. Effect of social determinants of health on cognition and risk of Alzheimer disease and related dementias. Clin. Ther. 43, 922–929 (2021). source

      The Education section reports an association between lower education and higher ADRD risk. The Social Engagement section links isolation to poorer cognition and reports that greater middle- and late-life engagement reduces later ADRD diagnosis risk. Loneliness appears in a cited study’s title, but our copy does not separately report an association between loneliness and ADRD onset. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [86] Barak, Y. & Glue, P. Do Social Isolation and Loneliness Kill People with Alzheimer’s Disease?. OBM Geriatrics 2, 1–5 (2018). source

      In “Do Social Isolation and Loneliness Kill People with Alzheimer's Disease?” (page unavailable), the paper describes social isolation as a risk factor for developing AD and reports an association between loneliness and incident mild cognitive impairment in older adults. It does not address education level, and its loneliness finding concerns MCI rather than the onset of AD or dementia. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    SBERT fine-tunes BERT in a siamese/triplet network architecture that produces sentence embeddings that are semantically meaningful.

    “SBERT fine-tunes BERT31 in a siamese/triplet network architecture that produces sentence embeddings that are semantically meaningful.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [91] Reimers, N. & Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3982–3992 (2019). source
    • cites [31] Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies. Vol. 1, 4171–4186 (2019). source

  • May combine two claims.

    Findingown data

    The Mayo dataset contained all clinical notes of a chronic pain cohort with much of the data structured as templates to be filled in resulting in high volumes of duplicate information.

    “The dataset contained all clinical notes of a chronic pain cohort with much of the data structured as templates to be filled in resulting in high volumes of duplicate information.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    A total of 2838 sentences were annotated from 500 notes in the MIMIC-III dataset, with 20 SDoH factors present.

    “A total of 2838 sentences were annotated from 500 notes with 20 SDoH factors present.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    When fine-tuned on HCPC, LLaMA achieved an F1 score of 0.91 on both UTP and Mayo datasets and 0.82 on MIMIC-III, demonstrating an overall best performance on cross-dataset generalizability.

    “When fine-tuned on HCPC, LLaMA achieved an F1 score of 0.91 on both UTP and Mayo datasets and 0.82 on MIMIC-III, demonstrating an overall best performance on cross-dataset generalizability.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For level 1, the UTP dataset has only 15 classes and has a comparatively better class balance.

    “For level 1, UTP dataset has only 15 classes (Fig. 1 top right) and has a comparatively better class balance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    18 SDoH factors were present in the Mayo dataset with 10 factors having less than 10 samples.

    “18 SDoH factors were present with 10 factors having less than 10 samples.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The Mayo dataset was the smallest with 964 sentences annotated.

    “The Mayo dataset was the smallest with 964 sentences annotated.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Other factors that were frequently reported in the MIMIC-III dataset included alcohol and drug use, sex, and employment.

    “Other factors that were frequently reported included alcohol and drug use, sex, and employment.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The instruction tuned LLaMA model achieved an F1 score of 0.94 on the Mayo dataset, closely followed by ClinicalBERT with 0.919.

    “It is interesting to note that the instruction tuned LLaMA model still achieved an F1 score of 0.94 closely followed by ClinicalBERT with 0.919.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The instruction-tuned LLaMA model performed better than all other models for all datasets except on the MIMIC-III dataset (level 1) for which the ClinicalBERT model was better.

    “We observe that the instruction-tuned LLaMA model performed better than all other models for all datasets except on the MIMIC-III dataset (level 1) for which the ClinicalBERT50 model was better.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [50] Alsentzer, E. et al. Publicly available clinical BERT embeddings. Proceedings of the 2nd Clinical Natural Language Processing Workshop. 72–78 (2019). source

  • May combine two claims.

    Findingown data

    The 7B-base model produced inconsistent outputs, while the 7B-chat model, though slightly improved, generated overly verbose responses with numerous false positives, requiring extensive post-processing.

    “The 7B-base model produced inconsistent outputs, while the 7B-chat model, though slightly improved, generated overly verbose responses with numerous false positives (outputting the entire list of SDoH factors in majority of the cases), requiring extensive post-processing.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    A total of 18 SDoH factors were annotated in the HCPC dataset.

    “A total of 18 SDoH factors were annotated in this dataset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the UTP dataset, sex and race were almost always mentioned in the beginning followed by some information regarding the patient's employment, living and marital status.

    “Sex and race were almost always mentioned in the beginning (e.g., The patient is an 80 yr old Caucasian F …) followed by some information regarding the patient’s employment, living and marital status.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    For the UTP dataset, a total of 1691 sentences were annotated from 1000 chart notes.

    “For the UT Physicians (UTP) dataset, a total of 1691 sentences were annotated from 1000 chart notes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Mayo corpus is highly imbalanced, is the smallest dataset, and has a total of 19 classes.

    “On the other hand, Mayo corpus (Fig. 1 bottom right) is highly imbalanced, is the smallest dataset, and has a total of 19 classes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    14 SDoH factors were annotated in the UTP dataset, showing high prevalence of sex, race, and substance abuse.

    “14 SDoH were annotated showing high prevalence of sex, race, and substance abuse.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    ClinicalBERT demonstrated better performance when trained on UTP and tested on MIMIC-III and Mayo datasets and when trained on MIMIC-III and tested on Mayo dataset.

    “ClinicalBERT demonstrated better performance when trained on UTP and tested on MIMIC-III and Mayo datasets and when trained on MIMIC-III and tested on Mayo dataset.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingreasoning

    This is reflected in the model performance with 5-fold cross-validation consistently yielding an F1 score above 0.95 for all models on the UTP dataset.

    “This is reflected in the model performance with 5-fold cross-validation consistently yielding an F1 score above 0.95 for all models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    We notice a widened performance gap between LLaMA and other models, with the LLM still performing better than other models with greater than 0.45 F1 across the datasets and levels of annotation.

    “We notice a widened performance gap between LLaMA and other models, with the LLM still performing better than other models with greater than 0.45 F1 across the datasets and levels of annotation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The inter-annotator agreement after the final round of training was in the range of 0.73-0.90 for the datasets, similar to the annotator agreement reported in other studies.

    “The inter-annotator agreement after the final round of training was in the range of 0.73–0.90 for the datasets (which is similar to the annotator agreement reported in other studies32).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] Han, S. et al. Classifying social determinants of health from unstructured electronic health records using deep learning-based natural language processing. J. Biomed. Inf. 127, 103984 (2022). source

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    UTP chart notes are comparatively longer and contain information regarding medications and different body systems which are irrelevant to this study.

    “Additionally, chart notes are comparatively longer and contain information regarding medications and different body systems which are irrelevant to this study.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Performance is usually good for most models when trained and tested on the same corpus.

    “It is observed that the performance is usually good for most models when trained and tested on the same corpus.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The primary reasons for the notably low performance on the Mayo dataset, when evaluated on the combined dataset, were instances of no responses and generation of irrelevant outputs such as numerical values.

    “The primary reasons for the notably low performance on the Mayo dataset, when evaluated on the combined dataset (Table 3), were the instances of no responses and the generation of irrelevant outputs such as numerical values (e.g., dates).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The lowest performance for any dataset pair was an F1 score of 0.40 for LLaMA 2, 0.08 for ClinicalBERT, 0.06 for SBERT, 0.04 for TextCNN, and 0.09 for XGBoost, demonstrating the superior ability of instruction tuned LLaMA models to generalize better to unseen data.

    “The lowest performance for any dataset pair was an F1score of 0.40 for LLaMA 2, 0.08 for ClinicalBERT, 0.06 for SBERT, 0.04 for TextCNN, and 0.09 for XGBoost, clearly demonstrating the superior ability of instruction tuned LLaMA models to generalize better to unseen data.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    LLaMA models outperformed all other models in cross-dataset generalizability, showing a wider performance gap when evaluated on macro-average precision, recall, and F1.

    “LLaMA models outperformed all other models in cross-dataset generalizability, showing a wider performance gap when evaluated on macro-average precision, recall, and F1 (Supplementary Table 3).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ClinicalBERT and XGBoost also showed competitive performance, achieving top performance in 13 and 5 experiments, respectively.

    “However, ClinicalBERT and XGBoost also showed competitive performance, achieving top performance in 13 and 5 experiments, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    During annotation, some of the 21 annotated factors had a notably low prevalence and those factors varied based on the corpus.

    “During the annotation process, it was observed that some of the factors among the 21 annotated had a notably low prevalence and that those factors varied based on the corpus.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The instruction-tuned LLaMA model failed to make any predictions, generated predictions outside the expected class labels, and struggled to differentiate between SDoH pertaining to the patient versus those referring to family members.

    “Analyzing the predictions, we observed instances where the model failed to make any predictions, generated predictions outside the expected class labels (e.g., predicting “6th grade” instead of “Education level”), and struggled to differentiate between SDoH pertaining to the patient versus those referring to family members.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ClinicalBERT performed on par or sometimes better on micro-averaged evaluation and when enough data is available for different factors.

    “The ClinicalBERT model performed on par or sometimes better on micro-averaged evaluation and when enough data is available for different factors.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The Mayo dataset is the smallest dataset and has high class imbalance with more than half of the samples belonging to the 'non SDoH' category.

    “As for Mayo, it is the smallest dataset and has high class imbalance with more than half of the samples belonging to ‘non SDoH’ category.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The TextCNN, SBERT, and ClinicalBERT models had more modest training overheads, requiring only a single GPU.

    “The TextCNN, SBERT, and ClinicalBERT models had more modest training overheads, requiring only a single GPU.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    LLaMA 2's in-context learning capabilities showed limitations in zero-/few-shot performance, with the 13B-chat model achieving a best micro-average F1 of only 0.518 on the HCPC dataset for level 1 annotations, and performance further declining in level 2 annotations and macro-average evaluations.

    “Furthermore, our exploration of LLaMA 2’s in-context learning capabilities (Supplementary Table 12) revealed limitations in its zero-/few-shot performance for this task, with the 13B-chat model achieving a best micro-average F1 of only 0.518 on the HCPC dataset (the highest among all datasets) for level 1 annotations, and performance further declining in level 2 annotations and macro-average evalu”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    In UTP chart notes, social history recorded mostly information related to substance use, and other SDoH information was scattered under sections such as "History of present illness" and "General observation".

    “Other SDoH information was scattered under sections titled “History of present illness”, “General observation”, etc.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In cross-dataset evaluations, LLaMA achieved top performance in 41 out of 60 experiments, demonstrating strong generalizability.

    “In these cross-dataset evaluations, LLaMA achieved top performance in 41 out of 60 experiments, demonstrating strong generalizability.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    While the work demonstrates some generalizability across datasets from different health systems, the highest performances are still achieved when models are tested in-domain.

    “As such, while our work demonstrates some generalizability across datasets from different health systems, the highest performances are still achieved when models are tested in-domain.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The XGBoost model could be trained efficiently on CPUs and exhibited the fastest runtimes overall.

    “The XGBoost model being conventional tree-based methods, could be trained efficiently on CPUs and exhibited the fastest runtimes overall.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The Mayo notes exhibited a semi-structured format characterized by template structures, particularly noticeable in sections concerning substance use, along with a huge amount of duplicated information.

    “These notes exhibited a semi-structured format characterized by template structures, particularly noticeable in sections concerning substance use, along with huge amount of duplicated information.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The Social history sections extracted from HCPC psychosocial assessment notes are rich in SDoH information.

    “We randomly selected 2000 assessment notes (corresponding to 1529 patients) and extracted the “Social history” sections (using MedSpacy sectionizer83) from these notes which are rich in SDoH information.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Even though ClinicalBERT, SBERT and XGBoost models performed well when trained and tested on the same datasets, this performance did not translate when tested on other datasets, especially on level 2 corpora and when evaluating per factor performance.

    “Even though ClinicalBERT, SBERT and XGBoost models performed well when trained and tested on the same datasets, this performance did not translate when tested on other datasets especially on level 2 corpora and when evaluating per factor performance.”
    • Also stated in: § Discussion · hedged
      “While all models perform relatively well when trained and tested on a single dataset, performance varied and dropped on cross-dataset evaluation, indicating the need for further research in this domain.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    Notes from UTP contained diverse SDoH information spread across different sections like "History of present illness" and "General observation," prompting a comprehensive review of note contents.

    “Conversely, notes from UTP contained diverse SDoH information spread across different sections like “History of present illness” and “General observation,” prompting a comprehensive review of note contents to ensure inclusion of relevant data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    To fine-tune the 7B LLaMA model on about 7300 instruction demonstrations required approximately 10 min utilizing four NVIDIA A100 80GB GPUs.

    “Specifically, to fine-tune the 7B LLaMA model on about 7300 instruction demonstrations (combined training on all corpora) required approximately 10 min utilizing four NVIDIA A100 80GB GPUs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The distribution of factors still varies, and class imbalance exists.

    “Nevertheless, the distribution of factors still varies, and class imbalance exists.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    A single instance of LLaMA's zero-shot performance was also observed for financial issues.

    “A single instance of LLaMA’s zero-shot performance was also observed (Supplementary Table 7) for financial issues.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Overall, LLaMA performs particularly well on low-prevalence factors; however, other models can achieve similar or even better performance in some cases when sufficient training data are available.

    “Overall, LLaMA performs particularly well on low-prevalence factors; however, other models can achieve similar or even better performance in some cases when sufficient training data are available.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Out of 60 experiments (5 factors, 4 data sources, and each tested across three other data sources), LLaMA achieved top performance on 41 experiments, ClinicalBERT achieved top performance on 13, XGBoost achieved top performance on 5, and TextCNN and SBERT on 1 each.

    “Out of 60 experiments (5 factors, 4 data sources, and each tested across three other data sources), LLaMA achieved top performance on 41 experiments, ClinicalBERT achieved top performance on 13, XGBoost achieved top performance on 5, and TextCNN and SBERT on 1 each.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    These tables include instances where LLaMA achieved a perfect F1 score of 1.0, as well as scenarios where all models, including LLaMA, performed subpar or where LLaMA's F1 score was lower than other models.

    “These tables include instances where LLaMA achieved a perfect F1 score of 1.0, as well as scenarios where all models, including LLaMA, performed subpar or where LLaMA’s F1 score was lower than other models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The instruction-tuned LLaMA model outperformed all other models, especially when the number of factors increases and performance is evaluated per factor considering high class imbalance.

    “Our results demonstrate the ability of instruction tuned LLaMA model to outperform all other models especially when the number of factors increases, and the performance is evaluated per factor considering the high class-imbalance.”
    • Also stated in: § Discussion · plain
      “Additionally, four classification models and a large language model were experimented with to detect SDoH factors with the LLM performing the best.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The difference ranged from 2.6% to 12.6% on level 1 and from 2% to 13.4% on level 2 annotations.

    “The difference ranged from 2.6% to 12.6% on level 1 and from 2% to 13.4% on level 2 annotations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the HCPC dataset, SDoH details were predominantly concentrated in psychosocial assessments within social history sections, leading to their sole consideration.

    “In the case of the HCPC dataset, SDoH details were predominantly concentrated in psychosocial assessments within social history sections, leading to their sole consideration.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    These results can be attributed to the high overlap of SDoH categories between these datasets, with a minimal impact from non-overlapping categories on micro-averaged metrics due to their limited sample sizes.

    “These results can be attributed to the high overlap of SDoH categories between these datasets, with a minimal impact from non-overlapping categories on micro-averaged metrics due to their limited sample sizes.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The lack of broader testing raises concerns about the generalizability and adaptability of these NLP methods to diverse healthcare settings.

    “The lack of broader testing raises concerns about the generalizability and adaptability of these NLP methods to diverse healthcare settings.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    These factors have a profound influence on the heterogeneity in the distribution of SDoH factors in clinical notes.

    “These factors have a profound influence on the heterogeneity in the distribution of SDoH factors in clinical notes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The performance further dropped on level 2 annotations and macro-average evaluations likely due to the complexity of this task and increase in the number of classes.

    “The performance further dropped on level 2 annotations and macro-average evaluations likely due to the complexity of this task and increase in the number of classes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    This explains the towering frequency of marital status and some inference that could be made regarding living status and social support in the MIMIC-III dataset.

    “This explains the towering frequency of marital status and some inference that could be made regarding living status and social support.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The maximum difference between ClinicalBERT and LLaMA (13.2%) was observed for the Mayo dataset (level 2) due to the high class-imbalance and most classes not having enough samples for learning.

    “The maximum difference between ClinicalBERT and LLaMA (13.2%) was observed for the Mayo dataset (level 2) due to the high class-imbalance (Supplementary Fig. 4) and most classes not having enough samples for learning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Other work has explored mapping SDoH documentation and area-level deprivation to assess individual-level social risks.

    “Other work has explored mapping SDoH documentation and area-level deprivation to assess individual-level social risks44.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [44] Brown, E. M. et al. Assessing area-level deprivation as a proxy for individual-level social risks. Am. J. Prev. Med 65, 1163–1171 (2023). source

  • May combine two claims.

    Interpretationreasoning

    ClinicalBERT demonstrated better performance than SBERT, possibly due to training on MIMIC notes.

    “Comparing ClinicalBERT and SBERT, ClinicalBERT demonstrated better performance possibly due to training on MIMIC notes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Techniques like parameter-efficient fine-tuning and quantization could further reduce the GPU and memory requirements considerably.

    “though techniques like parameter-efficient fine-tuning and quantization could further reduce the GPU and memory requirements considerably.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The HCPC notes were annotated by two annotators based on the annotation guidelines.

    “These notes were then annotated by two annotators based on the annotation guidelines.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    With sufficient training data, simpler models may achieve comparable or sometimes better generalizability for well-defined tasks.

    “With sufficient training data, simpler models may achieve comparable or sometimes better generalizability for well-defined tasks.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Fivefold cross validation was performed by dividing each corpus into 5 folds, recording performance of each fold, and averaging performance across all 5 folds.

    “We performed fivefold cross validation by dividing each corpus into 5 folds and performance of each fold was recorded and average performance across all 5 folds was calculated.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    To address class imbalance and enhance model performance, it is common practice to merge classes with fewer samples into a single category.

    “To address the issue of class imbalance and enhance model performance, it is common practice to merge classes with fewer samples into a single category.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Han et al. 32

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Variety in note types accounts for variety in documentation practices as notes are written by physicians, nurses, social workers, etc.

    “This variety in note types also accounts for the variety in documentation practices as they are written by physicians, nurses, social workers, etc.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Han et al. 32 consolidated their initial set of 14 annotation categories for SDoH classification into eight categories, with the less frequent ones merged into an "other-social" category.

    “Han et al. 32 employed this approach by consolidating their initial set of 14 annotation categories for SDoH classification into eight categories, with the less frequent ones merged into an “other-social” category.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Han et al. 32

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Because the authors do not constrain the SDoH factors, their model performance is a better reflection of real-world operating conditions.

    “Since we do not constrain the SDoH factors as mentioned above, our model performance is a better reflection of the real-world operating conditions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The authors cannot rule out that dataset-specific preprocessing may have inadvertently introduced biases that hindered cross-dataset generalization and performance difference between models.

    “However, we cannot rule out the possibility that these dataset-specific preprocessing may have inadvertently introduced biases that hindered cross-dataset generalization and performance difference between models.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The MIMIC-III dataset has high lexical and semantic variability, its sentences are diverse and do not follow patterns or templates, and it is mostly written in a style of reproducing conversations, which might have affected model performance.

    “MIMIC-III dataset has high lexical and semantic variability, and the sentences are diverse and do not follow any patterns/templates and is mostly written in a style of reproducing conversations with patients and their families, which might have affected the model performance.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    LLMs excel in zero- and few-shot settings on unseen tasks, making them suitable for novel SDoH extraction.

    “While LLMs excel in zero- and few-shot settings on unseen tasks, making them suitable for novel SDoH extraction, other models also show reasonable performance when evaluating generalizability on shared SDoH factors across datasets with sufficient training data.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Maintaining and training models on all classes is crucial for cross-dataset evaluation to ensure consistent and reliable assessment, as differences in the total number and types of classes would otherwise impact the evaluation process.

    “Furthermore, maintaining and training models on all classes is crucial for our cross-dataset evaluation to ensure consistent and reliable assessment, as any differences in the total number and types of classes would otherwise impact the evaluation process.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    Certain factors such as social support, isolation or adverse childhood experiences may not lend themselves well to strict entity identification as they often need to be inferred from context.

    “Furthermore, there are certain factors such as social support, isolation or adverse childhood experiences that may not lend themselves well to strict entity identification as these factors often need to be inferred from the context.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    This oversimplified approach fails to mirror the intricacies of real-world scenarios.

    “However, this oversimplified approach fails to mirror the intricacies of real-world scenarios.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The Mayo condition-specific cohort likely does not reflect the diverse SDoH profiles of Mayo's general clinical population, which may have impacted model performance on this dataset along with its small sample size and SDoH distribution.

    “This condition-specific cohort likely does not reflect the diverse SDoH profiles of Mayo’s general clinical population, which may have impacted model performance on this dataset along with its small sample size in addition to also having an impact on the SDoH distribution.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ClinicalBERT is a BERT model that has been further pre-trained on text from all note types in the MIMIC-III v1.4 database.

    “The fourth model, ClinicalBERT50, is yet another BERT model that has been further pre-trained on text from all note types in the MIMIC-III v1.4 database.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [50] Alsentzer, E. et al. Publicly available clinical BERT embeddings. Proceedings of the 2nd Clinical Natural Language Processing Workshop. 72–78 (2019). source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The annotators went through multiple rounds of training and calculated inter-annotator agreement (Cohen's Kappa) after each training round.

    “The annotators went through multiple rounds of training starting with a detailed discussion of the annotation guidelines and calculating the inter-annotator agreement (Cohen’s Kappa) after each training round.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    We randomly selected 2000 assessment notes (corresponding to 1529 patients) from HCPC.

    “We randomly selected 2000 assessment notes (corresponding to 1529 patients) and extracted the “Social history” sections (using MedSpacy sectionizer83) from these notes which are rich in SDoH information.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    After obtaining Institutional Review Board approval for the study, the datasets were annotated.

    “After obtaining the Institutional Review Board (IRB) approval for the study the datasets were annotated.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    We randomly selected and annotated 500 social work notes from MIMIC-III.

    “We randomly selected and annotated 500 such notes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The low cross-dataset performance could be attributed to differences in SDoH frequency distribution and to patterns in which SDoH factors are described differing significantly by hospital setting, practitioner preference, templates, and medical specialties.

    “This low performance could be attributed to factors such as difference in (1) SDoH frequency distribution, and (2) the patterns in which SDoH factors are described differing significantly depending on the hospital setting, preference of individual practitioner, templates in place and medical specialties.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Once a Kappa value greater than 0.7 was achieved, the remaining notes or sentences were annotated individually by the annotators and discrepancies were resolved.

    “Once a Kappa value greater than 0.7 was achieved the remaining notes/sentences were annotated individually by the annotators and any discrepancies were resolved as mentioned above.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Each corpus was randomly divided into a 7:1:2 ratio for train, validation, and test, respectively.

    “Each corpus was randomly divided into 7:1:2 ratio for train, validation, and test, respectively.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The base LLaMA model being a general-domain LLM and SDoH identification being a non-medical task might have made these models more suitable for this task than other biomedical NLP tasks requiring considerable biomedical knowledge.

    “The base LLaMA model being a general-domain LLM and the task of SDoH identification being a non-medical one might have rendered these models more suitable for this task compared to other biomedical NLP tasks which requires considerable biomedical knowledge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The requirement for informed consent was waived by all governing IRBs under exempt category 4, as the research involved only secondary use of clinical notes and did not involve any patient contact.

    “The requirement for informed consent was waived by all the governing IRBs under exempt category 4, as the research involved only secondary use of clinical notes and did not involve any patient contact.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Prior research has often constrained the number of SDoH factors within datasets due to manual annotation burden or integrated numerous factors with only a few samples into an "Other SDoH" category to potentially improve model performance.

    “Prior research has often constrained the number of SDoH factors within datasets due to the manual annotation burden or later integrated numerous factors with only a few samples into an “Other SDoH” category to potentially improve model performance32.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [32] Han, S. et al. Classifying social determinants of health from unstructured electronic health records using deep learning-based natural language processing. J. Biomed. Inf. 127, 103984 (2022). source
92 of 92

  • The paper relies on this without support.

    Clinical narratives contain more detailed characterization of several SDoH factors (e.g., living status, financial instability, social support) beyond the structured representation.

    “On the other hand, clinical narratives contain more detailed characterization of several SDoH factors (e.g., living status, financial instability, social support, etc.), beyond the structured representation.”

  • The paper relies on this without support.

    The unstructured text data often contains detailed information to describe how multiple socio-economic, behavioral factors, and situational variables interact with each other.

    “The unstructured text data often contains detailed information to describe how multiple socio-economic, behavioral factors, and situational variables interact with each other.”

  • The paper relies on this without support.

    Many academic and healthcare organizations face significant resource constraints that prohibit the utilization of LLMs at scales beyond a certain threshold.

    “We recognize that many academic and healthcare organizations face significant resource constraints that prohibit the utilization of LLMs at scales beyond a certain threshold including our own computational limitations.”

  • The paper relies on this without support.

    Many social and environmental risk factors are intricately tied to the unique circumstances, lived experiences, and cultural contexts of different communities.

    “Many social and environmental risk factors are intricately tied to the unique circumstances, lived experiences, and cultural contexts of different communities.”

  • The paper relies on this without support.

    The salience and manifestations of specific SDoH can vary drastically between patient populations.

    “The salience and manifestations of specific SDoH can vary drastically between patient populations.”

  • The paper relies on this without support.

    Extracting and classifying SDoH from EHR and clinical notes have the potential for developing effective treatment plans, improving population health, and reducing health disparities.

    “extracting and classifying social determinants of health from EHR and clinical notes have the potential for developing effective treatment plans, improving population health, and reducing health disparities.”

  • The paper relies on this without support.

    What SDoH factors are documented and where they are documented varies widely across healthcare settings and medical specialties, and the effects of these variations on model performance in a real-world scenario is rarely studied.

    “While what SDoH factors are documented and where they are documented varies widely across healthcare settings and medical specialties, the effects of these variations on the model performance in a real-world scenario is rarely studied, especially when developing generalizable models is the primary goal.”

  • The paper relies on this without support.

    The specific SDoH factors and linguistic patterns for expressing them may differ across other geographic and cultural contexts.

    “The specific SDoH factors and linguistic patterns for expressing them may differ across other geographic and cultural contexts.”

  • The paper relies on this without support.

    Differences in race, ethnicity, sex, age distributions and other traits across sites could contribute to language biases, disproportionate SDoH mention rates, or framing disparities stemming from clinician implicit biases.

    “Differences in race, ethnicity, sex, age distributions and other traits across sites could contribute to language biases, disproportionate SDoH mention rates, or framing disparities stemming from clinician implicit biases.”

  • The paper relies on this without support.

    MIMIC-III includes 2670 “social work” notes documenting the patient’s social and family history and the interactions of the social workers with the patient and family members during their stay at the hospital.

    “It includes 2670 “social work” notes documenting the patient’s social and family history and the interactions of the social workers with the patient and family members during their stay at the hospital.”
10 of 10

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Abstract overstates level 2 F1 across corpora

    Minor

    The abstract says the instruction-tuned LLM achieved micro-averaged F1 over 0.84 on level 2 corpora, but Table 1 reports 0.837 for LLaMA on the MIMIC-III level 2 corpus.

  • Abstract overstates level 1 F1 across corpora

    Minor

    The abstract says the instruction-tuned LLM achieved micro-averaged F1 over 0.9 on level 1 corpora, but Table 1 reports 0.883 for LLaMA on the MIMIC-III level 1 corpus.

  • Top-performance counts exceed the number of experiments

    Minor

    The reported model counts sum to 61 top-performance assignments, although the paper says there were 60 experiments. This would require at least one experiment to be counted for multiple models as a tie, which the text does not specify.

    Reported 60 · recomputed 61

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Dai, S. et al. in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 6437-6447.
  2. Patra, B. G. et al. Extracting social determinants of health from electronic health records using natural language processing: a systematic review - Supplementary Material. J. Am. Med Inf. Assoc. 28, 2716–2727 (2021). · DOI · Paper page

Cited by

Sorted by citations

Related

Authors
Keloth, V. K., Selek, S., Chen, Q., Gilman, C., Fu, S., Dang, Y., Chen, X., Hu, X., Zhou, Y., He, H., Fan, J. W., Wang, K., Brandt, C., Tao, C., Liu, H., & Xu, H.
Affiliations
  • Yale University (US)
  • The University of Texas Health Science Center at Houston (US)
  • Mayo Clinic (US)
Corresponding author
Xu, Hua
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2025-05-17
Volume
8
Issue
1
Pages
287-287
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-025-01645-8
PMC
PMC12084648
OpenAlex
W4410462384
Cited by
24
References
76
Citations by year
2026: 21 · 2025: 3
FWCI
9.79
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
  • Digital Commons-TMC (Texas Medical Center) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • U.S. Department of Health & Human Services | NIH | National Human Genome Research Institute (RM1HG011558)
  • U.S. Department of Health & Human Services | NIH | National Institute on Aging (R01AG080429, RF1AG072799, R01AG084236, R01AG083039)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
social determinants of health, clinical notes, large language models, cross-dataset evaluation, clinical text annotation, electronic health records, instruction-tuned LLMs

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-27.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-27.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review