Zero-shot Learning with Minimum Instruction to Extract Social Determinants and Family History from Clinical Notes using GPT Model
Open accessAbstract
Demographics, social determinants of health, and family history documented in the unstructured text within the electronic health records are increasingly being studied to understand how this information can be utilized with the structured data to improve healthcare outcomes. After the GPT models were released, many studies have applied GPT models to extract this information from the narrative clinical notes. Different from the existing work, our research focuses on investigating the zero-shot learning on extracting this information together by providing minimum information to the GPT model. We utilize de-identified real-world clinical notes annotated for demographics, various social determinants, and family history information. Given that the GPT model might provide text different from the text in the original data, we explore two sets of evaluation metrics, including the traditional NER evaluation metrics and semantic similarity evaluation metrics, to completely understand the performance. Our results show that the GPT-3.5 method achieved an average of 0.975 F1 on demographics extraction, 0.615 F1 on social determinants extraction, and 0.722 F1 on family history extraction. We believe these results can be further improved through model fine-tuning or few-shots learning. Through the case studies, we also identified the limitations of the GPT models, which need to be addressed in future research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
More recent natural language processing (NLP)-based methods, particularly LLMs like GPT-4, have shown success in capturing SDoH information from unstructured clinical notes and patient narratives, yielding tangible benefits for patient outcomes.
“More recent natural language processing (NLP)-based methods, particularly LLMs like GPT-4, have also shown success in capturing SDoH information from unstructured clinical notes and patient narratives, yielding tangible benefits for patient outcomes [78,79].”
From Bias in medical AI: Implications for clinical decision-making (2024) · cites this paper as [79]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · see citing works
Cardiovascular-kidney-metabolic health: a presidential advisory from the American Heart Association
Ndumele, C. E., Rangaswami, J., Chow, S. L., Neeland, I. J., Tuttle, K. R., Khan, S. S., Coresh, J., Mathew, R. O., Baker-Smith, C. M., Carnethon, M. R., et al. (2023) · Circulation · doi:10.1161/cir.0000000000001184 · Cited by 2,208 · cited 1× · see citing works
“With recent advances in natural language processing, other than the rule-based models [2], [12], various transformer-based language models [12], [15], [20] have been developed to extract the social determinants and family history from the narrative clinical notes.”
Johnson, D., Goodman, R., Patrinely, J., Stone, C., Zimmerman, E., Donald, R., Chang, S., Berkowitz, S., Finn, A., Jahangir, E., et al. (2023) · doi:10.21203/rs.3.rs-2566942/v1 · Cited by 598 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · see citing works
Large language models are few-shot clinical information extractors
Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.130 · Cited by 358 · see citing works
Patra, B. G., Sharma, M. M., Vekaria, V., Adekkanattu, P., Patterson, O. V., Glicksberg, B., Lepow, L. A., Ryu, E., Biernacka, J. M., Furmanchuk, A., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab170 · Cited by 248 · see citing works
A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?Preprint
Zhang, C.-N., Zhang, C.-S., Zheng, S., Qiao, Y., Li, C.-H., Zhang, M.-C., Dam, S. K., Thwal, C. M., Tun, Y. L., Huy, L. L., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2303.11717 · Cited by 225 · see citing works
News Summarization and Evaluation in the Era of GPT-3
Goyal, T., Li, J. J., & Durrett, G. (2022) · arXiv · doi:10.48550/arxiv.2209.12356 · Cited by 175 · see citing works
Various criteria in the evaluation of biomedical named entity recognition
Tsai, R. T., Wu, S.-H., Chou, W., Lin, Y., He, D., Hsiang, J., Sung, T., & Hsu, W. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-92 · Cited by 128 · see citing works
Medical image captioning via generative pretrained transformers
Selivanov, A., Rogov, O. Y., Chesakov, D., Shelmanov, A., Fedulova, I., & Dylov, D. V. (2023) · Scientific Reports · doi:10.1038/s41598-023-31223-5 · Cited by 119 · see citing works
A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?Preprint
Zhang, C., Zhang, C., Zheng, S., Qiao, Y., Li, C., Zhang, M., Dam, S. K., Thwal, C. M., Tun, Y. L., HUY, L. L., et al. (2023) · arXiv (Cornell University) · doi:10.13140/rg.2.2.29980.16001 · Cited by 110 · see citing works
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · arXiv · doi:10.48550/arxiv.1908.10084 · Cited by 86 · see citing works
Lybarger, K., Mari Ostendorf, E., & Yetişgen, M. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103631 · Cited by 78 · see citing works
Improving Large Language Models for Clinical Named Entity Recognition via Prompt EngineeringPreprint
Hu, Y., Chen, Q., Du, J., Peng, X., Keloth, V. K., Zuo, X., Zhou, Y., Li, Z., Jiang, X., Lu, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.16416 · Cited by 62 · see citing works
Automated extraction of family history information from clinical notes.
(2014) · PubMed Central · Cited by 35 · see citing works
Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliveryPreprint
Dash, D., Thapa, R., Banda, J. M., Swaminathan, A., Cheatham, M., Kashyap, M., Kotecha, N., Chen, J. H., Gombar, S., Downing, N. L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.13714 · Cited by 33 · see citing works
Yang, X., Zhang, H., He, X., Bian, J., & Wu, Y. (2020) · JMIR Medical Informatics · doi:10.2196/22982 · Cited by 29 · cited 1× · see citing works
“With recent advances in natural language processing, other than the rule-based models [2], [12], various transformerbased language models [12], [15], [20] have been developed to extract the social determinants and family history from the narrative clinical notes.”
Improving child health through Big Data and data science
Vesoulis, Z. A., Husain, A. N., & Cole, F. S. (2022) · Pediatric Research · doi:10.1038/s41390-022-02264-9 · Cited by 27 · see citing works
Romanowski, B., Ben Abacha, A., & Fan, Y. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad071 · Cited by 23 · see citing works
A probabilistic model for identifying protein names and their name boundaries
Seki, K., & Mostafa, J. (2004) · IEEE Computational Systems Bioinformatics Conference (CSB) · doi:10.1109/csb.2003.1227325 · Cited by 22 · see citing works
Mitchell, E., Noh, J. J., Li, S., Armstrong, W., Agarwal, A., Liu, P., Finn, C., & Manning, C. D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.115 · Cited by 21 · see citing works
Prompt-based Extraction of Social Determinants of Health Using Few-shot Learning
Ramachandran, G. K., Fu, Y., Han, B., Lybarger, K., Dobbins, N., Uzuner, O., & Yetisgen, M. (2023) · Proceedings of the 5th Clinical Natural Language Processing Workshop · doi:10.18653/v1/2023.clinicalnlp-1.41 · Cited by 18 · see citing works
Rule-based extraction of family history information from clinical notes
Almeida, J. R., & Matos, S. (2020) · ACM Symposium on Applied Computing · doi:10.1145/3341105.3374000 · Cited by 12 · see citing works
Task formulation for Extracting Social Determinants of Health from Clinical NarrativesPreprint
Torii, M., Finn, I. M., Doan, S., Wang, P. P., Yang, E. W., & Zisook, D. S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.11386 · Cited by 6 · see citing works
Mitchell, E., Noh, J. J., Li, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.11875 · Cited by 1 · see citing works
Prompt-based Extraction of Social Determinants of Health Using Few-shot LearningPreprint
Ramachandran, G. K., Fu, Y., Han, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.07170
Flagged first, then by citations
1 of the 27 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [7]Jain Siddhartha, Ma Xiaofei, Deoras Anoop, and Xiang Bing. Self-consistency for open-ended generations. arXiv preprint arXiv:2307.06857, 2023. · arXiv
Cited by
Sorted by citations
Bias in medical AI: Implications for clinical decision-making
Cross, J. L., Choma, M. A., & Onofrey, J. A. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000651 · Cited by 607 · see citing works
Alkhalaf, M., Yu, P., Yin, M., & Deng, C. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104662 · Cited by 156 · see citing works
Integrating LLMs With ITS: Recent Advances, Potentials, Challenges, and Future Directions
Mahmud, D., Hajmohamed, H., Almentheri, S., Alqaydi, S., Aldhaheri, L., Khalil, R. A., & Saeed, N. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3528116 · Cited by 100 · see citing works
Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis
Yu, H., Fan, L., Li, L., Zhou, J., Ma, Z., Xian, L., Hua, W., He, S., Jin, M., Zhang, Y., et al. (2024) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-024-00171-8 · Cited by 69 · see citing works
Bai, E., Luo, X., Zhang, Z., Adelgais, K., Ali, H., Finkelstein, J., & Kutzin, J. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02197-w · Cited by 24 · see citing works
Self-citationVithanage, D., Deng, C., Wang, L., Yin, M., Alkhalaf, M., Zhang, Z., Zhu, Y., & Yu, P. (2025) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-025-00190-z · Cited by 17 · see citing works
Decoding substance use disorder severity from clinical notes using a large language model
Mahbub, M., Dams, G. M., Srinivasan, S., Rizy, C., Danciu, I., Trafton, J. A., & Knight, K. (2025) · npj Mental Health Research · doi:10.1038/s44184-024-00114-6 · Cited by 15 · see citing works
Vithanage, D., Deng, C., Wang, L., Yin, M., Alkhalaf, M., Zhang, Z., Zhu, Y., Soewargo, A. C., & Yu, P. (2024) · medRxiv · doi:10.1101/2024.06.24.24309441 · Cited by 6 · see citing works
Kaddari, Z., Hachmi, I. E., Berrich, J., Amrani, R., & Bouchentouf, T. (2024) · Applied Computer Science · doi:10.35784/acs-2024-44 · Cited by 4 · see citing works
Assessing Personalized AI Mentoring with Large Language Models in the Computing Field
Luo, X., O'Connell, S., & Mithun, S. (2025) · IEEE Symposium on Computational Intelligence in Natural Language Processing and Social Media (CI-NLPSoMe) · doi:10.1109/ci-nlpsomecompanion65206.2025.10977933 · Cited by 3 · see citing works
Self-citationFarrag, A. N., Soliman, A., Hatef, E., Goodin, A. J., & Rouhizadeh, M. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf201 · Cited by 3 · see citing works
Soley, N., Bentil, M., Shah, J., Rouhizadeh, M., & Taylor, C. O. (2025) · Scientific Reports · doi:10.1038/s41598-025-13542-x · Cited by 2 · see citing works
Rajwal, S., Pandey, A. K., Zhang, Z., Chen, Y., Liu, M. X., Das, S., Rogers, H., Sarker, A., & Xiao, Y. (2026) · Journal of Medical Internet Research · doi:10.2196/83793 · Cited by 1 · see citing works
Applications of ChatGPT in the Health Sector: A Systematic Literature Review
Martínez‐Burgos, A., Enamorado-Díaz, E., Gutiérrez, J., & García-García, J. A. (2026) · Expert Systems · doi:10.1111/exsy.70296 · Cited by 1 · see citing works
Farrag, A. N., Soliman, A. S. A., Hatef, E., Goodin, A. J., & Rouhizadeh, M. (2025) · medRxiv · doi:10.1101/2025.07.04.25330866 · Cited by 1 · see citing works
Rajwal, S., Pandey, A. K., Zhang, Z., Chen, Y., Liu, M. X., Das, S., Rogers, H., Sarker, A., & Xiao, Y. (2025) · doi:10.2196/preprints.83793
ACKE: Attribute and Context Based Knowledge Encoding
Qin, Y., & Zhang, X. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402208
Natural Language Processing to Identify Substance Use in Electronic Health Records: A Scoping Review
Azubuike, C. D., Farrag, A. N., Zandbiglari, K., Soliman, A. S. A., Bagchi, C., Mehrabi, F., Adkins, L., Rouhizadeh, M., & Goodin, A. J. (2026) · Substance Use & Addiction Journal · doi:10.1177/29767342261463377
- Authors
- Bhate, N. J., Mittal, A., He, Z., & Luo, X.
- Affiliations
- Indiana University Indianapolis (US)
- Florida State University (US)
- Oklahoma State University (US)
- Venue
- 2023 IEEE International Conference on Big Data (BigData)
- Publisher
- IEEE
- Published
- 2023-12-15
- Volume
- 2023
- Pages
- 1476-1480
- PMC
- PMC11295958
- OpenAlex
- W4391093878
- Cited by
- 18
- References
- 27
- Citations by year
- 2026: 3 · 2025: 10 · 2024: 5
- FWCI
- 3.19
- Type
- conference paper
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- IEEE International Conference on Big Data (BigData) (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- IUScholarWorks (Indiana University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Health
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- zero-shot learning, GPT models, social determinants of health, clinical notes, clinical text, named entity recognition, electronic health records, GPT-3.5
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
