CoChat
Sign Up
Cited by 41Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareBiomedical Text Mining and Ontologies

Clinical Text Datasets for Medical Artificial Intelligence and Large Language Models — A Systematic Review

Wu, Jiageng, Liu, Xiaocong, Li, Minghui, Li, Wanxin, Su, Zichang, Lin, Shixu, Garay, Lucas, Zhang, Zhiyun +5 more·2024·NEJM AI
DOI 10.1056/aira2400012
Open this paper in CoChat — free account

Abstract

Privacy and ethical considerations limit access to large-scale clinical datasets, particularly clinical text data, which contain extensive and diverse information and serve as the foundation for building clinical large language models (LLMs). The limited accessibility of clinical text data impedes the development of clinical artificial intelligence systems and hampers research participation from resource-poor regions and medical institutions, thereby exacerbating health care disparities. In this review, we conduct a global review to identify publicly available clinical text datasets and elaborate on their accessibility, diversity, and usability for clinical LLMs. We screened 3962 papers across medical (PubMed and MEDLINE) and computational linguistic academic databases (the Association for Computational Linguistics Anthology) as well as 239 tasks from prevalent medical natural language processing (NLP) challenges, such as National NLP Clinical Challenges (n2c2). We identified 192 unique clinical text datasets that claimed to be publicly available. Following an institutional review board–approved data-requesting pipeline, access was granted to fewer than half (91 of 192 [47.4%]) of the identified datasets, with an additional 14 (7.3%) datasets being available for regulated access and 87 (45.3%) datasets remaining inaccessible. The publicly available datasets cover nine languages from 14 countries and over 10 million clinical text records, which mostly (88 [95.7%]) originated from the Americas, Europe, and Asia, with none originating from Oceania or Africa, leaving these regions significantly underrepresented. Distribution differences were also evident within the focused clinical context and supported NLP tasks, with intensive care unit (18 [16.8%]), respiratory disease (13 [12.1%]), and cardiovascular disease (11 [10.3%]) gaining significant attention. Named entity recognition (23 [21.7%]), text classification (22 [20.8%]), and event extraction (12 [11.3%]) were the most explored NLP tasks on clinical text datasets. To our knowledge, this is the first systematic review to characterize publicly available clinical text datasets, the foundation of clinical LLMs, highlighting the difficulty in accessibility, underrepresentation across regions and languages, and the challenges posed by the LLMs. Sharing diversified and large-scale clinical text data is necessary, with protection to promote health care research.

Integrity signals

Registry

Crossref · 10.1056/aira2400012

Cites

61 of 61 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Advanced LLMs often rely on general domain datasets and lack exposure to real-world EHR notes, which limits their effectiveness in clinical settings.

    “Furthermore, due to data access and privacy concerns34, advanced LLMs often rely on general domain datasets and lack exposure to real-world EHR notes, which limits their effectiveness in clinical settings24,36.”

    From Probabilistic medical predictions of large language models (2024) · cites this paper as [34]

  • In the clinical domain, LLMs often underperform due to limited training data for minority languages, lack of domain knowledge, and dataset imbalances.

    “In the clinical domain, due to limited training data for minority languages, lack of domain knowledge, and dataset imbalances, LLMs often underperforms34.”

    From Probabilistic medical predictions of large language models (2024) · cites this paper as [34]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Wu, J., Liu, X., Li, M., Li, W., Su, Z., Lin, S., Garay, L., Zhang, Z., Zhang, Y., Zeng, Q., Shen, J., Yuan, C., & Yang, J.
Affiliations
  • Second Affiliated Hospital of Zhejiang University (CN)
  • Zhejiang University (CN)
  • Northwestern University (US)
  • Harvard University (US)
  • Brigham and Women's Hospital (US)
ORCID
Venue
NEJM AI
Publisher
Massachusetts Medical Society
Published
2024-05-23
Volume
1
Issue
6
Pages
AIra2400012-AIra2400012
ISSN
2836-9386 (ISSN-L)
DOI
10.1056/aira2400012
OpenAlex
W4397003497
Cited by
41
References
61
Citations by year
2026: 7 · 2025: 24 · 2024: 10
FWCI
0.94
Type
journal article
Language
en
Open access
closed
Locations
  • NEJM AI (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
Keywords
clinical large language models, artificial intelligence in healthcare, data accessibility, data privacy, health disparities, medical natural language processing, named entity recognition, text classification, event extraction, intensive care unit data, clinical data sharing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review