CoChat
Sign Up
Cited by 358Machine Learning in HealthcareBiomedical Text Mining and OntologiesTopic Modeling

Clinical text data in machine learning: systematic review

Open access
Spasic, Irena, Nenadic, Goran·2020·JMIR Medical Informatics
DOI 10.2196/17984PMC PMC7157505
Open this paper in CoChat — free account

Abstract

BACKGROUND: Clinical narratives represent the main form of communication within health care, providing a personalized account of patient history and assessments, and offering rich information for clinical decision making. Natural language processing (NLP) has repeatedly demonstrated its feasibility to unlock evidence buried in clinical narratives. Machine learning can facilitate rapid development of NLP tools by leveraging large amounts of text data. OBJECTIVE: The main aim of this study was to provide systematic evidence on the properties of text data used to train machine learning approaches to clinical NLP. We also investigated the types of NLP tasks that have been supported by machine learning and how they can be applied in clinical practice. METHODS: Our methodology was based on the guidelines for performing systematic reviews. In August 2018, we used PubMed, a multifaceted interface, to perform a literature search against MEDLINE. We identified 110 relevant studies and extracted information about text data used to support machine learning, NLP tasks supported, and their clinical applications. The data properties considered included their size, provenance, collection methods, annotation, and any relevant statistics. RESULTS: The majority of datasets used to train machine learning models included only hundreds or thousands of documents. Only 10 studies used tens of thousands of documents, with a handful of studies utilizing more. Relatively small datasets were utilized for training even when much larger datasets were available. The main reason for such poor data utilization is the annotation bottleneck faced by supervised machine learning algorithms. Active learning was explored to iteratively sample a subset of data for manual annotation as a strategy for minimizing the annotation effort while maximizing the predictive performance of the model. Supervised learning was successfully used where clinical codes integrated with free-text notes into electronic health records were utilized as class labels. Similarly, distant supervision was used to utilize an existing knowledge base to automatically annotate raw text. Where manual annotation was unavoidable, crowdsourcing was explored, but it remains unsuitable because of the sensitive nature of data considered. Besides the small volume, training data were typically sourced from a small number of institutions, thus offering no hard evidence about the transferability of machine learning models. The majority of studies focused on text classification. Most commonly, the classification results were used to support phenotyping, prognosis, care improvement, resource management, and surveillance. CONCLUSIONS: We identified the data annotation bottleneck as one of the key obstacles to machine learning approaches in clinical NLP. Active learning and distant supervision were explored as a way of saving the annotation efforts. Future research in this field would benefit from alternatives such as data augmentation and transfer learning, or unsupervised learning, which do not require data annotation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/17984

Cites

145 of 145 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Machine learning models required structured inputs and extensive manual feature engineering to extract meaningful signals from clinical text.

    “However, these models required structured inputs and extensive manual feature engineering to extract meaningful signals from clinical text [13].”

    From Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review (2025) · cites this paper as [13]

    The Results (Availability; page not provided) say that most reviewed studies relied on manual annotation of clinical text, and the Discussion identifies annotation as a key bottleneck. Manual annotation is not the same as manual feature engineering, however, and the review does not say that machine-learning models required structured inputs. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Acquiring and labeling training datasets for supervised learning can be prohibitively expensive, especially for rare constructs.

    “However, acquiring and labeling training datasets for supervised learning can be prohibitively expensive, especially for rare constructs.”

    From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [2]

    The Results > Annotation section says that manually annotating training data for supervised learning is labor-intensive, and the Discussion calls data annotation a key bottleneck. The paper does not specifically say that acquiring and labeling data is especially expensive for rare constructs; its use of “rarely” concerns dataset reuse, not construct rarity. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The progress of applying NLP methods and resources to biomedical texts, including FH extraction, has been limited by difficulties in accessing shared tools and resources, partially caused by patient privacy and data confidentiality constraints.

    “Although the application of NLP methods and resources to biomedical texts has received increasing attention [6-8], with methods for FH extraction [9-11], the progress has been limited by difficulties in accessing shared tools and resources, partially caused by patient privacy and data confidentiality constraints.”

    From Family History Extraction From Synthetic Clinical Narratives Using Natural Language Processing: Overview and Evaluation of a Challenge Data Set and Solutions for the 2019 National NLP Clinical Challenges (n2c2)/Open Health Natural Language Processing (OHNLP) Competition (2021) · cites this paper as [7]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • The annotation bottleneck is an important barrier for the use of supervised learning in clinical applications of NLP.

    “This problem, which has been termed the “annotation bottleneck,” is an important barrier for the use of supervised learning in clinical applications of NLP.2”

    From Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions (2026) · cites this paper as [2]

  • Clinical NLP tasks are greatly limited by their overreliance on the same handful of data sources, small dataset sizes (typically thousands of examples), highly repetitive content, and low coverage of use cases.

    “However, these NLP tasks are greatly limited by their overreliance on the same handful of data sources74, small dataset sizes (typically thousands of examples)74,76, highly repetitive content77, and low coverage of use cases20.”

    From The shaky foundations of large language models and foundation models for electronic health records (2023) · cites this paper as [76]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. [139]Ho D Liang E Chen X Stoica I Abbeel P Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules Proceedings of the International Conference on Machine Learning 2019 CML'19 June 10-15, 2019 Long Beach, USA 2731 41
  2. [141]Xie Z Wang SI Li J Levy D Nie A Jurafsky D Ng A Data Noising as Smoothing in Neural Network Language Models Proceedings of the 5th International Conference on Learning Representations 2017 CLR'17 April 24-26, 2017 Toulon, France
  3. [145]Do C Ng A Transfer learning for text classification Adv Neural Inf Process Syst 2006 18 299 306

Cited by

Sorted by citations

Related

Authors
Spasic, I., & Nenadic, G.
ORCID
Venue
JMIR Medical Informatics
Publisher
JMIR Publications Inc.
Published
2020-03-31
Volume
8
Issue
3
Pages
e17984
ISSN
2291-9694 (ISSN-L)
DOI
10.2196/17984
PMC
PMC7157505
OpenAlex
W3013605954
Cited by
358
References
145
Citations by year
2026: 52 · 2025: 71 · 2024: 56 · 2023: 48 · 2022: 71 · 2021: 46 · 2020: 13
FWCI
1.54
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Medical Informatics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • ORCA Online Research @Cardiff (Cardiff University) (acceptedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Research Explorer (The University of Manchester) (submittedVersion, other-oa)
Funders
  • Medical Research Council (MR/K006665/1, MC_PC_13042)
  • Engineering and Physical Sciences Research Council (EP/N027280/1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
clinical natural language processing, machine learning training data, active learning (machine learning), distant supervision, crowdsourcing, electronic health records, text classification, phenotyping, transferability, data augmentation, unsupervised learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review