Cited by 124Artificial Intelligence in Healthcare and EducationAI in cancer detectionBiomedical Text Mining and Ontologies

Extracting structured information from unstructured histopathology reports using generative pre‐trained transformer 4 ( GPT ‐4)

Open access
Truhn, Daniel, Loeffler, Chiara ML, Müller‐Franzes, Gustav, Nebelung, Sven, Hewitt, Katherine J, Brandner, Sebastian, Bressem, Keno K, Foersch, Sebastian +1 more·2024·The Journal of Pathology

Abstract

Deep learning applied to whole-slide histopathology images (WSIs) has the potential to enhance precision oncology and alleviate the workload of experts. However, developing these models necessitates large amounts of data with ground truth labels, which can be both time-consuming and expensive to obtain. Pathology reports are typically unstructured or poorly structured texts, and efforts to implement structured reporting templates have been unsuccessful, as these efforts lead to perceived extra workload. In this study, we hypothesised that large language models (LLMs), such as the generative pre-trained transformer 4 (GPT-4), can extract structured data from unstructured plain language reports using a zero-shot approach without requiring any re-training. We tested this hypothesis by utilising GPT-4 to extract information from histopathological reports, focusing on two extensive sets of pathology reports for colorectal cancer and glioblastoma. We found a high concordance between LLM-generated structured data and human-generated structured data. Consequently, LLMs could potentially be employed routinely to extract ground truth data for machine learning from unstructured pathology reports in the future. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/path.6232

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. Bubeck S, Chandrasekaran V, Eldan R, et al. Sparks of artificial general intelligence: early experiments with GPT-4. arXiv 2023; 2303.12712 [cs.CL]. [Not peer reviewed].
  2. Chen L, Zaharia M, Zou J. How is ChatGPT's behavior changing over time? arXiv 2023; 2307.09009 [cs.CL]. [Not peer reviewed].
  3. Touvron H, Lavril T, Izacard G, et al. LLaMA: open and efficient foundation language models. arXiv 2023; 2302.13971 [cs.CL]. [Not peer reviewed].
  4. Han T, Adams LC, Papaioannou J-M, et al. MedAlpaca -an open- source collection of medical conversational AI models and training data. arXiv 2023; 2304.08247 [cs.CL]. [Not peer reviewed]. Large language models for pathology reports 9 © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland. www.pathsoc.org J Pathol 2023 www.thejournalofpathology.com

Cited by

Sorted by citations

Authors
Truhn, D., Loeffler, C. M., Müller‐Franzes, G., Nebelung, S., Hewitt, K. J., Brandner, S., Bressem, K. K., Foersch, S., & Kather, J. N.
Affiliations
  • Universitätsklinikum Aachen (DE)
  • Else Kroener Fresenius Center for Digital Health Technical University Dresden Dresden Germany
  • Universitätsklinikum Erlangen (DE)
  • Humboldt-Universität zu Berlin (DE)
  • University Medical Center of the Johannes Gutenberg University Mainz (DE)
  • University of Leeds (GB)
Corresponding author
Kather, Jakob Nikolas
Venue
The Journal of Pathology
Publisher
Wiley
Published
2023-12-14
Volume
262
Issue
3
Pages
310-319
ISSN
  • 0022-3417 (ISSN-L)
  • 1096-9896
OpenAlex
W4389792145
Cited by
124
References
15
Citations by year
2026: 37 · 2025: 58 · 2024: 29
FWCI
4.95
Type
journal article
Language
en
Open access
Locations
  • The Journal of Pathology (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Qucosa (Saxon State and University Library Dresden) (submittedVersion, cc-by)
  • RWTH Publications (RWTH Aachen) (submittedVersion, other-oa)
  • RWTH Publications (RWTH Aachen)
Funders
  • Deutscher Akademischer Austauschdienst (57616814)
  • National Institute for Health and Care Research (NIHR213331)
  • Leeds Biomedical Research Centre
  • Deutsche Forschungsgemeinschaft (FO 942/2‐1, NE 2136/3‐1, TR 1700/7‐1)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
Keywords
GPT-4, large language models, structured data extraction, histopathology reports, zero-shot learning, colorectal cancer, glioblastoma, ground truth annotation, whole slide images, precision oncology
MeSH terms
Machine Learning, Glioblastoma, United Kingdom, Humans, Precision Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review