CoChat
Sign Up
Cited by 142Machine Learning in HealthcareBiomedical Text Mining and OntologiesText and Document Classification Technologies

Natural language-based machine learning models for the annotation of clinical radiology reports

Zech, John, Pain, Margaret, Titano, Joseph, Badgeley, Marcus, Schefflein, Javin, Su, Andres, Costa, Anthony, Bederson, Joshua +2 more·2018·Radiology
DOI 10.1148/radiol.2018171093
Open this paper in CoChat — free account

Abstract

Purpose To compare different methods for generating features from radiology reports and to develop a method to automatically identify findings in these reports. Materials and Methods In this study, 96 303 head computed tomography (CT) reports were obtained. The linguistic complexity of these reports was compared with that of alternative corpora. Head CT reports were preprocessed, and machine-analyzable features were constructed by using bag-of-words (BOW), word embedding, and Latent Dirichlet allocation–based approaches. Ultimately, 1004 head CT reports were manually labeled for findings of interest by physicians, and a subset of these were deemed critical findings. Lasso logistic regression was used to train models for physician-assigned labels on 602 of 1004 head CT reports (60%) using the constructed features, and the performance of these models was validated on a held-out 402 of 1004 reports (40%). Models were scored by area under the receiver operating characteristic curve (AUC), and aggregate AUC statistics were reported for (a) all labels, (b) critical labels, and (c) the presence of any critical finding in a report. Sensitivity, specificity, accuracy, and F1 score were reported for the best performing model’s (a) predictions of all labels and (b) identification of reports containing critical findings. Results The best-performing model (BOW with unigrams, bigrams, and trigrams plus average word embeddings vector) had a held-out AUC of 0.966 for identifying the presence of any critical head CT finding and an average 0.957 AUC across all head CT findings. Sensitivity and specificity for identifying the presence of any critical finding were 92.59% (175 of 189) and 89.67% (191 of 213), respectively. Average sensitivity and specificity across all findings were 90.25% (1898 of 2103) and 91.72% (18 351 of 20 007), respectively. Simpler BOW methods achieved results competitive with those of more sophisticated approaches, with an average AUC for presence of any critical finding of 0.951 for unigram BOW versus 0.966 for the best-performing model. The Yule I of the head CT corpus was 34, markedly lower than that of the Reuters corpus (at 103) or I2B2 discharge summaries (at 271), indicating lower linguistic complexity. Conclusion Automated methods can be used to identify findings in radiology reports. The success of this approach benefits from the standardized language of these reports. With this method, a large labeled corpus can be generated for applications such as deep learning. © RSNA, 2018 Online supplemental material is available for this article.

Integrity signals

Registry

Crossref · 10.1148/radiol.2018171093

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Deep learning methods have been used for automated generation of radiology reports, typically incorporating LSTM network models to generate textual paragraphs, and to identify findings in radiology reports.

    “Recently, deep learning methods have also been used for automated generation of radiology reports , typically incorporating long-short-term-memory (LSTM) network models to generate the textual paragraphs [314, 315, 316, 317], and also to identify findings in radiology reports [318, 319, 320].”

    From An overview of deep learning in medical imaging focusing on MRI (2019) · cites this paper as [319]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Zech, J., Pain, M., Titano, J., Badgeley, M., Schefflein, J., Su, A., Costa, A., Bederson, J., Lehar, J., & Oermann, E. K.
Affiliations
  • Boston University (US)
  • From the Departments of Radiology (J.Z., J.T., J.S., A.S.) and Neurosurgery (M.P., M.B., A.C., J.B., E.K.O.), Icahn School of Medicine, 1 Gustave Levy Pl, New York, NY 10029; and Department of Bioengineering and Bioinformatics, Boston University, Boston, Mass (J.L.).
ORCID
Venue
Radiology
Publisher
Radiological Society of North America (RSNA)
Published
2018-05
Volume
287
Issue
2
Pages
570-580
ISSN
  • 0033-8419 (ISSN-L)
  • 1527-1315
DOI
10.1148/radiol.2018171093
OpenAlex
W2785863263
Cited by
142
References
37
Citations by year
2026: 4 · 2025: 13 · 2024: 8 · 2023: 18 · 2022: 14 · 2021: 28 · 2020: 27 · 2019: 21 · 2018: 9
FWCI
11.98
Type
journal article
Language
en
Open access
closed
Locations
  • Radiology (publishedVersion)
  • PubMed (publishedVersion)
Funders
State University of New York
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
Keywords
head CT, bag-of-words features, word embeddings, latent Dirichlet allocation, LASSO logistic regression, linguistic complexity
MeSH terms
Machine Learning, Humans, Natural Language Processing, Radiology, Sensitivity and Specificity, Tomography, X-Ray Computed, Databases, Factual, Area Under Curve, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review