CoChat

Detecting Laterality Errors in Combined Radiographic Studies by Enhancing the Traditional Approach With GPT-4o: Algorithm Development and Multisite Internal Validation

Open access

Published version of a preprint

The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index; 2 of the 46 sources its claims cite were checked for support.

Abstract

Laterality errors in radiology reports can endanger patient safety. Effective methods for screening for laterality errors in combined radiographic reports, which combine multiple studies into one, remain unexplored. First, we define and analyze the unstudied combined radiographic report format and its challenges. Second, we introduce a clinically deployable ensemble method (rule-based+GPT-4o), evaluated on large-scale, real-world, imbalanced data. Third, we demonstrate significant performance gaps between real-world imbalanced and synthetic balanced datasets, highlighting limitations of the benchmarking methodology commonly used in current studies. This retrospective study analyzed deidentified English radiology reports containing laterality terms in order. We split the data into TrainVal (combined training and validation dataset), Test-1 (both real-world, imbalanced), and Test-2 (synthetic, balanced). Test-1 comes from a distinct branch. Experiment 1 compared the baseline, workaround, and GPT-4o-augmented rule-based methods. Experiment 2 compared the rule-based method with the highest recall to fine-tuned RoBERTa, ClinicalBERT, and GPT-4o models. As of July 2024, our dataset included 10,000 real-world and 889 synthetic radiology reports. The laterality error rate in real-world reports was 1.20% (120/10,000), significantly higher in combined (103/7000, 1.47%) than in noncombined reports (17/3000, 0.57%; difference=0.90%; z=3.81; P < .001).

Overview

CoChat Summary

TL;DR

This retrospective multisite study tested laterality-error detection in radiology reports, focusing on a newly defined “combined report” format that merges multiple studies into one report. A hybrid rule-based + GPT-4o approach performed best on real-world imbalanced data, and the paper shows that synthetic balanced benchmarks substantially overestimate performance.

Key findings

  • The dataset included 10,000 real-world radiographic studies and 889 synthetic studies; real-world laterality errors were 1.20% (120/10,000) overall, with a higher rate in combined reports (1.47%, 103/7000) than noncombined reports (0.57%, 17/3000), a difference of 0.90% (z=3.81; P<.001).
  • In experiment 1, recall differed across the 3 rule-based versions (Q=6.0; P=.0498), and the rule-based + GPT-4o version had the highest recall (average rank 1), significantly better than baseline (average rank 3; P=.04).
  • In experiment 2 on Test-1 (real-world, imbalanced), rule-based + GPT information extraction outperformed pure deep learning: precision 0.696, recall 0.889, F1 0.780 versus GPT-4o 0.219/0.889/0.352, ClinicalBERT 0.047/0.667/0.088, and RoBERTa F1 0.000.
  • On Test-2 (synthetic, balanced), performance was much higher for several models: rule-based + GPT 0.996/0.925/0.959, GPT-4o 0.979/0.953/0.966, ClinicalBERT 0.984/0.749/0.851, RoBERTa F1 0.013. The paper reports strong precision drops for GPT-4o and ClinicalBERT on real-world imbalanced data.
  • In multivariable logistic regression, Test-1 membership and order count per study independently predicted false positives for both models: for GPT-4o OR 239.89 and 1.79; for ClinicalBERT OR 1924.07 and 2.50.

Methods

  • Retrospective multisite internal validation using deidentified electronic health records from 3 hospital branches (2012–2024), limited to English x-ray reports with orders specifying laterality; Chinese labor health exams were excluded.
  • Data were split into TrainVal, Test-1 (real-world, imbalanced, from a distinct branch), and Test-2 (synthetic, balanced). A board-certified radiologist annotated all reports, with a second radiologist independently annotating a 500-sample subsample.
  • The main analysis treated laterality detection as binary text classification and compared a rule-based parser, a GPT-4o–augmented rule-based method, fine-tuned RoBERTa, ClinicalBERT, and GPT-4o few-shot prompting; statistical testing included Friedman/Nemenyi, 2-proportion z tests, and logistic regression.

Limitations

  • The paper states that its metadata-based method does not use original images, so it cannot detect some laterality errors (for example, image-based mislabeling) and cannot fully rule out false positives from minimal contralateral-side images.
  • The authors note the study compares orders and free-text reports rather than findings and impressions, and it does not include CT, MRI, or ultrasound reports; these differences may affect comparability and generalizability.
  • The paper also notes that only 1 board-certified radiologist performed the primary annotations, which can introduce bias despite a near-perfect subsample agreement (κ=0.998); this is explicitly acknowledged by the paper.
  • Evident from the text: this is an internal validation study in one hospital system, so external generalizability remains uncertain, especially because reporting styles and class imbalance likely vary across sites.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/76384

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Research flow. TrainVal: combined training and validation dataset.
Figure 1 Research flow. TrainVal: combined training and validation dataset.CC BY · PubMed Central
Figure 2 A high-confidence false-positive prediction from ClinicalBERT (softmax probability≥0.9). Corresponding orders were anteroposterior and lateral views of the bilateral ankles, wrists, and cervical spine. Word attribution analysis shows that ClinicalBERT overlooked important keywords, such as “Lt” and assigned attention to irrelevant words, such as “otherwise” and “obvious.”.
Figure 2 A high-confidence false-positive prediction from ClinicalBERT (softmax probability≥0.9). Corresponding orders were anteroposterior and lateral views of the bilateral ankles, wrists, and cervical spine. Word attribution analysis shows that ClinicalBERT overlooked important keywords, such as “Lt” and assigned attention to irrelevant words, such as “otherwise” and “obvious.”.CC BY · PubMed Central
Figure 3 A conceptual message box for detecting laterality errors using our method for the example in Table 1. Line 4 is incompatible with the paragraph heading at line 3, which is left shoulder only. Our method permits a human-readable error message with line-level alignment among the problematic sentences, the GPT-extracted entities (not shown), and the parser-mapped anatomical parts with laterality.
Figure 3 A conceptual message box for detecting laterality errors using our method for the example in Table 1. Line 4 is incompatible with the paragraph heading at line 3, which is left shoulder only. Our method permits a human-readable error message with line-level alignment among the problematic sentences, the GPT-extracted entities (not shown), and the parser-mapped anatomical parts with laterality.CC BY · PubMed Central

Verification

Claims the paper makes

2 Valid · 1 Not valid · 16 Not checked · 152 Own results and interpretations

66 more claims

Create a free account to see all 171 claims and to verify the 18 references still open.

See all claims — free account
  • Probably not a substantial claim. 53
  • It may state two separate claims as one. 61

Create a free account to see these claims.

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentz = 3.81reported p < .001 · recomputed p < .001

    “z=3.81, p < .001”

  • Consistentz = 3.81reported p < .001 · recomputed p < .001

    “(z=3.81, p < .001”

2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Citations

Cites

Research works printed but not linked: 2

  1. Christensen, JD, 2006, AMIA Annu Symp Proc, 2006, 136
  2. 10.5555/3600270.3602070 · doi:10.5555/3600270.3602070

Metadata

Authors
Weng, K.-H., Chou, Y.-C., Kuo, Y.-T., Hsieh, T.-J., & Liu, C.-F.
Affiliations
  • Chi Mei Medical Center (TW)
  • National Sun Yat-sen University (TW)
Corresponding author
Liu, Chung-Feng
Venue
JMIR formative research
Published
2025-10-29
Volume
9
Pages
e76384
ISSN
2561-326X (ISSN-L)
OpenAlex
W4415677754
Cited by
0
References
29
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Formative Research (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Chi Mei Medical Center
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
Keywords
GPT-4o, rule-based extraction, ensemble methods, class imbalance, synthetic data, patient safety, RoBERTa, ClinicalBERT, error cancellation, false positives, internal validation, benchmarking methodology
MeSH terms
Algorithms, Humans, Radiography, Retrospective Studies, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.