Detecting Laterality Errors in Combined Radiographic Studies by Enhancing the Traditional Approach With GPT-4o: Algorithm Development and Multisite Internal Validation
Open accessPublished version of a preprint
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index; 2 of the 46 sources its claims cite were checked for support.
Abstract
Laterality errors in radiology reports can endanger patient safety. Effective methods for screening for laterality errors in combined radiographic reports, which combine multiple studies into one, remain unexplored. First, we define and analyze the unstudied combined radiographic report format and its challenges. Second, we introduce a clinically deployable ensemble method (rule-based+GPT-4o), evaluated on large-scale, real-world, imbalanced data. Third, we demonstrate significant performance gaps between real-world imbalanced and synthetic balanced datasets, highlighting limitations of the benchmarking methodology commonly used in current studies. This retrospective study analyzed deidentified English radiology reports containing laterality terms in order. We split the data into TrainVal (combined training and validation dataset), Test-1 (both real-world, imbalanced), and Test-2 (synthetic, balanced). Test-1 comes from a distinct branch. Experiment 1 compared the baseline, workaround, and GPT-4o-augmented rule-based methods. Experiment 2 compared the rule-based method with the highest recall to fine-tuned RoBERTa, ClinicalBERT, and GPT-4o models. As of July 2024, our dataset included 10,000 real-world and 889 synthetic radiology reports. The laterality error rate in real-world reports was 1.20% (120/10,000), significantly higher in combined (103/7000, 1.47%) than in noncombined reports (17/3000, 0.57%; difference=0.90%; z=3.81; P < .001).
Overview
CoChat Summary
TL;DR
This retrospective multisite study tested laterality-error detection in radiology reports, focusing on a newly defined “combined report” format that merges multiple studies into one report. A hybrid rule-based + GPT-4o approach performed best on real-world imbalanced data, and the paper shows that synthetic balanced benchmarks substantially overestimate performance.
Key findings
- The dataset included 10,000 real-world radiographic studies and 889 synthetic studies; real-world laterality errors were 1.20% (120/10,000) overall, with a higher rate in combined reports (1.47%, 103/7000) than noncombined reports (0.57%, 17/3000), a difference of 0.90% (z=3.81; P<.001).
- In experiment 1, recall differed across the 3 rule-based versions (Q=6.0; P=.0498), and the rule-based + GPT-4o version had the highest recall (average rank 1), significantly better than baseline (average rank 3; P=.04).
- In experiment 2 on Test-1 (real-world, imbalanced), rule-based + GPT information extraction outperformed pure deep learning: precision 0.696, recall 0.889, F1 0.780 versus GPT-4o 0.219/0.889/0.352, ClinicalBERT 0.047/0.667/0.088, and RoBERTa F1 0.000.
- On Test-2 (synthetic, balanced), performance was much higher for several models: rule-based + GPT 0.996/0.925/0.959, GPT-4o 0.979/0.953/0.966, ClinicalBERT 0.984/0.749/0.851, RoBERTa F1 0.013. The paper reports strong precision drops for GPT-4o and ClinicalBERT on real-world imbalanced data.
- In multivariable logistic regression, Test-1 membership and order count per study independently predicted false positives for both models: for GPT-4o OR 239.89 and 1.79; for ClinicalBERT OR 1924.07 and 2.50.
Methods
- Retrospective multisite internal validation using deidentified electronic health records from 3 hospital branches (2012–2024), limited to English x-ray reports with orders specifying laterality; Chinese labor health exams were excluded.
- Data were split into TrainVal, Test-1 (real-world, imbalanced, from a distinct branch), and Test-2 (synthetic, balanced). A board-certified radiologist annotated all reports, with a second radiologist independently annotating a 500-sample subsample.
- The main analysis treated laterality detection as binary text classification and compared a rule-based parser, a GPT-4o–augmented rule-based method, fine-tuned RoBERTa, ClinicalBERT, and GPT-4o few-shot prompting; statistical testing included Friedman/Nemenyi, 2-proportion z tests, and logistic regression.
Limitations
- The paper states that its metadata-based method does not use original images, so it cannot detect some laterality errors (for example, image-based mislabeling) and cannot fully rule out false positives from minimal contralateral-side images.
- The authors note the study compares orders and free-text reports rather than findings and impressions, and it does not include CT, MRI, or ultrasound reports; these differences may affect comparability and generalizability.
- The paper also notes that only 1 board-certified radiologist performed the primary annotations, which can introduce bias despite a near-perfect subsample agreement (κ=0.998); this is explicitly acknowledged by the paper.
- Evident from the text: this is an internal validation study in one hospital system, so external generalizability remains uncertain, especially because reporting styles and class imbalance likely vary across sites.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
2 Valid · 1 Not valid · 16 Not checked · 152 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The laterality error rate in real-world reports was 1.20% (120/10,000)”
§ Detecting Laterality Errors in Combined Radiographic Studies by Enhancing the Traditional Approach With GPT-4o: Algorithm Development and Multisite Internal Validation > Abstract > Results As of July 2024, our dataset included 10,000 real-world and 889 synthetic radiology reports. The laterality error rate in real-world reports was 1.20% (120/10,000), significantly higher in combined (103/7000, 1.47%) than in noncombined reports (17/3000, 0.57%; difference=0.90%; z=3.81; P<.001). In experiment 1, recall differed significantly among the 3 versions of rule-based methods (Q=6.0; P=.0498, Friedman test). The rule-based+GPT-4o method had the highest recall (average rank=1), significantly better than the baseline (average rank=3; P=.04, Nemenyi test). Most (5/6) of the false positives introduced by the GPT-4o information extraction were due to parser limitations hidden by error cancellation. In experiment 2, on Test-1, rule-based+GPT-4o (precision=0.696; recall=0.889; F1-score=0.780) outperformed GPT-4o (precision=0.219; recall=0.889; F1-score=0.352), ClinicalBERT (precision=0.047; recall=0.667; F1-score=0.088), and RoBERTa (F1-score=0.000). On Test-2, rule-based+GPT-4o (precision=0.996; recall=0.925; F1-score=0.959) and GPT-4o (precision=0.979; recall=0.953; F1-score=0.966) outperformed ClinicalBERT (precision=0.984; recall=0.749; F1-score=0.851) and RoBERTa (F1-score=0.013). Both ClinicalBERT and GPT-4o exhibited notable declines in precision on TrainVal and Test-1 compared to Test-2. Both Test-1 data membership (GPT-4o: odds ratio [OR] 239.89, 95% CI 111.05-518.01; P<.001; ClinicalBERT: OR 1924.07, 95% CI 687.46-5383.99; P<.001) and order count per study (GPT-4o: OR 1.79, 95% CI 1.38-2.31; P<.001; ClinicalBERT: OR 2.50, 95% CI 1.64-3.80; P<.001) independently predicted false positive errors in multivariate logistic regression. In subgroup analysis, all models showed reduced precision and F1 in combined-study subgroups.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“significantly higher in combined (103/7000, 1.47%) than in noncombined reports (17/3000, 0.57%; difference=0.90%; z=3.81; P<.001)”
§ Detecting Laterality Errors in Combined Radiographic Studies by Enhancing the Traditional Approach With GPT-4o: Algorithm Development and Multisite Internal Validation > Abstract > Results As of July 2024, our dataset included 10,000 real-world and 889 synthetic radiology reports. The laterality error rate in real-world reports was 1.20% (120/10,000), significantly higher in combined (103/7000, 1.47%) than in noncombined reports (17/3000, 0.57%; difference=0.90%; z=3.81; P<.001). In experiment 1, recall differed significantly among the 3 versions of rule-based methods (Q=6.0; P=.0498, Friedman test). The rule-based+GPT-4o method had the highest recall (average rank=1), significantly better than the baseline (average rank=3; P=.04, Nemenyi test). Most (5/6) of the false positives introduced by the GPT-4o information extraction were due to parser limitations hidden by error cancellation. In experiment 2, on Test-1, rule-based+GPT-4o (precision=0.696; recall=0.889; F1-score=0.780) outperformed GPT-4o (precision=0.219; recall=0.889; F1-score=0.352), ClinicalBERT (precision=0.047; recall=0.667; F1-score=0.088), and RoBERTa (F1-score=0.000). On Test-2, rule-based+GPT-4o (precision=0.996; recall=0.925; F1-score=0.959) and GPT-4o (precision=0.979; recall=0.953; F1-score=0.966) outperformed ClinicalBERT (precision=0.984; recall=0.749; F1-score=0.851) and RoBERTa (F1-score=0.013). Both ClinicalBERT and GPT-4o exhibited notable declines in precision on TrainVal and Test-1 compared to Test-2. Both Test-1 data membership (GPT-4o: odds ratio [OR] 239.89, 95% CI 111.05-518.01; P<.001; ClinicalBERT: OR 1924.07, 95% CI 687.46-5383.99; P<.001) and order count per study (GPT-4o: OR 1.79, 95% CI 1.38-2.31; P<.001; ClinicalBERT: OR 2.50, 95% CI 1.64-3.80; P<.001) independently predicted false positive errors in multivariate logistic regression. In subgroup analysis, all models showed reduced precision and F1 in combined-study subgroups.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In subgroup analysis, all models showed reduced precision and F1 in combined-study subgroups.”
§ Detecting Laterality Errors in Combined Radiographic Studies by Enhancing the Traditional Approach With GPT-4o: Algorithm Development and Multisite Internal Validation > Abstract > Results As of July 2024, our dataset included 10,000 real-world and 889 synthetic radiology reports. The laterality error rate in real-world reports was 1.20% (120/10,000), significantly higher in combined (103/7000, 1.47%) than in noncombined reports (17/3000, 0.57%; difference=0.90%; z=3.81; P<.001). In experiment 1, recall differed significantly among the 3 versions of rule-based methods (Q=6.0; P=.0498, Friedman test). The rule-based+GPT-4o method had the highest recall (average rank=1), significantly better than the baseline (average rank=3; P=.04, Nemenyi test). Most (5/6) of the false positives introduced by the GPT-4o information extraction were due to parser limitations hidden by error cancellation. In experiment 2, on Test-1, rule-based+GPT-4o (precision=0.696; recall=0.889; F1-score=0.780) outperformed GPT-4o (precision=0.219; recall=0.889; F1-score=0.352), ClinicalBERT (precision=0.047; recall=0.667; F1-score=0.088), and RoBERTa (F1-score=0.000). On Test-2, rule-based+GPT-4o (precision=0.996; recall=0.925; F1-score=0.959) and GPT-4o (precision=0.979; recall=0.953; F1-score=0.966) outperformed ClinicalBERT (precision=0.984; recall=0.749; F1-score=0.851) and RoBERTa (F1-score=0.013). Both ClinicalBERT and GPT-4o exhibited notable declines in precision on TrainVal and Test-1 compared to Test-2. Both Test-1 data membership (GPT-4o: odds ratio [OR] 239.89, 95% CI 111.05-518.01; P<.001; ClinicalBERT: OR 1924.07, 95% CI 687.46-5383.99; P<.001) and order count per study (GPT-4o: OR 1.79, 95% CI 1.38-2.31; P<.001; ClinicalBERT: OR 2.50, 95% CI 1.64-3.80; P<.001) independently predicted false positive errors in multivariate logistic regression. In subgroup analysis, all models showed reduced precision and F1 in combined-study subgroups.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In experiment 1, recall differed significantly among the 3 versions of rule-based methods (Q=6.0; P=.0498, Friedman test).”
§ Detecting Laterality Errors in Combined Radiographic Studies by Enhancing the Traditional Approach With GPT-4o: Algorithm Development and Multisite Internal Validation > Abstract > Results As of July 2024, our dataset included 10,000 real-world and 889 synthetic radiology reports. The laterality error rate in real-world reports was 1.20% (120/10,000), significantly higher in combined (103/7000, 1.47%) than in noncombined reports (17/3000, 0.57%; difference=0.90%; z=3.81; P<.001). In experiment 1, recall differed significantly among the 3 versions of rule-based methods (Q=6.0; P=.0498, Friedman test). The rule-based+GPT-4o method had the highest recall (average rank=1), significantly better than the baseline (average rank=3; P=.04, Nemenyi test). Most (5/6) of the false positives introduced by the GPT-4o information extraction were due to parser limitations hidden by error cancellation. In experiment 2, on Test-1, rule-based+GPT-4o (precision=0.696; recall=0.889; F1-score=0.780) outperformed GPT-4o (precision=0.219; recall=0.889; F1-score=0.352), ClinicalBERT (precision=0.047; recall=0.667; F1-score=0.088), and RoBERTa (F1-score=0.000). On Test-2, rule-based+GPT-4o (precision=0.996; recall=0.925; F1-score=0.959) and GPT-4o (precision=0.979; recall=0.953; F1-score=0.966) outperformed ClinicalBERT (precision=0.984; recall=0.749; F1-score=0.851) and RoBERTa (F1-score=0.013). Both ClinicalBERT and GPT-4o exhibited notable declines in precision on TrainVal and Test-1 compared to Test-2. Both Test-1 data membership (GPT-4o: odds ratio [OR] 239.89, 95% CI 111.05-518.01; P<.001; ClinicalBERT: OR 1924.07, 95% CI 687.46-5383.99; P<.001) and order count per study (GPT-4o: OR 1.79, 95% CI 1.38-2.31; P<.001; ClinicalBERT: OR 2.50, 95% CI 1.64-3.80; P<.001) independently predicted false positive errors in multivariate logistic regression. In subgroup analysis, all models showed reduced precision and F1 in combined-study subgroups.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most (5/6) of the false positives introduced by the GPT-4o information extraction were due to parser limitations hidden by error cancellation.”
§ Detecting Laterality Errors in Combined Radiographic Studies by Enhancing the Traditional Approach With GPT-4o: Algorithm Development and Multisite Internal Validation > Abstract > Results As of July 2024, our dataset included 10,000 real-world and 889 synthetic radiology reports. The laterality error rate in real-world reports was 1.20% (120/10,000), significantly higher in combined (103/7000, 1.47%) than in noncombined reports (17/3000, 0.57%; difference=0.90%; z=3.81; P<.001). In experiment 1, recall differed significantly among the 3 versions of rule-based methods (Q=6.0; P=.0498, Friedman test). The rule-based+GPT-4o method had the highest recall (average rank=1), significantly better than the baseline (average rank=3; P=.04, Nemenyi test). Most (5/6) of the false positives introduced by the GPT-4o information extraction were due to parser limitations hidden by error cancellation. In experiment 2, on Test-1, rule-based+GPT-4o (precision=0.696; recall=0.889; F1-score=0.780) outperformed GPT-4o (precision=0.219; recall=0.889; F1-score=0.352), ClinicalBERT (precision=0.047; recall=0.667; F1-score=0.088), and RoBERTa (F1-score=0.000). On Test-2, rule-based+GPT-4o (precision=0.996; recall=0.925; F1-score=0.959) and GPT-4o (precision=0.979; recall=0.953; F1-score=0.966) outperformed ClinicalBERT (precision=0.984; recall=0.749; F1-score=0.851) and RoBERTa (F1-score=0.013). Both ClinicalBERT and GPT-4o exhibited notable declines in precision on TrainVal and Test-1 compared to Test-2. Both Test-1 data membership (GPT-4o: odds ratio [OR] 239.89, 95% CI 111.05-518.01; P<.001; ClinicalBERT: OR 1924.07, 95% CI 687.46-5383.99; P<.001) and order count per study (GPT-4o: OR 1.79, 95% CI 1.38-2.31; P<.001; ClinicalBERT: OR 2.50, 95% CI 1.64-3.80; P<.001) independently predicted false positive errors in multivariate logistic regression. In subgroup analysis, all models showed reduced precision and F1 in combined-study subgroups.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
66 more claims
Create a free account to see all 171 claims and to verify the 18 references still open.
- Probably not a substantial claim. 53
- It may state two separate claims as one. 61
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentz = 3.81reported p < .001 · recomputed p < .001
“z=3.81, p < .001”
- Consistentz = 3.81reported p < .001 · recomputed p < .001
“(z=3.81, p < .001”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Self-check results appear here once an independent check has confirmed them.
Citations
Cites
Sensitivity analysis in observational research: Introducing the E-value
VanderWeele, T. J., & Ding, P. (2017) · Annals of Internal Medicine · doi:10.7326/m16-2607 · Cited by 6,350 · cited 1×
“We performed the sensitivity analysis using the E value [29] to evaluate the robustness of our findings to potential unmeasured confounding, with the calculation based on the observed odds ratio (OR) point estimate for rare outcome ( 15”
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · cited 1×
“We used the RoBERTa large model fine-tuned on the Multi-Genre Natural Language Inference corpus, which focuses on the sentence contradiction task and evaluates a model’s ability to determine whether a premise sentence entails, contradicts, or is neutral concerning a hypothesis sentence [25].”
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618 · cited 1×
“Min et al [20] used a fine-tuned ClinicalBERT [24] binary classifier to detect inconsistencies, including lesion locations, between findings and conclusions in chest x-ray reports, evaluating largely on synthetically generated errors plus a smaller, less balanced, manually edited set.”
Bhayana, R. (2024) · Radiology · doi:10.1148/radiol.232756 · Cited by 329 · cited 1×
“However, LLMs also have well-documented limitations and potential pitfalls, such as difficulties with the complex, multistep reasoning tasks that are often required in advanced applications [31].”
Wrong-Side/Wrong-Site, Wrong-Procedure, and Wrong-Patient Adverse Events: Are They Preventable?
Seiden, S. C. (2006) · Archives of Surgery · doi:10.1001/archsurg.141.9.931 · Cited by 283
Incidence, Patterns, and Prevention of Wrong-Site Surgery
Kwaan, M. R. (2006) · Archives of Surgery · doi:10.1001/archsurg.141.4.353 · Cited by 277
Fool Me Twice: Delayed Diagnoses in Radiology With Emphasis on Perpetuated Errors
Kim, Y. W., & Mansfield, L. T. (2014) · American Journal of Roentgenology · doi:10.2214/ajr.13.11493 · Cited by 259 · cited 2×
“For example, Kim and Mansfield [2] identified 12 primary error types, while Pinto et al [3] grouped them into 4 distinct categories.”
Better Zero-Shot Reasoning with Role-Play Prompting
Kong, A., Zhao, S., Chen, H., Li, Q., Qin, Y., Sun, R., Zhou, X., Wang, E., & Dong, X. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.228 · Cited by 171
Potential of GPT-4 for Detecting Errors in Radiology Reports: Implications for Reporting Accuracy
Gertz, R. J., Dratsch, T., Bunck, A. C., Lennartz, S., Iuga, A.-I., Hellmich, M. G., Persigehl, T., Pennig, L., Gietzen, C. H., Fervers, P., et al. (2024) · Radiology · doi:10.1148/radiol.232714 · Cited by 143 · cited 3×
“Gertz et al [21] introduced 5 common error types, including side confusion, into radiographic and XSL • FO RenderX cross-sectional reports and found that GPT-4 underperformed the best-performing radiologist overall but was comparable to other radiologists.”
ChatGPT in radiology: A systematic review of performance, pitfalls, and future perspectives
Keshavarz, P., Bagherieh, S., Nabipoorashrafi, S. A., Chalian, H., Rahsepar, A. A., Kim, G. H. J., Hassani, C., Raman, S. S., & Bedayat, A. (2024) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2024.04.003 · Cited by 132 · cited 2×
“Based on previous literature [30] and our experimental results, we infer that our hybrid approach is more effective because it leverages GPT’s strengths in transforming and summarizing reports into clearer formats while mitigating its weaknesses in the complex multistep reasoning required for our…”
Research works printed but not linked: 2
- Christensen, JD, 2006, AMIA Annu Symp Proc, 2006, 136
- 10.5555/3600270.3602070 · doi:10.5555/3600270.3602070
Cited by
Yi, W., Chen, Y., Feng, X., & Xia, T. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1927284
Metadata
- Authors
- Weng, K.-H., Chou, Y.-C., Kuo, Y.-T., Hsieh, T.-J., & Liu, C.-F.
- Affiliations
- Chi Mei Medical Center (TW)
- National Sun Yat-sen University (TW)
- Corresponding author
- Liu, Chung-Feng
- Venue
- JMIR formative research
- Publisher
- JMIR Publications Inc.
- Published
- 2025-10-29
- Volume
- 9
- Pages
- e76384
- ISSN
- 2561-326X (ISSN-L)
- PMC
- PMC12612642
- OpenAlex
- W4415677754
- Semantic Scholar
- eb32b1ecde82900b47cf5d2aa788bb5410a24d03
- Cited by
- 0
- References
- 29
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Formative Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Chi Mei Medical Center
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Keywords
- GPT-4o, rule-based extraction, ensemble methods, class imbalance, synthetic data, patient safety, RoBERTa, ClinicalBERT, error cancellation, false positives, internal validation, benchmarking methodology
- MeSH terms
- Algorithms, Humans, Radiography, Retrospective Studies, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.