Interrater Agreement in the Evaluation of Discrepant Imaging Findings With the Radpeer System
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: The Radpeer system is central to the quality assurance process in many radiology practices. Previous studies have shown poor agreement between physicians in the evaluation of their peers. The purpose of this study was to assess the reliability of the Radpeer scoring system. MATERIALS AND METHODS: A sample of 25 discrepant cases was extracted from our quality assurance database. Images were made anonymous; associated reports and identities of interpreting radiologists were removed. Indications for the studies and descriptions of the discrepancies were provided. Twenty-one subspecialist attending radiologists rated the cases using the Radpeer scoring system. Multirater kappa statistics were used to assess interrater agreement, both with the standard scoring system and with dichotomized scores to reflect the practice of further review for cases rated 3 and 4. Subgroup analyses were conducted to assess subspecialist evaluation of cases. RESULTS: Interrater agreement was slight to fair compared with that expected by chance. For the group of 21 raters, the kappa values were 0.11 (95% CI, 0.06-0.16) with the standard scoring system and 0.20 (95% CI, 0.13-0.27) with dichotomized scores. There was disagreement about whether a discrepancy had occurred in 20 cases. Subgroup analyses did not reveal significant differences in the degree of interrater agreement. CONCLUSION: The identification of discrepant interpretations is valuable for the education of individual radiologists and for larger-scale quality assurance and quality improvement efforts. Our results show that a ratings-based peer review system is unreliable and subjective for the evaluation of discrepant interpretations. Resources should be devoted to developing more robust and objective assessment procedures, particularly those with clear quality improvement goals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529786 · Cited by 4,332 · cited 4×
“Kappa statistics were interpreted for strength of agreement by use of the widely cited scale of Landis and Koch [21].”
MethodInfluentialThe Reliability of Peer Assessments: A Meta-Analysis
Goldman, R. L. (1994) · Evaluation & the Health Professions · doi:10.1177/016327879401700101 · Cited by 62 · cited 2×
“Previous studies have shown poor agreement between physicians in the evaluation of their peers [12–15].”
Result
Cited by
Error and discrepancy in radiology: inevitable or avoidable?
Brady, A. P. (2017) · Insights into Imaging · doi:10.1007/s13244-016-0534-1 · Cited by 570
Added value of double reading in diagnostic radiology,a systematic review
Geijer, H., & Geijer, M. (2018) · Insights into Imaging · doi:10.1007/s13244-018-0599-0 · Cited by 117
Khorashad, B. S., Baron-Cohen, S., Roshan, G. M., Kazemian, M., Khazai, L., Aghili, Z., Talaei, A., & Afkhamizadeh, M. (2015) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-015-2427-4 · Cited by 67
Inter-rater reliability in quality assurance (QA) of pediatric chest X-rays
Hlabangana, L. T., Elsingergy, M., Ahmed, A., Boschoff, P. E., Goodier, M., Bove, M., & Andronikou, S. (2021) · Journal of Medical Imaging and Radiation Sciences · doi:10.1016/j.jmir.2021.04.002 · Cited by 11
Metadata
- Authors
- Bender, L. C., Linnau, K. F., Meier, E. N., Anzai, Y., & Gunn, M. L.
- Venue
- American Journal of Roentgenology
- Publisher
- American Roentgen Ray Society
- Published
- 2012-12
- Volume
- 199
- Issue
- 6
- Pages
- 1320-1327
- ISSN
- 0002-9580 (ISSN-L)
- 0361-803X
- 1546-3141
- OpenAlex
- W2113891910
- Cited by
- 67
- References
- 2
- FWCI
- 2.23
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.