Evaluation of Reader Variability in the Interpretation of Follow-up CT Scans at Lung Cancer Screening
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
PURPOSE: To measure reader agreement in determining whether lung nodules detected at baseline screening computed tomography (CT) had changed at subsequent screening examinations and to evaluate the variability in recommendations for further follow-up. MATERIALS AND METHODS: All subjects were enrolled in the National Lung Screening Trial (NLST), and each participant consented to the use of their de-identified images for research purposes. The authors randomly selected 100 cases of nodules measuring at least 4.0 mm at 1-year screening CT that were considered by the original screening CT reader to be present on baseline CT scans; nodules considered by the original reader to have changed were oversampled. Selected images from each case showing the entire nodule at both examinations were preloaded on a picture archiving and communication system workstation. Nine radiologists served as readers, and they evaluated whether the nodule was present at baseline and recorded the bidimensional measurements and nodule characteristics at each examination, presence or absence of change, results of screening CT, and follow-up recommendations (high-level follow-up, low-level follow-up, no follow-up). RESULTS: On the basis of reviews during case selection, five nodules seen at follow-up were judged not to have been present at baseline; for 19 of the remaining 95 cases, at least one reader judged the nodule not to have been present at baseline. For the 76 nodules that were unanimously considered to have been present at baseline, 21%-47% (mean ± standard deviation, 30% ± 9) were judged to have grown. The κ values were similar for growth (κ = 0.55) and a positive screening result (κ = 0.51) and were lower for a change in margins and attenuation (κ = 0.27-0.31). The κ value in the recommendation of high- versus low-level follow-up was high (κ = 0.66). CONCLUSION: Reader agreement on nodule growth and screening result was moderate to substantial. Agreement on follow-up recommendations was lower.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529786 · Cited by 4,332
Cited by
Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0447-x · Cited by 2,167
Naidich, D. P., Bankier, A. A., MacMahon, H., Schaefer-Prokop, C. M., Pistolesi, M., Goo, J. M., Macchiarini, P., Crapo, J. D., Herold, C. J., Austin, J. H., et al. (2013) · Radiology · doi:10.1148/radiol.12120628 · Cited by 1,000
Lung Cancer Screening, Version 3.2018, NCCN Clinical Practice Guidelines in Oncology
Wood, D. E., Kazerooni, E. A., Baum, S. L., Eapen, G. A., Ettinger, D. S., Hou, L., Jackman, D. M., Klippenstein, D., Kumar, R., Lackner, R. P., et al. (2018) · Journal of the National Comprehensive Cancer Network · doi:10.6004/jnccn.2018.0020 · Cited by 611
Kumar, R., Wang, W., Kumar, J., Yang, T., Khan, A., Ali, W., & Ali, I. (2021) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2020.101812 · Cited by 172
Ziegelmayer, S., Graf, M., Makowski, M., Gawlitza, J., & Gassert, F. (2022) · Cancers · doi:10.3390/cancers14071729 · Cited by 63
“Even after a recent revision of the reporting system, observer variability will remain a relevant limitation [9,10].”
Background
Metadata
- Authors
- Singh, S., Pinsky, P., Fineberg, N. S., Gierada, D. S., Garg, K., Sun, Y., & Nath, P. H.
- Venue
- Radiology
- Published
- 2011-04
- Volume
- 259
- Issue
- 1
- Pages
- 263-270
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2135799475
- Cited by
- 79
- References
- 2
- FWCI
- 6.19
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.