The Subgroup Imperative: Chest Radiograph Classifier Generalization Gaps in Patient, Setting, and Pathology Subgroups

Abstract

Purpose To externally test four chest radiograph classifiers on a large, diverse, real-world dataset with robust subgroup analysis. Materials and Methods In this retrospective study, adult posteroanterior chest radiographs (January 2016–December 2020) and associated radiology reports from Trillium Health Partners in Ontario, Canada, were extracted and de-identified. An open-source natural language processing tool was locally validated and used to generate ground truth labels for the 197 540-image dataset based on the associated radiology report. Four classifiers generated predictions on each chest radiograph. Performance was evaluated using accuracy, positive predictive value, negative predictive value, sensitivity, specificity, F1 score, and Matthews correlation coefficient for the overall dataset and for patient, setting, and pathology subgroups. Results Classifiers demonstrated 68%–77% accuracy, 64%–75% sensitivity, and 82%–94% specificity on the external testing dataset. Algorithms showed decreased sensitivity for solitary findings (43%–65%), patients younger than 40 years (27%–39%), and patients in the emergency department (38%–60%) and decreased specificity on normal chest radiographs with support devices (59%–85%). Differences in sex and ancestry represented movements along an algorithm’s receiver operating characteristic curve. Conclusion Performance of deep learning chest radiograph classifiers was subject to patient, setting, and pathology factors, demonstrating that subgroup analysis is necessary to inform implementation and monitor ongoing performance to ensure optimal quality, safety, and equity. Keywords: Conventional Radiography, Thorax, Ethics, Supervised Learning, Convolutional Neural Network (CNN), Machine Learning Algorithms Supplemental material is available for this article. © RSNA, 2023 See also the commentary by Huisman and Hannink in this issue.