CoChat

Heterogeneity and predictors of the effects of AI assistance on radiologists

Open access

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.

Abstract

The integration of artificial intelligence (AI) in medical image interpretation requires effective collaboration between clinicians and AI algorithms. Although previous studies demonstrated the potential of AI assistance in improving overall clinician performance, the individual impact on clinicians remains unclear. This large-scale study examined the heterogeneous effects of AI assistance on 140 radiologists across 15 chest X-ray diagnostic tasks and identified predictors of these effects. Surprisingly, conventional experience-based factors, such as years of experience, subspecialty and familiarity with AI tools, fail to reliably predict the impact of AI assistance. Additionally, lower-performing radiologists do not consistently benefit more from AI assistance, challenging prevailing assumptions. Instead, we found that the occurrence of AI errors strongly influences treatment outcomes, with inaccurate AI predictions adversely affecting radiologist performance on the aggregate of all pathologies and on half of the individual pathologies investigated. Our findings highlight the importance of personalized approaches to clinician–AI collaboration and the importance of accurate AI models. By understanding the factors that shape the effectiveness of AI assistance, this study provides valuable insights for targeted implementation of AI, enabling maximum benefits for individual clinicians in clinical practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41591-024-02850-w

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Experience-based radiologist characteristics as predictors of treatment effect on all pathologies aggregated. a, Heterogeneous treatment effects of subgroups of radiologists on all pathologies aggregated. The treatment effects were shrunk toward the mean using the empirical Bayes method. Statistically significant heterogeneity was observed between subgroups (P = 3.50 × 10−34), where radiologists with a higher than median treatment effect had a significantly higher treatment effect of 0.472 (95% CI: 0.403 to 0.541) than those with a treatment effect lower than or equal to the median of −0.357 (95% CI: −0.429 to 0.284). A two-sided, unpaired t-test between the two subgroups of treatment effects was conducted. Testing for all pathologies aggregated did not constitute multiple hypothesis testing. The error bars show 95% CIs. b, Treatment effects on all pathologies aggregated of subgroups of radiologists based on combined characteristics of years of experience, subspecialty in thoracic radiology and experience with AI tools. No statistically significant difference was observed between the lower predicted treatment effect subgroup, 0.091 (95% CI: −0.231 to 0.413), and the higher predicted treatment effect subgroup, 0.070 (95% CI: −0.243 to 0.383) (P > 0.05). The Wald test was used to test regression coefficients that estimate treatment effects against the null hypothesis of joint equality among treatment effects of different subgroups. Details of the statistical models are available in the Methods. There are 136 radiologists with available survey data on the three characteristics. The error bars show 95% CIs. NS indicates no statistical significance (P > 0.05). c, Same subfigures as in a and b based on years of experience (left), subspecialty in thoracic radiology (middle) and experience with AI tools (right), respectively. The same statistical test as in b was used. d, Same subfigure as in b for AUROC on all pathologies aggregated. The same statistical test as in b was used. e, Same subfigures as in d based on years of experience (left), subspecialty in thoracic radiology (middle) and experience with AI tools (right), respectively. The same statistical test as in b was used.
Fig. 1 Experience-based radiologist characteristics as predictors of treatment effect on all pathologies aggregated. a, Heterogeneous treatment effects of subgroups of radiologists on all pathologies aggregated. The treatment effects were shrunk toward the mean using the empirical Bayes method. Statistically significant heterogeneity was observed between subgroups (P = 3.50 × 10−34), where radiologists with a higher than median treatment effect had a significantly higher treatment effect of 0.472 (95% CI: 0.403 to 0.541) than those with a treatment effect lower than or equal to the median of −0.357 (95% CI: −0.429 to 0.284). A two-sided, unpaired t-test between the two subgroups of treatment effects was conducted. Testing for all pathologies aggregated did not constitute multiple hypothesis testing. The error bars show 95% CIs. b, Treatment effects on all pathologies aggregated of subgroups of radiologists based on combined characteristics of years of experience, subspecialty in thoracic radiology and experience with AI tools. No statistically significant difference was observed between the lower predicted treatment effect subgroup, 0.091 (95% CI: −0.231 to 0.413), and the higher predicted treatment effect subgroup, 0.070 (95% CI: −0.243 to 0.383) (P > 0.05). The Wald test was used to test regression coefficients that estimate treatment effects against the null hypothesis of joint equality among treatment effects of different subgroups. Details of the statistical models are available in the Methods. There are 136 radiologists with available survey data on the three characteristics. The error bars show 95% CIs. NS indicates no statistical significance (P > 0.05). c, Same subfigures as in a and b based on years of experience (left), subspecialty in thoracic radiology (middle) and experience with AI tools (right), respectively. The same statistical test as in b was used. d, Same subfigure as in b for AUROC on all pathologies aggregated. The same statistical test as in b was used. e, Same subfigures as in d based on years of experience (left), subspecialty in thoracic radiology (middle) and experience with AI tools (right), respectively. The same statistical test as in b was used.CC BY · PubMed Central
Fig. 2 Unassisted error as a predictor of treatment effect. a, Unassisted error is a poor predictor of treatment effect on all pathologies aggregated (left). Without split sampling, there is a hallucinated association between treatment effect and unassisted error (right). The binscatter plots contain five evenly spaced bins containing 5,190 data points in total. The gray regression line is fitted on the raw data. The five bins are presented as −0.254 (95% CI: −0.701 to 0.211), −0.205 (95% CI: −0.431 to 0.017), 0.590 (95% CI: 0.291 to 0.878), 0.301 (95% CI: −0.487 to 1.093) and −0.419 (95% CI: −2.178 to 1.125) (left) and −1.148 (95% CI: −1.473 to −0.842), 0.078 (95% CI: −0.165 to 0.313), 0.661 (95% CI: 0.327 to 0.971), 0.979 (95% CI: 0.269 to 1.772) and 0.409 (95% CI: −0.982 to 1.919) (right). The blue dotted regression line is the final regression output after adjusting for attenuation bias. The translucent band around the blue regression line represents the 95% CI. * and ** indicate statistically significant difference from zero at a significance level of 0.05 and 0.01, respectively. NS indicates no statistical significance (P > 0.05). b, Unassisted error is a poor predictor of treatment effect on each individual pathology. The binscatter plots are designed in the same way as those in a. The significance of the slope coefficients is determined through the Benjamini–Hochberg procedure, respectively, to correct for multiple hypothesis testing (15 individual pathologies) at a false discovery rate of 0.05 (*) and 0.01 (**). NS indicates no statistical significance (Benjamini–Hochberg-adjusted P > 0.05). c, Same subfigures as in a for AUROC on all pathologies aggregated. Unassisted AUROC is a poor predictor of treatment effect on AUROC on all pathologies aggregated (left). Without split sampling, there is a hallucinated association between treatment effect on AUROC and unassisted AUROC (right). The five bins are presented as −0.073 (95% CI: −0.196 to 0.044), 0.057 (95% CI: −0.014 to 0.157), 0.065 (95% CI: 0.036 to 0.095), 0.022 (95% CI: 0.006 to 0.038) and 0.033 (95% CI: 0.016 to 0.052) (left) and 0.119 (95% CI: 0.058 to 0.187), 0.074 (95% CI: 0.034 to 0.111), 0.065 (95% CI: 0.050 to 0.079), 0.018 (95% CI: 0.007 to 0.028) and 0.001 (95% CI: −0.011 to 0.012) (right). w/o, without.
Fig. 2 Unassisted error as a predictor of treatment effect. a, Unassisted error is a poor predictor of treatment effect on all pathologies aggregated (left). Without split sampling, there is a hallucinated association between treatment effect and unassisted error (right). The binscatter plots contain five evenly spaced bins containing 5,190 data points in total. The gray regression line is fitted on the raw data. The five bins are presented as −0.254 (95% CI: −0.701 to 0.211), −0.205 (95% CI: −0.431 to 0.017), 0.590 (95% CI: 0.291 to 0.878), 0.301 (95% CI: −0.487 to 1.093) and −0.419 (95% CI: −2.178 to 1.125) (left) and −1.148 (95% CI: −1.473 to −0.842), 0.078 (95% CI: −0.165 to 0.313), 0.661 (95% CI: 0.327 to 0.971), 0.979 (95% CI: 0.269 to 1.772) and 0.409 (95% CI: −0.982 to 1.919) (right). The blue dotted regression line is the final regression output after adjusting for attenuation bias. The translucent band around the blue regression line represents the 95% CI. * and ** indicate statistically significant difference from zero at a significance level of 0.05 and 0.01, respectively. NS indicates no statistical significance (P > 0.05). b, Unassisted error is a poor predictor of treatment effect on each individual pathology. The binscatter plots are designed in the same way as those in a. The significance of the slope coefficients is determined through the Benjamini–Hochberg procedure, respectively, to correct for multiple hypothesis testing (15 individual pathologies) at a false discovery rate of 0.05 (*) and 0.01 (**). NS indicates no statistical significance (Benjamini–Hochberg-adjusted P > 0.05). c, Same subfigures as in a for AUROC on all pathologies aggregated. Unassisted AUROC is a poor predictor of treatment effect on AUROC on all pathologies aggregated (left). Without split sampling, there is a hallucinated association between treatment effect on AUROC and unassisted AUROC (right). The five bins are presented as −0.073 (95% CI: −0.196 to 0.044), 0.057 (95% CI: −0.014 to 0.157), 0.065 (95% CI: 0.036 to 0.095), 0.022 (95% CI: 0.006 to 0.038) and 0.033 (95% CI: 0.016 to 0.052) (left) and 0.119 (95% CI: 0.058 to 0.187), 0.074 (95% CI: 0.034 to 0.111), 0.065 (95% CI: 0.050 to 0.079), 0.018 (95% CI: 0.007 to 0.028) and 0.001 (95% CI: −0.011 to 0.012) (right). w/o, without.CC BY · PubMed Central
Fig. 3 Expected performance of radiologists after receiving AI assistance. a, Higher-performing radiologists are still higher performing after receiving AI assistance on all pathologies aggregated. The binscatter plots contain five evenly spaced bins containing 5,190 data points in total. The gray regression line is fitted on the raw data. The five bins are presented as 7.950 (95% CI: 7.579 to 8.339), 8.569 (95% CI: 8.342 to 8.827), 10.197 (95% CI: 9.870 to 10.531), 12.013 (95% CI: 11.245 to 12.857) and 14.870 (95% CI: 13.418 to 16.358). The red dotted regression line is the final regression output after adjusting for attenuation bias. The translucent band around the red regression line represents the 95% CI. * and ** indicate statistically significant difference from zero at a significance level of 0.05 and 0.01, respectively. b, Higher-performing radiologists are still higher performing after receiving AI assistance on each individual pathology except for atelectasis, pneumothorax and shoulder fracture, where the regression coefficients for the slope are not statistically significantly different from zero. The binscatter plots are designed in the same way as those in a. The five bins for abnormal are presented as 22.712 (95% CI: 21.050 to 24.323), 28.470 (95% CI: 27.411 to 29.528), 33.359 (95% CI: 32.298 to 34.497), 41.729 (95% CI: 40.406 to 43.045) and 43.904 (95% CI: 39.488 to 48.428). The four bins for airspace opacity are presented as 15.590 (95% CI: 14.809 to 16.401), 17.940 (95% CI: 17.198 to 18.718), 22.414 (95% CI: 20.828 to 24.185) and 41.076 (95% CI: 33.894 to 48.061). The five bins for atelectasis are presented as 11.550 (95% CI: 10.558 to 12.505), 10.638 (95% CI: 10.030 to 11.221), 11.366 (95% CI: 10.070 to 12.691), 9.777 (95% CI: 7.909 to 11.676) and 12.262 (95% CI: 9.144 to 15.681). The five bins for bacterial/lobar pneumonia are presented as 4.191 (95% CI: 3.645 to 4.745), 4.443 (95% CI: 4.069 to 4.842), 5.474 (95% CI: 4.499 to 6.499), 6.619 (95% CI: 4.202 to 9.317) and 9.861 (95% CI: 7.620 to 12.169). The five bins for cardiomediastinal abnormality are presented as 14.061 (95% CI: 13.295 to 14.817), 16.268 (95% CI: 15.609 to 16.867), 21.241 (95% CI: 19.556 to 22.881), 24.820 (95% CI: 20.349 to 29.464) and 34.358 (95% CI: 30.263 to 38.553). The five bins for cardiomegaly are presented as 10.169 (95% CI: 9.615 to 10.727), 14.611 (95% CI: 13.737 to 15.516), 19.483 (95% CI: 16.558 to 22.551), 39.736 (95% CI: 32.091 to 46.737) and 26.732 (95% CI: 17.879 to 35.100). The five bins for consolidation are presented as 5.553 (95% CI: 4.727 to 6.387), 6.230 (95% CI: 5.826 to 6.604), 7.219 (95% CI: 6.447 to 8.093), 9.631 (95% CI: 7.575 to 11.707) and 11.915 (95% CI: 9.683 to 14.186). The five bins for edema are presented as 7.761 (95% CI: 7.127 to 8.419), 12.777 (95% CI: 11.909 to 13.616), 18.254 (95% CI: 16.937 to 19.523), 19.268 (95% CI: 16.246 to 22.671) and 39.059 (95% CI: 27.886 to 49.837). The four bins for lesion are presented as 3.185 (95% CI: 2.973 to 3.386), 4.507 (95% CI: 3.856 to 5.170), 2.738 (95% CI: 1.476 to 4.203) and 22.835 (95% CI: 13.974 to 32.555). The five bins for pleural effusion are presented as 3.404 (95% CI: 2.794 to 4.077), 3.473 (95% CI: 2.950 to 4.013), 4.401 (95% CI: 3.644 to 5.173), 5.940 (95% CI: 4.794 to 7.119) and 6.073 (95% CI: 4.431 to 7.915). The five bins for pleural other are presented as 0.410 (95% CI: 0.348 to 0.473), 0.729 (95% CI: 0.529 to 0.938), 1.202 (95% CI: 0.626 to 2.120), 1.668 (95% CI: 0.549 to 3.153) and 6.883 (95% CI: 3.940 to 10.317). The five bins for pneumothorax are presented as 0.689 (95% CI: 0.534 to 0.880), 0.675 (95% CI: 0.493 to 0.891), 1.235 (95% CI: 0.697 to 2.004), 1.425 (95% CI: 0.166 to 3.263) and 1.345 (95% CI: 0.712 to 2.059). The five bins for rib fracture are presented as 2.614 (95% CI: 2.246 to 3.019), 3.405 (95% CI: 2.902 to 3.942), 3.357 (95% CI: 2.699 to 4.112), 2.707 (95% CI: 1.309 to 4.579) and 5.386 (95% CI: 3.032 to 8.266). The five bins for shoulder fracture are presented as 0.594 (95% CI: 0.434 to 0.768), 1.093 (95% CI: 0.582 to 1.765), 0.712 (95% CI: 0.477 to 0.968), 0.171 (95% CI: 0.012 to 0.462) and 0.050 (95% CI: 0.000 to 0.118). The five bins for support device hardware are presented as 9.389 (95% CI: 8.799 to 10.016), 12.652 (95% CI: 11.702 to 13.586), 14.477 (95% CI: 11.126 to 18.418), 36.781 (95% CI: 30.346 to 43.310) and 45.214 (95% CI: 35.945 to 54.983). The significance of the slope coefficients is determined through the Benjamini–Hochberg procedure, respectively, to account for multiple hypothesis testing (15 individual pathologies) at a false discovery rate of 0.05 (*) and 0.01 (**). NS indicates no statistical significance (Benjamini–Hochberg-adjusted P > 0.05). c, Same subfigure as in a for AUROC on all pathologies aggregated. Higher-performing radiologists as measured by AUROC are still higher performing after receiving AI assistance on all pathologies aggregated. The five bins are presented as 0.770 (95% CI: 0.697 to 0.834), 0.814 (95% CI: 0.775 to 0.855), 0.874 (95% CI: 0.859 to 0.888), 0.889 (95% CI: 0.877 to 0.900) and 0.924 (95% CI: 0.911 to 0.936).
Fig. 3 Expected performance of radiologists after receiving AI assistance. a, Higher-performing radiologists are still higher performing after receiving AI assistance on all pathologies aggregated. The binscatter plots contain five evenly spaced bins containing 5,190 data points in total. The gray regression line is fitted on the raw data. The five bins are presented as 7.950 (95% CI: 7.579 to 8.339), 8.569 (95% CI: 8.342 to 8.827), 10.197 (95% CI: 9.870 to 10.531), 12.013 (95% CI: 11.245 to 12.857) and 14.870 (95% CI: 13.418 to 16.358). The red dotted regression line is the final regression output after adjusting for attenuation bias. The translucent band around the red regression line represents the 95% CI. * and ** indicate statistically significant difference from zero at a significance level of 0.05 and 0.01, respectively. b, Higher-performing radiologists are still higher performing after receiving AI assistance on each individual pathology except for atelectasis, pneumothorax and shoulder fracture, where the regression coefficients for the slope are not statistically significantly different from zero. The binscatter plots are designed in the same way as those in a. The five bins for abnormal are presented as 22.712 (95% CI: 21.050 to 24.323), 28.470 (95% CI: 27.411 to 29.528), 33.359 (95% CI: 32.298 to 34.497), 41.729 (95% CI: 40.406 to 43.045) and 43.904 (95% CI: 39.488 to 48.428). The four bins for airspace opacity are presented as 15.590 (95% CI: 14.809 to 16.401), 17.940 (95% CI: 17.198 to 18.718), 22.414 (95% CI: 20.828 to 24.185) and 41.076 (95% CI: 33.894 to 48.061). The five bins for atelectasis are presented as 11.550 (95% CI: 10.558 to 12.505), 10.638 (95% CI: 10.030 to 11.221), 11.366 (95% CI: 10.070 to 12.691), 9.777 (95% CI: 7.909 to 11.676) and 12.262 (95% CI: 9.144 to 15.681). The five bins for bacterial/lobar pneumonia are presented as 4.191 (95% CI: 3.645 to 4.745), 4.443 (95% CI: 4.069 to 4.842), 5.474 (95% CI: 4.499 to 6.499), 6.619 (95% CI: 4.202 to 9.317) and 9.861 (95% CI: 7.620 to 12.169). The five bins for cardiomediastinal abnormality are presented as 14.061 (95% CI: 13.295 to 14.817), 16.268 (95% CI: 15.609 to 16.867), 21.241 (95% CI: 19.556 to 22.881), 24.820 (95% CI: 20.349 to 29.464) and 34.358 (95% CI: 30.263 to 38.553). The five bins for cardiomegaly are presented as 10.169 (95% CI: 9.615 to 10.727), 14.611 (95% CI: 13.737 to 15.516), 19.483 (95% CI: 16.558 to 22.551), 39.736 (95% CI: 32.091 to 46.737) and 26.732 (95% CI: 17.879 to 35.100). The five bins for consolidation are presented as 5.553 (95% CI: 4.727 to 6.387), 6.230 (95% CI: 5.826 to 6.604), 7.219 (95% CI: 6.447 to 8.093), 9.631 (95% CI: 7.575 to 11.707) and 11.915 (95% CI: 9.683 to 14.186). The five bins for edema are presented as 7.761 (95% CI: 7.127 to 8.419), 12.777 (95% CI: 11.909 to 13.616), 18.254 (95% CI: 16.937 to 19.523), 19.268 (95% CI: 16.246 to 22.671) and 39.059 (95% CI: 27.886 to 49.837). The four bins for lesion are presented as 3.185 (95% CI: 2.973 to 3.386), 4.507 (95% CI: 3.856 to 5.170), 2.738 (95% CI: 1.476 to 4.203) and 22.835 (95% CI: 13.974 to 32.555). The five bins for pleural effusion are presented as 3.404 (95% CI: 2.794 to 4.077), 3.473 (95% CI: 2.950 to 4.013), 4.401 (95% CI: 3.644 to 5.173), 5.940 (95% CI: 4.794 to 7.119) and 6.073 (95% CI: 4.431 to 7.915). The five bins for pleural other are presented as 0.410 (95% CI: 0.348 to 0.473), 0.729 (95% CI: 0.529 to 0.938), 1.202 (95% CI: 0.626 to 2.120), 1.668 (95% CI: 0.549 to 3.153) and 6.883 (95% CI: 3.940 to 10.317). The five bins for pneumothorax are presented as 0.689 (95% CI: 0.534 to 0.880), 0.675 (95% CI: 0.493 to 0.891), 1.235 (95% CI: 0.697 to 2.004), 1.425 (95% CI: 0.166 to 3.263) and 1.345 (95% CI: 0.712 to 2.059). The five bins for rib fracture are presented as 2.614 (95% CI: 2.246 to 3.019), 3.405 (95% CI: 2.902 to 3.942), 3.357 (95% CI: 2.699 to 4.112), 2.707 (95% CI: 1.309 to 4.579) and 5.386 (95% CI: 3.032 to 8.266). The five bins for shoulder fracture are presented as 0.594 (95% CI: 0.434 to 0.768), 1.093 (95% CI: 0.582 to 1.765), 0.712 (95% CI: 0.477 to 0.968), 0.171 (95% CI: 0.012 to 0.462) and 0.050 (95% CI: 0.000 to 0.118). The five bins for support device hardware are presented as 9.389 (95% CI: 8.799 to 10.016), 12.652 (95% CI: 11.702 to 13.586), 14.477 (95% CI: 11.126 to 18.418), 36.781 (95% CI: 30.346 to 43.310) and 45.214 (95% CI: 35.945 to 54.983). The significance of the slope coefficients is determined through the Benjamini–Hochberg procedure, respectively, to account for multiple hypothesis testing (15 individual pathologies) at a false discovery rate of 0.05 (*) and 0.01 (**). NS indicates no statistical significance (Benjamini–Hochberg-adjusted P > 0.05). c, Same subfigure as in a for AUROC on all pathologies aggregated. Higher-performing radiologists as measured by AUROC are still higher performing after receiving AI assistance on all pathologies aggregated. The five bins are presented as 0.770 (95% CI: 0.697 to 0.834), 0.814 (95% CI: 0.775 to 0.855), 0.874 (95% CI: 0.859 to 0.888), 0.889 (95% CI: 0.877 to 0.900) and 0.924 (95% CI: 0.911 to 0.936).CC BY · PubMed Central
Fig. 4 AI error as a predictor of treatment effect. a, AI has greater treatment effects on radiologists when the AI assistance has lower error on all pathologies aggregated. The five bins are presented as 0.679 (95% CI: 0.492 to 0.865), −1.509 (95% CI: −2.267 to −0.750), −3.556 (95% CI: −4.878 to −2.235), −6.569 (95% CI: −8.764 to −4.374) and −16.845 (95% CI: −24.288 to −9.403). The error bars show 95% CIs. The blue lines show the overall treatment effect across AI error. * and ** indicate statistically significant difference among subgroups of different AI error through a joint equality test at a significance level of 0.05 and 0.01, respectively. b, AI has greater treatment effects on radiologists when the AI assistance has lower error on abnormal, airspace opacity, cardiomediastinal abnormality, cardiomegaly, pleural effusion, pleural other, pneumothorax and support device hardware. The reverse trend holds for bacterial/lobar pneumonia and consolidation. The bar plots are designed in the same way as those in a. The significance of the subgroup joint test is determined through the Benjamini–Hochberg procedure, respectively, to correct for multiple hypothesis testing (15 individual pathologies) at a false discovery rate of 0.05 (*) and 0.01 (**). NS indicates no statistical significance (Benjamini–Hochberg-adjusted P > 0.05). c, AI has greater treatment effects on AUROC on radiologists when the AI assistance has an absolute error in the range [20, 100], whereas the trend is unclear with absolute error in the range [0, 20].
Fig. 4 AI error as a predictor of treatment effect. a, AI has greater treatment effects on radiologists when the AI assistance has lower error on all pathologies aggregated. The five bins are presented as 0.679 (95% CI: 0.492 to 0.865), −1.509 (95% CI: −2.267 to −0.750), −3.556 (95% CI: −4.878 to −2.235), −6.569 (95% CI: −8.764 to −4.374) and −16.845 (95% CI: −24.288 to −9.403). The error bars show 95% CIs. The blue lines show the overall treatment effect across AI error. * and ** indicate statistically significant difference among subgroups of different AI error through a joint equality test at a significance level of 0.05 and 0.01, respectively. b, AI has greater treatment effects on radiologists when the AI assistance has lower error on abnormal, airspace opacity, cardiomediastinal abnormality, cardiomegaly, pleural effusion, pleural other, pneumothorax and support device hardware. The reverse trend holds for bacterial/lobar pneumonia and consolidation. The bar plots are designed in the same way as those in a. The significance of the subgroup joint test is determined through the Benjamini–Hochberg procedure, respectively, to correct for multiple hypothesis testing (15 individual pathologies) at a false discovery rate of 0.05 (*) and 0.01 (**). NS indicates no statistical significance (Benjamini–Hochberg-adjusted P > 0.05). c, AI has greater treatment effects on AUROC on radiologists when the AI assistance has an absolute error in the range [20, 100], whereas the trend is unclear with absolute error in the range [0, 20].CC BY · PubMed Central
Fig. 5 AI signed error as a predictor of treatment effect. a, AI has greater treatment effects on radiologists when the AI assistance underestimates probabilities, rather than overestimates probabilities, given the same absolute error on all pathologies aggregated. The nine bins are presented as −3.560 (95% CI: −9.472 to 2.353), 2.089 (95% CI: 0.334 to 3.844), 2.698 (95% CI: 1.497 to 3.899), 2.483 (95% CI: 1.897 to 3.070), 0.398 (95% CI: 0.235 to 0.561), −2.933 (95% CI: −3.787 to −2.079), −5.018 (95% CI: −6.516 to −3.519), −6.968 (95% CI: −9.346 to −4.591) and −16.845 (95% CI: −24.288 to −9.403). The error bars show 95% CIs. The blue bars show the overall treatment effect across AI error. * and ** indicate statistically significant difference among subgroups of different AI signed error through a joint equality test at a significance level of 0.05 and 0.01, respectively. b, AI has greater treatment effects on radiologists when the AI assistance underestimates probabilities, rather than overestimates probabilities, given the same absolute error on airspace opacity, atelectasis, cardiomegaly, consolidation and lesion. The bar plots are designed in the same way as those in a. The significance of the subgroup joint test is determined through the Benjamini–Hochberg procedure, respectively, to correct for multiple hypothesis testing (15 individual pathologies) at a false discovery rate of 0.05 (*) and 0.01 (**). NS indicates no statistical significance (Benjamini–Hochberg-adjusted P > 0.05).
Fig. 5 AI signed error as a predictor of treatment effect. a, AI has greater treatment effects on radiologists when the AI assistance underestimates probabilities, rather than overestimates probabilities, given the same absolute error on all pathologies aggregated. The nine bins are presented as −3.560 (95% CI: −9.472 to 2.353), 2.089 (95% CI: 0.334 to 3.844), 2.698 (95% CI: 1.497 to 3.899), 2.483 (95% CI: 1.897 to 3.070), 0.398 (95% CI: 0.235 to 0.561), −2.933 (95% CI: −3.787 to −2.079), −5.018 (95% CI: −6.516 to −3.519), −6.968 (95% CI: −9.346 to −4.591) and −16.845 (95% CI: −24.288 to −9.403). The error bars show 95% CIs. The blue bars show the overall treatment effect across AI error. * and ** indicate statistically significant difference among subgroups of different AI signed error through a joint equality test at a significance level of 0.05 and 0.01, respectively. b, AI has greater treatment effects on radiologists when the AI assistance underestimates probabilities, rather than overestimates probabilities, given the same absolute error on airspace opacity, atelectasis, cardiomegaly, consolidation and lesion. The bar plots are designed in the same way as those in a. The significance of the subgroup joint test is determined through the Benjamini–Hochberg procedure, respectively, to correct for multiple hypothesis testing (15 individual pathologies) at a false discovery rate of 0.05 (*) and 0.01 (**). NS indicates no statistical significance (Benjamini–Hochberg-adjusted P > 0.05).CC BY · PubMed Central
Extended Data Fig. 1 Individual heterogeneity in treatment effects. a, b, Individual heterogeneity in treatment effects of 140 radiologists as determined using the empirical Bayes method on (a) all pathologies aggregated and (b) high-prevalence pathology labels (pathology labels with greater than 10% prevalence). The curve is the kernel density estimate (KDE).
Extended Data Fig. 1 Individual heterogeneity in treatment effects. a, b, Individual heterogeneity in treatment effects of 140 radiologists as determined using the empirical Bayes method on (a) all pathologies aggregated and (b) high-prevalence pathology labels (pathology labels with greater than 10% prevalence). The curve is the kernel density estimate (KDE).CC BY · PubMed Central
Extended Data Fig. 2 Individual heterogeneity in unassisted error. a, b, Individual heterogeneity in unassisted error of 140 radiologists as determined using the empirical Bayes method on (a) all pathologies aggregated and (b) high-prevalence pathology labels (pathology labels with greater than 10% prevalence). The curve is the kernel density estimate (KDE).
Extended Data Fig. 2 Individual heterogeneity in unassisted error. a, b, Individual heterogeneity in unassisted error of 140 radiologists as determined using the empirical Bayes method on (a) all pathologies aggregated and (b) high-prevalence pathology labels (pathology labels with greater than 10% prevalence). The curve is the kernel density estimate (KDE).CC BY · PubMed Central
Extended Data Fig. 3 Individual heterogeneity in treatment effects on sensitivity, sensitivities, treatment effects on specificity, and specificities. a-b, Individual heterogeneity in (a) improvement in sensitivities, (b) sensitivity, (c) improvement in specificity, and (d) specificity of 140 radiologists as determined using the empirical Bayes method on all pathologies aggregated. The curve is the kernel density estimate (KDE).
Extended Data Fig. 3 Individual heterogeneity in treatment effects on sensitivity, sensitivities, treatment effects on specificity, and specificities. a-b, Individual heterogeneity in (a) improvement in sensitivities, (b) sensitivity, (c) improvement in specificity, and (d) specificity of 140 radiologists as determined using the empirical Bayes method on all pathologies aggregated. The curve is the kernel density estimate (KDE).CC BY · PubMed Central
Extended Data Fig. 4 Conventional radiologist characteristics as indicators for treatment effect on individual pathologies. a, Difference in treatment effects of subgroups of radiologists on high-prevalence pathology labels (pathology labels with greater than 10% prevalence). The difference is computed between lower and higher improvement subgroups. The error bars show 95% confidence intervals. There are statistically significant differences between subgroups on high-prevalence pathology labels (abnormal B-H adjusted P = 1.66e-29, airspace opacity B-H adjusted P = 7.20e-29, atelectasis B-H adjusted P = 1.10e-30, cardiomediastinal abnormality B-H adjusted P = 4.85e-30, support device hardware B-H adjusted P = 1.57e-30; B-H adjusted P < 0.001). A two-sided, unpaired t-test between the two subgroups of treatment effects was conducted. The difference is -4.194 (95% CI: -4.753 to -3.636) for abnormal, -1.465 (95% CI: -1.664 to -1.266) for airspace opacity, -1.766 (95% CI: -1.991 to -1.541) for atelectasis, -1.571 (95% CI: -1.777 to -1.365) for cardiomediastinal abnormality, and -3.150 (95% CI: -3.552 to -2.748) for support device hardware. 136 radiologists with available survey data are used. b, Difference in treatment effects of subgroups of radiologists based on combined characteristics of years of experience, subspecialty in thoracic radiology and experience with AI tools on held-out test sets of radiologists. The difference is computed between lower and higher predicted improvement subgroups. The error bars show 95% confidence intervals. n.s. indicates no statistical significance (B-H adjusted P > 0.05). The Wald test was used to test regression coefficients that estimate treatment effects against the null hypothesis of joint equality among treatment effects of different subgroups. Details of the statistical models are available in Methods. There are 136 radiologists with available survey data on the three characteristics. c-e, Difference in treatment effects of subgroups of radiologists based on (c) years of experience, (d) subspecialty in thoracic radiology, and (e) experience with AI tools on 15 individual pathologies. The difference is computed between (c) subgroups of fewer versus more years of experience, (d) subgroups without versus with subspecialty in thoracic radiology, and (e) subgroups without versus with experience using AI tools. The error bars show 95% confidence intervals. n.s. indicates no statistical significance (B-H adjusted P > 0.05). The same statistical test as in b was used. There are 136 radiologists with available survey data.
Extended Data Fig. 4 Conventional radiologist characteristics as indicators for treatment effect on individual pathologies. a, Difference in treatment effects of subgroups of radiologists on high-prevalence pathology labels (pathology labels with greater than 10% prevalence). The difference is computed between lower and higher improvement subgroups. The error bars show 95% confidence intervals. There are statistically significant differences between subgroups on high-prevalence pathology labels (abnormal B-H adjusted P = 1.66e-29, airspace opacity B-H adjusted P = 7.20e-29, atelectasis B-H adjusted P = 1.10e-30, cardiomediastinal abnormality B-H adjusted P = 4.85e-30, support device hardware B-H adjusted P = 1.57e-30; B-H adjusted P < 0.001). A two-sided, unpaired t-test between the two subgroups of treatment effects was conducted. The difference is -4.194 (95% CI: -4.753 to -3.636) for abnormal, -1.465 (95% CI: -1.664 to -1.266) for airspace opacity, -1.766 (95% CI: -1.991 to -1.541) for atelectasis, -1.571 (95% CI: -1.777 to -1.365) for cardiomediastinal abnormality, and -3.150 (95% CI: -3.552 to -2.748) for support device hardware. 136 radiologists with available survey data are used. b, Difference in treatment effects of subgroups of radiologists based on combined characteristics of years of experience, subspecialty in thoracic radiology and experience with AI tools on held-out test sets of radiologists. The difference is computed between lower and higher predicted improvement subgroups. The error bars show 95% confidence intervals. n.s. indicates no statistical significance (B-H adjusted P > 0.05). The Wald test was used to test regression coefficients that estimate treatment effects against the null hypothesis of joint equality among treatment effects of different subgroups. Details of the statistical models are available in Methods. There are 136 radiologists with available survey data on the three characteristics. c-e, Difference in treatment effects of subgroups of radiologists based on (c) years of experience, (d) subspecialty in thoracic radiology, and (e) experience with AI tools on 15 individual pathologies. The difference is computed between (c) subgroups of fewer versus more years of experience, (d) subgroups without versus with subspecialty in thoracic radiology, and (e) subgroups without versus with experience using AI tools. The error bars show 95% confidence intervals. n.s. indicates no statistical significance (B-H adjusted P > 0.05). The same statistical test as in b was used. There are 136 radiologists with available survey data.CC BY · PubMed Central
Extended Data Fig. 5 Conventional radiologist characteristics as indicators for treatment effect on AUROC on individual pathologies. a, Difference in treatment effects on AUROC of subgroups of radiologists based on combined characteristics of years of experience, subspecialty in thoracic radiology and experience with AI tools on held-out test sets of radiologists. The difference is computed between lower and higher predicted improvement subgroups. The error bars show 95% confidence intervals. n.s. indicates no statistical significance (B-H adjusted P > 0.05). The difference is 0.034 (95% CI: -0.017 to 0.842) for abnormal and -0.023 (95% CI: -0.082 to 0.035) for airspace opacity. The Wald test was used to test regression coefficients that estimate treatment effects against the null hypothesis of joint equality among treatment effects of different subgroups. Details of the statistical models are available in Methods. 136 radiologists with available survey data are used. b-d, Difference in treatment effects of subgroups of radiologists based on (b) years of experience, (c) subspecialty in thoracic radiology, and (d) experience with AI tools on 2 individual pathologies on which the AUROC analysis could be computed. The difference is computed between (b) subgroups of fewer versus more years of experience, (c) subgroups without versus with subspecialty in thoracic radiology, and (d) subgroups without versus with experience using AI tools. The error bars show 95% confidence intervals. n.s. indicates no statistical significance (B-H adjusted P > 0.05). The same statistical test as in a was used. There are 136 radiologists with available survey data.
Extended Data Fig. 5 Conventional radiologist characteristics as indicators for treatment effect on AUROC on individual pathologies. a, Difference in treatment effects on AUROC of subgroups of radiologists based on combined characteristics of years of experience, subspecialty in thoracic radiology and experience with AI tools on held-out test sets of radiologists. The difference is computed between lower and higher predicted improvement subgroups. The error bars show 95% confidence intervals. n.s. indicates no statistical significance (B-H adjusted P > 0.05). The difference is 0.034 (95% CI: -0.017 to 0.842) for abnormal and -0.023 (95% CI: -0.082 to 0.035) for airspace opacity. The Wald test was used to test regression coefficients that estimate treatment effects against the null hypothesis of joint equality among treatment effects of different subgroups. Details of the statistical models are available in Methods. 136 radiologists with available survey data are used. b-d, Difference in treatment effects of subgroups of radiologists based on (b) years of experience, (c) subspecialty in thoracic radiology, and (d) experience with AI tools on 2 individual pathologies on which the AUROC analysis could be computed. The difference is computed between (b) subgroups of fewer versus more years of experience, (c) subgroups without versus with subspecialty in thoracic radiology, and (d) subgroups without versus with experience using AI tools. The error bars show 95% confidence intervals. n.s. indicates no statistical significance (B-H adjusted P > 0.05). The same statistical test as in a was used. There are 136 radiologists with available survey data.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 49 references

Research works printed but not linked: 3

  1. Norden, J. G. & Shah, N. R. What AI in health care can learn from the long road to autonomous vehicles. NEJM Catalyst https://catalyst.nejm.org/doi/full/10.1056/CAT.21.0458 (2022). · doi:10.1056/cat.21.0458
  2. Mozannar, H. et al. Who Should Predict? Exact Algorithms For Learning to Defer to Humans. in International Conference on Artificial Intelligence and Statistics 10520–10545 (PMLR, 2023).
  3. Endo, M., Krishnan, R., Krishna, V., Ng, A. Y. & Rajpurkar, P. Retrieval-based chest X-ray report generation using a pre-trained contrastive language-image model. in Proc. of Machine Learning for Health 209–219 (PMLR, 2021).

Cited by

See all 236

Metadata

Authors
Yu, F., Moehring, A., Banerjee, O., Salz, T., Agarwal, N., & Rajpurkar, P.
Affiliations
  • Harvard University (US)
  • Massachusetts Institute of Technology (US)
Corresponding author
Rajpurkar, Pranav
Venue
Nature Medicine
Published
2024-03
Volume
30
Issue
3
Pages
837-849
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
OpenAlex
W4392956046
Cited by
200
References
44
Citations by year
2026: 81 · 2025: 97 · 2024: 21
FWCI
18.60
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
Alfred P. Sloan Foundation (2022-17182)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
Keywords
radiologists, chest X-ray interpretation, clinician-AI collaboration, treatment effect heterogeneity, AI errors, radiologist performance, diagnostic accuracy
MeSH terms
Radiologists, Algorithms, Artificial Intelligence, Humans

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Heterogeneity and predictors of the effects of AI assistance on radiologists. https://app.cochat.ai/papers/10.1038/s41591-024-02850-w
The paper
Yu, F., Moehring, A., Banerjee, O., Salz, T., Agarwal, N., & Rajpurkar, P. (2024). Heterogeneity and predictors of the effects of AI assistance on radiologists. Nature Medicine. https://doi.org/10.1038/s41591-024-02850-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.