Heterogeneity and predictors of the effects of AI assistance on radiologists
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
The integration of artificial intelligence (AI) in medical image interpretation requires effective collaboration between clinicians and AI algorithms. Although previous studies demonstrated the potential of AI assistance in improving overall clinician performance, the individual impact on clinicians remains unclear. This large-scale study examined the heterogeneous effects of AI assistance on 140 radiologists across 15 chest X-ray diagnostic tasks and identified predictors of these effects. Surprisingly, conventional experience-based factors, such as years of experience, subspecialty and familiarity with AI tools, fail to reliably predict the impact of AI assistance. Additionally, lower-performing radiologists do not consistently benefit more from AI assistance, challenging prevailing assumptions. Instead, we found that the occurrence of AI errors strongly influences treatment outcomes, with inaccurate AI predictions adversely affecting radiologist performance on the aggregate of all pathologies and on half of the individual pathologies investigated. Our findings highlight the importance of personalized approaches to clinician–AI collaboration and the importance of accurate AI models. By understanding the factors that shape the effectiveness of AI assistance, this study provides valuable insights for targeted implementation of AI, enabling maximum benefits for individual clinicians in clinical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



![Fig. 4 AI error as a predictor of treatment effect. a, AI has greater treatment effects on radiologists when the AI assistance has lower error on all pathologies aggregated. The five bins are presented as 0.679 (95% CI: 0.492 to 0.865), −1.509 (95% CI: −2.267 to −0.750), −3.556 (95% CI: −4.878 to −2.235), −6.569 (95% CI: −8.764 to −4.374) and −16.845 (95% CI: −24.288 to −9.403). The error bars show 95% CIs. The blue lines show the overall treatment effect across AI error. * and ** indicate statistically significant difference among subgroups of different AI error through a joint equality test at a significance level of 0.05 and 0.01, respectively. b, AI has greater treatment effects on radiologists when the AI assistance has lower error on abnormal, airspace opacity, cardiomediastinal abnormality, cardiomegaly, pleural effusion, pleural other, pneumothorax and support device hardware. The reverse trend holds for bacterial/lobar pneumonia and consolidation. The bar plots are designed in the same way as those in a. The significance of the subgroup joint test is determined through the Benjamini–Hochberg procedure, respectively, to correct for multiple hypothesis testing (15 individual pathologies) at a false discovery rate of 0.05 (*) and 0.01 (**). NS indicates no statistical significance (Benjamini–Hochberg-adjusted P > 0.05). c, AI has greater treatment effects on AUROC on radiologists when the AI assistance has an absolute error in the range [20, 100], whereas the trend is unclear with absolute error in the range [0, 20].](https://figures.cochat.ai/paper-figures/0f731956d7554b028fc0fd34a0368d3a/heterogeneity-and-predictors-of-the-effects-of-ai-assistance-on-radiologists-figure-4.jpg)






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior studies using traditional AI models have documented the effects of cognitive biases,22-27 with inaccurate predictions impairing radiologist performance and erroneous suggestions prompting pathology experts to overturn correct diagnoses in 7% of cases.22, 23”
From Automation Bias in Large Language Model Assisted Diagnostic Reasoning Among AI-Trained Physicians (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies27,29,30,42 provide empirical evidence that GenAI tools can significantly improve clinical accuracy, workflow efficiency, and decision-support capacity when implemented under structured supervision.”
From Generative Artificial Intelligence in Healthcare: Automation Bias, Deskilling, and Cognitive Implications – A Systematic Review (2026) · cites this paper as [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical evidence27,29,42 indicated that clinician performance is optimized when AI outputs are analyzed rather than uncritically accepted.”
From Generative Artificial Intelligence in Healthcare: Automation Bias, Deskilling, and Cognitive Implications – A Systematic Review (2026) · cites this paper as [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, radiologists in one study found it challenging to determine the correctness of AI predictions, leading to poor decisions when inaccurate predictions were provided [78].”
From To Recommend or Not to Recommend: Designing and Evaluating AI-Enabled Decision Support for Time-Critical Medical Events (2025) · cites this paper as [78]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2009) · doi:10.1515/9781400829828 · Cited by 9,467
Mostly Harmless Econometrics: An Empiricist's Companion
Angrist, J. D., & Pischke, J.-S. (2008) · doi:10.2307/j.ctvcm4j72 · Cited by 5,051
A Practitioner’s Guide to Cluster-Robust Inference
Colin Cameron, A., & Miller, D. L. (2015) · Journal of Human Resources · doi:10.3368/jhr.50.2.317 · Cited by 5,037
Fleischner Society: Glossary of Terms for Thoracic Imaging
Hansell, D. M., Bankier, A. A., MacMahon, H., McLoud, T. C., Müller, N. L., & Remy, J. (2008) · Radiology · doi:10.1148/radiol.2462070712 · Cited by 4,198
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
Self-citationKey challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816
Experimental methods: Between-subject and within-subject design
Charness, G., Gneezy, U., & Kuhn, M. A. (2012) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2011.08.009 · Cited by 1,469
Rajpurkar, P., Irvin, J., Ball, R. L., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C. P., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002686 · Cited by 1,449
Self-citationCheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D. Y., Bagul, A., Langlotz, C. P., Shpanskaya, K. S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05225 · Cited by 1,219
Self-citationHuman-computer collaboration for skin cancer recognition
Tschandl, P., Rinner, C., Apalla, Z., Argenziano, G., Codella, N., Halpern, A., Janda, M., Lallas, A., Longo, C., Malvehy, J., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0942-0 · Cited by 1,024
Generating Radiology Reports via Memory-driven Transformer
Chen, Z., Song, Y., Chang, T.-H., & Wan, X. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.112 · Cited by 693
The current and future state of AI interpretation of medical images
Rajpurkar, P., & Lungren, M. P. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2301725 · Cited by 443
Self-citationTiu, E., Talius, E., Patel, P., Langlotz, C. P., Ng, A. Y., & Rajpurkar, P. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00936-9 · Cited by 442
Self-citationCheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392
Self-citationDratsch, T., Chen, X., Rezazade Mehrizi, M., Kloeckner, R., Mähringer-Kunz, A., Püsken, M., Baeßler, B., Sauer, S., Maintz, D., & Pinto dos Santos, D. (2023) · Radiology · doi:10.1148/radiol.222176 · Cited by 369
Experimental evidence of effective human–AI collaboration in medical decision-making
Reverberi, C., Rigon, T., Solari, A., Hassan, C., Cherubini, P., GI Genius CADx Study Group, Antonelli, G., Awadie, H., Bernhofer, S., Carballal, S., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-18751-2 · Cited by 281
Majkowska, A., Mittal, S., Steiner, D. F., Reicher, J. J., McKinney, S. M., Duggan, G. E., Eswaran, K., Cameron Chen, P.-H., Liu, Y., Kalidindi, S. R., et al. (2020) · Radiology · doi:10.1148/radiol.2019191293 · Cited by 277
Benchmarking saliency methods for chest X-ray interpretation
Saporta, A., Gui, X., Agrawal, A., Pareek, A., Truong, S. Q. H., Nguyen, C. D. T., Ngo, V.-D., Seekins, J., Blankenberg, F. G., Ng, A. Y., et al. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00536-x · Cited by 270
Self-citationRegression towards the mean, historically considered
Stigler, S. M. (1997) · Statistical Methods in Medical Research · doi:10.1177/096228029700600202 · Cited by 254
Fully Convolutional Architectures for Multiclass Segmentation in Chest Radiographs
Novikov, A. A., Lenis, D., Major, D., Hladuvka, J., Wimmer, M., & Buhler, K. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2806086 · Cited by 252
A Review of Challenges and Opportunities in Machine Learning for Health
Ghassemi, M., Naumann, T., Schulam, P., Beam, A. L., Chen, I. Y., & Ranganath, R. (2018) · PubMed · doi:10.48550/arxiv.1806.00388 · Cited by 248
Conant, E. F., Toledano, A. Y., Periaswamy, S., Fotin, S. V., Go, J., Boatsman, J. E., & Hoffmeister, J. W. (2019) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2019180096 · Cited by 241
Seah, J. C. Y., Tang, C. H. M., Buchlak, Q. D., Holt, X. G., Wardman, J. B., Aimoldin, A., Esmaili, N., Ahmad, H., Pham, H., Lambert, J. F., et al. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00106-0 · Cited by 229
Multi-institutional validation of a mammography-based breast cancer risk model
Yala, A., Mikhael, P. G., Strand, F., Lin, G., Satuluru, S., Kim, T., Banerjee, I., Gichoya, J., Trivedi, H., Lehman, C. D., et al. (2022) · Journal of Clinical Oncology · doi:10.1200/jco.21.01337 · Cited by 204
Pacilè, S., Lopez, J., Chone, P., Bertinotti, T., Grouin, J. M., & Fillard, P. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020190208 · Cited by 161
Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023) · doi:10.3386/w31422 · Cited by 161
Self-citationImproving chest X-ray report generation by leveraging warm starting
Nicolson, A., Dowling, J., & Koopman, B. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102633 · Cited by 161
Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023) · doi:10.2139/ssrn.4505053 · Cited by 161
Self-citationImproving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation
Miura, Y., Zhang, Y., Tsai, E., Langlotz, C., & Jurafsky, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.416 · Cited by 149
Empirical Bayes: Past, Present and Future
Carlin, B. P., & Louis, T. A. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10474331 · Cited by 138
Ahn, J. S., Ebrahimian, S., McDermott, S., Lee, S., Naccarato, L., Di Capua, J. F., Wu, M. Y., Zhang, E. W., Muse, V., Miller, B., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.29289 · Cited by 138
Dvijotham, K., Winkens, J., Barsbey, M., Ghaisas, S., Stanforth, R., Pawlowski, N., Strachan, P., Ahmed, Z., Azizi, S., Bachrach, Y., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02437-x · Cited by 120
Zheng, X., Wang, R., Zhang, X., Sun, Y., Zhang, H., Zhao, Z., Zheng, Y., Luo, J., Zhang, J., Wu, H., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30459-5 · Cited by 111
“Who goes first? Influences of human-AI workflow on decision making in clinical imaging,”
Fogliato, R., Chappidi, S., Lungren, M., Fisher, P., Wilson, D., Fitzke, M., Parkinson, M., Horvitz, E., Inkpen, K., & Nushi, B. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533193 · Cited by 103
Non-task expert physicians benefit from correct explainable AI advice when reviewing X-rays
Gaube, S., Suresh, H., Raue, M., Lermer, E., Koch, T. K., Hudecek, M. F. C., Ackery, A. D., Grover, S. C., Coughlin, J. F., Frey, D., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-28633-w · Cited by 92
Chi, E. A., Chi, G., Tsui, C. T., Jiang, Y., Jarr, K., Kulkarni, C. V., Zhang, M., Long, J., Ng, A. Y., Rajpurkar, P., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.17391 · Cited by 64
Self-citationPoulain, R., Bin Tarek, M. F., & Beheshti, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594102 · Cited by 63
Jones, C. M., Danaher, L., Milne, M. R., Tang, C., Seah, J., Oakden-Rayner, L., Johnson, A., Buchlak, Q. D., & Esmaili, N. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-052902 · Cited by 45
Farzaneh, N., Ansari, S., Lee, E., Ward, K. R., & Sjoding, M. W. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00797-9 · Cited by 45
Benchmarking saliency methods for chest X-ray interpretationPreprint
Saporta, A., Gui, X., Agrawal, A., et al. (2021) · medRxiv · doi:10.1101/2021.02.28.21252634 · Cited by 40
Weakly Supervised Contrastive Learning for Chest X-Ray Report GenerationPreprint
Yan, A., He, Z., Lü, X., Du, J., Chang, E., Gentili, A., McAuley, J., & Hsu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.12242 · Cited by 40
Lu, Z., Zhang, L., Yao, L., Gong, D., Wu, L., Xia, M., Zhang, J., Zhou, W., Huang, X., He, C., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.53840 · Cited by 16
Chi, E. A., Chi, G., Tsui, C. T., Jiang, Y. C., Jarr, K. J., Kulkarni, C. V., Zhang, M., Long, J., Ng, A. Y., Rajpurkar, P., et al. (2021) · Gastroenterology · doi:10.1016/s0016-5085(21)00897-0 · Cited by 8
Self-citationComparison of AI-integrated pathways with human-AI interaction for population mammographic screeningPreprint
Frazer, H. M. L., Peña‐Solórzano, C. A., Kwok, C. F., Elliott, M. S., Chen, Y., Wang, C., the BRAIx team, Lippey, J., Hopper, J. L., Brotchie, P., et al. (2022) · medRxiv · doi:10.1101/2022.11.23.22282646 · Cited by 5
Dvijotham, K., Winkens, J., Barsbey, M., et al. (2022) · Research Square · doi:10.21203/rs.3.rs-2231672/v1 · Cited by 4
Effects of AI Assistance on Radiologists: Code Release
Yu, F., Moehring, A., Banerjee, O., Salz, T., Agarwal, N., & Rajpurkar, P. (2024) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.10467491 · Cited by 1
Self-citation
Research works printed but not linked: 3
- Norden, J. G. & Shah, N. R. What AI in health care can learn from the long road to autonomous vehicles. NEJM Catalyst https://catalyst.nejm.org/doi/full/10.1056/CAT.21.0458 (2022). · doi:10.1056/cat.21.0458
- Mozannar, H. et al. Who Should Predict? Exact Algorithms For Learning to Defer to Humans. in International Conference on Artificial Intelligence and Statistics 10520–10545 (PMLR, 2023).
- Endo, M., Krishnan, R., Krishna, V., Ng, A. Y. & Rajpurkar, P. Retrieval-based chest X-ray report generation using a pre-trained contrastive language-image model. in Proc. ofMachine Learning for Health 209–219 (PMLR, 2021).
Cited by
Educational Strategies for Clinical Supervision of Artificial Intelligence Use
Abdulnour, R.-E. E., Gin, B., & Boscardin, C. K. (2025) · New England Journal of Medicine · doi:10.1056/nejmra2503232 · Cited by 209
Performance of a large language model on the reasoning tasks of a physician
Brodeur, P. G., Buckley, T. A., Kanjee, Z., Goh, E., Ling, E., Jain, P., Cabral, S., Abdulnour, R.-E. E., Haimovich, A. D., Freed, J. A., et al. (2026) · Science · doi:10.1126/science.adz4433 · Cited by 231
The generative era of medical AI
Fahrner, J., Chen, E., Topol, E. J., & Rajpurkar, P. (2025) · Cell · doi:10.1016/j.cell.2025.05.018 · Cited by 144
Self-citationMultinational Attitudes Toward AI in Health Care and Diagnostics Among Hospital Patients
Busch, F., Hoffmann, L., Xu, L., Zhang, L. J., Hu, B., García-Juárez, I., Toapanta-Yanchapaxi, L. N., Gorelik, N., Gorelik, V., Rodriguez-Granillo, G. A., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.14452 · Cited by 124
Efficiency and Quality of Generative AI–Assisted Radiograph Reporting
Huang, J., Wittbrodt, M. T., Teague, C. N., Karl, E., Galal, G., Thompson, M., Chapa, A. R., Chiu, M.-L., Herynk, B. A., Linchangco, R., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.13921 · Cited by 73
Metadata
- Authors
- Yu, F., Moehring, A., Banerjee, O., Salz, T., Agarwal, N., & Rajpurkar, P.
- Affiliations
- Harvard University (US)
- Massachusetts Institute of Technology (US)
- Corresponding author
- Rajpurkar, Pranav
- Venue
- Nature Medicine
- Published
- 2024-03
- Volume
- 30
- Issue
- 3
- Pages
- 837-849
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC10957478
- OpenAlex
- W4392956046
- Cited by
- 200
- References
- 44
- Citations by year
- 2026: 81 · 2025: 97 · 2024: 21
- FWCI
- 18.60
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor, 18 d)
- Locations
- Nature Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Alfred P. Sloan Foundation (2022-17182)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Keywords
- radiologists, chest X-ray interpretation, clinician-AI collaboration, treatment effect heterogeneity, AI errors, radiologist performance, diagnostic accuracy
- MeSH terms
- Radiologists, Algorithms, Artificial Intelligence, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.