Cited by 161Artificial Intelligence in Healthcare and EducationRadiology practices and educationClinical Reasoning and Diagnostic Skills

Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology

Open access
Agarwal, Nikhil, Moehring, Alex, Rajpurkar, Pranav, Salz, Tobias·2023·National Bureau of Economic Research

Abstract

Full automation using Artificial Intelligence (AI) predictions may not be optimal if humans have information not available to the AI (contextual information). We study human-AI collaboration using an information experiment with professional radiologists. Results show that providing (i) AI predictions does not improve performance on average, whereas (ii) contextual information does. Radiologists do not realize the gains from AI assistance because of errors in belief updating – they underweight AI predictions and treat their own information and AI predictions as statistically independent. Unless these mistakes can be corrected, the optimal human-AI collaboration design delegates cases either to humans or to AI, but rarely to AI assisted humans.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3386/w31422

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Doctors did not understand how to use the AI tool effectively; when AI offered confident predictions, doctors frequently overrode those predictions with their own, and when AI offered uncertain predictions, doctors frequently overrode their own better predictions with those supplied by the machine.

    “That's because doctors did not understand how to use the AI tool effectively; when AI offered confident predictions, doctors frequently overrode those predictions with their own.When AI offered uncertain predictions, doctors frequently overrode their own better predictions with those supplied by the machine.”

    From Applying AI to Rebuild Middle Class Jobs (2024) · cites this paper as Agarwal et al., 2023

  • Agarwal et al. found that AI did not improve the quality of radiologists' diagnoses even though AI's predictions were at least as accurate as nearly two-thirds of the doctors studied.

    “In an experiment that randomly allocated the availability of AI assistance to professional radiologists, Nikhil Agarwal of MIT and his coauthors have found that AI did not improve the quality of radiologists' diagnoseseven though AI's predictions were at least as accurate as nearly two-thirds of the doctors studied (Agarwal et al., 2023).”

    From Applying AI to Rebuild Middle Class Jobs (2024) · cites this paper as Agarwal et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 52

  1. Jason Abaluck, Leila Agha, Chris Kabrhel, Ali Raja, and Arjun Venkatesh. The determinants of productivity in medical testing: Intensity and allocation of care. Am. Econ. Rev., 106 (12):3730-3764, December 2016.
  2. Ajay Agrawal, Joshua Gans, and Avi Goldfarb. Prediction Machines: The Simple Economics of Artificial Intelligence. Harvard Business Press, April 2018.
  3. Eugenio Alberdi, Lorenzo Strigini, Andrey A Povyakalo, and Peter Ayton. Why are people's decisions sometimes worse with computer support? In Lecture Notes in Computer Sci- ence, Lecture notes in computer science, pages 18-31. Springer Berlin Heidelberg, Berlin, Heidelberg, 2009.
  4. Donald W K Andrews and Biao Lu. Consistent model and moment selection procedures for GMM estimation with application to dynamic panel data models. J. Econom., 101(1): 123-164, March 2001.
  5. Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S Weld. Is the most accurate AI the best teammate? optimizing AI for teamwork. AAAI, 35(13):11405-11414, May 2021.
  6. Nicholas Barberis, Andrei Shleifer, and Robert Vishny. A model of investor sentiment. J. financ. econ., 49(3):307-343, September 1998.
  7. Daniel J Benjamin. Chapter 2 -errors in probabilistic reasoning and judgment biases. In Handbook of Behavioral Economics: Applications and Foundations 1, volume 2, pages 69- 186. January 2019.
  8. Dirk Bergemann and Stephen Morris. Information design: A unified perspective. J. Econ. Lit., 57(1):44-95, March 2019.
  9. David Blackwell. Equivalent comparisons of experiments. Ann. Math. Stat., 24(2):265-272, 1953.
  10. Erik Brynjolfsson and Tom Mitchell. What can machine learning do? workforce implications. 358(6370):1530-, 2017.
  11. David C Chan, Matthew Gentzkow, and Chuan Yu. Selection with variation in diagnostic skill: Evidence from radiologists. Q. J. Econ., 137(2):729-783, May 2022.
  12. Amitabh Chandra and Douglas O Staiger. Identifying sources of inefficiency in healthcare. Q. J. Econ., 135(2):785-843, May 2020.
  13. Daniel L Chen, Martin Schonger, and Chris Wickens. oTree-An open-source platform for laboratory, online, and field experiments. Journal of Behavioral and Experimental Finance, 9:88-97, March 2016.
  14. Emily F Conant, Alicia Y Toledano, Senthil Periaswamy, Sergei V Fotin, Jonathan Go, Justin E Boatsman, and Jeffrey W Hoffmeister. Improving accuracy and efficiency with concurrent use of artificial intelligence for digital breast tomosynthesis. Radiol Artif Intell, 1(4):e180096, July 2019.
  15. Janet Currie and W Bentley MacLeod. Diagnosing expertise: Human capital, decision mak- ing, and performance among physicians. J. Labor Econ., 35(1), 2017.
  16. Riccardo Fogliato, Shreya Chappidi, Matthew Lungren, Paul Fisher, Diane Wilson, Michael Fitzke, Mark Parkinson, Eric Horvitz, Kori Inkpen, and Besmira Nushi. Who goes first? influences of Human-AI workflow on decision making in clinical imaging. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT '22, pages 1362-1374. Association for Computing Machinery, June 2022.
  17. Morgan R Frank, David Autor, James E Bessen, Erik Brynjolfsson, Manuel Cebrian, David J Deming, Maryann Feldman, Matthew Groh, José Lobo, Esteban Moro, Dashun Wang, Hyejin Youn, and Iyad Rahwan. Toward understanding the impact of artificial intelligence on labor. Proc. Natl. Acad. Sci. U. S. A., 116(14):6531-6539, April 2019.
  18. Avi Goldfarb, Bledi Taska, and Florenta Teodoridis. Could machine learning be a general purpose technology? a comparison of emerging technologies using data from online job postings. Res. Policy, 52(1):104653, January 2023.
  19. David M Grether. Bayes rule as a descriptive model: The representativeness heuristic. Q. J. Econ., 95(3):537-557, November 1980.
  20. David M Grether. Testing bayes rule and the representativeness heuristic: Some experimental evidence. J. Econ. Behav. Organ., 17(1):31-57, January 1992.
  21. Dale Griffin and Amos Tversky. The weighing of evidence and the determinants of confidence. Cogn. Psychol., 24(3):411-435, July 1992.
  22. H Benjamin Harvey and Vrushab Gowda. How the FDA regulates AI. Acad. Radiol., 27(1): 58-61, January 2020.
  23. Ahmed Hosny, Chintan Parmar, John Quackenbush, Lawrence H Schwartz, and Hugo J W L Aerts. Artificial intelligence in radiology. Nat. Rev. Cancer, 18(8):500-510, August 2018.
  24. Tanjim Hossain and Ryo Okui. The binarized scoring rule. Rev. Econ. Stud., 80(3):984-1001, February 2013.
  25. Nicholas C Hunt and Andrea M Scheetz. Using MTurk to distribute a survey or experiment: Methodological considerations. Journal of Information Systems, 33(1):43-65, March 2019.
  26. Emir Kamenica and Matthew Gentzkow. Bayesian persuasion. Am. Econ. Rev., 101(6): 2590-2615, October 2011.
  27. Hyo Eun Kim, Hak Hee Kim, Boo Kyung Han, Ki Hwan Kim, Kyunghwa Han, Hyeonseob Nam, Eun Hye Lee, and Eun Kyung Kim. Changes in cancer detection and False-Positive recall in mammography using artificial intelligence: a retrospective, multireader study. The Lancet Digital Health, 2(3):e138-e148, March 2020.
  28. Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mul- lainathan. Human decisions and machine predictions. Q. J. Econ., 133(1):237-293, August 2017.
  29. Barnett S Kramer, Christine D Berg, Denise R Aberle, and Philip C Prorok. Lung cancer screening with low-dose helical CT: results from the national lung screening trial (NLST). J. Med. Screen., 18(3):109-111, 2011.
  30. Curtis P Langlotz. Will artificial intelligence replace radiologists? Radiology: Artificial Intelligence, 1(3):e190058, May 2019.
  31. Xiaoxuan Liu, Livia Faes, Aditya U Kale, Siegfried K Wagner, Dun Jack Fu, Alice Bruynseels, Thushika Mahendiran, Gabriella Moraes, Mohith Shamdas, Christoph Kern, Joseph R Ledsam, Martin K Schmid, Konstantinos Balaskas, Eric J Topol, Lucas M Bachmann, Pearse A Keane, and Alastair K Denniston. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. The Lancet Digital Health, 1(6):e271-e297, October 2019.
  32. Robert Mccluskey, A Enshaei, and B A S Hasan. Finding the Ground-Truth from multiple labellers: Why parameters of the task matter. ArXiv, 2021.
  33. Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7076-7087. PMLR, 2020.
  34. Justin G Norden and Nirav R Shah. What AI in health care can learn from the long road to autonomous vehicles. NEJM Catalyst Innovations in Care Delivery, 3(2), 2022.
  35. Ziad Obermeyer and Ezekiel J Emanuel. Predicting the future -big data, machine learning, and clinical medicine. N. Engl. J. Med., 375(13):1216-1219, September 2016.
  36. Serena Pacilè, January Lopez, Pauline Chone, Thomas Bertinotti, Jean Marie Grouin, and Pierre Fillard. Improving breast cancer detection accuracy of mammography with the concurrent use of an artificial intelligence tool. Radiol Artif Intell, 2(6):e190208, November 2020.
  37. David M Panicek and Hedvig Hricak. How sure are you, doctor? a standardized lexicon to describe the radiologist's level of certainty. AJR Am. J. Roentgenol., 207(1):2-3, July 2016.
  38. Allison Park, Chris Chute, Pranav Rajpurkar, Joe Lou, Robyn L Ball, Katie Shpanskaya, Rashad Jabarkheel, Lily H Kim, Emily McKenna, Joe Tseng, Jason Ni, Fidaa Wishah, Fred Wittber, David S Hong, Thomas J Wilson, Safwan Halabi, Sanjay Basu, Bhavik N Patel, Matthew P Lungren, Andrew Y Ng, and Kristen W Yeom. Deep Learning-Assisted diagnosis of cerebral aneurysms using the HeadXNet model. JAMA network open, 2(6): e195600, June 2019.
  39. Bhavik N Patel, Louis Rosenberg, Gregg Willcox, David Baltaxe, Mimi Lyons, Jeremy Irvin, Pranav Rajpurkar, Timothy Amrhein, Rajan Gupta, Safwan Halabi, Curtis Langlotz, Ed- ward Lo, Joseph Mammarappallil, A J Mariano, Geoffrey Riley, Jayne Seekins, Luyao Shen, Evan Zucker, and Matthew Lungren. Human-Machine partnership with artificial intelligence for chest radiograph diagnosis. npj Digital Medicine, 2(1):111, December 2019.
  40. Matthew Rabin. Incorporating limited rationality into economics. J. Econ. Lit., 51(2): 528-543, June 2013.
  41. Maithra Raghu, Katy Blumer, Greg Corrado, Jon Kleinberg, Ziad Obermeyer, and Sendhil Mullainathan. The algorithmic automation problem: Prediction, triage, and human effort. arXiv, March 2019.
  42. Pranav Rajpurkar, Emma Chen, Oishi Banerjee, and Eric J Topol. AI in health and medicine. Nat. Med., 28(1):31-38, January 2022.
  43. Andrew B Rosenkrantz, Tarek N Hanna, Scott D Steenburg, Mary Jo Tarrant, Robert S Pyatt, and Eric B Friedberg. The current state of teleradiology across the united states: A national survey of radiologists' habits, attitudes, and perceptions on teleradiology practice. J. Am. Coll. Radiol., 16(12):1677-1687, December 2019.
  44. Jarrel C Y Seah, Cyril H M Tang, Quinlan D Buchlak, Xavier G Holt, Jeffrey B Ward- man, Anuar Aimoldin, Nazanin Esmaili, Hassan Ahmad, Hung Pham, John F Lambert, Ben Hachey, Stephen J F Hogg, Benjamin P Johnston, Christine Bennett, Luke Oakden- Rayner, Peter Brotchie, and Catherine M Jones. Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, mul- tireader multicase study. Lancet Digit Health, 3(8):e496-e506, August 2021.
  45. Victor S Sheng, Foster Provost, and Panagiotis G Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '08, pages 614-622, New York, NY, USA, August 2008. Association for Computing Machinery.
  46. Yongsik Sim, Myung Jin Chung, Elmar Kotter, Sehyo Yune, Myeongchan Kim, Synho Do, Kyunghwa Han, Hanmyoung Kim, Seungwook Yang, Dong-Jae Lee, and Byoung Wook Choi. Deep convolutional neural network-based software improves radiologist detection of malignant lung nodules on chest radiographs. Radiology, 294(1):199-209, January 2020.
  47. Christopher A Sims. Implications of rational inattention. J. Monet. Econ., 50(3):665-690, April 2003.
  48. Yasasvi Tadavarthi, Brianna Vey, Elizabeth Krupinski, Adam Prater, Judy Gichoya, Nabile Safdar, and Hari Trivedi. The state of radiology AI: Considerations for purchase decisions and current market offerings. Radiol Artif Intell, 2(6):e200004, November 2020.
  49. Philipp Tschandl, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel Codella, Allan Halpern, Monika Janda, Aimilios Lallas, Caterina Longo, Josep Malvehy, John Paoli, Susana Puig, Cliff Rosendahl, H Peter Soyer, Iris Zalaudek, and Harald Kittler. Human- computer collaboration for skin cancer recognition. Nat. Med., 26(8):1229-1234, June 2020.
  50. A Tversky and D Kahneman. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124-1131, September 1974.
  51. Thomas S Wallsten and Adele Diederich. Understanding pooled subjective probability esti- mates. Math. Soc. Sci., 41(1):1-18, January 2001.
  52. S Kevin Zhou, Hayit Greenspan, Christos Davatzikos, James S Duncan, Bram Van Ginneken, Anant Madabhushi, Jerry L Prince, Daniel Rueckert, and Ronald M Summers. A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises. Proc. IEEE, 109(5):820-838, May 2021.

Cited by

Sorted by citations

Authors
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T.
Affiliations
National Bureau of Economic Research (US)
Corresponding author
Agarwal, Nikhil
Venue
National Bureau of Economic Research
Published
2023-07
OpenAlex
W4383744587
Cited by
161
References
17
Citations by year
2026: 43 · 2025: 66 · 2024: 37 · 2023: 15
Type
report
Language
en
Open access
Locations
  • National Bureau of Economic Research (publishedVersion)
  • SSRN Electronic Journal (acceptedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
Alfred P. Sloan Foundation (2022-17182)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
human-AI collaboration, radiologists, AI-assisted diagnosis, contextual information, belief updating, Bayesian updating, diagnostic accuracy, radiology, clinical decision-making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review