CoChat
Sign Up
Cited by 342Explainable Artificial Intelligence (XAI)Artificial Intelligence in Healthcare and EducationDecision-Making and Behavioral Economics

Understanding, explaining, and utilizing medical artificial intelligence

Open access
Cadario, Romain, Longoni, Chiara, Morewedge, Carey K.·2021·Nature Human Behaviour
DOI 10.1038/s41562-021-01146-0
Open this paper in CoChat — free accountOpen PDF

Abstract

Medical artificial intelligence is cost-effective and scalable and often outperforms human providers, yet people are reluctant to use it. We show that resistance to the utilization of medical artificial intelligence is driven by both the subjective difficulty of understanding algorithms (the perception that they are a 'black box') and by an illusory subjective understanding of human medical decision-making. In five pre-registered experiments (1-3B: N = 2,699), we find that people exhibit an illusory understanding of human medical decision-making (study 1). This leads people to believe they better understand decisions made by human than algorithmic healthcare providers (studies 2A,B), which makes them more reluctant to utilize algorithmic than human providers (studies 3A,B). Fortunately, brief interventions that increase subjective understanding of algorithmic decision processes increase willingness to utilize algorithmic healthcare providers (studies 3A,B). A sixth study on Google Ads for an algorithmic skin cancer detection app finds that the effectiveness of such interventions generalizes to field settings (study 4: N = 14,013).

AI snapshot

Written from the paper’s full text.

TL;DR

People report greater subjective understanding of medical decisions made by human than algorithmic providers, but their objective understanding is similarly poor; this illusory understanding contributes to algorithm aversion. Brief explanations of how medical AI works increase subjective understanding and willingness to use algorithmic providers, including in a Google Ads field study.

Key findings

  • In study 1 (N = 297), making people explain provider decision processes reduced subjective understanding for human providers (Mpre = 5.08 vs Mpost = 4.52, ΔH = −0.56, P < 0.001) but not for algorithmic providers (Mpre = 3.99 vs Mpost = 3.83, ΔA = −0.16, P = 0.14); the interaction was significant (ΔH − ΔA = −0.40, P = 0.009).
  • In studies 2A/B, participants claimed greater subjective understanding of human than algorithmic providers (2A: β = −0.82, P < 0.001; 2B: β = −0.46, P = 0.001), while objective understanding did not differ by provider (2A: β = 0.02, P = 0.91; 2B: β = 0.11, P = 0.43).
  • In studies 3A/B, interventions explaining decision processes increased subjective understanding and reduced algorithm aversion; utilization intentions increased more for algorithmic providers (3A: β = 0.83, P < 0.001; 3B: β = 0.70, P < 0.001) than for human providers (3A: β = 0.34, P = 0.008; 3B: β = 0.14, P = 0.41).
  • In study 4 (N = 14,013 impressions), the intervention advertisement had a higher click-through rate (6.36%) than the control advertisement (3.29%; β = 0.69, z = 8.17, P < 0.001, d = 0.38).

Methods

  • Five pre-registered online experiments (studies 1, 2A/B, 3A/B; N = 2,699) plus a sixth pre-registered online field study on Google Ads (study 4; N = 14,013). Participants were US residents recruited from Lucid, Amazon Mechanical Turk, and Google Ads, in a skin cancer detection context.
  • Designs included: study 1, 2 (provider: human vs algorithm) × 2 (rating order: pre- vs post-intervention) mixed design; studies 2A/B, 2 (provider) × 2 (understanding type: subjective vs objective) between-subjects; studies 3A/B, 2 (provider) × 2 (intervention: control vs intervention) between-subjects; study 4, intervention vs control advertisement.
  • Analyses used regressions, moderated mediation with 5,000 bootstrap samples, and logistic regression for click-through rate. Pre-registrations, raw data, and Stata syntax are available at https://osf.io/taqp8/.

Limitations

  • The paper states that mediation analysis has limitations and invites replication with alternative manipulations, designs, and measures; it also notes that the ancillary objective understanding measure may have been sensitive, and that wider-range questions might detect subtle changes. For study 4, the authors state that ad assignment was not random because Google Ads optimized for clicks, which may have generated unintended variance, though they believe replication in studies 3A/B makes it unlikely the results are solely due to ad optimization.
  • Evident from the text: studies 1–3B measure subjective understanding and healthcare utilization intentions rather than actual clinical uptake, and study 4 measures ad click-through rather than real patient utilization; the samples are online panels and Google Ads users, not a clinical patient population.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41562-021-01146-0

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

0 of 28 cited sources checked · 0 supported · 0 flagged

  • Findingown data
    § Study 4.

    In a field study using Google Ads for SkinVision, the ad that explained how the app works had a higher click-through rate (6.36%) than the control ad (3.29%; β=0.69, z=8.17, P<0.001, d=0.38).

    “A logistic regression revealed that the click-through rate was higher in the intervention advertisement (6.36%) compared with the control advertisement (3.29%, β = 0.69, z = 8.17, P < 0.001, d = 0.38).”
    • Also stated in: § Discussion · plain
      “Indeed, we find increased advertising efficacy for SkinVision, a skin cancer detection app, when advertising included language explaining how it works.”
    Quote in the paperSubstantial claim
  • Findingown data
    § Abstract

    People exhibit similar objective understanding of medical decisions made by human and AI providers.

    “but they exhibit similar objective understanding of decisions made by both providers (people actually know as little about both decision processes).”
    • Also stated in: § Studies 2A,B. · plain
      “whereas their objective understanding did not differ by provider (2A: β = 0.02, t = 0.11, P = 0.91; 2B: β = 0.11, t = 0.79, P = 0.43)”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' self-reported subjective understanding accurately reflects their subjective understanding.
  • Findingown data
    § Studies 2A,B.

    In studies 3A and 3B, the interventions increased subjective understanding of decisions made by algorithmic providers and human providers.

    “It increased subjective understanding of the decision made by algorithmic providers (3A: Δ = 1.42, t = 12.51, P < 0.001; 3B: Δ = 0.56, t = 6.21,P < 0.001) and human providers (3A: Δ = 1.04, t = 11.08,P < 0.001; 3B: Δ = 1.39, t = 11.85,P < 0.001).”
    Quote in the paperSubstantial claim
  • Findingown data
    § Studies 2A,B.

    In studies 3A and 3B, the indirect effect of healthcare provider on utilization intentions through subjective understanding was stronger in the control condition than in the intervention condition.

    “the indirect effect of healthcare provider on utilization intentions through subjective understanding was significantly stronger (3A: Δ = 0.20, 95% CI [0.05, 0.37]; 3B: Δ = 0.60, 95% CI [0.35, 0.89]) in the control condition”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Healthcare utilization intentions are a valid proxy for actual healthcare utilization.
  • Findingown data
    § Abstract

    People claim greater subjective understanding for medical decisions made by human than AI providers.

    “Indeed, we find that people claim greater subjective understanding for medical decisions made by human than AI providers (people feel that they know more), but they exhibit similar objective understanding of decisions made by both providers”
    • Also stated in: § Greater reductions in pre-intervention to post-intervention ratings indicate a larger illusion of understanding. · plain
      “Participants claimed greater subjective understanding of a human than an algorithmic decision processes (pre-intervention: β = 1.08, z = 5.02, P < 0.001; post-intervention: β = 0.69, z = 3.19, P = 0.001).”
    • Also stated in: § Studies 2A,B. · plain
      “Participants claimed greater subjective understanding of medical decisions made by human than algorithmic providers (2A: β = -0.82,t = -6.01,P < 0.001; 2B: β = -0.46,t = -3.33,P = 0.001)”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' self-reported subjective understanding accurately reflects their subjective understanding.
  • May combine two claims. 22
  • Classified as probably not a substantial claim. 24
  • The quote only partly states this claim. 3

  • May combine two claims.

    Findingown data
    § Greater reductions in pre-intervention to post-intervention ratings indicate a larger illusion of understanding.

    In study 1, the intervention reduced subjective understanding of human providers' decisions but not algorithmic providers' decisions.

    “the intervention significantly reduced reported subjective understanding of decisions made by human providers (M pre = 5.08 versus M post = 4.52, Δ H = -0.56,z = -5.27,P < 0.001), but not of decisions made by algorithmic providers (M pre = 3.99 versus M post = 3.83, Δ A = -0.16,z = -1.47,P = 0.14).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Study 4.

    In study 4, the Google Ads campaign generated 14,013 impressions and 698 clicks.

    “The advertisements generated 14,013 impressions (that is, how many times the ad was viewed across the following demographics: age: 18-24 years: 6.9%, 25-35 years: 5.7%, 35-44 years: 7.3%, 45-54 years: 15.1%, 55-64 years: 35.6%, 65+ years: 29.4%; gender: 78.8% female) and generated 698 clicks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Click-through rate on Google Ads is a valid measure of healthcare utilization.

  • The quote only partly states this claim.

    Findingown data
    § Studies 2A,B.

    In studies 2A and 2B, the provider by understanding-type interaction was significant.

    “creating the predicted provider × understanding interaction (2A: β = -0.80,t = -4.19,P < 0.001; 2B: β = -0.35,t = -1.78,P = 0.08).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Discussion

    In an ancillary study of skin cancer smartphone app marketing, performance-related keywords described 57-64% of apps, while process-related keywords described 21%.

    “Indeed, in an ancillary study examining the marketing of skin cancer smartphone applications (Supplementary Appendix 8), we find that performance-related keywords were used to describe 57-64% of the applications, whereas process-related keywords were used to describe 21% of the applications.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Studies 2A,B.

    In the ancillary experiment, the interventions used in studies 3A and 3B significantly increased subjective understanding but did not influence objective understanding.

    “Results from this experiment show that, relative to a control condition, the interventions used in studies 3A and 3B significantly increased subjective understanding (respectively, β = 0.67, t = 4.90, P < 0.001; β = 0.82, t = 6.05,P < 0.001) but did not influence objective understanding (respectively, P = 0.78 and P = 0.89; all interactions, ts > 3.60, ps < 0.001).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Greater reductions in pre-intervention to post-intervention ratings indicate a larger illusion of understanding.

    In study 1, the provider by rating-order interaction on subjective understanding was significant, with a larger reduction for human than algorithm.

    “The analyses revealed the predicted significant interaction (Δ H -Δ A = -0.40,z = -2.61,P = 0.009; Fig. 1).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Software and code

    In study 1, asking participants to explain a human provider's decision process reduced their subjective understanding of decisions made by human providers, but did not reduce it for decisions made by algorithmic providers.

    “Making people explain provider decision processes reduces their subjective understanding of decisions made by human providers, but not decisions made by algorithmic providers (study 1; N = 297).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § Abstract

    Algorithmic decision processes are perceived as opaque by patients because they are often actually opaque to medical professionals and even their creators, who cannot explain them.

    “Decision processes used by algorithms are perceived to be opaque by patients, because they are often actually opaque to medical professionals and even their creators, who cannot explain them 15,16 .”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [15] Castelvecchi, D. Can we open the black box of AI? Nature 538, 20–23 (2016). source

      Not checked in the free run.

    • cites [16] Kroll, J. A. et al. Accountable algorithms. Univ. Pa. Law Rev. 165, 633 (2016).

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § Abstract

    Medical AI is cost-effective, scalable, and often provides expert-level accuracy or outperforms human providers.

    “Algorithmic-based healthcare services (medical 'artificial intelligence' (AI)) are cost-effective and scalable and provide expert-level accuracy in applications ranging from the detection of skin cancer 5 and emergency department triage 6,7 to diagnoses of COVID-19 from chest X-rays 8 .”
    • Also stated in: § Understanding, explaining, and utilizing medical artificial intelligence · plain
      “Medical artificial intelligence is cost-effective and scalable and often outperforms human providers, yet people are reluctant to use it.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [5] Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017). source

      Not checked in the free run.

    • cites [6] Laranjo, L. et al. Conversational agents in healthcare: a systematic review. J. Am. Med. Inform. Assoc. 25, 1248–1258 (2018). source

      Not checked in the free run.

    • cites [7] Goto, T., Camargo, C. A. Jr, Faridi, M. K., Freishtat, R. J. & Hasegawa, K. Machine learning-based prediction of clinical outcomes for children during emergency department triage. JAMA Netw. Open 2, e186937–e186937 (2019). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § Studies 2A,B.

    The objective understanding quiz was developed in consultation with dermatologists and app developers.

    “we developed a quiz in consultation with experts in the relevant medical domain (preliminary visual inspection of moles): a team of dermatologists at a medical school in the Netherlands, and a team of developers of a popular skin cancer detection application in Europe.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data
    § Methods

    Condition assignments were random in all studies, with randomization administered by Qualtrics software.

    “Condition assignments were random in all our studies, with randomization administered by Qualtrics software.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § Studies 2A,B.

    The ancillary experiment comparing interventions and understanding types was pre-registered.

    “we ran an additional three (intervention: control, study 3A, study 3B) × 2 (understanding: subjective, objective) between-subjects pre-registered experiment with 601 participants on Amazon Mechanical Turk.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § Abstract

    All conditions, measures, and exclusions are reported, and data and pre-registrations are available.

    “All conditions, measures and exclusions are reported; data and pre-registrations are available at https://osf.io/taqp8/.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § Methods

    Amazon Mechanical Turk is not nationally representative but yields high data quality through attention checks and worker reputation scores.

    “Amazon Mechanical Turk is an online sample recruiting platform that, although not nationally representative, yields high data quality through the use of attention checks 44 and worker reputation scores 45 .”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [44] Paolacci, G., Chandler, J. & Ipeirotis, P. G. Running experiments on Amazon Mechanical Turk. Judgm. Decis. Mak. 5, 411–419 (2010). source

      Not checked in the free run.

    • cites [45] Peer, E., Vosgerau, J. & Acquisti, A. Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behav. Res. Methods 46, 1023–1031 (2014). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Study design and conductcitation
    § Methods

    In studies 1-3B, participants were recruited from Lucid and Amazon Mechanical Turk, and MTurk participants were selected with an approval rating above 95%.

    “Following prior research, we selected participants with an approval rating above 95% on Amazon Mechanical Turk 45 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] Peer, E., Vosgerau, J. & Acquisti, A. Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behav. Res. Methods 46, 1023–1031 (2014). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § Methods

    Studies 1-3B received IRB approval from Boston University's Charles River Campus (protocol 3632E), and all participants gave informed consent.

    “Studies 1-3B were approved for use with human participants by the Institutional Review Board on the Charles River Campus at Boston University (protocol 3632E); informed consent was obtained for all participants.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § Methods

    All manipulations and measures are reported.

    “All manipulations and measures are reported.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning
    § Discussion

    The results suggest an egocentric bias in the illusion of understanding, such that the illusion may be strongest for assessments of people, the causal systems most similar to ourselves.

    “Our results suggest an egocentric bias in this illusion of understanding, that the illusion may loom largest in assessments of people-the causal systems most similar to ourselves.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § Abstract

    Coupled effects make people aware of their limited understanding of algorithmic medical decisions but lead them to overestimate their understanding of human medical decisions.

    “We suggest that these coupled effects make people aware of their limited understanding of medical decisions made by algorithmic providers, but also lead them to overestimate their understanding of (similarly opaque) decisions made by human providers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § Methods

    Pre-registrations, raw data, and Stata syntax files are available on the Open Science Framework.

    “Pre-registrations, raw data and Stata syntax files are available on the Open Science Framework at https://osf.io/ taqp8/.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data
    § Abstract

    The five online experiments and the sixth field study were pre-registered.

    “We tested our predictions in a set of five pre-registered online experiments with nationally representative and convenience samples (studies 1, 2A,B and 3A,B: N = 2,699), and a sixth pre-registered online field study on Google Ads (study 4: N = 14,013).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § Discussion

    The one-page visual or sentence intervention could be easily adapted to other domains and procedures.

    “which could be easily adapted to other domains and procedures.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § Study 1.

    Participants were randomly assigned to conditions in the experiments.

    “Participants were randomly assigned to one of the two provider conditions between subjects, and ratings of subjective understanding were repeated within subjects.”
    • Also stated in: § Studies 2A,B. · plain
      “Participants were randomly assigned to one of four conditions in a 2 (provider: human, algorithm) × 2 (understanding type: subjective, objective) between-subject design.”
    • Also stated in: § Studies 2A,B. · plain
      “In both studies, participants were randomly assigned to one of four conditions in a 2 (provider: human, algorithm) × 2 (intervention: control, intervention) between-subject design.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § Discussion

    Improving subjective understanding of how medical AI works may benefit both consumer adoption and firms' positioning.

    “Improving subjective understanding of how medical AI works may then not only provide beneficent insights for increasing consumer adoption but also for firms seeking to improve their positioning.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Providers tend to present AI provider decisions as a 'black box' because the technologies used in algorithmic-based medical applications are complex.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § Abstract

    All experiments were approved by institutional review boards for use with human subjects.

    “All experiments were approved for use with human subjects by institutional review boards.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § Methods

    Study 4 was approved by the Erasmus University Erasmus Research Institute of Management Internal Review Board.

    “This study was approved by the Erasmus University Erasmus Research Institute of Management Internal Review Board.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning
    § Methods

    In Study 4, individual participants could not be identified, interacted with, or asked for consent.

    “It is impossible to identify, interact with and obtain consent from individual participants.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Study design and conductown data
    § Studies 2A,B.

    The objective understanding quiz had acceptable internal consistency in both provider conditions.

    “Cronbach's α = 0.65 in the human condition and α = 0.67 in the algorithm condition”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § Additional information

    For Study 4, allocation was performed using the Google platform.

    “Study 4: Allocation from Google platform”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § Methods

    The present research involved no more than minimal risks, and all study participants were 18 years of age or older.

    “The present research involved no more than minimal risks, and all study participants were 18 years of age or older.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data
    § Methods

    The number of participants screened out for failing attention checks was N=211 in study 1, N=255 in study 2A, N=260 in study 3A, and N=378 in study 3B.

    “We screened out N = 211 participants in study 1, N = 255 participants in study 2A, N = 260 participants in study 3A and N = 378 participants in study 3B.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § Data availability

    No exclusions were made in the data analyses.

    “No exclusions in data analyses”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § Additional information

    No participant declined participation.

    “No participant declined participation”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § Software and code

    Pre-registrations, raw data, and syntax files for the studies are available on the Open Science Framework.

    “Pre-registrations, raw data and syntax files are available on the Open Science Framework at https://osf.io/taqp8/”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § Methods

    In studies 1, 2A, and 3A,B, as pre-registered, surveys automatically excluded participants who failed an attention check at the beginning and prior to manipulation.

    “In studies 1, 2A and 3A,B, as pre-registered, we programmed our surveys to automatically exclude participants who failed an attention check at the very beginning of the survey and prior to any manipulation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Discussion

    People exhibit an illusion of explanatory depth for causal systems in many domains, claiming to understand ideas, technologies, and biological systems better than they objectively do.

    “Finally, people do exhibit an illusion of explanatory depth of causal systems [21][22][23]42 in many domains, claiming to better understand a variety of ideas, technologies and biological systems than they objectively do.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21][22][23]42

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § Additional information

    For Studies 1, 2A, 2B, 3A, and 3B, randomization was performed using the Qualtrics randomizer.

    “Randomization Studies 1, 2A, 2B, 3A, 3B: Qualtrics randomizer”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § Discussion

    People are generally averse to using algorithms for tasks usually done by humans, because algorithms are perceived as unable to learn and improve from mistakes, unsuitable for subjective or experiential tasks, unable to adapt to unique or mutable circumstances, and unable to carry responsibility for negative outcomes.

    “The literature on algorithm aversion finds that people are generally averse to using algorithms for tasks that are usually done by humans 34,35 , on the grounds that algorithms are perceived as unable to learn and improve from their mistakes 36 , unsuitable for subjective or experiential tasks 37,38 , unable to adapt to unique or mutable circumstances 10,39 and unable to carry responsibility for”
    Quote in the paperSubstantial claim0 of 5 sources checked
    • cites [34] Dawes, R., Faust, D. & Meehl, P. Clinical versus actuarial judgment. Science 243, 1668–1674 (1989). source

      Not checked in the free run.

    • cites [35] Yeomans, M., Shah, A., Mullainathan, S. & Kleinberg, J. Making sense of recommendations. J. Behav. Decis. Mak. 32, 403–414 (2019). source

      Not checked in the free run.

    • cites [36] Dietvorst, B. J., Simmons, J. P. & Massey, C. Algorithm aversion: people erroneously avoid algorithms after seeing them err. J. Exp. Psychol. Gen. 144, 114 (2015). source

      Not checked in the free run.

    • cites [37] Castelo, N., Bos, M. W. & Lehmann, D. R. Task-dependent algorithm aversion. J. Mark. Res. 56, 809–825 (2019). source

      Not checked in the free run.

    • cites [38] Longoni, C. & Cian, L. Artificial intelligence in utilitarian vs. hedonic contexts: the “word-of-machine” effect. J. Market. https://doi.org/10.1177/0022242920957347 (2020). source

      Not checked in the free run.

38 of 38

  • The paper relies on this without support.

    Healthcare utilization intentions are a valid proxy for actual healthcare utilization.

    “Healthcare utilization intentions were measured in study 3A with the item”
    “they reported their healthcare utilization intentions.”

  • The paper relies on this without support.

    The skin cancer detection context is representative of medical AI decision-making generally.

    “In the same skin cancer detection context as study 1”
    “medical AI”

  • The paper relies on this without support.

    Human and algorithmic providers in the studies are comparable in quality or accuracy.

    “comparable human providers”

  • The paper relies on this without support.

    Participants' self-reported subjective understanding accurately reflects their subjective understanding.

    “participants reported their subjective understanding”
    “Subjective understanding was measured in study 3A with the item used in study 2A.”

  • The paper relies on this without support.

    Click-through rate on Google Ads is a valid measure of healthcare utilization.

    “tested the impact of that intervention on ad click-through rates.”

  • The paper relies on this without support.

    The Google Ads platform serves ads according to the campaign goal (maximizing clicks), so the frequency of appearance of different ads was not random.

    “The frequency of appearance of the different ads was not random, because the Google Ads platform serves ads according to the goal of the campaign (maximizing clicks, in our case).”

  • The paper relies on this without support.

    Providers tend to present AI provider decisions as a 'black box' because the technologies used in algorithmic-based medical applications are complex.

    “Because the technologies used in algorithmic-based medical applications are complex, providers tend to present AI provider decisions as a 'black box' .”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Lack of transparency may prevent formation of reliable case-by-case expectations of AI accuracy.

    “First, it may prevent the formation of reliable case-by-case expectations of AI accuracy [65 , 66] .”

    From Enhancing human-AI collaboration: The case of colonoscopy (2024) · cites this paper as [65]

    The Abstract says AI decision processes are perceived as opaque and that patients poorly understand how algorithms make medical decisions. The Discussion says explaining the criteria or process increased acceptance of a diagnostic tool. Neither passage addresses whether opacity prevents reliable case-by-case expectations of AI accuracy; the paper supports the transparency concern, but not that specific consequence. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Algorithmic aversion refers to a phenomenon where clinicians or healthcare professionals exhibit reluctance or skepticism toward relying on AI algorithms for making diagnostic or treatment decisions in medical imaging.

    “Algorithmic aversion refers to a phenomenon where clinicians or healthcare professionals exhibit reluctance or skepticism toward relying on AI algorithms for making diagnostic or treatment decisions in medical imaging.46”

    From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [46]

  • Patients penalize and resist the clinical use of AI because they feel they can understand human providers' decision-making as explained, even though human and AI decisions are probably equally unobservable to them.

    “Decisions made by human physicians or AI are probably equally unobservable to a patient—but because patients feel that they can understand the decision-making as explained by human providers, they ultimately penalize and resist the clinical use of AI (122).”

    From The impact of generative artificial intelligence on socioeconomic inequalities and policy making (2024) · cites this paper as [122]

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentz = -2.61reported p = .009 · recomputed p = .009

    “,z = -2.61,p = .009”

  • Consistentz = 5.02reported p < .001 · recomputed p < .001

    “z = 5.02, p < .001”

  • Consistentz = 3.19reported p = .001 · recomputed p = .001

    “z = 3.19, p = .001”

  • Consistentz = -5.27reported p < .001 · recomputed p < .001

    “,z = -5.27,p < .001”

  • Consistentz = -1.47reported p = .14 · recomputed p = .142

    “,z = -1.47,p = .14”

  • Consistentz = 8.17reported p < .001 · recomputed p < .001

    “z = 8.17, p < .001”

6 of 6

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Reported understanding differences are larger, not smaller, after intervention

    Major

    For both studies, the absolute reported provider difference is larger in the intervention condition than in the control condition, contrary to the stated direction of the result.

  • Subjective-understanding coefficients conflict with the stated provider coding

    Minor

    With algorithm coded 0 and human coded 1, a negative provider coefficient indicates lower understanding for human providers. The text interprets the negative coefficients for both studies as greater understanding for human providers.

2 of 2

Cites

Flagged first, then by citations

2 of the 48 references couldn’t be matched to a known paper.

Research works printed but not linked: 2

  1. Hao, K. Doctors are using AI to triage covid-19 patients. The tools may be here to stay. MIT Technology Review (23 April 2020).
  2. Kroll, J. A. et al. Accountable algorithms. Univ. Pa. Law Rev. 165, 633 (2016).

Cited by

Sorted by citations

Related

Authors
Cadario, R., Longoni, C., & Morewedge, C. K.
Affiliations
  • Erasmus University Rotterdam (NL)
  • Boston University (US)
Corresponding author
Cadario, Romain
ORCID
Venue
Nature Human Behaviour
Publisher
Springer Science and Business Media LLC
Published
2021-06-28
Volume
5
Issue
12
Pages
1636-1642
ISSN
2397-3374 (ISSN-L)
DOI
10.1038/s41562-021-01146-0
OpenAlex
W3095202475
Semantic Scholar
2dbd832cce2b900341e0634995e5a093c23ccb0b
Cited by
342
References
48
Citations by year
2026: 74 · 2025: 93 · 2024: 87 · 2023: 54 · 2022: 23 · 2021: 11
FWCI
15.46
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Human Behaviour (publishedVersion, public-domain)
  • PubMed (publishedVersion)
  • EUR Research Repository (Erasmus University Rotterdam) (acceptedVersion, public-domain)
  • EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, other-oa)
  • RePEc: Research Papers in Economics (submittedVersion)
  • Archivio istituzionale della Ricerca - Bocconi (submittedVersion)
  • PsyArXiv (OSF Preprints) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion)
Funders
Erasmus Research Institute of Management
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
Keywords
artificial intelligence in healthcare, algorithmic decision-making, trust in AI, resistance to AI adoption, algorithmic transparency, brief intervention, skin cancer detection, Google AdWords
MeSH terms
Clinical Decision-Making, Adult, Algorithms, Artificial Intelligence, Delivery of Health Care, Female, Humans, Male, Middle Aged

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review