CoChat

Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems

Open access
Panigutti, Cecilia, Beretta, Andrea, Giannotti, Fosca, Pedreschi, Dino · 2022 · CHI Conference on Human Factors in Computing Systems
Cited by 126

The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.

Abstract

The field of eXplainable Artificial Intelligence (XAI) focuses on providing explanations for AI systems’ decisions. XAI applications to AI-based Clinical Decision Support Systems (DSS) should increase trust in the DSS by allowing clinicians to investigate the reasons behind its suggestions. In this paper, we present the results of a user study on the impact of advice from a clinical DSS on healthcare providers’ judgment in two different cases: the case where the clinical DSS explains its suggestion and the case it does not. We examined the weight of advice, the behavioral intention to use the system, and the perceptions with quantitative and qualitative measures. Our results indicate a more significant impact of advice when an explanation for the DSS decision is provided. Additionally, through the open-ended questions, we provide some insights on how to improve the explanations in the diagnosis forecasts for healthcare assistants, nurses, and doctors.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3491102.3502104

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 24 references

Research works printed but not linked: 57

  1. Vijay Arya Rachel KE Bellamy Pin-Yu Chen Amit Dhurandhar Michael Hind Samuel C Hoffman Stephanie Houde Q Vera Liao Ronny Luss Aleksandra Mojsilović 2019. One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques. arXiv preprint arXiv:1909.03012(2019). Vijay Arya Rachel KE Bellamy Pin-Yu Chen Amit Dhurandhar Michael Hind Samuel C Hoffman Stephanie Houde Q Vera Liao Ronny Luss Aleksandra Mojsilović 2019. One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques. arXiv preprint arXiv:1909.03012(2019). · arXiv
  2. 10.1145/3351095.3375624 · DOI · Paper page
  3. Alan F. Blackwell . 2021 . Ethnographic artificial intelligence . Interdisciplinary Science Reviews 46 , 1 - 2 (2021), 198–211. https://doi.org/10.1080/03080188.2020.1840226 Alan F. Blackwell. 2021. Ethnographic artificial intelligence. Interdisciplinary Science Reviews 46, 1-2 (2021), 198–211. https://doi.org/10.1080/03080188.2020.1840226 · DOI
  4. Francesco Bodria Fosca Giannotti Riccardo Guidotti Francesca Naretto Dino Pedreschi and Salvatore Rinzivillo. 2021. Benchmarking and survey of explanation methods for black box models. arXiv preprint arXiv:2102.13076(2021). Francesco Bodria Fosca Giannotti Riccardo Guidotti Francesca Naretto Dino Pedreschi and Salvatore Rinzivillo. 2021. Benchmarking and survey of explanation methods for black box models. arXiv preprint arXiv:2102.13076(2021). · arXiv
  5. 10.1016/j.ifacol.2016.10.517 · DOI
  6. 10.1145/3334480.3383047 · DOI · Paper page
  7. 10.1145/3449287 · doi:10.1145/3449287 · Paper page
  8. Béatrice Cahour and Jean-François Forzy . 2009. Does projection into use improve trust and exploration? An example with a cruise control system. Safety science 47, 9 ( 2009 ), 1260–1270. Béatrice Cahour and Jean-François Forzy. 2009. Does projection into use improve trust and exploration? An example with a cruise control system. Safety science 47, 9 (2009), 1260–1270.
  9. Edward Choi Mohammad Taha Bahadori Andy Schuetz Walter F Stewart and Jimeng Sun. 2016. Doctor ai: Predicting clinical events via recurrent neural networks. In Machine learning for healthcare conference. PMLR 301–318. Edward Choi Mohammad Taha Bahadori Andy Schuetz Walter F Stewart and Jimeng Sun. 2016. Doctor ai: Predicting clinical events via recurrent neural networks. In Machine learning for healthcare conference. PMLR 301–318.
  10. Giovanni Comandé . 2020. Unfolding the legal component of trustworthy AI: a must to avoid ethics washing.Version Accepted for Annuario di Diritto Comparato e di Studi Legislativi, forthcoming ( 2020 ). Giovanni Comandé. 2020. Unfolding the legal component of trustworthy AI: a must to avoid ethics washing.Version Accepted for Annuario di Diritto Comparato e di Studi Legislativi, forthcoming (2020).
  11. Ian Covert Scott Lundberg and Su-In Lee. 2020. Explaining by removing: A unified framework for model explanation. arXiv preprint arXiv:2011.14878(2020). Ian Covert Scott Lundberg and Su-In Lee. 2020. Explaining by removing: A unified framework for model explanation. arXiv preprint arXiv:2011.14878(2020). · arXiv
  12. Berkeley Dietvorst and Soaham Bharti. 2019. People Reject Even the Best Possible Algorithm in Uncertain Decision Domains. SSRN Electronic Journal(2019). https://doi.org/10.2139/ssrn.3424158 Berkeley Dietvorst and Soaham Bharti. 2019. People Reject Even the Best Possible Algorithm in Uncertain Decision Domains. SSRN Electronic Journal(2019). https://doi.org/10.2139/ssrn.3424158 · DOI
  13. Berkeley J Dietvorst and Soaham Bharti . 2020. People reject algorithms in uncertain decision domains because they have diminishing sensitivity to forecasting error. Psychological science 31, 10 ( 2020 ), 1302–1314. Berkeley J Dietvorst and Soaham Bharti. 2020. People reject algorithms in uncertain decision domains because they have diminishing sensitivity to forecasting error. Psychological science 31, 10 (2020), 1302–1314.
  14. Berkeley J Dietvorst , Joseph P Simmons , and Cade Massey . 2015 . Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology : General 144 , 1 (2015), 114 . Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. 2015. Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology: General 144, 1 (2015), 114.
  15. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608(2017). Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608(2017). · arXiv
  16. Jinyun Duan Yue Xu and Lyn M Van Swol. 2020. Influence of self-concept clarity on advice seeking and utilisation. Asian Journal of Social Psychology(2020). Jinyun Duan Yue Xu and Lyn M Van Swol. 2020. Influence of self-concept clarity on advice seeking and utilisation. Asian Journal of Social Psychology(2020). · DOI
  17. Wenjuan Fan Jingnan Liu Shuwan Zhu and Panos M Pardalos. 2018. Investigating the impacting factors for the healthcare professionals to adopt artificial intelligence-based medical diagnosis support system (AIMDSS). Annals of Operations Research(2018) 1–26. Wenjuan Fan Jingnan Liu Shuwan Zhu and Panos M Pardalos. 2018. Investigating the impacting factors for the healthcare professionals to adopt artificial intelligence-based medical diagnosis support system (AIMDSS). Annals of Operations Research(2018) 1–26.
  18. Bhavya Ghai , Q Vera Liao , Yunfeng Zhang , Rachel Bellamy , and Klaus Mueller . 2021 . Explainable active learning (xal) toward ai explanations as interfaces for machine teachers . Proceedings of the ACM on Human-Computer Interaction 4, CSCW3(2021) , 1–28. Bhavya Ghai, Q Vera Liao, Yunfeng Zhang, Rachel Bellamy, and Klaus Mueller. 2021. Explainable active learning (xal) toward ai explanations as interfaces for machine teachers. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3(2021), 1–28.
  19. Francesca Gino and Maurice E Schweitzer . 2008 . Take this advice and shove it. In Academy of Management Proceedings, Vol. 2008. Academy of Management Briarcliff Manor , NY 10510 , 1 – 5 . Francesca Gino and Maurice E Schweitzer. 2008. Take this advice and shove it. In Academy of Management Proceedings, Vol. 2008. Academy of Management Briarcliff Manor, NY 10510, 1–5.
  20. Riccardo Guidotti , Anna Monreale , Salvatore Ruggieri , Franco Turini , Dino Pedreschi , and Fosca Giannotti . 2018. A Survey Of Methods For Explaining Black Box Models. ACM CSUR 51, 5 , Article 93 (Aug. 2018 ), 42 pages. Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Dino Pedreschi, and Fosca Giannotti. 2018. A Survey Of Methods For Explaining Black Box Models. ACM CSUR 51, 5, Article 93 (Aug. 2018), 42 pages.
  21. David Gunning . 2017. Explainable artificial intelligence (xai) . Defense Advanced Research Projects Agency (DARPA) , nd Web 2 ( 2017 ). David Gunning. 2017. Explainable artificial intelligence (xai). Defense Advanced Research Projects Agency (DARPA), nd Web 2 (2017).
  22. 10.1145/3442188.3445917 · DOI
  23. 10.3390/ijerph18115562 · DOI
  24. Nigel Harvey and Ilan Fischer . 1997. Taking advice: Accepting help, improving judgment, and sharing responsibility. Organizational behavior and human decision processes 70, 2 ( 1997 ), 117–133. Nigel Harvey and Ilan Fischer. 1997. Taking advice: Accepting help, improving judgment, and sharing responsibility. Organizational behavior and human decision processes 70, 2 (1997), 117–133.
  25. 10.1177/0272989X9501500202 · DOI · Paper page
  26. Robert R Hoffman Shane T Mueller Gary Klein and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608(2018). Robert R Hoffman Shane T Mueller Gary Klein and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608(2018). · arXiv
  27. Maia Jacobs , Melanie F Pradier , Thomas H McCoy , Roy H Perlis , Finale Doshi-Velez , and Krzysztof Z Gajos . 2021. How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection. Translational psychiatry 11, 1 ( 2021 ), 1–9. Maia Jacobs, Melanie F Pradier, Thomas H McCoy, Roy H Perlis, Finale Doshi-Velez, and Krzysztof Z Gajos. 2021. How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection. Translational psychiatry 11, 1 (2021), 1–9.
  28. C Krittanawong . 2018. The rise of artificial intelligence and the uncertain future for physicians. European journal of internal medicine 48 ( 2018 ), e13–e14. C Krittanawong. 2018. The rise of artificial intelligence and the uncertain future for physicians. European journal of internal medicine 48 (2018), e13–e14.
  29. Himabindu Lakkaraju and Osbert Bastani . 2020. ” How Do I Fool You?”: Manipulating User Trust via Misleading Black Box Explanations(AIES ’20) . Association for Computing Machinery , New York, NY, USA , 79–85. https://doi.org/10.1145/3375627.3375833 Himabindu Lakkaraju and Osbert Bastani. 2020. ”How Do I Fool You?”: Manipulating User Trust via Misleading Black Box Explanations(AIES ’20). Association for Computing Machinery, New York, NY, USA, 79–85. https://doi.org/10.1145/3375627.3375833 · DOI · Paper page
  30. Jean-Baptiste Lamy , Boomadevi Sekar , Gilles Guezennec , Jacques Bouaud , and Brigitte Séroussi . 2019. Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach. Artificial intelligence in medicine 94 ( 2019 ), 42–53. Jean-Baptiste Lamy, Boomadevi Sekar, Gilles Guezennec, Jacques Bouaud, and Brigitte Séroussi. 2019. Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach. Artificial intelligence in medicine 94 (2019), 42–53.
  31. John D Lee and Katrina A See . 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 ( 2004 ), 50–80. John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
  32. Thomas Lindow , Josefine Kron , Hans Thulesius , Erik Ljungström , and Olle Pahlm . 2019. Erroneous computer-based interpretations of atrial fibrillation and atrial flutter in a Swedish primary health care setting. Scandinavian journal of primary health care 37, 4 ( 2019 ), 426–433. Thomas Lindow, Josefine Kron, Hans Thulesius, Erik Ljungström, and Olle Pahlm. 2019. Erroneous computer-based interpretations of atrial fibrillation and atrial flutter in a Swedish primary health care setting. Scandinavian journal of primary health care 37, 4 (2019), 426–433.
  33. Scott M Lundberg and Su-In Lee . 2017 . A unified approach to interpreting model predictions . In Proceedings of the 31st international conference on neural information processing systems. 4768–4777 . Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the 31st international conference on neural information processing systems. 4768–4777.
  34. Gianclaudio Malgieri and Giovanni Comandé. 2017. Why a right to legibility of automated decision-making exists in the general data protection regulation. International Data Privacy Law(2017). Gianclaudio Malgieri and Giovanni Comandé. 2017. Why a right to legibility of automated decision-making exists in the general data protection regulation. International Data Privacy Law(2017). · DOI · Paper page
  35. 10.1038/s42256-021-00323-0 · DOI
  36. 10.1109/ISCC53001.2021.9631485 · DOI
  37. 10.1145/3411763.3451759 · DOI · Paper page
  38. Emanuele Neri Francesca Coppola Vittorio Miele Corrado Bibbolino and Roberto Grassi. 2020. Artificial intelligence: Who is responsible for the diagnosis? Emanuele Neri Francesca Coppola Vittorio Miele Corrado Bibbolino and Roberto Grassi. 2020. Artificial intelligence: Who is responsible for the diagnosis? · DOI · Paper page
  39. Mahsan Nourani , Joanie King , and Eric Ragan . 2020 . The role of domain expertise in user trust and the impact of first impressions with intelligent systems . In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 8. 112–121 . Mahsan Nourani, Joanie King, and Eric Ragan. 2020. The role of domain expertise in user trust and the impact of first impressions with intelligent systems. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 8. 112–121.
  40. Cecilia Panigutti , Riccardo Guidotti , Anna Monreale , and Dino Pedreschi . 2019 . Explaining multi-label black-box classifiers for health applications . In International Workshop on Health Intelligence. Springer, 97–110 . Cecilia Panigutti, Riccardo Guidotti, Anna Monreale, and Dino Pedreschi. 2019. Explaining multi-label black-box classifiers for health applications. In International Workshop on Health Intelligence. Springer, 97–110.
  41. Cecilia Panigutti , Anna Monreale , Giovanni Comandé , and Dino Pedreschi . 2022. Ethical , societal and legal issues in deep learning for healthcare . In Deep Learning in Biology and Medicine . World Scientific Publishing . Cecilia Panigutti, Anna Monreale, Giovanni Comandé, and Dino Pedreschi. 2022. Ethical, societal and legal issues in deep learning for healthcare. In Deep Learning in Biology and Medicine. World Scientific Publishing.
  42. 10.1145/3351095.3372855 · DOI · Paper page
  43. Michael G Pratt . 2009. From the editors : For the lack of a boilerplate: Tips on writing up (and reviewing) qualitative research. Michael G Pratt. 2009. From the editors: For the lack of a boilerplate: Tips on writing up (and reviewing) qualitative research.
  44. 10.1145/2939672.2939778 · DOI · Paper page
  45. 10.1007/978-3-030-57321-8_24 · DOI · Paper page
  46. 10.1093/jamia/ocaa296 · DOI
  47. Lucy Shinners , Christina Aggar , Sandra Grace , and Stuart Smith . 2020. Exploring healthcare professionals ’ understanding and experiences of artificial intelligence technology use in the delivery of healthcare: an integrative review. Health informatics journal 26, 2 ( 2020 ), 1225–1236. Lucy Shinners, Christina Aggar, Sandra Grace, and Stuart Smith. 2020. Exploring healthcare professionals’ understanding and experiences of artificial intelligence technology use in the delivery of healthcare: an integrative review. Health informatics journal 26, 2 (2020), 1225–1236.
  48. Linda J Skitka , Kathleen L Mosier , and Mark Burdick . 1999. Does automation bias decision-making?International Journal of Human-Computer Studies 51, 5 ( 1999 ), 991–1006. Linda J Skitka, Kathleen L Mosier, and Mark Burdick. 1999. Does automation bias decision-making?International Journal of Human-Computer Studies 51, 5 (1999), 991–1006.
  49. Janet A Sniezek and Timothy Buckley . 1995. Cueing and cognitive conflict in judge-advisor decision making. Organizational behavior and human decision processes 62, 2 ( 1995 ), 159–174. Janet A Sniezek and Timothy Buckley. 1995. Cueing and cognitive conflict in judge-advisor decision making. Organizational behavior and human decision processes 62, 2 (1995), 159–174.
  50. Janet A Sniezek and Lyn M Van Swol . 2001. Trust , confidence, and expertise in a judge-advisor system. Organizational behavior and human decision processes 84, 2 ( 2001 ), 288–307. Janet A Sniezek and Lyn M Van Swol. 2001. Trust, confidence, and expertise in a judge-advisor system. Organizational behavior and human decision processes 84, 2 (2001), 288–307.
  51. MT Spil and WR Schuring. 2006. E-Health Systems Diffusion and Use: The Innovation. The Users and the Use IT Model(2006). MT Spil and WR Schuring. 2006. E-Health Systems Diffusion and Use: The Innovation. The Users and the Use IT Model(2006). · doi:10.4018/978-1-59140-423-1
  52. Sana Tonekaboni , Shalmali Joshi , Melissa D McCradden , and Anna Goldenberg . 2019 . What clinicians want: contextualizing explainable machine learning for clinical end use . In Machine Learning for Healthcare Conference. PMLR, 359–380 . Sana Tonekaboni, Shalmali Joshi, Melissa D McCradden, and Anna Goldenberg. 2019. What clinicians want: contextualizing explainable machine learning for clinical end use. In Machine Learning for Healthcare Conference. PMLR, 359–380.
  53. Viswanath Venkatesh and Hillol Bala . 2008. Technology acceptance model 3 and a research agenda on interventions. Decision sciences 39, 2 ( 2008 ), 273–315. Viswanath Venkatesh and Hillol Bala. 2008. Technology acceptance model 3 and a research agenda on interventions. Decision sciences 39, 2 (2008), 273–315.
  54. Viswanath Venkatesh , Michael G Morris , Gordon B Davis , and Fred D Davis . 2003. User acceptance of information technology: Toward a unified view. MIS quarterly ( 2003 ), 425–478. Viswanath Venkatesh, Michael G Morris, Gordon B Davis, and Fred D Davis. 2003. User acceptance of information technology: Toward a unified view. MIS quarterly (2003), 425–478.
  55. Himanshu Verma , Roger Schaer , Julien Reichenbach , Jreige Mario , John O Prior , Florian Evéquoz , and Adrien Raphaël Depeursinge . 2021. On Improving Physicians ’ Trust in AI: Qualitative Inquiry with Imaging Experts in the Oncological Domain. ( 2021 ). Himanshu Verma, Roger Schaer, Julien Reichenbach, Jreige Mario, John O Prior, Florian Evéquoz, and Adrien Raphaël Depeursinge. 2021. On Improving Physicians’ Trust in AI: Qualitative Inquiry with Imaging Experts in the Oncological Domain. (2021).
  56. 10.1145/2930238.2930290 · DOI · Paper page
  57. 10.1145/3394486.3403067 · DOI

Cited by

See all 126

Metadata

Authors
Panigutti, C., Beretta, A., Giannotti, F., & Pedreschi, D.
Venue
CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2022-04-29
Pages
1-9
OpenAlex
W4225105010
Cited by
126
References
24
FWCI
15.73
Type
conference paper
Language
en
Open access
hybrid
Funders
European Commission (GA-834756-XAI)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems. https://app.cochat.ai/papers/10.1145/3491102.3502104
The paper
Panigutti, C., Beretta, A., Giannotti, F., & Pedreschi, D. (2022). Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems. CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3491102.3502104

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.