CoChat
Sign Up
Cited by 117Artificial Intelligence in Healthcare and EducationExplainable Artificial Intelligence (XAI)Human-Automation Interaction and Safety

Rams, hounds and white boxes: investigating human–AI collaboration protocols in medical diagnosis

Open access
Cabitza, Federico, Campagner, Andrea, Ronzio, Luca, Cameli, Matteo, Mandoli, Giulia Elena, Pastore, Maria Concetta, Sconfienza, Luca Maria, Folgado, Duarte +2 more·2023·Artificial Intelligence in Medicine
DOI 10.1016/j.artmed.2023.102506
Open this paper in CoChat — free account

Abstract

In this paper, we study human-AI collaboration protocols, a design-oriented construct aimed at establishing and evaluating how humans and AI can collaborate in cognitive tasks. We applied this construct in two user studies involving 12 specialist radiologists (the knee MRI study) and 44 ECG readers of varying expertise (the ECG study), who evaluated 240 and 20 cases, respectively, in different collaboration configurations. We confirm the utility of AI support but find that XAI can be associated with a "white-box paradox", producing a null or detrimental effect. We also find that the order of presentation matters: AI-first protocols are associated with higher diagnostic accuracy than human-first protocols, and with higher accuracy than both humans and AI alone. Our findings identify the best conditions for AI to augment human diagnostic skills, rather than trigger dysfunctional responses and cognitive biases that can undermine decision effectiveness.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.artmed.2023.102506

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Lack of transparency can induce high variability in a-priori tendencies toward evaluation of AI competence, leading to general trust (algorithm appreciation) or general mistrust (algorithm aversion).

    “Second, it can induce high variability in the a-priori tendencies toward the evaluation of AI competence, leading to general trust (algorithm appreciation) or general mistrust (algorithm aversion) towards AI technology [67][68][69][70] .”

    From Enhancing human-AI collaboration: The case of colonoscopy (2024) · cites this paper as [68]

    The Introduction (p. 2) mentions algorithmic aversion, and the Discussion (p. 18) reports that explanations increased clinicians’ trust in AI advice. The paper does not establish that lack of transparency causes variability in a-priori assessments of AI competence or leads to both general appreciation and general aversion. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • An evidence-based cognitive model for human-AI interaction would allow designing the most effective human-AI collaboration protocols.

    “First and foremost, establishing an evidence-based cognitive model for human-AI interaction is imperative, as this would allow for designing the most effective human-AI collaboration protocols [68 , 97 , 98] .”

    From Enhancing human-AI collaboration: The case of colonoscopy (2024) · cites this paper as [68]

    The Discussion (p. 16) proposes human–AI collaboration protocols as a way to design and evaluate interactions, and the Conclusion (p. 21) reports that testing different protocols identified a more accurate configuration in two diagnostic settings. The paper does not establish an evidence-based cognitive model or show that such a model would enable the design of the most effective protocols generally. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. [1][1] Alufaisan, Y., Marusich, L. R., Bakdash, J. Z., Zhou, Y., & Kantarcioglu, M. (2021). Does explainable artificial intelligence improve human decisionmaking? In Proceedings of the AAAI Conference on Artificial Intelligence (pp. 6618-6626). volume 35. -
  2. [3][3] Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019). Beyond accuracy: The role of mental models in human-ai team performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (pp. 2-11). volume 7. -
  3. [8][8] Bertrand, A., Belloum, R., Eagan, J. R., & Maxwell, W. (2022). How cognitive biases affect xai-assisted decision-making: A systematic review. In Proc. of the AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society AIES. New York, NY: ACM. To appear. -
  4. [11][11] Cabitza, F., Campagner, A., & Simone, C. (2021). The need to move away from agential-ai: Empirical investigations, useful concepts and open issues. International Journal of Human-Computer Studies , 155 , 102696. -
  5. [12][12] Cabitza, F., Campagner, A., Zotti, F., Ravizza, A., & Sternini, F. (2020). All you need is higher accuracy? on the quest for minimum acceptable accuracy for medical artificial intelligence. (pp. 159-166). -
  6. [13][13] Cabitza, F., & Zeitoun, J.-D. (2019). The proof of the pudding: in praise of a culture of real-world validation for medical artificial intelligence. Annals of translational medicine , 7 . -
  7. [19][19] Dey, S., Karahalios, K., & Fu, W.-T. (2018). Getting there and beyond: Incidental learning of spatial knowledge with turn-by-turn directions and location updates in navigation interfaces. In Proceedings of the symposium on spatial user interaction (pp. 100-110). -
  8. [23][23] Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I., Muller, M., Riedl, M. O. et al. (2021). The who in explainable ai: how ai background shapes perceptions of ai explanations. arXiv preprint arXiv:2107.13509 , . - · arXiv
  9. [27][27] Friedman, C. P. (2009). A 'fundamental theorem' of biomedical informatics. Journal of the American Medical Informatics Association , 16 , 169-170. -
  10. [31][31] Gigerenzer, G. (2007). Gut feelings: The intelligence of the unconscious . Penguin. -
  11. [35][35] Hemmer, P., Schemmer, M., Vössing, M., & Kühl, N. (2021). Humanai complementarity in hybrid intelligence systems: A structured literature review. PACIS , (p. 78). -
  12. [36][36] Hoff, T. (2011). Deskilling and adaptation among primary care physicians using two work innovations. Health Care Management Review , 36 , 338348. -
  13. [37][37] Holzinger, A. (2021). The next frontier: AI we can really trust. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 427-440). Springer. -
  14. [40][40] Inkpen, K. (2020). Does my ai help or hurt? exploring human-ai complementarity. In Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization (pp. 2-2). -
  15. [41][41] Itoh, M., & Tanaka, K. (2000). Mathematical modeling of trust in automation: Trust, distrust, and mistrust. In Proceedings of the human factors and ergonomics society annual meeting (pp. 9-12). SAGE Publications Sage CA: Los Angeles, CA volume 44. -
  16. [45][45] Klein, G. (2022). Snapshots of the Mind . The MIT Press. -
  17. [47][47] Mann, H. B., & Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics , (pp. 50-60). -
  18. [48][48] Newell, B. R., & Shanks, D. R. (2014). Unconscious influences on decision making: A critical review. Behavioral and brain sciences , 37 , 1-19. -
  19. [49][49] Ooge, J., & Verbert, K. (2022). Explaining artificial intelligence with tailored interactive visualisations. In 27th International Conference on Intelligent User Interfaces (pp. 120-123). -
  20. [50][50] Paleja, R., Ghuy, M., Ranawaka Arachchige, N., Jensen, R., & Gombolay, M. (2021). The utility of explainable ai in ad hoc human-machine teaming. Advances in Neural Information Processing Systems , 34 . -
  21. [53][53] Rastogi, C., Zhang, Y., Wei, D., Varshney, K. R., Dhurandhar, A., & Tomsett, R. (2020). Deciding fast and slow: The role of cognitive biases in ai-assisted decision-making. arXiv preprint arXiv:2010.07938 , . - · arXiv
  22. [54][54] Ronzio, L., Campagner, A., Cabitza, F., & Gensini, G. F. (2021). Unity is intelligence: A collective intelligence experiment on ecg reading to improve diagnostic performance in cardiology. Journal of Intelligence , 9 , 17. -
  23. [56][56] Schemmer, M., Kühl, N., Benz, C., & Satzger, G. (2022). On the influence of explainable ai on automation bias. In Thirtieth European Conference on Information Systems (ECIS 2022) . Preprint at https://arxiv.org/abs/2204.08859. - · arXiv
  24. [58][58] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradientbased localization. In Proceedings of the IEEE international conference on computer vision (pp. 618-626). -
  25. [60][60] Siau, K., & Wang, W. (2018). Building trust in artificial intelligence, machine learning, and robotics. Cutter business technology journal , 31 , 47-53. -
  26. [67][67] Vodrahalli, K., Gerstenberg, T., & Zou, J. (2022). Uncalibrated models can improve human-ai collaboration. arXiv preprint arXiv:2202.05983 , . - · arXiv
  27. [68][68] Woolson, R. F. (2007). Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials , (pp. 1-3). -
  28. [71][71] Zhang, Q., Lee, M. L., & Carter, S. (2022). You complete me: Human-ai teams and complementary expertise. In CHI Conference on Human Factors in Computing Systems (pp. 1-28).

Cited by

Sorted by citations

Authors
Cabitza, F., Campagner, A., Ronzio, L., Cameli, M., Mandoli, G. E., Pastore, M. C., Sconfienza, L. M., Folgado, D., Barandas, M., & Gamboa, H.
Affiliations
  • IRCCS Ospedale Galeazzi - Sant'Ambrogio (IT)
  • IRCCS Ospedale San Raffaele (IT)
  • University of Siena (IT)
  • Azienda Ospedaliero Universitaria Maggiore della Carita (IT)
  • University of Milan (IT)
  • Fraunhofer Portugal Research (PT)
Corresponding author
Cabitza, Federico
ORCID
Venue
Artificial Intelligence in Medicine
Publisher
Elsevier BV
Published
2023-04
Volume
138
Pages
102506
ISSN
0933-3657 (ISSN-L)
DOI
10.1016/j.artmed.2023.102506
OpenAlex
W4319339070
Cited by
117
References
22
Citations by year
2026: 42 · 2025: 42 · 2024: 24 · 2023: 9
FWCI
4.34
Type
journal article
Language
en
Open access
Licence
Locations
  • Artificial Intelligence in Medicine (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • BOA (University of Milano-Bicocca) (submittedVersion, other-oa)
  • Use Siena air (University of Siena) (submittedVersion, cc-by)
Funders
Ministero della Salute
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
Keywords
human-AI collaboration, collaboration protocol, explainable AI, ECG interpretation, diagnostic accuracy, cognitive biases, radiologists
MeSH terms
Artificial Intelligence, Humans

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review