"Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making
Open accessThe DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index.
Abstract
Although rapid advances in machine learning have made it increasingly applicable to expert decision-making, the delivery of accurate algorithmic predictions alone is insufficient for effective human-AI collaboration. In this work, we investigate the key types of information medical experts desire when they are first introduced to a diagnostic AI assistant. In a qualitative lab study, we interviewed 21 pathologists before, during, and after being presented deep neural network (DNN) predictions for prostate cancer diagnosis, to learn the types of information that they desired about the AI assistant. Our findings reveal that, far beyond understanding the local, case-specific reasoning behind any model decision, clinicians desired upfront information about basic, global properties of the model, such as its known strengths and limitations, its subjective point-of-view, and its overall design objective--what it's designed to be optimized for. Participants compared these information needs to the collaborative mental models they develop of their medical colleagues when seeking a second opinion: the medical perspectives and standards that those colleagues embody, and the compatibility of those perspectives with their own diagnostic patterns. These findings broaden and enrich discussions surrounding AI transparency for collaborative decision-making, providing a richer understanding of what experts find important in their introduction to AI assistants before integrating them into routine practice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The training included learning optimal ways of using the AI in their settings and pointing out possible errors [18].”
From Explainable recommendation: when design meets trust calibration (2021) · cites this paper as [18]
In the citing paper
“This finding relates to earlier work in Human-AI trust calibration [18], which showed that training clinicians to use the AI-based decision-making tool reduced users’ errors and increased the Human-AI collaborative decision-making performance.”
From Explainable recommendation: when design meets trust calibration (2021) · cites this paper as [18]
In the citing paper
“there has been increasing recent research efforts [8, 15, 22, 41, 58, 65] that highlight the importance of involving stakeholders to understand their practices and needs [15, 41, 65] and socio-environmental factors [8] for the design and evaluation of a system.”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [15]
In the citing paper
“researchers have discussed the importance of an AI explanation to communicate an AI output to a user [5, 20, 29] and realize human-AI collaboration [48] for a decision-making task [15, 42].”
From Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making (2023) · cites this paper as [15]
In the citing paper
“Third, this allows uncertainty to be expressed seamlessly within the natural language interactions of LLM-infused applications, rather than on the side or in onboarding materials [21, 74, 79] that users might overlook.”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [21]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Braun, V., & Clarke, V. (2012) · APA handbook of research methods in psychology, Vol 2: Research designs: Quantitative, qualitative, neuropsychological, and biological. · doi:10.1037/13620-004 · Cited by 10,299
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“Recent work on fairness and accountability has proposed the use of short documents accompanying trained ML models to disclose the intended use of models, details of their performance evaluation procedures, and the potential biases that they may embody [25, 51].”
BackgroundGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
Shared mental models in expert team decision making.
Cannon‐Bowers, J. A., Salas, E., & Converse, S. A. (1993) · Cited by 1,545
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467
A Contemporary Prostate Cancer Grading System: A Validated Alternative to the Gleason Score
Epstein, J. I., Zelefsky, M. J., Sjoberg, D. D., Nelson, J. B., Egevad, L., Magi-Galluzzi, C., Vickers, A. J., Parwani, A. V., Reuter, V. E., Fine, S. W., et al. (2016) · European Urology · doi:10.1016/j.eururo.2015.06.046 · Cited by 1,374
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240
Self-citationSirinukunwattana, K., Raza, S. E. A., Tsang, Y.-W., Snead, D. R. J., Cree, I. A., & Rajpoot, N. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2525803 · Cited by 1,224
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084
Machine learning in medicine: Addressing ethical challenges
Vayena, E., Blasimme, A., & Cohen, I. G. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002689 · Cited by 912
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
Abdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743
Carruthers, P., & Smith, P. K. (1996) · Cited by 718
Giger (2018) · doi:10.1016/j.jacr.2017.12.028 · Cited by 634
Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., & Sandvig, C. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702556 · Cited by 627
Ström, P., Kartasalo, K., Olsson, H., Solorzano, L., Delahunt, B., Berney, D. M., Bostwick, D. G., Evans, A. J., Grignon, D. J., Humphrey, P. A., et al. (2020) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30738-7 · Cited by 620
The Appropriation of Interactive Technologies: Some Lessons from Placeless Documents
Dourish, P. (2003) · Computer Supported Cooperative Work (CSCW) · doi:10.1023/a:1026149119426 · Cited by 548
Nagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495
What This Computer Needs Is a Physician: Humanism and Artificial Intelligence
Verghese, A., Shah, N. H., & Harrington, R. A. (2018) · JAMA · doi:10.1001/jama.2017.19198 · Cited by 468
Computer Technology and Clinical Work: Still Waiting for Godot
Wears, R. L., & Berg, M. (2005) · JAMA · doi:10.1001/jama.293.10.1261 · Cited by 448
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441
Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405
Self-citationHAPPÉ, F. (1996) · Mind & Language · doi:10.1111/j.1468-0017.1996.tb00057.x · Cited by 392
Questions for artificial intelligence in health care
Maddox, T. M., Rumsfeld, J. S., & Payne, P. R. O. (2019) · JAMA · doi:10.1001/jama.2018.18932 · Cited by 385 · cited 1×
“For example, users may resist adopting a tool if they do not understand its capabilities, its intended use, or its utility over existing practices [48, 72, 74].”
BackgroundReasons For Physicians Not Adopting Clinical Decision Support Systems: Critical Analysis
Khairat, S., Marc, D., Crosby, W., & Al Sanousi, A. (2018) · JMIR Medical Informatics · doi:10.2196/medinform.8912 · Cited by 335
Compatible and Incompatible Relationships
(1985) · Springer eBooks · doi:10.1007/978-1-4612-5044-9 · Cited by 297
A finger on the pulse: temporal rhythms and information seeking in medical work
Reddy, M., & Dourish, P. (2002) · Proceedings of the 2002 ACM conference on Computer supported cooperative work · doi:10.1145/587078.587126 · Cited by 282
Yang, Q., Steinfeld, A., & Zimmerman, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300468 · Cited by 247
Value-Sensitive Algorithm Design: Method, Case Study, and Lessons
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274463 · Cited by 220
The effects of example-based explanations in a machine learning interface
Cai, C. J., Jongejan, J., & Holbrook, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302289 · Cited by 220
Self-citationUnderstanding together: sensemaking in collaborative information seeking
Paul, S. A., & Reddy, M. C. (2010) · Proceedings of the 2010 ACM conference on Computer supported cooperative work · doi:10.1145/1718918.1718976 · Cited by 207
Clinical Decision Support Systems
(2016) · Health Informatics · doi:10.1007/978-3-319-31913-1 · Cited by 196
Communicating Algorithmic Process in Online Behavioral Advertising
Eslami, M., Krishna Kumaran, S. R., Sandvig, C., & Karahalios, K. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174006 · Cited by 171
Temporality in Medical Work: Time also Matters
Reddy, M. C., Dourish, P., & Pratt, W. (2006) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-005-9010-z · Cited by 164
Collaboration and Trust in Healthcare Innovation: The eDiaMoND Case Study
JIROTKA, M., PROCTER, R., HARTSWOOD, M., SLACK, R., SIMPSON, A., COOPMANS, C., HINDS, C., & VOSS, A. (2005) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-005-9001-0 · Cited by 158
Similar image search for histopathology: SMILY
Hegde, N., Hipp, J. D., Liu, Y., Emmert-Buck, M., Reif, E., Smilkov, D., Terry, M., Cai, C. J., Amin, M. B., Mermel, C. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0131-z · Cited by 157
Self-citationTan, J., Ung, M. H., Cheng, C., & Greene, C. S. (2014) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789814644730_0014 · Cited by 152
Lee, M. K., Kim, J. T., & Lizarondo, L. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025884 · Cited by 129
MOSQUERA-LOPEZ, C. M., Agaian, S. С., Hoyos, A. V., & Thompson, I. M. (2014) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2014.2340401 · Cited by 118
Kaufman, Z., Shpitz, B., Shapiro, M., Rona, R., Lew, S., & Dinbar, A. (1994) · Journal of Surgical Oncology · doi:10.1002/jso.2930560413 · Cited by 117
Kohli, A., & Jha, S. (2018) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2017.12.029 · Cited by 114
Automatic Gleason grading of prostate cancer using quantitative phase imaging and machine learning
Nguyen, T. H., Sridharan, S., Macias, V., Kajdacsy‐Balla, A., Melamed, J., Do, M. N., & Popescu, G. (2017) · Journal of Biomedical Optics · doi:10.1117/1.jbo.22.3.036015 · Cited by 110
Bardram, J. E., & Hansen, T. R. (2010) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-010-9110-2 · Cited by 83
Yang, Q., Steinfeld, A. M., & Zimmerman, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09612 · Cited by 80
Medical Imaging 2014: Image Perception, Observer Performance, and Technology Assessment
Mello-Thoms, C. R., & Kupinski, M. A. (2014) · SPIE eBooks · Cited by 80
Usefulness of the Triple Test Score for Palpable Breast Masses
Morris, K., Pommier, R. F., Morris, A., Schmidt, W. A., Beagle, G., Alexander, P. W., Toth-Fejel, S., Schmidt, J., & Vetto, J. T. (2001) · Archives of Surgery · doi:10.1001/archsurg.136.9.1008 · Cited by 72
(2007) · Elsevier eBooks · doi:10.1016/b978-0-12-369377-8.x5000-4 · Cited by 72
Collocated Use of Imaging Systems in Coordinated Surgical Practice
Mentis, H. M. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134713 · Cited by 72
Achieving Diagnosis by Consensus
Kane, B. T., & Luz, S. F. (2009) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-009-9094-y · Cited by 60
Work Practices Surrounding PACS: The Politics of Space in Hospitals
Tellioğlu, H., & Wagner, I. (2001) · Computer Supported Cooperative Work (CSCW) · doi:10.1023/a:1011298824442 · Cited by 60
Artefactual Multiplicity: A Study of Emergency-Department Whiteboards
Bjørn, P., & Hertzum, M. (2010) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-010-9126-7 · Cited by 57
Zamora, A., de Bobadilla, F. F., Carrión, C., Vázquez, G., Paluzié, G., Elosúa, R., Vilaseca, M., Martín-Urda, A., Rivera, A., Plana, N., et al. (2013) · Atherosclerosis · doi:10.1016/j.atherosclerosis.2013.09.029 · Cited by 49
Hartswood, M., Procter, R., Rouncefield, M., Slack, R., Soutter, J., & Voß, A. (2003) · Springer eBooks · doi:10.1007/978-94-010-0068-0_20 · Cited by 48
Alberdi, E., Povyakalo, A. A., Strigini, L., Ayton, P., Hartswood, M., Procter, R., & Slack, R. (2005) · British Journal of Radiology · doi:10.1259/bjr/37646417 · Cited by 48
Veloso, S. G., de Lima, M. F. R., Salles, P. G. D. O., Berenstein, C. K., Scalon, J. D., & Bambirra, E. A. (2007) · International braz j urol · doi:10.1590/s1677-55382007000500005 · Cited by 47
Performance Management in Breast Screening: A Case Study of Professional Vision
Hartswood, M., Procter, R., Rouncefield, M., & Slack, R. (2002) · Cognition Technology & Work · doi:10.1007/s101110200008 · Cited by 46
Supporting coordination in surgical suites: physical aspects of common information spaces
Scupelli, P. G., Xiao, Y., Fussell, S. R., Kiesler, S. B., & Gross, M. D. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753593 · Cited by 44
Re-coordinating activities: an investigation of articulation work in patient transfers
Abraham, J., & Reddy, M. C. (2013) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2441776.2441787 · Cited by 42
Provider Response to Computer-Based Care Suggestions for Chronic Heart Failure
Keeffe, B. G., Subramanian, U., Tierney, W. M., Udris, E. M., Willems, J., McDonell, M. B., & Fihn, S. D. (2005) · Medical Care · doi:10.1097/01.mlr.0000160378.53326.f3 · Cited by 37
Moving beyond local practice: Reconfiguring the adoption of a breast cancer diagnostic technology
Maniatopoulos, G., Procter, R., Llewellyn, S., Harvey, G., & Boyd, A. (2015) · Social Science & Medicine · doi:10.1016/j.socscimed.2015.02.036 · Cited by 34
Pathologist-Level Grading of Prostate Biopsies with Artificial IntelligencePreprint
Ström, P., Kartasalo, K., Olsson, H., Solorzano, L., Delahunt, B., Berney, D. M., Bostwick, D. G., Evans, A. J., Grignon, D. J., Humphrey, P. A., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.01368 · Cited by 28
Persson, J., Wilderäng, U., Jiborn, T., Wiklund, P. N., Damber, J., Hugosson, J., Steineck, G., Haglind, E., & Bjartell, A. S. (2013) · Scandinavian Journal of Urology · doi:10.3109/21681805.2013.820788 · Cited by 27
Coordination Mechanisms for Self-Organized Work in an Emergency Communication Center
Zhang, Z., & Sarcevic, A. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274468 · Cited by 20
Working IT out in e-Science : experiences of requirements capture in a HealthGrid project
Hartswood, M., Jirotka, M. D., Procter, R., Slack, R., Voß, A., & Lloyd, S. (2005) · PubMed · doi:10.82444/warw.33682234 · Cited by 19
Information Work in Bone Marrow Transplant: Reducing Misalignment of Perspectives
Büyüktür, A. G., & Ackerman, M. S. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998361 · Cited by 19
Larsen, S. B., & Bardram, J. E. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357144 · Cited by 18
Understanding Design for Automated Image Analysis in Digital Pathology
Molin, J., Woźniak, P. W., Lundström, C. F., Treanor, D. E., & Fjeld, M. (2016) · Nordic Conference on Human-Computer Interaction · doi:10.1145/2971485.2971561 · Cited by 15
Magnetic Resonance Image Analysis for Brain CAD Systems with Machine Learning
Arimura, H., Tokunaga, C., Yamashita, Y., & Kuwazuru, J. (2012) · Advances in bioinformatics and biomedical engineering book series · doi:10.4018/978-1-4666-0059-1.ch013 · Cited by 14
Schalekamp, S., van Ginneken, B., Schaefer‐Prokop, C. M., & Karssemeijer, N. (2014) · Journal of medical imaging · doi:10.1117/1.jmi.1.1.015501 · Cited by 9
When do people rely on algorithms
Logg, J. M. (2016) · ProQuest Demo Repository · Cited by 8
Grid-based mammography training
Soutter, J., Campos, J., Hartswood, M., Jirotka, M. D., Procter, R., Slack, R., & Taylor, P. M. (2003) · Research Explorer (The University of Manchester) · doi:10.82444/warw.33682237 · Cited by 7
Astakhov, V. (2009) · Methods in molecular biology · doi:10.1007/978-1-59745-524-4 · Cited by 7
Drawing the line between perception and interpretation in computer-aided mammography
Hartswood, M., Procter, R., Williams, L. J., Prescott, R. J., & Dixon, P. (1997) · University of Warwick · doi:10.82444/warw.33682342 · Cited by 6
Prompting in mammography : computer-aided detection or computer-aided diagnosis?
Hartswood, M., Procter, R., & Williams, L. J. (1998) · University of Warwick · doi:10.82444/warw.33682324 · Cited by 5
Subjective responses to prompting in screening mammography
Hartswood, M., Procter, R., Williams, L. J., Prescott, R. J., & Dixon, P. (1997) · University of Warwick · doi:10.82444/warw.33682351 · Cited by 3
Reading the lesson: eliciting requirements for a mammography training application
Hartswood, M., Blot, L., Taylor, P., Anderson, S., Procter, R., Wilkinson, L., & Smart, L. (2009) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.813920 · Cited by 3
Research works printed but not linked: 7
- Wouter Bulten , Hans Pinckaers , Hester van Boven , Robert Vink , Thomas de Bel , Bram van Ginneken , Jeroen van der Laak , Christina Hulsbergen-van de Kaa, and Geert Litjens . 2019 . Automated Gleason Grading of Prostate Biopsies using Deep Learning . arXiv preprint arXiv:1907.07980 (2019). Wouter Bulten, Hans Pinckaers, Hester van Boven, Robert Vink, Thomas de Bel, Bram van Ginneken, Jeroen van der Laak, Christina Hulsbergen-van de Kaa, and Geert Litjens. 2019. Automated Gleason Grading of Prostate Biopsies using Deep Learning. arXiv preprint arXiv:1907.07980 (2019). · arXiv
- Dumitru Erhan , Yoshua Bengio , Aaron Courville , and Pascal Vincent . 2009 . Visualizing higher-layer features of a deep network . University of Montreal 1341 , 3 (2009), 1 . Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2009. Visualizing higher-layer features of a deep network. University of Montreal 1341, 3 (2009), 1.
- Timnit Gebru , Jamie Morgenstern , Briana Vecchione , Jennifer Wortman Vaughan , Hanna Wallach, Hal Daumeé III, and Kate Crawford. 2018 . Datasheets for datasets. arXiv preprint arXiv:1803.09010 (2018). Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumeé III, and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010 (2018). · arXiv
- Ken Hinckley ( Ed .). 2018 . ACM Trans. Comput.-Hum. Interact. 25 , 1 (2018). Ken Hinckley (Ed.). 2018. ACM Trans. Comput.-Hum. Interact. 25, 1 (2018).
- 10.5555/3305381.3305576 · doi:10.5555/3305381.3305576
- Forough Poursabzi-Sangdeh , Daniel G Goldstein , Jake M Hofman , Jennifer Wortman Vaughan, and Hanna Wallach . 2018 . Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (2018). Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (2018). · arXiv
- Ming Yin , Jennifer Wortman Vaughan, and Hanna Wallach . 2019 . Understanding the Effect of Accuracy on Trust in Machine Learning Models . (2019). Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. 2019. Understanding the Effect of Accuracy on Trust in Machine Learning Models. (2019).
Cited by
Human-computer collaboration for skin cancer recognition
Tschandl, P., Rinner, C., Apalla, Z., Argenziano, G., Codella, N., Halpern, A., Janda, M., Lallas, A., Longo, C., Malvehy, J., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0942-0 · Cited by 1,024
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
“Indeed, at first glance, it appears that researchers have already confirmed the utility of explanations on tasks ranging from medical diagnosis [10, 47] and data annotation [57] to deception detection [36].”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629
Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614
“In subsequent work, Cai et al. examined what information pathologists (in a lab setting) found to be important when being introduced to AI assistants, before integrating these assistants into routine prostate cancer screening practice [9].”
BackgroundSelf-citation
Metadata
- Authors
- Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M.
- Affiliations
- Google (United States) (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2019-11-07
- Volume
- 3
- Issue
- CSCW
- Pages
- 1-24
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W2970837303
- Cited by
- 594
- References
- 79
- Citations by year
- 2026: 84 · 2025: 158 · 2024: 97 · 2023: 75 · 2022: 80 · 2021: 70 · 2020: 28 · 2019: 2
- FWCI
- 20.77
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- human-AI collaboration, physicians, prostate cancer diagnosis, pathologists, AI transparency, model limitations, second opinion, deep neural network inference, clinical decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.