Lessons Learned from Designing an AI-Enabled Diagnosis Tool for Pathologists
Open accessAbstract
Despite the promises of data-driven artificial intelligence (AI), little is known about how we can bridge the gulf between traditional physician-driven diagnosis and a plausible future of medicine automated by AI. Specifically, how can we involve AI usefully in physicians' diagnosis workflow given that most AI is still nascent and error-prone (\eg in digital pathology)? To explore this question, we first propose a series of collaborative techniques to engage human pathologists with AI given AI's capabilities and limitations, based on which we prototype Impetus --- a tool where an AI takes various degrees of initiatives to provide various forms of assistance to a pathologist in detecting tumors from histological slides. We summarize observations and lessons learned from a study with eight pathologists and discuss recommendations for future work on human-centered medical AI systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There is still limited evidence-based knowledge of the context, design, and psychological mechanisms that would craft an optimal human-AI team.
“Notwithstanding a clear agenda on the issues that we should explore further, we still have limited evidence-based knowledge of the context, design, and psychological mechanisms that would craft an optimal human–ai team15–17.”
From Experimental evidence of effective human–AI collaboration in medical decision-making (2022) · cites this paper as [17]
The Introduction says little work has addressed how physicians can use imperfect AI in diagnosis, and §2.1 identifies a lack of research on how medical human–AI collaboration can provide new knowledge to AI. Those passages support a narrower research gap, but not the claim of limited evidence-based knowledge about context, design, and psychological mechanisms for optimal human–AI teams generally. Page numbers are unavailable in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Fiji: an open-source platform for biological-image analysis
Schindelin, J., Arganda-Carreras, I., Frise, E., Kaynig, V., Longair, M., Pietzsch, T., Preibisch, S., Rueden, C., Saalfeld, S., Schmid, B., et al. (2012) · Nature Methods · doi:10.1038/nmeth.2019 · Cited by 66,890 · see citing works
NIH Image to ImageJ: 25 years of image analysis
Schneider, C. A., Rasband, W. S., & Eliceiri, K. W. (2012) · Nature Methods · doi:10.1038/nmeth.2089 · Cited by 64,638 · see citing works
A Threshold Selection Method from Gray-Level Histograms
Otsu, N. (1979) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1979.4310076 · Cited by 43,748 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
(1996) · Cited by 20,179 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
The cancer genome atlas pan-cancer analysis project
The Cancer Genome Atlas Research Network, Weinstein, J. N., Collisson, E. A., Mills, G. B., Shaw, K. R. M., Ozenberger, B. A., Ellrott, K., Shmulevich, I., Sander, C., & Stuart, J. M. (2013) · Nature Genetics · doi:10.1038/ng.2764 · Cited by 9,092 · see citing works
Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2008) · 2008 Eighth IEEE International Conference on Data Mining · doi:10.1109/icdm.2008.17 · Cited by 6,601 · see citing works
ImageJ2: ImageJ for the next generation of scientific image data
Rueden, C., Schindelin, J., Hiner, M., DeZonia, B. E., Walter, A., Arena, E. T., & Eliceiri, K. W. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1934-z · Cited by 5,895 · see citing works
CellProfiler: image analysis software for identifying and quantifying cell phenotypes
Carpenter, A. E., Jones, T. R., Lamprecht, M. R., Clarke, C., Kang, I. H., Friman, O., Guertin, D. A., Chang, J. H., Lindquist, R. A., Moffat, J., et al. (2006) · Genome biology · doi:10.1186/gb-2006-7-10-r100 · Cited by 5,077 · see citing works
Artificial intelligence in healthcare: past, present and future
Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., & Wang, Y. (2017) · Stroke and Vascular Neurology · doi:10.1136/svn-2017-000101 · Cited by 4,837 · see citing works
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791 · see citing works
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293 · see citing works
Clinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799 · see citing works
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · see citing works
Collins, T. J. (2007) · BioTechniques · doi:10.2144/000112517 · Cited by 2,202 · see citing works
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · see citing works
Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks
Cireşan, D. C., Giusti, A., Gambardella, L. M., & Schmidhuber, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_51 · Cited by 1,536 · see citing works
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · see citing works
Attention-based Deep Multiple Instance LearningPreprint
Ilse, M., Tomczak, J. M., & Welling, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04712 · Cited by 1,235 · see citing works
Ilastik: Interactive learning and segmentation toolkit
Sommer, C., Straehle, C., Köthe, U., & Hamprecht, F. A. (2011) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2011.5872394 · Cited by 1,145 · see citing works
Classification of breast cancer histology images using Convolutional Neural Networks
Araújo, T., Aresta, G., Castro, E., Rouco, J., Aguiar, P., Eloy, C., Polónia, A., & Campilho, A. (2017) · PLoS ONE · doi:10.1371/journal.pone.0177544 · Cited by 1,017 · see citing works
Quantitative Analysis of Histological Staining and Fluorescence Using ImageJ
Jensen, E. C. (2013) · The Anatomical Record · doi:10.1002/ar.22641 · Cited by 946 · see citing works
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906 · see citing works
Explicit factor models for explainable recommendation based on phrase-level sentiment analysis
Zhang, Y., Lai, G., Zhang, M., Zhang, Y., Liu, Y., & Ma, S. (2014) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2600428.2609579 · Cited by 814 · see citing works
Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376301 · Cited by 622 · cited 2× · see citing works
“Due to the “uncertainty surrounding AI’s capabilities” and “complexity in AI’s output”, it is hard for humans to control and trust AI systems [65].”
BackgroundCai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · see citing works
Assessment of algorithms for mitosis detection in breast cancer histopathology images
Veta, M., van Diest, P. J., Willems, S. M., Wang, H., Madabhushi, A., Cruz-Roa, Á., Gonzalez, F. A., Larsen, A., Vestergaard, J. S., Dahl, A. B., et al. (2014) · Medical Image Analysis · doi:10.1016/j.media.2014.11.010 · Cited by 480 · see citing works
Automated Gleason grading of prostate cancer tissue microarrays via deep learning
Arvaniti, E., Fricker, K. S., Moret, M., Rupp, N. J., Hermanns, T., Fankhauser, C. D., Wey, N., Wild, P. J., Rüschoff, J. H., & Claassen, M. (2018) · Scientific Reports · doi:10.1038/s41598-018-30535-1 · Cited by 447 · see citing works
Classification of Breast Cancer Based on Histology Images Using Convolutional Neural Networks
Bardou, D., Zhang, K., & Ahmad, S. M. (2018) · IEEE Access · doi:10.1109/access.2018.2831280 · Cited by 417 · see citing works
Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405 · see citing works
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392 · see citing works
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · arXiv · doi:10.48550/arxiv.1901.07031 · Cited by 392
MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network
Zhang, Z., Xie, Y., Xing, F., McGough, M., & Yang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.378 · Cited by 354 · see citing works
Predicting breast tumor proliferation from whole-slide images: The TUPAC16 challenge
Veta, M., Heng, Y. J., Stathonikos, N., Bejnordi, B. E., de Beça, F. F., Wollmann, T., Rohr, K., Shah, M., Wang, D., Rousson, M., et al. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.02.012 · Cited by 309 · see citing works
Mitosis detection in breast cancer histological images An ICPR 2012 contest
Roux, L., Racoceanu, D., Loménie, N., Kulikova, M. S., Irshad, H., Jacques, K., Capron, F., Genestie, C., Naour, G. L., & Gurcan Metin, N. (2013) · Journal of Pathology Informatics · doi:10.4103/2153-3539.112693 · Cited by 303 · see citing works
Ertosun, M. G., & Rubin, D. L. (2015) · PubMed · Cited by 255 · see citing works
Yang, Q., Steinfeld, A., & Zimmerman, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300468 · Cited by 247 · see citing works
Guan, Q., Huang, Y., Zhong, Z., Zheng, Z., Zheng, L., & Yang, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09927 · Cited by 235 · see citing works
Joint Prostate Cancer Detection and Gleason Score Prediction in mp-MRI via FocalNet
Cao, R., Bajgiran, A. M., Mirak, S. A., Shakeri, S., Zhong, X., Enzmann, D. R., Raman, S. S., & Sung, K. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2901928 · Cited by 219 · see citing works
Collaborative analysis of multi-gigapixel imaging data using Cytomine
Marée, R., Rollus, L., Stévens, B., Hoyoux, R., Louppe, G., Vandaele, R., Begon, J.-M., Kainz, P., Geurts, P., & Wehenkel, L. A. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw013 · Cited by 196 · see citing works
Chau, D. H., Kittur, A., Hong, J., & Faloutsos, C. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978967 · Cited by 190 · see citing works
Drew, T. W., Evans, K. K., Võ, M. L. ‐., Jacobson, F. L., & Wolfe, J. M. (2012) · Radiographics · doi:10.1148/rg.331125023 · Cited by 182 · see citing works
Gutman, D. A., Khalilia, M. A., Lee, S., Nalisnik, M., Mullen, Z., Beezley, J., Chittajallu, D. R., Manthey, D., & Cooper, L. (2017) · Cancer Research · doi:10.1158/0008-5472.can-17-0629 · Cited by 180 · see citing works
The Demise of the “Greek Oracle” Model for Medical Diagnostic Systems
Miller, R. A., & Masarie, F. E. (1990) · Methods of Information in Medicine · doi:10.1055/s-0038-1634767 · Cited by 170 · see citing works
The adolescence of AI in Medicine: Will the field come of age in the '90s?
Shortliffe, E. H. (1993) · Artificial Intelligence in Medicine · doi:10.1016/0933-3657(93)90011-q · Cited by 153 · see citing works
Similar image search for histopathology: SMILY
Hegde, N., Hipp, J. D., Liu, Y., Emmert-Buck, M., Reif, E., Smilkov, D., Terry, M., Cai, C. J., Amin, M. B., Mermel, C. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0131-z · Cited by 153 · see citing works
Diagnosis of Ovarian Carcinoma Cell Type is Highly Reproducible
Köbel, M., Kalloger, S. E., Baker, P. M., Ewanowich, C. A., Arseneau, J., Zherebitskiy, V. A., Abdulkarim, S., Leung, S. C. Y., Duggan, M. A., Fontaine, D., et al. (2010) · The American Journal of Surgical Pathology · doi:10.1097/pas.0b013e3181e1a3bb · Cited by 148 · see citing works
Clinical Judgment and Computers
Blois, M. S. (1980) · New England Journal of Medicine · doi:10.1056/nejm198007243030405 · Cited by 143 · see citing works
MIForests: Multiple-Instance Learning with Randomized Trees
Leistner, C., Saffari, A., & Bischof, H. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15567-3_3 · Cited by 141 · see citing works
Xie, Y., Chen, M., Kao, D., Gao, G., & Chen, X. '. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376807 · Cited by 141 · see citing works
Self-citationMachine Teaching: A New Paradigm for Building Machine Learning SystemsPreprint
Simard, P. Y., Amershi, S., Chickering, D. M., Pelton, A. E., Ghorashi, S., Meek, C., Ramos, G., Suh, J., Verwey, J., Wang, M., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06742 · Cited by 123 · see citing works
Towards the Augmented Pathologist: Challenges of Explainable-AI in Digital PathologyPreprint
Holzinger, A., Malle, B., Kieseberg, P., Roth, P., Müller, H., Reihs, R., & Zatloukal, K. (2017) · arXiv.org · doi:10.48550/arxiv.1712.06657 · Cited by 113 · see citing works
Nguyen, A. T., Kharosekar, A., Krishnan, S., Krishnan, S., Elizabeth, T., Wallace, B., & Lease, M. (2018) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3242587.3242666 · Cited by 109 · see citing works
Yang, Q., Zimmerman, J., Steinfeld, A., Carey, L., & Antaki, J. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858373 · Cited by 105 · see citing works
Xu, Y. L., Zhu, J.-Y., Chang, E., & Tu, Z. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247772 · Cited by 100 · see citing works
Yang, Q., Steinfeld, A. M., & Zimmerman, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09612 · Cited by 80 · see citing works
Interactive phenotyping of large-scale histology imaging data with HistomicsML
Nalisnik, M., Amgad, M. T., Lee, S., Halani, S. H., Vega, J. E. V., Brat, D. J., Gutman, D. A., & Cooper, L. (2017) · Scientific Reports · doi:10.1038/s41598-017-15092-3 · Cited by 67 · see citing works
MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis NetworkPreprint
Zhang, Z., Xie, Y., Xing, F., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02485 · Cited by 64 · see citing works
Martel, A. L., Hosseinzadeh, D., Senaras, C., Zhou, Y., Yazdanpanah, A., Shojaii, R., Patterson, E. S., Madabhushi, A., & Gurcan, M. N. (2017) · Cancer Research · doi:10.1158/0008-5472.can-17-0323 · Cited by 63 · see citing works
Saltz, J., Sharma, A., Iyer, G., Bremer, E., Wang, F., Jasniewski, A., DiPrima, T., Almeida, J. S., Gao, Y., Zhao, T., et al. (2017) · Cancer Research · doi:10.1158/0008-5472.can-17-0316 · Cited by 60 · see citing works
Automated Gleason grading of prostate cancer tissue microarrays via deep learningPreprint
Arvaniti, E., Fricker, K. S., Moret, M., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/280024 · Cited by 60 · see citing works
Chen, X. '., Tao, Y., Wang, G., Kang, R., Grossman, T., Coros, S., & Hudson, S. E. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174070 · Cited by 59 · see citing works
Self-citationSentinel Lymph Node in Breast Cancer: Review Article from a Pathologist’s Point of View
Apple, S. K. (2016) · Journal of Pathology and Translational Medicine · doi:10.4132/jptm.2015.11.23 · Cited by 51 · see citing works
Outlining the Design Space of Explainable Intelligent Systems for Medical DiagnosisPreprint
Xie, Y., Gao, G., & Chen, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06019 · Cited by 47 · see citing works
Self-citationOverview based example selection in end user interactive concept learning
Amershi, S., Fogarty, J. A., Kapoor, A., & Tan, D. S. (2009) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1622176.1622222 · Cited by 41 · see citing works
The Exploratory Labeling Assistant
Felix, C., Dasgupta, A., & Bertini, E. (2018) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3242587.3242596 · Cited by 30 · see citing works
Expert Discussions Improve Comprehension of Difficult Cases in Medical Image Assessment
Schaekermann, M., Cai, C. J., Huang, A. E., & Sayres, R. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376290 · Cited by 28 · see citing works
Scalable Histopathological Image Analysis via Active Learning
Zhu, Y., Zhang, S., Liu, W., & Metaxas, D. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10443-0_47 · Cited by 21 · see citing works
Diagnosis of Ovarian Carcinoma Cell Type is Highly Reproducible: A Transcanadian Study
Post, M. D. (2011) · Yearbook of Pathology and Laboratory Medicine · doi:10.1016/j.ypat.2010.11.027 · Cited by 20 · see citing works
Licklider, J. C. R. (2021) · The MIT Press eBooks · doi:10.7551/mitpress/12274.003.0022 · Cited by 18 · see citing works
Believe it or not Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking
An, T., Kharosekar, A., Krishnan, S., Krishnan, S., Elizabeth, T., Byron, C., & Matthew, L. (2018) · ACM Proceedings · Cited by 17 · see citing works
The label complexity of mixed-initiative classifier training
Suh, J., Zhu, X., & Amershi, S. (2016) · International Conference on Machine Learning · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Leo Breiman. 2001. Random forests. Machine learning 45, 1 (2001), 5-32.
- Duen Horng Chau, Aniket Kittur, Jason I Hong, and Christos Faloutsos. 2011. Apolo: making sense of large network data by combining rich user interaction and machine learning. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 167-176.
- Geert Litjens, Peter Bandi, Babak Ehteshami Bejnordi, Oscar Geessink, Maschenka Balkenhol, Peter Bult, Altuna Halilovic, Meyke Hermsen, Rob van de Loo, Rob Vogels, et al. 2018. 1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset. GigaScience 7, 6 (2018), giy065.
Cited by
Sorted by citations
AI-assisted CT imaging analysis for COVID-19 screening: Building and deploying a medical AI system
Wang, B., Jin, S., Yan, Q., Xu, H., Luo, C. W., Wei, L., Zhao, W. Z., Hou, X., Ma, W., Xu, Z., et al. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106897 · Cited by 539 · see citing works
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice
Delgado, F., Yang, S., Madaio, M., & Yang, Q. (2023) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3617694.3623261 · Cited by 256 · see citing works
Experimental evidence of effective human–AI collaboration in medical decision-making
Reverberi, C., Rigon, T., Solari, A., Hassan, C., Cherubini, P., GI Genius CADx Study Group, Antonelli, G., Awadie, H., Bernhofer, S., Carballal, S., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-18751-2 · Cited by 235 · see citing works
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228 · see citing works
Understanding Human-Centred AI: a review of its defining elements and a research agenda
Schmager, S., Pappas, I. O., & Vassilakopoulou, P. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2448719 · Cited by 191 · see citing works
Computational pathology: A survey review and the way forward
Hosseini, M. S., Bejnordi, B. E., Trinh, V. Q., Chan, L., Hasan, D., Li, X., Yang, S. C., Kim, T., Zhang, H., Wu, T., et al. (2024) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100357 · Cited by 124 · see citing works
Zając, H. D., Li, D., Dai, X., Carlsen, J. F., Kensing, F., & Andersen, T. O. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3582430 · Cited by 76 · see citing works
Verma, H., Mlynář, J., Schaer, R., Reichenbach, J., Jreige, M., Prior, J. O., Évéquoz, F., & Depeursinge, A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581506 · Cited by 63 · see citing works
Gu, H., Liang, Y., Xu, Y., Williams, C. K., Magaki, S., Khanlou, N., Vinters, H. V., Chen, Z., Ni, S., Yang, C., et al. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3577011 · Cited by 56 · see citing works
Self-citationPublicly available datasets of breast histopathology H&E whole-slide images: A scoping review
Tafavvoghi, M., Bongo, L. A., Shvetsov, N., Busund, L., & Møllersen, K. (2024) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2024.100363 · Cited by 53 · see citing works
Unpacking Human-AI Interaction in Safety-Critical Industries: A Systematic Literature Review
Bach, T. A., Kristiansen, J. K., Babić, A., & Jacovi, A. (2024) · IEEE Access · doi:10.1109/access.2024.3437190 · Cited by 43 · see citing works
(2022) · doi:10.1145/3491102.3501868 · Cited by 36 · see citing works
Gu, H., Yang, C., Haeri, M., Wang, J., Tang, S., Yan, W., He, S., Williams, C. K., Magaki, S., & Chen, X. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580694 · Cited by 36 · see citing works
Self-citationConfigurations of human-centered AI at work: seven actor-structure engagements in organizations
Wilkens, U., Lupp, D., & Langholf, V. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1272159 · Cited by 32 · see citing works
Göndöcs, D., Horváth, S., & Dörfler, V. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103622 · Cited by 32 · see citing works
Marvista: Exploring the Design of a Human-AI Collaborative News Reading Tool
Chen, X., Wu, C.-S., Murakhovs’ka, L., Laban, P., Niu, T., Liu, W., & Xiong, C. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3609331 · Cited by 29 · see citing works
Self-citationRezaev, A. V., & Трегубова, Н. Д. (2023) · Current Sociology · doi:10.1177/00113921231211580 · Cited by 25 · see citing works
Lindvall, M., Lundström, C. F., & Löwgren, J. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450681 · Cited by 25 · see citing works
Current AI Applications and Challenges in Oral Pathology
Xu, Z., Lin, A., & Han, X. (2025) · Oral · doi:10.3390/oral5010002 · Cited by 22 · see citing works
GANravel: User-Driven Direction Disentanglement in Generative Adversarial Networks
Evirgen, N., & Chen, X. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581226 · Cited by 17 · see citing works
Self-citationContractMind: Trust-calibration interaction design for AI contract review tools
Zeng, J., Chen, K., Wang, R., Li, Y., Fan, M., Wu, K., Qi, X., & Wang, L. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103411 · Cited by 16 · see citing works
The AI-DEC: A Card-based Design Method for User-centered AI Explanations
Lee, C. P., Lee, M. K., & Mutlu, B. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661576 · Cited by 15 · see citing works
Jo, E., Ryu, M., Kenderova, G., So, S., Shapiro, B. B., Papoutsaki, A., & Epstein, D. A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502206 · Cited by 15 · see citing works
"It depends": Configuring AI to Improve Clinical Usefulness Across Contexts
Zając, H. D., Ribeiro, J., Ingala, S., Gentile, S., Wanjohi, R., Gitau, S. N., Carlsen, J. F., Nielsen, M. B., & Andersen, T. O. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660707 · Cited by 14 · see citing works
Xu, Y., Shao, Y., Dong, J., Shi, S., Jiang, C., & Li, Q. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713772 · Cited by 13 · see citing works
The Future of Work is Blended, Not Hybrid
Constantinides, M., Verma, H., Sadeghian, S., & Ali, A. E. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729202 · Cited by 13 · see citing works
Zając, H. D., Andersen, T. O., Kwasa, E., Wanjohi, R., Onyinkwa, M. K., Mwaniki, E. K., Gitau, S. N., Yaseen, S. S., Carlsen, J. F., Fraccaro, M., et al. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3715115 · Cited by 12 · see citing works
Perceptions of the Helpfulness of Unexpected Agent Assistance
Candon, K., Hsu, Z., Kim, Y., Chen, J., Tsoi, N., & Vázquez, M. (2022) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3527188.3561915 · Cited by 12 · see citing works
Ahmed, A. A., Brychcy, A., Abouzid, M., Witt, M., & Kaczmarek, E. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13060962 · Cited by 11 · see citing works
Wang, X., Tian, Y., Huang, K., & Liang, B. (2024) · Computers & Security · doi:10.1016/j.cose.2024.104113 · Cited by 10 · see citing works
Arakawa, R., Maeda, K., & Yakura, H. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3709012 · Cited by 10 · see citing works
Majority voting of doctors improves appropriateness of AI reliance in pathology
Gu, H., Yang, C., Magaki, S., Zarrin‐Khameh, N., Lakis, N. S., Cobos, I., Khanlou, N., Zhang, X. R., Assi, J. S., Byers, J. T., et al. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2024.103315 · Cited by 6 · see citing works
Self-citationHuman-Centered AI for Medical Imaging
Liang, Y., He, L., & Chen, X. (2021) · Human-computer interaction series · doi:10.1007/978-3-030-82681-9_16 · Cited by 6 · see citing works
Self-citationKanak: Automating the Generation of Accessible STEM Materials for Blind and Low-vision Students
Palani, H. P., Adnin, R., & Nagar, S. (2025) · Australian Conference on Human-Computer Interaction (OzCHI) · doi:10.1145/3764687.3764694 · Cited by 5 · see citing works
From Text to Pixels: Enhancing User Understanding through Text-to-Image Model Explanations
Evirgen, N., Wang, R., & Chen, X. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645173 · Cited by 5 · see citing works
Self-citationTeng, C., Lee, L. H., Lander, J., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2022) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3517031.3529637 · Cited by 5 · see citing works
D’Abbronzo, G., D’Antonio, A., Chiara, A. D., Panico, L., Sparano, L., Diluvio, A., Sica, A., Svanera, G., Chiara, G. D., Fuggi, M., et al. (2025) · Archiv für Pathologische Anatomie und Physiologie und für Klinische Medicin · doi:10.1007/s00428-025-04122-5 · Cited by 4 · see citing works
Singh, P., Budhiraja, R., Jalote, P., Kumar, M. J., & Singh, P. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711093 · Cited by 4 · see citing works
AI-Augmented Art Psychotherapy through a Hierarchical Co-Attention Mechanism
Jin, S., Choi, H.-Y., & Han, K. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557542 · Cited by 4 · see citing works
A Human-AI Collaborative System to Support Mitosis Assessment in Pathology
Yang, C., Haeri, M., Magaki, S., Zarrin‐Khameh, N., Gu, H., & Chen, X. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640544.3645233 · Cited by 3 · see citing works
Self-citationArakawa, R., Yakura, H., Akuzawa, K., & Kubo, S. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3706686 · Cited by 2 · see citing works
Nallamothu, T. K. (2025) · INTERNATIONAL JOURNAL OF COMPUTER ENGINEERING & TECHNOLOGY · doi:10.34218/ijcet_16_03_038 · Cited by 2 · see citing works
ClinSegNet: Towards Reliable and Enhanced Histopathology Screening
Yu, B., Markham, H., Moutasim, K. A., Foria, V., & Liu, H. (2025) · Bioengineering · doi:10.3390/bioengineering12111156 · Cited by 2 · see citing works
Human-AI Collaboration in Generating Graphical Museum Descriptions
Kim, J., Han, M., Kim, S., & Hong, J.-H. (2025) · Journal on Computing and Cultural Heritage · doi:10.1145/3786327 · Cited by 2 · see citing works
Xu, Y., Zhu, Y., Wang, H., Wu, Y. G., Ouyang, Y., Li, H., Zhou, W., Liu, X., Jiang, C., & Li, Q. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790835 · Cited by 2 · see citing works
Human-Centered Artificial Intelligence, a reviewPreprint
Domfeh, E. A. (2022) · doi:10.36227/techrxiv.19174772 · Cited by 2 · see citing works
ReviseMate: Exploring Contextual Support for Digesting STEM Paper Reviews
Xu, Y., Zhang, S., Fan, Y., Shi, S., Peng, Z., & Li, Q. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757502 · Cited by 1 · see citing works
Designing a human-centered AI solution with radiologists to improve prostate cancer diagnosis
Saßmannshausen, S. M., Ontika, N. N., Tran, K., Ukleja, K., de Carvalho, A. F. P., & Rohde, M. (2025) · i-com · doi:10.1515/icom-2025-0043 · Cited by 1 · see citing works
Duan, W., Zhou, S., Scalia, M. J., Weng, N., Yin, X., Hao, R., Freeman, G., Funke, G. J., Tolston, M. T., Gorman, J., et al. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3788055 · Cited by 1 · see citing works
Human-Centered Artificial Intelligence, a reviewPreprint
Domfeh, E. A. (2022) · doi:10.36227/techrxiv.19174772.v1 · Cited by 1 · see citing works
Krishnan, G., & Hofmann, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.01131
Mudhol, B., Krishna, M., & Kademane, A. (2023) · Salud Ciencia y Tecnología · doi:10.56294/saludcyt2023463
Human-Centered AI at Work: Common Ground in Theories and Methods
(2024) · Frontiers research topics · doi:10.3389/978-2-8325-4840-0
Liu, X., Yu, X., & Liu, J. (2024) · doi:10.2196/preprints.66172
Liu, X., Yu, X., Oda, T., Liu, J., & Guo, Y. (2025) · Research Square · doi:10.21203/rs.3.rs-6708114/v1
Prototyping Framework for Interactive Annotation Interfaces in Medical Domain
Miková, P., & Laco, M. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-07989-3_11
Charting a Path Toward a Responsible and Human-Centered AI Future
Shafik, W. (2026) · Springer eBooks · doi:10.1007/978-3-032-09130-7_10
Otify, A., & Nabney, I. T. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1878569
- Authors
- Gu, H., Huang, J., Hung, L., & Chen, X. '.
- Affiliations
- University of California, Los Angeles (US)
- University of California, Los Angeles, Los Angeles, CA, USA
- Carnegie Mellon University (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2021-04-13
- Volume
- 5
- Issue
- CSCW1
- Pages
- 1-25
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3036375901
- Cited by
- 56
- References
- 73
- Citations by year
- 2026: 4 · 2025: 16 · 2024: 12 · 2023: 11 · 2022: 9 · 2021: 3 · 2020: 1
- FWCI
- 2.59
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (IIS-1850183)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- AI-assisted diagnosis, digital pathology, human-AI collaboration, tumor detection, histological slides, human-centered AI, impetus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
