A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Open accessThe DOI matches its Crossref record; 107 of the 107 references it cites were found in a registry or an index.
Abstract
Recent advances in the performance of machine learning algorithms have led to the adoption of AI models in decision making contexts across various domains such as healthcare, finance, and education. Different research communities have attempted to optimize and evaluate human-AI team performance through empirical studies by increasing transparency of AI systems, or providing explanations to aid human understanding of such systems. However, the variety in decision making tasks considered and their operationalization in prior empirical work, has led to an opacity around how findings from one task or domain carry forward to another. The lack of a standardized means of considering task attributes prevents straightforward comparisons across decision tasks, thereby limiting the generalizability of findings. We argue that the lens of ‘task complexity’ can be used to tackle this problem of under-specification and facilitate comparison across empirical research in this area. To retrospectively explore how different HCI communities have considered the influence of task complexity in designing experiments in the realm of human-AI decision making, we survey literature and provide an overview of empirical studies on this topic. We found a serious dearth in the consideration of task complexity across various studies in this realm of research. Inspired by Robert Wood’s seminal work on the construct, we operationalized task complexity with respect to three dimensions (component, coordinative, and dynamic) and quantified the complexity of decision tasks in existing work accordingly. We then summarized current trends and proposed research directions for the future. Our study highlights the need to account for task complexity as an important design choice. This is a first step to help the scientific community in drawing meaningful comparisons across empirical studies in human-AI decision making and to provide opportunities to generalize findings across diverse domains and experimental settings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1956) · Psychological Review · doi:10.1037/h0043158 · Cited by 17,812
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 2×
“Doshi-Velez and Kim [30] have argued that to create and advance a ‘rigorous science’ in the realm of human-AI decision making, there is room for empirical work that considers functionally-grounded explanations with proxy users and tasks, human-grounded evaluation with real users and simple tasks, and application-groun…”
BackgroundHuman trust in artificial intelligence: Review of empirical research
Glikson, E., & Woolley, A. W. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0057 · Cited by 2,921 · cited 1×
“User factors like expertise [28], machine learning literacy [24], and task characteristics like task subjectivity and proximity [15], and task types [34, 49, 55, 61, 74] have also been broadly investigated.”
BackgroundTask Complexity: A Review and Analysis
Campbell, D. J. (1988) · The Academy of Management Review · doi:10.5465/amr.1988.4306775 · Cited by 1,634 · cited 5×
“Among all works, the theories introduced by Campbell [20], Wood [100] gained popularity with more than 2000 citations and became the basis of other frameworks.”
BackgroundInfluentialTask complexity: Definition of the construct
Wood, R. E. (1986) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(86)90044-0 · Cited by 1,393 · cited 9×
“posed in seminal work by Wood [100] in the realm of human-AI decision making.”
BackgroundMethodInfluentialLee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338 · cited 2×
“Even though some studies incorporate tasks with different characteristics [53, 55, 61], task attributes haven’t been identified systematically in the literature, so their impact on human-AI complementary performance has not been fully investigated.”
BackgroundProceedings of the 2021 CHI Conference on Human Factors in Computing Systems
(2021) · doi:10.1145/3411764 · Cited by 1,240
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
The accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071 · cited 1×
“For example, recidivism prediction algorithms have been used to help judges determine whether defendants are likely to re-offend [26, 31, 65, 69], medical diseases are being diagnosed with AI systems [19, 57, 62], and loan risk prediction algorithms are employed to approve or reject loan applications [11, 25, 38, 95].”
MethodZhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · cited 1×
“By increasing task complexity, such approaches affect human cognitive processes by: (I) asking humans to make decisions before seeing model predictions [97, 107], (II) varying AI systems response time [16, 80], and (III) providing feedback to humans [8, 40, 101, 103].”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 3×
“In our example, race, gender, age, prior count, charge name, charge degree, days in custody, and model predictions form component complexity as 8.”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754 · cited 3×
“In our example, race, gender, age, prior count, charge name, charge degree, days in custody, and model predictions form component complexity as 8.”
BackgroundManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 5×
“House Price Prediction (9,15,0)[1],(11,24,12)[83], (12,40,10)[24]”
BackgroundInfluentialManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711 · cited 5×
“House Price Prediction (9,15,0)[1],(11,24,12)[83], (12,40,10)[24]”
BackgroundInfluential'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · cited 2×
“For example, recidivism prediction algorithms have been used to help judges determine whether defendants are likely to re-offend [26, 31, 65, 69], medical diseases are being diagnosed with AI systems [19, 57, 62], and loan risk prediction algorithms are employed to approve or reject loan applications [11, 25, 38, 95].”
BackgroundMethodInterpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687 · cited 1×
“For example, recidivism prediction algorithms have been used to help judges determine whether defendants are likely to re-offend [26, 31, 65, 69], medical diseases are being diagnosed with AI systems [19, 57, 62], and loan risk prediction algorithms are employed to approve or reject loan applications [11, 25, 38, 95].”
MethodRapp, A., Curti, L., & Boldi, A. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102630 · Cited by 643 · cited 1×
“We followed a semi-systematic literature review, widely adopted in prior studies [77, 84], including the four stages summarized in Figure 1 below.”
MethodBeede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614 · cited 2×
“tent moderation system [94], movie recommendation [104, 104], and medical diagnosis [9].”
BackgroundNauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590 · cited 1×
“User factors like expertise [28], machine learning literacy [24], and task characteristics like task subjectivity and proximity [15], and task types [34, 49, 55, 61, 74] have also been broadly investigated.”
BackgroundDeep learning for financial applications : A survey
Ozbayoglu, A. M., Gudelek, M. U., & Sezer, O. B. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106384 · Cited by 545
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 2×
“Explanations [65, 97], stated performance [7, 102, 106], risk perception [39], and uncertainty [92] have been studied extensively”
BackgroundDeep Learning and Medical Diagnosis: A Review of Literature
Bakator, M., & Radosav, D. (2018) · Multimodal Technologies and Interaction · doi:10.3390/mti2030047 · Cited by 510 · cited 1×
“nosis [6], a growing number of researchers and practitioners have begun to propose such AI systems in augmenting human decision making [54].”
BackgroundBeyond Accuracy: The Role of Mental Models in Human-AI Team Performance
Bansal, G., Nushi, B., Kamar, E., Lasecki, W. S., Weld, D. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5285 · Cited by 466 · cited 6×
“Although several studies have examined the role of human factors in shaping interactions with AI systems, there is a limited understanding of task characteristics in the human-AI decision making context [2, 7, 83, 93].”
BackgroundInfluentialFrontal theta activity reflects distinct aspects of mental fatigue
Wascher, E., Rasch, B., Sänger, J., Hoffmann, S., Schneider, D., Rinkenauer, G., Heuer, H., & Gutberlet, I. (2014) · Biological Psychology · doi:10.1016/j.biopsycho.2013.11.010 · Cited by 454 · cited 1×
“This is the result of prolonged periods of demanding cognitive activity [50] and has been shown to negatively affect performance [73, 99].”
BackgroundTask complexity: A review and conceptualization framework
Liu, P., & Li, Z. (2012) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2012.09.001 · Cited by 438 · cited 3×
“In the late-1980s, some frameworks were proposed to define and analyze task complexity; they were adapted in many domains such as psychology, management, information systems, and etc. [20, 48, 66, 100].”
BackgroundHuman-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405 · cited 2×
“Incorporating various features can ensure that human decision-makers access salient features required to make better decisions, especially in high-stake domains [13, 19, 40, 60].”
BackgroundMethodThe Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 2×
“For example, recidivism prediction algorithms have been used to help judges determine whether defendants are likely to re-offend [26, 31, 65, 69], medical diseases are being diagnosed with AI systems [19, 57, 62], and loan risk prediction algorithms are employed to approve or reject loan applications [11, 25, 38, 95].”
BackgroundMethodLai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377
Wang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320 · cited 5×
“Medical Diagnosis (7,4,2)[79],(6,1,0)[9], (6,240,0)[32] Nutrition Prediction (5,26,156)[16],(3,24,0)[15], (4,48,24)[33] Recidivism Prediction (10,64,32)[97],(11,200,50)[40], (8,12,6)[29],(132,10,0)[95], (11,50,0)[69] Monitoring and Administration (15,130,390)[81] Job Application Approval (5,24,0)[82]”
BackgroundInfluentialIslam, M. R., Ahmed, M. U., Barua, S., & Begum, S. (2022) · Applied Sciences · doi:10.3390/app12031353 · Cited by 311 · cited 1×
“User factors like expertise [28], machine learning literacy [24], and task characteristics like task subjectivity and proximity [15], and task types [34, 49, 55, 61, 74] have also been broadly investigated.”
BackgroundProxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307 · cited 7×
“User factors like expertise [28], machine learning literacy [24], and task characteristics like task subjectivity and proximity [15], and task types [34, 49, 55, 61, 74] have also been broadly investigated.”
BackgroundInfluentialExplaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · cited 1×
“Additionally, to fill the knowledge gap between domain experts and laypeople or improve AI literacy, empirical studies engage with more explanations [22, 62, 87, 105].”
BackgroundExplaining models: an empirical study of how explanations impact fairness judgment
Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K. E., & Dugan, C. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302310 · Cited by 286
Toward understanding the role of tasks in behavioral research
Hackman, J. R. (1969) · Acta Psychologica · doi:10.1016/0001-6918(69)90073-0 · Cited by 282 · cited 1×
“The complexity of tasks is an important dimension differentiating one task from the other, playing a significant role in determining the performance of a human-AI team [2, 43, 66].”
BackgroundSelf-Care Technologies in HCI: Trends, Tensions, and Opportunities
Nunes, F., Verdezoto, N., Fitzpatrick, G., Kyng, M., Grönvall, E., & Storni, C. (2015) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2803173 · Cited by 258 · cited 1×
“We followed a semi-systematic literature review, widely adopted in prior studies [77, 84], including the four stages summarized in Figure 1 below.”
MethodThe effects of example-based explanations in a machine learning interface
Cai, C. J., Jongejan, J., & Holbrook, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302289 · Cited by 220 · cited 4×
“Sketch Recognition (4,6,0)[18],(6,84,42)[105] Movie Recommendation (2,4,0)[52],(28,1,0)[104], (4,15,0)[63], (10,4,0)[10] Place Recommendation (12,2,0)[96], (51,12,0)[46] Food Recommendation (2,51,0)[37], (9,3,0)[72] Image Classification (3,12,0)[3],(5,216,0)[101], (4,90,0)[47], (3,40,0)[75] Sentiment Analysis (4,3,0)[…”
BackgroundInfluentialHow do visual explanations foster end users' appropriate trust in machine learning?
Yang, F., Huang, Z., Scholtz, J., & Arendt, D. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377480 · Cited by 200 · cited 4×
“Sketch Recognition (4,6,0)[18],(6,84,42)[105] Movie Recommendation (2,4,0)[52],(28,1,0)[104], (4,15,0)[63], (10,4,0)[10] Place Recommendation (12,2,0)[96], (51,12,0)[46] Food Recommendation (2,51,0)[37], (9,3,0)[72] Image Classification (3,12,0)[3],(5,216,0)[101], (4,90,0)[47], (3,40,0)[75] Sentiment Analysis (4,3,0)[…”
BackgroundInfluentialRapid Trust Calibration through Interpretable and Uncertainty-Aware AI
Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., & Kaplan, L. (2020) · Patterns · doi:10.1016/j.patter.2020.100049 · Cited by 186 · cited 1×
“Explanations [65, 97], stated performance [7, 102, 106], risk perception [39], and uncertainty [92] have been studied extensively in this context.”
BackgroundTask Complexity: Extending a Core Concept
Hærem, T., Pentland, B. T., & Miller, K. D. (2015) · Academy of Management Review · doi:10.5465/amr.2013.0350 · Cited by 180
The Effects of Time on Task in Response Selection - An ERP Study of Mental Fatigue
Möckel, T., Beste, C., & Wascher, E. (2015) · Scientific Reports · doi:10.1038/srep10113 · Cited by 172 · cited 1×
“This is the result of prolonged periods of demanding cognitive activity [50] and has been shown to negatively affect performance [73, 99].”
BackgroundEvaluating saliency map explanations for convolutional neural networks: a user study
Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., & Berthouze, N. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377519 · Cited by 163 · cited 2×
“Sketch Recognition (4,6,0)[18],(6,84,42)[105] Movie Recommendation (2,4,0)[52],(28,1,0)[104], (4,15,0)[63], (10,4,0)[10] Place Recommendation (12,2,0)[96], (51,12,0)[46] Food Recommendation (2,51,0)[37], (9,3,0)[72] Image Classification (3,12,0)[3],(5,216,0)[101], (4,90,0)[47], (3,40,0)[75] Sentiment Analysis (4,3,0)[…”
BackgroundFairness and Explanation in AI-Informed Decision Making
Angerschmid, A., Zhou, J., Theuermann, K., Chen, F., & Holzinger, A. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4020026 · Cited by 161 · cited 1×
“factors, the choice of decision tasks also affects the performance of human-AI teams [4, 8, 95].”
BackgroundLet Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems
Kunkel, J., Donkers, T., Michael, L., Barbu, C.-M., & Ziegler, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300717 · Cited by 156 · cited 4×
“Sketch Recognition (4,6,0)[18],(6,84,42)[105] Movie Recommendation (2,4,0)[52],(28,1,0)[104], (4,15,0)[63], (10,4,0)[10] Place Recommendation (12,2,0)[96], (51,12,0)[46] Food Recommendation (2,51,0)[37], (9,3,0)[72] Image Classification (3,12,0)[3],(5,216,0)[101], (4,90,0)[47], (3,40,0)[75] Sentiment Analysis (4,3,0)[…”
BackgroundInfluentialI Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450644 · Cited by 144 · cited 1×
“For example, recidivism prediction algorithms have been used to help judges determine whether defendants are likely to re-offend [26, 31, 65, 69], medical diseases are being diagnosed with AI systems [19, 57, 62], and loan risk prediction algorithms are employed to approve or reject loan applications [11, 25, 38, 95].”
MethodKawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517439 · Cited by 143 · cited 1×
“For instance, we found two scenarios of child clinical decision making [51] and nutrition prediction [17] tasks with the same component complexity; the first high-stakes task requires extensive domain knowledge, while the second task”
BackgroundA Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445472 · Cited by 137 · cited 2×
“Stroke Rehabilitation Assessment (25,60,0)[59, 60] Medical Image Retrieval (14,6,1023)[19]”
BackgroundJury Learning: Integrating Dissenting Voices into Machine Learning Models
Gordon, M. L., Lam, M. S., Park, J. S., Patel, K., Hancock, J., Hashimoto, T., & Bernstein, M. S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502004 · Cited by 132 · cited 1×
“Researchers have shown that explanations can effectively inform mental models of humans and improve their understanding, especially for laypeople [35, 36, 44, 90].”
BackgroundVisual, textual or hybrid: the effect of user expertise on different explanations
Szymanski, M., Millecamp, M., & Verbert, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450662 · Cited by 131 · cited 2×
“complicated and diverse visuals of task features are presented [90].”
BackgroundHuman Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131 · cited 3×
“Prior work has also examined how human trust and reliance on AI systems is affected by different design choices through empirical studies [16, 52, 67, 76].”
BackgroundDo People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
Gajos, K. Z., & Mamykina, L. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511138 · Cited by 127
Panigutti, C., Beretta, A., Giannotti, F., & Pedreschi, D. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502104 · Cited by 126 · cited 1×
“Medical Diagnosis (7,4,2)[79],(6,1,0)[9], (6,240,0)[32] Nutrition Prediction (5,26,156)[16],(3,24,0)[15], (4,48,24)[33] Recidivism Prediction (10,64,32)[97],(11,200,50)[40], (8,12,6)[29],(132,10,0)[95], (11,50,0)[69] Monitoring and Administration (15,130,390)[81] Job Application Approval (5,24,0)[82]”
"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Lai, V., Liu, H., & Tan, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376873 · Cited by 123
I can do better than your AI: expertise and explanations
Schaffer, J., O'Donovan, J., Michaelis, J., Raglin, A., & Höllerer, T. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302308 · Cited by 119 · cited 1×
“Additionally, to fill the knowledge gap between domain experts and laypeople or improve AI literacy, empirical studies engage with more explanations [22, 62, 87, 105].”
BackgroundNourani, M., King, J., & Ragan, E. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7469 · Cited by 116
Dikmen, M., & Burns, C. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102792 · Cited by 116 · cited 1×
“User factors like expertise [28], machine learning literacy [24], and task characteristics like task subjectivity and proximity [15], and task types [34, 49, 55, 61, 74] have also been broadly investigated.”
BackgroundSpringer, A., & Whittaker, S. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302322 · Cited by 115 · cited 3×
“Sketch Recognition (4,6,0)[18],(6,84,42)[105] Movie Recommendation (2,4,0)[52],(28,1,0)[104], (4,15,0)[63], (10,4,0)[10] Place Recommendation (12,2,0)[96], (51,12,0)[46] Food Recommendation (2,51,0)[37], (9,3,0)[72] Image Classification (3,12,0)[3],(5,216,0)[101], (4,90,0)[47], (3,40,0)[75] Sentiment Analysis (4,3,0)[…”
BackgroundMethodHuman Decision Making with Machine Assistance: An Experiment on Bailing and Jailing
Grgić-Hlača, N., Engel, C., & Gummadi, K. P. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359280 · Cited by 113 · cited 4×
“Medical Diagnosis (7,4,2)[79],(6,1,0)[9], (6,240,0)[32] Nutrition Prediction (5,26,156)[16],(3,24,0)[15], (4,48,24)[33] Recidivism Prediction (10,64,32)[97],(11,200,50)[40], (8,12,6)[29],(132,10,0)[95], (11,50,0)[69] Monitoring and Administration (15,130,390)[81] Job Application Approval (5,24,0)[82]”
BackgroundInfluentialThe limits of human predictions of recidivism
Lin, Z. “., Jung, J., Goel, S., & Skeem, J. (2020) · Science Advances · doi:10.1126/sciadv.aaz0652 · Cited by 112 · cited 1×
“[64], AI systems outperform humans when they have access to extensive amount of information.”
BackgroundSmith-Renner, A., Fan, R., Birchfield, M., Wu, T., Boyd-Graber, J., Weld, D. S., & Findlater, L. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376624 · Cited by 112 · cited 1×
“Tasks with low complexity can help human decision makers have a better understanding of AI systems [18, 88].”
BackgroundLiu, H., Lai, V., & Tan, C. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479552 · Cited by 111 · cited 4×
“Explanations [65, 97], stated performance [7, 102, 106], risk perception [39], and uncertainty [92] have been studied extensively”
BackgroundMethodInfluentialAn empirical study on the perceived fairness of realistic, imperfect machine learning models
Harrison, G., Hanson, J., Jacinto, C., Ramirez, J., & Ur, B. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372831 · Cited by 110 · cited 1×
“It is also common to present information about the AI systems to create a better perception of their functionality among users [18, 45, 45, 55].”
BackgroundAnchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems
Nourani, M., Roy, C., Block, J. E., Honeycutt, D. R., Rahman, T., Ragan, E., & Gogate, V. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450639 · Cited by 107 · cited 2×
“Prior work has also examined how human trust and reliance on AI systems is affected by different design choices through empirical studies [16, 52, 67, 76].”
BackgroundMethodAnik, A. I., & Bunt, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445736 · Cited by 103 · cited 1×
“4% set up their studies in both conditions, either by changing the decision task [5] or artificially modifying the scenario.”
Background“Who goes first? Influences of human-AI workflow on decision making in clinical imaging,”
Fogliato, R., Chappidi, S., Lungren, M., Fisher, P., Wilson, D., Fitzke, M., Parkinson, M., Horvitz, E., Inkpen, K., & Nushi, B. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533193 · Cited by 103
Bell, A., Solano-Kamaiko, I., Nov, O., & Stoyanovich, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533090 · Cited by 103 · cited 1×
“Sketch Recognition (4,6,0)[18],(6,84,42)[105] Movie Recommendation (2,4,0)[52],(28,1,0)[104], (4,15,0)[63], (10,4,0)[10] Place Recommendation (12,2,0)[96], (51,12,0)[46] Food Recommendation (2,51,0)[37], (9,3,0)[72] Image Classification (3,12,0)[3],(5,216,0)[101], (4,90,0)[47], (3,40,0)[75] Sentiment Analysis (4,3,0)[…”
Vaccaro, K., Sandvig, C., & Karahalios, K. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415238 · Cited by 102 · cited 1×
“tent moderation system [94], movie recommendation [104, 104], and medical diagnosis [9].”
BackgroundRechkemmer, A., & Yin, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501967 · Cited by 97 · cited 1×
“in another because it has been used in previous studies [85].”
ResultCOGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations
Abdul, A., von der Weth, C., Kankanhalli, M., & Lim, B. Y. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376615 · Cited by 96 · cited 1×
“House Price Prediction (9,15,0)[1],(11,24,12)[83], (12,40,10)[24]”
ViCE: visual counterfactual explanations for machine learning models
Gomez, O., Holter, S., Yuan, J., & Bertini, E. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377536 · Cited by 91 · cited 1×
“Researchers have shown that explanations can effectively inform mental models of humans and improve their understanding, especially for laypeople [35, 36, 44, 90].”
BackgroundTask complexity moderates group synergy
Almaatouq, A., Alsobay, M., Yin, M., & Watts, D. J. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2101062118 · Cited by 87 · cited 2×
“It has been found to be an essential predictor of human performance and behaviour [2, 21, 66], affecting the success of team work [7].”
BackgroundYou Complete Me: Human-AI Teams and Complementary Expertise
Zhang, Q., Lee, M. L., & Carter, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517791 · Cited by 85 · cited 4×
“Additionally, to fill the knowledge gap between domain experts and laypeople or improve AI literacy, empirical studies engage with more explanations [22, 62, 87, 105].”
BackgroundInfluentialEffect of Information Presentation on Fairness Perceptions of Machine Learning Predictors
van Berkel, N., Goncalves, J., Russo, D., Hosio, S., & Skov, M. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445365 · Cited by 84 · cited 7×
“For instance, in a recent study, a specific task was selected due to the abundance of datasets in its domain [95], and”
BackgroundMethodInfluentialTowards a Science of Human-AI Decision Making: A Survey of Empirical Studies
Lai, V., Chen, C., Liao, Q. V., Smith-Renner, A., & Tan, C. (2021) · arXiv · doi:10.48550/arxiv.2112.11471 · Cited by 75 · cited 3×
“High-Stake and Low-Stake : We also analyzed the risk of tasks as this is identified as one of four dimensions that vary in decision tasks by Lai et al. [54].”
BackgroundMethodLevy, A., Agrawal, M., Satyanarayan, A., & Sontag, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445522 · Cited by 73 · cited 3×
“Additionally, to fill the knowledge gap between domain experts and laypeople or improve AI literacy, empirical studies engage with more explanations [22, 62, 87, 105].”
BackgroundMethodHoneycutt, D., Nourani, M., & Ragan, E. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7464 · Cited by 72
Do I trust my machine teammate?: an investigation from perception to decision
Yu, K., Berkovsky, S., Taib, R., Zhou, J., & Chen, F. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302277 · Cited by 71 · cited 2×
“In the task with a score of 420, participants were presented with 210 questions regarding quality control in a drinking glass-making factory scenario [103].”
BackgroundSecond Chance for a First Impression? Trust Development in Intelligent System Interaction
Tolmeijer, S., Gadiraju, U., Ghantasala, R., Gupta, A., & Bernstein, A. (2021) · Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3450613.3456817 · Cited by 67
Self-citationA Slow Algorithm Improves Users' Assessments of the Algorithm's Accuracy
Park, J. S., Barber, R., Kirlik, A., & Karahalios, K. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359204 · Cited by 65 · cited 2×
“Our examination, established that around half of the tasks were still artificial [80, 105] or do not necessarily require human intervention.”
BackgroundSoames Job, R. F., & Dalziel, J. (2000) · Stress, Workload, and Fatigue · doi:10.1201/b12791-3.2 · Cited by 60 · cited 1×
“This is the result of prolonged periods of demanding cognitive activity [50] and has been shown to negatively affect performance [73, 99].”
BackgroundLee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415227 · Cited by 57
Visualizing Uncertainty and Alternatives in Event Sequence Predictions
Guo, S., Du, F., Malik, S., Koh, E., Kim, S., Liu, Z., Kim, D., Zha, H., & Cao, N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300803 · Cited by 56 · cited 1×
“…are sometimes simplified for lay people to complete, some knowledge and familiarity, such as AI literacy [24], numeracy, and statistics background [41], may still be required to accomplish tasks, which may not be feasible to expect from all crowd workers — expert recruitment on-demand remains a…”
BackgroundYou’d Better Stop! Understanding Human Reliance on Machine Learning Models under Covariate Shift
Chiang, C.-W., & Yin, M. (2021) · 13th ACM Web Science Conference 2021 · doi:10.1145/3447535.3462487 · Cited by 53 · cited 1×
“To promote appropriate reliance on AI systems, different interventions including explanations of AI advice [97], cognitive forcing functions [16], and user tutorials [23, 24] have been proposed in empirical studies of human-AI decision mak-”
BackgroundLeveraging rationales to improve human task performance
Das, D., & Chernova, S. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377512 · Cited by 52 · cited 2×
“On one hand, more complex tasks may imply a greater need for humans to rely on AI systems [27] as a result of increased information overload in such tasks [21].”
BackgroundLiao, M., Sundar, S. S., & B. Walther, J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501936 · Cited by 52 · cited 1×
“Sketch Recognition (4,6,0)[18],(6,84,42)[105] Movie Recommendation (2,4,0)[52],(28,1,0)[104], (4,15,0)[63], (10,4,0)[10] Place Recommendation (12,2,0)[96], (51,12,0)[46] Food Recommendation (2,51,0)[37], (9,3,0)[72] Image Classification (3,12,0)[3],(5,216,0)[101], (4,90,0)[47], (3,40,0)[75] Sentiment Analysis (4,3,0)[…”
What's in a User? Towards Personalising Transparency for Music Recommender Interfaces
Millecamp, M., Htun, N. N., Conati, C., & Verbert, K. (2020) · Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3340631.3394844 · Cited by 50
Why does my model fail?: contrastive local explanations for retail forecasting
Lucic, A., Haned, H., & de Rijke, M. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372824 · Cited by 45 · cited 1×
“For instance, many studies provide explanations along with AI decisions to help humans interpret AI systems’ decisions [11, 12, 38, 40, 68].”
BackgroundShared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behavior
Boggust, A., Hoover, B., Satyanarayan, A., & Strobelt, H. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501965 · Cited by 45 · cited 2×
“Lastly, we can see distinct types of images in image classification tasks [3, 13], from clinical, nutrition-related, animal, and animal images.”
BackgroundChan, S. H., Song, Q., & Yao, L. J. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.04.059 · Cited by 43 · cited 2×
“It has been found to be an essential predictor of human performance and behaviour [2, 21, 66], affecting the success of team work [7].”
BackgroundExploring the Effects of Natural Language Justifications in Food Recommender Systems
Musto, C., Starke, A. D., Trattner, C., Rapp, A., & Semeraro, G. M. (2021) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3450613.3456827 · Cited by 43
Chiang, C.-W., & Yin, M. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511121 · Cited by 42
What You See Is What You Get? The Impact of Representation Criteria on Human Bias in Hiring
Peng, A., Nushi, B., Kıcıman, E., Inkpen, K., Suri, S., & Kamar, E. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5281 · Cited by 41
Improving understandability of feature contributions in model-agnostic explainable AI tools
Hadash, S., Willemsen, M. C., Snijders, C., & IJsselsteijn, W. A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517650 · Cited by 33 · cited 1×
“Researchers have shown that explanations can effectively inform mental models of humans and improve their understanding, especially for laypeople [35, 36, 44, 90].”
BackgroundNauta, M., Trienes, J., Pathak, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.08164 · Cited by 30 · cited 1×
“User factors like expertise [28], machine learning literacy [24], and task characteristics like task subjectivity and proximity [15], and task types [34, 49, 55, 61, 74] have also been broadly investigated.”
BackgroundThe challenges of providing explanations of AI systems when they do not behave like users expect
Riveiro, M., & Thill, S. (2022) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3503252.3531306 · Cited by 26 · cited 1×
“For instance, sentiment analysis [89] and text classification [86]”
MethodZehrung, R., Singhal, A., Correll, M., & Battle, L. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445195 · Cited by 26 · cited 4×
“There are four tasks with the lowest coordinative complexity, where participants were asked to react to the hypothetical scenario in which their Facebook account is suspended by an algorithmic content moderation system [94], movie recommendation [104, 104], and medical diagnosis [9].”
BackgroundInfluentialDo I Look Like a Criminal? Examining how Race Presentation Impacts Human Judgement of Recidivism
Mallari, K., Inkpen, K., Johns, P., Tan, S., Ramesh, D., & Kamar, E. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376257 · Cited by 24 · cited 2×
“(5,26,156)[16],(3,24,0)[15],(4,48,24)[33] Recidivism Prediction (10,64,32)[97],(11,200,50)[40],(8,12,6)[29],(132,10,0)[95], (11,50,0)[69] Monitoring and Administration (15,130,390)[81] Job Application Approval (5,24,0)[82] (a) Task complexity in high-stake domains”
MethodA User Study on Groups Interacting with Tourist Trip Recommender Systems in Public Spaces
Herzog, D., & Wörndl, W. (2019) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3320435.3320449 · Cited by 24 · cited 2×
“Sketch Recognition (4,6,0)[18],(6,84,42)[105] Movie Recommendation (2,4,0)[52],(28,1,0)[104], (4,15,0)[63], (10,4,0)[10] Place Recommendation (12,2,0)[96], (51,12,0)[46] Food Recommendation (2,51,0)[37], (9,3,0)[72] Image Classification (3,12,0)[3],(5,216,0)[101], (4,90,0)[47], (3,40,0)[75] Sentiment Analysis (4,3,0)[…”
BackgroundQ-Chef: The impact of surprise-eliciting systems on food-related decision-making
Grace, K. S., Finch, E., Gulbransen-Diaz, N., & Henderson, H. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501862 · Cited by 19 · cited 1×
“Sketch Recognition (4,6,0)[18],(6,84,42)[105] Movie Recommendation (2,4,0)[52],(28,1,0)[104], (4,15,0)[63], (10,4,0)[10] Place Recommendation (12,2,0)[96], (51,12,0)[46] Food Recommendation (2,51,0)[37], (9,3,0)[72] Image Classification (3,12,0)[3],(5,216,0)[101], (4,90,0)[47], (3,40,0)[75] Sentiment Analysis (4,3,0)[…”
Can Computers Outperform Humans in Detecting User Zone-Outs? Implications for Intelligent Interfaces
Bosch, N., & D’Mello, S. K. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3481889 · Cited by 18 · cited 1×
“The task with the component complexity score of 1 is related to a mind wandering detection task in which crowdworkers were asked to identify whether people’s attention in the presented video clip drifted away [14].”
BackgroundProceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization
Tsvi Kuflik, I. T., & Tsvi Kuflik, I. T. (2020) · doi:10.1145/3340631 · Cited by 16
Viswanathan, S., Omidvar-Tehrani, B., & Renders, J.-M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501912 · Cited by 11 · cited 1×
“Sketch Recognition (4,6,0)[18],(6,84,42)[105] Movie Recommendation (2,4,0)[52],(28,1,0)[104], (4,15,0)[63], (10,4,0)[10] Place Recommendation (12,2,0)[96], (51,12,0)[46] Food Recommendation (2,51,0)[37], (9,3,0)[72] Image Classification (3,12,0)[3],(5,216,0)[101], (4,90,0)[47], (3,40,0)[75] Sentiment Analysis (4,3,0)[…”
Play for Real(ism) - Using Games to Predict Human-AI interactions in the Real World
Guttman, R. D., Hammer, J., Harpstead, E., & Smith, C. J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3474655 · Cited by 10
Relationship between Device Performance, Trust and User Behaviour in a Care-taking Scenario
Partovi, A., Ingrid Zukerman, A., Zhan, K., Hamacher, N., & Hohwy, J. (2019) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3320435.3320440 · Cited by 6
Ulusan, A., Narayan, U., Snodgrass, S., Ergün, Ö., & Harteveld, C. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511163 · Cited by 5
Research works printed but not linked: 7
- 10.5555/3172077.3172090 · doi:10.5555/3172077.3172090
- 10.1145/3449287 · doi:10.1145/3449287 · Paper page
- Ben Green and Yiling Chen. 2020. Algorithmic risk assessments can alter human decision-making processes in high-stakes government contexts. arXiv preprint arXiv:2012.05370 (2020). · arXiv
- Vivian Lai, Chacha Chen, Qingzi Vera Liao, Alison Smith-Renner, and Chenhao Tan. 2021. Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies. ArXiv abs/2112.11471 (2021). · arXiv
- John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
- 10.1145/3397481.3450650 · DOI · Paper page
- 10.1145/3351095.3372852 · DOI · Paper page
Cited by
Benk, M., Kerstan, S., von Wangenheim, F., & Ferrario, A. (2024) · AI & Society · doi:10.1007/s00146-024-02059-y · Cited by 122
“…level (Simmler and Frischknecht 2021; c) the interaction’s relational dynamics (e.g., goals, success criteria, trustworthiness criteria of the AI(Salimzadeh et al. 2023), socio-technical context (Ehsan and Riedl 2020; Benk et al. 2022), and finally (d) the risk assessment (e.g., interaction…”
Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
He, G., Aishwarya, N., & Gadiraju, U. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712133 · Cited by 115
“This simulates a realistic scenario where participants interact with an AI system and may rely on AI advice and XAI methods due to the inherent complexity in decision-making [99].”
Self-citationKim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., & Russakovsky, O. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714020 · Cited by 58
“…128, 129] and (un)certainty [6, 14, 17, 41, 132], explanations of outputs [6, 14, 17, 40, 41, 69, 132], and onboarding materials [18, 68] impact user reliance, as well as the roles played by human intuition [20], task complexity [101, 102], and other human, AI, and context-related factors [63].”
Explainable AI Frameworks: Navigating the Present Challenges and Unveiling Innovative Applications
Sharma, N. A., Chand, R. R., Buksh, Z., Ali, A. B. M. S., Hanif, A., & Beheshti, A. (2024) · Algorithms · doi:10.3390/a17060227 · Cited by 60
“Consider an AI system as a complicated puzzle [14], in which each piece signifies a step in the algorithm’s decision-making protocol [15].”
BackgroundMethodTrust Development and Repair in AI-Assisted Decision-Making during Complementary Expertise
Pareek, S., Velloso, E., & Gonçalves, J. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658924 · Cited by 54
Metadata
- Authors
- Salimzadeh, S., He, G., & Gadiraju, U.
- Affiliations
- Delft University of Technology (NL)
- Venue
- Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization
- Publisher
- ACM
- Published
- 2023-06-18
- Pages
- 215-227
- OpenAlex
- W4381189853
- Cited by
- 42
- References
- 100
- Citations by year
- 2026: 7 · 2025: 24 · 2024: 11
- FWCI
- 21.01
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on User Modeling, Adaptation and Personalization (UMAP) (publishedVersion, cc-by)
- Research Repository (Delft University of Technology) (submittedVersion, cc-by)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- task complexity, human-AI decision-making, human-AI teaming, construct operationalization, component complexity, dynamic complexity, AI transparency, explainable AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.