A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making
Open accessThe DOI matches its Crossref record; 107 of the 107 references it cites were found in a registry or an index.
Abstract
Recent advances in the performance of machine learning algorithms have led to the adoption of AI models in decision making contexts across various domains such as healthcare, finance, and education. Different research communities have attempted to optimize and evaluate human-AI team performance through empirical studies by increasing transparency of AI systems, or providing explanations to aid human understanding of such systems. However, the variety in decision making tasks considered and their operationalization in prior empirical work, has led to an opacity around how findings from one task or domain carry forward to another. The lack of a standardized means of considering task attributes prevents straightforward comparisons across decision tasks, thereby limiting the generalizability of findings. We argue that the lens of ‘task complexity’ can be used to tackle this problem of under-specification and facilitate comparison across empirical research in this area. To retrospectively explore how different HCI communities have considered the influence of task complexity in designing experiments in the realm of human-AI decision making, we survey literature and provide an overview of empirical studies on this topic. We found a serious dearth in the consideration of task complexity across various studies in this realm of research. Inspired by Robert Wood’s seminal work on the construct, we operationalized task complexity with respect to three dimensions (component, coordinative, and dynamic) and quantified the complexity of decision tasks in existing work accordingly. We then summarized current trends and proposed research directions for the future. Our study highlights the need to account for task complexity as an important design choice. This is a first step to help the scientific community in drawing meaningful comparisons across empirical studies in human-AI decision making and to provide opportunities to generalize findings across diverse domains and experimental settings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1956) · Psychological Review · doi:10.1037/h0043158 · Cited by 17,812
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 2×
“Doshi-Velez and Kim [30] have argued that to create and advance a ‘rigorous science’ in the realm of human-AI decision making, there is room for empirical work that considers functionally-grounded explanations with proxy users and tasks, human-grounded evaluation with real users and simple tasks, and application-groun…”
BackgroundHuman trust in artificial intelligence: Review of empirical research
Glikson, E., & Woolley, A. W. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0057 · Cited by 2,921 · cited 1×
“User factors like expertise [28], machine learning literacy [24], and task characteristics like task subjectivity and proximity [15], and task types [34, 49, 55, 61, 74] have also been broadly investigated.”
BackgroundTask Complexity: A Review and Analysis
Campbell, D. J. (1988) · The Academy of Management Review · doi:10.5465/amr.1988.4306775 · Cited by 1,634 · cited 5×
“Among all works, the theories introduced by Campbell [20], Wood [100] gained popularity with more than 2000 citations and became the basis of other frameworks.”
BackgroundInfluentialTask complexity: Definition of the construct
Wood, R. E. (1986) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(86)90044-0 · Cited by 1,393 · cited 9×
“posed in seminal work by Wood [100] in the realm of human-AI decision making.”
BackgroundMethodInfluentialLee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338 · cited 2×
“Even though some studies incorporate tasks with different characteristics [53, 55, 61], task attributes haven’t been identified systematically in the literature, so their impact on human-AI complementary performance has not been fully investigated.”
BackgroundProceedings of the 2021 CHI Conference on Human Factors in Computing Systems
(2021) · doi:10.1145/3411764 · Cited by 1,240
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
The accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071 · cited 1×
“For example, recidivism prediction algorithms have been used to help judges determine whether defendants are likely to re-offend [26, 31, 65, 69], medical diseases are being diagnosed with AI systems [19, 57, 62], and loan risk prediction algorithms are employed to approve or reject loan applications [11, 25, 38, 95].”
Method
Research works printed but not linked: 7
- 10.5555/3172077.3172090 · doi:10.5555/3172077.3172090
- 10.1145/3449287 · doi:10.1145/3449287 · Paper page
- Ben Green and Yiling Chen. 2020. Algorithmic risk assessments can alter human decision-making processes in high-stakes government contexts. arXiv preprint arXiv:2012.05370 (2020). · arXiv
- Vivian Lai, Chacha Chen, Qingzi Vera Liao, Alison Smith-Renner, and Chenhao Tan. 2021. Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies. ArXiv abs/2112.11471 (2021). · arXiv
- John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
- 10.1145/3397481.3450650 · DOI · Paper page
- 10.1145/3351095.3372852 · DOI · Paper page
Cited by
Benk, M., Kerstan, S., von Wangenheim, F., & Ferrario, A. (2024) · AI & Society · doi:10.1007/s00146-024-02059-y · Cited by 122
“…level (Simmler and Frischknecht 2021; c) the interaction’s relational dynamics (e.g., goals, success criteria, trustworthiness criteria of the AI(Salimzadeh et al. 2023), socio-technical context (Ehsan and Riedl 2020; Benk et al. 2022), and finally (d) the risk assessment (e.g., interaction…”
Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
He, G., Aishwarya, N., & Gadiraju, U. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712133 · Cited by 115
“This simulates a realistic scenario where participants interact with an AI system and may rely on AI advice and XAI methods due to the inherent complexity in decision-making [99].”
Self-citationKim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., & Russakovsky, O. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714020 · Cited by 58
“…128, 129] and (un)certainty [6, 14, 17, 41, 132], explanations of outputs [6, 14, 17, 40, 41, 69, 132], and onboarding materials [18, 68] impact user reliance, as well as the roles played by human intuition [20], task complexity [101, 102], and other human, AI, and context-related factors [63].”
Explainable AI Frameworks: Navigating the Present Challenges and Unveiling Innovative Applications
Sharma, N. A., Chand, R. R., Buksh, Z., Ali, A. B. M. S., Hanif, A., & Beheshti, A. (2024) · Algorithms · doi:10.3390/a17060227 · Cited by 60
“Consider an AI system as a complicated puzzle [14], in which each piece signifies a step in the algorithm’s decision-making protocol [15].”
BackgroundMethodTrust Development and Repair in AI-Assisted Decision-Making during Complementary Expertise
Pareek, S., Velloso, E., & Gonçalves, J. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658924 · Cited by 54
Metadata
- Authors
- Salimzadeh, S., He, G., & Gadiraju, U.
- Affiliations
- Delft University of Technology (NL)
- Venue
- Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization
- Publisher
- ACM
- Published
- 2023-06-18
- Pages
- 215-227
- OpenAlex
- W4381189853
- Cited by
- 42
- References
- 100
- Citations by year
- 2026: 7 · 2025: 24 · 2024: 11
- FWCI
- 21.01
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on User Modeling, Adaptation and Personalization (UMAP) (publishedVersion, cc-by)
- Research Repository (Delft University of Technology) (submittedVersion, cc-by)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- task complexity, human-AI decision-making, human-AI teaming, construct operationalization, component complexity, dynamic complexity, AI transparency, explainable AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making. https://app.cochat.ai/papers/10.1145/3565472.3592959
- The paper
- Salimzadeh, S., He, G., & Gadiraju, U. (2023). A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making. Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization. https://doi.org/10.1145/3565472.3592959
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.