CoChat

A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making

Open access
Salimzadeh, Sara, He, Gaole, Gadiraju, Ujwal · 2023 · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization

The DOI matches its Crossref record; 107 of the 107 references it cites were found in a registry or an index.

Abstract

Recent advances in the performance of machine learning algorithms have led to the adoption of AI models in decision making contexts across various domains such as healthcare, finance, and education. Different research communities have attempted to optimize and evaluate human-AI team performance through empirical studies by increasing transparency of AI systems, or providing explanations to aid human understanding of such systems. However, the variety in decision making tasks considered and their operationalization in prior empirical work, has led to an opacity around how findings from one task or domain carry forward to another. The lack of a standardized means of considering task attributes prevents straightforward comparisons across decision tasks, thereby limiting the generalizability of findings. We argue that the lens of ‘task complexity’ can be used to tackle this problem of under-specification and facilitate comparison across empirical research in this area. To retrospectively explore how different HCI communities have considered the influence of task complexity in designing experiments in the realm of human-AI decision making, we survey literature and provide an overview of empirical studies on this topic. We found a serious dearth in the consideration of task complexity across various studies in this realm of research. Inspired by Robert Wood’s seminal work on the construct, we operationalized task complexity with respect to three dimensions (component, coordinative, and dynamic) and quantified the complexity of decision tasks in existing work accordingly. We then summarized current trends and proposed research directions for the future. Our study highlights the need to account for task complexity as an important design choice. This is a first step to help the scientific community in drawing meaningful comparisons across empirical studies in human-AI decision making and to provide opportunities to generalize findings across diverse domains and experimental settings.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3565472.3592959

Cites

107 of 107 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 A workflow diagram of the semi-systematic literature review process that we followed.
Figure 1 A workflow diagram of the semi-systematic literature review process that we followed.CC-BY · From the PDF
Figure 2 A decision task study. The features of the defendant profile are recognized as information cues, each step is an act, and the final decision is considered the product.
Figure 2 A decision task study. The features of the defendant profile are recognized as information cues, each step is an act, and the final decision is considered the product.CC-BY · From the PDF
Image 3
CC-BY · From the PDF
Figure 4 Distribution of component, coordinative, and dynamic complexity in the decision tasks corresponding to our corpus.
Figure 4 Distribution of component, coordinative, and dynamic complexity in the decision tasks corresponding to our corpus.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 107 references

Research works printed but not linked: 7

  1. 10.5555/3172077.3172090 · doi:10.5555/3172077.3172090
  2. 10.1145/3449287 · doi:10.1145/3449287 · Paper page
  3. Ben Green and Yiling Chen. 2020. Algorithmic risk assessments can alter human decision-making processes in high-stakes government contexts. arXiv preprint arXiv:2012.05370 (2020). · arXiv
  4. Vivian Lai, Chacha Chen, Qingzi Vera Liao, Alison Smith-Renner, and Chenhao Tan. 2021. Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies. ArXiv abs/2112.11471 (2021). · arXiv
  5. John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
  6. 10.1145/3397481.3450650 · DOI · Paper page
  7. 10.1145/3351095.3372852 · DOI · Paper page

Cited by

See all 57

Metadata

Authors
Salimzadeh, S., He, G., & Gadiraju, U.
Affiliations
Delft University of Technology (NL)
Venue
Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization
Publisher
ACM
Published
2023-06-18
Pages
215-227
OpenAlex
W4381189853
Cited by
42
References
100
Citations by year
2026: 7 · 2025: 24 · 2024: 11
FWCI
21.01
Type
conference paper
Language
en
Open access
Locations
  • ACM Conference on User Modeling, Adaptation and Personalization (UMAP) (publishedVersion, cc-by)
  • Research Repository (Delft University of Technology) (submittedVersion, cc-by)
Topics
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
task complexity, human-AI decision-making, human-AI teaming, construct operationalization, component complexity, dynamic complexity, AI transparency, explainable AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making. https://app.cochat.ai/papers/10.1145/3565472.3592959
The paper
Salimzadeh, S., He, G., & Gadiraju, U. (2023). A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making. Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization. https://doi.org/10.1145/3565472.3592959

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.