The DOI matches its Crossref record; 150 of the 150 references it cites were found in a registry or an index; 37 of the 76 sources its claims cite were checked for support.
Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making
Open accessAbstract
While existing literature has explored and revealed several insights pertaining to the role of human factors (e.g., prior experience, domain knowledge) and attributes of AI systems (e.g., accuracy, trustworthiness), there is a limited understanding around how the important task characteristics of complexity and uncertainty shape human decision-making and human-AI team performance. In this work, we aim to address this research and empirical gap by systematically exploring how task complexity and uncertainty influence human-AI decision-making. Task complexity refers to the load of information associated with a task, while task uncertainty refers to the level of unpredictability associated with the outcome of a task. We conducted a between-subjects user study (N = 258) in the context of a trip-planning task to investigate the impact of task complexity and uncertainty on human trust and reliance on AI systems. Our results revealed that task complexity and uncertainty have a significant impact on user reliance on AI systems. When presented with complex and uncertain tasks, users tended to rely more on AI systems while demonstrating lower levels of appropriate reliance compared to tasks that were less complex and uncertain. In contrast, we found that user trust in the AI systems was not influenced by task complexity and uncertainty. Our findings can help inform the future design of empirical studies exploring human-AI decision-making. Insights from this work can inform the design of AI systems and interventions that are better aligned with the challenges posed by complex and uncertain tasks. Finally, the lens of diagnostic versus prognostic tasks can inspire the operationalization of uncertainty in human-AI decision-making studies.
Overview
CoChat Summary
TL;DR
This CHI 2024 study examines how task complexity and task uncertainty (diagnostic vs. prognostic) shape trust and reliance in human-AI decision-making using a trip-planning task. Across 258 participants, higher complexity and uncertainty increased reliance on AI but did not significantly change subjective trust.
Key findings
- Task complexity significantly affected reliance: Switch Fraction rose from 0.18 ± 0.32 (low) to 0.26 ± 0.30 (medium) to 0.34 ± 0.36 (high), adjusted p = .003; Accuracy fell from 0.79 ± 0.22 (low) to 0.58 ± 0.27 (medium) and 0.61 ± 0.29 (high), p < .001.
- Appropriate reliance declined with complexity: Accuracy-wid was 0.61 ± 0.40 (low), 0.40 ± 0.37 (medium), and 0.50 ± 0.35 (high), p = .001; RAIR increased from 0.22 ± 0.41 (low) to 0.33 ± 0.41 (medium) and 0.43 ± 0.45 (high), p = .001.
- Task uncertainty significantly affected reliance: prognostic tasks showed higher Switch Fraction (0.31 ± 0.34 vs. 0.22 ± 0.32, p = .02) and RAIR (0.38 ± 0.44 vs. 0.27 ± 0.42, p = .02), but lower Accuracy (0.60 ± 0.24 vs. 0.72 ± 0.30, p < .001) and Accuracy-wid (0.45 ± 0.33 vs. 0.56 ± 0.43, p = .04) than diagnostic tasks.
- Trust in the AI system was not significantly influenced by task complexity or uncertainty; hypotheses H1b and H2b were rejected. A significant interaction on Accuracy-wid was found, with a turning point where appropriate reliance increased in high-complexity prognostic tasks, rejecting H3.
Methods
- Between-subjects 3 × 2 factorial user study with six conditions: task complexity (low, medium, high) × task uncertainty (diagnostic, prognostic). Participants completed three trip-planning tasks with an imperfect AI system fixed at 66.7% accuracy.
- Final sample: 258 participants from Prolific after excluding 27 of 285 who failed attention checks or a quiz; inclusion required native English speakers, ≥95% approval rate, and ≥100 completed studies. Average completion time was about 25 minutes.
- Complexity was manipulated by number of constraints (4 low, 8 medium, 12 high); uncertainty was manipulated via diagnostic vs. prognostic scenarios (present trip with exact values vs. trip two weeks later with ranges/probabilities). Reliance metrics included Agreement Fraction, Switch Fraction, Accuracy with Disagreement, RAIR, and RSR; trust was measured with the Trust in Automation questionnaire. Analyses used Kruskal-Wallis with Dunn post-hoc and Bonferroni correction, plus ANCOVA/ANOVA.
Limitations
- Stated: findings are not universally applicable to all decision-making tasks; other operationalizations of task uncertainty were not explored; traffic features were assumed unrelated and equally weighted; traffic conditions were included in both diagnostic and prognostic scenarios even though real-world traffic is predominantly prognostic.
- Stated: cognitive biases may affect results, including familiarity bias, availability heuristic, confirmation bias, overconfidence bias, and self-interest bias.
- Evident from the text: the study uses a single trip-planning domain, a fixed AI accuracy of 66.7%, artificial routes, and a Prolific sample of native English speakers, which may limit generalizability beyond this task and population.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
16 Valid · 2 Not valid · 14 Not checked · 91 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“On the other hand, when individuals are faced with tasks that have lower uncertainty, such as annotation and classifcation task [4, 77, 117], they tend to rely less on the AI advice and rely more on their expertise and judgment.”
What the sources say
- Not checked — no full text of the sourcecites [4] [4] Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza, and Nadia Berthouze. 2020. Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI '20) . Association for Computing Machinery, New York, NY, USA, 275-285. https://doi.org/10.1145/3377325.3377519 - source
- Disputedcites [77] [77] Ariel Levy, Monica Agrawal, Arvind Satyanarayan, and David Sontag. 2021. Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiative. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21) . Association for Computing Machinery, New York, NY, USA, Article 72, 13 pages. https://doi.org/10.1145/3411764.3445522 - sourceExperimental — accuracy not yet measured
The Introduction reports that clinicians given fully pre-populated annotations exercised less agency, not more: they were more hesitant to correct spans and less likely to add annotations. The Discussion likewise says pre-populated suggestions caused a loss of agency and cautions that the findings may not extend to simpler classification tasks. The paper does not characterize its annotation task as lower-uncertainty. (LLM, conf=0.84)
Checked by a reasoning model — may be wrong.
- Not checked — no full text of the sourcecites [117] [117] Alison Smith-Renner, Ron Fan, Melissa Birchfeld, Tongshuang Wu, Jordan BoydGraber, Daniel S. Weld, and Leah Findlater. 2020. No Explainability without Accountability: An Empirical Study of Explanations and Feedback in Interactive ML. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20) . Association for Computing Machinery, New York, NY, USA, 1-13. https://doi.org/10.1145/3313831.3376624 - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“there is a limited understanding around how the important task characteristics of complexity and uncertainty shape human decision-making and human-AI team performance.”
- Also stated in: § 1 INTRODUCTION AND BACKGROUND · plain
“However, limited task characteristics have been systematically explored and their impact on human reliance on AI systems is not yet fully understood [68, 109].”
What the sources say
- Partly supportedcites [68] [68] Vivian Lai, Chacha Chen, Alison Smith-Renner, Q. Vera Liao, and Chenhao Tan. 2023. Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA) (FAccT '23) . Association for Computing Machinery, New York, NY, USA, 1369-1385. https://doi.org/10.1145/3593013.3594087 - sourceExperimental — accuracy not yet measured
The Task characteristics section says task properties may affect reliance on AI and that studies often fail to justify task choices or establish how findings generalize. But it does not identify task complexity and task uncertainty as the two characteristics whose effects on human decision-making and human-AI team performance are poorly understood. Its discussion of uncertainty in Model predictions concerns AI prediction estimates, not task uncertainty. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [109] [109] Sara Salimzadeh, Gaole He, and Ujwal Gadiraju. 2023. A Missing Piece in the Puzzle: Considering the Role of Task Complexity in Human-AI Decision Making. In Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization (Limassol, Cyprus) (UMAP '23) . Association for Computing Machinery, New York, NY, USA, 215-227. https://doi.org/10.1145/3565472. 3592959 - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION AND BACKGROUND · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prior work has highlighted that appropriate trust and reliance play a critical role in achieving complementary human-AI team performance [58, 90, 139, 141].”
What the sources say
- Not checked — no full text of the sourcecites [58] [58] Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. 2021. Formalizing Trust in Artifcial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT '21) . Association for Computing Machinery, New York, NY, USA, 624-635. https://doi.org/10.1145/3442188. 3445923 - source
- Not checked — no full text of the sourcecites [90] [90] David Miller, Mishel Johns, Brian Mok, Nikhil Gowda, David Sirkin, Key Lee, and Wendy Ju. 2016. Behavioral Measurement of Trust in Automation: The Trust Fall. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 60, 1 (2016), 1849-1853. https://doi.org/10.1177/1541931213601422 arXiv:https://doi.org/10.1177/1541931213601422 - source
- Partly supportedcites [139] [139] Kun Yu, Shlomo Berkovsky, Ronnie Taib, Jianlong Zhou, and Fang Chen. 2019. Do I Trust My Machine Teammate? An Investigation from Perception to Decision. In Proceedings of the 24th International Conference on Intelligent User Interfaces (Marina del Ray, California) (IUI '19) . Association for Computing Machinery, New York, NY, USA, 460-468. https://doi.org/10.1145/3301275.3302277 - sourceExperimental — accuracy not yet measured
The Introduction calls trust a critical factor in human–machine decisions and performance, and Related Work says appropriate trust benefits collaboration. Results §4.2 links trust to reliance on machine recommendations. The paper does not establish that appropriate trust and reliance achieve *complementary* team performance, as distinct from affecting collaboration and decisions; page numbers are unavailable in our copy. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [141] [141] Qiaoning Zhang, Matthew L Lee, and Scott Carter. 2022. You Complete Me: Human-AI Teams and Complementary Expertise. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22) . Association for Computing Machinery, New York, NY, USA, Article 114, 28 pages. https://doi.org/10.1145/3491102.3517791 - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Furthermore, the level of trust and reliance on AI may difer across various domains and applications due to the attributes associated with decision tasks [42, 127].”
What the sources say
- Not checked — no full text of the sourcecites [42] [42] Ella Glikson and Anita Williams Woolley. 2020. Human trust in artifcial intelligence: Review of empirical research. Academy of Management Annals 14, 2 (2020), 627-660. - source
- Supportedcites [127] [127] Michael Veale, Max Van Kleek, and Reuben Binns. 2018. Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18) . Association for Computing Machinery, New York, NY, USA, 1-14. https://doi.org/10.1145/3173574.3174014 - source
The discussion on p. 9 says that whether people trust and rely on decision-support systems is highly contextual. It also identifies differences in tasks, stakes, and contexts as factors that may mediate how algorithms are perceived, supporting the claim's qualified wording. (LLM, conf=0.93)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This design choice is guided by prior neuroscience research by Miller [91], suggesting that human cognitive capacity for processing information is limited to around seven ( ± two) chunks of information at a time.”
What the sources say
- Partly supportedcites [91] [91] George A. Miller. 1956. The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. The Psychological Review 63, 2 (March 1956), 81-97. http://www.musanim.com/miller1956/ - sourceExperimental — accuracy not yet measured
In the immediate-memory discussion (pp. 91–93), Miller describes a span of about seven items, with the number of chunks—not bits of information—limiting immediate memory. The cited sentence extends that specific finding to human cognitive processing generally; Miller explicitly distinguishes immediate memory from other processing limits. (LLM, conf=0.88)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
59 more claims · 4 with their cited sources checked
Create a free account to see all 123 claims and to verify the 4 references still open.
- Probably not a substantial claim. 40
- It may state two separate claims as one. 28
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Timely feedback and task-oriented support appear to strengthen students' functional evaluations of conversational AI (Ng et al., 2024; Salimzadeh et al., 2024).”
From The effects of responsiveness, perceived warmth, and anthropomorphism on university students' use of conversational AI for learning support: a chain mediation analysis based on S-O-R framework (2026) · cites this paper as Salimzadeh et al., 2024
Citations
Cites
The magical number seven, plus or minus two: Some limits on our capacity for processing information.
Miller, G. A. (1956) · Psychological Review · doi:10.1037/h0043158 · Cited by 17,812 · cited 2×
“This design choice is guided by prior neuroscience research by Miller [91], suggesting that human cognitive capacity for processing information is limited to around seven ( ± two) chunks of information at a time.”
Kruger, J., & Dunning, D. (1999) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.77.6.1121 · Cited by 7,731
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Human trust in artificial intelligence: Review of empirical research
Glikson, E., & Woolley, A. W. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0057 · Cited by 2,921 · cited 3×
“Furthermore, the level of trust and reliance on AI may differ across various domains and applications due to the attributes associated with decision tasks [42, 127].”
Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817
Foundations for an Empirically Determined Scale of Trust in Automated Systems
Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000) · International Journal of Cognitive Ergonomics · doi:10.1207/s15327566ijce0401_04 · Cited by 1,799
Task complexity: Definition of the construct
Wood, R. E. (1986) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(86)90044-0 · Cited by 1,393 · cited 2×
“Task complexity pertains to the characteristics of a task that contribute to an increased load of information [133], and it is distinct from task difficulty [100], which relates to an individual’s perception of the task-based on their capabilities and previous experience [133].”
Lee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338 · cited 3×
“We also considered traffic conditions in both diagnostic and prognostic scenarios, although, in the real world, traffic conditions can change over time and at the time of decision-making, making them predominantly prognostic.”
BackgroundResultBuçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Research works printed but not linked: 4
- Jason Burton, Mari-Klara Stein, and Tina Blegind Jensen. 2019. A systematic review of algorithm aversion in augmented decision making. Journal of Behavioral Decision Making (2019). https://api.semanticscholar.org/CorpusID:210439660
- Gaole He and Ujwal Gadiraju. 2022. Walking on Eggshells: Using Analogies to Promote Appropriate Reliance in Human-AI Decision Making. In Proceedings of the Workshop on Trust and Reliance on AI-Human Teams at the ACM Conference on Human Factors in Computing Systems (CHI’22).
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
Cited by
Kim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., & Russakovsky, O. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714020 · Cited by 58
“…128, 129] and (un)certainty [6, 14, 17, 41, 132], explanations of outputs [6, 14, 17, 40, 41, 69, 132], and onboarding materials [18, 68] impact user reliance, as well as the roles played by human intuition [20], task complexity [101, 102], and other human, AI, and context-related factors [63].”
Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
He, G., Aishwarya, N., & Gadiraju, U. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712133 · Cited by 115
“Others have explored the role of task factors such as task complexity and uncertainty in shaping trust and reliance in human-AI decision-making [98, 100].”
Self-citationEnhancing AI-Human Collaborative Decision-Making in Industry 4.0 Management Practices
Alam, S., & Khan, M. F. (2024) · IEEE Access · doi:10.1109/access.2024.3449415 · Cited by 49
“One of the key challenges lies in the development of an approach that, while deploying the computational power of AI, still respects and captures human judgment [21], especially in management [22].”
BackgroundUnderstanding Choice Independence and Error Types in Human-AI Collaboration
Erlei, A., Sharma, A., & Gadiraju, U. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641946 · Cited by 38
“People fail to adequately judge their own performance level in relation to the task’s difculty, complexity [92], or uncertainty [93], and therefore do not implement efective delegation rules.”
BackgroundSelf-citationHe, G., Demartini, G., & Gadiraju, U. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713218 · Cited by 23
“In our study, all tasks are executed in a simulation setup, which has been a popular method for orchestrating meaningful human-centered AI studies [16, 86].”
Self-citation
Metadata
- Authors
- Salimzadeh, S., He, G., & Gadiraju, U.
- Affiliations
- Delft University of Technology (NL)
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-17
- OpenAlex
- W4396832356
- Semantic Scholar
- 360a8fa00bad3d0c054d7c38ccfa1ff910e5059c
- Cited by
- 41
- References
- 147
- Citations by year
- 2026: 12 · 2025: 26 · 2024: 3
- FWCI
- 27.54
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- Research Repository (Delft University of Technology) (submittedVersion, cc-by)
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- human-AI decision-making, task complexity, task uncertainty, trust in AI, AI reliance, diagnostic tasks, trip planning, AI system design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making. https://app.cochat.ai/papers/10.1145/3613904.3641905
- The paper
- Salimzadeh, S., He, G., & Gadiraju, U. (2024). Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3641905
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.