The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index; 10 of the 12 sources its claims cite were checked for support.
Emerging Reliance Behaviors in Human-AI Content Grounded Data Generation: The Role of Cognitive Forcing Functions and Hallucinations
Open accessAbstract
We investigate the impact of hallucinations and Cognitive Forcing Functions in human-AI collaborative content grounded data generation, focusing on the use of Large Language Models (LLMs) to assist in generating high-quality conversational data. Through a study with 34 users who each completed 8 tasks (n=272), we found that hallucinations significantly reduce data quality. While Cognitive Forcing Functions do not always alleviate these effects, their presence influences how users integrate AI responses. Specifically, we observed emerging reliance behaviors, with users often appending AI-generated responses to their correct answers, even when the AI’s suggestions conflicted. This points to a potential drawback of Cognitive Forcing Functions, particularly when AI suggestions are inaccurate. Users who overrelied on AI-generated text produced lower-quality data, emphasizing the nuanced dynamics of overreliance in human-LLM collaboration compared to traditional human-AI decision-making.
Overview
CoChat Summary
TL;DR
In a 34-participant study (8 tasks each, n=272) of LLM-assisted content-grounded dialogue creation, hallucinated AI responses significantly reduced data quality and significantly reduced users' incorporation of AI text, while cognitive forcing functions (CFFs) did not mitigate either effect. The authors identify new reliance behaviors in co-creation, including users appending hallucinated AI text to their own already-correct answers.
Key findings
- Hallucinations significantly lowered data quality (F(1, 229) = 73.07, p < .001): quality was M = 4.34 (SE = 0.137) with hallucinations present vs. M = 5.77 (SE = 0.137) absent, mean difference −1.43 (t(229) = −8.548, p < .0001). H1 accepted.
- Hallucinations significantly decreased AI use (F(1, 229) = 463.519, p < .001): AI usage M = 0.133 (SE = 0.0299) with hallucinations vs. M = 0.916 (SE = 0.0299) without (t(229) = −21.529, p < .0001). H3 accepted.
- CFFs did not mitigate hallucination effects: no significant main effect of CFF type (F(2, 31) = 0.19, p = .83) or CFF presence (F(1, 229) = 3.12, p = .079), and no significant CFF × CFF presence × hallucination interaction on quality (F(2, 229) = 1.870, p = .15); H2A–H2C and H4A–H4C rejected. A significant CFF Presence × CFF interaction on AI use (F(2, 229) = 4.944, p = 0.008) showed more AI use in Formulate when the CFF was absent (M = 0.614, SE = 0.0523) than in Highlight (M = 0.423, SE = 0.0481), t(74.5) = 2.684, p = 0.0907.
- Reliance classification: 13 underreliers, 9 appropriate reliers, 6 both over- and underreliers, 6 solely overreliers. Reliance type significantly affected summed data quality (F(3, 268) = 110.05, p < 0.0001): appropriate reliers (M = 43.33) and underreliers (M = 42.77) outperformed overreliers (M = 36) and overreliers-underreliers (M = 34.33) (p < 0.01 and p < 0.05 respectively). Five hallucination-reliance patterns were coded: exact copy, copy and minor edit, exact copy and append with correct response, copy and removal of information, copy and added content from the reference document.
Methods
- Mixed between-within web experiment: 34 participants recruited via internal Slack channels and word of mouth, each completing 8 HR/customer-support tasks (n = 272). Between-subjects variable was CFF Type (Formulate, Read-First, Highlight); within-subjects variables were CFF presence (visible or not) and hallucination presence. Four tasks contained hallucinated AI responses and four did not; task order was randomized.
- Tasks were drawn from real HR support documents (8 unique reference documents, M = 555.25 words, SD = 274.41) with prior annotator-created conversations; final turns used AI responses from Flan-T5-XXL and a fine-tuned content-grounded variant. Gold responses were produced independently by three authors and reconciled.
- Data quality was rated with a 6-item yes/no rubric spanning faithfulness, accuracy, and completeness (adapted from Grice's maxims), plus one item on whether any AI-generated text appeared in the final response. Analysis used linear mixed-effects models with random intercepts per participant and F-tests with Satterthwaite approximations; reliance-type effects were tested with ANOVA and Tukey post-hoc comparisons.
Limitations
- Stated: the focus on conversational customer support may limit applicability to other domains; gold-response creation may not be scalable; the participant pool was restricted to internal company members due to HR document confidentiality; the small sample size limits generalizability, and future work should test larger, more diverse pools.
- Stated: only ~25% of participants reported being from HR, though participants on average reported high confidence in composing high-quality data.
- Evident from the text: reliance was measured as a binary indicator of whether any AI text appeared in the final response, and the authors note that logged copy-button events were not a reliable proxy for AI text usage, requiring manual rubric-based evaluation that they describe as neither scalable nor efficient.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 0 Not valid · 2 Not checked · 60 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“In the condition where users had to compose their own response before seeing the AI-generated response (Formulate), which was adapted from the decision making context in [7], we found that users relied on the AI more when they were not presented with the CFF.”
§ 7 Discussion > 7.1 Unintended Consequences of Cognitive Forcing Functions on AI Use
What the sources say
- Supportedcites [7] [7] Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–21. Google Scholar a [...] ] on overreliance b [...] counterparts across various domains c [...] versus not following incorrect AI advice d [...] and when to access the AI recommendation e [...] most effective at reducing overreliance f [...] insights distinct from existing studies g [...] outcome in human-AI interaction h [...] to various qualit source
The citation supports the Formulate condition’s design, not the citing authors’ finding about reliance on AI. In §3.2.2 (page unavailable), the source describes an Update condition in which participants made an initial decision before seeing the AI suggestion and could then revise it. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, to define underreliance, we calculated the median number of times users ignored AI-generated text when the response was non-hallucinated (median = 0, range = 0 to 4).”
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“with users often appending AI-generated responses to their correct answers, even when the AI’s suggestions conflicted.”
- Also stated in: § 1 Introduction · plain
“users append AI responses to their already correct responses, even when the information is conflicting”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further Tukey post-hoc analysis revealed several significant pairwise comparisons: Appropriate reliers (M=43.33) had higher quality scores than overreliers (M = 36) p < 0.01, and overreliers-underreliers (M=34.33) p< 0.01.”
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our study, we observed cases where participants did not use the copy button—an action we logged—but still incorporated portions of AI-generated text into their final responses.”
§ 7 Discussion > 7.2 New Patterns of AI Reliance in Co-creation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
31 more claims
Create a free account to see all 70 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 12
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Not confirmedF(2, 229) = 1.87reported p = .15 · recomputed p = .156
“F(2, 229) = 1.870, p = .15”
- ConsistentF(1, 229) = 73.07reported p < .001 · recomputed p < .001
“F(1, 229) = 73.07, p < .001”
- Consistentt(229) = -8.548reported p < .0001 · recomputed p < .001
“t(229) = -8.548, p < .0001”
- ConsistentF(2, 31) = 0.19reported p = .83 · recomputed p = .828
“F(2, 31) = 0.19, p = .83”
- ConsistentF(1, 229) = 3.12reported p = .079 · recomputed p = .079
“F(1, 229) = 3.12, p = .079”
6 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
“We conducted an ANOVA to examine the effects of reliance type, CFF, and their interaction on the sum of data quality for all tasks completed by a user. The results revealed a statistically significant effect of reliance type on data quality F(3,268)=110.05, p<0.0001.”
§ 5 Methodology > 5.1 Participants
“We recruited 34 participants via multiple slack channels and word of mouth.”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
“when hallucinations were absent (M = 0.916, SE = 0.0299).”
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
“Across our participants, 13 were classified as underreliers, 9 as appropriate reliers, 6 as both overreliers and underreliers, and 6 solely as overreliers.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5 Methodology > 5.2 Procedure
“Participants were randomly assigned to one of the four Cognitive Forcing Function conditions: Formulate, Read-First, or Highlight.”
§ 5 Methodology > 5.2 Procedure
“one between-subject variables: CFF Type (Formulate, Read-First, Highlight)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
“Similarly, to define underreliance, we calculated the median number of times users ignored AI-generated text when the response was non-hallucinated (median = 0, range = 0 to 4).”
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
“Across our participants, 13 were classified as underreliers, 9 as appropriate reliers, 6 as both overreliers and underreliers, and 6 solely as overreliers.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
“Appropriate Relier: Used non-hallucinated AI response at least once.”
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
“Never used non-hallucinated response.”
1 more finding
Create a free account to see every finding of this check.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“For use of LLMs in more specialized domains [20] and language tasks [14], fine-tuning pre-trained language models has been the dominant methodology.”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2004.10964 · Cited by 3,348
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 2×
“Overreliance has been extensively studied in critical industries, including medicine [18] and aviation [44].”
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 1×
“However, there is limited evidence to suggest that transparency improves trust, whether through the use of interpretable models or by allowing users to inspect model behavior, providing explanations, or reducing the number of features presented [9, 31, 39].”
Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions
Dale, R., & Reiter, E. (1995) · Cognitive Science · doi:10.1207/s15516709cog1902_3 · Cited by 721 · cited 1×
“These areas were influenced and adapted from Grice’s maxims which stress clarity (manner), relevance (relation), truthfulness (quality), and informativeness (quantity) [12, 25] and were reduced to yes/no questions to prevent survey fatigue and improve consistency of evaluations.”
Reactance to Recommendations: When Unsolicited Advice Yields Contrary Responses
Fitzsimons, G. J., & Lehmann, D. R. (2004) · Marketing Science · doi:10.1287/mksc.1030.0033 · Cited by 604
Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing
Lapata, M., & Ng, H. T. (2008) · Cited by 567
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · cited 1×
“The study found that there was often a mismatch between users mental models of the tool’s abilities and the tool’s actual capabilities [30].”
Recommender Systems in the Era of Large Language Models (LLMs)
Zhao, Z., Fan, W., Li, J., Liu, Y., Mei, X., Wang, Y., Wen, Z., Wang, F., Zhao, X., Tang, J., et al. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3392335 · Cited by 481
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
Nested by design: model fitting and interpretation in a mixed model era
Schielzeth, H., & Nakagawa, S. (2013) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2012.00251.x · Cited by 394 · cited 1×
“Mixed-effects models are particularly useful for handling hierarchical data structures [42] and incorporating random effects [4], making them well-suited for our study, where each participant completed multiple tasks.”
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 2×
“Despite the remarkable accuracy of AI, human-AI teams sometimes fall short of expectations [8], particularly in areas where AI-alone teams outperform human-alone counterparts across various domains [6, 7, 19, 24, 55].”
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356
BARTScore: Evaluating Generated Text as Text Generation
Yuan, W., Neubig, G., & Liu, P. (2021) · arXiv · doi:10.48550/arxiv.2106.11520 · Cited by 353
Human heuristics for AI-generated language are flawed
Jakesch, M., Hancock, J. T., & Naaman, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208839120 · Cited by 342
Linear and generalized linear mixed models
Bolker, B. M. (2015) · Ecological Statistics · doi:10.1093/acprof:oso/9780199672547.003.0014 · Cited by 339 · cited 1×
“Participants skewed to rating the AI generated responses highly for accuracy (median=4), completeness (median=4), and usefulness (median=5).”
Wang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320
Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292
Zhou, J., Zhang, Y., Luo, Q., Parker, A. G., & De Choudhury, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581318 · Cited by 288
Co-Writing with Opinionated Language Models Affects Users’ Views
Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581196 · Cited by 265
Jacobs, M., Pradier, M. F., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01224-x · Cited by 247 · cited 1×
“Despite the remarkable accuracy of AI, human-AI teams sometimes fall short of expectations [8], particularly in areas where AI-alone teams outperform human-alone counterparts across various domains [6, 7, 19, 24, 55].”
A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
De-Arteaga, M., Fogliato, R., & Chouldechova, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376638 · Cited by 221
Dual-process cognitive interventions to enhance diagnostic reasoning: a systematic review
Lambe, K. A., O'Reilly, G., Kelly, B. D., & Curristan, S. (2016) · BMJ Quality & Safety · doi:10.1136/bmjqs-2015-004417 · Cited by 215 · cited 1×
“We rely on prior work in the AI-decision making that explored the role of Cognitive Forcing Functions, interventions applied at the decision making point that disrupt routine thinking prompting analytical reasoning [32] on overreliance [7].”
The Fact Extraction and VERification (FEVER) Shared Task
Thorne, J., Vlachos, A., Cocarascu, O., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the First Workshop on Fact Extraction and VERification (FEVER) · doi:10.18653/v1/w18-5501 · Cited by 212
Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings
Shing, H.-C., Nair, S., Zirikly, A., Friedenberg, M., Daumé III, H., & Resnik, P. (2018) · Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic · doi:10.18653/v1/w18-0603 · Cited by 208 · cited 1×
“Prior work has recruited subject matter experts for labeling data to yield more reliable data and has demonstrated that in some contexts, SME annotated data is more reliable than crowsdsourced data [43, 45].”
Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584066 · Cited by 208
How do visual explanations foster end users' appropriate trust in machine learning?
Yang, F., Huang, Z., Scholtz, J., & Arendt, D. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377480 · Cited by 200
Are chatbots really useful for human resource management?
Majumder, S., & Mondal, A. (2021) · International Journal of Speech Technology · doi:10.1007/s10772-021-09834-y · Cited by 189
Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI
Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., & Kaplan, L. (2020) · Patterns · doi:10.1016/j.patter.2020.100049 · Cited by 186 · cited 1×
“Extensive work has been done on achieving calibrated trust in AI systems [48, 55].”
Let Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems
Kunkel, J., Donkers, T., Michael, L., Barbu, C.-M., & Ziegler, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300717 · Cited by 156
Inkpen, K., Chappidi, S., Mallari, K., Nushi, B., Ramesh, D., Michelucci, P., Mandava, V., Vepřek, L. H., & Quinn, G. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3534561 · Cited by 132
Automation Bias and Errors: Are Crews Better Than Individuals?
Skitka, L. J., Mosier, K. L., Burdick, M., & Rosenblatt, B. (2000) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap1001_5 · Cited by 127 · cited 1×
“Overreliance has been extensively studied in critical industries, including medicine [18] and aviation [44].”
The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123
Does Explainable Artificial Intelligence Improve Human Decision-Making?
Alufaisan, Y., Marusich, L. R., Bakdash, J. Z., Zhou, Y., & Kantarcioglu, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16819 · Cited by 106
The imitation game: Detecting human and AI-generated texts in the era of ChatGPT and BARD
Hayawi, K., Shahriar, S., & Mathew, S. S. (2024) · Journal of Information Science · doi:10.1177/01655515241227531 · Cited by 84
Designing Ground Truth and the Social Life of Labels
Muller, M., Wolf, C. T., Andres, J., Desmond, M., Joshi, N. N., Ashktorab, Z., Sharma, A., Brimijoin, K., Pan, Q., Duesterwald, E., et al. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445402 · Cited by 83
Self-citationPrabhudesai, S., Yang, L., Asthana, S., Huan, X., Liao, Q. V., & Banovic, N. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584033 · Cited by 82
A Slow Algorithm Improves Users' Assessments of the Algorithm's Accuracy
Park, J. S., Barber, R., Kirlik, A., & Karahalios, K. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359204 · Cited by 65 · cited 1×
“Delaying the presentation of AI-generated suggestions is based on previous research suggesting that postponing the AI’s recommendation can potentially enhance decision outcomes [38].”
Does Explainable Artificial Intelligence Improve Human Decision-Making?Preprint
Alufaisan, Y., Marusich, L. R., Bakdash, J. Z., Zhou, Y., & Kantarcıoğlu, M. (2020) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/d4r9t · Cited by 46
AI-Assisted Human Labeling: Batching for Efficiency without Overreliance
Ashktorab, Z., Desmond, M., Andres, J., Muller, M., Joshi, N. N., Brachman, M., Sharma, A., Brimijoin, K., Pan, Q., Wolf, C. T., et al. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449163 · Cited by 45
Self-citationLeiser, F., Eckhardt, S., Knaeble, M., Maedche, A., Schwabe, G., & Sunyaev, A. (2023) · Mensch und Computer 2023 · doi:10.1145/3603555.3603565 · Cited by 44 · cited 1×
“Thenegativeimpactofhallucinationsondata quality iswell-documented [33] —incorrect AI recommendations have been shown to harm users in decision-making contexts [13].”
Analyzing and Evaluating Faithfulness in Dialogue Summarization
Wang, B., Zhang, C., Zhang, Y., Chen, Y., & Li, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.325 · Cited by 40
The Impact of the Gricean Maxims of Quality, Quantity and Manner in Chatbots
Jacquet, B., Hullin, A., Baratgin, J., & Jamet, F. (2019) · International Conference on Information and Digital Technologies (IDT) · doi:10.1109/dt.2019.8813473 · Cited by 35 · cited 1×
“These areas were influenced and adapted from Grice’s maxims which stress clarity (manner), relevance (relation), truthfulness (quality), and informativeness (quantity) [12, 25] and were reduced to yes/no questions to prevent survey fatigue and improve consistency of evaluations.”
An Empirical Evaluation of Predicted Outcomes as Explanations in Human-AI Decision-Making
Jakubik, J., Schoeffer, J., Hoge, V., Vössing, M., & Kühl, N. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-23618-1_24 · Cited by 24
A Test for Evaluating Performance in Human-Computer Systems
Campero, A., Vaccaro, M., Song, J., Wen, H., Almaatouq, A., & Malone, T. W. (2022) · arXiv · doi:10.48550/arxiv.2206.12390 · Cited by 21
Analyzing and Evaluating Faithfulness in Dialogue Summarization
Wang, B., Zhang, C., Zhang, Y., Chen, Y., & Li, H.-Z. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.11777 · Cited by 21
How Do Visual Explanations Foster End Users' Appropriate Trust in Machine Learning?Preprint
Yang, F. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/s9q8t · Cited by 18
An Empirical Evaluation of Predicted Outcomes as Explanations in Human-AI Decision-Making
Jakubik, J., Schoffer, J., Hoge, V., Vossing, M., & Kühl, N. (2022) · PKDD/ECML Workshops · doi:10.48550/arxiv.2208.04181 · Cited by 16
Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i09.7115 · Cited by 15
Hayward, S. (2006) · Strategic HR Review · doi:10.1108/14754390680000861 · Cited by 2 · cited 1×
“Em-ployees often approach HR with questions about various benefits and services, including healthcare similar to how customers might approach customer services about products or services [22, 35, 50].”
Go/No Go Criteria in Formative E-Rubrics
Pedro Company, Otey, J., Torres, M. J. A., Contero, M., & Camba, J. D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-91152-6_20 · Cited by 2 · cited 1×
“Our choice of a three-way rating system ("yes", "no", "I don’t know") both avoids forcing raters into a binary decision when uncertain and reduces complexity of likert scale [11].”
AI interventions as hypothesis cues: rethinking overreliance in human-AI decision-making
Pham, M. P., Illingworth-Vera, D. A., & Thomas, R. P. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1967356
Defamation in the Age of Artificial Intelligence
Tenzer, L. Y. G. (2023) · doi:10.2139/ssrn.4545070
Research works printed but not linked: 3
- Tim Fischer, Steffen Remus, and Chris Biemann. 2022. Measuring Faithfulness of Abstractive Summaries. In Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022), Robin Schaefer, Xiaoyu Bai, Manfred Stede, and Torsten Zesch (Eds.). KONVENS 2022 Organizers, Potsdam, Germany, 63–73. https://aclanthology.org/2022.konvens-1.8/
- Daniel Kahneman. 2011. Thinking, Fast and Slow. Macmillan, New York, NY.
- Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013/
Cited by
Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making
Chappidi, S., Singh, J., & Krauze, A. V. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791428 · Cited by 8
“By encouraging or supporting implicit reasoning, this arche-type seeks to avoid issues of overreliance or automation bias [15, 25].”
InfluentialUnderstanding Digital Religion in the Lives of Black Christian Young Adults
Smith, A. N., Otoo, N., Beach, R., Eaves, J., Jennings, K., Lyons, G. W., Thompson, D., McCall, T., O’Leary, T. K., & Ogbonnaya-Ogburu, I. F. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790509 · Cited by 6
“These minor inconveniences may seem like isolated errors; however, past research indicates that generative AI has the tendency to hallucinate, or present false information as a factual [12].”
Balancing utopian and dystopian paradigms: trust, values, and AI literacy in generative AI adoption
Kim, D., Bak, S., Kim, S., Lee, B. H., Lee, Y., & Park, S. (2026) · Asian Journal of Communication · doi:10.1080/01292986.2026.2639672 · Cited by 2
“While utopian perspectives emphasize gains in efficiency and enhanced service delivery (Ameen et al., 2021; Radonjić et al., 2024), the rapid proliferation of GAI raises critical concerns regarding algorithmic bias and reliability (Ashktorab et al., 2024).”
Guided Reflection in AI-Assisted Decision-Making: Effects on AI Overreliance and Decision Accuracy
Li, S., Li, J., Li, H., Zhu, H., & Li, X. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790632 · Cited by 8
Zhang, Z. T., von Felten, N., Reicherts, L., Tankelevitch, L., Guan, Z., Rintel, S., Fu, Y., He, J., Holstein, K., Sarkar, A., et al. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3778733
“For example, Ashktorab et al. [2] found that users had negative impressions of a cognitive forcing function that created more work for them.”
Metadata
- Authors
- Ashktorab, Z., Desmond, M., Pan, Q., Johnson, J., Brachman, M., Dugan, C., Danilevsky, M., & Geyer, W.
- Affiliations
- IBM (United States) (US)
- IBM Research - Cambridge (United States) (US)
- RAND Corporation (US)
- IBM Research - Almaden (US)
- ORCID
- Venue
- Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work
- Publisher
- ACM
- Published
- 2025-06-22
- Pages
- 1-17
- OpenAlex
- W4411505989
- Semantic Scholar
- a23bed4f695eb558c50c77eb46f72c07b1e7eda7
- Cited by
- 10
- References
- 57
- Citations by year
- 2026: 10
- FWCI
- 10.33
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Annual Symposium on Human-Computer Interaction for Work (CHIWORK) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- human-AI collaboration, hallucinations, human-AI interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.