Emerging Reliance Behaviors in Human-AI Content Grounded Data Generation: The Role of Cognitive Forcing Functions and Hallucinations
Open accessThe DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index; 10 of the 12 sources its claims cite were checked for support.
Abstract
We investigate the impact of hallucinations and Cognitive Forcing Functions in human-AI collaborative content grounded data generation, focusing on the use of Large Language Models (LLMs) to assist in generating high-quality conversational data. Through a study with 34 users who each completed 8 tasks (n=272), we found that hallucinations significantly reduce data quality. While Cognitive Forcing Functions do not always alleviate these effects, their presence influences how users integrate AI responses. Specifically, we observed emerging reliance behaviors, with users often appending AI-generated responses to their correct answers, even when the AI’s suggestions conflicted. This points to a potential drawback of Cognitive Forcing Functions, particularly when AI suggestions are inaccurate. Users who overrelied on AI-generated text produced lower-quality data, emphasizing the nuanced dynamics of overreliance in human-LLM collaboration compared to traditional human-AI decision-making.
Overview
CoChat Summary
TL;DR
In a 34-participant study (8 tasks each, n=272) of LLM-assisted content-grounded dialogue creation, hallucinated AI responses significantly reduced data quality and significantly reduced users' incorporation of AI text, while cognitive forcing functions (CFFs) did not mitigate either effect. The authors identify new reliance behaviors in co-creation, including users appending hallucinated AI text to their own already-correct answers.
Key findings
- Hallucinations significantly lowered data quality (F(1, 229) = 73.07, p < .001): quality was M = 4.34 (SE = 0.137) with hallucinations present vs. M = 5.77 (SE = 0.137) absent, mean difference −1.43 (t(229) = −8.548, p < .0001). H1 accepted.
- Hallucinations significantly decreased AI use (F(1, 229) = 463.519, p < .001): AI usage M = 0.133 (SE = 0.0299) with hallucinations vs. M = 0.916 (SE = 0.0299) without (t(229) = −21.529, p < .0001). H3 accepted.
- CFFs did not mitigate hallucination effects: no significant main effect of CFF type (F(2, 31) = 0.19, p = .83) or CFF presence (F(1, 229) = 3.12, p = .079), and no significant CFF × CFF presence × hallucination interaction on quality (F(2, 229) = 1.870, p = .15); H2A–H2C and H4A–H4C rejected. A significant CFF Presence × CFF interaction on AI use (F(2, 229) = 4.944, p = 0.008) showed more AI use in Formulate when the CFF was absent (M = 0.614, SE = 0.0523) than in Highlight (M = 0.423, SE = 0.0481), t(74.5) = 2.684, p = 0.0907.
- Reliance classification: 13 underreliers, 9 appropriate reliers, 6 both over- and underreliers, 6 solely overreliers. Reliance type significantly affected summed data quality (F(3, 268) = 110.05, p < 0.0001): appropriate reliers (M = 43.33) and underreliers (M = 42.77) outperformed overreliers (M = 36) and overreliers-underreliers (M = 34.33) (p < 0.01 and p < 0.05 respectively). Five hallucination-reliance patterns were coded: exact copy, copy and minor edit, exact copy and append with correct response, copy and removal of information, copy and added content from the reference document.
Methods
- Mixed between-within web experiment: 34 participants recruited via internal Slack channels and word of mouth, each completing 8 HR/customer-support tasks (n = 272). Between-subjects variable was CFF Type (Formulate, Read-First, Highlight); within-subjects variables were CFF presence (visible or not) and hallucination presence. Four tasks contained hallucinated AI responses and four did not; task order was randomized.
- Tasks were drawn from real HR support documents (8 unique reference documents, M = 555.25 words, SD = 274.41) with prior annotator-created conversations; final turns used AI responses from Flan-T5-XXL and a fine-tuned content-grounded variant. Gold responses were produced independently by three authors and reconciled.
- Data quality was rated with a 6-item yes/no rubric spanning faithfulness, accuracy, and completeness (adapted from Grice's maxims), plus one item on whether any AI-generated text appeared in the final response. Analysis used linear mixed-effects models with random intercepts per participant and F-tests with Satterthwaite approximations; reliance-type effects were tested with ANOVA and Tukey post-hoc comparisons.
Limitations
- Stated: the focus on conversational customer support may limit applicability to other domains; gold-response creation may not be scalable; the participant pool was restricted to internal company members due to HR document confidentiality; the small sample size limits generalizability, and future work should test larger, more diverse pools.
- Stated: only ~25% of participants reported being from HR, though participants on average reported high confidence in composing high-quality data.
- Evident from the text: reliance was measured as a binary indicator of whether any AI text appeared in the final response, and the authors note that logged copy-button events were not a reliable proxy for AI text usage, requiring manual rubric-based evaluation that they describe as neither scalable nor efficient.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 0 Not valid · 2 Not checked · 60 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“In the condition where users had to compose their own response before seeing the AI-generated response (Formulate), which was adapted from the decision making context in [7], we found that users relied on the AI more when they were not presented with the CFF.”
§ 7 Discussion > 7.1 Unintended Consequences of Cognitive Forcing Functions on AI Use
What the sources say
- Supportedcites [7] [7] Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–21. Google Scholar a [...] ] on overreliance b [...] counterparts across various domains c [...] versus not following incorrect AI advice d [...] and when to access the AI recommendation e [...] most effective at reducing overreliance f [...] insights distinct from existing studies g [...] outcome in human-AI interaction h [...] to various qualit source
The citation supports the Formulate condition’s design, not the citing authors’ finding about reliance on AI. In §3.2.2 (page unavailable), the source describes an Update condition in which participants made an initial decision before seeing the AI suggestion and could then revise it. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, to define underreliance, we calculated the median number of times users ignored AI-generated text when the response was non-hallucinated (median = 0, range = 0 to 4).”
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“with users often appending AI-generated responses to their correct answers, even when the AI’s suggestions conflicted.”
- Also stated in: § 1 Introduction · plain
“users append AI responses to their already correct responses, even when the information is conflicting”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further Tukey post-hoc analysis revealed several significant pairwise comparisons: Appropriate reliers (M=43.33) had higher quality scores than overreliers (M = 36) p < 0.01, and overreliers-underreliers (M=34.33) p< 0.01.”
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our study, we observed cases where participants did not use the copy button—an action we logged—but still incorporated portions of AI-generated text into their final responses.”
§ 7 Discussion > 7.2 New Patterns of AI Reliance in Co-creation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
31 more claims
Create a free account to see all 70 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 12
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Not confirmedF(2, 229) = 1.87reported p = .15 · recomputed p = .156
“F(2, 229) = 1.870, p = .15”
- ConsistentF(1, 229) = 73.07reported p < .001 · recomputed p < .001
“F(1, 229) = 73.07, p < .001”
- Consistentt(229) = -8.548reported p < .0001 · recomputed p < .001
“t(229) = -8.548, p < .0001”
- ConsistentF(2, 31) = 0.19reported p = .83 · recomputed p = .828
“F(2, 31) = 0.19, p = .83”
- ConsistentF(1, 229) = 3.12reported p = .079 · recomputed p = .079
“F(1, 229) = 3.12, p = .079”
6 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
“We conducted an ANOVA to examine the effects of reliance type, CFF, and their interaction on the sum of data quality for all tasks completed by a user. The results revealed a statistically significant effect of reliance type on data quality F(3,268)=110.05, p<0.0001.”
§ 5 Methodology > 5.1 Participants
“We recruited 34 participants via multiple slack channels and word of mouth.”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
“when hallucinations were absent (M = 0.916, SE = 0.0299).”
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
“Across our participants, 13 were classified as underreliers, 9 as appropriate reliers, 6 as both overreliers and underreliers, and 6 solely as overreliers.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5 Methodology > 5.2 Procedure
“Participants were randomly assigned to one of the four Cognitive Forcing Function conditions: Formulate, Read-First, or Highlight.”
§ 5 Methodology > 5.2 Procedure
“one between-subject variables: CFF Type (Formulate, Read-First, Highlight)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
“Similarly, to define underreliance, we calculated the median number of times users ignored AI-generated text when the response was non-hallucinated (median = 0, range = 0 to 4).”
§ 6 Results > 6.3 Reliance in Human-AI Collaborative Content Grounded Data Generation (RQ2)
“Across our participants, 13 were classified as underreliers, 9 as appropriate reliers, 6 as both overreliers and underreliers, and 6 solely as overreliers.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5 Methodology > 5.1 Participants
“All questions were asked on a 7-point likert scale.”
§ 6 Results > 6.1 Participant Demographics and Self Reported Measures
“Participants in the study rated their exposure to Artificial Intelligence on a Likert scale from 1 (“I have never heard of AI”) to 5 (“I have extensive experience in AI research and/or development.”)”
1 more finding
Create a free account to see every finding of this check.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“For use of LLMs in more specialized domains [20] and language tasks [14], fine-tuning pre-trained language models has been the dominant methodology.”
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2004.10964 · Cited by 3,348
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 2×
“Overreliance has been extensively studied in critical industries, including medicine [18] and aviation [44].”
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Research works printed but not linked: 3
- Tim Fischer, Steffen Remus, and Chris Biemann. 2022. Measuring Faithfulness of Abstractive Summaries. In Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022), Robin Schaefer, Xiaoyu Bai, Manfred Stede, and Torsten Zesch (Eds.). KONVENS 2022 Organizers, Potsdam, Germany, 63–73. https://aclanthology.org/2022.konvens-1.8/
- Daniel Kahneman. 2011. Thinking, Fast and Slow. Macmillan, New York, NY.
- Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013/
Cited by
Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making
Chappidi, S., Singh, J., & Krauze, A. V. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791428 · Cited by 8
“By encouraging or supporting implicit reasoning, this arche-type seeks to avoid issues of overreliance or automation bias [15, 25].”
InfluentialUnderstanding Digital Religion in the Lives of Black Christian Young Adults
Smith, A. N., Otoo, N., Beach, R., Eaves, J., Jennings, K., Lyons, G. W., Thompson, D., McCall, T., O’Leary, T. K., & Ogbonnaya-Ogburu, I. F. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790509 · Cited by 6
“These minor inconveniences may seem like isolated errors; however, past research indicates that generative AI has the tendency to hallucinate, or present false information as a factual [12].”
Balancing utopian and dystopian paradigms: trust, values, and AI literacy in generative AI adoption
Kim, D., Bak, S., Kim, S., Lee, B. H., Lee, Y., & Park, S. (2026) · Asian Journal of Communication · doi:10.1080/01292986.2026.2639672 · Cited by 2
“While utopian perspectives emphasize gains in efficiency and enhanced service delivery (Ameen et al., 2021; Radonjić et al., 2024), the rapid proliferation of GAI raises critical concerns regarding algorithmic bias and reliability (Ashktorab et al., 2024).”
Guided Reflection in AI-Assisted Decision-Making: Effects on AI Overreliance and Decision Accuracy
Li, S., Li, J., Li, H., Zhu, H., & Li, X. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790632 · Cited by 8
Zhang, Z. T., von Felten, N., Reicherts, L., Tankelevitch, L., Guan, Z., Rintel, S., Fu, Y., He, J., Holstein, K., Sarkar, A., et al. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3778733
“For example, Ashktorab et al. [2] found that users had negative impressions of a cognitive forcing function that created more work for them.”
Metadata
- Authors
- Ashktorab, Z., Desmond, M., Pan, Q., Johnson, J., Brachman, M., Dugan, C., Danilevsky, M., & Geyer, W.
- Affiliations
- IBM (United States) (US)
- IBM Research - Cambridge (United States) (US)
- RAND Corporation (US)
- IBM Research - Almaden (US)
- ORCID
- Venue
- Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work
- Publisher
- ACM
- Published
- 2025-06-22
- Pages
- 1-17
- OpenAlex
- W4411505989
- Semantic Scholar
- a23bed4f695eb558c50c77eb46f72c07b1e7eda7
- Cited by
- 10
- References
- 57
- Citations by year
- 2026: 10
- FWCI
- 10.33
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Annual Symposium on Human-Computer Interaction for Work (CHIWORK) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- human-AI collaboration, hallucinations, human-AI interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Emerging Reliance Behaviors in Human-AI Content Grounded Data Generation: The Role of Cognitive Forcing Functions and Hallucinations. https://app.cochat.ai/papers/10.1145/3729176.3729179
- The paper
- Ashktorab, Z., Desmond, M., Pan, Q., Johnson, J., Brachman, M., Dugan, C., Danilevsky, M., & Geyer, W. (2025). Emerging Reliance Behaviors in Human-AI Content Grounded Data Generation: The Role of Cognitive Forcing Functions and Hallucinations. Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work. https://doi.org/10.1145/3729176.3729179
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.