The Trust Recovery Journey. The Effect of Timing of Errors on the Willingness to Follow AI Advice.
Open accessThe DOI matches its Crossref record; 53 of the 54 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Complementing human decision-making with AI advice offers substantial advantages. However, humans do not always trust AI advice appropriately and are overly sensitive to incidental AI errors, even in cases with overall good performance. Today’s research still needs to uncover the underlying aspects of trust decline and recovery over time in repeated human-AI interactions. Our work investigates the consequences of incidental AI error on (self-reported) trust and participants’ reliance on AI advice. Results from our experiment, where 208 participants evaluated 14 legal cases before and after receiving algorithmic advice, showed that trust significantly decreased after early and late errors but was rapidly restored in both scenarios. Reliance significantly dropped only for early errors but not for late errors. In both scenarios, reliance was able to be restored. Results suggest that late (compared to early) errors are less drastic in trust loss and allow quicker recovery. These findings align with an interpretation in which humans can build up trust over time if a system is performing well, making them more tolerant of incidental AI errors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Schematic reliance (A, left) and trust (B, right) trajectories based on our three experimental conditions, Baseline, Early Error, and Late Error. The vertical thick lines represent the early (trial 4) and late (trial 11) errors. The trajectories represent assumed progressions of reliance and trust based on previous research, for example, trust increases over time [32, 68] where reliance rather stabilizes ([32], including expected reliance and trust drops, and expected recovery. We note that we omit any realistic fluctuations by using straight lines.](https://figures.cochat.ai/paper-figures/dbdd7ae94eaf4eefa4bb3e0598498099/the-trust-recovery-journey-the-effect-of-timing-of-errors-on-the-willingness-to-figure-1.png)




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736 · cited 1×
“…the resulting manifestation of what and how a person thinks about the system’s competencies (labelled “trust”); the second is based on the behavior elicited by the AI advice and is defined by the extent to which the AI advice influences a person’s decision-making (labelled “reliance”) [26, 35, 44].”
BackgroundTrust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 2×
“…the resulting manifestation of what and how a person thinks about the system’s competencies (labelled “trust”); the second is based on the behavior elicited by the AI advice and is defined by the extent to which the AI advice influences a person’s decision-making (labelled “reliance”) [26, 35, 44].”
BackgroundResultAlgorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Prospect Theory: An Analysis of Decision Under Risk
Kahneman, D., & Tversky, A. (2013) · World Scientific Handbook in Financial Economics Series · doi:10.1142/9789814417358_0006 · Cited by 3,091
Human trust in artificial intelligence: Review of empirical research
Glikson, E., & Woolley, A. W. (2020) · Academy of Management Annals · doi:10.5465/annals.2018.0057 · Cited by 2,921 · cited 2×
“Glikson and Woolley’s [23] comprehensive review mentions several cavities and the lack of clear research directions.”
BackgroundTrust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817
Gray, H. M., Gray, K., & Wegner, D. M. (2007) · Science · doi:10.1126/science.1134475 · Cited by 2,342 · cited 1×
“Gray et al. [25] propose that AI systems can be perceived as having abilities, such as the ability to think, plan, and act autonomously.”
Background(2014) · Choice Reviews Online · doi:10.5860/choice.51-5559 · Cited by 2,342 · cited 1×
“Among other reasons, this may happen because people have (limited) capabilities for a specific (complex) task [7, 33, 46], act based on (unconsciously) biased perceptions, traits or beliefs [4, 38], lack information or access to understanding the algorithm and its recommendation [20, 47], or are…”
BackgroundNass, C., Steuer, J., & Tauber, E. R. (1994) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191703 · Cited by 2,148
Algorithm appreciation: People prefer algorithmic to human judgment
Logg, J. M., Minson, J. A., & Moore, D. A. (2019) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2018.12.005 · Cited by 2,061
Trust, control strategies and allocation of function in human-machine systems
LEE, J., & MORAY, N. (1992) · Ergonomics · doi:10.1080/00140139208967392 · Cited by 1,459 · cited 2×
“Previous work shows that trust behavior (reliance) and trust beliefs are correlated [34] but still often experienced and expressed differently: people can state to trust a system but may nevertheless not act upon it (or differently).”
BackgroundZimmermann, U. (2003) · Public Integrity · doi:10.1080/15580989.2003.11770952 · Cited by 1,146
Almost human: Anthropomorphism increases trust resilience in cognitive agents.
de Visser, E. J., Monfort, S. S., McKendrick, R., Smith, M. A. B., McKnight, P. E., Krueger, F., & Parasuraman, R. (2016) · Journal of Experimental Psychology: Applied · doi:10.1037/xap0000092 · Cited by 614
Nass, C., Steuer, J., & Tauber, E. R. (1994) · Conference companion on Human factors in computing systems - CHI '94 · doi:10.1145/259963.260288 · Cited by 579
Similarities and differences between human–human and human–automation trust: an integrative review
Madhavan, P., & Wiegmann, D. A. (2007) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220500337708 · Cited by 577
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445923 · Cited by 545
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527
Algorithms in practice: Comparing web journalism and criminal justice
Christin, A. (2017) · Big Data & Society · doi:10.1177/2053951717718855 · Cited by 379 · cited 1×
“A negative first impression may steer people away from using automated systems at all [10], which can result in detrimental consequences.”
BackgroundUnderstanding algorithm aversion: When is advice from automation discounted?
Prahl, A., & Van Swol, L. (2017) · Journal of Forecasting · doi:10.1002/for.2464 · Cited by 308 · cited 1×
“Just like people were found to value losses more than comparable gains [31], algorithmic failure seems to weigh heavier than algorithmic correctness [53] and slows down the recovery of trust [43].”
BackgroundManzey, D., Reichenbach, J., & Onnasch, L. (2012) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/1555343411433844 · Cited by 277
Impact of robot failures and feedback on real-time trust
Desai, M., Kaniarasu, P., Medvedev, M., Steinfeld, A., & Yanco, H. (2013) · 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2013.6483596 · Cited by 234 · cited 2×
“Then again, there is evidence that first impressions can be “sticky”, leading people into biased behaviour: overly optimistic first or general impressions may help promote trust in poor AI systems [14].”
BackgroundDesigning Interactive Systems Conference 2021
(2021) · doi:10.1145/3461778 · Cited by 200
Artificial Intelligence, Trust, and Perceptions of AgencyPreprint
Vanneste, B. S., & Puranam, P. (2024) · Social Science Research Network · doi:10.2139/ssrn.3897704 · Cited by 192 · cited 1×
“In the following, we discuss our hypotheses and extend our considerations with regard to reliance and trust dynamics and recovery.”
BackgroundArtificial Intelligence, Trust, and Perceptions of Agency
Vanneste, B. S., & Puranam, P. (2025) · Academy of Management Review · doi:10.5465/amr.2022.0041 · Cited by 149
A Revelation Principle for Competing Mechanisms
Epstein, L. G., & Peters, M. (1999) · Journal of Economic Theory · doi:10.1006/jeth.1999.2542 · Cited by 142
Explainable Artificial Intelligence: An Introduction to Interpretable Machine Learning
Kamath, U., & Liu, J. (2021) · doi:10.1007/978-3-030-83356-5 · Cited by 126
Nourani, M., King, J., & Ragan, E. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7469 · Cited by 116
The role of domain expertise in trusting and following explainable AI decision support systems
Bayer, S., Gimpel, H., & Markgraf, M. (2022) · Journal of Decision Systems · doi:10.1080/12460125.2021.1958505 · Cited by 103
User Trust Dynamics: An Investigation Driven by Differences in System Performance
Yu, K., Berkovsky, S., Taib, R., Conway, D., Zhou, J., & Chen, F. (2017) · Proceedings of the 22nd International Conference on Intelligent User Interfaces · doi:10.1145/3025171.3025219 · Cited by 90
User Trust in Intelligent Systems: A Journey Over Time
Holliday, D., Wilson, S., & Stumpf, S. (2016) · Proceedings of the 21st International Conference on Intelligent User Interfaces · doi:10.1145/2856767.2856811 · Cited by 83
Do You Still Trust Me? Human-Robot Trust Repair Strategies
Esterwood, C., & Robert, L. P. (2021) · 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN) · doi:10.1109/ro-man50785.2021.9515365 · Cited by 75
Calhoun, C. S., Bobko, P., Gallimore, J. J., & Lyons, J. B. (2019) · Journal of Trust Research · doi:10.1080/21515581.2019.1579730 · Cited by 69 · cited 1×
“Building trust can be hindered due to system unfamiliarity or insight: systems often lack transparency and give little to no insight into the reasoning behind decision-making processes [5, 8].”
BackgroundSecond Chance for a First Impression? Trust Development in Intelligent System Interaction
Tolmeijer, S., Gadiraju, U., Ghantasala, R., Gupta, A., & Bernstein, A. (2021) · Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3450613.3456817 · Cited by 67 · cited 1×
“Tolmeijer et al. [63] found that a good first impression works substantially better for the level of trust in the system in comparison to efforts to repair low trust.”
ResultThe influence of agent reliability on trust in human-agent collaboration
Fan, X., Oh, S., McNeese, M., Yen, J., Cuevas, H., Strater, L., & Endsley, M. R. (2008) · Proceedings of the 15th European conference on Cognitive ergonomics: the ergonomics of cool interaction · doi:10.1145/1473018.1473028 · Cited by 59
How model accuracy and explanation fidelity influence user trust
Papenmeier, A., Englebienne, G., & Seifert, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.12652 · Cited by 58
On Defining “Reliance” and “Trust”: Purposes, Conditions of Adequacy, and New Definitions
Licht, K. D. F., & Brülde, B. (2021) · Philosophia · doi:10.1007/s11406-021-00339-1 · Cited by 54
Chao, C., Chang, T.-C., Wu, H.-C., Lin, Y.-S., & Chen, P.-C. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.06.031 · Cited by 45 · cited 1×
“Among other reasons, this may happen because people have (limited) capabilities for a specific (complex) task [7, 33, 46], act based on (unconsciously) biased perceptions, traits or beliefs [4, 38], lack information or access to understanding the algorithm and its recommendation [20, 47], or are…”
BackgroundKahr, P. K., Rooks, G., Willemsen, M. C., & Snijders, C. C. P. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584058 · Cited by 39
Self-citationSnijders, C. C. P., Conijn, R., de Fouw, E., & van Berlo, K. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2097601 · Cited by 38 · cited 1×
“For example, system accuracy can be stated before or during an interaction [62].”
BackgroundSelf-citationWang, X., & Yin, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581366 · Cited by 38
Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Making
Schemmer, M., Hemmer, P., Kühl, N., Benz, C., & Satzger, G. (2022) · arXiv · doi:10.48550/arxiv.2204.06916 · Cited by 33 · cited 1×
“Schemmer et al. [58] define appropriate trust as the means to complement people’s expertise with the support of systems: "Human decision-makers should not simply rely on AI advice, but should be empowered to differentiate when to rely on AI advice and when to rely on their own [strength]".”
BackgroundA Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures
Baughan, A., Wang, X., Liu, A., Mercurio, A., Chen, J., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581152 · Cited by 32
Trust, Shared Understanding and Locus of Control in Mixed-Initiative Robotic Systems
Chiou, M., McCabe, F., Grigoriou, M., & Stolkin, R. (2021) · IEEE RO-MAN · doi:10.1109/ro-man50785.2021.9515476 · Cited by 32 · cited 1×
“Other longitudinal studies also support the notion that trust stabilizes after understanding the system sufficiently [6, 9].”
Human-Autonomy Teaming and Agent Transparency
Chen, J. Y. C., Barnes, M. J., Selkowitz, A. R., Stowers, K., Lakhmani, S. G., & Kasdaglis, N. (2016) · Companion Publication of the 21st International Conference on Intelligent User Interfaces · doi:10.1145/2876456.2879479 · Cited by 30 · cited 1×
“Building trust can be hindered due to system unfamiliarity or insight: systems often lack transparency and give little to no insight into the reasoning behind decision-making processes [5, 8].”
BackgroundAkgun, M., Cagiltay, K., & Zeyrek, D. (2010) · Journal of Pragmatics · doi:10.1016/j.pragma.2009.12.011 · Cited by 26
How Accurate Does It Feel? – Human Perception of Different Types of Classification Mistakes
Papenmeier, A., Kern, D., Hienert, D., Kammerer, Y., & Seifert, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501915 · Cited by 23
The Influence of Feedback on Automation Use, Misuse, and Disuse
Dzindolet, M. T., Pierce, L. G., Peterson, S., Purcell, L., Beck, H. P., & Beck, H. (2002) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120204600370 · Cited by 21 · cited 1×
“Dzindolet et al. [16] found that reliance and trust levels significantly decrease after errors.”
BackgroundTrust and Reliance in XAI -- Distinguishing Between Attitudinal and Behavioral Measures
Scharowski, N., Perrig, S. A. C., von Felten, N., & Brühlmann, F. (2022) · arXiv · doi:10.48550/arxiv.2203.12318 · Cited by 18
(2021) · doi:10.1145/3461778.3461997 · Cited by 14
Is Overreliance on AI Provoked by Study Design?
Zhang, Z. T., Tong, S., Liu, Y., & Butz, A. M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42286-7_3 · Cited by 11
Guiding User Choice During Discussion by Silence, Examples and Justifications
Maier, F., Zuckerman, I., & Kraus, S. (2012) · Frontiers in artificial intelligence and applications · doi:10.3233/978-1-61499-098-7-330 · Cited by 9
Visually Explaining Uncertain Price Predictions in Agrifood: A User-Centred Case-Study
Ooge, J., & Verbert, K. (2022) · Agriculture · doi:10.3390/agriculture12071024 · Cited by 6
In Groups We Trust: Lower Betrayal Aversion Toward a Group than Toward an IndividualPreprint
Lee, S. Y., Kinias, Z., & Vanneste, B. S. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3983438 · Cited by 1
Research works printed but not linked: 14
- Phoebe E Bailey, Tarren Leon, Natalie C Ebner, Ahmed A Moustafa, and Gabrielle Weidemann. 2022. A meta-analysis of the weight of advice in decision-making. Current Psychology (2022), 1–26.
- Alvaro Chacon, Edgar E Kausel, and Tomas Reyes. 2022. A longitudinal approach for understanding algorithm use. Journal of Behavioral Decision Making (2022).
- Franz Faul, Edgar Erdfelder, Axel Buchner, and Albert-Georg Lang. 2009. Statistical power analyses using G* Power 3.1: Tests for correlation and regression analyses. Behavior research methods 41, 4 (2009), 1149–1160.
- Heather M Gray, Kurt Gray, and Daniel M Wegner. 2007. Dimensions of mind perception. science 315, 5812 (2007), 619–619.
- Stephan Lewandowsky, Michael Mundy, and Gerard Tan. 2000. The dynamics of trust: comparing humans to automation.Journal of Experimental Psychology: Applied 6, 2 (2000), 104.
- Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust. In 11th australasian conference on information systems, Vol. 53. Citeseer, 6–8.
- JB Manchon, Mercedes Bueno, and Jordan Navarro. 2021. Calibration of Trust in Automated Driving: A Matter of Initial Level of Trust and Automated Driving Style?Human Factors (2021), 00187208211052804.
- Raja Parasuraman and Victor Riley. 1997. Humans and automation: Use, misuse, disuse, abuse. Human factors 39, 2 (1997), 230–253.
- Tobias M Peters and Roel W Visser. 2023. The Importance of Distrust in AI. arXiv preprint arXiv:2307.13601 (2023). · arXiv
- Victor Andrew Riley. 1994. Human use of automation. University of Minnesota.
- Daniel B Shank and Alexander Gott. 2019. People’s self-reported encounters of Perceiving Mind in Artificial Intelligence. Data in brief 25 (2019), 104220.
- Keng Siau and Weiyu Wang. 2018. Building trust in artificial intelligence, machine learning, and robotics. Cutter business technology journal 31, 2 (2018), 47–53.
- Janet A Sniezek and Timothy Buckley. 1995. Cueing and cognitive conflict in judge-advisor decision making. Organizational behavior and human decision processes 62, 2 (1995), 159–174.
- X Jessie Yang, Christopher Schemanske, and Christine Searle. 2021. Toward quantifying trust dynamics: How people adjust their trust after moment-to-moment interaction with automation. arXiv preprint arXiv:2107.07374 (2021). · arXiv
Cited by
de Jong, S., Paananen, V., Tag, B., & van Berkel, N. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710946 · Cited by 32
Personalising AI Assistance Based on Overreliance Rate in AI-Assisted Decision Making
Swaroop, S., Buçinca, Z., Gajos, K. Z., & Doshi-Velez, F. (2025) · Proceedings of the 30th International Conference on Intelligent User Interfaces · doi:10.1145/3708359.3712128 · Cited by 20
Kahr, P. K., Rooks, G., Snijders, C. C. P., & Willemsen, M. C. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713099 · Cited by 26
Self-citationTo Solicit or Not to Solicit? Impact of AI Assistance Delivery Mechanisms on Decision-Making
Nair, P., Gressel, G., Anand, M. N., & Achuthan, K. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2536617 · Cited by 17
“This is important to consider, especially in light of Kahr et al. who highlighted the role of trust recovery mechanisms in AI-assisted decision-making (Kahr et al., 2024).”
AI systems should be trustworthy, not trusted
Zanotti, G. (2025) · AI & Society · doi:10.1007/s00146-025-02728-6 · Cited by 13
Metadata
- Authors
- Kahr, P. K., Rooks, G., Snijders, C., & Willemsen, M. C.
- Affiliations
- Eindhoven University of Technology (NL)
- Venue
- Proceedings of the 29th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2024-03-18
- Pages
- 609-622
- OpenAlex
- W4393970495
- Cited by
- 21
- References
- 44
- Citations by year
- 2026: 9 · 2025: 12
- FWCI
- 14.08
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by)
- TU/e Research Portal (publishedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
- Funders
- Technische Universiteit Eindhoven
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- trust in AI, trust repair, human-AI interaction, algorithmic advice, trust loss
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.