CoChat

The Trust Recovery Journey. The Effect of Timing of Errors on the Willingness to Follow AI Advice.

Open access
Kahr, Patricia K., Rooks, Gerrit, Snijders, Chris, Willemsen, Martijn C. · 2024 · Proceedings of the 29th International Conference on Intelligent User Interfaces

The DOI matches its Crossref record; 53 of the 54 references it cites were found in a registry or an index, and 1 are not checked yet.

Abstract

Complementing human decision-making with AI advice offers substantial advantages. However, humans do not always trust AI advice appropriately and are overly sensitive to incidental AI errors, even in cases with overall good performance. Today’s research still needs to uncover the underlying aspects of trust decline and recovery over time in repeated human-AI interactions. Our work investigates the consequences of incidental AI error on (self-reported) trust and participants’ reliance on AI advice. Results from our experiment, where 208 participants evaluated 14 legal cases before and after receiving algorithmic advice, showed that trust significantly decreased after early and late errors but was rapidly restored in both scenarios. Reliance significantly dropped only for early errors but not for late errors. In both scenarios, reliance was able to be restored. Results suggest that late (compared to early) errors are less drastic in trust loss and allow quicker recovery. These findings align with an interpretation in which humans can build up trust over time if a system is performing well, making them more tolerant of incidental AI errors.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3640543.3645167

Cites

53 of 54 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Schematic reliance (A, left) and trust (B, right) trajectories based on our three experimental conditions, Baseline, Early Error, and Late Error. The vertical thick lines represent the early (trial 4) and late (trial 11) errors. The trajectories represent assumed progressions of reliance and trust based on previous research, for example, trust increases over time [32, 68] where reliance rather stabilizes ([32], including expected reliance and trust drops, and expected recovery. We note that we omit any realistic fluctuations by using straight lines.
Figure 1 Schematic reliance (A, left) and trust (B, right) trajectories based on our three experimental conditions, Baseline, Early Error, and Late Error. The vertical thick lines represent the early (trial 4) and late (trial 11) errors. The trajectories represent assumed progressions of reliance and trust based on previous research, for example, trust increases over time [32, 68] where reliance rather stabilizes ([32], including expected reliance and trust drops, and expected recovery. We note that we omit any realistic fluctuations by using straight lines.CC-BY · From the PDF
Figure 2 Study task procedure visual which also includes reliance (weight on advice) and (self-reported) trust measurements. The legal decision-making task was repeated for 14 trials and was followed by additional questions at the end of the study.
Figure 2 Study task procedure visual which also includes reliance (weight on advice) and (self-reported) trust measurements. The legal decision-making task was repeated for 14 trials and was followed by additional questions at the end of the study.CC-BY · From the PDF
Figure 3 Line plot representing mean reliance levels (calculated based on WoA) for the Baseline (green), Early Error (yellow), and Late Error condition (purple) overall 14 trial sequences (random order per participant). Line graph A) illustrates the averaged reliance trials, whereas line graph B) illustrates the reliance trials when controlling for interpersonal reliance effects.
Figure 3 Line plot representing mean reliance levels (calculated based on WoA) for the Baseline (green), Early Error (yellow), and Late Error condition (purple) overall 14 trial sequences (random order per participant). Line graph A) illustrates the averaged reliance trials, whereas line graph B) illustrates the reliance trials when controlling for interpersonal reliance effects.CC-BY · From the PDF
Figure 4 Line plot representing averaged trust levels for the Baseline (green), Early Error (yellow), and Late Error conditions (purple) over all 14 trial sequences (random order per participant). Line graph A) illustrates the averaged trust trials, whereas line graph B) illustrates the trust trials when controlling for inter-personal trust effects.
Figure 4 Line plot representing averaged trust levels for the Baseline (green), Early Error (yellow), and Late Error conditions (purple) over all 14 trial sequences (random order per participant). Line graph A) illustrates the averaged trust trials, whereas line graph B) illustrates the trust trials when controlling for inter-personal trust effects.CC-BY · From the PDF
Figure 5 Experimental Sequence of Legal Decision-Making Task in Four Steps
Figure 5 Experimental Sequence of Legal Decision-Making Task in Four StepsCC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 54 references

Research works printed but not linked: 14

  1. Phoebe E Bailey, Tarren Leon, Natalie C Ebner, Ahmed A Moustafa, and Gabrielle Weidemann. 2022. A meta-analysis of the weight of advice in decision-making. Current Psychology (2022), 1–26.
  2. Alvaro Chacon, Edgar E Kausel, and Tomas Reyes. 2022. A longitudinal approach for understanding algorithm use. Journal of Behavioral Decision Making (2022).
  3. Franz Faul, Edgar Erdfelder, Axel Buchner, and Albert-Georg Lang. 2009. Statistical power analyses using G* Power 3.1: Tests for correlation and regression analyses. Behavior research methods 41, 4 (2009), 1149–1160.
  4. Heather M Gray, Kurt Gray, and Daniel M Wegner. 2007. Dimensions of mind perception. science 315, 5812 (2007), 619–619.
  5. Stephan Lewandowsky, Michael Mundy, and Gerard Tan. 2000. The dynamics of trust: comparing humans to automation.Journal of Experimental Psychology: Applied 6, 2 (2000), 104.
  6. Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust. In 11th australasian conference on information systems, Vol. 53. Citeseer, 6–8.
  7. JB Manchon, Mercedes Bueno, and Jordan Navarro. 2021. Calibration of Trust in Automated Driving: A Matter of Initial Level of Trust and Automated Driving Style?Human Factors (2021), 00187208211052804.
  8. Raja Parasuraman and Victor Riley. 1997. Humans and automation: Use, misuse, disuse, abuse. Human factors 39, 2 (1997), 230–253.
  9. Tobias M Peters and Roel W Visser. 2023. The Importance of Distrust in AI. arXiv preprint arXiv:2307.13601 (2023). · arXiv
  10. Victor Andrew Riley. 1994. Human use of automation. University of Minnesota.
  11. Daniel B Shank and Alexander Gott. 2019. People’s self-reported encounters of Perceiving Mind in Artificial Intelligence. Data in brief 25 (2019), 104220.
  12. Keng Siau and Weiyu Wang. 2018. Building trust in artificial intelligence, machine learning, and robotics. Cutter business technology journal 31, 2 (2018), 47–53.
  13. Janet A Sniezek and Timothy Buckley. 1995. Cueing and cognitive conflict in judge-advisor decision making. Organizational behavior and human decision processes 62, 2 (1995), 159–174.
  14. X Jessie Yang, Christopher Schemanske, and Christine Searle. 2021. Toward quantifying trust dynamics: How people adjust their trust after moment-to-moment interaction with automation. arXiv preprint arXiv:2107.07374 (2021). · arXiv

Cited by

See all 30

Metadata

Authors
Kahr, P. K., Rooks, G., Snijders, C., & Willemsen, M. C.
Affiliations
Eindhoven University of Technology (NL)
Venue
Proceedings of the 29th International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2024-03-18
Pages
609-622
OpenAlex
W4393970495
Cited by
21
References
44
Citations by year
2026: 9 · 2025: 12
FWCI
14.08
Type
conference paper
Language
en
Open access
Locations
  • ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion, cc-by)
  • TU/e Research Portal (publishedVersion, cc-by)
  • OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
Funders
Technische Universiteit Eindhoven
Topics
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
trust in AI, trust repair, human-AI interaction, algorithmic advice, trust loss

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.