CoChat
Sign Up
Cited by 73Human-Automation Interaction and SafetySocial Robot Interaction and HRIUsability and User Interface Design

Do I trust my machine teammate?: an investigation from perception to decision

Open access
Yu, Kun, Berkovsky, Shlomo, Taib, Ronnie, Zhou, Jianlong, Chen, Fang·2019·Proceedings of the 24th International Conference on Intelligent User Interfaces
DOI 10.1145/3301275.3302277
Open this paper in CoChat — free accountOpen PDF

Abstract

In the human-machine collaboration context, understanding the reason behind each human decision is critical for interpreting the performance of the human-machine team. Via an experimental study of a system with varied levels of accuracy, we describe how human trust interplays with system performance, human perception and decisions. It is revealed that humans are able to perceive the performance of automatic systems and themselves, and adjust their trust levels according to the accuracy of systems. The 70% system accuracy suggests to be a threshold between increasing and decreasing human trust and system usage. We have also shown that trust can be derived from a series of users' decisions rather than from a single one, and relates to the perceptions of users. A general framework depicting how trust and perception affect human decision making is proposed, which can be used as future guidelines for human-machine collaboration design.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3301275.3302277

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Appropriate trust and reliance play a critical role in achieving complementary human-AI team performance.

    “Prior work has highlighted that appropriate trust and reliance play a critical role in achieving complementary human-AI team performance [58, 90, 139, 141].”

    From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [139]

    The Introduction calls trust a critical factor in human–machine decisions and performance, and Related Work says appropriate trust benefits collaboration. Results §4.2 links trust to reliance on machine recommendations. The paper does not establish that appropriate trust and reliance achieve *complementary* team performance, as distinct from affecting collaboration and decisions; page numbers are unavailable in our copy. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Several mechanisms have been proposed to optimize the combination of human and AI decisions and facilitate appropriate reliance, including interpretable explanations, cognitive forcing functions, and feedback loops.

    “These mechanisms include providing interpretable explanations for AI advice [21, 72, 123], using cognitive forcing functions[17, 47, 99], and incorporating feedback loops to enhance the interaction between humans and AI systems [10, 11, 139].”

    From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [139]

    The Methodology (§3.2 Tasks; page unavailable) describes a repeated decision-and-feedback process: after each decision, users saw the actual outcome and whether they were correct. The Discussion (§5; page unavailable) reports that users adapted their trust to system performance over repeated trials. This supports the feedback aspect of the cited clause, but the paper does not establish that incorporating feedback loops enhances human–AI interaction. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • The effect of a mental model of AI accuracy is unclear, as some studies found that any accuracy score increases trust, others found high accuracy scores increased trust and low accuracy scores decreased trust, while another found that these effects are mediated by an individual's innate likelihood of trusting automation.

    “The effect of a mental model of AI accuracy is unclear, as some studies found that any accuracy score increases trust [55], others found high accuracy scores increased trust and low accuracy scores decreased trust [96,97], while another found that these effects are mediated by an individual's innate likelihood of trusting automation [77].”

    From Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2023) · cites this paper as [96]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • The objective metrics used are commonly adapted in literature to capture the level of reliance within the human-AI interaction context.

    “These parameters are commonly adapted in literature to capture the level of reliance within the human-AI interaction context.”

    From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [139]

  • Previous work has suggested that trust increases someone's likelihood of using a machine.

    “Previous work has suggested that trust increases someone's likelihood of using a machine [96,97].”

    From Explanations Can Reduce Overreliance on AI Systems During Decision-Making (2023) · cites this paper as [96]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. [17]17. S. Zuboff (1988). In the age of the smart machine: the future of power and work. New York: Basic.FNM Surname (2018). Article Title. Journal Title, 10(3), 1-10.

Cited by

Sorted by citations

Related

Authors
Yu, K., Berkovsky, S., Taib, R., Zhou, J., & Chen, F.
Affiliations
  • University of Technology Sydney (AU)
  • Commonwealth Scientific and Industrial Research Organisation (AU)
ORCID
Venue
Proceedings of the 24th International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2019-03-17
Pages
460-468
DOI
10.1145/3301275.3302277
OpenAlex
W2918027211
Cited by
71
References
32
Citations by year
2026: 7 · 2025: 13 · 2024: 14 · 2023: 12 · 2022: 11 · 2021: 11 · 2020: 3
FWCI
10.32
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
  • UTS ePRESS (University of Technology Sydney) (submittedVersion)
Funders
Asian Office of Aerospace Research and Development
Topics
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
human-machine collaboration, system accuracy, human perception, human decision-making, trust threshold, trust framework

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review