CoChat
Sign Up
Cited by 103Social Robot Interaction and HRIHuman-Automation Interaction and SafetyRobot Manipulation and Learning

Taxonomy of Trust-Relevant Failures and Mitigation Strategies

Open access
Tolmeijer, Suzanne, Weiss, Astrid, Hanheide, Marc, Lindner, Felix, Powers, Thomas M., Dixon, Clare, Tielman, Myrthe L.·2020·Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction
DOI 10.1145/3319502.3374793
Open this paper in CoChat — free accountOpen PDF

Abstract

We develop a taxonomy that categorizes HRI failure types and their impact on trust to structure the broad range of knowledge contributions. We further identify research gaps in order to support fellow researchers in the development of trustworthy robots. Studying trust repair in HRI has only recently been given more interest and we propose a taxonomy of potential trust violations and suitable repair strategies to support researchers during the development of interaction scenarios. The taxonomy distinguishes four failure types: Design, System, Expectation, and User failures and outlines potential mitigation strategies. Based on these failures, strategies for autonomous failure detection and repair are presented, employing explanation, verification and validation techniques. Finally, a research agenda for HRI is outlined, discussing identified gaps related to the relation of failures and HR-trust.

Integrity signals

Registry

Crossref · 10.1145/3319502.3374793

Cites

75 of 75 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 94 cited sources checked · 0 supported · 0 flagged

  • Findingcitation

    In the STRANDS case study, a variety of problems were detected automatically, including navigation issues, forceful pushes to the robot, and hardware failures.

    “In the specific instance, a variety of problems were detected automatically, such as navigation issues [15], forceful pushes to the robot, and hardware failures.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15]

      Not checked in the free run.

  • Findingown data

    Feedback from on-site interviews showed that commission failures have a very high impact on trust.

    “For instance, feedback from on-site interviews showed that commission failures have a very high impact on trust.”
    Quote in the paperNot a substantial claim
  • Findingcitation

    In 1605 instances the robot had to ask for help because it could not recover from a navigation problem, making the failure obvious to interacting humans and potentially affecting trust.

    “An analysis of software failures, in particular navigation failures (which account for more than 99% of all software-related issues in this particular use case) in [15] reveals that in 1605 instances the robot had to ask for help as it could not recover from a navigation problem, making its failure obvious to the interacting humans, and hence potentially having an impact on trust.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15]

      Not checked in the free run.

  • Findingcitation

    Navigation failures account for more than 99% of all software-related issues in this use case.

    “An analysis of software failures, in particular navigation failures (which account for more than 99% of all software-related issues in this particular use case) in [15] reveals that in 1605 instances the robot had to ask for help as it could not recover from a navigation problem, making its failure obvious to the interacting humans, and hence potentially having an impact on trust.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15]

      Not checked in the free run.

  • Findingcitation

    System data indicated about 3.5 interactions per operational hour with users actively using the robot.

    “The system data in [36] indicated that there were about 3.5 interactions per operational hour (i.e.time the robot is not resting or charging) with users that are actively using the robot.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [36]

      Not checked in the free run.

  • May combine two claims. 38
  • Classified as probably not a substantial claim. 56

  • May combine two claims.

    Findingcitation

    Such failure was observed about every 2 hours of autonomous operation, giving a ratio of 7:1 for software failures to interactions and a Probability score of 3 in Table 2.

    “Thus, we observed such failure about every 2 hours of autonomous operation, leading to a ratio of 7 : 1 for Software failures to interactions, leading to a Probability score of “3” in Tab. 2.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Intentional User failure was observed in about 1 out of 200 interactions and scored 2.

    “In the case of the latter, by counting the number of forceful 1 http://strands-project.eu/ robot pushes and deliberate tampering, “intentional User failure” was observed in about 1 out of 200 interactions, scoring “2”.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    A participant complained that the robot appears stupid because it would start talking to a wall, a consequence of misclassification leading to a commission failure.

    “As an example, we quote a participant, who complained that the robot appears “stupid”, because it would “start talking to a wall”, a consequence of misclassification leading to a commission failure.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The scores in Table 2 come from a real-world use case.

    “To show how such a risk table can be used in a HRI context, the scores in Table 2 are from a real world use case.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Table 2 presents case study results as Probability and Impact scores derived from retrospective analysis of the deployment data sets.

    “Tab. 2 presents the results of this case study analysis, in terms of Probability and Impact scores derived from the retrospective analysis of the data sets from the deployments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    [33] revealed some Design failures that led to iterative improvements between annual deployments.

    “For instance, [33] revealed some of the Design failures that lead to the iterative improvements between annual deployments.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] source

      Not checked in the free run.

  • May combine two claims.

    Findingnone given

    Empirical HRI research is increasingly identifying suitable mitigation strategies, but it rarely considers the type of failure that caused the trust violation.

    “Our efforts revealed that empirical research in HRI tries to more and more identify suitable mitigation strategies, but hardly considers the type of failure that caused the trust violation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Brooks et al. explored reactions to failures in autonomous robots by manipulating context risk, failure severity, task support, and human support.

    “In an online survey, Brooks et al. [7] explored people's reactions to failures in autonomous robots, namely a vacuum cleaner and a self-driving taxi, by manipulating four variables: context risk, failure severity, task support and human support.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Tversky and Kahneman and other ethicists regard the action-omission distinction as an omission bias, because the consequences of acting and not acting can be the same.

    “However, psychologists Tversky and Kahneman [74] as well as other ethicists find in this to be an omission bias, since the consequences of (not) acting can be the same.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [74]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Because robots have seen little employment in society, long-term studies have not been conducted so far.

    “Also due to their little employment in society, long-term studies have not been conducted so far.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    A promise to change behaviour can significantly speed trust recovery, but prior deception harms the effectiveness of a promise.

    “Additionally, a promise to change behavior can significantly speed the trust recovery process, but prior deception harms the effectiveness of a promise.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [68]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Trust repair and trust violations have been understudied in HRI.

    “The aspects of trust repair and trust violations have been understudied in the field of HRI [4].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [4]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Trust harmed by deception and the same untrustworthy actions never fully recovers, even with trust repair actions.

    “However, trust harmed by deception and the same untrustworthy actions never fully recovers, even with actions of trust repair [68].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [68]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Verification and validation improve reliability, safety, and trustworthiness and help mitigate some failures that lead to loss of trust.

    “This improves reliability, safety and trustworthiness of the systems (see for example [23,77]) and will help mitigate some of the failures leading to loss of trust.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [23]

      Not checked in the free run.

    • cites [77]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Premisecitation

    According to some, actions that fail expectations and damage trust are worse than failures to act.

    “Actions (commissions) that fail expectations and thus damage trust are, according to some [19], worse than failures to act (omissions).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Hoff and Bashir suggested a three-layered model in which trust factors contribute to dispositional, situational, and learned trust.

    “Based on the three-factor model by Hancock et al. [29], Hoff and Bashir [39] have also suggested a three-layered model in which these factors contribute to dispositional, situational and learned trust.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Major reasons for limited trust in industrial robot collaboration were communication problems that made participants uncertain about their own ability to collaborate, and these problems were strongly linked to interface design.

    “Interestingly, major identified reasons for limited trust were communication problems during collaboration resulting in participant's uncertainty of their own ability to collaborate with the robot.The communication problems were strongly linked to the interface design.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [59] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Wagner et al. proposed a risk-based trust model defining trust as a belief that the trustee will act to mitigate the trustor's risk when the trustor's outcomes are at risk.

    “Wagner et al. [75] propose a trust model based on risk.They define trust as “a belief, held by the trustor, that the trustee will act in a manner that mitigates the trustor's risk in a situation in which the trustor has put its outcomes at risk” [75, p.26:4].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [75] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Promises lead to better trust recovery than apologies, and message timing and exact content are crucial.

    “Promises lead to a better trust recovery than apologies, and in general, the message timing and exact content was shown to be crucial [63].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [63] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Several trust repair actions have been proposed, including apologies, promises, internal or external attribution, and showing a consistent series of trustworthy actions.

    “In ongoing studies, several actions of trust repair have been proposed, including apologies, promises, internal or external attribution, and the showing of consistent series of trustworthy actions [4,14].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [4]

      Not checked in the free run.

    • cites [14]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Trust has static and dynamic components in human-robot interaction; static components such as gender do not change, while system-related dynamic components can be influenced.

    “The challenge becomes more complex as trust has both static and dynamic components in human-robot interaction.Static components such as gender do not change, but dynamic components related to the system can be influenced [39].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [39]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Billings proposed a three-factor model of trust in robots comprising human characteristics, environmental characteristics, and robot characteristics.

    “Billings proposed a three-factor model of trust in robots, including human characteristics such as ability and personality, environmental characteristics such as task and team, and robot characteristics such as performance and attributes [5].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Participants initially had confidence in the robot itself but were insecure about their own ability to collaborate because they could not predict the robot's intentions and instructions.

    “Further, the participants initially had confidence in the robot itself, but were insecure of their own ability to collaborate due to their inability to predict the robot's intentions and instructions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [59] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In an emergency setting, an apology immediately after a trust violation did not recover trust, but an apology right before the next trust decision point repaired trust.

    “In an emergency setting, where an apology right after violated trust has not recovered trust, an apology right before the next trust decision point has repaired trust.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [63] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A meta-analysis identified the same three trust factors and stressed that too few studies have examined environmental and human-related factors, although robot-related factors affect trust most.

    “These three factors have also been identified in a meta-analysis on trust [29], where the authors stressed that too few studies have yet been conducted on environmental and human-related factors, although robot-related factors have been shown to affect trust the most.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Humans who misplace trust through under- or over-reliance may face serious danger, which makes safety concerns highly important.

    “Humans who misplace trust, understood as both underand over-reliance, might be exposed to serious danger, which is the reason safety concerns are of high consideration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Nordqvist and Lindblom's operator experience framework for industrial robot trustworthiness uses ability, benevolence, integrity, perceived safety, time on task, and errors, with 12 UX goals.

    “Nordqvist and Lindblom [59] analyze trustworthiness of industrial robots with an operators' experience framework.The evaluation framework consists of the factors ability, benevolence, integrity, perceived safety, time on task and errors, where in total 12 user experience (UX) goals were characterized, 2 for each component.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [59] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The need for research on trust in artificial agents in cases of inevitable failure has been highlighted.

    “The need for research on trust in artificial agents in cases of inevitable failure has been highlighted as well [5].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The authors reported a non-significant tendency for a preference for both task and human support in high-severity situations and only task support in low-severity situations.

    “However, the authors reported an interesting but non-significant tendency showing a preference for both task and human support in high severity situations, and a preference for only task support in low severity situations.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Intentional misbehavior is harder to address because the user acted purposefully.

    “Intentional misbehavior is harder to address, since the user acted purposefully.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Explanations for end users can be a suitable means to repair trust.

    “Explanations for the end user can be a suitable mean to repair trust.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductcitation

    Rich data sets comprising task and error logs, user demographics, and navigation failures were obtained from the deployments and analysed for the case study.

    “Rich data sets, comprising task and error logs [32], user demographics [36], and navigation failures [15] have been obtained from these deployments, and analysed for the case study for this paper.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [32]

      Not checked in the free run.

    • cites [36]

      Not checked in the free run.

    • cites [15]

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Many failure types can be detected from system logs and dedicated anomaly or failure detection modules that allow their probability to be estimated.

    “Consequently, many failure types can be detected from system logs and from dedicated anomaly or failure detection modules that allow to estimate the probability of them occurring.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Lewicki and Wiethoff explain that restoring trust after a violation is a three-step process: exchanging information about the perceived violation, willingness to forgive the violator, and reaffirming commitment.

    “As Lewicki and Wiethoff [49] explain restoring trust after a violation is a three-step process: (1) exchanging information about the perceived trust violation, (2) willingness to forgive the violator, and (3) reaffirm their commitment.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [49]

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    Impact assessment was based on qualitative analysis in the context of the care deployment within the STRANDS project.

    “To assess the impact of individual failures, we base our assessment of a qualitative analysis in the context of the care deployment within the STRANDS project [24,33,34].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [24] source

      Not checked in the free run.

    • cites [33] source

      Not checked in the free run.

    • cites [34] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The taxonomy distinguishes four failure types: Design, System, Expectation, and User.

    “We differentiate four different failure types with respect to their impact on trust and the related mitigation strategies: (1) Design, (2) System, (3) Expectation, and (4) User (see Table 1 for condensed failure type descriptions).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Expectation failures could lead to unintentional User failures.

    “Of course, Expectation failures could lead to unintentional User failures.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    When a user misbehaves, the response the system gives will influence the user's future behaviour towards the system.

    “When a user misbehaves, the response the system gives will influence future behavior of the user towards the system.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Only some scores in Table 2 were robustly computed from logs; others are informed guesses based on the authors' experience.

    “It shall be noted that this constitutes merely a case study, based on available data, allowing only some scores to be robustly computed from logs, while others have to be informed guesses, based on the authors' experience.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductcitation

    Other probability scores are much harder to obtain in a post-mortem analysis of long-term deployment and require more focused studies involving users directly.

    “The other probability scores are much harder to obtain in a post-mortem analysis of long-term deployment, and require more focused studies, e.g.[33], involving the users directly.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    De Visser et al. stress the importance of trust repair and list possible mitigation strategies, but these strategies had not previously been linked to different failure types.

    “While De Visser et al. [14] stress the importance of trust repair and list possible mitigation strategies, these strategies have not been linked to different failure types before.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [14]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Improving trust through interaction design for other prospective users remains a viable approach.

    “However, improving trust through the interaction design for other prospective users will still be a viable way to go.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The risk table scores are specific to the system and scenario.

    “These scores are system-and scenario-specific.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Hancock et al. provide a detailed overview of HRI studies on the impact of robot design features on trust.

    “Hancock et al. [29] provide a detailed overview on HRI studies on the impact of robot design features on trust in HRI.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Explanations to end users do not necessarily need natural language and can use cues such as closed eyes, blinking lights, or nodding head.

    “Explanations to end users do not necessarily need to be in natural language, but can use cues such as closed eyes, blinking lights, nodding head etc.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    In a system breakdown, trust lost in the system can be minimized when alternatives are available.

    “In case of a system breakdown, the trust lost in the system can be minimized when alternatives are available.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Models of human-automation and human-human trust are a helpful starting point but do not account for the complexities of building and maintaining trust in HRI.

    “As mentioned by Baker et al. [4], models of human-automation and human-human trust are a helpful starting point, but do not account for the complexities of building and maintaining trust in HRI.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [4]

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisenone given

    Discussions on User Failures revealed little to no research on how to mitigate this type of failure.

    “Our discussions on User Failures revealed that there is little to no research on how to mitigate this type of failure.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    The correct level of detail for abstractions and human-comprehensible explanations is challenging.

    “However, the correct level of detail of abstractions and human-comprehensible explanations are challenging.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Proper design allows smooth interactions, and substantial HRI research exists on robot-related factors affecting trust in interaction design, such as social skills, robot role, and communication style.

    “Proper design allows for smooth interactions and substantial research is available in HRI on understanding robot-related factors affecting trust in the interaction design, such as social skills [35], robot role [27], and communication style [60].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [35] source

      Not checked in the free run.

    • cites [27]

      Not checked in the free run.

    • cites [60]

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Interaction Design is important for fostering trust and can also be a tool for reestablishing trust.

    “While it is intuitive that Interaction Design is important to foster trust, it can also be a tool in reestablishing trust.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The impact assessment is not exact, and the case study does not aim for a comprehensive analysis of the STRANDS system but presents concepts of considering trust loss as a risk open to systematic analysis.

    “The assessment is not an exact science; within this case study we do not aim for a comprehensive analysis of this STRANDS system, but rather present the concepts of considering trust loss as a risk open to a systematic analysis.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Different mitigation strategies can help regain user trust depending on the type of failure that occurred.

    “Depending on the type of failure that has taken place, there are different possible mitigation strategies that can help regain the trust of the user.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    If a user was not aware of any misbehavior, asking for justification can create awareness of their mistakes.

    “If the user was not aware of any misbehavior, asking the user for justification of their actions can create awareness of their mistakes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    Fix applies only to System failures and ensures the issue does not occur again.

    “This is a very practical mitigation strategy to ensure the issue does not occur again and only applies to System failures.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Marinaccio et al.'s framework is based on Reason's error taxonomy, which does not account for the interactive nature of HRI.

    “However, they base their framework on the error taxonomy of Reason [61] which does not account for the interactive nature of HRI.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [61] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    User failures can be caused by either a Design failure or an Expectation failure, which affects their impact on trust and possible mitigation strategies.

    “This type of failure can be caused either by a Design failure or an Expectation failure which influences its impact on trust and potential mitigation strategies.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    How failure situations are framed affects trust repair.

    “We argue that a framing of failure situations will have an impact on trust repair and needs to be considered in future studies, but above all in future interaction designs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The belief aspects from [11,18] as modeled in the logics discussed above appear to correspond to the expectation failure type.

    “The belief aspects from [11,18] and modelled in the logics mentioned above seem to match the expectation failure type discussed above.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [11]

      Not checked in the free run.

    • cites [18]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Those belief aspects do not correspond to the more complex trust models presented in Section 2.2.

    “However, they do not match the more complex models of trust as introduced in Section 2.2.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [11]

      Not checked in the free run.

    • cites [18]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The logic of trust in [37] has a complete axiomatisation, but its complexity and decidability are not addressed.

    “A complete axiomatisation is provided but complexity and decidability are not considered.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Failure classifications often come with severity rankings, for example Carlson and Murphy's classification.

    “Failure classifications often come with severity rankings, such as the failure classification by Carlson and Murphy [10].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Little research has been done on how users can be trained in HRI, since research mostly focuses on how users can train robots, but existing work shows that training is essential.

    “So far, little research has been done on how users can be trained in HRI (since research mostly focuses on how users can train robots [1]), but existing work shows that “training is essential” [9].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [1]

      Not checked in the free run.

    • cites [9]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Failure detection is considered a classification problem in which a model of the failure can be defined or learned.

    “In other words, the failure detection problem is considered a classification problem, where a model of the failure itself can either be defined or learned.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    STRIPS and its derivatives are commonly used to represent a robot's deliberate actions and external events.

    “STRIPS and its derivatives are widely used to describe a robot's deliberate actions and external events.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20]

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Counterfactual explanation approaches for plans only partly satisfy Miller's selectivity requirement, because many candidate explanations exist and selecting the most suitable one is not clear.

    “These approaches only partially fulfill Miller's criteria of selectivity, though: although minimality criteria are considered, there are generally many possible explanations and it is not necessarily clear how to pick the most appropriate ones.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [25]

      Not checked in the free run.

    • cites [22]

      Not checked in the free run.

    • cites [51]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Identification of a potentially known problem can be deferred to explanation generation using domain knowledge.

    “The identification of a potentially known problem can then be deferred to approaches to generation explanations, utilising domain knowledge as formally defined in the following section and also explored in [30].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Some studies indicate that repeatedly helping a robot in need when the suggested repair strategy works does not reduce likability.

    “Some studies on people's willingness to help robots after repeated failure indicate that repeatedly helping robots in need when the suggested repair strategy is successful does not reduce likability [3].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Automated detection of a failure is the first necessary step in mitigating failures.

    “While automated detection of a failure is the first necessary step in failure mitigation, the next goal should be automated trust loss detection to be able to respond appropriately.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Miller holds that explanations need to be contrastive, selective, and social.

    “According to Miller [57], explanations should be contrastive, selective, and social.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [57]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Formal verification, simulation-based testing, and end-user experiments can help mitigate system failure, design failure, and expectation failure, with end-user experiments relevant to the latter two.

    “Formal verification, simulation based testing and end user experiments can help improve the safety, reliability and trust in robotic systems [23,77], as well as help mitigate as system failure (all), design failure and expectation failure (end user experiments).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [23]

      Not checked in the free run.

    • cites [77]

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    The template-based POMDP explanation approach is less generic and needs hand-crafted templates.

    “The downside of this approach is its being less generic and its requiring hand-crafted template modeling.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [76]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    To detect a user's loss of trust in the system, different cues in the user's behavior need to be distinguishable.

    “Furthermore, different cues in user behavior need to be distinguishable to detect trust loss of the user in the system.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Counterfactual explanations have been used to explain plan failures, why an action plan includes a particular action, and whether an action plan follows moral principles.

    “Counterfactual explanations have recently been applied to generating explanations for plan failures [25], for explaining why an action plan contains a specific action [22], and to explain why an action plan does (not) adhere to moral principles [51].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [25]

      Not checked in the free run.

    • cites [22]

      Not checked in the free run.

    • cites [51]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Anomaly detection aims to detect any deviation from normal system behaviour without necessarily classifying a problem.

    “A complementary approach relates to anomaly detection (e.g. recently surveyed in [28]).It aims to detect any deviation from a normal behaviour of a system, without necessarily classifying a problem.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Those studies do not reveal how much overall trust is harmed.

    “However, this does not give insights into how much overall trust is harmed.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The probabilistic rational temporal logic from [43,44] is generally undecidable, but decidable fragments have been identified.

    “The resulting logic is, in general, undecidable but decidable fragments are identified.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [43] source

      Not checked in the free run.

    • cites [44]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The current logics for modeling trust and trust loss are fairly simplistic.

    “As mentioned before, the current logics that allow trust (loss) modeling are fairly simplistic.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationown data

    Proper recognition and standardized scoring of trust loss could greatly benefit trust research in HRI, as the STRANDS use case has shown.

    “As the STRANDS use case has shown, proper recognition and standardized scoring of trust loss could greatly benefit trust research in HRI.”
    Quote in the paperNot a substantial claim
79 of 79

  • The paper relies on this without support.

    Human responses to robotic systems are complex and depend on factors such as the system's morphology, behaviour, and deployment context.

    “Human responses towards robotic systems are very complex in their nature and depend on many factors, such as the morphology and behavior of the system and the context in which they are deployed.”

  • The paper relies on this without support.

    Information about limitations following automation errors aids trust recovery.

    “Following errors of automation, information related to limitations further aid in trust recovery.”

  • The paper relies on this without support.

    No taxonomy for trust repair in HRI exists.

    “A taxonomy for trust repair in HRI does not exist, but a framework for rebuilding trust in automation has been proposed by Marinaccio et al. [55].”

  • The authors state this as an assumption.

    Once trust is broken because of a Design failure, redesigning the system becomes even more challenging.

    “However, we can assume that once trust is broken due to a Design failure, the redesign of the system becomes even more challenging.”

  • The paper relies on this without support.

    In human-human interaction, apologies are more effective than shifting blame elsewhere.

    “In human-human interaction, they are more effective than shifting the blame elsewhere.”

  • The authors state this as an assumption.

    Unintentional negative behaviour will not be repeated once the user becomes aware of it.

    “We assume that unintentional negative behavior will not be repeated once the user becomes aware of it.”

  • The paper relies on this without support.

    Robots that interact with humans in the wild will at some point face failure situations inflicted by the robot, the human, or unexpected environmental events.

    “Robots that interact with humans in the wild will at some point face failure situations, which can either be inflicted by the robot, the human, or by unexpected environmental events.”

  • The paper relies on this without support.

    It is in human nature to anthropomorphize robots, for example by projecting a personality onto them or reading emotions into their output.

    “It is in our nature to anthropomorphize robots, for example by projecting a personality onto the robot or reading emotions into its output.”

  • The authors state this as an assumption.

    There exists an empty action in the operator set with no preconditions, no delete list, and no add list.

    “We assume the existence of the empty action ϵ ∈ O, which has an empty precondition, an empty delete list, and an empty add list.”
9 of 9

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Software-failure ratio is reversed

    Minor

    At 3.5 interactions per operational hour and one software failure about every two hours, there are about seven interactions per failure-not seven software failures per interaction.

  • Marinaccio citation points to a different reference

    Minor

    The text attributes the trust-rebuilding framework to Marinaccio et al. at reference [55], but entry 55 is a paper by Malle et al. The Marinaccio paper is listed as entry 56.

2 of 2

Cites

Flagged first, then by citations

3 of the 78 references couldn’t be matched to a known paper.

Research works printed but not linked: 8

  1. Paul Gainer , Clare Dixon , Kerstin Dautenhahn , Michael Fisher , Ullrich Hustadt , Joe Saunders , and Matt Webster . 2017. CRutoN: Automatic Verification of aRobotic Assistant's Behaviours. InCritical Systems: Formal Methods and Auto-mated Verification - Joint 22nd International Workshop on Formal Methods forIndustrial Critical Systems - and - 17th International Workshop on AutomatedVerification of Critical Systems , FMICS-AVoCS 2017 , Turin, Italy, September 18--20,2017, Proceedings (Lecture Notes in Computer Science), Laure Petrucci, CristinaSeceleanu, and Ana Cavalcanti (Eds.), Vol. 10471 . Springer , Turin, Italy, 119--133.https://doi.org/10.1007/978--3--319--67113-0_8 10.1007/978--3--319--67113-0_8 Paul Gainer, Clare Dixon, Kerstin Dautenhahn, Michael Fisher, Ullrich Hustadt,Joe Saunders, and Matt Webster. 2017. CRutoN: Automatic Verification of aRobotic Assistant's Behaviours. InCritical Systems: Formal Methods and Auto-mated Verification - Joint 22nd International Workshop on Formal Methods forIndustrial Critical Systems - and - 17th International Workshop on AutomatedVerification of Critical Systems, FMICS-AVoCS 2017, Turin, Italy, September 18--20,2017, Proceedings (Lecture Notes in Computer Science), Laure Petrucci, CristinaSeceleanu, and Ana Cavalcanti (Eds.), Vol. 10471. Springer, Turin, Italy, 119--133.https://doi.org/10.1007/978--3--319--67113-0_8 · DOI
  2. Xiaowei Huang and Marta Zofia Kwiatkowska . 2017. Reasoning about CognitiveTrust in Stochastic Multiagent Systems. InProceedings of the Thirty-First AAAIConference on Artificial Intelligence , February 4--9 , 2017 , San Francisco, California ,USA., Satinder P. Singh and Shaul Markovitch (Eds.). AAAI Press , 3768--3774.http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14566 Xiaowei Huang and Marta Zofia Kwiatkowska. 2017. Reasoning about CognitiveTrust in Stochastic Multiagent Systems. InProceedings of the Thirty-First AAAIConference on Artificial Intelligence, February 4--9, 2017, San Francisco, California,USA., Satinder P. Singh and Shaul Markovitch (Eds.). AAAI Press, 3768--3774.http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14566
  3. John D. Lee and Katrina A . See . 2004 . Trust in Automation : Designing forAppropriate Reliance.Human Factors 46, 1 (2004), 50--80. John D. Lee and Katrina A. See. 2004. Trust in Automation: Designing forAppropriate Reliance.Human Factors46, 1 (2004), 50--80.
  4. Michael Lewis , Katia Sycara , and Phillip Walker . 2018. The Role of Trust inHuman-Robot Interaction. InFoundations of Trusted Autonomy , Hussein A.Abbass , Jason Scholz, and Darryn J. Reid (Eds.). Springer International Publishing ,Cham, 135--159. https://doi.org/10.1007/978--3--319--64816--3_8 10.1007/978--3--319--64816--3_8 Michael Lewis, Katia Sycara, and Phillip Walker. 2018. The Role of Trust inHuman-Robot Interaction. InFoundations of Trusted Autonomy, Hussein A.Abbass, Jason Scholz, and Darryn J. Reid (Eds.). Springer International Publishing,Cham, 135--159. https://doi.org/10.1007/978--3--319--64816--3_8 · DOI
  5. John K Rempel , John G Holmes , and Mark P Zanna . 1985. Trust in close relation-ships.Journal of personality and social psychology49, 1 ( 1985 ), 95. John K Rempel, John G Holmes, and Mark P Zanna. 1985. Trust in close relation-ships.Journal of personality and social psychology49, 1 (1985), 95.
  6. Laurence Thomas . 1987. Friendship. Synthese72, 2 ( 1987 ), 217--236 . Laurence Thomas. 1987. Friendship. Synthese72, 2 (1987), 217--236.
  7. Ning Wang , David V. Pynadath , and Susan G. Hill . 2016. Trust Calibration within a Human-Robot Team: Comparing Automatically Generated Explanations . In11th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE , New York, NY,USA, 109--116. Ning Wang, David V. Pynadath, and Susan G. Hill. 2016. Trust Calibration within a Human-Robot Team: Comparing Automatically Generated Explanations. In11th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE,New York, NY,USA, 109--116.
  8. M. Webster , D. Western , D. AraizaIllan , C. Dixon , K. Eder , M. Fisher , and A. Pipe . 2020. A Corroborative Approach to Verification and Validation of Human--RobotTeams.International Journal of Robotics Research39 ( 2020 ), 73--99. M. Webster, D. Western, D. AraizaIllan, C. Dixon, K. Eder, M. Fisher, and A. Pipe.2020. A Corroborative Approach to Verification and Validation of Human--RobotTeams.International Journal of Robotics Research39 (2020), 73--99.

Cited by

Sorted by citations

Related

Authors
Tolmeijer, S., Weiss, A., Hanheide, M., Lindner, F., Powers, T. M., Dixon, C., & Tielman, M. L.
Affiliations
  • University of Zurich (CH)
  • TU Wien (AT)
  • University of Lincoln (GB)
  • Universität Ulm (DE)
  • University of Delaware (US)
  • University of Liverpool (GB)
  • Delft University of Technology (NL)
ORCID
Venue
Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction
Publisher
ACM
Published
2020-03-09
Pages
3-12
DOI
10.1145/3319502.3374793
OpenAlex
W3010109960
Semantic Scholar
0daae809a2018e7b2124570a8c031e310891ee98
Cited by
103
References
78
Citations by year
2026: 6 · 2025: 22 · 2024: 18 · 2023: 17 · 2022: 14 · 2021: 20 · 2020: 6
FWCI
20.10
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM/IEEE International Conference on Human-Robot Interaction (HRI) (publishedVersion, other-oa)
  • Research Explorer (The University of Manchester) (publishedVersion, other-oa)
  • Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, other-oa)
  • Research Repository (Delft University of Technology) (submittedVersion, other-oa)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • Zurich Open Repository and Archive (University of Zurich)
Funders
UKRIHubs for Robotics and AI in Hazardous Environments (EP/R026092(FAIR-SPACE), EP/R026084 (RAIN))
Topics
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
Keywords
human-robot interaction, trust in robots, breach of trust, failure taxonomy, trust repair, design failure, system failure, explanation, verification and validation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review