The Development of Overtrust: An Empirical Simulation and Psychological Analysis in the Context of Human–Robot Interaction
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
With impressive developments in human–robot interaction it may seem that technology can do anything. Especially in the domain of social robots which suggest to be much more than programmed machines because of their anthropomorphic shape, people may overtrust the robot's actual capabilities and its reliability. This presents a serious problem, especially when personal well-being might be at stake. Hence, insights about the development and influencing factors of overtrust in robots may form an important basis for countermeasures and sensible design decisions. An empirical study [N= 110] explored the development of overtrust using the example of a pet feeding robot. A 2 × 2 experimental design and repeated measurements contrasted the effect of one's own experience, skill demonstration, and reputation through experience reports of others. The experiment was realized in a video environment where the participants had to imagine they were going on a four-week safari trip and leaving their beloved cat at home, making use of a pet feeding robot. Every day, the participants had to make a choice: go to a day safari without calling options (risk and reward) or make a boring car trip to another village to check if the feeding was successful and activate an emergency call if not (safe and no reward). In parallel to cases of overtrust in other domains (e.g., autopilot), the feeding robot performed flawlessly most of the time until in the fourth week; it performed faultily on three consecutive days, resulting in the cat's death if the participants had decided to go for the day safari on these days. As expected, with repeated positive experience about the robot's reliability on feeding the cat, trust levels rapidly increased and the number of control calls decreased. Compared to one's own experience, skill demonstration and reputation were largely neglected or only had a temporary effect. We integrate these findings in a conceptual model of (over)trust over time and connect these to related psychological concepts such as positivism, instant rewards, inappropriate generalization, wishful thinking, dissonance theory, and social concepts from human–human interaction. Limitations of the present study as well as implications for robot design and future research are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 5×
“Accordingly, Lee and See (2004), p. 51) de fi ne trust as "the attitude that an agent will help achieve an individual ’ s goals in a situation characterized by uncertainty and vulnerability."”
BackgroundInfluentialHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 1×
“…the de fi nition of automation as "the execution by a machine agent (usually a computer) of a function that was previously carried out by a human" (Parasuraman and Riley, 1997, p. 231), robots expand the fi eld by perception and intelligence and other important factors (Feil-Seifer and Mataric,…”
BackgroundA new scale for the measurement of interpersonal trust1
Rotter, J. B. (1967) · Journal of Personality · doi:10.1111/j.1467-6494.1967.tb01454.x · Cited by 3,503
Rempel, J. K., Holmes, J. G., & Zanna, M. P. (1985) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.49.1.95 · Cited by 3,146 · cited 1×
“A review of trust de fi nitions in general (e.g., Rotter, 1967; Barber, 1983; Rempel et al., 1985; Luhmann, 2018) highlights the multidimensionality of the concept, each focusing on different aspects of people ’ s everyday usage of trust.”
BackgroundThe role of the basal ganglia in habit formation
Yin, H. H., & Knowlton, B. J. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1919 · Cited by 2,402
Nass, C., Steuer, J., & Tauber, E. R. (1994) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191703 · Cited by 2,148 · cited 1×
“Related to the discussion of robots and computers as social actors, as it has already started in the 90 ’ s (Nass et al., 1994; Lee and Nass, 2003), is the factor of anthropomorphism.”
A Meta-Analysis of Factors Affecting Trust in Human-Robot Interaction
Hancock, P. A., Billings, D. R., Schaefer, K. E., Chen, J. Y. C., de Visser, E. J., & Parasuraman, R. (2011) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720811417254 · Cited by 2,051 · cited 3×
“In a literature review on trust in the domain of robots (Hancock et al., 2011), one of the most dominant in fl uencing factors turned out to be reputation in the sense of knowledge about the robot ’ s reliance (Bagheri and Jamieson, 2004) or knowledge about the robot ’ past performance (Lee and See,…”
BackgroundInfluentialFoundations for an Empirically Determined Scale of Trust in Automated Systems
Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000) · International Journal of Cognitive Ergonomics · doi:10.1207/s15327566ijce0401_04 · Cited by 1,799 · cited 1×
“To some degree, trust in humans, automation, and robots are based on similar fundamental characteristics such as reliability, predictability, and ability (Jian et al., 2000).”
BackgroundLearning and Memory Functions of the Basal Ganglia
Packard, M. G., & Knowlton, B. J. (2002) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.25.112701.142937 · Cited by 1,717 · cited 1×
“…on rather vague and intuitive associations based on unconscious, non-declarative memory systems, such as in the case of perceptual learning (e.g., Packard and Knowlton, 2002; Gazzaniga et al., 2006; Yin and Knowlton, 2006) and the improved abilities of sensory systems to respond to stimuli…”
BackgroundJustification effects on consumer choice of hedonic and utilitarian goods.
Okada, E. M. (2005) · Journal of Marketing Research · doi:10.1509/jmkr.42.1.43.56889 · Cited by 1,069 · cited 1×
“First, it is well known from consumer choice that people have a natural preference for hedonic, experiential options (here: the safari) over pragmatic options (here: the control call), especially if they can fi nd a reason to justify their choice (e.g., Böhm and P fi ster, 1996; Okada, 2005).”
Background
Research works printed but not linked: 30
- Barber, 1983, The logic and limits of trust
- Bartneck, 2009, Does the design of a robot influence its animacy and perceived intelligence?, Int. J. Soc. Robotics., 1, 195, 10.1007/s12369-009-0013-7 · DOI · Paper page
- Böhm, 1996, Instrumental or emotional evaluations: what determines preferences?, Acta Psychol (Amst)., 93, 135, 10.1016/0001-6918(96)00017-0 · DOI
- Carlson, 2014, Identifying factors that influence trust in automated cars and medical diagnosis systems
- Cathcart, 1997, The media equation: how people treat computers, television, and new media like real people and places, 305
- Chen, 2010, Supervisory control of unmanned vehicles
- Coeckelbergh, 2012, Can we trust robots?, Ethics Inf. Technol., 14, 53, 10.1007/s10676-011-9279-1 · DOI · Paper page
- Darling, 2012, Extending legal rights to social robots, 10.2139/ssrn.2044797 · DOI
- Diefenbach, 2019, Combining model-based analysis with phenomenological insight: a case study on hedonic product quality, Qual. Psychol., 6, 3, 10.1037/qup0000096 · DOI
- Festinger, 1957, A theory of cognitive dissonance, 10.1515/9781503620766 · DOI · Paper page
- Gazzaniga, 2006, Cognitive Neuroscience. The biology of the mind
- Hoffman, 2009, The dynamics of trust in cyberdomains, IEEE Intell. Syst., 24, 5, 10.1109/mis.2009.124 · DOI · Paper page
- Johns, 1996, A concept analysis of trust, J. Adv. Nurs., 24, 76, 10.1046/j.1365-2648.1996.16310.x · DOI · Paper page
- Luhmann, 2018, Trust and power
- Monroe, 2018, AI, explain yourself, Commun. ACM., 61, 11, 10.1145/3276742 · DOI
- Moorman, 1993, Factors affecting trust in market research relationships, J. mark., 57, 81, 10.2307/1252059 · DOI · Paper page
- Mosier, 1992, Electronic checklists: implications for decision making, 7
- Muir, 1987, Trust between humans and machines, and the design of decision aids, Int. J. Man-Machine Stud., 27, 527, 10.1016/s0020-7373(87)80013-5 · DOI · Paper page
- Parasuraman, 2008, Situation awareness, mental workload, and trust in automation: viable, empirically supported cognitive engineering constructs, J. Cogn. Eng. Decis. making., 2, 140, 10.1518/155534308x284417 · DOI · Paper page
- Percival, 2014, Confirmation versus falsificationism, encyclopedia Clin. Psychol., 10.1002/9781118625392.wbecp388 · doi:10.1002/9781118625392.wbecp388
- Rousseau, 1998, Not so different after all: a cross-discipline view of trust, Acad. Manag. Review Amr., 23, 393, 10.5465/amr.1998.926617 · DOI · Paper page
- Sanchez, 2006, Factors that affect trust and reliance on an automated aid
- Sheridan, 2002, Humans and automation: system design and research issues
- Slater, 2006, A virtual reprise of the Stanley Milgram obedience experiments, PloS one, 1, e39, 10.1371/journal.pone.0000039 · DOI · Paper page
- Sparaco, 1995, Airbus seeks to keep pilot, new technology in harmony, Aviation Week Space Tech., 142, 62
- Syrdal, 2007, Looking good? Appearance preferences and robot personality inferences at zero acquaintance, 86
- Wagner, 2018, Overtrust in the robotic age, Commun. ACM., 61, 22, 10.1145/3241365 · DOI · Paper page
- Wagner, 2009, The role of trust and relationships in human-robot social interaction
- Yagoda, 2012, You want me to trust a ROBOT? The development of a human–robot interaction trust scaleYou Want Me to Trust a ROBOT? The Development of a Human-Robot Interaction Trust Scale, Int J Soc Robotics., 4, 235, 10.1007/s12369-012-0144-0 · DOI · Paper page
- Zuboff, 1989, The age of the smart machine: the future of work and power
Cited by
Christoforakos, L., Gallucci, A., Surmava-Große, T., Ullrich, D., & Diefenbach, S. (2021) · Frontiers in Robotics and AI · doi:10.3389/frobt.2021.640444 · Cited by 154
Self-citationWischnewski, M., Krämer, N., & Müller, E. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581197 · Cited by 146
“…141], hotel reviews [71], clearing tables [18], fnd housing [43, 124], navigation through buildings [101, 112], long-distance management of robots [126] studies of this survey, it was difcult to gain insights into what powered the automated systems, as many authors merely described their systems…”
BackgroundHuman control of AI systems: from supervision to teaming
Tsamados, A., Floridi, L., & Taddeo, M. (2025) · AI and Ethics · doi:10.1007/s43681-024-00489-4 · Cited by 92
Virvou, M., Tsihrintzis, G. A., & Tsichrintzi, E.-A. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120759 · Cited by 55
Metadata
- Authors
- Ullrich, D., Butz, A., & Diefenbach, S.
- Venue
- Frontiers in Robotics and AI
- Publisher
- Frontiers Media SA
- Published
- 2021-04-13
- Volume
- 8
- Pages
- 554578-554578
- ISSN
- 2296-9144 (ISSN-L)
- OpenAlex
- W3129716156
- Cited by
- 69
- References
- 18
- FWCI
- 8.03
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Development of Overtrust: An Empirical Simulation and Psychological Analysis in the Context of Human–Robot Interaction. https://app.cochat.ai/papers/10.3389/frobt.2021.554578
- The paper
- Ullrich, D., Butz, A., & Diefenbach, S. (2021). The Development of Overtrust: An Empirical Simulation and Psychological Analysis in the Context of Human–Robot Interaction. Frontiers in Robotics and AI. https://doi.org/10.3389/frobt.2021.554578
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.