Examining Profiles for Robotic Risk Assessment: Does a Robot's Approach to Risk Affect User Trust?
Open accessAbstract
As autonomous robots move towards ubiquity, the need for robots to make decisions under risk that are trustworthy becomes increasingly significant; both to aid acceptance and to fully utilise their autonomous capabilities. We propose that incorporating a human approach to risk assessment into a robot's decision making process will increase user trust. This work investigates four robotic approaches to risk and, through a user study, explores the levels of trust placed in each. These approaches are: risk averse, risk seeking, risk neutral and a human approach to risk. Risk is artificially stimulated through performance-based compensation, in line with previous studies. The study was conducted in a virtual nuclear environment created using the Unity games engine. Forty participants were asked to complete a robot supervision task, in which they observed a robot making risk based decisions and were able to question the robot, question the robot further and ultimately accept or alter the robot's decision. It is shown that a robot that is risk seeking is significantly less trusted than a risk averse robot, a risk neutral robot and a robot utilising human approach to risk. There was found to be no significant difference between the levels of trust placed in the risk averse, risk neutral and human approach to risk. It is also found that the level to which participants question a robot's decisions does not form an accurate measure of trust. The results suggest that when designing a robot that must make risk based decisions during teleoperation in a hazardous environment, an engineer should avoid a risk seeking robot. However, that same engineer may choose whichever of the remaining risk profiles best suits the implementation, with knowledge that the trust in their system is unlikely to be significantly affected.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Such behavioral measures include decision time, number of recommendation requests, and number of agreements with AI recommendations.
“Such measures include, but are not limited to measuring end-users' decision time [19,62], how many times end-users requested AI recommendation [52], and how many times they agreed with AI recommendations [7,23,58].”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [7]
Cites
Advances in prospect theory: Cumulative representation of uncertainty
Tversky, A., & Kahneman, D. (1992) · Journal of Risk and Uncertainty · doi:10.1007/bf00122574 · Cited by 15,467 · see citing works
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 1× · see citing works
“For a review detailing the different aspects contributing to trust in an automated system, the interested reader may refer to work by Lee and See [15].”
MethodHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · see citing works
Prospect Theory: An Analysis of Decision Under Risk
Kahneman, D., & Tversky, A. (2013) · World Scientific Handbook in Financial Economics Series · doi:10.1142/9789814417358_0006 · Cited by 3,091 · see citing works
Gamification in theory and action: a survey
Seaborn, K., & Fels, D. I. (2015) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2014.09.006 · Cited by 2,515 · see citing works
A Meta-Analysis of Factors Affecting Trust in Human-Robot Interaction
Hancock, P. A., Billings, D. R., Schaefer, K. E., Chen, J. Y. C., de Visser, E. J., & Parasuraman, R. (2011) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720811417254 · Cited by 2,051 · cited 5× · see citing works
“This was so that the only factor taken into account was the approach to decision making that the robot took, as it has been shown that robot performance can affect user trust [11].”
BackgroundResultInfluentialFoundations for an Empirically Determined Scale of Trust in Automated Systems
Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000) · International Journal of Cognitive Ergonomics · doi:10.1207/s15327566ijce0401_04 · Cited by 1,799 · see citing works
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · see citing works
International Journal of Human-Computer Studies
Sdol, E. (2009) · Cited by 1,095 · see citing works
Choice Under Uncertainty: Problems Solved and Unsolved
Machina, M. J. (1992) · Hübner international series on risk, insurance, and economic security · doi:10.1007/978-94-015-7957-5_2 · Cited by 950 · see citing works
Trust, self-confidence, and operators' adaptation to automation
Lee, J. D., & Moray, N. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1007 · Cited by 932 · see citing works
Almost human: Anthropomorphism increases trust resilience in cognitive agents.
de Visser, E. J., Monfort, S. S., McKendrick, R., Smith, M. A. B., McKnight, P. E., Krueger, F., & Parasuraman, R. (2016) · Journal of Experimental Psychology: Applied · doi:10.1037/xap0000092 · Cited by 614 · see citing works
Nagatani, K., Kiribayashi, S., Okada, Y., Otake, K., Yoshida, K., Tadokoro, S., Nishimura, T., Yoshida, T., Koyanagi, E., Fukushima, M., et al. (2012) · Journal of Field Robotics · doi:10.1002/rob.21439 · Cited by 605 · see citing works
Similarities and differences between human–human and human–automation trust: an integrative review
Madhavan, P., & Wiegmann, D. A. (2007) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220500337708 · Cited by 577 · see citing works
An experimental test of several generalized utility theories
Camerer, C. F. (1989) · Journal of Risk and Uncertainty · doi:10.1007/bf00055711 · Cited by 549 · see citing works
Salem, M., Lakatos, G., Amirabdollahian, F., & Dautenhahn, K. (2015) · Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/2696454.2696497 · Cited by 494 · see citing works
Nonlinear Preference and Utility Theory.
Sugden, R., & Fishburn, P. C. (1989) · The Economic Journal · doi:10.2307/2234100 · Cited by 493 · see citing works
The Benefits of Interactions with Physically Present Robots over Video-Displayed Agents
Bainbridge, W. A., Hart, J., Kim, E. S., & Scassellati, B. M. (2010) · International Journal of Social Robotics · doi:10.1007/s12369-010-0082-7 · Cited by 478 · see citing works
Overtrust of robots in emergency evacuation scenarios
Robinette, P., Li, W., Allen, R., Howard, A. M., & Wagner, A. R. (2016) · 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2016.7451740 · Cited by 468 · see citing works
The Perceived Utility of Human and Automated Aids in a Visual Detection Task
Dzindolet, M. T., Pierce, L. G., Beck, H. P., & Dawe, L. A. (2002) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/0018720024494856 · Cited by 386 · see citing works
The effect of presence on human-robot interaction
Bainbridge, W. A., Hart, J., Kim, E. S., & Scassellati, B. (2008) · RO-MAN 2008 - The 17th IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/roman.2008.4600749 · Cited by 300 · see citing works
Effect of Robot Performance on Human–Robot Trust in Time-Critical Situations
Robinette, P., Howard, A. M., & Wagner, A. R. (2017) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2017.2648849 · Cited by 205 · see citing works
Effects of changing reliability on trust of robot systems
Desai, M., Medvedev, M., Vázquez, M., McSheehy, S., Gadea-Omelchenko, S., Bruggeman, C., Steinfeld, A., & Yanco, H. (2012) · Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction · doi:10.1145/2157689.2157702 · Cited by 188 · see citing works
How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics
Majumdar, A., & Pavone, M. (2019) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-28619-4_10 · Cited by 180 · see citing works
Consideration of Risk in Reinforcement Learning
Heger, M. (1994) · Elsevier eBooks · doi:10.1016/b978-1-55860-335-6.50021-0 · Cited by 179 · see citing works
Human-robot interaction: Developing trust in robots
Billings, D., Schaefer, K. E., Chen, J. Y. C., & Hancock, P. (2012) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1145/2157689.2157709 · Cited by 136 · see citing works
Rae, I., Takayama, L. A., & Mutlu, B. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466253 · Cited by 99 · see citing works
Unity Game Development Essentials
Goldstone, W. (2009) · Cited by 98 · see citing works
Effects of changing reliability on trust of robot systems
Desai, Medvedev, Vázquez, McSheehy, Gadea-Omelchenko, Bruggeman, Steinfeld, & Yanco (2012) · Cited by 72 · see citing works
Wang, N., Pynadath, D., Rovira, E., Barnes, M., & Hill, S. (2018) · International Conference on Persuasive Technology · doi:10.1007/978-3-319-78978-1_5 · Cited by 56 · see citing works
FACTORS THAT AFFECT TRUST AND RELIANCE ON AN AUTOMATED AID
Sánchez, J. (2006) · UPT. Syiah Kuala University Library (Syiah Kuala University) · Cited by 24 · see citing works
The Influence of a Robot's Embodiment on Trust: A Longitudinal Study
van Maris, A., Lehmann, H., Natale, L., & Grzyb, B. (2017) · Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3029798.3038435 · Cited by 24 · see citing works
Self-citationFoundations of Insurance Economics
Cross, M. L., Dionne, G., & Harrington, S. E. (1993) · Journal of Risk & Insurance · doi:10.2307/253046 · Cited by 10 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
How to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies
Vereschak, O., Bailly, G., & Caramiaux, B. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476068 · Cited by 229 · see citing works
Banovic, N., Yang, Z., Ramesh, A., & Liu, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579460 · Cited by 55 · see citing works
- Authors
- Bridgwater, T., Giuliani, M., van Maris, A., Baker, G., Winfield, A., & Pipe, T.
- Affiliations
- Bristol Robotics Laboratory (GB)
- University of the West of England (GB)
- Bristol Robotics Laboratory, Bristol, United Kingdom
- Venue
- Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction
- Publisher
- ACM
- Published
- 2020-03-09
- Pages
- 23-31
- OpenAlex
- W3010330612
- Cited by
- 14
- References
- 33
- Citations by year
- 2025: 4 · 2024: 3 · 2023: 1 · 2022: 3 · 2021: 2 · 2020: 1
- FWCI
- 2.47
- Type
- conference paper
- Language
- en
- Open access
- Locations
- ACM/IEEE International Conference on Human-Robot Interaction (HRI) (publishedVersion)
- UWE Research Repository (UWE Bristol) (submittedVersion, cc-by)
- Funders
- EPSRC (EP/P01366X/1)
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Physical Sciences › Engineering › Mechanical Engineering › Teleoperation and Haptic Systems
- Keywords
- risk-based decision making, trust in robots, risk aversion, risk seeking, risk neutrality, teleoperation, hazardous environments, performance-based pay
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
