Trust calibration within a human-robot team: Comparing automatically generated explanations
Abstract
Trust is a critical factor for achieving the full potential of human-robot teams. Researchers have theorized that people will more accurately trust an autonomous system, such as a robot, if they have a more accurate understanding of its decision-making process. Studies have shown that hand-crafted explanations can help maintain trust when the system is less than 100% reliable. In this work, we leverage existing agent algorithms to provide a domain-independent mechanism for robots to automatically generate such explanations. To measure the explanation mechanism's impact on trust, we collected self-reported survey data and behavioral data in an agent-based online testbed that simulates a human-robot team task. The results demonstrate that the added explanation capability led to improvement in transparency, trust, and team performance. Furthermore, by observing the different outcomes due to variations in the robot's explanation content, we gain valuable insight that can help lead to refinement of explanation algorithms to further improve human-robot trust calibration.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Erratic behaviors of robots negatively influence people's trust towards them.
“In terms of the capability of a robot, erratic behaviors of robots were shown to negatively influence people's trust towards them [176].”
From A Taxonomy of Social Errors in Human-Robot Interaction (2021) · cites this paper as [176]
Model trustworthiness emerges as the primary predictor of AI trust, in line with previous studies in human-robot interaction scenarios.
“Model trustworthiness emerges as the primary predictor of AI trust, in line with previous studies in human-robot interaction scenarios [43].”
From AI Trust: Can Explainable AI Enhance Warranted Trust? (2023) · cites this paper as [43]
Some authors use correlations between perceived trustworthiness and system capabilities to measure calibration; larger correlation indicates better calibration.
“Some authors work with correlations to get a better sense of perceived trustworthiness and system capabilities [5,12,24,39,66,67,81,94,130,139,142].Here the larger the correlation between perceived trustworthiness and system capabilities, the better the calibration.”
From Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions (2023) · cites this paper as [130]
Wang et al. found explanations were only beneficial for low reliable systems and very reliable systems did not benefit.
“Wang et al. [130] found, for example, that explanations were only benefcial for low reliable systems and that very reliable systems did not beneft from the explanation.”
From Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions (2023) · cites this paper as [130]
Results support that users can and do differentiate between low and highly reliable systems, with more reliable systems perceived as more trustworthy than less reliable systems.
“Results of these eforts support the notion that users can and do diferentiate between low and highly reliable systems (see e.g., [19,20,69,130]), with more reliable systems being perceived as more trustworthy than less reliable systems.”
From Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions (2023) · cites this paper as [130]
A counterfactual explanation of an ML-based algorithm prediction may describe the smallest change to the feature values that changes the prediction to a predefined output.
“For instance, a counterfactual explanation of an ML-based algorithm prediction may describe ''the smallest change to the feature values that changes the prediction to a predefined output'' [53].”
From A Survey of Contrastive and Counterfactual Explanation Generation Methods for Explainable Artificial Intelligence (2021) · cites this paper as [53]
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · see citing works
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736 · see citing works
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · see citing works
Developing and validating trust measures for e-commerce: an integrative typology
McKnight, D. H., Choudhury, V., & Kacmar, C. (2002) · Information Systems Research · doi:10.1287/isre.13.3.334.81 · Cited by 5,550 · see citing works
Humans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · see citing works
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149 · see citing works
The effect of the performance appraisal system on trust for management: A field quasi-experiment.
Mayer, R. C., & Davis, J. H. (1999) · Journal of Applied Psychology · doi:10.1037/0021-9010.84.1.123 · Cited by 2,248 · see citing works
Trust, control strategies and allocation of function in human-machine systems
LEE, J., & MORAY, N. (1992) · Ergonomics · doi:10.1080/00140139208967392 · Cited by 1,459 · see citing works
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · see citing works
Trust between humans and machines, and the design of decision aids
Muir, B. M. (1987) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(87)80013-5 · Cited by 767 · see citing works
Towards robotic assistants in nursing homes: Challenges and results
Pineau, J., Montemerlo, M., Pollack, M., Roy, N., & Thrun, S. (2003) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(02)00381-0 · Cited by 640 · see citing works
Situational Awareness Rating Technique (Sart): The Development of a Tool for Aircrew Systems Design
Taylor, R. (2017) · Routledge eBooks · doi:10.4324/9781315087924-8 · Cited by 530 · see citing works
Salem, M., Lakatos, G., Amirabdollahian, F., & Dautenhahn, K. (2015) · Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/2696454.2696497 · Cited by 494 · see citing works
Acting under uncertainty: discrete Bayesian models for mobile-robot navigation
Cassandra, A. R., Kaelbling, L. P., & Kurien, J. (2002) · doi:10.1109/iros.1996.571080 · Cited by 463 · see citing works
The effects of transparency on trust in and acceptance of a content-based art recommender
Cramer, H., Evers, V., Ramlal, S., van Someren, M., Rutledge, L., Stash, N., Aroyo, L., & Wielinga, B. (2008) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-008-9051-3 · Cited by 448 · see citing works
Probability Information in Risk Communication: A Review of the Research Literature
Visschers, V. H. M., Meertens, R. M., Passchier, W. F., & de Vries, N. K. (2008) · Risk Analysis · doi:10.1111/j.1539-6924.2008.01137.x · Cited by 390 · see citing works
Measurement of trust in human-robot collaboration
Freedy, A., deVisser, E., Weltman, G., & Coeyman, N. (2007) · International Conference on Collaboration Technologies and Systems (CTS) · doi:10.1109/cts.2007.4621745 · Cited by 308 · see citing works
Groom, V., & Nass, C. (2007) · Interaction Studies Social Behaviour and Communication in Biological and Artificial Systems · doi:10.1075/is.8.3.10gro · Cited by 306 · see citing works
The Impact of Explanation Facilities on User Acceptance of Expert Systems Advice
Ye, L. R., & Johnson, P. E. (1995) · MIS Quarterly · doi:10.2307/249686 · Cited by 290 · see citing works
Coping with uncertainty: the construction and validation of a new measure
Greco, V., & Roger, D. (2001) · Personality and Individual Differences · doi:10.1016/s0191-8869(00)00156-2 · Cited by 279 · see citing works
de Vries, P., Midden, C., & Bouwhuis, D. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00039-9 · Cited by 264 · see citing works
You Want Me to Trust a ROBOT? The Development of a Human–Robot Interaction Trust Scale
Yagoda, R. E., & Gillan, D. (2012) · International Journal of Social Robotics · doi:10.1007/s12369-012-0144-0 · Cited by 252 · see citing works
Impact of robot failures and feedback on real-time trust
Desai, M., Kaniarasu, P., Medvedev, M., Steinfeld, A., & Yanco, H. (2013) · 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2013.6483596 · Cited by 234 · see citing works
Syrdal, D. S., Dautenhahn, K., Koay, K. L., & Walters, M. L. (2009) · University of Hertfordshire Research Archive (University of Hertfordshire) · Cited by 188 · see citing works
Effects of changing reliability on trust of robot systems
Desai, M., Medvedev, M., Vázquez, M., McSheehy, S., Gadea-Omelchenko, S., Bruggeman, C., Steinfeld, A., & Yanco, H. (2012) · Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction · doi:10.1145/2157689.2157702 · Cited by 188 · see citing works
PsychSim: modeling theory of mind with decision-theoretic agents
Pynadath, D. V., & Marsella, S. (2005) · Cited by 162 · see citing works
Self-citationExplanation in Second Generation Expert Systems
Swartout, W. R., & Moore, J. D. (1993) · Second Generation Expert Systems · doi:10.1007/978-3-642-77927-5_24 · Cited by 160 · see citing works
Formats for Improving Risk Communication in Medical Tradeoff Decisions
Waters, E. A., Weinstein, N. D., Colditz, G. A., & Emmons, K. M. (2006) · Journal of Health Communication · doi:10.1080/10810730500526695 · Cited by 159 · see citing works
Relative importance of scenario information and frequency information in the judgment of risk
Hendrickx, L., Vlek, C. A. J., & Oppewal, H. (1989) · Acta Psychologica · doi:10.1016/0001-6918(89)90050-4 · Cited by 158 · see citing works
BiLAT: A Game-Based Environment for Practicing Negotiation in a Cultural Context
Kim, J. M., Hill, R. W., Durlach, P. J., Lane, H. C., Forbell, E., Core, M. G., Marsella, S., Pynadath, D. V., & Hart, J. (2009) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2009-19(3)03 · Cited by 155 · see citing works
Self-citationXavier: a robot navigation architecture based on partially observable Markov decision process models
Koenig, S., & Simmons, R. G. (1998) · Cited by 147 · see citing works
Xu, A., & Dudek, G. (2015) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/2696454.2696492 · Cited by 143 · see citing works
PsychSim: Agent-based Modeling of Social Interactions and Influence.
Pynadath, D. V. (2004) · International Conference on Cognitive Modelling · Cited by 141 · see citing works
Self-citationThe Perception And Measurement Of Human-robot Trust
Schaefer, K. E. (2013) · Journal of International Crisis and Risk Communication Research · Cited by 138 · see citing works
(1995) · Cited by 128 · see citing works
Tactical Language and Culture Training Systems: Using AI to Teach Foreign Languages and Cultures
Johnson, W. L., & Valente, A. X. C. N. (2009) · AI Magazine · doi:10.1609/aimag.v30i2.2240 · Cited by 114 · see citing works
(2018) · doi:10.1201/9781315137957-2 · Cited by 108 · see citing works
Building appropriate trust in human-‐robot teams
Ososky, S., Schuster, D. M., Phillips, E. K., & Jentsch, F. G. (2013) · Journal of International Crisis and Risk Communication Research · Cited by 79 · see citing works
MEASURING TRUST OF HUMAN OPERATORS IN NEW GENERATION RESCUE ROBOTS
Park, E., JENKINS, Q., & Jiang, X. (2008) · Proceedings of the JFPS International Symposium on Fluid Power · doi:10.5739/isfp.2008.489 · Cited by 39 · see citing works
A Natural Language Argumentation Interface for Explanation Generation in Markov Decision Processes
Dodson, T. B., Mattei, N., & Goldsmith, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24873-3_4 · Cited by 37 · see citing works
Socially Optimized Learning in Virtual Environments (SOLVE)
Miller, L. C., Marsella, S., Dey, T., et al. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-25289-1_20 · Cited by 27 · see citing works
Negotiations in the Context of AIDS Prevention: An Agent-Based Model Using Theory of Mind
Klatt, J., Marsella, S., & Krämer, N. C. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23974-8_23 · Cited by 21 · see citing works
Trust, self-confidence and supervisory control in a process control simulation
Lee, J., & Moray, N. (2002) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.1991.169700 · Cited by 17 · see citing works
UrbanSim: A Game-based Simulation for Counterinsurgency and Stability-focused Operations
McAlinden, R., Gordon, A. S., Lane, H. C., et al. (2009) · Cited by 17 · see citing works
Intelligent Agents for Virtual Simulation of Human-Robot Interaction
Wang, N., Pynadath, D. V., Unnikrishnan, K. V., Shankar, S., & Merchant, C. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-21067-4_24 · Cited by 14 · see citing works
Self-citationAutomatically Generated Explanations for Markov Decision Processes
Khan, O. Z., Poupart, P., & Black, J. P. (2011) · IGI Global eBooks · doi:10.4018/978-1-60960-165-2.ch007 · Cited by 13 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Stepin, I., Alonso, J. M., Catala, A., & Pereira-Farina, M. (2021) · IEEE Access · doi:10.1109/access.2021.3051315 · Cited by 376 · see citing works
“Furthermore, it prevents users from a responsible exploitation of their decisions [3].”
BackgroundSystematic Review: Trust-Building Factors and Implications for Conversational Agent Design
Rheu, M., Shin, J. Y., Peng, W., & Huh-Yoo, J. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1807710 · Cited by 360 · see citing works
Naiseh, M., Al-Thani, D., Jiang, N., & Ali, R. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102941 · Cited by 170 · see citing works
Wischnewski, M., Krämer, N., & Müller, E. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581197 · Cited by 146 · see citing works
“Some authors work with correlations to get a better sense of perceived trustworthiness and system capabilities [5, 12, 24, 39, 66, 67, 81, 94, 130, 139, 142].”
BackgroundMethodInfluentialA Taxonomy of Social Errors in Human-Robot Interaction
Tian, L., & Oviatt, S. (2021) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3439720 · Cited by 133 · see citing works
Esterwood, C., & Jr, L. P. R. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.107658 · Cited by 98 · see citing works
Establishing Appropriate Trust via Critical States
Huang, S. H., Bhatia, K., Abbeel, P., & Dragan, A. D. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593649 · Cited by 94 · see citing works
Aroyo, A. M., Rea, F., Sandini, G., & Sciutti, A. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2856272 · Cited by 85 · see citing works
Effective human–AI work design for collaborative decision-making
Jain, R., Garg, N., & Khera, S. N. (2023) · Kybernetes · doi:10.1108/k-04-2022-0548 · Cited by 85 · see citing works
Do You Still Trust Me? Human-Robot Trust Repair Strategies
Esterwood, C., & Robert, L. P. (2021) · 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN) · doi:10.1109/ro-man50785.2021.9515365 · Cited by 75 · see citing works
AI Trust: Can Explainable AI Enhance Warranted Trust?
de Brito Duarte, R., Correia, F., Arriaga, P., & Paiva, A. (2023) · Human Behavior and Emerging Technologies · doi:10.1155/2023/4637678 · Cited by 52 · see citing works
McGrath, M. J., Duenser, A., Lacey, J., & Paris, C. (2025) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2025.100200 · Cited by 51 · see citing works
Bobko, P., Hirshfield, L., Eloy, L., Spencer, C., Doherty, E., Driscoll, J., & Obolsky, H. (2023) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922x.2022.2086644 · Cited by 33 · see citing works
Cheon, E., Zaga, C., Lee, H. R., Lupetti, M. L., Dombrowski, L., & Jung, M. F. (2021) · Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/3462204.3481726 · Cited by 29 · see citing works
An explanation is not an excuse: Trust calibration in an age of transparent robots
Wagner, A. R., & Robinette, P. (2021) · Trust in Human-Robot Interaction · doi:10.1016/b978-0-12-819472-0.00009-5 · Cited by 20 · see citing works
McMahon, G., Akash, K., Reid, T., & Jain, N. (2020) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2021.04.113 · Cited by 10 · see citing works
- Authors
- Wang, N., Pynadath, D. V., & Hill, S. G.
- Affiliations
- University of Southern California (US)
- ORCID
- Pynadath, David V.
- Venue
- 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI)
- Publisher
- IEEE
- Published
- 2016-03
- Pages
- 109-116
- OpenAlex
- W4238869533
- Cited by
- 220
- References
- 46
- Citations by year
- 2026: 21 · 2025: 21 · 2024: 32 · 2023: 31 · 2022: 34 · 2021: 26 · 2020: 31 · 2019: 15 · 2018: 6 · 2017: 3
- FWCI
- 4.23
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Human-Robot Interaction (HRI) (publishedVersion)
- Funders
- Oak Ridge Associated Universities
- Army Research Laboratory
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Keywords
- human-robot teams, trust calibration, explainable AI, robot transparency, team performance, autonomous systems, human-robot interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
