Establishing Appropriate Trust via Critical States
The DOI matches its Crossref record; 28 of the 31 references it cites were found in a registry or an index.
Abstract
In order to effectively interact with or supervise a robot, humans need to have an accurate mental model of its capabilities and how it acts. Learned neural network policies make that particularly challenging. We propose an approach for helping end-users build a mental model of such policies. Our key observation is that for most tasks, the essence of the policy is captured in a few critical states: states in which it is very important to take a certain action. Our user studies show that if the robot shows a human what its understanding of the task's critical states is, then the human can make a more informed decision about whether to deploy the policy, and if she does deploy it, when she needs to take control from it at execution time.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Human-AI collaborative decision-making tools are usually built to support trust calibration through the recommendations interface design [10, 35, 70].”
From Explainable recommendation: when design meets trust calibration (2021) · cites this paper as [35]
In the citing paper
“A third way to measure the trust calibration was to measure over-, under-, and appropriate trust as the behavioral deviation (reliance) from the ideal behavior [27,50,55,104,110,132,137].”
From Measuring and Understanding Trust Calibrations for Automated Systems: A Survey of the State-Of-The-Art and Future Directions (2023) · cites this paper as [50]
Citations
Cites
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 1×
“In particular, giving human end-users an accurate mental model of a robot’s capabilities is key to establishing an appropriate level of trust in the robot [8], [9], [10].”
BackgroundSoft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594
Self-citationEnd to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198
What trust means in e-commerce customer relationships: an interdisciplinary conceptual typology
McKnight, D. H., & Chervany, N. L. (2001) · International Journal of Electronic Commerce · doi:10.1080/10864415.2001.11044235 · Cited by 2,282
Maximum Entropy Inverse Reinforcement Learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2018) · doi:10.1184/r1/6555512 · Cited by 1,869
End-to-End Training of Deep Visuomotor Policies
Levine, S., Finn, C., Darrell, T. J., & Abbeel, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00702 · Cited by 1,344
Self-citationThe role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276
Action understanding as inverse planning
Baker, C. L., Saxe, R., & Tenenbaum, J. B. (2009) · Cognition · doi:10.1016/j.cognition.2009.07.005 · Cited by 920
Trust between humans and machines, and the design of decision aids
Muir, B. M. (1987) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(87)80013-5 · Cited by 762
Research works printed but not linked: 2
- 10.1518/hfes.46.1.50.30392 · doi:10.1518/hfes.46.1.50.30392 · Paper page
- israelsen, 2017, I can assure you…that its going to be all right, A definition case for and survey of algorithmic assurances in human-autonomy trust relationships
Cited by
DARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360
Towards a Theory of Longitudinal Trust Calibration in Human–Robot Teams
de Visser, E. J., Peeters, M. M. M., Jung, M. F., Kohn, S., Shaw, T. H., Pak, R., & Neerincx, M. A. (2020) · International Journal of Social Robotics · doi:10.1007/s12369-019-00596-x · Cited by 447
Explainable Reinforcement Learning: A Survey and Comparative Review
Milani, S., Topin, N., Veloso, M., & Fang, F. (2024) · ACM Computing Surveys · doi:10.1145/3616864 · Cited by 224
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
Trust in Robots: Challenges and Opportunities
Kok, B. C., & Soh, H. (2020) · Current Robotics Reports · doi:10.1007/s43154-020-00029-y · Cited by 221
Metadata
- Authors
- Huang, S. H., Bhatia, K., Abbeel, P., & Dragan, A. D.
- Affiliations
- University of California, Berkeley (US)
- Venue
- 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Publisher
- IEEE
- Published
- 2018-10
- Pages
- 3929-3936
- ISSN
- 2153-0858 (ISSN-L)
- OpenAlex
- W2897798332
- Cited by
- 94
- References
- 31
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 13 · 2023: 14 · 2022: 7 · 2021: 16 · 2020: 13 · 2019: 12
- FWCI
- 18.35
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- trust in robots, critical state, mental models, neural network policy, robot capabilities, human-robot interaction, human-in-the-loop control
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Establishing Appropriate Trust via Critical States. https://app.cochat.ai/papers/10.1109/iros.2018.8593649
- The paper
- Huang, S. H., Bhatia, K., Abbeel, P., & Dragan, A. D. (2018). Establishing Appropriate Trust via Critical States. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). https://doi.org/10.1109/iros.2018.8593649
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.