Demand-Driven Transparency for Monitoring Intelligent Agents
Abstract
In autonomous multiagent or multirobotic systems, the ability to quickly and accurately respond to threats and uncertainties is important for both mission outcomes and survivability. Such systems are never truly autonomous, often operating as part of a human-agent team. Artificial intelligent agents (IAs) have been proposed as tools to help manage such teams; e.g., proposing potential courses of action to human operators. However, they are often underutilized due to a lack of trust. Designing transparent agents, who can convey at least some information regarding their internal reasoning processes, is considered an effective method of increasing trust. How people interact with such transparency information to gain situation awareness while avoiding information overload is currently an unexplored topic. In this article, we go part way to answering this question, by investigating two forms of transparency: sequential transparency, which requires people to step through the IA's explanation in a fixed order; and demand-driven transparency, which allows people to request information as needed. In an experiment using a multivehicle simulation, our results show that demand-driven interaction improves the operators' trust in the system while maintaining, and at times improving, performance and usability.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Evaluative AI provides decision makers with better control and follows a process they will naturally follow, unlike recommendation-driven approaches which disrupt the decision-making process.
“However, one argument against this is that this evaluative AI provides decision makers with better control [36,43], and follows a process that they will naturally follow, rather than recommendation-driven approaches, which somewhat disrupt the decision-making process.”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [43]
Cites
Toward a Theory of Situation Awareness in Dynamic Systems
Endsley, M. R. (1995) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872095779049543 · Cited by 8,875 · see citing works
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 3× · see citing works
“These results are consistent with previous studies that suggest that the more the participants trust the IA, the more likely they are to accept the IA’s recommendations [4].”
BackgroundResultInfluentialExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · see citing works
Self-citationHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · see citing works
Multiple Resources and Mental Workload
Wickens, C. D. (2008) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872008x288394 · Cited by 2,164 · see citing works
Foundations for an Empirically Determined Scale of Trust in Automated Systems
Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000) · International Journal of Cognitive Ergonomics · doi:10.1207/s15327566ijce0401_04 · Cited by 1,799 · see citing works
An overview of online trust: Concepts, elements, and implications
Wang, Y. D., & Emurian, H. H. (2004) · Computers in Human Behavior · doi:10.1016/j.chb.2003.11.008 · Cited by 998 · see citing works
Casper, J. L., & Murphy, R. R. (2003) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2003.811794 · Cited by 942 · see citing works
Trust, self-confidence, and operators' adaptation to automation
Lee, J. D., & Moray, N. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1007 · Cited by 932 · see citing works
MUIR, B. M. (1994) · Ergonomics · doi:10.1080/00140139408964957 · Cited by 873 · cited 1× · see citing works
“For this collaboration to work, the human operator must rely on the IAs to perform part of the monitoring and supervision task [22].”
BackgroundMaking Sense of Sensemaking 1: Alternative Perspectives
Klein, G. A., Moon, B., & Hoffman, R. R. (2006) · IEEE Intelligent Systems · doi:10.1109/mis.2006.75 · Cited by 812 · see citing works
Wizard of Oz Studies in HRI: A Systematic Review and New Reporting Guidelines
Riek, L. D. (2012) · Journal of Human-Robot Interaction · doi:10.5898/jhri.1.1.riek · Cited by 778 · see citing works
Controlling the Information Flow: Effects on Consumers' Decision Making and Preferences
Ariely, D. (2000) · Journal of Consumer Research · doi:10.1086/314322 · Cited by 724 · see citing works
How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615 · see citing works
The Cambridge Handbook of Expertise and Expert Performance
(2018) · Cambridge University Press eBooks · doi:10.1017/9781316480748 · Cited by 614 · see citing works
Nass, C., & Lee, K. M. (2001) · Journal of Experimental Psychology: Applied · doi:10.1037/1076-898x.7.3.171 · Cited by 522 · see citing works
A Cognitive Approach to Situation Awareness: Theory and Application
Banbury, S. P. (2017) · Routledge eBooks · doi:10.4324/9781315263977 · Cited by 408 · see citing works
Intelligent agent transparency in human–agent teaming for multi-UxV management
Mercado, J. E., Rupp, M. A., Chen, J. Y. C., Barnes, M. J., Barber, D., & Procci, K. (2016) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720815621206 · Cited by 403 · see citing works
Too much, too little, or just right? Ways explanations impact end users' mental models
Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., & Wong, W.-K. (2013) · 2013 IEEE Symposium on Visual Languages and Human Centric Computing · doi:10.1109/vlhcc.2013.6645235 · Cited by 388 · see citing works
Improving Robot Controller Transparency Through Autonomous Policy Explanation
Hayes, B., & Shah, J. A. (2017) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1145/2909824.3020233 · Cited by 359 · see citing works
Situation awareness-based agent transparency and human-autonomy teaming effectiveness
Chen, J. Y. C., Lakhmani, S. G., Stowers, K., Selkowitz, A. R., Wright, J. L., & Barnes, M. (2018) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922x.2017.1315750 · Cited by 341 · see citing works
Information Displays and Decision Processes
Kleinmuntz, D. N., & Schkade, D. A. (1993) · Psychological Science · doi:10.1111/j.1467-9280.1993.tb00265.x · Cited by 309 · see citing works
A predictive model of menu performance
Cockburn, A., Gutwin, C., & Greenberg, S. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240723 · Cited by 306 · cited 1× · see citing works
“interaction [11], [12], [32].”
Bahner, J. E., Hüper, A.-D., & Manzey, D. (2008) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2008.06.001 · Cited by 279 · see citing works
Superior performance and neural efficiency: The impact of intelligence and expertise
Grabner, R. H., Neubauer, A. C., & Stern, E. (2006) · Brain Research Bulletin · doi:10.1016/j.brainresbull.2006.02.009 · Cited by 228 · see citing works
Situation Awareness: Proceed with Caution
Flach, J. M. (1995) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872095779049480 · Cited by 198 · see citing works
FORUM: Why are people turning off our alarms?
Sorkin, R. D. (1988) · The Journal of the Acoustical Society of America · doi:10.1121/1.397232 · Cited by 158 · see citing works
Predicting Controller Capacity in Supervisory Control of Multiple UAVs
Cummings, M. L., & Mitchell, P. J. (2008) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/tsmca.2007.914757 · Cited by 140 · see citing works
Dynamic versus static menus: an exploratory comparison
Mitchell, J., & Shneiderman, B. (1989) · ACM SIGCHI Bulletin · doi:10.1145/67243.67247 · Cited by 120 · see citing works
Intelligent agent supporting human–multi-robot team collaboration
Rosenfeld, A., Agmon, N., Maksimov, O., & Kraus, S. (2017) · Artificial Intelligence · doi:10.1016/j.artint.2017.08.005 · Cited by 113 · see citing works
Review and reappraisal of adaptive interfaces: Toward biologically inspired paradigms
Rothrock, L., Koubek, R. J., Fuchs, F., Haas, M., & Salvendy, G. (2002) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220110110342 · Cited by 104 · see citing works
Integrated Data Management for a Fleet of Search‐and‐rescue Robots
Balta, H., Będkowski, J., Govindaraj, S., Majek, K., Musialik, P., Serrano, D., Alexis, K., Siegwart, R., & Cubber, G. D. (2016) · Journal of Field Robotics · doi:10.1002/rob.21651 · Cited by 104 · see citing works
Chen, J. Y. C., & Terrence, P. (2009) · Ergonomics · doi:10.1080/00140130802680773 · Cited by 99 · see citing works
Human Interaction With Multiple Remote Robots
Lewis, M. (2013) · Reviews of Human Factors and Ergonomics · doi:10.1177/1557234x13506688 · Cited by 80 · see citing works
On using mixed-initiative control
Hardin, B., & Goodrich, M. A. (2009) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/1514095.1514126 · Cited by 70 · see citing works
Engineering Trust in Complex Automated Systems
Lyons, J. B., Koltai, K., Ho, N., Johnson, W. B., Smith, D. E., & Shively, R. J. (2016) · Ergonomics in Design The Quarterly of Human Factors Applications · doi:10.1177/1064804615611272 · Cited by 63 · see citing works
Current Trends in Adaptive User Interfaces: Challenges and Applications
Álvarez-Cortés, V., Zayas-Pérez, B. E., Zarate, V. H., & Uresti, J. A. R. (2007) · Electronics, Robotics and Automotive Mechanics Conference · doi:10.1109/cerma.2007.4367705 · Cited by 54 · see citing works
Squire, P., & Parasuraman, R. (2010) · Ergonomics · doi:10.1080/00140139.2010.489969 · Cited by 52 · see citing works
Gigerenzer, G. (2020) · Topics in Cognitive Science · doi:10.1111/tops.12480 · Cited by 48 · see citing works
Multi‐robot team formation control in the GUARDIANS project
Sàez-Pons, J., Alboul, L., Penders, J., & Nomdedeu, L. (2010) · Industrial Robot the international journal of robotics research and application · doi:10.1108/01439911011044831 · Cited by 45 · see citing works
Intelligent Agent Transparency
Stowers, K. L., Kasdaglis, N., Newton, O. B., Lakhmani, S. G., Wohleber, R. W., & Chen, J. Z.-J. (2016) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1541931213601392 · Cited by 42 · see citing works
Helldin, T. (2014) · University Library of Skövde (University of Skövde) · Cited by 35 · see citing works
Dodge, J. E., Penney, S., Hilderbrand, C., Anderson, A. W., & Burnett, M. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174136 · Cited by 31 · see citing works
Autonomous Self-Explanation of Behavior for Interactive Reinforcement Learning Agents
Fukuchi, Y., Osawa, M., Yamakawa, H., & Imai, M. (2017) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3125739.3125746 · Cited by 30 · see citing works
Toward Foraging for Understanding of StarCraft Agents
Penney, S., Dodge, J. E., Hilderbrand, C., Anderson, A. W., Simpson, L., & Burnett, M. (2018) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3172944.3172946 · Cited by 30 · see citing works
Coordination in Human-Agent-Robot Teamwork
Bradshaw, J. M., Feltovich, P. J., Johnson, M., Bunch, L., Breedy, M. R., Eskridge, T. C., Jung, H., Lott, J., & Uszok, A. (2008) · International Conference on Collaboration Technologies and Systems (CTS) · doi:10.1109/cts.2008.4543966 · Cited by 28 · see citing works
Transparency of Automated Combat Classification
Helldin, T., Ohlander, U., Falkman, G., & Riveiro, M. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07515-0_3 · Cited by 18 · see citing works
Situation awareness-based agent transparency for human-autonomy teaming effectiveness
Chen, J. Y. C., Barnes, M. J., Wright, J. L., Stowers, K. L., & Lakhmani, S. G. (2017) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2263194 · Cited by 18 · see citing works
Current Challenges and Applications for Adaptive User Interfaces
Álvarez-Cortés, V., Victor, H., Uresti, J. A. R., & Benjamin, E. (2009) · InTech eBooks · doi:10.5772/7745 · Cited by 17 · see citing works
Definitions and Dimensions of Etiquette
Miller, C. A. (2002) · Cited by 13 · see citing works
Design and Experimental Validation of Transparent Behavior for a Workload-Adaptive Cognitive Agent
Brand, Y., Ebersoldt, M., Barber, D. J., Chen, J. Y. C., & Schulte, A. (2017) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-73888-8_28 · Cited by 6 · see citing works
An Adaptive User Interface Based on Psychological Test and Task-Relevance
Riascos, J. A., Nedel, L., & Barone, D. C. (2017) · Communications in computer and information science · doi:10.1007/978-3-319-71011-2_12 · Cited by 4 · see citing works
Flagged first, then by citations
1 of the 53 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172 · see citing works
Self-citationGraziani, M., Dutkiewicz, L., Calvaresi, D., Amorim, J. P., Yordanova, K., Vered, M., Nair, R. S., Abreu, P. H., Blanke, T., Pulignano, V., et al. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10256-8 · Cited by 138 · see citing works
Self-citationCaldwell, S., Sweetser, P., O’Donnell, N., Knight, M. J., Aitchison, M., Gedeon, T., Johnson, D. M., Brereton, M. F., Gallagher, M. R., & Conroy, D. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3514257 · Cited by 82 · see citing works
The effects of explanations on automation bias
Vered, M., Livni, T., Howe, P. D. L., Miller, T., & Sonenberg, L. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.103952 · Cited by 71 · see citing works
Self-citationSaghafian, M., Vatn, D. M. K., Moltubakk, S. T., Bertheussen, L. E., Petermann, F. M., Johnsen, S. O., & Alsos, O. A. (2024) · Safety Science · doi:10.1016/j.ssci.2024.106730 · Cited by 20 · see citing works
Chen, J. Y. C., & Barnes, M. (2021) · Handbook of Human Factors and Ergonomics · doi:10.1002/9781119636113.ch44 · Cited by 1 · see citing works
van de Merwe, K., Mallam, S., Nazir, S., & Engelhardtsen, Ø. (2024) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/15553434241240553 · Cited by 17 · see citing works
A Systematic Review and Taxonomy of Human–Agent Teaming Testbeds
Chung, H., Holder, T., Shah, J., & Yang, X. J. (2025) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208251376898 · Cited by 12 · see citing works
Examining and Promoting Explainable Recommendations for Personal Sensing Technology Acceptance
Newn, J., Kelly, R., D’Alfonso, S., & Lederman, R. (2022) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3550297 · Cited by 12 · see citing works
Behaviour Recognition with Kinodynamic Planning Over Continuous Domains
Fitzpatrick, G., Lipovetzky, N., Papasimeon, M., Ramírez, M., & Vered, M. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.717003 · Cited by 10 · see citing works
Self-citationFang, R., Zhou, Q., Zhou, C., Yuan, S., Wang, K., Li, Q., Zhang, Y., & Li, J. (2025) · PsyCh Journal · doi:10.1002/pchj.70027 · Cited by 6 · see citing works
Nahulanthran, S., Vered, M., Tian, L., & Kulić, D. (2024) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii63134.2024.00017 · Cited by 3 · see citing works
Self-citationTatasciore, M., Bennett, L., Bowden, V. K., Bell, J., Visser, T. A. W., McAnally, K. I., McCarley, J. S., Thompson, M. B., Shanahan, C., Morris, R. A., et al. (2025) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/00187208251349269 · Cited by 3 · see citing works
Explainable Goal Recognition: A Framework Based on Weight of Evidence
Alshehri, A. S., Miller, T., & Vered, M. (2023) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v33i1.27173 · Cited by 3 · see citing works
Self-citationNavigators’ Perspective on Information Requirements for Supervisory Control of Autonomous Ships
van de Merwe, K., Mallam, S., Nazir, S., & Engelhardtsen, Ø. (2024) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813241261686 · Cited by 3 · see citing works
Agent Transparency for Human-Autonomy Teaming
Chen, J. Y. C., & Barnes, M. J. (2022) · Automation, collaboration, and e-services · doi:10.1007/978-3-031-10788-7_15 · Cited by 2 · see citing works
Fairhurst, M., Kaesling, K., & Klös, V. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-08330-2_17 · Cited by 1 · see citing works
Sonenberg, L. (2023) · Logic Journal of IGPL · doi:10.1093/jigpal/jzad006
Self-citationAgent Transparency and Human Performance in the Context of Autonomous Collision Avoidance
van de Merwe, K., Nazir, S., Mallam, S., & Engelhardtsen, Ø. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-12392-3_17
Explaining Facial Expression Recognition for Human Understanding and Model PersonalisationPreprint
Nahulanthran, S., Vered, M., Kulić, D., & Tian, L. (2026) · Research Square · doi:10.21203/rs.3.rs-9582396/v1
Self-citationHart, S., Steane, V., Parnell, K. J., Plant, K. L., Soorati, M. D., & Young, M. S. (2026) · Theoretical Issues in Ergonomics Science · doi:10.1080/1463922x.2026.2718460
- Authors
- Vered, M., Howe, P., Miller, T., Sonenberg, L., & Velloso, E.
- Affiliations
- Monash University (AU)
- The University of Melbourne (AU)
- Venue
- IEEE Transactions on Human-Machine Systems
- Published
- 2020-06
- Volume
- 50
- Issue
- 3
- Pages
- 264-275
- ISSN
- 2168-2291 (ISSN-L)
- 2168-2305
- OpenAlex
- W3027378508
- Cited by
- 21
- References
- 53
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 4 · 2023: 4 · 2022: 4 · 2021: 2
- FWCI
- 2.73
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Human-Machine Systems (publishedVersion)
- Funders
- Australia Defence Science and Technology
- Defence Science Institute
- Topics
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Engineering › Control and Systems Engineering › Military Strategy and Technology
- Keywords
- intelligent agents, human-agent teams, trust in autonomous systems, situational awareness, information overload, human-machine interaction, multi-robot systems, operator performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
