Assessing Demand for Transparency in Intelligent Systems Using Machine Learning
Open accessAbstract
Intelligent systems offering decision support can lessen cognitive load and improve the efficiency of decision making in a variety of contexts. These systems assist users by evaluating multiple courses of action and recommending the right action at the right time. Modern intelligent systems using machine learning introduce new capabilities in decision support, but they can come at a cost. Machine learning models provide little explanation of their outputs or reasoning process, making it difficult to determine when it is appropriate to trust, or if not, what went wrong. In order to improve trust and ensure appropriate reliance on these systems, users must be afforded increased transparency, enabling an understanding of the systems reasoning, and an explanation of its predictions or classifications. Here we discuss the salient factors in designing transparent intelligent systems using machine learning, and present the results of a user-centered design study. We propose design guidelines derived from our study, and discuss next steps for designing for intelligent system transparency.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Vorm reported that XAI could provide different information types to make intelligent systems explainable and more acceptable and trustworthy to users.
“They reported that XAI could provide different information types to make intelligent systems explainable and more acceptable and trustworthy to users.”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [106]
Therefore, XAI can support compliance with the GDPR.
“Therefore, XAI can support compliance with the GDPR [98,103,106].”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [106]
In 2016 the European Union passed the GDPR which has been interpreted as a requirement for any decision made based on an algorithm to be explainable to the user.
“As discussed in Section 2.2.1, in 2016 the European Union passed the GDPR which has been interpreted as a requirement for any decision made based on an algorithm to be explainable to the user [106].”
From Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review (2021) · cites this paper as [106]
Cites
Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory
Corbin, J., & Strauss, A. (2008) · SAGE · doi:10.4135/9781452230153 · Cited by 12,136 · see citing works
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · see citing works
Clark, H. H., & Brennan, S. E. (1991) · Perspectives on socially shared cognition. · doi:10.1037/10096-006 · Cited by 4,555 · see citing works
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · see citing works
Human-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015) · Science · doi:10.1126/science.aab3050 · Cited by 2,817 · see citing works
European Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · see citing works
Explaining collaborative filtering recommendations
Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000) · Proceedings of the 2000 ACM conference on Computer supported cooperative work · doi:10.1145/358916.358995 · Cited by 1,731 · see citing works
Rule-based expert systems: The mycin experiments of the stanford heuristic programming project
Swartout, W. (1985) · Artificial Intelligence · doi:10.1016/0004-3702(85)90067-0 · Cited by 1,248 · see citing works
The Study of Behavior. Q Technique and Its Methodology
Glueck, B. (1954) · American Journal of Psychiatry · doi:10.1176/ajp.111.6.479 · Cited by 644 · see citing works
Ten challenges for making automation a ‘team player’ in joint human-agent activity
Klein, G., Woods, D. D., Bradshaw, J. M., Hoffman, R. R., & Feltovich, P. J. (2004) · IEEE Intelligent Systems · doi:10.1109/mis.2004.74 · Cited by 590 · see citing works
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · see citing works
Explanations From Intelligent Systems: Theoretical Foundations and Implications for Practice1
Gregor, S., & Benbasat, I. (1999) · MIS Quarterly · doi:10.2307/249487 · Cited by 535 · see citing works
The role of transparency in recommender systems
Sinha, R. R., & Swearingen, K. (2002) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/506443.506619 · Cited by 480 · see citing works
Intelligibility and Accountability: Human Considerations in Context-Aware Systems
Bellotti, V., & Edwards, K. (2001) · Human–Computer Interaction · doi:10.1207/s15327051hci16234_05 · Cited by 471 · see citing works
Rogers, Y. (2012) · Synthesis lectures on human-centered informatics · doi:10.2200/s00418ed1v01y201205hci014 · Cited by 385 · see citing works
Decision theory in expert systems and artificial intelligence
Horvitz, E., Breese, J. S., & Henrion, M. (1988) · International Journal of Approximate Reasoning · doi:10.1016/0888-613x(88)90120-x · Cited by 368 · see citing works
Trust and etiquette in high-criticality automated systems
Parasuraman, R., & Miller, C. A. (2004) · Communications of the ACM · doi:10.1145/975817.975844 · Cited by 311 · see citing works
Reasoning in explanation-based decision making
Pennington, N. (1993) · Cognition · doi:10.1016/0010-0277(93)90038-w · Cited by 293 · see citing works
Assessing demand for intelligibility in context-aware applications
Lim, B. Y., & Dey, A. K. (2009) · Proceedings of the 11th international conference on Ubiquitous computing · doi:10.1145/1620545.1620576 · Cited by 246 · see citing works
Toward harnessing user feedback for machine learning
Stumpf, S., Rajaram, V., Li, L., Burnett, M., Dietterich, T. G., Sullivan, E., Drummond, R., & Herlocker, J. L. (2007) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1216295.1216316 · Cited by 134 · see citing works
Holtzblatt, K., & Beyer, H. R. (2014) · Synthesis lectures on human-centered informatics · doi:10.2200/s00597ed1v01y201409hci024 · Cited by 86 · see citing works
Trust in new decision aid systems
Atoyan, H., Duquet, J.-R., & Robert, J.-M. (2006) · Conference on l'Interaction Homme-Machine (IHM) · doi:10.1145/1132736.1132751 · Cited by 69 · see citing works
A co-operative computer based on the principles of human co-operation
Clarke, A., & Smyth, M. (1993) · International Journal of Man-Machine Studies · doi:10.1006/imms.1993.1002 · Cited by 55 · see citing works
Representation of electronic mail filtering profiles
Pazzani, M. J. (2000) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/325737.325843 · Cited by 46 · see citing works
Berendt, B., & Preibusch, S. (2014) · Artificial Intelligence and Law · doi:10.1007/s10506-013-9152-0 · Cited by 44 · see citing works
The role of transparency in recommender systems
Sinha, R. R., & Swearingen, K. (2002) · CHI '01 extended abstracts on Human factors in computer systems - CHI '01 · doi:10.1145/506558.506619 · Cited by 41 · see citing works
Making machine learning useable by revealing internal states update - a transparent approach
Zhou, J. J., Khawaja, M. A., Li, Z., Sun, J., Wang, Y., & Chen, F. (2016) · International Journal of Computational Science and Engineering · doi:10.1504/ijcse.2016.080214 · Cited by 27 · see citing works
Kaltenbach, E. K., & Dolgov, I. (2017) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1541931213601558 · Cited by 21 · see citing works
Toward Harnessing User Feedback For Machine Learning
Stumpf, S., Rajaram, V., Li, L., Burnett, M., Dietterich, T. G., Sullivan, E., Drummond, R., & Herlocker, J. L. (2006) · doi:10.21236/ada471484 · Cited by 20 · see citing works
The Trust Value Calculating for Social Network Based on Machine Learning
Yu-ji, W. (2017) · International Conference on Intelligent Human-Machine Systems and Cybernetics · doi:10.1109/ihmsc.2017.145 · Cited by 18 · see citing works
A Categorization of Explanation Questions for Task Processing Systems.
McGuinness, D. L., Glass, A., Wolverton, M. J., & da Silva, P. P. (2007) · Cited by 12 · see citing works
Transparency of Computational Intelligence Models
Owotoki, P., & Mayer-Lindenberg, F. (2007) · Springer eBooks · doi:10.1007/978-1-84628-663-6_29 · Cited by 7 · see citing works
The Crash of Korean Air Lines Flight 007
Degani, A. (2003) · Palgrave Macmillan US eBooks · doi:10.1057/9781403982520_5 · Cited by 1 · see citing works
Flagged first, then by citations
1 of the 34 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629 · see citing works
A Review of Trust in Artificial Intelligence: Challenges, Vulnerabilities and Future Directions
Lockey, S., Gillespie, N. A., Holm, D., & Someh, I. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.664 · Cited by 216 · see citing works
Utilization of Self-Diagnosis Health Chatbots in Real-World Settings: Case Study
Fan, X., Chao, D., Zhang, Z., Wang, D., Li, X., & Tian, F. (2020) · Journal of Medical Internet Research · doi:10.2196/19928 · Cited by 164 · see citing works
It's complicated: the relationship between user trust, model accuracy and explanations in AI
Papenmeier, A., Kern, D., Englebienne, G., & Seifert, C. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3495013 · Cited by 144 · see citing works
Effect of AI Explanations on Human Perceptions of Patient-Facing AI-Powered Healthcare Systems
Zhang, Z., Genc, Y., Wang, D., Ahsen, M. E., & Fan, X. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01743-6 · Cited by 108 · see citing works
Zhang, Z., Citardi, D., Wang, D., Genc, Y., Shan, J., & Fan, X. (2021) · Health Informatics Journal · doi:10.1177/14604582211011215 · Cited by 95 · see citing works
The effect of machine learning explanations on user trust for automated diagnosis of COVID-19
Goel, K., Sindhgatta, R., Kalra, S., Goel, R., & Mutreja, P. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105587 · Cited by 40 · see citing works
Explainable software systems: from requirements analysis to system evaluation
Chazette, L., Brunotte, W., & Speith, T. (2022) · Requirements Engineering · doi:10.1007/s00766-022-00393-5 · Cited by 44 · see citing works
Lay individuals' perceptions of artificial intelligence ( AI )‐empowered healthcare systems
Zhang, Z., Genc, Y., Xing, A., Wang, D., Fan, X., & Citardi, D. (2020) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.326 · Cited by 37 · see citing works
Requirements for Explainability and Acceptance of Artificial Intelligence in Collaborative Work
Theis, S., Jentzsch, S. F., Deligiannaki, F., Berro, C., Raulf, A., & Bruder, C. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35891-3_22 · Cited by 34 · see citing works
Liefgreen, A., Weinstein, N., Wachter, S., & Mittelstadt, B. (2023) · AI & Society · doi:10.1007/s00146-023-01684-3 · Cited by 26 · see citing works
Towards Generating Consumer Labels for Machine Learning Models
Seifert, C., Scherzinger, S., & Wiese, L. (2019) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi48466.2019.00033 · Cited by 19 · see citing works
A pragmatic perspective on AI transparency at workplace
Al-Sulaiti, G., Sadeghi, M. A., Chauhan, L. P. S., Lucas, J., Chawla, S., & Elmagarmid, A. K. (2023) · AI and Ethics · doi:10.1007/s43681-023-00257-w · Cited by 12 · see citing works
Kim, S., Yi, S., & Park, S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0326028 · Cited by 14 · see citing works
Breda, P., Markova, R., Abdin, A., Mantı, N. P., Carlo, A., & Jha, D. (2023) · Journal of Space Safety Engineering · doi:10.1016/j.jsse.2023.08.003 · Cited by 10 · see citing works
Zhang, T., Chung, T. A., Dey, A. K., & Bae, S. W. (2024) · International Conference on Activity and Behavior Computing (ABC) · doi:10.1109/abc61795.2024.10652070 · Cited by 10 · see citing works
Bemthuis, R., Wang, W., Iacob, M., & Havinga, P. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.03.057 · Cited by 9 · see citing works
Aladi, C. C. (2024) · ACM SIGCPR Conference on Computer Personnel Research (SIGCPR) · doi:10.1145/3632634.3655852 · Cited by 10 · see citing works
Fompeyrine, D. A., Vorm, E. S., Ricka, N., Rose, F., & Pellegrin, G. (2021) · Human-Intelligent Systems Integration · doi:10.1007/s42454-021-00037-z · Cited by 8 · see citing works
Self-citationLi, W., Fan, X., Zhu, H., Wu, J., & Teng, D. (2020) · ACM Turing Celebration Conference - China (ACM TURC) · doi:10.1145/3393527.3393561 · Cited by 7 · see citing works
Scalable Information Fusion Trust
Blasch, E., & Braines, D. (2021) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion49465.2021.9626986 · Cited by 4 · see citing works
Corporate training for developing social media literacy skills: personalized approach
Андреевна, О. Е., & Savelyeva, A. A. (2023) · RUDN Journal of Studies in Literature and Journalism · doi:10.22363/2312-9220-2023-28-1-157-164 · Cited by 4 · see citing works
When Mitigating Bias is Unfair: Multiplicity and Arbitrariness in Algorithmic Group Fairness
Krčo, N., Laugel, T., Grari, V., Loubes, J., & Detyniecki, M. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00046 · Cited by 2 · see citing works
Interactive Learning To Rank And Visual Rank Interpretation
Pereira, M. M. (2020) · DalSpace (Dalhousie University)
Blenk, E. R. (2021) · NCSU Libraries Repository (North Carolina State University Libraries)
- Authors
- Vorm, E. S.
- Affiliations
- Indiana University Indianapolis (US)
- Corresponding author
- Vorm, Eric S.
- ORCID
- Vorm, Eric S.
- Venue
- 2018 Innovations in Intelligent Systems and Applications (INISTA)
- Publisher
- IEEE
- Published
- 2018-07
- Pages
- 1-7
- OpenAlex
- W2892099700
- Cited by
- 25
- References
- 34
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 6 · 2022: 3 · 2021: 7 · 2020: 4 · 2019: 1
- FWCI
- 1.22
- Type
- conference paper
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- public-domain
- Locations
- International Conference on Innovations in Intelligent Systems and Applications (INISTA) (publishedVersion)
- IUScholarWorks (Indiana University) (submittedVersion, public-domain)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Keywords
- transparency, intelligent systems, explainable AI, decision support, trust in AI, user-centered design, design guidelines, cognitive load, machine learning models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review