Explainable AI for Designers: A Human-Centered Perspective on Mixed-Initiative Co-Creation
Open accessThe DOI matches its Crossref record; 31 of the 35 references it cites were found in a registry or an index.
Abstract
Growing interest in eXplainable Artificial Intelligence (XAI) aims to make AI and machine learning more understandable to human users. However, most existing work focuses on new algorithms, and not on usability, practical interpretability and efficacy on real users. In this vision paper, we propose a new research area of eXplainable AI for Designers (XAID), specifically for game designers. By focusing on a specific user group, their needs and tasks, we propose a human-centered approach for facilitating game designers to co-create with AI/ML techniques through XAID. We illustrate our initial XAID framework through three use cases, which require an understanding both of the innate properties of the AI techniques and users' needs, and we identify key open challenges.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, investigations into mediating factors (e.g., interaction design [140], explainability [209]) typically occur in isolation, limiting the structured mapping of how agency patterns, control mechanism, and contextual factors interrelate.”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [209]
In the citing paper
“Transparency and explainability regarding AI actions [70, 187, 209] are crucial for enabling humans to appropriately trust and engage with AI contributions [191, 201].”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [209]
In the citing paper
“Zhu et al. [88] noticed that instead of focusing on the practicality and efficacy of current XAI practices, most works are coming up with new methods of explanations.”
From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [88]
Citations
Cites
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“Techniques that aim to explain relationships between neurons are known as attribution and a variety of different approaches exists [2], [5], [6].”
Background(2011) · Cited by 14,317
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“Techniques that aim to explain relationships between neurons are known as attribution and a variety of different approaches exists [2], [5], [6].”
BackgroundDeep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258 · cited 1×
“Visualizing hidden layers’ features [1]–[4] can give insights into what network inputs would cause a certain reaction in an internal neuron or in the network output.”
BackgroundDeep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images
Nguyen, A., Yosinski, J., & Clune, J. (2014) · arXiv · doi:10.48550/arxiv.1412.1897 · Cited by 3,258 · cited 1×
“Visualizing hidden layers’ features [1]–[4] can give insights into what network inputs would cause a certain reaction in an internal neuron or in the network output.”
BackgroundInterpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097 · cited 1×
“Techniques that aim to explain relationships between neurons are known as attribution and a variety of different approaches exists [2], [5], [6].”
BackgroundAbdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743 · cited 1×
“However, most existing work focuses on new algorithms of XAI rather than on usability, practical interpretability and efficacy on real users [26]–[28].”
BackgroundThe Building Blocks of Interpretability
Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., & Mordvintsev, A. (2018) · Distill · doi:10.23915/distill.00010 · Cited by 591 · cited 1×
“In a recent article [8], Olah et al.”
BackgroundPlug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
Nguyen, A., Clune, J., Bengio, Y., Dosovitskiy, A., & Yosinski, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.374 · Cited by 578
Common ground at the understanding of demonstrative reference
Clark, H. H., Schreuder, R., & Buttrick, S. (1983) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(83)90189-5 · Cited by 445 · cited 1×
“Essentially, this is the notion of establishing ‘common ground’ as forwarded and explored by Herb Clark [31].”
Background
Research works printed but not linked: 4
- straatman, 2005, Killzones ai: dynamic procedural combat tactics, Game Developers Conference
- miller, 2017, Explainable AI: Beware of inmates running the asylum, Proceedings of the IJCAI Workshop on Workshop on Explainable Artificial Intelligence
- erhan, 2009, Visualizing higher-layer features of a deep network, Universite De Montreal Tech Rep, 1341
- biran, 2017, Explanation and justification in machine learning: A survey, Proceedings of the IJCAI Workshop on Explainable Artificial Intelligence
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
“In general, humans are reticent to adopt techniques that are not directly interpretable, tractable and trustworthy [9] , given the increasing demand for ethical AI [3] .”
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
“[157] noted that most existing works focus on new explainingmethods, and not on usability, practical interpretability and efficacy on real users.”
BackgroundInterpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
“Zhu et al. [88] noticed that instead of focusing on the practicality and efficacy of current XAI practices, most works are coming up with new methods of explanations.”
BackgroundQuestioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
“Future work could study other roles involved in AI product development (e.g., data scientists) to better understand the challenges to create XAI products.”
BackgroundA Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785
“gn, Eiband et al. [56] present a stage-based participatory process for integration of transparency in existing intelligent systems using explanations. Another design framework is XAID from Zhu et al. [225], which presents a human-centered approach for facilitating game designers to co-create with machine learning tech…”
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
“XAI interface design, Eiband et al. [47] present a stage-based participatory process for integration of transparency in existing intelligent systems. Another design framework is XAID from Zhu et al. [201], which presents a human-centered approach for facilitating game designers to co-create with machine learning techn…”
MethodLanger, M., Oster, D., Speith, T., Hermanns, H., Kästner, L., Schmidt, E., Sesing, A., & Baum, K. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103473 · Cited by 652
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564
“[60] provide domain specific results for challenges of designing iML systems for software developers and game designers, respectively.”
BackgroundExplanations in Autonomous Driving: A Survey
Omeiza, D., Webb, H., Jirotka, M., & Kunze, L. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3122865 · Cited by 355
“Second, researchers suggested (in [77]) that understanding users’ requirements might be helpful in explainable AI research.”
BackgroundLowe, M., Qin, R., & Mao, X. (2022) · Water · doi:10.3390/w14091384 · Cited by 354
“Humans are generally hesitant to adopt models and methods that are not easily interpretable and trustworthy [134], and with the rise in deep learning and black-box methods (as seen in the models reviewed in this very paper), trans-parency and explainability are increasingly difficult to come by in…”
BackgroundUnderstanding artificial intelligence ethics and safety
Leslie, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05684 · Cited by 134
Ahmad, K., Maabreh, M., Ghaly, M., Khan, K., Qadir, J., & Al-Fuqaha, A. (2022) · Computer Science Review · doi:10.1016/j.cosrev.2021.100452 · Cited by 291
“Explainable AI is also a need for modern entertainment and businesses [162].”
BackgroundExplainable Artificial Intelligence (XAI) in auditing
Zhang, C. (., Cho, S., & Vasarhelyi, M. (2022) · International Journal of Accounting Information Systems · doi:10.1016/j.accinf.2022.100572 · Cited by 164
Shi, Y., Gao, T., Jiao, X., & Cao, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610217 · Cited by 170
“As one of the widely recognized principles for developing trusted AI, explainability also applies when AI is generating design output [211].”
BackgroundUddin, M. Z., & Soylu, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-95947-y · Cited by 147
Artificial intelligence in the water domain: Opportunities for responsible use
Doorn, N. (2021) · Science of The Total Environment · doi:10.1016/j.scitotenv.2020.142561 · Cited by 142
“J Partly in response to this problem of opacity, DARPA’s third wave of AI techniques focuses on explainability and the development of so-called “white box AI” (Zhu et al. 2018).”
How to explain AI systems to end users: a systematic literature review and research agenda
Laato, S., Tiainen, M., Najmul Islam, A. K. M., & Mäntymäki, M. (2022) · Internet Research · doi:10.1108/intr-08-2021-0600 · Cited by 155
Guzdial, M., Liao, N., Chen, J., Chen, S.-Y., Shah, S., Shah, V., Reno, J., Smith, G., & Riedl, M. O. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300854 · Cited by 128
“While the majority of these extant AI systems cover rote or repetitive human interactions, there has been increased interest in using AI to assist creative expression [32, 56].”
BackgroundProcedural Content Generation through Quality Diversity
Gravina, D., Khalifa, A., Liapis, A., Togelius, J., & Yannakakis, G. N. (2019) · IEEE Conference on Games (CoG) · doi:10.1109/cig.2019.8848053 · Cited by 179
“Explainable AI for Designers [33] is an important research area that aims to aid the game designer in understanding AI algorithms applied to games.”
BackgroundSelf-citationDesigning Participatory AI: Creative Professionals’ Worries and Expectations about Generative AI
Inie, N., Falk, J., & Tanimoto, S. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3585657 · Cited by 137
A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning
Kaluarachchi, T., Reis, A., & Nanayakkara, S. (2021) · Sensors · doi:10.3390/s21072514 · Cited by 124
“Trust Explainability [63], [70], [165], [174], [211]”
BackgroundMethodInfluentialA survey on artificial intelligence assurance
Batarseh, F. A., Freeman, L., & Huang, C. (2021) · Journal Of Big Data · doi:10.1186/s40537-021-00445-7 · Cited by 156
Ochmann, J., Michels, L., Tiefenbeck, V., Maier, C., & Laumer, S. (2024) · Information Systems Journal · doi:10.1111/isj.12482 · Cited by 95
Explainable AI for earth observation: A review including societal and regulatory perspectives
Gevaert, C. M. (2022) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2022.102869 · Cited by 177
“Thirdly, research tends to present innovative explainability methods without actually focusing on the utility of these new methods to foster explainability for real-world problems and users (Murdoch et al., 2019; J. Zhu et al., 2018).”
BackgroundHofeditz, L., Clausen, S., Rieß, A., Mirbabaie, M., & Stieglitz, S. (2022) · Electronic Markets · doi:10.1007/s12525-022-00600-9 · Cited by 95
“One reason might be that the origin of recommendations made by an AI-based system is often incomprehensible (Adadi & Berrada, 2018), which makes it difficult for people to trust the underlying technology (Zhu et al., 2018).”
BackgroundBayesian networks for interpretable machine learning and optimization
Mihaljević, B., Bielza, C., & Larrañaga, P. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.01.138 · Cited by 123
“In addition, experts in many fields — ranging from medical diagnosis [6] and bioinformatics [7] to finan-cial applications [8]— require understanding the model’s decisions in order to use it [9].”
BackgroundGenerative AI-enhanced human-AI collaborative conceptual design: A systematic literature review
Fang, C., Zhu, Y., Fang, L., Long, Y., Lin, H., Cong, Y., & Wang, S. J. (2025) · Design Studies · doi:10.1016/j.destud.2025.101300 · Cited by 135
Nourani, M., Kabir, S., Mohseni, S., & Ragan, E. D. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5284 · Cited by 100
Barda, A. J., Horvat, C. M., & Hochheiser, H. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01276-x · Cited by 105
LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing
Wang, B., Li, Y., Lv, Z., Xia, H., Xu, Y., & Sodhi, R. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645143 · Cited by 63
A Review on Human–Machine Trust Evaluation: Human-Centric and Machine-Centric Perspectives
Gebru, B., Zeleke, L., Blankson, D., Nabil, M., Nateghi, S., Homaifar, A., & Tunstel, E. (2022) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2022.3144956 · Cited by 101
“pouse techniques that are not interpretable and trustworthy [82].”
BackgroundExplanation of Machine-Learning Solutions in Air-Traffic Management
Xie, Y., Pongsakornsathien, N., Gardi, A., & Sabatini, R. (2021) · Aerospace · doi:10.3390/aerospace8080224 · Cited by 110
“Experienced human operators tend to be reluctant to adopt suggested solutions from highly autonomous DSS if these are not trustworthy, traceable, and interpretable, especially in very complex situations [14].”
XAIR: A Framework of Explainable AI in Augmented Reality
Xu, X., Yu, A., Jonker, T. R., Todi, K., Lu, F., Qian, X., Belo, J. M. E., Wang, T., Li, M., Mun, A., et al. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581500 · Cited by 109
“proposed a co-creation design space between game designers using ML techniques and investigated the usability of XAI algorithms to support game designers [246].”
MethodHorvath, A.-S., & Pouliou, P. (2024) · Frontiers of Architectural Research · doi:10.1016/j.foar.2024.02.006 · Cited by 113
“…to make the re-sults of AI systems easier to understand, and to ensure that those using them trust these results, explainable AI research proposes creating different types of visualizations, or verbalizations of the inner workings of neural networks (Halilovic and Lindner, 2023; Zhu et al., 2018).”
BackgroundSensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
Kaur, H., Adar, E., Gilbert, E., & Lampe, C. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533135 · Cited by 92
“For example, [42, 45, 70, 113, 126] all highlight the challenges of operating within an organization that either develops or employs an AI-based system.”
BackgroundWhen Post Hoc Explanation Knocks: Consumer Responses to Explainable AI Recommendations
Chen, C., Tian, A. D., & Jiang, R. (2023) · Journal of Interactive Marketing · doi:10.1177/10949968231200221 · Cited by 98
“Trust, an individual ’ s perceived con fi dence and belief in AI recommendations (Tintarev and Masthoff 2011), is critical because people are generally reluctant to consider such emerging technologies trustworthy (Zhu et al. 2018).”
Scenario-Based Requirements Elicitation for User-Centric Explainable AI: A Case in Fraud Detection
Cirqueira, D., Nedbal, D., Helfert, M., & Bezbradica, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-57321-8_18 · Cited by 110
“[79,83,84] 3”
Nagahisarchoghaei, M., Nur, N., Cummins, L., Nur, N., Karimi, M. M., Nandanwar, S., Bhattacharyya, S., & Rahimi, S. (2023) · Electronics · doi:10.3390/electronics12051092 · Cited by 103
From Chess and Atari to StarCraft and Beyond: How Game AI is Driving the World of AI
Risi, S., & Preuss, M. (2020) · Künstliche Intell. · doi:10.1007/s13218-020-00647-w · Cited by 94
Self-citationPersonaFlow: Designing LLM-Simulated Expert Perspectives for Enhanced Research Ideation
Liu, Y., Sharma, P., Oswal, M., Xia, H., & Huang, Y. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735789 · Cited by 87
“In future iterations, designers could explore using AI-simulated personas that pose reflective questions [54], prompting users to elaborate further on their thought processes and rationales.”
The Use of Responsible Artificial Intelligence Techniques in the Context of Loan Approval Processes
Purificato, E., Lorenzo, F., Fallucchi, F., & Luca, E. W. D. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2081284 · Cited by 89
Explaining sentiment analysis results on social media texts through visualization
Jain, R., Kumar, A., Nayyar, A., Dewan, K., Garg, R., Raman, S., & Ganguly, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-14432-y · Cited by 68
“on socio-technical systems, several of which are based on artificial intelligence [61].”
BackgroundA User-centered Framework for Human-AI Co-creativity
Moruzzi, C., & Margarido, S. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650929 · Cited by 68
What Are People Doing About XAI User Experience? A Survey on AI Explainability Research and Practice
Ferreira, J. J., & Monteiro, M. S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-49760-6_4 · Cited by 68
Melhárt, D., Togelius, J., Mikkelsen, B., Holmgård, C., & Yannakakis, G. N. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3276425 · Cited by 79
“While explainable AI frameworks generally advocate for open communication towards the user about model predictions [46], [48], when it comes to in-game adjustments this can be detrimental to the player experience [42].”
BackgroundMaking AI Explainable in the Global South: A Systematic Review
Okolo, C. T., Dell, N. L., & Vashistha, A. (2022) · ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies (COMPASS) · doi:10.1145/3530190.3534802 · Cited by 81
“Additionally, frameworks that guide the design, development, and use of XAI techniques have also become more common [40, 149, 155].”
Designing Human-Agent Collaborations: Commitment, responsiveness, and support
Cila, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517500 · Cited by 65
“AI agents have been collaborating with musicians to compose music [101], supporting people in writing stories and slogans [33], helping actors to create introductions for improvised theatrical scenes [112], and supporting game designers to create levels in a game [164].”
BackgroundExplainability and white box in drug discovery
Kırboğa, K. K., Abbasi, S. W., & Küçüksille, E. U. (2023) · Chemical Biology & Drug Design · doi:10.1111/cbdd.14262 · Cited by 68
“Thus neural networking requires transparency and adds justification to the generated predictions (Zhu et al., 2018).”
BackgroundA new approach based on association rules to add explainability to time series forecasting models
Troncoso-García, A. R., Martínez-Ballesteros, M., Alvarez, F. M., & Troncoso, A. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.01.021 · Cited by 68
“If the end users are able to understand the computations made by ML and DL algorithms, the model is explainable and post-hoc XAI techniques are achieving their goal [29].”
MethodThe Pixels and Sounds of Emotion: General-Purpose Representations of Arousal in Games
Makantasis, K., Liapis, A., & Yannakakis, G. N. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3060877 · Cited by 64
“The GCAM visualization increases the explainability of the models [82] and can be very useful for game designers when adjusting the appearance or in-game function of the game elements to increase or decrease the elicited emotion of certain events.”
MethodSelf-citationWoodcock, C., Mittelstadt, B., Busbridge, D., & Blank, G. (2021) · Journal of Medical Internet Research · doi:10.2196/29386 · Cited by 54
“If AI systems are not trusted, they are less likely to be adopted by their users, limiting the efficacy of the technology [19,58].”
BackgroundDeng, M., Liu, Y., & Chen, L. (2023) · Electronic Research Archive · doi:10.3934/era.2023295 · Cited by 59
An Interaction Framework for Studying Co-Creative AIPreprint
Guzdial, M., & Riedl, M. O. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.09709 · Cited by 60
“[9, 19].”
Co-Creative Level Design via Machine Learning
Guzdial, M., Liao, N., & Riedl, M. (2018) · arXiv · doi:10.48550/arxiv.1809.09420 · Cited by 56
“nts comparing these existing approaches to approaches designed for co-creation. Related Work The concept of co-creative PCGML has been previously discussed in the literature (Summerville et al. 2017; Zhu et al. 2018), but no prior approaches or systems exist. Comparatively there exist many prior approaches to co-creat…”
BackgroundMethodEvaluating eXplainable artificial intelligence tools for hard disk drive predictive maintenance
Ferraro, A., Galli, A., Moscato, V., & Sperlí, G. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10354-7 · Cited by 62
“Nevertheless, humans are reluctant to adopt techniques that are not directly interpretable, tractable, and reliable (Zhu et al. 2018), in view of the growing demand for ethical AI (Goodman and Flax-man 2017).”
BackgroundTrustworthy AI for human-centric smart manufacturing: A survey
Li, D., Liu, S., Wang, B., Yu, C., Zheng, P., & Li, W. (2024) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2024.11.020 · Cited by 59
Player-AI Interaction: What Neural Network Games Reveal About AI as Play
Zhu, J., Villareale, J., Javvaji, N., Risi, S., Löwe, M., Weigelt, R., & Harteveld, C. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445307 · Cited by 65
“Furthermore, new interdisciplinary research areas have emerged around topics such as explainable AI [9, 66, 83, 101], ethics & fairness [12, 23, 38], and machine learning (ML) as a design material for UX [93, 94].”
BackgroundSelf-citationTorre-Bastida, A. I., Díaz-de-Arcaya, J., Osaba, E., Muhammad, K., Camacho, D., & Ser, J. D. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06332-9 · Cited by 56
“reported by the academia and the real-world problems faced by practitioners in their respective sectors [235].”
BackgroundData Privacy Threat Modelling for Autonomous Systems: A Survey From the GDPR's Perspective
Azam, N., Michala, A. L., Ansari, S., & Truong, N. B. (2022) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2022.3227336 · Cited by 63
“The AC should be comprehensible and trustworthy for its successful implementation in the real world[248].”
BackgroundAdvances in low-altitude airspace management for uncrewed aircraft and advanced air mobility
Pongsakornsathien, N., Safwat, N. E., Xie, Y., Gardi, A., & Sabatini, R. (2025) · Progress in Aerospace Sciences · doi:10.1016/j.paerosci.2025.101085 · Cited by 59
Applying Genetic Programming to Improve Interpretability in Machine Learning Models
Ferreira, L. A., Guimarães, F. G., & Silva, R. (2020) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec48606.2020.9185620 · Cited by 60
“Humans tend to be less willing to accept decisions that are not directly interpretable and trustworthy [3].”
BackgroundOprescu, A. M., Amarante, G. M., García‐Díaz, L., Rey, V. E., Chimenea, Á., Martinez-Martinez, R. L., & Ternero, M. D. C. R. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.011 · Cited by 55
Multilayer dynamic ensemble model for intensive care unit mortality prediction of neonate patients
Juraev, F., El–Sappagh, S., Abdukhamidov, E., Ali, F., & Abuhmed, T. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104216 · Cited by 47
“Humans do not accept adopting techniques that are not directly interpretable and trustworthy [94].”
BackgroundFritz, A., Brandt, W., Gimpel, H., & Bayer, S. (2020) · De Ethica · doi:10.3384/de-ethica.2001-8819.20613 · Cited by 37
Rezazadeh, A., Jafarian, Y., & Valeshabad, A. K. (2022) · Forecasting · doi:10.3390/forecast4010015 · Cited by 46
Behl, A., Sampat, B. H., Pereira, V. E., & Jabbour, C. J. C. (2023) · Annals of Operations Research · doi:10.1007/s10479-023-05624-8 · Cited by 49
Bennetot, A., Franchi, G., Ser, J. D., Chatila, R., & Díaz-Rodríguez, N. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109947 · Cited by 53
“Common needs for requiring interpretability in Deep Learning are the need for ethics [3], safety when using AI in critical settings [4] and the need to enable the end user to trust the system [5].”
BackgroundHan, J. W., Kim, T., Lee, S., Kang, T., & Im, J. K. (2024) · Ecological Indicators · doi:10.1016/j.ecolind.2024.112361 · Cited by 42
Data-Driven Game Development: Ethical Considerations
El‐Nasr, M. S., & Kleinman, E. (2020) · International Conference on the Foundations of Digital Games (FDG) · doi:10.1145/3402942.3402964 · Cited by 44
“ata,explainable AIhas dedicatedmuchtimeand resources towards achieving the goals of transparency and interpretability [19, 46], and existing work has shown potential in applying XAI concepts to games [86]. However, it has yet to be seen how XAI can guide more ethical game data science. Going forward, we encourage work…”
BackgroundMethodTowards a Diffractive Analysis of Prompt-Based Generative AI
Rajcic, N., Llano Rodriguez, M. T., & McCormack, J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641971 · Cited by 37
“Mixed-initiative approaches have also been explored, in which both human and computational agents can initiate content exchanges through a mutual exploration of the design space [43, 66, 68].”
BackgroundToward Human-centered XAI in Practice: A survey
Kong, X., Liu, S., & Zhu, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-022-1407-3 · Cited by 50
Zhang, S., Wang, H., & Yi, X. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757594 · Cited by 36
“Furthermore, mediating factors like interaction design [140] or explainability [209] are often analyzed separately, obscuring their integrated role in agency dynamics.”
InfluentialConfigurations of human-centered AI at work: seven actor-structure engagements in organizations
Wilkens, U., Lupp, D., & Langholf, V. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1272159 · Cited by 38
Explainable Computational Creativity
Llano, M. T., d’Inverno, M., Yee-King, M. J., McCormack, J., Ilsar, A., Pease, A., & Colton, S. (2022) · Goldsmiths (University of London) · doi:10.48550/arxiv.2205.05682 · Cited by 36
“In (Zhu et al. 2018) the authors defined a new area of XAI which they called Explainable AI for Designers (XAID).”
BackgroundOn Mixed-Initiative Content Creation for Video Games
Lai, G., Leymarie, F. F., & Latham, W. (2022) · IEEE Transactions on Games · doi:10.1109/tg.2022.3176215 · Cited by 42
Artificial everyday creativity: creative leaps with AI through critical making
Reddy, A. (2022) · Digital Creativity · doi:10.1080/14626268.2022.2138452 · Cited by 29
“Some AI researchers are already working to make existing AI systems understandable and explainable to their users (Zhu et al. 2018; Hassan et al. 2021).”
BackgroundExplainable AI for the Arts: XAIxArts
Bryan–Kinns, N., Ford, C., Chamberlain, A., Benford, S., Kennedy, H., Li, Z., Wu, Q., Xia, G., & Rezwana, J. (2023) · Creativity and Cognition · doi:10.1145/3591196.3593517 · Cited by 43
“There is little research on how User Centred Design (UCD) can be used in the creation of XAI systems [41], and no current research on UCD for XAIxArts.”
BackgroundMixed-Initiative Level Design with RL Brush
Delarosa, O., Dong, H., Ruan, M., Khalifa, A., & Togelius, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72914-1_27 · Cited by 41
Nguyen, H. D., & Tran, K. P. (2023) · Springer series in reliability engineering · doi:10.1007/978-3-031-30510-8_2 · Cited by 44
Luo, Y. (2025) · Advances in Human-Computer Interaction · doi:10.1155/ahci/3869207 · Cited by 40
“Defning objectives related to human needs was stressed to avoid excessive automation [47, 104], especially considering the required resources and ethical risks of data collection [47].”
Rapp, A., Boldi, A., Curti, L., Perrucci, A., & Simeoni, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580995 · Cited by 19
“music [62], support people in writing stories [14] or game designers to create games [103], help individuals select the best energy tarif in the home [1], give runners motivation and companionship during their run [35], help players feel less lonely [80], and provide support in the context of healthcare [48].”
BackgroundTowards Explainable Neural-Symbolic Visual Reasoning
Bennetot, A., Laurent, J.-L., Chatila, R., & Díaz-Rodríguez, N. (2019) · arXiv · doi:10.48550/arxiv.1909.09065 · Cited by 32
“Some common cases are a need for ethics [Goodman and Flaxman, 2017], for safety when using AI in high-risk environments [Caruana et al., 2015] or a need to allow the final user to trust the system [Zhu et al., 2018].”
BackgroundBaskar, G., Omer, S. N., Saravanan, P., Rajeshkannan, R., Saravanan, V., Rajasimman, M., & Shanmugam, V. (2024) · Chemosphere · doi:10.1016/j.chemosphere.2024.142477 · Cited by 33
Where AI and Design Meet - Designers’ Perceptions of AI Tools
Kalving, M., Colley, A., & Häkkilä, J. (2024) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3679318.3685388 · Cited by 38
“Indeed, in Zhu et al.’s Spectrum of Initiative, the most advanced level positions the AI as a collaborator, actively participating in the design process alongside human designers [35].”
The Co-Creative Design Framework for Hybrid Intelligence
Davis, N., Sherson, J. F., & Rafner, J. F. (2025) · Creativity and Cognition · doi:10.1145/3698061.3726934 · Cited by 44
“The field designs, analyzes, and evaluates co-creative AI systems in a wide variety of creative domains [12], such as drawing [11, 16, 19], design [28, 31, 32, 34, 68], dance [36], writing [63], poetry [24, 30, 47], music [2, 26, 27, 37], and video games [22, 40].”
Explainable Bayesian networks applied to transport vulnerability
de Waal, A., & Joubert, J. W. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118348 · Cited by 34
Metrics for Robot Proficiency Self-assessment and Communication of Proficiency in Human-robot Teams
Norton, A., Admoni, H., Crandall, J. W., Fitzgerald, T., Gautam, A., Goodrich, M. A., Saretsky, A., Scheutz, M. J., Simmons, R. G., Steinfeld, A. M., et al. (2022) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3522579 · Cited by 36
“, visualizations of input-output relationships of a neural network [172] or heat maps to convey image saliency [22, 65]) can be rich in information and can be transmitted instantaneously, but may require some additional interpretation from the human.”
BackgroundGetting Playful with Explainable AI: Games with a Purpose to Improve Human Understanding of AI
Fulton, L. B., Lee, J. Y., Wang, Q., Yuan, Z., Hammer, J., & Perer, A. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3382831 · Cited by 35
Assuring Safe and Efficient Operation of UAV Using Explainable Machine Learning
Alharbi, A., Petrunin, I. V., & Panagiotakopoulos, D. (2023) · Drones · doi:10.3390/drones7050327 · Cited by 32
Toward Designer Modeling Through Design Style Clustering
Alvarez, A., Font, J., & Togelius, J. (2022) · IEEE Transactions on Games · doi:10.1109/tg.2022.3143800 · Cited by 27
“[32] proposed the field of eXplainable AI for Designers (XAID) as a human-centered perspective on MI-CC tools.”
BackgroundEmotion Recognition Using Explainable Genetically Optimized Fuzzy ART Ensembles
Liew, W. S., Loo, C. K., & Wermter, S. (2021) · IEEE Access · doi:10.1109/access.2021.3072120 · Cited by 28
“Machine-generated and human-interpretable explanations that are generated alongside decisions would help to gain users’ trust [5] and support post-hoc debugging and correction when the system produces a harmful decision.”
BackgroundA Multifaceted Surrogate Model for Search-Based Procedural Content Generation
Karavolos, D., Liapis, A., & Yannakakis, G. N. (2019) · IEEE Transactions on Games · doi:10.1109/tg.2019.2931044 · Cited by 27
“Alternatively, the model can explain to the designer [40] which sections of the level most impact the duration prediction and thus should be changed [41].”
BackgroundSelf-citationProcedural generation of problems for elementary math education
Xu, Y., Smeets, R., & Rafael Bidarra, E. (2021) · International Journal of Serious Games · doi:10.17083/ijsg.v8i2.396 · Cited by 23
“Furthermore, it would be interesting to explore and introduce AI-based methods to automatically generate the ontology used in the text generation process [24].”
MethodFox, S., & Rey, V. F. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6030071 · Cited by 22
“This is because within tools that are used by people [59].”
MethodWang, B., Yuan, T., & Rau, P. P. (2024) · International Journal of Social Robotics · doi:10.1007/s12369-024-01132-2 · Cited by 25
“…critical explanation strategies and propose hypotheses about possible effects of the explanation strategies and agent autonomy on human–AI collaboration. et al. [21] proposed three explanation methods, including explaining the evidence, explaining the model, and explaining the reasoning process.”
SOYSAL, F. (2023) · Karamanoğlu Mehmetbey Üniversitesi Uluslararası Filoloji ve Çeviribilim Dergisi · doi:10.55036/ufced.1402649 · Cited by 30
Regulating Explainable Artificial Intelligence (XAI) May Harm Consumers
Mohammadi, B., Malik, N., Derdenger, T. P., & Srinivasan, K. (2024) · Marketing Science · doi:10.1287/mksc.2022.0396 · Cited by 35
A survey of Semantic Reasoning frameworks for robotic systems
Liu, W., Daruna, A., Patel, M., Ramachandruni, K., & Chernova, S. (2022) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2022.104294 · Cited by 29
Explanation sets: A general framework for machine learning explainability
Fernández, R. R., de Diego, I. M., Moguerza, J. M., & Herrera, F. (2022) · Information Sciences · doi:10.1016/j.ins.2022.10.084 · Cited by 26
Metadata
- Authors
- Zhu, J., Liapis, A., Risi, S., Bidarra, R., & Youngblood, G. M.
- Affiliations
- Drexel University (US)
- University of Malta (MT)
- IT University of Copenhagen (DK)
- Palo Alto Research Center (US)
- Venue
- 2018 IEEE Conference on Computational Intelligence and Games (CIG)
- Publisher
- IEEE
- Published
- 2018-08
- Pages
- 1-8
- OpenAlex
- W2896010852
- Cited by
- 255
- References
- 35
- Citations by year
- 2026: 14 · 2025: 35 · 2024: 38 · 2023: 33 · 2022: 46 · 2021: 41 · 2020: 33 · 2019: 12 · 2018: 3
- FWCI
- 9.05
- Type
- conference paper
- Language
- en
- Open access
- green
- pure.itu.dk
- Licence
- other-oa
- Locations
- IEEE Conference on Computational Intelligence and Games (CIG) (publishedVersion)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Research Repository (Delft University of Technology) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- explainable AI, game designers, human-centered design, interpretability, game design, AI-assisted design, user needs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.