Bridging the Gap Between UX Practitioners’ Work Practices and AI-Enabled Design Support Tools
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
User interface (UI) and user experience (UX) design have become an indispensable part of today’s tech industry. Recently, much progress has been made in machine-learning-enabled design support tools for UX designers. However, few of these tools have been adopted by practitioners. To learn the underlying reasons and understand user needs for bridging this gap, we conducted a retrospective analysis with 8 UX professionals to understand their practice and identify opportunities for future research. We found that the current AI-enabled systems to support UX work mainly work on graphical interface elements, while design activities that involve more ‘design thinking” such as user interviews and user testings are more helpful for designers. Many current systems were also designed for overly-simplistic and generic use scenarios. We identified 4 areas in the UX workflow that can benefit from additional AI-enabled assistance: design inspiration search, design alternative exploration, design system customization, and design guideline violation check.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Goodman, L. A. (1961) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177705148 · Cited by 3,606
A Neural Algorithm of Artistic Style
Gatys, L., Ecker, A., & Bethge, M. (2016) · Journal of Vision · doi:10.1167/16.12.326 · Cited by 2,612
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · cited 1×
“The general goal remains the same and many design principles in human-AI co-creation, such as mixed-initiative interactions [16], are applicable.”
BackgroundMuller, M. J., & Kuhn, S. (1993) · Communications of the ACM · doi:10.1145/153571.255960 · Cited by 909
Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376301 · Cited by 622
Rico: A Mobile App Dataset for Building Data-Driven Design Applications
Deka, B., Huang, Z., Franzen, C., Hibschman, J., Afergan, D., Li, Y., Nichols, J., & Kumar, R. (2017) · Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3126594.3126651 · Cited by 482
Oh, C., Song, J., Choi, J., Kim, S., Lee, S., & Suh, B. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174223 · Cited by 301
Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models
Louie, R., Coenen, A., Huang, C. Z., Terry, M., & Cai, C. J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376739 · Cited by 271 · cited 1×
“With the advancement of ML, many researchers have explored human-AI co-creation in creative domains including writing [4, 12, 22], music [29], drawing [7, 15, 20, 37], etc.”
BackgroundChen, C., Su, T., Meng, G., Xing, Z., & Liu, Y. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180240 · Cited by 188
Metaphoria: An Algorithmic Companion for Metaphor Creation
Gero, K. I., & Chilton, L. B. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300526 · Cited by 146
Davis, N., Hsiao, C.-P., Yashraj Singh, K., Li, L., & Magerko, B. (2016) · Proceedings of the 21st International Conference on Intelligent User Interfaces · doi:10.1145/2856767.2856795 · Cited by 138
Four Principles of Explainable Artificial Intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Broniatowski, D. A., & Przybocki, M. A. (2020) · doi:10.6028/nist.ir.8312-draft · Cited by 136 · cited 1×
“Researchers can get inspiration from the area of explainable AI [28, 39], especially”
BackgroundWhy Are We Like This?: The AI Architecture of a Co-Creative Storytelling Game
Kreminski, M., Dickinson, M., Mateas, M., & Wardrip-Fruin, N. (2020) · International Conference on the Foundations of Digital Games · doi:10.1145/3402942.3402953 · Cited by 50 · cited 1×
“With the advancement of ML, many researchers have explored human-AI co-creation in creative domains including writing [4, 12, 22], music [29], drawing [7, 15, 20, 37], etc.”
BackgroundWordcraft: a Human-AI Collaborative Editor for Story Writing
Coenen, A., Davis, L., Ippolito, D., Reif, E., & Yuan, A. (2021) · arXiv · doi:10.48550/arxiv.2107.07430 · Cited by 35 · cited 1×
“With the advancement of ML, many researchers have explored human-AI co-creation in creative domains including writing [4, 12, 22], music [29], drawing [7, 15, 20, 37], etc.”
BackgroundIntroduction to Explainable AI
Liao, Q. V., Singh, M., Zhang, Y., & Bellamy, R. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3445016 · Cited by 34
Research works printed but not linked: 37
- 10.1145/3419249.3420163 · DOI
- 10.1145/3064663.3064667 · DOI · Paper page
- 10.1007/978-3-540-74853-3_25 · DOI
- 10.1007/978-3-030-86365-4_5 · DOI
- Nishit Gajjar , Vinoth Pandian Sermuga Pandian , Sarah Suleri , and Matthias Jarke . 2021 . Akin: Generating UI Wireframes From UI Design Patterns Using Deep Learning. (2021), 40–42. https://doi.org/10.1145/3397482.3450727 Nishit Gajjar, Vinoth Pandian Sermuga Pandian, Sarah Suleri, and Matthias Jarke. 2021. Akin: Generating UI Wireframes From UI Design Patterns Using Deep Learning. (2021), 40–42. https://doi.org/10.1145/3397482.3450727 · DOI
- 10.1145/2556288.2557264 · DOI
- David Ha and Douglas Eck. 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477(2017). David Ha and Douglas Eck. 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477(2017). · arXiv
- 10.1145/3290605.3300334 · DOI
- Forrest Huang Gang Li Xin Zhou John F Canny and Yang Li. 2021. Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models. arXiv preprint arXiv:2110.07775(2021). Forrest Huang Gang Li Xin Zhou John F Canny and Yang Li. 2021. Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models. arXiv preprint arXiv:2110.07775(2021). · arXiv
- 10.1145/3491101.3504030 · DOI
- 10.1145/3377325.3377522 · DOI · Paper page
- Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114(2013). Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114(2013). · arXiv
- 10.1145/257089.257396 · DOI
- 10.1016/j.ijhcs.2013.09.006 · DOI · Paper page
- Hsin-Ying Lee Lu Jiang Irfan Essa Phuong B Le Haifeng Gong Ming-Hsuan Yang and Weilong Yang. 2020. Neural Design Network: Graphic Layout Generation with Constraints. (2020). https://doi.org/10.1007/978-3-030-58580-8 Hsin-Ying Lee Lu Jiang Irfan Essa Phuong B Le Haifeng Gong Ming-Hsuan Yang and Weilong Yang. 2020. Neural Design Network: Graphic Layout Generation with Constraints. (2020). https://doi.org/10.1007/978-3-030-58580-8 · DOI
- Jianan Li , Jimei Yang , Aaron Hertzmann , Jianming Zhang , and Tingfa Xu . 2019 . LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators. 7th International Conference on Learning Representations, ICLR 2019 (jan 2019 ). arxiv:1901.06767 Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. 2019. LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators. 7th International Conference on Learning Representations, ICLR 2019 (jan 2019). arxiv:1901.06767 · arXiv
- 10.1109/TSE.2018.2844788 · DOI · Paper page
- 10.1145/344949.344959 · DOI · Paper page
- 10.1145/200968.200971 · DOI
- Brad A Myers , Dario A Giuse , Roger B Dannenberg , Brad Vander Zanden , David S Kosbie , Edward Pervin , Andrew Mickish , and Philippe Marchal . 1995. GARNET comprehensive support for graphical, highly interactive user interfaces . In Readings in Human–Computer Interaction . Elsevier , 357–371. Brad A Myers, Dario A Giuse, Roger B Dannenberg, Brad Vander Zanden, David S Kosbie, Edward Pervin, Andrew Mickish, and Philippe Marchal. 1995. GARNET comprehensive support for graphical, highly interactive user interfaces. In Readings in Human–Computer Interaction. Elsevier, 357–371.
- 10.1145/1806491.1806494 · DOI · Paper page
- Kara Pernice . 2018 . Affinity Diagramming: Collaboratively Sort UX Findings & Design Ideas. https://www.nngroup.com/articles/affinity-diagram/ Kara Pernice. 2018. Affinity Diagramming: Collaboratively Sort UX Findings & Design Ideas. https://www.nngroup.com/articles/affinity-diagram/
- 10.1145/3188173.3188175 · DOI
- 10.1145/3090152.3090160 · DOI
- Daniel M Russell and Ed H Chi . 2014. Looking back: Retrospective study methods for HCI . In Ways of Knowing in HCI . Springer , 373–393. Daniel M Russell and Ed H Chi. 2014. Looking back: Retrospective study methods for HCI. In Ways of Knowing in HCI. Springer, 373–393.
- 10.1145/3411764.3445784 · DOI
- Ben Shneiderman , Michael Leavitt , 2006. Research-based web design & usability guidelines. Washington DC , Department of Health and Human Services ( 2006 ). Ben Shneiderman, Michael Leavitt, 2006. Research-based web design & usability guidelines. Washington DC, Department of Health and Human Services (2006).
- Erik Stolterman . 2008 . The nature of design practice and implications for interaction design research . International Journal of Design 2 , 1 (2008). Erik Stolterman. 2008. The nature of design practice and implications for interaction design research. International Journal of Design 2, 1 (2008).
- Hariharan Subramonyam , Colleen Seifert , and Eytan Adar . 2021 . ProtoAI: Model-Informed Prototyping for AI-Powered Interfaces. In 26th International Conference on Intelligent User Interfaces. Association for Computing Machinery , New York, NY, USA, 48–58. https://doi.org/10.1145/3397481.3450640 Hariharan Subramonyam, Colleen Seifert, and Eytan Adar. 2021. ProtoAI: Model-Informed Prototyping for AI-Powered Interfaces. In 26th International Conference on Intelligent User Interfaces. Association for Computing Machinery, New York, NY, USA, 48–58. https://doi.org/10.1145/3397481.3450640 · DOI
- 10.1145/3313831.3376593 · DOI · Paper page
- 10.1145/3472749.3474765 · DOI · Paper page
- 10.1007/978-3-030-49760-6_12 · DOI
- 10.1145/1952383.1952384 · DOI · Paper page
- 10.1109/ICSE43902.2021.00075 · DOI
- 10.1145/3196709.3196730 · DOI · Paper page
- ZiCheng Zhang CongYing Han and TianDe Guo. 2021. ExSinGAN: Learning an Explainable Generative Model from a Single Image. arXiv preprint arXiv:2105.07350(2021). ZiCheng Zhang CongYing Han and TianDe Guo. 2021. ExSinGAN: Learning an Explainable Generative Model from a Single Image. arXiv preprint arXiv:2105.07350(2021). · arXiv
- 10.1109/ICSE43902.2021.00074 · DOI
Cited by
Suh, S., Chen, M., Min, B., Li, T. J.-J., & Xia, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642400 · Cited by 148
Self-citationZhang, Z., Gao, J., Dhaliwal, R. S., & Li, T. J.-J. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606800 · Cited by 108
“Recently, there are some works exploring to use AI models to support people’s workfow of creative work [28, 35, 47].”
BackgroundSelf-citationLimitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks
Szymanski, A., Ziems, N., Eicher-Miller, H. A., Li, T. J.-J., Jiang, M., & Metoyer, R. A. (2025) · Proceedings of the 30th International Conference on Intelligent User Interfaces · doi:10.1145/3708359.3712091 · Cited by 77
Self-citationLiu, Y.-L. E., & Huang, Y.-M. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2311975 · Cited by 47
“The design of user interfaces necessitates considerable innovation and creativity, yet the process is often protracted and constrained (Lu et al., 2022).”
Kamath, P., Morreale, F., Bagaskara, P. L., Wei, Y., & Nanayakkara, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642040 · Cited by 37
“Interactive CSTs have been studied for their co-creation capabilities in visual arts [17, 27, 43, 53, 71], in writing [14, 23], in fashion [50], in UX design and engineering [38, 54, 63, 67], and in new musical interface design [91].”
Background
Metadata
- Authors
- Lu, Y., Zhang, C., Zhang, I., & Li, T. J.-J.
- Venue
- CHI Conference on Human Factors in Computing Systems Extended Abstracts
- Publisher
- ACM
- Published
- 2022-04-27
- Pages
- 1-7
- OpenAlex
- W4225129570
- Cited by
- 63
- References
- 16
- FWCI
- 10.70
- Type
- conference paper
- Language
- en
- Open access
- closed
- Funders
- Asia Research Collaboration Grant from Notre Dame International
- Google Cloud Research Credit Grant
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.