Vibe Design: Human-in-the-loop Agentic Framework for UI Design with Large Language Models
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Advances in LLMs are driving structural changes across the software development landscape, particularly in software product development processes. These transformations extend beyond programming and increasingly influence established UI design workflows. Conventional UI design relies on iterative cycles of prototyping and user testing. However, such processes are often time- and resource-intensive, creating tension with real-world environments that demand rapid iteration. Although prior research has explored AI-assisted approaches to support UI design, most studies focus on isolated functionalities or depend on code-based inputs that are rarely available in practical design settings. To address these limitations, this study proposes Vibe Design, an LLM-based agentic framework that enables iterative interface prototyping and usability test simulation without relying on code-based data. The framework integrates LLM-assisted prototyping and usability test simulation grounded in the Persona-Scenario-Goal methodology. An empirical study was conducted using three types of user data, namely demographic profiles, persona descriptions, and interview transcripts, to evaluate simulation performance. Results indicate that the interview-based agent achieved the highest performance, reproducing 76% of the usability themes identified in human usability testing. By integrating the Persona-Scenario-Goal methodology and Human-in-the-Loop mechanisms, this study demonstrates both the practical applicability and methodological implications for LLM-based UI design agents. While the findings highlight the potential of AI-driven user simulation to reduce repetitive validation costs, they also indicate that higher-order judgment and creative reframing remain the domain of human designers. These results underscore the need for complementary human-AI collaboration in the reconfiguration of contemporary design processes.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“while domain-specific studies clarify which metric trade-offs are acceptable in practice [80,118].”
From Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications (2026) · cites this paper as [118]
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
Wamba-Taguimdje, S.-L., Fosso Wamba, S., Kala Kamdjoug, J. R., & Tchatchouang Wanko, C. E. (2020) · Business Process Management Journal · doi:10.1108/bpmj-10-2019-0411 · Cited by 1,280
Dellermann, D., Ebel, P., Söllner, M., & Leimeister, J. M. (2019) · Business & Information Systems Engineering · doi:10.1007/s12599-019-00595-2 · Cited by 826
Rico: A Mobile App Dataset for Building Data-Driven Design Applications
Deka, B., Huang, Z., Franzen, C., Hibschman, J., Afergan, D., Li, Y., Nichols, J., & Kumar, R. (2017) · Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3126594.3126651 · Cited by 482
Guiding goal modeling using scenarios
Rolland, C., Souveyet, C., & Achour, C. B. (1998) · IEEE Transactions on Software Engineering · doi:10.1109/32.738339 · Cited by 478
Akata, Z., Balliet, D., de Rijke, M., Dignum, F., Dignum, V., Eiben, G., Fokkens, A., Grossi, D., Hindriks, K., Hoos, H., et al. (2020) · Computer · doi:10.1109/mc.2020.2996587 · Cited by 415
Auernhammer, J., & Roth, B. (2021) · Journal of Product Innovation Management · doi:10.1111/jpim.12594 · Cited by 260
Human–computer interaction: psychology as a science of design
Carroll, J. M. (1997) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1996.0101 · Cited by 226
Learning Visual Importance for Graphic Designs and Data Visualizations
Bylinskii, Z., Kim, N. W., O'Donovan, P., Alsheikh, S., Madan, S., Pfister, H., Durand, F., Russell, B., & Hertzmann, A. (2017) · Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3126594.3126653 · Cited by 156
Research works printed but not linked: 44
- I F Alexander, 2005, Persona-scenario-goal methodology for user-centered requirements engineering, 15th IEEE International Requirements Engineering Conference, 185
- A N Badre, 2002, Shaping Web usability: interaction design in context. Ubiquity
- N Bevan, 2015, ISO 9241-11 revised: What have we learnt about usability since 1998? Human-Computer Interaction: Design and Evaluation: 17th International Conference, Proceedings, Part I, 17, 143
- T Brown, 2020, Language models are few-shot learners, Advances in neural information processing systems, 33, 1877
- J M Carroll, 2003, Making use: scenario-based design of human-computer interactions
- A Cooper, 1999, The inmates are running the asylum
- A Cooper, 2003, About Face 2.0: The Essentials of Interaction Design
- J J Garrett, 2002, Uncovering the dynamics of human-AI hybrid performance: A qualitative meta-analysis of empirical studies, International Journal of Human-Computer Studies, 205
- T Gr�nsund, 2020, Augmenting the algorithm: Emerging human-in-the-loop work configurations, The Journal of Strategic Information Systems, 29
- E Hwang, 2023, Aligning language models to user opinions
- J S Hwang, 2026, Vibe Design: Human-in-the-loop AI Agents for UI Design with Large Language Models, Proceedings of the 59th Hawaii International Conference on System Sciences, 4453
- Iso, 1998, ISO 9241-11, Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) -Guidance on Usability, Ergonomics of human-system interaction -Usability methods supporting human-centred design
- Y Jiang, 2023, The Future of Computational Approaches for Understanding and Adapting User Interfaces. Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems
- K Jinwoo, 2012, Human Computer Interaction
- T Kojima, 2022, Large language models are zeroshot reasoners, Advances in neural information processing systems, 35, 22199
- M Kosinski, 2023, Theory of mind may have spontaneously emerged in large language models
- C Lee, 2020, GUIComp: A GUI Design Assistant with Real-Time, Multi-Faceted Feedback, Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1
- J Li, 2025, Vibe Coding for UX Design: Understanding UX Professionals' Perceptions of AI-Assisted Design and Development
- Z Liu, 2023, Fill in the blank: Contextaware automated text input generation for mobile gui testing, IEEE/ACM 45th International Conference on Software Engineering (ICSE), 1355
- E Mollick, 2022, ChatGPT is a tipping point for AI, Harvard Business Review, 1
- J Moriarty, 2024, Expanding scope and blurred boundaries of AI design, Medium. Moriarty, J., 2025. Design in the age of vibes, Medium
- S Mulder, 2006, The user is always right: A practical guide to creating and using personas for the web
- R S Nickerson, 1997, Human-computer interaction: Background and issues. Handbook of human-computer interaction, 3
- J Nielsen, 1993, A mathematical model of the finding of usability problems, Proceedings of the INTERACT'93 and CHI'93 conference on Human factors in computing systems, 206
- J Nielsen, 1990, Heuristic evaluation of user interfaces, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 249
- M Paris, 2023, ChatGPT Hits 100 Million Users, Google Invests in AI Bot and CatGPT Goes Viral
- J S Park, 2024, Generative agent simulations of 1,000 people
- J Pruitt, 2010, Measuring visual user interface complexity of mobile applications with metrics, Interacting with Computers, 207
- L Rukonic, 2024, Social Practices of Creating and Using UX Artifacts in Agile Organizations, HICSS, 6667
- L Salewski, 2023, In-context impersonation reveals large language models' strengths and biases, Advances in neural information processing systems, 36, 72044
- B Settles, 2009, Active learning literature survey
- Y Shao, 2024, Assisting in writing wikipedia-like articles from scratch with large language models
- 2010, An Introduction to Design Thinking PROCESS GUIDE
- L Sun, 2022, Meta-gui: Towards multi-modal conversational agents on mobile gui
- A Sutcliffe, 2002, Reciprocal human-machine learning: A theory and an instantiation for the case of message classification, Springer Science & Business Media. Te'eni
- T Teubner, 2003, Synthesis of behavioral models from scenarios, Business & Information Systems Engineering, 65, 99
- A Van Lamsweerde, 2001, Goal-oriented requirements engineering: A guided tour, Proceedings fifth ieee international symposium on requirements engineering, 249
- M W Van Someren, 1994, The think aloud method: a practical approach to modelling cognitive processes
- S Wadinambiarachchi, 2024, The effects of generative ai on design fixation and divergent thinking, Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 1
- B Wang, 2023, Enabling Conversational Interaction with Mobile UI using Large Language Models, Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 432
- L Wang, 2023, Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models
- C Wharton, 1994, SimUser: Generating Usability Feedback by Simulating Various Users Interacting with Mobile Applications, Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, Article 9
- B Xu, 2023, Expertprompting: Instructing large language models to be distinguished experts
- P Zaphiris, 2007, International design engineering technical conferences and computers and information in engineering conference 87318
Cited by
Lazaros, K., Vrahatis, A. G., & Kotsiantis, S. (2026) · Entropy · doi:10.3390/e28040377 · Cited by 50
Metadata
- Authors
- Hwang, J., & Kang, J.
- Affiliations
- Ajou University (KR)
- Corresponding author
- Kang, Juyoung
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2026
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W7131234329
- Cited by
- 1
- References
- 15
- Citations by year
- 2026: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Persona Design and Applications
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- agentic framework, human-in-the-loop, user interface prototyping, personas, interview transcripts, human-AI collaboration, design process
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.