UISGPT: Automated Mobile UI Design Smell Detection with Large Language Models
Open accessAbstract
Manual inspection and remediation of guideline violations (UI design smells) is a knowledge-intensive, time-consuming, and context-related task that requires a high level of expertise. This paper proposes UISGPT, a novel end-to-end approach for automatically detecting user interface (UI) design smells and explaining each violation of specific design guidelines in natural language. To avoid hallucinations in large language models (LLMs) and achieve interpretable results, UISGPT uses few-shot learning and least-to-most prompting strategies to formalize design guidelines. To prevent the model from exceeding the input window size and for the enhancement of the logic in responses, UISGPT divides design smell detection into the following three subtasks: design guideline formalization, UI component information extraction, and guideline validation. The experimental results show that UISGPT performs effectively in automatically detecting design violations (F1 score of 0.729). In comparison to the latest LLM methods, the design smell reports generated by UISGPT have higher contextual consistency and user ratings.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent generative models can offload parts of UI evaluation: metrics quantify visual properties, predictive models simulate user behaviors, and LLMs detect heuristic violations from screenshots.
“Metrics can quantify visual properties [71], predictive models can simulate user behaviors [17, 91], and Large Language Models (LLMs) can detect heuristic violations directly from screenshots [17, 85, 93].”
From Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation (2026) · cites this paper as [93]
Most prior work on AI-supported UI evaluation-including the single-pass baseline condition-treats models as automated judges: systems that take a screen as input and output a fixed list of issues, with users interacting mainly through coarse-grained, post-hoc prompts to revise it.
“Most prior work on AI-supported UI evaluation-including our single-pass baseline condition-treats models as automated judges : systems that take a screen as input and output a fi xed list of issues, with users interacting mainly through coarse-grained, post-hoc prompts to revise it [17, 85, 93].”
From Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation (2026) · cites this paper as [93]
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529786 · Cited by 4,332 · see citing works
Comparing Individual Means in the Analysis of Variance
Tukey, J. W. (1949) · Biometrics · doi:10.2307/3001913 · Cited by 4,302 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752 · see citing works
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573 · see citing works
Statistical Methods for Research Workers.
Jowett, G. H., & Fisher, R. A. (1956) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.2307/2985747 · Cited by 517 · see citing works
Rico: A Mobile App Dataset for Building Data-Driven Design Applications
Deka, B., Huang, Z., Franzen, C., Hibschman, J., Afergan, D., Li, Y., Nichols, J., & Kumar, R. (2017) · Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3126594.3126651 · Cited by 482 · see citing works
Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey
Buhrmester, V., Münch, D., & Arens, M. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3040048 · Cited by 403 · see citing works
Guided, stochastic model-based GUI testing of Android apps
Su, T., Meng, G., Chen, Y., Wu, K., Yang, W., Yao, Y., Pu, G., Liu, Y., & Su, Z. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106298 · Cited by 378 · see citing works
DroidBot: a lightweight UI-Guided test input generator for android
Li, Y., Yang, Z., Guo, Y., & Chen, X. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-c.2017.8 · Cited by 243 · see citing works
Few-shot training LLMs for project-specific code-summarization
Ahmed, T., & Devanbu, P. (2022) · Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3551349.3559555 · Cited by 240 · see citing works
Impact of Code Language Models on Automated Program Repair
Jiang, N., Liu, K., Lutellier, T., & Tan, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00125 · Cited by 202 · see citing works
Reinforcement learning based curiosity-driven testing of Android applications
Pan, M., Huang, A., Wang, G., Zhang, T. Y., & Li, X. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397354 · Cited by 180 · see citing works
Prompting Is All You Need: Automated Android Bug Replay with Large Language Models
Feng, S., & Chen, C. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3608137 · Cited by 180 · see citing works
Retrieval-Based Prompt Selection for Code-Related Few-Shot Learning
Nashid, N., Sintaha, M., & Mesbah, A. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00205 · Cited by 178 · see citing works
Enabling Conversational Interaction with Mobile UI using Large Language Models
Wang, B., Li, G., & Li, Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580895 · Cited by 156 · see citing works
An extensive study on pre-trained models for program understanding and generation
Zeng, Z., Tan, H., Zhang, H., Li, J., Zhang, Y., & Zhang, L. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534390 · Cited by 153 · see citing works
Liu, Z., Chen, C., Wang, J., Chen, M., Wu, B., Che, X., Wang, D., & Wang, Q. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639180 · Cited by 143 · see citing works
Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing
Liu, Z., Chen, C., Wang, J., Che, X., Huang, Y.-K., Hu, J., & Wang, Q. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00119 · Cited by 130 · see citing works
Guided GUI testing of android apps with minimal restart and approximate learning
Choi, W., Necula, G. C., & Sen, K. (2013) · ACM SIGPLAN Notices · doi:10.1145/2544173.2509552 · Cited by 125 · see citing works
Liu, Z., Chen, C., Wang, J., Huang, Y., Hu, J., & Wang, Q. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416547 · Cited by 104 · see citing works
Automated reporting of GUI design violations for mobile apps
Moran, K., Li, B., Bernal-Cárdenas, C., Jelf, D., & Poshyvanyk, D. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180246 · Cited by 101 · see citing works
How developers perceive smells in source code: A replicated study
Taibi, D., Janes, A., & Lenarduzzi, V. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.08.008 · Cited by 100 · see citing works
Generating Automatic Feedback on UI Mockups with Large Language Models
Duan, P., Warner, J., Li, Y., & Hartmann, B. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642782 · Cited by 83 · see citing works
Wireframe-based UI Design Search through Image Autoencoder
Chen, J., Chen, C., Xing, Z., Xia, X., Zhu, L., Grundy, J., & Wang, J. (2020) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3391613 · Cited by 82 · see citing works
Gallery D.C.: Design Search and Knowledge Discovery through Auto-created GUI Component Gallery
Chen, C., Feng, S., Xing, Z., Liu, L. L., Zhao, S., & Wang, J. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359282 · Cited by 82 · see citing works
Benchmarking automated GUI testing for Android against real-world bugs
Su, T., Wang, J., & Su, Z. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468620 · Cited by 79 · see citing works
LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities
Yu, S., Fang, C., Ling, Y., Wu, C., & Chen, Z. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00029 · Cited by 77 · see citing works
GUIComp: A GUI Design Assistant with Real-Time, Multi-Faceted Feedback
Lee, C., Kim, S.-H., Han, D., Yang, H., Park, Y., Kwon, B. C., & Ko, S. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376327 · Cited by 63 · see citing works
Nighthawk: Fully Automated Localizing UI Display Issues via Visual Understanding
Liu, Z., Chen, C., Wang, J., Huang, Y.-K., Hu, J., & Wang, Q. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3150876 · Cited by 59 · see citing works
Zhao, D., Xing, Z., Chen, C., Xu, X., Zhu, L., Li, G., & Wang, J. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380411 · Cited by 55 · see citing works
Alégroth, E., Gao, Z., de Oliveira, R. A. P., & Memon, A. M. (2015) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2015.7102584 · Cited by 54 · see citing works
AXNav: Replaying Accessibility Tests from Natural Language
Taeb, M., Swearngin, A., Schoop, E., Cheng, R., Jiang, Y., & Nichols, J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642777 · Cited by 52 · see citing works
Don’t Do That! Hunting Down Visual Design Smells in Complex UIs Against Design Guidelines
Yang, B. W., Xing, Z., Xia, X., Chen, C., Ye, D., & Li, S. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00075 · Cited by 52 · see citing works
Self-citationA Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility
Burns, A., Arsan, D., Agrawal, S., Kumar, R., Saenko, K., & Plummer, B. A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20074-8_18 · Cited by 32 · see citing works
GLIB: towards automated test oracle for graphically-rich applications
Chen, K., Li, Y., Chen, Y., Fan, C., Hu, Z., & Yang, W. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468586 · Cited by 31 · see citing works
Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis
Schoop, E., Zhou, X., Li, G., Chen, Z., Hartmann, B., & Li, Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517497 · Cited by 30 · see citing works
Enhancing Exploratory Testing by Large Language Model and Knowledge Graph
Su, Y., Liao, D., Xing, Z., Huang, Q. Y., Xie, M., Lu, Q., & Xu, X. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639157 · Cited by 23 · see citing works
Automated Repair of Size-Based Inaccessibility Issues in Mobile Applications
Alotaibi, A. S., Chiou, P., & Halfond, W. G. J. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678625 · Cited by 23 · see citing works
Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents
Deng, S., Xu, W., Sun, H., Liu, W., Tan, T., Liujianfeng, L., Li, A., Luan, J., Wang, B., Yan, R., et al. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.478 · Cited by 22 · see citing works
The Metamorphosis: Automatic Detection of Scaling Issues for Mobile Apps
Su, Y., Chen, C., Wang, J., Liu, Z., Wang, D., Li, S., & Wang, Q. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556935 · Cited by 20 · see citing works
Self-citationIssa, A. A., Sillito, J., & Garousi, V. (2012) · IEEE International Symposium on Web Site Evolution (WSE) · doi:10.1109/wse.2012.6320526 · Cited by 20 · see citing works
Automating GUI-based Software Testing with GPT-3
Zimmermann, D., & Koziolek, A. (2023) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw58534.2023.00022 · Cited by 18 · see citing works
OwlEyes-online: a fully automated platform for detecting and localizing UI display issues
Su, Y., Liu, Z., Chen, C., Wang, J., & Wang, Q. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473109 · Cited by 16 · see citing works
Mobile-UI-Repair: a deep learning based UI smell detection technique for mobile user interface
Ali, A., Xia, Y., Navid, Q., Khan, Z. A., Khan, J. A., Aldakheel, E. A., & Khafaga, D. S. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2028 · Cited by 11 · see citing works
Checking conformance of applications against GUI policies
Zhang, Z., Feng, Y., Ernst, M. D., Porst, S., & Dillig, I. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468561 · Cited by 9 · see citing works
Liu, Z. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510454.3522681 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Using LLMs to enhance code quality: A systematic literature review
Alomari, N., Redah, M., Ashraf, A., & Alshayeb, M. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107960 · Cited by 4 · see citing works
Large language modeling of hallucinatory problem mitigation based on the wheel of emotions
Wang, Z., Wang, J., Zhou, L., & You, F. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107996 · Cited by 4 · see citing works
Massenon, R., Gambo, I., & Khan, J. A. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107996 · Cited by 2 · see citing works
Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation
Ko, J., Choi, J., Morales, C., Kim, D., & Ko, M. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790929 · Cited by 1 · see citing works
Intelligent Meal Planning: A Generative LLM-Based Autonomous Agent Application
Howlett, E., Catlett, J., & Memari, M. (2025) · Intermountain Engineering, Technology and Computing (IETC) · doi:10.1109/ietc64455.2025.11039478 · Cited by 1 · see citing works
Aljohani, B., & Aljuhani, A. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3642 · Cited by 1 · see citing works
CritiqueCrew: Orchestrating Multi-Perspective Conversational Design Critique
Chen, X., Zhou, J., Shu, Y., Wang, R., & Liu, Q. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791391 · Cited by 1 · see citing works
Accessibility Issue Detection and Repair in Mobile Applications: A Systematic Literature Review
Liu, Z., Yao, D., Liu, H., Xu, C., & Huang, H. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3809496
mUXSmellDetection: An LLM-Assisted Framework for Automated UX Smell Detection in Mobile Apps
Al‐Shammare, H., Alshayeb, M., & Baslyman, M. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2730793
Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach
Mao, R., Fang, Z., Guo, T., Yang, H., Li, Y., Peng, S., Huang, M., Lin, X., Liu, S., Li, X., et al. (2026) · doi:10.1109/cvprf71909.2026.00862
- Authors
- Yang, B., & Li, S.
- Affiliations
- Zhejiang University (CN)
- Corresponding author
- Yang, Bo
- ORCID
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2024-08-07
- Volume
- 13
- Issue
- 16
- Pages
- 3127
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W4401422911
- Cited by
- 10
- References
- 50
- Citations by year
- 2026: 6 · 2025: 4
- FWCI
- 3.77
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Mobile and Web Applications
- Keywords
- mobile interface design, large language models, natural language explanations, few-shot learning, least-to-most prompting, hallucination mitigation, explainable AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
