CoChat
Sign Up
Cited by 24Usability and User Interface DesignInnovative Human-Technology InteractionSoftware Engineering Research

UICrit: Enhancing Automated Design Evaluation with a UI Critique Dataset

Open access
Duan, Peitong, Cheng, Chin-Yi, Li, Gang, Hartmann, Bjoern, Li, Yang·2024·Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology
DOI 10.1145/3654777.3676381
Open this paper in CoChat — free account

Abstract

Automated UI evaluation can be beneficial for the design process; for example, to compare different UI designs, or conduct automated heuristic evaluation. LLM-based UI evaluation, in particular, holds the promise of generalizability to a wide variety of UI types and evaluation tasks. However, current LLM-based techniques do not yet match the performance of human evaluators. We hypothesize that automatic evaluation can be improved by collecting a targeted UI feedback dataset and then using this dataset to enhance the performance of general-purpose LLMs. We present a targeted dataset of 3,059 design critiques and quality ratings for 983 mobile UIs, collected from seven designers, each with at least a year of professional design experience. We carried out an in-depth analysis to characterize the dataset’s features. We then applied this dataset to achieve a 55% performance gain in LLM-generated UI feedback via various few-shot and visual prompting techniques. We also discuss future applications of this dataset, including training a reward model for generative UI techniques, and fine-tuning a tool-agnostic multi-modal LLM that automates UI evaluation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3654777.3676381

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Each UI in the UICrit dataset was annotated by three professional designers, yielding 8,387 expert-authored heuristic critique comments grounded in bounding boxes.

    “Each UI was annotated by three professional designers, yielding 8,387 expert-authored heuristic critique comments grounded in bounding boxes.”

    From Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation (2026) · cites this paper as [16]

    The Data Collection section (p. 4) says only one worker was assigned per UI, not three. Results (p. 4) reports 3,059 critiques, not 8,387. The paper does support that the annotators had professional design experience, used design guidelines, and supplied bounding boxes for critiques. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • The UICrit dataset contains 1,000 mobile UI screenshots with task descriptions sampled from Rico.

    “We used the UICrit dataset [16], which contains 1,000 mobile UI screenshots with task descriptions sampled from Rico [15].”

    From Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation (2026) · cites this paper as [16]

    The Design Quality Rubric (p. 4) says 1,000 UIs were selected for annotation from CLAY, a cleaned subset of Rico. But Results (p. 4) says the completed UICrit dataset has 983 annotated screens, each with a main task. Thus the source supports the Rico connection and task descriptions, but not the claim that the dataset contains 1,000 screens. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Part of this paper’s reference list could not be split into single entries.

Flagged first, then by citations

Research works printed but not linked: 39

  1. [3][3] Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. 2022. A Dataset for Interactive Vision Language Navigation with Unknown Command Feasibility. In European Conference on Computer Vision (ECCV) . -
  2. [4][4] Souradeep Chakraborty, Zijun Wei, Conor Kelton, Seoyoung Ahn, Aruna Balasubramanian, Gregory J. Zelinsky, and Dimitris Samaras. 2023. Predicting Visual Attention in Graphic Design Documents. IEEE Transactions on Multimedia 25 (2023), 4478-4493. https://doi.org/10.1109/TMM.2022.3176942 - · DOI
  3. [5][5] Chin-Yi Cheng, Forrest Huang, Gang Li, and Yang Li. 2023. PLay: Parametrically conditioned layout generation using latent diffusion. arXiv preprint arXiv:2301.11529 (2023). - · arXiv
  4. [6][6] Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual Symposium on User Interface Software and Technology (UIST '17) . -
  5. [7][7] Michael Dorkenwald, Nimrod Barazani, Cees G. M. Snoek, and Yuki M. Asano. 2024. PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs. arXiv:2402.08657 [cs.CV] - · arXiv
  6. [8][8] Peitong Duan, Björn Hartmann, Karina Nguyen, Yang Li, Marti Hearst, and Meredith Ringel Morris. 2023. Towards Semantically-Aware UI Design Tools: Design, Implementation and Evaluation of Semantic Grouping Guidelines. In ICML 2023 Workshop on Artificial Intelligence and Human-Computer Interaction . https://research.google/pubs/pub52594/ -
  7. [9][9] Peitong Duan, Jeremy Warner, Yang Li, and Bjoern Hartmann. 2024. Generating Automatic Feedback on UI Mockups with Large Language Models. arXiv preprint arXiv:2403.13139 (2024). - · arXiv
  8. [10][10] Peitong Duan, Casimir Wierzynski, and Lama Nachman. 2020. Optimizing User Interface Layouts via Gradient Descent. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (<conf-loc>, <city>Honolulu</city>, <state>HI</state>, <country>USA</country>, </conf-loc>) (CHI '20) . Association for Computing Machinery, New York, NY, USA, 1-12. https://doi.org/10.1145/ 3313831.3376589 -
  9. [11][11] J.L. Fleiss et al. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin 76, 5 (1971), 378-382. -
  10. [12][12] Camilo Fosco, Vincent Casser, Amish Kumar Bedi, Peter O'Donovan, Aaron Hertzmann, and Zoya Bylinskii. 2020. Predicting Visual Importance Across Graphic Design Types. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '20) . Association for Computing Machinery, New York, NY, USA, 249-260. https://doi.org/10. 1145/3379337.3415825 -
  11. [13][13] Gemini Team Google. 2023. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023). - · arXiv
  12. [15][15] Jan Hartmann, Alistair Sutcliffe, and Antonella De Angeli. 2008. Towards a theory of user judgment of aesthetics and user interface quality. ACM Trans. Comput.-Hum. Interact. 15 (11 2008). https://doi.org/10.1145/1460355.1460357 - · DOI
  13. [16][16] Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. 2023. Layoutdm: Discrete diffusion model for controllable layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 10167-10176. -
  14. [17][17] Yue Jiang, Luis A. Leiva, Hamed Rezazadegan Tavakoli, Paul R. B. Houssel, Julia Kylmälä, and Antti Oulasvirta. 2023. UEyes: Understanding Visual Saliency across User Interface Types. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23) . Association for Computing Machinery, New York, NY, USA, Article 285, 21 pages. https://doi. org/10.1145/3544548.3581096 - · DOI
  15. [18][18] Chunggi Lee, Sanghoon Kim, Dongyun Han, Hongjun Yang, Young-Woo Park, Bum Chul Kwon, and Sungahn Ko. 2020. GUIComp: A GUI Design Assistant with Real-Time, Multi-Faceted Feedback. arXiv:2001.05684 [cs.HC] - · arXiv
  16. [20][20] Luis A. Leiva, Asutosh Hota, and Antti Oulasvirta. 2021. Enrico: A Dataset for Topic Modeling of Mobile UI Designs. In 22nd International Conference on HumanComputer Interaction with Mobile Devices and Services (Oldenburg, Germany) (MobileHCI '20) . Association for Computing Machinery, New York, NY, USA, Article 9, 4 pages. https://doi.org/10.1145/3406324.3410710 - · DOI · Paper page
  17. [21][21] Luis A. Leiva, Yunfei Xue, Avya Bansal, Hamed R. Tavakoli, Tuðçe Köroðlu, Jingzhou Du, Niraj R. Dayama, and Antti Oulasvirta. 2020. Understanding Visual Saliency in Mobile User Interfaces. In 22nd International Conference on HumanComputer Interaction with Mobile Devices and Services (Oldenburg, Germany) (MobileHCI '20) . Association for Computing Machinery, New York, NY, USA, Article 3, 12 pages. https://doi.org/10.1145/3379503.3403557 - · DOI
  18. [22][22] Gang Li, Gilles Baechler, Manuel Tragut, and Yang Li. 2022. Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems . Association for Computing Machinery, New Orleans, LA, USA, 1-13. https://doi.org/10.1145/ 3491102.3502042 -
  19. [23][23] Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. 2020. Mapping Natural Language Instructions to Mobile UI Action Sequences. CoRR abs/2005.03776 (2020). arXiv:2005.03776 https://arxiv.org/abs/2005.03776 - · arXiv
  20. [24][24] Jiawei Lin, Jiaqi Guo, Shizhao Sun, Zijiang Yang, Jian-Guang Lou, and Dongmei Zhang. 2024. LayoutPrompter: Awaken the Design Ability of Large Language Models. Advances in Neural Information Processing Systems 36 (2024). -
  21. [25][25] Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. 2024. MOKA: Open-Vocabulary Robotic Manipulation through Mark-Based Visual Prompting. arXiv:2403.03174 [cs.RO] - · arXiv
  22. [26][26] Kurt Luther, Jari-Lee Tolentino, Wei Wu, Amy Pavel, Brian P. Bailey, Maneesh Agrawala, Björn Hartmann, and Steven P. Dow. 2015. Structuring, Aggregating, and Evaluating Crowdsourced Design Critique. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (Vancouver, BC, Canada) (CSCW '15) . Association for Computing Machinery, New York, NY, USA, 473-485. https://doi.org/10.1145/2675133.2675283 - · DOI
  23. [27][27] Jean-bernard Martens and L.M.J. Meesters. 1998. Image dissimilarity. Signal Processing 70 (11 1998), 155-176. https://doi.org/10.1016/S0165-1684(98)00123-6 - · DOI
  24. [28][28] Walter T. Nakamura, Edson Cesar de Oliveira, Elaine H.T. de Oliveira, David Redmiles, and Tayana Conte. 2022. What factors affect the UX in mobile apps? A systematic mapping study on the analysis of app store reviews. Journal of Systems and Software 193 (2022), 111462. https://doi.org/10.1016/j.jss.2022.111462 - · DOI
  25. [29][29] Jakob Nielsen. 1994. Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, Massachusetts, USA) (CHI '94) . Association for Computing Machinery, New York, NY, USA, 152-158. https://doi.org/10.1145/191666.191729 - · DOI · Paper page
  26. [31][31] Jakob Nielsen and Rolf Molich. 1990. Heuristic Evaluation of User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, USA) (CHI '90) . Association for Computing Machinery, New York, NY, USA, 249-256. https://doi.org/10.1145/97243.97281 - · DOI · Paper page
  27. [33][33] Antti Oulasvirta, Samuli De Pascale, Janin Koch, Thomas Langerak, Jussi Jokinen, Kashyap Todi, Markku Laine, Manoj Kristhombuge, Yuxi Zhu, Aliaksei Miniukovich, Gregorio Palmas, and Tino Weinkauf. 2018. Aalto Interface Metrics (AIM): A Service and Codebase for Computational GUI Evaluation. In Adjunct Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST '18 Adjunct) . Association for Computing Machinery, New York, NY, USA, 16-19. https://doi.org/10.1145/3266037.3266087 - · DOI · Paper page
  28. [34][34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020 [cs.CV] - · arXiv
  29. [35][35] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. CoRR abs/1910.10683 (2019). arXiv:1910.10683 http://arxiv.org/abs/1910.10683 - · arXiv
  30. [36][36] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023. Android in the Wild: A Large-Scale Dataset for Android Device Control. arXiv:2307.10088 [cs.LG] - · arXiv
  31. [37][37] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics. https://arxiv.org/abs/1908.10084 - · arXiv
  32. [38][38] D Royce Sadler. 1989. Formative assessment and the design of instructional systems. Instructional science 18 (1989), 119-144. -
  33. [39][39] Chengyao Shen and Qi Zhao. 2014. Webpage Saliency. In ECCV . IEEE. -
  34. [40][40] Robert L Thorndike. 1953. Who belongs in the family? Psychometrika 18, 4 (1953), 267-276. -
  35. [41][41] Kashyap Todi, Daryl Weir, and Antti Oulasvirta. 2016. Sketchplore: Sketch and Explore with a Layout Optimiser. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (Brisbane, QLD, Australia) (DIS '16) . Association for Computing Machinery, New York, NY, USA, 543-555. https://doi.org/10. 1145/2901790.2901817 -
  36. [42][42] Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research 9 (2008), 2579-2605. http://www. jmlr.org/papers/v9/vandermaaten08a.html -
  37. [43][43] Jason Wu, Siyan Wang, Siman Shen, Yi-Hao Peng, Jeffrey Nichols, and Jeffrey P Bigham. 2023. WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (, Hamburg, Germany,) (CHI '23) . Association for Computing Machinery, New York, NY, USA, Article 286, 14 pages. https://doi.org/10.1145/ 3544548.3581158 -
  38. [44][44] Ziming Wu, Yulun Jiang, Yiding Liu, and Xiaojuan Ma. 2020. Predicting and Diagnosing User Engagement with Mobile UI Animation via a Data-Driven Approach. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (<conf-loc>, <city>Honolulu</city>, <state>HI</state>, <country>USA</country>, </conf-loc>) (CHI '20) . Association for Computing Machinery, New York, NY, USA, 1-13. https://doi.org/10.1145/3313831.3376324 - · DOI
  39. [45][45] Weiwen Xu, Deng Cai, Zhisong Zhang, Wai Lam, and Shuming Shi. 2023. Reasons to reject? aligning language models with judgments. arXiv preprint arXiv:2312.14591 (2023). · arXiv

Cited by

Sorted by citations

Authors
Duan, P., Cheng, C.-Y., Li, G., Hartmann, B., & Li, Y.
Affiliations
  • University of California, Berkeley (US)
  • Google (United States) (US)
  • EECS, UC Berkeley, United States
ORCID
Venue
Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology
Publisher
ACM
Published
2024-10-11
Pages
1-17
DOI
10.1145/3654777.3676381
OpenAlex
W4403333969
Cited by
24
References
3
Citations by year
2026: 16 · 2025: 8
FWCI
7.73
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Symposium on User Interface Software and Technology (UIST) (publishedVersion, cc-by-nd)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
design criticism, heuristic evaluation, few-shot prompting, visual prompting, reward modeling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review