Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering
Open accessAbstract
Generative AI has made rapid advancements in recent years, achieving unprecedented capabilities in multimodal understanding and code generation. This can enable a new paradigm of front-end development in which multimodal large language models (MLLMs) directly convert visual designs into code implementations. In this work, we construct Design2Code - the first real-world benchmark for this task. Specifically, we manually curate 484 diverse real-world webpages as test cases and develop a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input. We also complement automatic metrics with comprehensive human evaluations to validate the performance ranking. To rigorously benchmark MLLMs, we test various multimodal prompting methods on frontier models such as GPT-4o, GPT-4V, Gemini, and Claude. Our fine-grained break-down metrics indicate that models mostly lag in recalling visual elements from the input webpages and generating correct layout designs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
To ensure a fair comparison, all baselines employ the one-pass generation strategy.
“Therefore, to ensure a fair comparison, all baselines will employ the one-pass generation strategy.”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [45]
The paper’s Self-Revision Prompting section describes a second generation step: the model receives its initial code and rendered webpage, then revises the code. Table 1 also evaluates self-revision alongside direct and text-augmented prompting. Thus, the paper does not say that all its baselines use one-pass generation; the citing sentence’s future-tense choice about its own baselines is not established by this source. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredExisting UI code generation benchmarks only contain static web pages and ignore dynamic interaction.
“However, existing benchmarks only contain static web pages for evaluation and ignore the dynamic interaction, limiting the practicality, usability and user engagement of the generated webpages.”
From Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping (2024) · cites this paper as [22]
The Introduction describes screenshot-based visual evaluation, and ‘Making Webpages Stand-alone’ says Design2Code removes scripts and links. The Conclusion identifies dynamic user interaction as future work. These passages support the point for Design2Code, but do not establish that all existing benchmarks contain only static pages or that this limits practicality, usability, and user engagement. (LLM, conf=0.82)
Experimental — accuracy not yet measuredThe complexity of code generation increases with the growth in the total number of HTML tags, the diversity of unique tags, and the depth of the DOM tree.
“as highlighted in a recent study [45], the complexity of code generation increases with the growth in the total number of HyperText Markup Language (HTML) tags, the diversity of unique tags, and the depth of the Document Object Model (DOM) tree.”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [45]
Section 4.5 and the Table 7 caption say that webpages with more HTML tags tend to be harder for the model to generate. Section 2.2 treats DOM-tree depth as a measure of complexity and records the number of unique tags, but the cited passages do not establish that generation difficulty increases as either of those measures grows. (LLM, conf=0.85)
Experimental — accuracy not yet measuredDesign2Code contains only 484 real-world samples, intended solely for testing and insufficient for effective fine-tuning.
“For instance, Design2Code [45] contains only 484 real-world samples, intended solely for testing and insufficient for effective fine-tuning.”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [45]
Section 2.1 (Test Set Curation; page unavailable) says the authors curated 484 real-world webpages for their benchmark. Section E (Finetuning details; page unavailable) says they fine-tuned on WebSight instead, but does not establish that 484 samples are insufficient for effective fine-tuning. The claim of insufficiency is therefore unsupported by this source. (LLM, conf=0.88)
Experimental — accuracy not yet measuredModern MLLMs, such as GPT-4V, exhibit a notable decline in performance when confronted with real-world webpage designs that feature complex structures and a larger variety of unique HTML tags.
“Modern MLLMs, such as GPT-4V, also exhibit a notable decline in performance when confronted with real-world webpage designs that feature complex structures and a larger variety of unique HTML tags [45].”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [45]
Section 4.5 reports a significant performance drop on hard real-world webpages for GPT-4o, and Appendix Table 7 identifies the total number of HTML tags as a strong difficulty indicator for GPT-4V. The copy does not establish that a larger variety of unique tags causes the decline, and the explicitly reported drop is for GPT-4o rather than GPT-4V. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Existing datasets are either too small to provide meaningful generalization or consist of synthetic data that does not fully capture the complexity and variability of real-world webpage designs.
“However, this approach faces a significant limitation since existing datasets are either too small to provide meaningful generalization [21,45] or consist of synthetic data that does not fully capture the complexity and variability of real-world webpage designs [27,54].”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [45]
A significant disparity exists between WebSight samples and real-world data.
“However, a significant disparity exists between these samples and real-world data [45].”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [45]
WebCoder, fine-tuned from the smaller Pix2Struct-1.3B, outperforms both Design2Code-18B and WebSight VLM-8B across all evaluation metrics, including CLIP-based visual similarity, low-level appearance accuracy, and TreeBLEU.
“Compared to the two fine-tuned baselines, Design2Code-18B [45] and WebSight VLM-8B [27], our model, fine-tuned from the smaller Pix2Struct-1.3B,outperforms both across all evaluation metrics, including CLIP-based visual similarity [35], low-level appearance accuracy [45], and our proposed TreeBLEU to measure the structural hierarchy recall.”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [45]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 38
- Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Miko- laj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204.14198. · arXiv
- Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision- language model for understanding, localization, text reading, and beyond. Preprint, arXiv:2308.12966. · arXiv
- Tony Beltramelli. 2018. pix2code: Generating code from a graphical user interface screenshot. In Pro- ceedings of the ACM SIGCHI Symposium on Engi- neering Interactive Computing Systems, pages 1-6.
- Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. 2023. Nougat: Neural optical understanding for academic documents. Preprint, arXiv:2308.13418. · arXiv
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Ed- wards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Moham- mad Bavarian, Clemens Winter, Philippe Tillet, Fe- lipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluat- ing large language models trained on code. ArXiv, abs/2107.03374. · arXiv
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. Preprint, arXiv:2010.11929. · arXiv
- Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. ArXiv, abs/2305.14387. · arXiv
- Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida I. Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. ArXiv, abs/2204.05999. · arXiv
- Charles D Gilbert and Wu Li. 2013. Top-down influ- ences on visual processing. Nature Reviews Neuro- science, 14(5):350-363.
- Google. 2023. Gemini: A family of highly capable multimodal models. ArXiv, abs/2312.11805. · arXiv
- Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wen- feng Liang. 2024. Deepseek-coder: When the large language model meets programming -the rise of code intelligence. Preprint, arXiv:2401.14196. · arXiv
- Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. 2023. Cogagent: A visual language model for gui agents. Preprint, arXiv:2312.08914. · arXiv
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685. · arXiv
- Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xiny- ing Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. ArXiv, abs/2310.01798. · arXiv
- Melody Y Ivory and Rodrick Megraw. 2005. Evolution of web site design patterns. ACM Transactions on Information Systems (TOIS), 23(4):463-497.
- Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. 2024a. Mantis: Interleaved multi-image instruction tuning. Preprint, arXiv:2405.01483. · arXiv
- Yue Jiang, Changkong Zhou, Vikas Garg, and Antti Oulasvirta. 2024b. Graph4gui: Graph neural net- works for representing graphical user interfaces. In International Conference on Human Factors in Com- puting Systems.
- Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Won- seok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. Ocr-free document under- standing transformer. In European Conference on Computer Vision, pages 498-517. Springer.
- Hugo Laurencon, Léo Tronchon, and Victor Sanh. 2024b. Unlocking the conversion of web screenshots into html code with the websight dataset. Preprint, arXiv:2403.09029. · arXiv
- Triet HM Le, Hao Chen, and Muhammad Ali Babar. 2020. Deep learning for source code modeling and generation: Models, applications, and challenges. ACM Computing Surveys (CSUR), 53(3):1-38.
- Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nourhan Fahmy, Urvashi Bhat- tacharyya, W. Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jana Ebert, Tri Dao, Mayank Mishra, Alexander Gu, Jennifer Robinson, Carolyn Jane Anderson, Bren- dan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Car- los Muñoz Ferrandis, Sean M. Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023. Starcoder: may the source be with you! ArXiv, abs/2305.06161. · arXiv
- Jenny T Liang, Chenyang Yang, and Brad A. Myers. 2023. A large-scale survey on the usability of ai pro- gramming assistants: Successes and challenges. In International Conference on Software Engineering.
- Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhu- oshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. 2024. Deepseek-vl: Towards real-world vision-language understanding. Preprint, arXiv:2403.05525. · arXiv
- M Ronnier Luo, Guihua Cui, and Bryan Rigg. 2001. The development of the cie 2000 colour-difference formula: Ciede2000. Color Research & Applica- tion: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Cen- tre Français de la Couleur, 26(5):340-350.
- Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wen- hui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, and Furu Wei. 2023. Kosmos-2.5: A multimodal literate model. Preprint, arXiv:2309.11419. · arXiv
- Shunji Mori, Ching Y Suen, and Kazuhiko Yamamoto. 1992. Historical review of ocr research and develop- ment. Proceedings of the IEEE, 80(7):1029-1058.
- Tuan Anh Nguyen and Christoph Csallner. 2015. Re- verse engineering mobile application user interfaces with remaui (t). In 2015 30th IEEE/ACM Interna- tional Conference on Automated Software Engineer- ing (ASE), pages 248-259. IEEE Computer Society.
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. Preprint, arXiv:2103.00020. · arXiv
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former. arXiv e-prints.
- Alex Robinson. 2019. Sketch2code: Generating a web- site from a paper mockup. ArXiv, abs/1905.13750. · arXiv
- Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, I. Evtimov, Joanna Bitton, Manish P Bhatt, Cristian Cantón Ferrer, Aaron Grattafiori, Wen- han Xiong, Alexandre D'efossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code llama: Open foundation models for code. ArXiv, abs/2308.12950. · arXiv
- Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Preprint, arXiv:2210.08402. · arXiv
- Alexandru Telea. 2004. An image inpainting technique based on the fast marching method. Journal of graph- ics tools, 9(1):23-34.
- Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971. · arXiv
- Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
- Helena Vasconcelos, Gagan Bansal, Adam Fourney, Qingzi Vera Liao, and Jennifer Wortman Vaughan. 2023. Generation probabilities are not enough: Exploring the effectiveness of uncertainty high- lighting in ai-powered code completions. ArXiv, abs/2302.07248. · arXiv
- Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. arXiv preprint arXiv:1704.01696. · arXiv
- Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less is more for alignment. Preprint, arXiv:2305.11206. · arXiv
Cited by
Sorted by citations
Office-in-the-Loop: an investigation into Agentic AI for advanced building HVAC control systems
Sawada, T., Mizuno, M., Hasegawa, T., Yokoyama, K., & Kono, M. (2025) · Data-Centric Engineering · doi:10.1017/dce.2025.10010 · Cited by 23 · see citing works
DeclarUI: Bridging Design and Development with Automated Declarative UI Code Generation
Zhou, T., Zhao, Y., Hou, X., Sun, X., Chen, K., & Wang, H. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715726 · Cited by 19 · see citing works
WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs
Gui, Y., Li, Z., Wan, Y., Shi, Y., Zhang, H., Chen, B., Su, Y., Chen, D., Wu, S., Zhou, X., et al. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714889 · Cited by 17 · see citing works
Xiao, J., Wan, Y., Huo, Y., Wang, Z., Xu, X., Wang, W., Xu, Z., Wang, Y., & Lyu, M. R. (2024) · arXiv · doi:10.48550/arxiv.2411.03292 · Cited by 7 · see citing works
Yang, C., Liu, Y., & Yin, C. (2024) · Computer Science Review · doi:10.1016/j.cosrev.2024.100637 · Cited by 5 · see citing works
GUIMigrator: Semantics-Preserving Transpilation from Android XML to Compose and SwiftUI
橘高, 義., Hu, X., Yang, X., & Xia, X. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808215 · Cited by 1 · see citing works
- Authors
- Si, C., Zhang, Y., Li, R., Yang, Z., Liu, R., & Yang, D.
- Affiliations
- Georgia Institute of Technology (US)
- Stanford University (US)
- Microsoft (United States) (US)
- Google DeepMind (United Kingdom) (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2403.03163
- OpenAlex
- W4392538933
- Cited by
- 4
- References
- 0
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Office of Naval Research
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Information Systems › Mobile and Web Applications
- Keywords
- multimodal large language models, front-end web development, evaluation metrics, human evaluation, multimodal prompting, GPT-4V, Gemini, Claude, web page layout
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
