Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset
Open accessAbstract
Using vision-language models (VLMs) in web development presents a promising strategy to increase efficiency and unblock no-code solutions: by providing a screenshot or a sketch of a UI, a VLM could generate the code to reproduce it, for instance in a language like HTML. Despite the advancements in VLMs for various tasks, the specific challenge of converting a screenshot into a corresponding HTML has been minimally explored. We posit that this is mainly due to the absence of a suitable, high-quality dataset. This work introduces WebSight, a synthetic dataset consisting of 2 million pairs of HTML codes and their corresponding screenshots. We fine-tune a foundational VLM on our dataset and show proficiency in converting webpage screenshots to functional HTML code. To accelerate the research in this area, we open-source WebSight.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
WebSight is a dataset comprising approximately 0.8 million synthesized samples generated by LLMs.
“WebSight [27] is another dataset comprising approximately 0.8 million synthesized samples generated by LLMs.”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [27]
The Related work passage says the earlier WebSight-v0.1 release contained 823K synthetic screenshot–HTML pairs, and the dataset-construction passages describe using LLMs to generate the HTML. But the paper describes the current WebSight-v0.2 dataset as containing 2 million pairs; the citing sentence omits the v0.1 qualifier needed for its 0.8 million figure. (LLM, conf=0.88)
Experimental — accuracy not yet measuredExisting datasets are either too small to provide meaningful generalization or consist of synthetic data that does not fully capture the complexity and variability of real-world webpage designs.
“However, this approach faces a significant limitation since existing datasets are either too small to provide meaningful generalization [21,45] or consist of synthetic data that does not fully capture the complexity and variability of real-world webpage designs [27,54].”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [27]
The paper describes WebSight as a synthetic dataset, and §5.1 reports that a model trained on it struggles with complex layouts and designs substantially different from its training data. This supports a limitation of this synthetic dataset, but the paper does not establish that synthetic datasets generally fail to capture the complexity and variability of real-world webpage designs. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe benchmark examples are much more challenging and cover a wider spectrum of complexities than WebSight.
“Overall, the examples in our benchmark are much more challenging and cover a wider spectrum of complexities than prior efforts like WebSight.”
From Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering (2025) · cites this paper as Laurençon et al., 2024
The benchmark uses real-world webpages rather than synthetically generated ones to best reflect realistic use cases.
“To best reflect realistic use cases, we use realworld webpages (examples in Figure 1) in the wild as our test examples rather than synthetically generated ones as in prior works (Soselia et al., 2023; Laurençon et al., 2024).”
From Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering (2025) · cites this paper as Laurençon et al., 2024
Webpage code from online websites is often lengthy and includes redundant elements, which significantly impairs the model's ability to accurately learn the correlation between webpage code and screenshots.
“Our investigation reveals that webpage code from online websites is often lengthy and includes redundant elements, which significantly impairs the model's ability to accurately learn the correlation between webpage code and screenshots [27].”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [27]
WebCoder, fine-tuned from the smaller Pix2Struct-1.3B, outperforms both Design2Code-18B and WebSight VLM-8B across all evaluation metrics, including CLIP-based visual similarity, low-level appearance accuracy, and TreeBLEU.
“Compared to the two fine-tuned baselines, Design2Code-18B [45] and WebSight VLM-8B [27], our model, fine-tuned from the smaller Pix2Struct-1.3B,outperforms both across all evaluation metrics, including CLIP-based visual similarity [35], low-level appearance accuracy [45], and our proposed TreeBLEU to measure the structural hierarchy recall.”
From WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs (2025) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 5
- Tony Beltramelli. pix2code: Generating code from a graphical user interface screenshot. arXiv preprint arXiv:1705.07962, 2017. · arXiv
- Hugo Laurençon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. In Thirty- seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=SKN2hflBIZ.
- Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2Struct: Screenshot parsing as pretraining for visual language understanding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 18893-18912. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr. press/v202/lee23g.html.
- Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation, 2024b. Tuan Anh Nguyen and Christoph Csallner. Reverse engineering mobile application user interfaces with remaui (t). 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 248-259, 2015. URL https://api.semanticscholar.org/CorpusID:7499368.
- Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion- scale corpus of images interleaved with text. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum? id=tOd8rSjcWz.
Cited by
Sorted by citations
WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs
Gui, Y., Li, Z., Wan, Y., Shi, Y., Zhang, H., Chen, B., Su, Y., Chen, D., Wu, S., Zhou, X., et al. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714889 · Cited by 17 · see citing works
Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering
Si, C., Zhang, Y., Li, R., Yang, Z., Liu, R., & Yang, D. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.199 · Cited by 17 · see citing works
“To best reflect realistic use cases, we use real-world webpages (examples in Figure 1) in the wild as our test examples rather than synthetically generated ones as in prior works (Soselia et al., 2023; Laurencon et al., 2024b).”
InfluentialXiao, J., Wan, Y., Huo, Y., Wang, Z., Xu, X., Wang, W., Xu, Z., Wang, Y., & Lyu, M. R. (2024) · arXiv · doi:10.48550/arxiv.2411.03292 · Cited by 7 · see citing works
IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web
Guo, H., Zhang, W., Chen, J., Gu, Y., Yang, J., Du, J., Cao, S., Hui, B., Liu, T., Ma, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.18980
“WebSight is the top performer in balancing both element and layout accuracy.”
Influential
- Authors
- Laurençon, H., Tronchon, L., & Sanh, V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2403.09029
- OpenAlex
- W4392867139
- Cited by
- 2
- References
- 0
- Citations by year
- 2025: 1 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Mobile and Web Applications
- Keywords
- vision-language models, synthetic dataset, web development, no-code
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
