pix2code: Generating Code from a Graphical User Interface Screenshot
The DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index.
Abstract
Transforming a graphical user interface screenshot created by a designer into computer code is a typical task conducted by a developer in order to build customized software, websites, and mobile applications. In this paper, we show that deep learning methods can be leveraged to train a model end-to-end to automatically reverse engineer user interfaces and generate code from a single input image with over 77% of accuracy for three different platforms (i.e. iOS, Android and web-based technologies).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 2×
“n. Therefore, pre-training the language model to learn vectorial representations would allow the relationships between tokens in the DSL to be inferred (i.e. learning word embeddings such as word2vec [13]) and as a result alleviate semantical error in the generated code. Furthermore, one-hot encoding does not scale to…”
BackgroundMethodShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021 · cited 1×
“Generative Adversarial Networks (GANs) [5] have shown to be extremely powerful at generating images and sequences [2, 20].”
BackgroundRecurrent Neural Network Regularization
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 2,123
Towards Diverse and Natural Image Descriptions via a Conditional GAN
Dai, B., Fidler, S., Urtasun, R., & Lin, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.323 · Cited by 477
Research works printed but not linked: 5
- Matej Balog Alexander L Gaunt Marc Brockschmidt Sebastian Nowozin and Daniel Tarlow. 2016. DeepCoder: Learning to Write Programs. arXiv preprint arXiv:1611.01989 (2016). Matej Balog Alexander L Gaunt Marc Brockschmidt Sebastian Nowozin and Daniel Tarlow. 2016. DeepCoder: Learning to Write Programs. arXiv preprint arXiv:1611.01989 (2016). · arXiv
- Gaunt Alexander L, 2016, Terpret: A probabilistic programming language for program induction. arXiv preprint arXiv:1608.04428 · arXiv
- Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680. Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.
- Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 (2013). Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 (2013). · arXiv
- 10.5555/2627435.2670313 · doi:10.5555/2627435.2670313
Cited by
Items 201–239 of 239
Haptic Auditory Feedback for Enhanced Image Description: A Study of User Preferences and Performance
Alkhathlan, M., Tlachac, M., & Rundensteiner, E. A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42280-5_14 · Cited by 2
On Improving Management of Duplicate Video-Based Bug Reports
Yan, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3639786
Draw2Web: From Hand-Drawn Images to HTML + CSS
Immanuel, M. R., & Mr. Rohan, S. (2024) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-18728
Hsueh, Y.-T., Chen, H., & Chen, T.-Y. (2024) · International Conference on Information and Education Technology (ICIET) · doi:10.1109/iciet60671.2024.10542829
مسح لتقنيات إنشاء الكود تلقائيًا من تصميمات واجهة المستخدم ذات الدقة المختلفة
جاد, م. س. ز., مرعي, م., & يعقوب, أ. ا. (2024) · النشرة المعلوماتية في الحاسبات والمعلومات · doi:10.21608/fcihib.2024.262296.1106 · Cited by 1
RESEARCH AND DEVELOPMENT OF METHODS OF DYNAMIC ANALYSIS OF CODE FOR CREATING SELF-ADAPTIVE SOFTWARE
Бершадский, А. М., Сергеевич, Б. А., Евсеева, Ю. И., & Gudkov, A. A. (2018) · Modelirovanie, optimizaciâ i informacionnye tehnologii. · doi:10.26102/2310-6018/2018.23.4.009
Merging Creativity with Computation in Sketch-to-Code Transitions
Calò, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-59235-5_16
A novel hybrid CNN-GRU model for website-based user interface code generator
Alhasyim, A. G., & Shabrina, N. H. (2024) · International Journal of Computers and Applications · doi:10.1080/1206212x.2024.2416955
DropMismatch: removing mismatched UI elements for better pixel to code generation
Li, M., & Lin, T. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06384-7
Automatic Code Generation from GUI Screenshots with Vision-Language Models
Liang, J., Liang, J., & Li, S. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-2684-7_19
Web-Based Application for GUI Programming Learning Through Reverse Engineering Method
Tunardi, Y., Yulianto, B., & Layona, R. (2024) · International Conference on Information Technology Systems and Innovation · doi:10.1109/icitsi65188.2024.10929136
Guetari, R., Kaabi, R. S., Hamzaoui, H., & Kraïem, N. (2025) · Research Square · doi:10.21203/rs.3.rs-6353409/v1
REVOLUTIONIZING WEB DEVELOPMENT: HARNESSING GENERATIVE AI FOR IMAGE –TO-CODE TRANSFORMATION
Velmurugan., N. S. A. (2024) · International Journal of Engineering Research and Sustainable Technologies (IJERST) · doi:10.63458/ijerst.v2i3.90
Review and Mapping of Search-Based Approaches for Program Synthesis
Saber, T., & Tao, N. (2025) · Information · doi:10.3390/info16050401 · Cited by 2
Automating Generation of Web GUI from a Design Image
Singh, A., & Bansal, A. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-4273-1_17
From Mock-ups to Code: A Conceptual Synthesis of AI-Driven Automatic Website Generation
Kaluarachchi, T. (2025) · doi:10.38124/ijisrt/25nov339 · Cited by 1
AutoStructGUI: Bridging Design and Implementation of GUI through Structured Layout Generation
Ren, J., & Xu, P. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789058
Athena: Intermediate Representations for Iterative Scaffolded App Generation with an LLM
Beason, J.-T., Cheng, R., Schoop, E., & Nichols, J. W. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789133 · Cited by 6
No Code, No Cloud: On-Device Mockup-to-Code with Lightweight Vision-Language AI
Kuganathan, A., Purandare, M., & Stolze, M. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789144
UCS2GUI: GUIs generator from use case specification
Raharjana, I. K., Zaman, B., Ermawati, Siregar, R. T. A., Radiks, A. S. B. M., Chandra, L. E. P., Sari, L. F., Sihombing, M. L. D., Witania, A., & Husna, O. I. (2026) · AIP conference proceedings · doi:10.1063/5.0308834
Automatic learning of next generation human-computer interactions
Debard, Q. (2020) · theses.fr (ABES) · doi:10.70675/8aa26513zedabz4492zbfacz4c4613e74be6
Apprentissage profond appliqué à la détection d'événements anormaux dans les flux vidéos
Bouindour, S. (2019) · theses.fr (ABES) · doi:10.70675/798aa8cdz1462z4e55zacd5z4041132c1594
Incremental approach for application GUI migration using metamodels
Verhaeghe, B. (2021) · theses.fr (ABES) · doi:10.70675/0782cdfcza667z4ee7z9dcez7772a888bf96 · Cited by 3
Foundation Vision-Language models
Laurençon, H. (2025) · theses.fr (ABES) · doi:10.70675/ec0311a0zb6a4z4249zb9fbz01bf51a02c7c · Cited by 1
Hyper Web AI: A Natural Language Processing Based AI-Powered Web Code Generator
Kolli Neharika, V. H., & Ganapathi, G. S. A. (2026) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-32590
SPRITE:From Static Mockups to Engine-Ready GameUI
Bai, Y., Ruihao, L., Zhang, H., Lim, C. H., Yan, M., & Li, M. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798742
WebUI-95: A Large-Scale Dataset of Normalized Web Interfaces via UI-to-Code Generation
Calò, T., & Russis, L. D. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3799359 · Cited by 2
Park, H., & Kim, H. K. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2664690
DevAI: Multi-agent Co-pilot for Intelligent Assistance in Software Development Life Cycle
Firdausi, M., Parulekar, S., Bist, S., Dhumane, A. V., Tiwari, V., & Joshi, R. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20609-1_24
Automatic GUI Code Generation via Convolutional Neural Networks from Low-Fidelity Wireframes
Mathivanan, P., Inban, P., Kevin, A., & Pravin, G. (2026) · Algorithms for intelligent systems · doi:10.1007/978-3-032-12243-8_16
Rizk, C., Khatoonabadi, S., & Shihab, E. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793371
Xiao, J., Qin, J., Li, S., Lam, M. H., Wan, Y., Huang, J.-T., Huo, Y., & Lyu, M. R.-T. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817689 · Cited by 7
Learning to Design Interfaces: GUI Generation Using GAN
Yadav, A., Rastogi, R., & Kumar, R. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-19160-1_7
Babu, M. H., & Vadakkan, D. J. (2026) · International Conference on Smart Systems and Inventive Technology (ICSSIT) · doi:10.1109/icssit69151.2026.11656639
Bilgeturk, G., & Saran, M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7404671
Structgen: A Multi-Agent Generative Service for Automated Web UI Prototyping
Luo, J., Zhao, S., Su, Y., Jinjian, Y., Tang, H., Zhao, C., & Li, Y. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00094
Liu, M., & Li, Y. (2026) · International Conference on Digital Society and Artificial Intelligence (DSAI) · doi:10.1145/3833541.3833788
Wan, Y., Liang, T., Xu, J., Xiao, J., Huo, Y., & Lyu, M. R.-T. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834388 · Cited by 5
Generating PlantUML from UML class and sequence diagram images with a compact vision encoder–decoderPreprint
Baťalík, P., Kordos, T., Dubec, J., & Lang, J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7674418
Metadata
- Authors
- Beltramelli, T.
- Affiliations
- Uizard Technologies, Copenhagen, Denmark
- Corresponding author
- Beltramelli, Tony
- Venue
- Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems
- Publisher
- ACM
- Published
- 2018-06-19
- Pages
- 1-6
- OpenAlex
- W2962724148
- Cited by
- 237
- References
- 15
- Citations by year
- 2026: 22 · 2025: 32 · 2024: 25 · 2023: 25 · 2022: 30 · 2021: 38 · 2020: 30 · 2019: 23 · 2018: 12
- FWCI
- 20.21
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Mobile and Web Applications
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- deep learning, end-to-end learning, iOS, Android, web application, software development automation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.