The DOI matches its Crossref record; 14 of the 15 references it cites were found in a registry or an index.
pix2code: Generating Code from a Graphical User Interface Screenshot
Abstract
Transforming a graphical user interface screenshot created by a designer into computer code is a typical task conducted by a developer in order to build customized software, websites, and mobile applications. In this paper, we show that deep learning methods can be leveraged to train a model end-to-end to automatically reverse engineer user interfaces and generate code from a single input image with over 77% of accuracy for three different platforms (i.e. iOS, Android and web-based technologies).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021
Recurrent Neural Network Regularization
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 2,123
Towards Diverse and Natural Image Descriptions via a Conditional GAN
Dai, B., Fidler, S., Urtasun, R., & Lin, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.323 · Cited by 477
Research works printed but not linked: 5
- Matej Balog Alexander L Gaunt Marc Brockschmidt Sebastian Nowozin and Daniel Tarlow. 2016. DeepCoder: Learning to Write Programs. arXiv preprint arXiv:1611.01989 (2016). Matej Balog Alexander L Gaunt Marc Brockschmidt Sebastian Nowozin and Daniel Tarlow. 2016. DeepCoder: Learning to Write Programs. arXiv preprint arXiv:1611.01989 (2016). · arXiv
- Gaunt Alexander L, 2016, Terpret: A probabilistic programming language for program induction. arXiv preprint arXiv:1608.04428 · arXiv
- Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680. Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.
- Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 (2013). Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 (2013). · arXiv
- 10.5555/2627435.2670313 · doi:10.5555/2627435.2670313
Cited by
Chen, C., Su, T., Meng, G., Xing, Z., & Liu, Y. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180240 · Cited by 188
Stige, Å., Zamani, E. D., Mikalef, P., & Zhu, Y. (2024) · Information Technology & People · doi:10.1108/itp-07-2022-0519 · Cited by 167
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154
“In specific domains such as user interface design and prototyping, recent advances in deep learning models also enable developers to sketch the user interface visually and then automatically generates user interface code with the sketch [13], or using existing screenshots [84].”
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Zhang, X., de Greef, L., Swearngin, A., White, S., Murray, K. I., Yu, L., Shan, Q., Nichols, J. W., Wu, J., Fleizach, C., et al. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445186 · Cited by 178
Object detection for graphical user interface: old fashioned or deep learning or a combination?
Chen, J., Xie, M., Xing, Z., Chen, C., Xu, X., Zhu, L., & Li, G. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409691 · Cited by 163
Metadata
- Authors
- Beltramelli, T.
- Affiliations
- Uizard Technologies, Copenhagen, Denmark
- Corresponding author
- Beltramelli, Tony
- Venue
- Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems
- Publisher
- ACM
- Published
- 2018-06-19
- Pages
- 1-6
- OpenAlex
- W2962724148
- Cited by
- 237
- References
- 15
- Citations by year
- 2026: 22 · 2025: 32 · 2024: 25 · 2023: 25 · 2022: 30 · 2021: 38 · 2020: 30 · 2019: 23 · 2018: 12
- FWCI
- 20.21
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Mobile and Web Applications
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- deep learning, end-to-end learning, iOS, Android, web application, software development automation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: pix2code: Generating Code from a Graphical User Interface Screenshot. https://app.cochat.ai/papers/10.1145/3220134.3220135
- The paper
- Beltramelli, T. (2018). pix2code: Generating Code from a Graphical User Interface Screenshot. Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems. https://doi.org/10.1145/3220134.3220135
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.