The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
From UI design image to GUI skeleton: a neural machine translator to bootstrap mobile GUI implementation
Abstract
A GUI skeleton is the starting point for implementing a UI design image. To obtain a GUI skeleton from a UI design image, developers have to visually understand UI elements and their spatial layout in the image, and then translate this understanding into proper GUI components and their compositions. Automating this visual understanding and translation would be beneficial for bootstraping mobile GUI implementation, but it is a challenging task due to the diversity of UI designs and the complexity of GUI skeletons to generate. Existing tools are rigid as they depend on heuristically-designed visual understanding and GUI generation rules. In this paper, we present a neural machine translator that combines recent advances in computer vision and machine translation for translating a UI design image into a GUI skeleton. Our translator learns to extract visual features in UI images, encode these features' spatial layouts, and generate GUI skeletons in a unified neural network framework, without requiring manual rule development. For training our translator, we develop an automated GUI exploration method to automatically collect large-scale UI data from real-world applications. We carry out extensive experiments to evaluate the accuracy, generality and usefulness of our approach.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Learning-based GUI code generation methods struggle with generalizing to diverse web elements.
“However, these methods struggle with generalizing to diverse web elements.”
From Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping (2024) · cites this paper as [9]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472
Distributed hierarchical processing in the primate cerebral cortex
Felleman, D. J., & Van Essen, D. C. (1991) · Cerebral Cortex · doi:10.1093/cercor/1.1.1 · Cited by 7,783
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
pix2code: Generating Code from a Graphical User Interface Screenshot
Beltramelli, T. (2018) · Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems · doi:10.1145/3220134.3220135 · Cited by 237
Reverse Engineering Mobile Application User Interfaces with REMAUI (T)
Nguyen, T. A., & Csallner, C. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.32 · Cited by 200 · cited 2×
“At inference time, our translator can generate GUI skeletons for 20 UI images per second, which is about 180 times faster than existing UI reverse engineering techniques [45] based on traditional computer vision techniques (e.”
Method
Flagged first, then by citations
Research works printed but not linked: 42
- 2017. Sketch:Professional digital design for Mac. (2017). https://www.sketchapp.com/
- Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional attention network for extreme summarization of source code. In International Conference on Machine Learning. 2091--2100.
- Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2016. Deepcoder: Learning to write programs. arXiv preprint arXiv.1611.01989 (2016). · arXiv
- 10.1145/2259051.2259056 · DOI
- Léon Bottou. 2012. Stochastic gradient descent tricks. In Neural networks: Tricks of the trade. Springer, 421--436.
- 10.1109/TPAMI.1986.4767851 · DOI · Paper page
- 10.1145/2393596.2393606 · DOI
- Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014). · arXiv
- 10.1109/ASE.2015.89 · DOI
- Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. 2011. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop.
- Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M Rush. 2017. Image-to-Markup Generation with Coarse-to-Fine Attention. In International Conference on Machine Learning. 980--989.
- 10.1145/1753326.1753554 · DOI
- Michael P Fay and Michael A Proschan. 2010. Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Statistics surveys 4 (2010), 1.
- 10.1145/2950290.2950334 · DOI
- 10.5555/3172077.3172403 · DOI
- Ruozi Huang Yonghao Long and Xiangping Chen. 2016. Automaticly Generating Web Page From A Mockup.. In SEKE. 589--594. · DOI
- 10.1113/jphysiol.1962.sp006837 · DOI · Paper page
- 10.1113/jphysiol.1968.sp008455 · DOI
- 10.1145/2568225.2568292 · DOI
- Philipp Koehn. 2004. Pharaoh: a beam search decoder for phrase-based statistical machine translation models. Machine translation: From real users to research (2004), 115--124.
- 10.1007/978-3-319-10602-1_48 · DOI · Paper page
- 10.5555/850924.851523 · DOI
- 10.1145/1985793.1985809 · DOI
- Guozhu Meng, Yinxing Xue, Jing Kai Siow, Ting Su, Annamalai Narayanan, and Yang Liu. 2017. AndroVault: Constructing Knowledge Graph from Millions of Android Apps for Automated Analysis. CoRR abs/1711.07451 (2017). arXiv:1711.07451 http://arxiv.org/abs/1711.07451 · arXiv
- Lili Mou, Rui Men, Ge Li, Lu Zhang, and Zhi Jin. 2015. On end-to-end program generation from user intention by deep neural networks. arXiv preprint arXiv:1510.07211 (2015). · arXiv
- 10.1109/CVPR.2014.317 · DOI
- 10.5555/3104322.3104425 · DOI
- 10.5555/2544000 · DOI
- 10.1145/2642937.2642976 · DOI
- 10.5555/938980.939477 · DOI
- 10.5555/3155562.3155583 · DOI
- 10.5555/1304596.1304846 · DOI
- 10.1145/2889160.2891043 · DOI
- 10.1145/3106237.3106298 · DOI
- 10.1145/3020266 · DOI
- Ilya Sutskever Oriol Vinyals and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. 3104--3112. · DOI
- Seyyed Ehsan Salamati Taba, Iman Keivanloo, Ying Zou, Joanna Ng, and Tinny Ng. 2014. An exploratory study on the relation between user interface complexity and the perceived quality. In International Conference on Web Engineering. Springer, 370--379.
- 10.1109/CVPR.2014.220 · DOI · Paper page
- 10.5555/2820518.2820559 · DOI
- Yonghui Wu Mike Schuster Zhifeng Chen Quoc V Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey et al. 2016. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint arXiv:1609.08144 (2016). · arXiv
- 10.1007/978-3-319-10590-1_53 · DOI · Paper page
- 10.1145/267580.267590 · DOI
Cited by
Sorted by citations
A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research
Watson, C., Cooper, N., Palacio, D. N., Moran, K., & Poshyvanyk, D. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3485275 · Cited by 137
A 2030 Roadmap for Software Engineering
Pezzè, M., Abrahão, S., Penzenstadler, B., Poshyvanyk, D., Roychoudhury, A., & Yue, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731559 · Cited by 13
Xiao, J., Wan, Y., Huo, Y., Wang, Z., Xu, X., Wang, W., Xu, Z., Wang, Y., & Lyu, M. R. (2024) · arXiv · doi:10.48550/arxiv.2411.03292 · Cited by 7
“The design and development of Graphical User Interfaces (GUIs) play a critical role in shaping a website’s aesthetics and functionality, improving user satisfaction through effective layout, colors, and typography [2], [3].”
Metadata
- Authors
- Chen, C., Su, T., Meng, G., Xing, Z., & Liu, Y.
- Affiliations
- Nanyang Technological University (SG)
- Australian National University (AU)
- Venue
- Proceedings of the 40th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2018-05-27
- Pages
- 665-676
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W2794908093
- Cited by
- 188
- References
- 11
- Citations by year
- 2026: 11 · 2025: 18 · 2024: 23 · 2023: 22 · 2022: 37 · 2021: 27 · 2020: 28 · 2019: 21 · 2018: 1
- FWCI
- 15.51
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Funders
- Nvidia
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Mobile and Web Applications
- Keywords
- neural machine translation, visual feature extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
