Sketch2Code: Transformation of Sketches to UI in Real-time Using Deep Neural Network
Open accessAbstract
User Interface (UI) prototyping is a necessary step in the early stages of application development. Transforming sketches of a Graphical User Interface (UI) into a coded UI application is an uninspired but time-consuming task performed by a UI designer. An automated system that can replace human efforts for straightforward implementation of UI designs will greatly speed up this procedure. The works that propose such a system primarily focus on using UI wireframes as input rather than hand-drawn sketches. In this paper, we put forward a novel approach wherein we employ a Deep Neural Network that is trained on our custom database of such sketches to detect UI elements in the input sketch. Detection of objects in sketches is a peculiar visual recognition task that requires a specific solution that our deep neural network model attempts to provide. The output from the network is a platform-independent UI representation object. The UI representation object is a dictionary of key-value pairs to represent the UI elements recognized along with their properties. This is further consumed by our UI parser which creates code for different platforms. The intrinsic platform-independence allows the model to create a UI prototype for multiple platforms with single training. This two-step approach without the need for two trained models improves over other methods giving time-efficient results (average time: 129 ms) with good accuracy.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · see citing works
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · see citing works
A Computational Approach to Edge Detection
Canny, J. (1986) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1986.4767851 · Cited by 30,004 · see citing works
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105 · see citing works
YOLOv3: An Incremental Improvement
Redmon, J., & Farhadi, A. (2018) · arXiv · doi:10.48550/arxiv.1804.02767 · Cited by 17,141 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858826 · Cited by 9,556 · see citing works
Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks
Sainath, T. N., Vinyals, O., Senior, A., & Sak, H. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178838 · Cited by 1,678 · see citing works
Deep Neural Networks for Object Detection
Szegedy, C., Toshev, A., & Erhan, D. (2013) · Cited by 1,313 · see citing works
Scalable Object Detection Using Deep Neural Networks
Erhan, D., Szegedy, C., Toshev, A., & Anguelov, D. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.276 · Cited by 1,161 · cited 1× · see citing works
“[12] proposed a saliency-inspired [13] neural network that could predict class-agnostic bounding box and assign a score to each box which corresponds to the probability of them containing any useful object in them.”
BackgroundVisual Tracking with Fully Convolutional Networks
Wang, L., Ouyang, W., Wang, X., & Lu, H. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.357 · Cited by 1,001 · see citing works
Past, present, and future of user interface software tools
Myers, B. A., Hudson, S. E., & Pausch, R. F. (2000) · ACM Transactions on Computer-Human Interaction · doi:10.1145/344949.344959 · Cited by 809 · see citing works
DeepCoder: Learning to Write ProgramsPreprint
Balog, M., Gaunt, A. L., Brockschmidt, M., Nowozin, S., & Tarlow, D. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01989 · Cited by 641 · see citing works
Fette, I., & Melnikov, A. (2011) · doi:10.17487/rfc6455 · Cited by 572 · see citing works
Fast, Accurate Detection of 100,000 Object Classes on a Single Machine
Dean, T., Ruzon, M. A., Segal, M., Shlens, J., Vijayanarasimhan, S., & Yagnik, J. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.237 · Cited by 307 · see citing works
A review of object detection based on convolutional neural network
Wang, Z., & Liu, J. (2017) · Chinese Control Conference (CCC) · doi:10.23919/chicc.2017.8029130 · Cited by 254 · see citing works
pix2code: Generating Code from a Graphical User Interface Screenshot
Beltramelli, T. (2018) · Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems · doi:10.1145/3220134.3220135 · Cited by 237 · see citing works
Robust Visual Tracking via Hierarchical Convolutional Features
Ma, C., Huang, J., Yang, X., & Yang, M. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2865311 · Cited by 209 · see citing works
Reverse Engineering Mobile Application User Interfaces with REMAUI (T)
Nguyen, T. A., & Csallner, C. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.32 · Cited by 200 · cited 3× · see citing works
“introduced REMAUI [22] which implements the user interface for a mobile application in Android and the iOS environment from screen-shots.”
MethodOptical character recognition: an illustrated guide to the frontier
Nagy, G., Nartker, T. A., & Rice, S. V. (1999) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.373511 · Cited by 118 · see citing works
A review of the use of examples for automating architectural design tasks
Sönmez, N. O. (2017) · Computer-Aided Design · doi:10.1016/j.cad.2017.10.005 · Cited by 69 · see citing works
Rian, I. M., Sassone, M., & Asayama, S. (2018) · Computer-Aided Design · doi:10.1016/j.cad.2018.01.004 · Cited by 38 · see citing works
Algebraic 3D graphic statics: Reciprocal constructions
Hablicsek, M., Akbarzadeh, M., & Guo, Y. (2018) · Computer-Aided Design · doi:10.1016/j.cad.2018.08.003 · Cited by 29 · see citing works
On the architecture of a human-centered CAD agent system
Modi, S., Tiwari, M. K., Lin, Y., & Zhang, W. (2010) · Computer-Aided Design · doi:10.1016/j.cad.2010.10.006 · Cited by 18 · see citing works
Deep saliency: What is learnt by a deep network about saliency?
He, S., & Pugeault, N. (2018) · Open Research Exeter (University of Exeter) · doi:10.48550/arxiv.1801.04261 · Cited by 6 · see citing works
Sketch classification with deep learning models
Eyiokur, F. İ., Yaman, D., & Ekenel, H. K. (2018) · Signal Processing and Communications Applications Conference (SIU) · doi:10.1109/siu.2018.8404417 · Cited by 3 · see citing works
Flagged first, then by citations
1 of the 30 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- [7][7] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, 'Overfeat: Integrated recognition, localization and detection using convolutional networks,' 2013. [Online]. Available: http://arxiv.org/abs/1312.6229 - · arXiv
- [14][14] R. Girshick, 'Fast r-cnn,' in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1440-1448. -
- [25][25] N. Sengar and D. Kapoor, 'Edge detection by combination of morphological operators with different edge detection operators,' International Journal of Information & Computation Technology , vol. 4, no. 11, pp. 1051-1056, 2014. -
Cited by
Sorted by citations
Code Generation Using Machine Learning: A Systematic Review
Dehaerne, E., Dey, B., Halder, S., Gendt, S. D., & Meert, W. (2022) · IEEE Access · doi:10.1109/access.2022.3196347 · Cited by 95 · see citing works
Improving Collaboration Efficiency Between UX/UI Designers and Developers in a Low-Code Platform
Pacheco, J., Garbatov, S., & Goulão, M. (2021) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C) · doi:10.1109/models-c53483.2021.00025 · Cited by 32 · see citing works
Closing the gap between designers and developers in a low code ecosystem
Bexiga, M. G., Garbatov, S., & Seco, J. C. (2020) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C) · doi:10.1145/3417990.3420195 · Cited by 29 · see citing works
Recent Progress in Automated Code Generation from GUI Images Using Machine Learning Techniques
Baulé, D. D. S., von Wangenheim, C. G., von Wangenheim, A., & Hauck, J. C. R. (2020) · JUCS - Journal of Universal Computer Science · doi:10.3897/jucs.2020.058 · Cited by 23 · see citing works
Deep Learning for Free-Hand Sketch: A Survey and A ToolboxPreprint
Xu, P., Hospedales, T. M., Yin, Q., Song, Y.-Z., Xiang, T., & Wang, L. (2020) · arXiv (Cornell University) · Cited by 21 · see citing works
Deep learning‐based prototyping of android GUI from hand‐drawn mockups
Abdelhamid, A. A., Alotaibi, S. R., & Mousa, A. (2020) · IET Software · doi:10.1049/iet-sen.2019.0378 · Cited by 18 · see citing works
Exploring the Role of AI in Web Design and Development: A Voyage through Automated Code Generation
Muthazhagu, V. H., & Surendiran, B. (2024) · International Conference on Intelligent and Innovative Technologies in Computing, Electrical and Electronics (IITCEE) · doi:10.1109/iitcee59897.2024.10467409 · Cited by 10 · see citing works
Xiao, J., Wan, Y., Huo, Y., Wang, Z., Xu, X., Wang, W., Xu, Z., Wang, Y., & Lyu, M. R. (2024) · arXiv · doi:10.48550/arxiv.2411.03292 · Cited by 7 · see citing works
Yashaswini, D., Sneha, S., & Kumar, N. (2022) · IEEE International Conference on Cybernetics, Cognition and Machine Learning Applications (ICCCMLA) · doi:10.1109/icccmla56841.2022.9989298 · Cited by 7 · see citing works
Automatic Code Generation From Low Fidelity Graphical User Interface Sketches Using Deep Learning
Adefris, B. B., Habtie, A. B., & Taye, Y. G. (2022) · International Conference on Information and Communication Technology for Development for Africa (ICT4DA) · doi:10.1109/ict4da56482.2022.9971204 · Cited by 7 · see citing works
Feng, Z., Fang, J., Cai, B., & Zhang, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86365-4_5 · Cited by 7 · see citing works
A Deep Learning Based Object Detection System for User Interface Code Generation
Aşıroğlu, B., Senan, S., Görgel, P., Isenkul, M. E., Ensarı, T., Sezen, A., & Dağtekin, M. (2022) · International Congress on Human-Computer Interaction, Optimization and Robotic Applications (ICHORA) · doi:10.1109/hora55278.2022.9799941 · Cited by 4 · see citing works
Transforming Ideas into Code: A Novel Approach to Design-to-Development Workflow
Raptis, G. E., & Katsini, C. (2023) · International Conference on Computer and Applications (ICCA) · doi:10.1109/icca59364.2023.10401412 · Cited by 4 · see citing works
pic2code: Generating HTML Code from Handwritten Picture
Arefin, M., Hossen, K. M., & Khan, R. (2023) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit60459.2023.10441098 · Cited by 3 · see citing works
Automated Front-End Code Generation Using OpenAI: Empowering Web Development EfficiencyPreprint
Ikumapayi, N. A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4590704 · Cited by 3 · see citing works
UIBee: An improved deep instance segmentation and classification of UI elements in wireframes
KAZANGİRLER, C. B., Özcan, C., & Tekin, B. Y. (2023) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.55730/1300-0632.3999 · Cited by 2 · see citing works
Text2App: A Framework for Creating Android Apps from Text DescriptionsPreprint
Hasan, M., Mehrab, K. S., Ahmad, W. U., & Shahriyar, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08301 · Cited by 4 · see citing works
IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web
Guo, H., Zhang, W., Chen, J., Gu, Y., Yang, J., Du, J., Cao, S., Hui, B., Liu, T., Ma, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.18980
“Sketch2Code (Jain et al., 2019) divides the problem into three parts: object recognition, bounding box creation, and the creation of a functional prototype application.”
A Deep Learning Approach for Generating Mark-up Code from Sketch Images
Lohana, B., Tanna, M., Pamnani, G., Sahijwani, T., & Temkar, R. (2022) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2022.40675
Implementing Machine Learning-Based Simulation in Physics Virtual Laboratory
Cheddi, F. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-26852-6_27
Görüntülerden veya Çizimlerden Otomatik Kod Oluşturma Teknikleri: Bir Derleme Çalışması
Kunduracı, M. S., & Bilgin, T. T. (2023) · Türkiye Bilişim Vakfı Bilgisayar Bilimleri ve Mühendisliği Dergisi · doi:10.54525/tbbmd.1190177
مسح لتقنيات إنشاء الكود تلقائيا من تصميمات واجهة المستخدم ذات الدقة المختلفة
جاد, م. س. ز., مرعي, م., & يعقوب, أ. ا. (2024) · النشرة المعلوماتية في الحاسبات والمعلومات · doi:10.21608/fcihib.2024.262296.1106
- Authors
- Jain, V., Agrawal, P., Banga, S., Kapoor, R., & Gulyani, S.
- Affiliations
- Bharati Vidyapeeth's College of Engineering, Delhi (IN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1910.08930
- OpenAlex
- W2981274875
- Cited by
- 20
- References
- 30
- Citations by year
- 2024: 2 · 2023: 6 · 2022: 5 · 2021: 3 · 2020: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- deep neural network, code generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
