CoChat
Sign Up
PreprintCited by 18Software Engineering ResearchHandwritten Text Recognition TechniquesWeb Data Mining and Analysis

Sketch2code: Generating a website from a paper mockup

Open access
Robinson, Alex·2019·arXiv
DOI 10.48550/arxiv.1905.13750arXiv 1905.13750
Open this paper in CoChat — free accountOpen PDF

Abstract

An early stage of developing user-facing applications is creating a wireframe to layout the interface. Once a wireframe has been created it is given to a developer to implement in code. Developing boiler plate user interface code is time consuming work but still requires an experienced developer. In this dissertation we present two approaches which automates this process, one using classical computer vision techniques, and another using a novel application of deep semantic segmentation networks. We release a dataset of websites which can be used to train and evaluate these approaches. Further, we have designed a novel evaluation framework which allows empirical evaluation by creating synthetic sketches. Our evaluation illustrates that our deep learning approach outperforms our classical computer vision approach and we conclude that deep learning is the most promising direction for future research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1905.13750

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 77

  1. [1][1] Pedro Campos and Nuno Nunes. 'Practitioner Tools and Workstyles for User-Interface Design'. In: 24 (Feb. 2007), pp. 73-80. -
  2. [2][2] James A. Landay and Brad A. Myers. 'Interactive Sketching for the Early Stages of User Interface Design'. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . CHI '95. Denver, Colorado, USA: ACM Press/Addison-Wesley Publishing Co., 1995, pp. 43-50. isbn : 0-201-84705-1. doi : 10.1145/223904.223910 . url : http: //dx.doi.org/10.1145/223904.223910 . - · DOI
  3. [3][3] T. Silva da Silva et al. 'User-Centered Design and Agile Methods: A Systematic Review'. In: 2011 Agile Conference . Aug. 2011, pp. 77-86. doi : 10.1109/AGILE.2011.24 . - · DOI
  4. [4][4] J. A. Landay and B. A. Myers. 'Sketching interfaces: toward more human interface design'. In: Computer 34.3 (Mar. 2001), pp. 56-64. issn : 0018-9162. doi : 10.1109/2.910894 . - · DOI
  5. [5][5] James Lin et al. 'DENIM: Finding a Tighter Fit Between Tools and Practice for Web Site Design'. In: (Apr. 2000), pp. 510-517. -
  6. [7][7] Chao Dong et al. 'Image Super-Resolution Using Deep Convolutional Networks'. In: CoRR abs/1501.00092 (2015). arXiv: 1501.00092 . url : http://arxiv.org/abs/1501.00092 . - · arXiv
  7. [8][8] Balakrishnan Varadarajan et al. 'Efficient Large Scale Video Classification'. In: CoRR abs/1505.06250 (2015). arXiv: 1505.06250 . url : http://arxiv.org/abs/1505.06250 . - · arXiv
  8. [9][9] Oriol Vinyals et al. 'Show and Tell: A Neural Image Caption Generator'. In: CoRR abs/1411.4555 (2014). arXiv: 1411.4555 . url : http://arxiv.org/abs/1411.4555 . - · arXiv
  9. [10][10] Andrej Karpathy and Fei-Fei Li. 'Deep Visual-Semantic Alignments for Generating Image Descriptions'. In: CoRR abs/1412.2306 (2014). arXiv: 1412.2306 . url : http://arxiv. org/abs/1412.2306 . - · arXiv
  10. [11][11] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 'A Neural Algorithm of Artistic Style'. In: CoRR abs/1508.06576 (2015). arXiv: 1508.06576 . url : http://arxiv.org/ abs/1508.06576 . - · arXiv
  11. [14][14] Brad Myers, Scott E. Hudson, and Randy Pausch. 'Past, Present, and Future of User Interface Software Tools'. In: ACM Trans. Comput.-Hum. Interact. 7.1 (Mar. 2000), pp. 3-28. issn : 1073-0516. doi : 10.1145/344949.344959 . url : http://doi.acm.org/10.1145/ 344949.344959 . - · DOI
  12. [19][19] Yin Yin Wong. 'Rough and Ready Prototypes: Lessons from Graphic Design'. In: Posters and Short Talks of the 1992 SIGCHI Conference on Human Factors in Computing Systems . CHI '92. Monterey, California: ACM, 1992, pp. 83-84. doi : 10.1145/1125021.1125094 . url : http://doi.acm.org/10.1145/1125021.1125094 . - · DOI
  13. [20][20] Richard Davis et al. 'SketchWizard: Wizard of Oz Prototyping of Pen-based User Interfaces'. In: (Jan. 2007), pp. 119-128. -
  14. [21][21] Julian Seifert et al. 'MobiDev: A Tool for Creating Apps on Mobile Phones'. In: (Aug. 2011), pp. 109-112. -
  15. [22][22] Tony Beltramelli. 'pix2code: Generating Code from a Graphical User Interface Screenshot'. In: CoRR abs/1705.07962 (2017). arXiv: 1705.07962 . url : http://arxiv.org/abs/1705. 07962 . - · arXiv
  16. [23][23] Kevin Moran et al. 'Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps'. In: CoRR abs/1802.02312 (2018). - · arXiv
  17. [24][24] Ajay Kumar Boyat and Brijendra Kumar Joshi. 'A Review Paper: Noise Models in Digital Image Processing'. In: CoRR abs/1505.03489 (2015). arXiv: 1505.03489 . url : http:// arxiv.org/abs/1505.03489 . - · arXiv
  18. [25][25] S. Singh and B. Singh. 'Effects of noise on various edge detection techniques'. In: 2015 2nd International Conference on Computing for Sustainable Global Development (INDIACom) . Mar. 2015, pp. 827-830. -
  19. [26][26] Junyuan Xie, Linli Xu, and Enhong Chen. 'Image denoising and inpainting with deep neural networks'. In: Advances in neural information processing systems . 2012, pp. 341-349. -
  20. [27][27] HS Shukla, Narendra Kumar, and RP Tripathi. 'Gaussian Noise Filtering Techniques using New Median Filter'. In: International Journal of Computer Applications 95.12 (2014). -
  21. [28][28] E. Arias-Castro and D. L. Donoho. 'Does median filtering truly preserve edges better than linear filtering?' In: ArXiv Mathematics e-prints (Dec. 2006). eprint: math/0612422 . - · arXiv
  22. [29][29] Hani Hunud A Kadouf and Yasir Mohd Mustafah. 'Colour-based object detection and tracking for autonomous quadrotor UAV'. In: IOP Conference Series: Materials Science and Engineering . Vol. 53. 1. IOP Publishing. 2013, p. 012086. -
  23. [30][30] Dibya Jyoti Bora, Anil Kumar Gupta, and Fayaz Ahmad Khan. 'Comparing the Performance of L*A*B* and HSV Color Spaces with Respect to Color Image Segmentation'. In: CoRR abs/1506.01472 (2015). arXiv: 1506.01472 . url : http://arxiv.org/abs/1506.01472 . - · arXiv
  24. [31][31] Dibya Jyoti Bora, Anil Kumar Gupta, and Fayaz Ahmad Khan. 'Comparing the Performance of L*A*B* and HSV Color Spaces with Respect to Color Image Segmentation'. In: CoRR abs/1506.01472 (2015). arXiv: 1506.01472 . url : http://arxiv.org/abs/1506.01472 . - · arXiv
  25. [32][32] SB Park, JW Lee, and SK Kim. 'Pattern Recognit'. In: Lett 25 (2004), pp. 287-300. -
  26. [34][34] John Canny. 'A computational approach to edge detection'. In: Readings in Computer Vision . Elsevier, 1987, pp. 184-203. -
  27. [35][35] Yun Liu et al. 'Richer Convolutional Features for Edge Detection'. In: CoRR abs/1612.02103 (2016). arXiv: 1612.02103 . url : http://arxiv.org/abs/1612.02103 . - · arXiv
  28. [36][36] Jin Zhou and Baoxin Li. 'Automatic generation of pencil-sketch like drawings from personal photos'. In: Multimedia and Expo, 2005. ICME 2005. IEEE International Conference on . IEEE. 2005, pp. 1026-1029. -
  29. [37][37] Sunil Kumar Katiyar and P. V. Arun. 'Comparative analysis of common edge detection techniques in context of object extraction'. In: CoRR abs/1405.6132 (2014). arXiv: 1405. 6132 . url : http://arxiv.org/abs/1405.6132 . - · arXiv
  30. [39][39] Satoshi Suzuki and KeiichiA be. 'Topological structural analysis of digitized binary images by border following'. In: Computer Vision, Graphics, and Image Processing 30.1 (1985), pp. 32-46. issn : 0734-189X. doi : https://doi.org/10.1016/0734-189X(85)90016-7 . url : http://www.sciencedirect.com/science/article/pii/0734189X85900167 . - · DOI
  31. [40][40] David H Douglas and Thomas K Peucker. 'Algorithms for the reduction of the number of points required to represent a digitized line or its caricature'. In: Cartographica: The International Journal for Geographic Information and Geovisualization 10.2 (1973), pp. 112122. -
  32. [41][41] Boris Epshtein, Eyal Ofek, and Yonathan Wexler. 'Stroke Width Transform'. In: IEEE, Jan. 2010. url : https://www.microsoft.com/en-us/research/publication/strokewidth-transform/ . -
  33. [42][42] Yingying Zhu, Cong Yao, and Xiang Bai. 'Scene text detection and recognition: Recent advances and future trends'. In: Frontiers of Computer Science 10.1 (2016), pp. 19-36. -
  34. [43][43] Yingying Jiang et al. 'R2CNN: rotational region CNN for orientation robust scene text detection'. In: arXiv preprint arXiv:1706.09579 (2017). - · arXiv
  35. [44][44] Tao Wang et al. 'End-to-end text recognition with convolutional neural networks'. In: Pattern Recognition (ICPR), 2012 21st International Conference on . IEEE. 2012, pp. 33043308. -
  36. [46][46] Minghui Liao et al. 'TextBoxes: A Fast Text Detector with a Single Deep Neural Network'. In: CoRR abs/1611.06779 (2016). arXiv: 1611.06779 . url : http://arxiv.org/abs/1611. 06779 . - · arXiv
  37. [47][47] Wenhao He et al. 'Deep Direct Regression for Multi-Oriented Scene Text Detection'. In: CoRR abs/1703.08289 (2017). arXiv: 1703.08289 . url : http://arxiv.org/abs/1703. 08289 . - · arXiv
  38. [48][48] H. Cho, M. Sung, and B. Jun. 'Canny Text Detector: Fast and Robust Scene Text Localization Algorithm'. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . June 2016, pp. 3566-3573. doi : 10.1109/CVPR.2016.388 . - · DOI
  39. [49][49] Nasser M Nasrabadi. 'Pattern recognition and machine learning'. In: Journal of electronic imaging 16.4 (2007), p. 049901. -
  40. [50][50] Yann LeCun et al. 'A theoretical framework for back-propagation'. In: Proceedings of the 1988 connectionist models summer school . CMU, Pittsburgh, Pa: Morgan Kaufmann. 1988, pp. 21-28. -
  41. [51][51] Marti A. Hearst et al. 'Support vector machines'. In: IEEE Intelligent Systems and their applications 13.4 (1998), pp. 18-28. -
  42. [52][52] Leo Breiman. 'Random forests'. In: Machine learning 45.1 (2001), pp. 5-32. -
  43. [54][54] M. Everingham et al. 'The Pascal Visual Object Classes (VOC) Challenge'. In: International Journal of Computer Vision 88.2 (June 2010), pp. 303-338. -
  44. [55][55] Alberto Garcia-Garcia et al. 'A review on deep learning techniques applied to semantic segmentation'. In: arXiv preprint arXiv:1704.06857 (2017). - · arXiv
  45. [56][56] Liang-Chieh Chen et al. 'Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation'. In: arXiv:1802.02611 (2018). - · arXiv
  46. [57][57] Vincent Dumoulin and Francesco Visin. 'A guide to convolution arithmetic for deep learning'. In: ArXiv e-prints (Mar. 2016). eprint: 1603.07285 . -
  47. [58][58] Fisher Yu and Vladlen Koltun. 'Multi-Scale Context Aggregation by Dilated Convolutions'. In: CoRR abs/1511.07122 (2015). arXiv: 1511.07122 . url : http://arxiv.org/abs/1511. 07122 . - · arXiv
  48. [59][59] Shuchang Zhou et al. 'Exploiting Local Structures with the Kronecker Layer in Convolutional Networks'. In: CoRR abs/1512.09194 (2015). arXiv: 1512 . 09194 . url : http : //arxiv.org/abs/1512.09194 . - · arXiv
  49. [61][61] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. 'SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation'. In: CoRR abs/1511.00561 (2015). arXiv: 1511.00561 . url : http://arxiv.org/abs/1511.00561 . - · arXiv
  50. [62][62] Hengshuang Zhao et al. 'Pyramid Scene Parsing Network'. In: CoRR abs/1612.01105 (2016). arXiv: 1612.01105 . url : http://arxiv.org/abs/1612.01105 . - · arXiv
  51. [64][64] Liang-Chieh Chen et al. 'Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs'. In: CoRR abs/1412.7062 (2014). arXiv: 1412.7062 . url : http: //arxiv.org/abs/1412.7062 . - · arXiv
  52. [65][65] Liang-Chieh Chen et al. 'DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs'. In: CoRR abs/1606.00915 (2016). arXiv: 1606.00915 . url : http://arxiv.org/abs/1606.00915 . - · arXiv
  53. [66][66] Liang-Chieh Chen et al. 'Rethinking Atrous Convolution for Semantic Image Segmentation'. In: CoRR abs/1706.05587 (2017). arXiv: 1706.05587 . url : http://arxiv.org/abs/1706. 05587 . - · arXiv
  54. [70][70] Waleed Abu-Ain et al. 'Skeletonization algorithm for binary images'. In: Procedia Technology 11 (2013), pp. 704-709. -
  55. [72][72] Alon Halevy, Peter Norvig, and Fernando Pereira. 'The unreasonable effectiveness of data'. In: IEEE Intelligent Systems 24.2 (2009), pp. 8-12. -
  56. [74][74] Gabriel L. Muller. 'HTML5 WebSocket protocol and its application to distributed computing'. In: CoRR abs/1409.3367 (2014). arXiv: 1409.3367 . url : http://arxiv.org/abs/ 1409.3367 . - · arXiv
  57. [76][76] Chris Harris and Mike Stephens. 'A combined corner and edge detector'. In: In Proc. of Fourth Alvey Vision Conference . 1988, pp. 147-151. -
  58. [78][78] Yann LeCun et al. 'Efficient backprop'. In: Neural networks: Tricks of the trade . Springer, 1998, pp. 9-50. -
  59. [79][79] Vinod Nair and Geoffrey E Hinton. 'Rectified linear units improve restricted boltzmann machines'. In: Proceedings of the 27th international conference on machine learning (ICML10) . 2010, pp. 807-814. -
  60. [80][80] Diederik P. Kingma and Jimmy Ba. 'Adam: A Method for Stochastic Optimization'. In: CoRR abs/1412.6980 (2014). arXiv: 1412.6980 . url : http://arxiv.org/abs/1412.6980 . - · arXiv
  61. [81][81] Sebastian Ruder. 'An overview of gradient descent optimization algorithms'. In: CoRR abs/1609.04747 (2016). arXiv: 1609.04747 . url : http://arxiv.org/abs/1609.04747 . - · arXiv
  62. [82][82] Adam P. Piotrowski and Jarosglyph[suppress] law J. Napiorkowski. 'A comparison of methods to avoid overfitting in neural networks training in the case of catchment runoff modelling'. In: Journal of Hydrology 476 (2013), pp. 97-111. issn : 0022-1694. doi : https://doi.org/10.1016/j. jhydrol.2012.10.019 . url : http://www.sciencedirect.com/science/article/pii/ S0022169412008931 . - · DOI
  63. [84][84] Shaoqing Ren et al. 'Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks'. In: CoRR abs/1506.01497 (2015). arXiv: 1506.01497 . url : http: //arxiv.org/abs/1506.01497 . - · arXiv
  64. [85][85] Sepp Hochreiter and J¨ urgen Schmidhuber. 'Long short-term memory'. In: Neural computation 9.8 (1997), pp. 1735-1780. -
  65. [87][87] Fran¸ cois Chollet. 'Xception: Deep Learning with Depthwise Separable Convolutions'. In: CoRR abs/1610.02357 (2016). arXiv: 1610.02357 . url : http://arxiv.org/abs/1610. 02357 . - · arXiv
  66. [88][88] Christian Szegedy et al. 'Rethinking the inception architecture for computer vision'. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 2016, pp. 2818-2826. -
  67. [90][90] M. Everingham et al. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results . http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html. -
  68. [91][91] Liang-Chieh Chen et al. 'DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs'. In: CoRR abs/1606.00915 (2016). arXiv: 1606.00915 . url : http://arxiv.org/abs/1606.00915 . - · arXiv
  69. [93][93] H. B. Mann and D. R. Whitney. 'On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other'. In: Ann. Math. Statist. 18.1 (Mar. 1947), pp. 50-60. doi : 10.1214/aoms/1177730491 . url : https://doi.org/10.1214/aoms/1177730491 . - · DOI · Paper page
  70. [95][95] Norman Cliff. 'Dominance statistics: Ordinal analyses to answer ordinal questions.' In: Psychological Bulletin 114.3 (1993), p. 494. -
  71. [97][97] Jacob Benesty et al. 'Mean-Squared Error Criterion'. In: Noise Reduction in Speech Processing . Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, pp. 1-6. isbn : 978-3-642-00296-0. doi : 10.1007/978-3-642-00296-0\_4 . url : https://doi.org/10.1007/978-3-64200296-0\_4 . - · DOI
  72. [98][98] Bernd Girod. 'Digital Images and Human Vision'. In: ed. by Andrew B. Watson. Cambridge, MA, USA: MIT Press, 1993. Chap. What's Wrong with Mean-squared Error?, pp. 207-220. isbn : 0-262-23171-9. url : http://dl.acm.org/citation.cfm?id=197765.197784 . -
  73. [99][99] Robert A. Wagner and Michael J. Fischer. 'The String-to-String Correction Problem'. In: J. ACM 21.1 (Jan. 1974), pp. 168-173. issn : 0004-5411. doi : 10.1145/321796.321811 . url : http://doi.acm.org/10.1145/321796.321811 . - · DOI
  74. [104][104] Matthew D. Zeiler and Rob Fergus. 'Visualizing and Understanding Convolutional Networks'. In: CoRR abs/1311.2901 (2013). arXiv: 1311.2901 . url : http://arxiv.org/abs/ 1311.2901 . - · arXiv
  75. [105][105] Karen Simonyan and Andrew Zisserman. 'Very Deep Convolutional Networks for LargeScale Image Recognition'. In: CoRR abs/1409.1556 (2014). arXiv: 1409.1556 . url : http: //arxiv.org/abs/1409.1556 . - · arXiv
  76. [106][106] Ross B. Girshick et al. 'Rich feature hierarchies for accurate object detection and semantic segmentation'. In: CoRR abs/1311.2524 (2013). arXiv: 1311.2524 . url : http://arxiv. org/abs/1311.2524 . - · arXiv
  77. [107][107] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 'Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps'. In: CoRR abs/1312.6034 (2013). arXiv: 1312.6034 . url : http://arxiv.org/abs/1312.6034 . · arXiv

Cited by

Sorted by citations

Authors
Robinson, A.
Affiliations
University of Bristol (GB)
Corresponding author
Robinson, Alex
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1905.13750
arXiv
1905.13750
OpenAlex
W2947973062
Cited by
18
References
8
Citations by year
2025: 1 · 2024: 3 · 2023: 4 · 2022: 3 · 2021: 3 · 2020: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
  • Physical Sciences › Computer Science › Information Systems › Web Data Mining and Analysis
Keywords
semantic segmentation, computer vision, evaluation framework

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review