Sikuli: using GUI screenshots for search and automation
The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
We present Sikuli, a visual approach to search and automation of graphical user interfaces using screenshots. Sikuli allows users to take a screenshot of a GUI element (such as a toolbar button, icon, or dialog box) and query a help system using the screenshot instead of the element's name. Sikuli also provides a visual scripting API for automating GUI interactions, using screenshot patterns to direct mouse and keyboard events. We report a web-based user study showing that searching by screenshot is easy to learn and faster to specify than keywords. We also demonstrate several automation tasks suitable for visual scripting, such as map navigation and bus tracking, and show how visual scripting can improve interactive help systems previously proposed in the literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472 · cited 5×
“The three-feature indexing scheme is written in C++ to index these screenshots, using SIFT [11] to extract visual features, Tesseract(5) for OCR, and Ferret(6) for indexing the text surrounding the screenshots.”
MethodInfluentialVideo Google: a text retrieval approach to object matching in videos
Sivic, & Zisserman (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238663 · Cited by 6,367 · cited 1×
“Recent advances in computer vision have demonstrated the effectiveness of representing an image as a set of visual words [18].”
BackgroundComputer Vision: A Modern Approach
Forsyth, D. A., & Ponce, J. (2002) · Cited by 3,739
Robust wide-baseline stereo from maximally stable extremal regions
Matas, J., Chum, O., Urban, M., & Pajdla, T. (2004) · Image and Vision Computing · doi:10.1016/j.imavis.2004.02.006 · Cited by 3,686
Robust Wide Baseline Stereo from Maximally Stable Extremal Regions
Matas, J., Chum, O., Urban, M., & Pajdla, T. (2002) · Procedings of the British Machine Vision Conference 2002 · doi:10.5244/c.16.36 · Cited by 1,350
CoScripter: automating & sharing how-to knowledge in the enterprise
Leshed, G., Haber, E. M., Matthews, T., & Lau, T. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357323 · Cited by 311 · cited 1×
“DocWizards [2], Chickenfoot [3], and CoScripter [10]).”
Automation and customization of rendered web pages
Bolin, M., Webber, M., Rha, P., Wilson, T., & Miller, R. C. (2005) · Proceedings of the 18th annual ACM symposium on User interface software and technology · doi:10.1145/1095034.1095062 · Cited by 223 · cited 2×
“Current approaches tend to require support from application developers (e.g., Ap-pleScript and Windows Scripting, which require applications to provide APIs) or accessible text labels for GUI elements (e.g. DocWizards [2], Chickenfoot [3], and Co-Scripter [10]).”
Self-citationMultiple Object Class Detection with a Generative Model
Mikolajczyk, K., Leibe, B., & Schiele, B. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR'06) · doi:10.1109/cvpr.2006.202 · Cited by 220 · cited 1×
“The particular algorithm we have adapted for our purpose [13] was originally used for”
MethodStencils-based tutorials: design and evaluation
Kelleher, C., & Pausch, R. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055047 · Cited by 187 · cited 3×
“Creating Stencils-based Tutorials The Stencils system described in [8] is a tutorial presentation technique that seeks to correct interface components by displaying translucent colored stencils with holes.”
BackgroundMethodAssieme: finding and leveraging implicit references in a web search interface for programmers
Hoffmann, R., Fogarty, J., & Weld, D. S. (2007) · Proceedings of the 20th annual ACM symposium on User interface software and technology · doi:10.1145/1294211.1294216 · Cited by 130 · cited 1×
“The questions were adapted from the evaluation of the Assieme search interface [6] and are listed below:”
MethodCoupling a UI framework with automatic generation of context-sensitive animated help
Sukaviriya, P., & Foley, J. D. (1990) · Proceedings of the 3rd annual ACM SIGGRAPH symposium on User interface software and technology - UIST '90 · doi:10.1145/97924.97942 · Cited by 105 · cited 1×
“For example, Sukaviriya & Foley [19] proposed a framework called Cartoonists for generating animated help as a sequence of user actions needed to perform the task by demonstrating these actions first.”
BackgroundDocWizards: a system for authoring follow-me documentation wizards
Bergman, L., Castelli, V., Lau, T., & Oblinger, D. (2005) · Proceedings of the 18th annual ACM symposium on User interface software and technology · doi:10.1145/1095034.1095067 · Cited by 96 · cited 2×
“DocWizards [2], Chickenfoot [3], and CoScripter [10]).”
BackgroundHarrison, S. M. (1995) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '95 · doi:10.1145/223904.223915 · Cited by 83 · cited 1×
“Harrison [ 5 ] found that users learned more quickly following tutorials illustrated by screenshots than reading textual-only tutorials.”
BackgroundAutomatic generation of task-oriented help
Pangoli, S., & Paternò, F. (1995) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/215585.215971 · Cited by 53 · cited 1×
“Furthermore, Pangoli & Paterno [ 15 ] demonstrated how to generate task-oriented help from user interface specification.”
BackgroundApple guide: a case study in user-aided design of online help
Knabe, K. (1995) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/223355.223677 · Cited by 46 · cited 1×
“However, Knabe [9] found that users often find it difficult to locate the interface components pictured in the tutorials.”
BackgroundA visual medium for programmatic control of interactive applications
Zettlemoyer, L., & Amant, R. S. (1999) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/302979.303039 · Cited by 43 · cited 1×
“Amant [20] described VisMap and VisScript.”
Automatic generation of help from interface design models
Moriyón, R., Szekely, P. A., & Neches, R. T. (1994) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191751 · Cited by 36 · cited 1×
“Moriyon et al [14] proposed a technique to generate hypertext help messages from a user interface design model.”
MethodGraphstract: minimal graphical help for computers
Huang, J. J. S., & Twidale, M. B. (2007) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1294211.1294248 · Cited by 33 · cited 9×
“However, as mentioned in [7], the fidelity of the replay depends on the absolute locations of the GUI elements and may break if the windows containing those elements are moved.”
BackgroundMethodInfluentialProgramming by example: visual generalization in programming by example
Amant, R. S., Lieberman, H. A., Potter, R., & Zettlemoyer, L. (2000) · Communications of the ACM · doi:10.1145/330534.330549 · Cited by 30
Prabaker, M. K., Bergman, L. D., & Castelli, V. (2006) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124809 · Cited by 18
Shneiderman, B., & Potter, R. (1999) · Cited by 11
Research works printed but not linked: 2
- 10.5555/850924.851523 · doi:10.5555/850924.851523
- 10.5555/946247.946751 · doi:10.5555/946247.946751
Cited by
PLUS: Open-Source Toolkit for Ultrasound-Guided Intervention Systems
Lassó, A., Heffter, T., Rankin, A., Pintér, C., Ungi, T., & Gabor Fichtinger, A. (2014) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2322864 · Cited by 421
“Automatic testing of the application graphical user interface is performed using Sikuli [36].”
Evaluation Strategies for HCI Toolkit Research
Ledo, D., Houben, S., Vermeulen, J., Marquardt, N., Oehlberg, L., & Greenberg, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173610 · Cited by 315
GUI testing using computer vision
Chang, T.-H., Yeh, T., & Miller, R. C. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753555 · Cited by 202
“Since Sikuli Script is based on a full scripting language, Python, it is possible for QA testers to programmatically simulate a large variety of user interactions, simple or complex.”
MethodInfluentialSelf-citationDirectGPT: A Direct Manipulation Interface to Interact with Large Language Models
Masson, D., Malacria, S., Casiez, G., & Vogel, D. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642462 · Cited by 107
SUGILITE: Creating Multimodal Smartphone Automation by Demonstration
Li, T. J.-J., Azaria, A., & Myers, B. A. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025483 · Cited by 184
“For future work, we plan to use visual references similar to SIKULI [43] to identify the graphical elements in the script.”
Method
Metadata
- Authors
- Yeh, T., Chang, T.-H., & Miller, R. C.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Venue
- Proceedings of the 22nd annual ACM symposium on User interface software and technology
- Publisher
- ACM
- Published
- 2009-10-04
- Pages
- 183-192
- OpenAlex
- W2141125339
- Cited by
- 346
- References
- 21
- Citations by year
- 2026: 11 · 2025: 12 · 2024: 11 · 2023: 15 · 2022: 17 · 2021: 28 · 2020: 28 · 2019: 25 · 2018: 29 · 2017: 33 · 2016: 24 · 2015: 22 · 2014: 30 · 2013: 21 · 2012: 15
- FWCI
- 10.19
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Symposium on User Interface Software and Technology (UIST) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Keywords
- visual scripting, map-based navigation, bus tracking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.