Sikuli: using GUI screenshots for search and automation
The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
We present Sikuli, a visual approach to search and automation of graphical user interfaces using screenshots. Sikuli allows users to take a screenshot of a GUI element (such as a toolbar button, icon, or dialog box) and query a help system using the screenshot instead of the element's name. Sikuli also provides a visual scripting API for automating GUI interactions, using screenshot patterns to direct mouse and keyboard events. We report a web-based user study showing that searching by screenshot is easy to learn and faster to specify than keywords. We also demonstrate several automation tasks suitable for visual scripting, such as map navigation and bus tracking, and show how visual scripting can improve interactive help systems previously proposed in the literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472
Video Google: a text retrieval approach to object matching in videos
Sivic, & Zisserman (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238663 · Cited by 6,367 · cited 1×
“Recent advances in computer vision have demonstrated the effectiveness of representing an image as a set of visual words [18].”
BackgroundComputer Vision: A Modern Approach
Forsyth, D. A., & Ponce, J. (2002) · Cited by 3,739
Robust wide-baseline stereo from maximally stable extremal regions
Matas, J., Chum, O., Urban, M., & Pajdla, T. (2004) · Image and Vision Computing · doi:10.1016/j.imavis.2004.02.006 · Cited by 3,686
Robust Wide Baseline Stereo from Maximally Stable Extremal Regions
Matas, J., Chum, O., Urban, M., & Pajdla, T. (2002) · Procedings of the British Machine Vision Conference 2002 · doi:10.5244/c.16.36 · Cited by 1,350
CoScripter: automating & sharing how-to knowledge in the enterprise
Leshed, G., Haber, E. M., Matthews, T., & Lau, T. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357323 · Cited by 311
Automation and customization of rendered web pages
Bolin, M., Webber, M., Rha, P., Wilson, T., & Miller, R. C. (2005) · Proceedings of the 18th annual ACM symposium on User interface software and technology · doi:10.1145/1095034.1095062 · Cited by 223
Self-citationMultiple Object Class Detection with a Generative Model
Mikolajczyk, K., Leibe, B., & Schiele, B. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR'06) · doi:10.1109/cvpr.2006.202 · Cited by 220
Stencils-based tutorials: design and evaluation
Kelleher, C., & Pausch, R. (2005) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055047 · Cited by 187
Hoffmann, R., Fogarty, J. A., & Weld, D. S. (2007) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1294211.1294216 · Cited by 131
Research works printed but not linked: 2
- 10.5555/850924.851523 · doi:10.5555/850924.851523
- 10.5555/946247.946751 · doi:10.5555/946247.946751
Cited by
PLUS: Open-Source Toolkit for Ultrasound-Guided Intervention Systems
Lassó, A., Heffter, T., Rankin, A., Pintér, C., Ungi, T., & Gabor Fichtinger, A. (2014) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2322864 · Cited by 421
Evaluation Strategies for HCI Toolkit Research
Ledo, D., Houben, S., Vermeulen, J., Marquardt, N., Oehlberg, L., & Greenberg, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173610 · Cited by 315
GUI testing using computer vision
Chang, T.-H., Yeh, T., & Miller, R. C. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753555 · Cited by 202
Self-citationDirectGPT: A Direct Manipulation Interface to Interact with Large Language Models
Masson, D., Malacria, S., Casiez, G., & Vogel, D. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642462 · Cited by 107
SUGILITE: Creating Multimodal Smartphone Automation by Demonstration
Li, T. J.-J., Azaria, A., & Myers, B. A. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025483 · Cited by 184
Metadata
- Authors
- Yeh, T., Chang, T.-H., & Miller, R. C.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Venue
- Proceedings of the 22nd annual ACM symposium on User interface software and technology
- Publisher
- ACM
- Published
- 2009-10-04
- Pages
- 183-192
- OpenAlex
- W2141125339
- Cited by
- 346
- References
- 21
- Citations by year
- 2026: 11 · 2025: 12 · 2024: 11 · 2023: 15 · 2022: 17 · 2021: 28 · 2020: 28 · 2019: 25 · 2018: 29 · 2017: 33 · 2016: 24 · 2015: 22 · 2014: 30 · 2013: 21 · 2012: 15
- FWCI
- 10.19
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Symposium on User Interface Software and Technology (UIST) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Keywords
- visual scripting, map-based navigation, bus tracking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Sikuli: using GUI screenshots for search and automation. https://app.cochat.ai/papers/10.1145/1622176.1622213
- The paper
- Yeh, T., Chang, T.-H., & Miller, R. C. (2009). Sikuli: using GUI screenshots for search and automation. Proceedings of the 22nd annual ACM symposium on User interface software and technology. https://doi.org/10.1145/1622176.1622213
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.