Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Current chasms between applications implemented with different user interface toolkits make it difficult to implement and explore potentially important interaction techniques in new and existing applications, limiting the progress and impact of human-computer interaction research. We examine an approach based in the single most common characteristic of all graphical user interface toolkits, that they ultimately paint pixels to a display. We present Prefab, a system for implementing advanced behaviors through the reverse engineering of the pixels in graphical interfaces. Informed by how user interface toolkits paint interfaces, Prefab features a separation of the modeling of widget layout from the recognition of widget appearance. We validate Prefab in implementations of three applications: target-aware pointing techniques, Phosphor transitions, and Side Views parameter spectrums. Working only from pixels, we demonstrate a single implementation of these enhancements in complex existing applications created in different user interface toolkits running on different windowing systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Sikuli: using GUI screenshots for search and automation
Yeh, T., Chang, T.-H., & Miller, R. C. (2009) · Proceedings of the 22nd annual ACM symposium on User interface software and technology · doi:10.1145/1622176.1622213 · Cited by 346 · cited 2×
“’s recently developed Sikuli system uses a combination of template matching and voting based on invariant local features, and it requires a reported 200msec to identify all occurrences of a single target [25].”
BackgroundAutomatically identifying targets users interact with during real world tasks
Hurst, A., Hudson, S. E., & Mankoff, J. (2010) · Proceedings of the 15th international conference on Intelligent user interfaces · doi:10.1145/1719970.1719973 · Cited by 50
Cited by
Ability-Based Design: Concept, Principles and Examples
Wobbrock, J. O., Kane, S. K., Gajos, K. Z., Harada, S., & Froehlich, J. (2011) · ACM Transactions on Accessible Computing · doi:10.1145/1952383.1952384 · Cited by 567
How to evaluate technologies for health behavior change in HCI research
Klasnja, P., Consolvo, S., & Pratt, W. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979396 · Cited by 414
“…than current techniques that address the same problem; similarly, HCI systems papers try to show that the new system can do what it is intended to do—help users better understand their privacy settings, manipulate GUI applications by recognizing their widgets at the pixel level [14], and so on.”
BackgroundAutomatically generating personalized user interfaces with Supple
Gajos, K. Z., Weld, D. S., & Wobbrock, J. O. (2010) · Artificial Intelligence · doi:10.1016/j.artint.2010.05.005 · Cited by 317
“The feasibility of this approach has been previously demonstrated for traditional (non-AJAX) web sites [9,64] and more recently for desktop user interfaces [12,45].”
BackgroundEvaluation Strategies for HCI Toolkit Research
Ledo, D., Houben, S., Vermeulen, J., Marquardt, N., Oehlberg, L., & Greenberg, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173610 · Cited by 315
Generative Adversarial Networks for face generation: A survey
Kammoun, A., Slama, R., Tabia, H., Ouni, T., & Abid, M. (2022) · ACM Computing Surveys · doi:10.1145/1122445.1122456 · Cited by 311
pix2code: Generating Code from a Graphical User Interface Screenshot
Beltramelli, T. (2018) · Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems · doi:10.1145/3220134.3220135 · Cited by 237
Reverse Engineering Mobile Application User Interfaces with REMAUI (T)
Nguyen, T. A., & Csallner, C. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.32 · Cited by 200
Chen, C., Su, T., Meng, G., Xing, Z., & Liu, Y. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180240 · Cited by 188
MixT: automatic generation of step-by-step mixed media tutorials
Chi, P.-Y., Ahn, S., Ren, A., Dontcheva, M., Li, W., & Hartmann, B. (2012) · Proceedings of the 25th annual ACM symposium on User interface software and technology · doi:10.1145/2380116.2380130 · Cited by 113
“Methods that analyze user action traces [8,3,10] often capture application-level semantics about specific tools and their purpose, while techniques that analyze screen recordings [18,20,4] use computer vision to identify GUI interactions such as clicking on an icon or button.”
Enhanced area cursors: reducing fine pointing demands for people with motor impairments
Findlater, L., Jansen, A., Shinohara, K., Dixon, M., Kamb, P., Rakita, J., & Wobbrock, J. O. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866055 · Cited by 98
Self-citationPatina: dynamic heatmaps for visualizing application usage
Matejka, J., Grossman, T., & Fitzmaurice, G. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466442 · Cited by 67
Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility
Zhang, X., Ross, A. S., Caspi, A., Fogarty, J., & Wobbrock, J. O. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025846 · Cited by 59
“Runtime modification can be implemented using only pixel-level analysis [15,16,17,18,19,61] or in combination with information from an accessibility API [13].”
BackgroundMethodSelf-citationAXNav: Replaying Accessibility Tests from Natural Language
Taeb, M., Swearngin, A., Schoop, E., Cheng, R., Jiang, Y., & Nichols, J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642777 · Cited by 52
“Using pixels to detect UI elements makes AXNav agnostic to the underlying UI framework [18].”
BackgroundStateLens: A Reverse Engineering Solution for Making Existing Dynamic Touchscreens Accessible
Guo, A., Kong, J., Rivera, M., Xu, F. F., & Bigham, J. P. (2019) · Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3332165.3347873 · Cited by 35
“rentiating minor interface changes such as toggle buttons or color indicators. One solution may be to detect and factor in UI widgets that are expected to change using approaches like those in PreFab [12] and TapShoe [32]. Furthermore, StateLens cannot currently handle major updates and layout changes of the interface…”
BackgroundMethodCodeA11y: Making AI Coding Assistants Useful for Accessible Web Development
Mowar, P., Peng, Y.-H., Wu, J., Steinfeld, A., & Bigham, J. P. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713335 · Cited by 31
ShowMeHow: translating user interface instructions between applications
Ramesh, V., Hsu, C., Agrawala, M., & Hartmann, B. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047212 · Cited by 27
(2019) · doi:10.1145/3308561.3353808 · Cited by 14
GUIPilot: A Consistency-Based Mobile GUI Testing Approach for Detecting Application-Specific Bugs
Liu, R., Teoh, X., Lin, Y., Chen, G., Ren, R., Poshyvanyk, D., & Dong, J. S. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3728909 · Cited by 12
CLI-Anything: Towards Agent-Native Computer Use
Yang, Y., Fan, T., & Huang, C. (2026) · arXiv · doi:10.48550/arxiv.2606.03854
“Earlier GUI automation systems showed that screenshot search and pixel-based interface recovery can be useful when structure is unavailable [24, 25].”
Metadata
- Authors
- Dixon, M., & Fogarty, J.
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2010-04-10
- Pages
- 1525-1534
- OpenAlex
- W2158447593
- Cited by
- 137
- References
- 2
- FWCI
- 9.07
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.