Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring accessibility metadata for mobile apps from their pixels, as the visual interfaces often best reflect an app’s full functionality. We trained a robust, fast, memory-efficient, on-device model to detect UI elements using a dataset of 77,637 screens (from 4,068 iPhone apps) that we collected and annotated. To further improve UI detections and add semantic information, we introduced heuristics (e.g., UI grouping and ordering) and additional models (e.g., recognize UI content, state, interactivity). We built Screen Recognition to generate accessibility metadata to augment iOS VoiceOver. In a study with 9 screen reader users, we validated that our approach improves the accessibility of existing mobile apps, enabling even previously inaccessible apps to be used.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
SSD: Single Shot MultiBox Detector
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_2 · Cited by 22,128
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
A Comprehensive Survey of Deep Learning for Image Captioning
Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H. (2019) · ACM Computing Surveys · doi:10.1145/3295748 · Cited by 879 · cited 1×
“To describe content in pictures, prior work used deep learning models to generate natural language descriptions of images [44, 46], and some accessibility improvement research has also leveraged crowdsourcing to generate image captions [35, 39, 40].”
MethodLeakage in data mining: Formulation, detection, and avoidance
Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/2382577.2382579 · Cited by 791
Rico: A Mobile App Dataset for Building Data-Driven Design Applications
Deka, B., Huang, Z., Franzen, C., Hibschman, J., Afergan, D., Li, Y., Nichols, J., & Kumar, R. (2017) · Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3126594.3126651 · Cited by 482
Self-citationGUI ripping: reverse engineering of graphical user interfaces for testing
Memon, A., Banerjee, I., & Nagarajan, A. (2004) · 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings. · doi:10.1109/wcre.2003.1287256 · Cited by 379 · cited 1×
“Other work extracts interface structure through system APIs [52].”
BackgroundSikuli: using GUI screenshots for search and automation
Yeh, T., Chang, T.-H., & Miller, R. C. (2009) · Proceedings of the 22nd annual ACM symposium on User interface software and technology · doi:10.1145/1622176.1622213 · Cited by 346
Guidelines are only half of the story: accessibility problems encountered by blind users on the web
Power, C., Freire, A., Petrie, H., & Swallow, D. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207736 · Cited by 327 · cited 1×
“Prior approaches for addressing this problem include encouraging developers to x accessibility problems through education [16], standards [15, 62], and better testing tools [14, 18].”
Backgroundpix2code: Generating Code from a Graphical User Interface Screenshot
Beltramelli, T. (2018) · Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems · doi:10.1145/3220134.3220135 · Cited by 237
Cited by
Enabling Conversational Interaction with Mobile UI using Large Language Models
Wang, B., Li, G., & Li, Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580895 · Cited by 156
“For example, Widget Captioning [38] and Screen Recognition [58] predict semantically meaningful alt-text labels for GUI components.”
BackgroundStangl, A., Verma, N., Fleischmann, K. R., Morris, M. R., & Gurari, D. (2021) · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3441852.3471233 · Cited by 81
AXNav: Replaying Accessibility Tests from Natural Language
Taeb, M., Swearngin, A., Schoop, E., Cheng, R., Jiang, Y., & Nichols, J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642777 · Cited by 52
“Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle.”
BackgroundMethodInfluentialSelf-citationZhang, Z., Yao, Y., Zhang, A., Tang, X., Ma, X., He, Z., Wang, Y., Gerstein, M., Wang, R., Liu, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3719341 · Cited by 51
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
You, K., Zhang, H., Schoop, E., Weers, F., Swearngin, A., Nichols, J., Yang, Y., & Gan, Z. (2024) · arXiv · doi:10.48550/arxiv.2404.05719 · Cited by 40
“For iPhone screens, we use the AMP dataset [58], which spans a broad spectrum of applications.”
MethodInfluential
Metadata
- Authors
- Zhang, X., de Greef, L., Swearngin, A., White, S., Murray, K., Yu, L., Shan, Q., Nichols, J., Wu, J., Fleizach, C., Everitt, A., & Bigham, J. P.
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-15
- OpenAlex
- W3163374010
- Cited by
- 144
- References
- 16
- FWCI
- 30.31
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. https://app.cochat.ai/papers/10.1145/3411764.3445186
- The paper
- Zhang, X., de Greef, L., Swearngin, A., White, S., Murray, K., Yu, L., Shan, Q., Nichols, J., Wu, J., Fleizach, C., Everitt, A., & Bigham, J. P. (2021). Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445186
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.