Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility
The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Abstract
We introduce interaction proxies as a strategy for runtime repair and enhancement of the accessibility of mobile applications. Conceptually, interaction proxies are inserted between an application's original interface and the manifest interface that a person uses to perceive and manipulate the application. This strategy allows third-party developers and researchers to modify an interaction without an application's source code, without rooting the phone, without otherwise modifying an application, while retaining all capabilities of the system (e.g., Android's full implementation of the TalkBack screen reader). This paper introduces interaction proxies, defines a design space of interaction re-mappings, identifies necessary implementation abstractions, presents details of implementing those abstractions in Android, and demonstrates a set of Android implementations of interaction proxies from throughout our design space. We then present a set of interviews with blind and low-vision people interacting with our prototype interaction proxies, using these interviews to explore the seamlessness of interaction, the perceived usefulness and potential of interaction proxies, and visions of how such enhancements could gain broad usage. By allowing third-party developers and researchers to improve an interaction, interaction proxies offer a new approach to personalizing mobile application accessibility and a new approach to catalyzing development, deployment, and evaluation of mobile accessibility enhancements.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In this paper, we describe our implementation which, although it leveraged concepts from Interaction Proxies, was entirely novel.”
From Interactiles: 3D Printed Tactile Interfaces to Enhance Mobile Touchscreen Accessibility (2018) · cites this paper as [32]
Citations
Cites
del Carmen Malbrán, M. (2011) · Journal of Policy and Practice in Intellectual Disabilities · doi:10.1111/j.1741-1130.2011.00320.x · Cited by 1,674
Direct Manipulation Interfaces
Hutchins, E. L., Hollan, J. D., & Norman, D. A. (1985) · Human–Computer Interaction · doi:10.1207/s15327051hci0104_2 · Cited by 1,307
Past, present, and future of user interface software tools
Myers, B., Hudson, S. E., & Pausch, R. (2000) · ACM Transactions on Computer-Human Interaction · doi:10.1145/344949.344959 · Cited by 808 · cited 1×
“Although such personalization is promising for many accessibility needs, adoption of such methods is limited by a need for interfaces to be re-written as abstract specifications [45].”
BackgroundVizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601 · cited 1×
“VizWiz examines supporting blind people in using a mobile device to take a picture and ask a crowd a question about that picture [5,11].”
BackgroundAbility-Based Design: Concept, Principles and Examples
Wobbrock, J. O., Kane, S. K., Gajos, K. Z., Harada, S., & Froehlich, J. (2011) · ACM Transactions on Accessible Computing · doi:10.1145/1952383.1952384 · Cited by 567 · cited 5×
“Figure 10 shows an example of a personalized interface layout, motivated by SUPPLE’s approach to arranging entire interfaces to match an individual’s motor abilities [23,58].”
BackgroundMethodInfluentialSelf-citationVizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Tatarowicz, A., White, B., White, S., & Yeh, T. (2010) · Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806020 · Cited by 536
Kane, S. K., Jayant, C., Wobbrock, J. O., & Ladner, R. E. (2009) · Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1639642.1639663 · Cited by 383 · cited 1×
“Prior research examines how people adapt to accessibility barriers in mobile devices and how devices can enhance access in the physical world [35].”
BackgroundSelf-citationMaking computers easier for older adults to use: area cursors and sticky icons
Worden, A., Walker, N., Bharat, K., & Hudson, S. (1997) · Proceedings of the ACM SIGCHI Conference on Human factors in computing systems · doi:10.1145/258549.258724 · Cited by 376 · cited 1×
“Examples include making targets “sticky” [29,59], dynamically modifying cursor behavior according to a person’s movement profile [57], and breaking a pointing interaction into multiple interactions that disambiguate the intended target [22,33,63].”
BackgroundKane, S. K., Bigham, J. P., & Wobbrock, J. O. (2008) · Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/1414471.1414487 · Cited by 374 · cited 3×
“, Slide Rule [34], Android’s TalkBack, iOS’s VoiceOver), which re-maps touch to support navigation separate from activation and may completely disable visual display of an app.”
BackgroundInfluentialSelf-citationEnsuring Digital Accessibility Through Process and Policy
Lazar, J., Goldstein, D., & Taylor, A. (2015) · Elsevier eBooks · doi:10.1016/c2013-0-13367-3 · Cited by 359 · cited 1×
“, Android’s TalkBack, iOS’s VoiceOver), and now improve accessibility of apps on those platforms [42].”
Background
Cited by
SeeingVR: A Set of Tools to Make Virtual Reality More Accessible to People with Low Vision
Zhao, Y., Cutrell, E., Holz, C., Morris, M. R., Ofek, E., & Wilson, A. D. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300341 · Cited by 207
“introduced interaction proxies [81], which improved the accessibility of mobile apps by adding an overlay that modified the input and output of the underlying app.”
BackgroundScreen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Zhang, X., de Greef, L., Swearngin, A., White, S., Murray, K., Yu, L., Shan, Q., Nichols, J., Wu, J., Fleizach, C., et al. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445186 · Cited by 144
“tory of such information for runtime repair [80, 81].”
BackgroundSelf-citationLi, T. J.-J., Radensky, M., Jia, J., Singarajah, K., Mitchell, T. M., & Myers, B. A. (2019) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3332165.3347899 · Cited by 188
“P UMICE uses an interaction proxy overlay [53] for recording, so that it can record all values visible on the screen, not limited to the selectable or clickable ones.”
Unblind your apps: predicting natural-language labels for mobile GUI components by deep learning
Chen, J., Chen, C., Xing, Z., Xu, X., Zhu, L., Li, G., & Wang, J. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380327 · Cited by 182
“For other accessibility issues, they further develop an approach to deploy the interaction proxies for runtime repair and enhancement of mobile application accessibility [77]”
BackgroundCaption Crawler: Enabling Reusable Alternative Text Descriptions using Reverse Image Search
Guinness, D., Cutrell, E., & Morris, M. R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174092 · Cited by 108
“Zhang et al. presented a proxy-based method for runtime repair of Android applications [39].”
Method
Metadata
- Authors
- Zhang, X., Ross, A. S., Caspi, A., Fogarty, J., & Wobbrock, J. O.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2017-05-02
- Pages
- 6024-6037
- OpenAlex
- W2610917376
- Semantic Scholar
- 8577f8716ef27982a47963c498bd12f891c70702
- Cited by
- 59
- References
- 59
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 3 · 2023: 5 · 2022: 5 · 2021: 11 · 2020: 10 · 2019: 8 · 2018: 6 · 2017: 3
- FWCI
- 14.34
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- National Science Foundation (IIS-1053868)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- mobile application accessibility, Android, blind and visually impaired users
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility. https://app.cochat.ai/papers/10.1145/3025453.3025846
- The paper
- Zhang, X., Ross, A. S., Caspi, A., Fogarty, J., & Wobbrock, J. O. (2017). Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3025453.3025846
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.