Repurposing Visual Input Modalities for Blind Users: A Case Study of Word Processors
Open accessAbstract
Visual `point-and-click' interaction artifacts such as mouse and touchpad are tangible input modalities, which are essential for sighted users to conveniently interact with computer applications. In contrast, blind users are unable to leverage these visual input modalities and are thus limited while interacting with computers using a sequentially narrating screen-reader assistive technology that is coupled to keyboards. As a consequence, blind users generally require significantly more time and effort to do even simple application tasks (e.g., applying a style to text in a word processor) using only keyboard, compared to their sighted peers who can effortlessly accomplish the same tasks using a point-and-click mouse. This paper explores the idea of repurposing visual input modalities for non-visual interaction so that blind users too can draw the benefits of simple and efficient access from these modalities. Specifically, with word processing applications as the representative case study, we designed and developed NVMouse as a concrete manifestation of this repurposing idea, in which the spatially distributed word-processor controls are mapped to a virtual hierarchical `Feature Menu' that is easily traversable non-visually using simple scroll and click input actions. Furthermore, NVMouse enhances the efficiency of accessing frequently-used application commands by leveraging a data-driven prediction model that can determine what commands the user will most likely access next, given the current `local' screen-reader context in the document. A user study with 14 blind participants comparing keyboard-based screen readers with NVMouse, showed that the latter significantly reduced both the task-completion times and user effort (i.e., number of user actions) for different word-processing activities.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Blind users are frequently about ten times slower than sighted users on similar computer tasks.
“with research indicating that they are frequently ten times slower than sighted individuals when performing similar computer tasks [13, 26, 46].”
From Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models (2024) · cites this paper as [26]
The Abstract says blind screen-reader users generally need significantly more time than sighted peers for simple application tasks. It gives no tenfold estimate. The Evaluation > Results compares two input methods used by blind participants, not blind and sighted participants. (LLM, conf=0.88)
Experimental — accuracy not yet measuredBlind people often have difficulty performing basic tasks in computer applications.
“As a result, blind people often face difficulties in performing basic tasks (such as accessing ribbon commands) in applications [24].”
From Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models (2024) · cites this paper as [24]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · see citing works
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989 · see citing works
Complete Counterbalancing of Immediate Sequential Effects in a Latin Square Design
Bradley, J. V. (1958) · Journal of the American Statistical Association · doi:10.1080/01621459.1958.10501456 · Cited by 343 · cited 1× · see citing works
“In order to minimize learning effect, the ordering of tasks and conditions were counterbalanced using the Latin Square method [17].”
MethodUser interface of a Home Page Reader
Asakawa, C., & Itoh, T. (1998) · doi:10.1145/274497.274526 · Cited by 186 · cited 4× · see citing works
“To compensate for the lack of non-visual alternatives to convenient-to-use visual input modalities, alternative input modalities for blind users have been previously explored [6], [7].”
BackgroundMethodInfluentialEmployment Outcomes for Blind and Visually Impaired Adults
Bell, E. C., & Mino, N. M. (2015) · Journal of Blindness Innovation and Research · doi:10.5241/5-85 · Cited by 51 · cited 1× · see citing works
“Furthermore, the word processing applications are commonly used in everyday lives of blind users, and proficiency with the applications has been recognized as an important skill for the employment of blind individuals [4], [5].”
BackgroundNonvisual Interaction Techniques at the Keyboard Surface
Khurana, R., McIsaac, D., Lockerman, E., & Mankoff, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173585 · Cited by 34 · cited 2× · see citing works
“[10], on the other hand, propose spatially interaction techniques that leverage the keyboard surface to easily facilitate nonvisual interaction with 2D structures.”
BackgroundDoush, I. A., Pontelli, E., Simon, D. A., Son, T. C., & Ma, O. (2009) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1639642.1639669 · Cited by 33 · cited 3× · see citing works
“To compensate for the lack of non-visual alternatives to convenient-to-use visual input modalities, alternative input modalities for blind users have been previously explored [6], [7].”
BackgroundBillah, S. M., Ashok, V. G., Porter, D. E., & Ramakrishnan, I. V. (2017) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3132525.3132531 · Cited by 26 · cited 1× · see citing works
“Speed-Dial [9], like VoiceOver’s rotor, also supports easy hierarchical navigation of content via its external Microsoft Surface Dial input interface.”
MethodSelf-citationMuhammad, W., Aslam, M., & Farhan, M. (2019) · Symmetry · doi:10.3390/sym11020238 · Cited by 26 · see citing works
Designing E-Learning Collaborative Tools for Blind People
Claudia, M., Buzzi, M., & Mori, B. L. G. (2012) · InTech eBooks · doi:10.5772/31377 · Cited by 21 · cited 1× · see citing works
“As also mentioned in Section II, these features have also been individually explored by other works [6], [13]–[15].”
BackgroundWang, K., Barron, L. G., & Hebl, M. R. (2010) · Rehabilitation Psychology · doi:10.1037/a0018546 · Cited by 21 · cited 1× · see citing works
“Furthermore, the word processing applications are commonly used in everyday lives of blind users, and proficiency with the applications has been recognized as an important skill for the employment of blind individuals [4], [5].”
BackgroundApple Products: A Discussion of the Product Life Cycle
Więcek-Janka, E., Milosz, P., Kornecka, M., & Michal, N. (2017) · International Conference on Management Science and Management Innovation (MSMI) · doi:10.2991/msmi-17.2017.36 · Cited by 20 · cited 1× · see citing works
“Analogous to HPR’s numeric pad, Apple’s MacBook Touch Bar [8] provides contextual menus and navigation shortcuts for a variety of productivity applications.”
MethodDesign guidelines of a tool to help blind authors independently format their word documents
Morales, L., Arteaga, S. M., & Kurniawan, S. (2013) · CHI '13 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/2468356.2468363 · Cited by 16 · cited 1× · see citing works
“[14], on the other hand, focused only on formatting; they propose guidelines for a support tool in Microsoft Word that can assist blind users format their documents independently.”
MethodUsability evaluation of an accessible collaborative writing prototype for blind users
Schoeberlein, J. G., & Wang, Y. (2014) · Journal of Usability Studies archive · doi:10.5555/2817310.2817313 · Cited by 14 · see citing works
Making “Google Docs” User Interface More Accessible for Blind People
Mori, G., Buzzi, M. C., Buzzi, M., Leporini, B., & Penichet, V. M. R. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-20810-2_4 · Cited by 11 · cited 2× · see citing works
“Despite the importance of word processing applications, there is a dearth of studies on improving the usability of these applications for blind users [11]–[15].”
BackgroundNew accessibility model for Microsoft Windows and cross platform development
Haverty, R. (2005) · ACM SIGACCESS Accessibility and Computing · doi:10.1145/1077238.1077240 · Cited by 9 · cited 1× · see citing works
“However, the modular structure of NVMouse easily lets it replace Interop with other alternatives such as UI Automation accessibility framework [20], thereby enabling NVMouse to work with other applications.”
MethodDesigning E-Learning Collaborative Tools for Blind People
Buzzi, M. C., Buzzi, M., & Mori, B. L. G. (2012) · Contact-less Assessment of In-vivo Body Signals Using Microwave Doppler Radar (InTech) · Cited by 8 · see citing works
Document resizing for visually impaired students
Connolly, M., Lutteroth, C., & Plimmer, B. (2010) · doi:10.1145/1952222.1952248 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The Potential of Artificial Intelligence for Assistive Technology in Education
Zdravkova, K. (2022) · Learning and analytics in intelligent systems · doi:10.1007/978-3-031-04662-9_4 · Cited by 68 · see citing works
Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models
Kodandaram, S. R., Uckun, U., Bi, X., Ramakrishnan, IV, & Ashok, V. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675605 · Cited by 24 · see citing works
“As a result, blind people often face difficulties in performing basic tasks (such as accessing ribbon commands) in applications [24].”
BackgroundSelf-citationPerera, M., Ananthanarayan, S., Goncu, C., & Marriott, K. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713634 · Cited by 14 · see citing works
“Furthermore, SR users encounter issues fnding and accessing features due to the need to navigate through menus and toolbars [25, 66, 99, 106] or because of unlabelled controls [106].”
Taming User-Interface Heterogeneity with Uniform Overlays for Blind Users
Uckun, U., Tumkur Suresh, R., Ferdous, M. J., Bi, X., Ramakrishnan, I. V., & Ashok, V. (2022) · Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3503252.3531317 · Cited by 7 · see citing works
“screen reader users have to expend significantly more time and effort compared to their sighted peers to do the same tasks such as accessing application controls [29].”
BackgroundSelf-citationWheeler: A Three-Wheeled Input Device for Usable, Efficient, and Versatile Non-Visual Interaction
Islam, M. T., Sojib, N., Kabir, I., Amit, A. R., Amin, M. R., & Billah, S. M. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676396 · Cited by 6 · see citing works
“For example, Windows OS allows users to move the mouse cursor by pressing Num keys [4], commonly known as Mouse Keys; MacOS supports controlling the native VoiceOver [13] screen reader through trackpad inputs [3].”
MethodScreen Reading Enabled by Large Language Models
Ghosh, A., Reddy, M. P., Kodandaram, S. R., Uckun, U., Ashok, V. G., Bi, X., & Ramakrishnan, I. V. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3688491 · Cited by 6 · see citing works
“This design focus often makes it cumbersome for blind users to navigate and interact with these interfaces using keyboard shortcuts [5].”
Self-citationLeporini, B., Buzzi, M., & Penna, G. D. (2025) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3733155.3737910 · Cited by 4 · see citing works
Falcon, J. N., Kiss, J., Poncet, F., Wittich, W., Edwards, G., & Morales, E. (2025) · Assistive Technology · doi:10.1080/10400435.2025.2509699 · Cited by 3 · see citing works
Islam, M. T., Sojib, N., Kabir, I., Amit, A. R., Amin, M. R., & Billah, S. M. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3672539.3686749 · Cited by 1 · see citing works
“While prior research has proposed alternate input modalities with faster task completion times in specific scenarios [4, 8, 9], the challenge of navigating to UI items that belong to a different sub-tree remains.”
Background
- Authors
- Lee, H.-N., Ashok, V., & Ramakrishnan, I. V.
- Affiliations
- Stony Brook University (US)
- Old Dominion University (US)
- Venue
- 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)
- Publisher
- IEEE
- Published
- 2020-10-11
- Volume
- 2020
- Pages
- 2714-2721
- ISSN
- 1062-922X (ISSN-L)
- PMC
- PMC7871701
- OpenAlex
- W3111424221
- Cited by
- 9
- References
- 17
- Citations by year
- 2025: 3 · 2024: 4 · 2022: 2
- FWCI
- 0.68
- Type
- conference paper
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- Locations
- Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- ODU Digital Commons (Old Dominion University) (submittedVersion)
- Funders
- National Eye Institute (R01EY026621, R01EY030085)
- Eunice Kennedy Shriver National Institute of Child Health and Human Development (R01HD097188)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- blind users, word processing software, screen readers, non-visual interaction, assistive technology, accessibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
