Accessible Gesture Typing for Non-Visual Text Entry on Smartphones
Open accessAbstract
Gesture typing--entering a word by gliding the finger sequentially over letter to letter-- has been widely supported on smartphones for sighted users. However, this input paradigm is currently inaccessible to blind users: it is difficult to draw shape gestures on a virtual keyboard without access to key visuals. This paper describes the design of accessible gesture typing, to bring this input paradigm to blind users. To help blind users figure out key locations, the design incorporates the familiar screen-reader supported touch exploration that narrates the keys as the user drags the finger across the keyboard. The design allows users to seamlessly switch between exploration and gesture typing mode by simply lifting the finger. Continuous touch-exploration like audio feedback is provided during word shape construction that helps the user glide in the right direction of the key locations constituting the word. Exploration mode resumes once word shape is completed. Distinct earcons help distinguish gesture typing mode from touch exploration mode, and thereby avoid unintended mix-ups. A user study with 14 blind people shows 35% increment in their typing speed, indicative of the promise and potential of gesture typing technology for non-visual text entry.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Assistive technology research for improving computer application usability has largely focused on web and mobile applications.
“However, these works have largely centered on the web [6, 12, 15, 16, 43] and mobile applications [11, 22, 27–29].”
From Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models (2024) · cites this paper as [11]
Experimental — accuracy not yet measuredOther reviewed studies aimed to improve the usability of QWERTY keyboards for VIBPs or to propose novel gestural or alternative layout keyboard-based input methods.
“Other studies aimed to improve the usability of QWERTY keyboards for VIBPs (e.g., [38]) or to propose novel gestural (stroke-based) [39] or alternative layout keyboard-based input methods [40].”
From A Review of Design and Evaluation Practices in Mobile Text Entry for Visually Impaired and Blind Persons (2023) · cites this paper as [39]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Kristensson, P.-O., & Zhai, S. (2004) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1029632.1029640 · Cited by 419 · see citing works
Kane, S. K., Bigham, J. P., & Wobbrock, J. O. (2008) · Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/1414471.1414487 · Cited by 374 · cited 3× · see citing works
“For exploring keys, we used the "exploration-by-touch" gesture [9].”
BackgroundMethodShorthand writing on stylus keyboard
Zhai, S., & Kristensson, P.-O. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642630 · Cited by 371 · see citing works
Usable gestures for blind people: understanding preference and performance
Kane, S. K., Wobbrock, J. O., & Ladner, R. E. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979001 · Cited by 314 · cited 1× · see citing works
“[10] explored the differences between how blind and sighted people use gestures for interacting with touchscreens.”
BackgroundZhao, S., Dragicevic, P., Chignell, M., Balakrishnan, R., & Baudisch, P. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240836 · Cited by 185 · see citing works
Yu, C., Gu, Y., Yang, Z., Yi, X., Luo, H., & Shi, Y. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025964 · Cited by 170 · see citing works
Zhai, S., & Kristensson, P. O. (2012) · Communications of the ACM · doi:10.1145/2330667.2330689 · Cited by 169 · see citing works
Markussen, A., Jakobsen, M. R., & Hornbæk, K. (2014) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/2556288.2556964 · Cited by 158 · see citing works
Investigating touchscreen accessibility for people with visual impairments
McGookin, D., Brewster, S., & Jiang, W. (2008) · Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges · doi:10.1145/1463160.1463193 · Cited by 151 · cited 1× · see citing works
“Problems with touch-screen accessibility and recommendations for addressing them appear in McGookin et al. [15].”
BackgroundAn evaluation of BrailleTouch: mobile touchscreen text entry for the visually impaired
Southern, C., Clawson, J., Frey, B., Abowd, G., & Romero, M. (2012) · Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services · doi:10.1145/2371574.2371623 · Cited by 125 · see citing works
Blind people and mobile touch-based text-entry: acknowledging the need for different flavors
Oliveira, J., Guerreiro, T., Nicolau, H., Jorge, J., & Gonçalves, D. (2011) · The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/2049536.2049569 · Cited by 123 · cited 1× · see citing works
“[17] describe a study around mobile touch-based text entry tasks with blind users and conclude that individual differences impact performance.”
BackgroundBrailleType: Unleashing Braille over Touch Screen Mobile Phones
Oliveira, J., Guerreiro, T., Nicolau, H., Jorge, J. A., & Gonçalves, D. J. V. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23774-4_10 · Cited by 111 · see citing works
(2012) · Graphics Interface · Cited by 103 · see citing works
From Tapping to Touching: Making Touch Screens Accessible to Blind Users
Guerreiro, T., Lagoá, P., Nicolau, H., Gonalves, D., & Jorge, J. A. (2008) · IEEE Multimedia · doi:10.1109/mmul.2008.88 · Cited by 89 · see citing works
Brown, B. A. T., McGregor, M., & McMillan, D. (2014) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/2628363.2628377 · Cited by 76 · see citing works
Adaptive blind interaction technique for touchscreens
Yfantidis, G., & Evreinov, G. (2006) · Universal Access in the Information Society · doi:10.1007/s10209-004-0109-7 · Cited by 73 · cited 1× · see citing works
“In Yfantidis and Evreinov [27], a single finger directional gesture in any of the 8 compass directions (North, Northeast, South,..) corresponds to a unique character.”
BackgroundCommand strokes with and without preview
Kristensson, P. O., & Zhai, S. (2007) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/1240624.1240797 · Cited by 62 · see citing works
Discrete and Continuous Shape Writing for Text Entry and Control
Kristensson, P. O. (2007) · DiVA (Linkoping University) · Cited by 59 · see citing works
Bi, X., Chelba, C. I., Ouyang, T. Y., Partridge, K., & Zhai, S. (2012) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2380116.2380136 · Cited by 55 · see citing works
Self-citationInvestigating Tilt-based Gesture Keyboard Entry for Single-Handed Text Entry on Large Devices
Yeo, H.-S., Phang, X.-S., Castellucci, S. J., Kristensson, P. O., & Quigley, A. J. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025520 · Cited by 52 · see citing works
Eyes-free text entry with error correction on touchscreen mobile devices
Tinwala, H., & MacKenzie, I. S. (2010) · Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries · doi:10.1145/1868914.1868972 · Cited by 51 · see citing works
Mobile Messenger for the Blind
Sánchez, J., & Aguayo, F. (2007) · Springer eBooks · doi:10.1007/978-3-540-71025-7_24 · Cited by 48 · see citing works
Use of Audio-Haptic Interface Techniques to Allow Nonvisual Access to Touchscreen Appliances
Vanderheiden, G. C. (1996) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193129604002430 · Cited by 47 · see citing works
Subway Mobility Assistance Tools for Blind Users
Sánchez, J., & Maureira, E. (2007) · Springer eBooks · doi:10.1007/978-3-540-71025-7_25 · Cited by 40 · see citing works
Human interface design and the handicapped user
Buxton, W. A. S., Foulds, R., Rosen, M., Scadden, L. A., & Shein, F. (1986) · ACM SIGCHI Bulletin · doi:10.1145/22339.22386 · Cited by 38 · see citing works
Typing Performance of Blind Users
Nicolau, H., Montague, K., Guerreiro, T., Rodrigues, A., & Hanson, V. L. (2015) · Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility - ASSETS '15 · doi:10.1145/2700648.2809861 · Cited by 37 · cited 1× · see citing works
“A longitudinal study of the typing performance of blind users is reported in Nicolau et al. [16].”
BackgroundBi, X., Azenkot, S., Partridge, K., & Zhai, S. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470732 · Cited by 35 · see citing works
Self-citationWord Clarity as a Metric in Sampling Keyboard Test Sets
Yi, X., Yu, C., Shi, W., Bi, X., & Shi, Y. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025701 · Cited by 26 · see citing works
Self-citationTabLETS Get Physical: Non-Visual Text Entry on Tablet Devices
Guerreiro, J., Rodrigues, A., Montague, K., Guerreiro, T., Nicolau, H., & Gonçalves, D. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702373 · Cited by 24 · cited 1× · see citing works
“The SpatialTouch system in Guerreiro et al. [7] leverages the full-size Qwerty keyboards in tablets for two-handed multitouch exploration for text entry by blind users.”
MethodLearning shape writing by game playing
Kristensson, P. O., & Zhai, S. (2007) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1240866.1240934 · Cited by 22 · see citing works
Zhai, S., & Kristensson, P. O. (2007) · Elsevier eBooks · doi:10.1016/b978-012373591-1/50007-3 · Cited by 19 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
TipText: Eyes-Free Text Entry on a Fingertip Keyboard
Xu, Z., Wong, P. C., Gong, J., Wu, T.-Y., Nittala, A. S., Bi, X., Steimle, J., Fu, H., Zhu, K., & Yang, X.-D. (2019) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3332165.3347865 · Cited by 95 · see citing works
“However, most of this work has been focused on enabling novel text entry schemes for visually impaired users [9, 52, 56] or for touch-screen devices [34, 66, 81, 82] where the screen real-estate is considerably larger than that of the finger-tip.”
BackgroundSelf-citationReview of Quantitative Empirical Evaluations of Technology for People with Visual Impairments
Brulé, E., Tomlinson, B. J., Metatla, O., Jouffrais, C., & Serrano, M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376749 · Cited by 76 · see citing works
GlanceWriter: Writing Text by Glancing Over Letters with Gaze
Cui, W., Liu, R., Li, Z., Wang, Y., Wang, A., Zhao, X., Rashidian, S., Baig, F., Ramakrishnan, I. V., Wang, F., et al. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581269 · Cited by 32 · see citing works
“There are also some works investigated on gesture typing without visual feedback [4, 59].”
BackgroundSelf-citationEnabling Uniform Computer Interaction Experience for Blind Users through Large Language Models
Kodandaram, S. R., Uckun, U., Bi, X., Ramakrishnan, IV, & Ashok, V. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675605 · Cited by 24 · see citing works
“However, these works have largely centered on the web [6, 12, 15, 16, 43] and mobile applications [11, 22, 27– 29].”
BackgroundSelf-citationCollaborative Tabletops for Blind People
Mendes, D., Reis, S., Guerreiro, J., & Nicolau, H. (2020) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3427325 · Cited by 18 · see citing works
“Over the last decade, numerous projects aimed at improving touchscreen interfaces, including understanding the performance of gestures [41], proposing new text input techniques [6, 24, 51, 52], and leveraging Braille knowledge [4, 58].”
BackgroundKomninos, A., Stefanis, V., & Garofalakis, J. (2023) · Multimodal Technologies and Interaction · doi:10.3390/mti7020022 · Cited by 16 · see citing works
Understanding VR Accessibility Practices of VR Professionals
Wáng, Y. X. J., Liu, X., Arora, C., Grundy, J., & Hoang, T. N. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713927 · Cited by 14 · see citing works
“This includes performing basic tasks such as reading the text, using the keyboard for typing, and perceiving sounds [8, 9, 20].”
Senorita: A Chorded Keyboard for Sighted, Low Vision, and Blind Mobile Users
Rakhmetulla, G., & Arif, A. S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376576 · Cited by 14 · see citing works
Limitations in speech recognition for young adults with down syndrome
Cibrian, F. L., Chen, Y. ‘., Anderson, K., Abrahamsson, C. M., & Motti, V. G. (2025) · Universal Access in the Information Society · doi:10.1007/s10209-025-01197-4 · Cited by 9 · see citing works
Islam, M. T., Porter, D. E., & Billah, S. M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581400 · Cited by 8 · see citing works
“In the future, we will explore how to adapt our model to measure the hard-to-type words on a gesture-based touchscreen keyboard [7] and hard-to-navigate printed forms with a wearable device [6].”
Self-citationText entry for the Blind on Smartwatches: A study of Braille code input methods for a novel device
Luna, M. M., Nascimento, H. A. D., Quigley, A., & Soares, F. (2023) · Universal Access in the Information Society · doi:10.1007/s10209-022-00870-2 · Cited by 7 · see citing works
“More recently, the usefulness of gesture input over the QWERTY layout has also been noted in a pilot study named AGTex [8], with promising early results.”
BackgroundEnhancing Older Adults’ Gesture Typing Experience Using the T9 Keyboard on Small Touchscreen Devices
Kuang, E., Chen, R., & Fan, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581105 · Cited by 5 · see citing works
“It is implemented by setting the threshold of the number of swiping direction changes on the X or Y axis to more than three [9].”
BackgroundMethodInfluentialAccessible Gesture Typing on Smartphones for People with Low Vision
Zhang, D., Li, Z., Ashok, V. G., Seiple, W., Ramakrishnan, I. V., & Bi, X. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676447 · Cited by 4 · see citing works
Self-citationModeling Gliding-based Target Selection for Blind Touchscreen Users
Ko, Y.-J., Putkonen, A., Aydin, A. S., Feiz, S., Wang, Y., Ashok, V., Ramakrishnan, IV, Oulasvirta, A., & Bi, X. (2021) · Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction · doi:10.1145/3447526.3472022 · Cited by 3 · see citing works
“In practice, both Android’s TalkBack [42] and Apple’s VoiceOver [2] screen readers support gliding for reaching a target (Figure 1), and many blind users rely on gliding to type on soft keyboards, as described in previous work such as [3, 8].”
Self-citationYim, J., Ko, D., Kim, T., Kim, T., Hong, J., & Lee, G. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747801 · Cited by 3 · see citing works
“In fact, Billah et al. [9]’s study attempted gesture typing without auditory confirmation, finding that error rates increased significantly and users felt less confident without audio feedback and strongly preferred providing auditory cues.”
Viago: Exploring Visual-Audio Modality Transitions for Social Media Consumption on the Go
Chang, R.-C., Grossman, T., Rognon, C., Glueck, M., Collins, C. M., Karlson, A. K., & Surale, H. B. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747761 · Cited by 1 · see citing works
“Prior research has explored promising methods for non-visual text input [15, 44, 91], which could potentially enhance or complement our current interaction techniques for on-the-go scenarios.”
Enabling Auto-Correction on Soft Braille Keyboard
Zhang, D., Ma, Y., Dausch, G., Seiple, W., Gu, D., Ramakrishnan, I. V., & Bi, X. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747699 · Cited by 1 · see citing works
“This performance is comparable to existing methods such as Qwerty with search-and-confirm (4–5 WPM, 25% WER) [43] and AGTex with gesture typing (5.66 WPM, 14.87% WER) [13].”
InfluentialSelf-citationA Systematic Review and Meta-analysis of Text Entry Studies
Iarygina, O., & Mottelson, A. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720117 · Cited by 3 · see citing works
Development of a Smartphone Operation and Text-Input Training App for Visually Impaired Individuals
Takechi, Y., Yamashita, A., & Matsubayashi, K. (2026) · International Conference on Knowledge and Smart Technology · doi:10.1109/kst67832.2026.11432403
“Further studies aiming to improve the typing experience of visually impaired users include the introduction of gesture typing techniques [9], development and evaluation of applications for Braille reading and writing practice [10], and investigations into enhancing app icon recognition accuracy…”
Vasilyev, D. (2026) · Программные системы и вычислительные методы · doi:10.7256/2454-0714.2026.2.79112
In-Air Hand Gesture Recognition Using Multi-Channel BLE Advertising RSSI
Mukherjee, A., Almohseni, H., & Najeeb, O. (2026) · IEEE International Conference on Electro Information Technology · doi:10.1109/eit68936.2026.11670427
“For users with visual impairments, gesture input has also been explored as an alternative to PIN entry [7]–[9], where reliance on proprioception rather than visual feedback is a natural fit.”
- Authors
- Billah, S. M., Ko, Y.-J., Ashok, V., Bi, X., & Ramakrishnan, IV
- Affiliations
- Stony Brook University (US)
- Venue
- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Volume
- 2019
- Pages
- 1-12
- PMC
- PMC7881442
- OpenAlex
- W2941364481
- Cited by
- 21
- References
- 31
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 2 · 2023: 4 · 2022: 1 · 2021: 1 · 2020: 3 · 2019: 1
- FWCI
- 2.59
- Type
- conference paper
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Eye Institute (R01EY026621)
- National Institute on Disability, Independent Living, and Rehabilitation Research
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Keywords
- blind users, accessibility, smartphones, virtual keyboard, screen readers, audio feedback, earcons, typing speed, assistive technology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
