Let's Talk About X: Combining Image Recognition and Eye Gaze to Support Conversation for People with ALS
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Communicating at a natural speed is a significant challenge for users of augmentative and alternative communication (AAC) devices, especially when input is provided by eye gaze, as is common for people with ALS and similar conditions. One way to improve AAC throughput is by drawing on contextual information from the outside world. Toward this goal, we present SceneTalk, a prototype gaze-based AAC system that uses computer vision to identify objects in the user's field of view and suggests words and phrases related to the current scene. We conducted a formative evaluation of SceneTalk with six people with ALS, in which we evaluated their preference for user interface modes and output preferences. Participants agreed that integrating contextual awareness into their AAC device could be helpful across a diverse range of situations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Prior research about AAC users has often identifed challenges with specifc forms of communication, such as when talking to a physician or telling a long story [18]”
From “The less I type, the better”: How AI Language Models can Enhance or Impede Communication for AAC Users (2023) · cites this paper as [18]
In the citing paper
“The social timing pressures that exist for AAC device users make it hard to add nuance and talk about other things beyond requests or short utterances [18].”
From “The less I type, the better”: How AI Language Models can Enhance or Impede Communication for AAC Users (2023) · cites this paper as [18]
Citations
Cites
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298754 · Cited by 1,287
Beukelman, D. R., & Mirenda, P. (2006) · Medical Entomology and Zoology · Cited by 757
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601 · cited 1×
“For example, if the user captured an image of a light switch, a crowd worker might suggest the phrase “Can you turn off the lights?” While crowd-based recognition is slower than automated recognition, crowd workers can be used to label images within minutes for a reasonable cost [6], which may still be faster than typ…”
BackgroundVizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Tatarowicz, A., White, B., White, S., & Yeh, T. (2010) · Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806020 · Cited by 536
Twenty years of eye typing: systems and design issues
Majaranta, P., & Räihä, K.-J. (2002) · Proceedings of the symposium on Eye tracking research & applications - ETRA '02 · doi:10.1145/507072.507076 · Cited by 483 · cited 1×
“Using gaze for real-time communication can be challenging, as eye-typing typically enables users to communicate at 10 to 20 words per minute (wpm) [18], while spoken conversation occurs at about 180 wpm [28].”
Fast hands-free writing by gaze direction
Ward, D. J., & MacKay, D. J. C. (2002) · Nature · doi:10.1038/418838a · Cited by 329 · cited 1×
“Alternative text entry methods such as Dasher [24,25] may offer improved performance, but learning an alternative text entry method requires extensive practice.”
MethodFast gaze typing with an adjustable dwell time
Majaranta, P., Ahola, U.-K., & Špakov, O. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518758 · Cited by 261 · cited 1×
“Prior approaches to speeding up AAC input include reducing dwell time [17,20], integrating word and phrase predictions [16], and supporting dwell-free eye-typing [15,22].”
Method(2017) · doi:10.1145/2998181.2998364 · Cited by 258 · cited 1×
“Automatic image captioning has recently been used to provide alternative descriptions of images for blind users [27], but has not previously been used to support communication.”
MethodToward Everyday Gaze Input: Accuracy and Precision of Eye Tracking and Implications for Design
Feit, A. M., Williams, S., Toledo, A., Paradiso, A., Kulkarni, H., Kane, S., & Morris, M. R. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025599 · Cited by 234 · cited 1×
“However, participants sometimes encountered difficulties using our prototype device due to calibration issues [9], positioning, and fatigue.”
BackgroundSelf-citationFast Hands-free Writing by Gaze Direction
Ward, D. J., & Mackay, D. (2002) · PubMed · doi:10.48550/arxiv.cs/0204030 · Cited by 214 · cited 1×
“Alternative text entry methods such as Dasher [24,25] may offer improved performance, but learning an alternative text entry method requires extensive practice.”
Method
Research works printed but not linked: 1
- ALS Association . Epidemiology of ALS and Suspected Clusters. ALSA.org . Retrieved May 25, 2016 from http://www.alsa.org/alscare/resources/publicationsvideos/factsheets/epidemiology.html ALS Association. Epidemiology of ALS and Suspected Clusters. ALSA.org. Retrieved May 25, 2016 from http://www.alsa.org/alscare/resources/publicationsvideos/factsheets/epidemiology.html
Cited by
Valencia, S., Cave, R., Kallarackal, K., Seaver, K., Terry, M., & Kane, S. K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581560 · Cited by 97
“Prior research about AAC users has often identifed challenges with specifc forms of communication, such as when talking to a physician or telling a long story [18], and that AAC users often conduct extensive work before a meeting to prepare what they wish to communicate [20].”
BackgroundInfluentialSelf-citationState of the Art in AAC: A Systematic Review and Taxonomy
Curtis, H., Neate, T., & Gonzalez, C. V. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3544810 · Cited by 60
“Knowledge of the context i.e., environment, location or communication partner have been harnessed in high-tech AAC systems for: natural language generation (NLG) [152], synthesised vocabulary searches, discourse prediction, improved adaptation to topics and to capture experiences [90, 91].”
BackgroundKane, S. K., Guo, A., & Morris, M. R. (2020) · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3373625.3416990 · Cited by 35
“Related paradigms like context-aware computing [18] expand the vision of how sensing infrastructures can support myriad implicit and explicit interactions; context-awareness has shown promise for supporting accessibility applications such as predictive text suggestions for AAC systems [36, 37].”
BackgroundSelf-citationConversational Agency in Augmentative and Alternative Communication
Valencia, S., Pavel, A., Santa Maria, J., Yu, S. (., Bigham, J. P., & Admoni, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376376 · Cited by 31
“As a first step towards making communication possible and efficient, designers and researchers of AAC have focused on improving device throughput, for example via text prediction [34], and vocabulary sorting [2, 21], and improving Figure 1.”
BackgroundWho Are You Asking?: Qualitative Methods for Involving AAC Users as Primary Research Participants
Beneteau, E. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376883 · Cited by 28
“At times, studies incorporate two or more of these methods [11,29,45].”
BackgroundBreaking Badge: Augmenting Communication with Wearable AAC Smartbadges and Displays
Curtis, H., Lau, Y. H., & Neate, T. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642327 · Cited by 26
“In contrast, the – often exerting [98] – communication interactions facilitated by mainstream AAC interventions are often found to be too slow [31, 46, 60, 84, 92, 93], unfulflling [41, 99] and depersonalised [97, 102].”
BackgroundThe Design Space of Nonvisual Word Completion
Nicolau, H., Rodrigues, A., Santos, A., Guerreiro, T., Montague, K., & Guerreiro, J. (2019) · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3308561.3353786 · Cited by 21
Why So Serious? Exploring Timely Humorous Comments in AAC Through AI-Powered Interfaces
Weinberg, T. M., Kadoma, K., Penuela, R. E. G., Valencia, S., & Roumen, T. J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714102 · Cited by 21
“See our project and demo at: tobiwg.github.io/research/why_so_serious”
Telehealth experiences of providers and patients who use augmentative and alternative communication
Beneteau, E., Paradiso, A., & Pratt, W. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab273 · Cited by 16
SpeakFaster Observer: Long-Term Instrumentation of Eye-Gaze Typing for Measuring AAC Communication
Cai, S., Venugopalan, S., Tomanek, K., Kane, S. K., Morris, M. R., Cave, R., MacDonald, R., Campbell, J., Casey, B., Kornman, E., et al. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3573870 · Cited by 16
“eye-gaze computer and geolocation, which may contain useful contextual signals for improving text prediction [9].”
BackgroundSelf-citationCognitive Load during Eye-typing
Bafna, T., Hansen, J. P., & Bækgaard, P. (2020) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3379155.3391333 · Cited by 16
“…but also design interfaces involving key selection [Blattgerste et al. 2018], develop adaptive interfaces [Katidioti et al. 2016], detect cognitive load for people with neuro-muscular problems who use eye-typing systems on a daily basis [Kane and Morris 2017] and overall replace self-reporting.”
BackgroundI, Robot? Exploring Ultra-Personalized AI-Powered AAC; an Autoethnographic Account
Weinberg, T. M., Penuela, R. E. G., Valencia, S., & Roumen, T. J. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790310 · Cited by 13
“More recent systems move beyond these strategies by inferring suggestions directly from context, drawing on cues like location, conversation transcripts, or surrounding images [42, 43, 68].”
Ivanyi, B. A., Tjemsland, T. B., Tsalidis, C., Tóth, L., Dyrholm, M. A., Naylor, S. J., Paradiso, A., Lamb, D., Chudge, J., Adjorlu, A., et al. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581285 · Cited by 11
“[32] developed an advanced augmentative and alternative communication (AAC) device, combining eye tracking with computer vision.”
BackgroundMethodA Design Space for Intelligent Dialogue Augmentation
Chan, R. S. M., Marx, A., Kim, A., & El‐Assady, M. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712096 · Cited by 9
“For instance, visual cues such as eye-gaze [66], mimics, or deictic gestures [105, 141] may be used to capture user attention, engagement, or non-verbal content.”
Investigating Speech Recognition for Improving Predictive AAC
Adhikary, J. K., Watling, R., Fletcher, C., Stanage, A. M., & Vertanen, K. (2019) · Workshop on Speech and Language Processing for Assistive Technologies · doi:10.18653/v1/w19-1706 · Cited by 8
“, 2012), vision (Kane and Morris, 2017), and location (Demmans Epp et al.”
BackgroundDesign Probes for AI-Driven AAC: Addressing Complex Communication Needs in Aphasia
Mao, L., Lee, J. H., Faroqi‐Shah, Y., & Valencia, S. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735736 · Cited by 6
“…further improve communication efficiency and autonomy, including using computer vision in AAC to identify objects and suggest phrases for ALS [39], generating symbolic boards from photos for autism [23], and integrating LLMs into AAC systems to create sentences from user-provided keywords [67].”
InfluentialVilla, S., Weiß, Y., Lu, M. Y., Ziarko, M., Schmidt, A., & Niess, J. (2024) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3678957.3685731 · Cited by 6
“Contextual information has further served as input for user support, with Kane and Morris [23] using object context to suggest relevant words for individuals with Amyotrophic Lateral Sclerosis (ALS) and Fontana de Vargas et al. [14] generating context-specifc suggestions for users with Autistic…”
Beneteau, E., Hiniker, A., Tench, B., Ibrahim, S. B., & Pratt, W. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2436038 · Cited by 5
“…for people with motor and communication disabilities who use AT frequently involve end users as expert consultants who inform design through interviews or evaluate prototypes, rather than as co-designers (e.g., Fiannaca et al., 2018; Ivanyi et al., 2023; Kane & Morris, 2017; Sobel et al., 2017).”
Designing AAC for use in social and community contexts: a scoping review
Frisch, B., & Vertanen, K. (2025) · Augmentative and Alternative Communication · doi:10.1080/07434618.2025.2558851 · Cited by 5
“Other designs have explored the impact that technologies like context-aware computing (Kane et al., 2012), image recognition (Kane & Morris, 2017), and natural language processing (Dempster et al., 2010; Higginbotham et al., 2012; Reiter et al., 2009) have on communicating with AAC.”
Design Probes for AI-Driven AAC: Addressing Complex Communication Needs in AphasiaPreprint
Mao, L., Lee, J. H., Shah, Y. F., & Valencia, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.09435 · Cited by 5
“…further improve communication efficiency and autonomy, including using computer vision in AAC to identify objects and suggest phrases for ALS [39], generating symbolic boards from photos for autism [23], and integrating LLMs into AAC systems to create sentences from user-provided keywords [67].”
InfluentialCondado, P. A., & Lobo, F. G. (2026) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3797867.3832761 · Cited by 1
Guidelines for Gaze-based Neural Preliminary DiagnosisPreprint
Elfares, M., Younis, S., Reisert, P., Küsters, R., Renner, T., & Bulling, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.08517 · Cited by 1
Internet of Things-Based Digital Life to Provide Assisted Living for Visually Challenged Persons
Udgata, S. K., & Kumar, G. S. (2023) · Algorithms for intelligent systems · doi:10.1007/978-981-99-3076-0_1
Frisch, B., Peters, B., & Vertanen, K. D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.00202
“Other research has focused on integrating specific technologies, such as context-aware computing [31], image recognition [32], and natural language processing [15, 28, 57].”
Yang, B., Jiang, P., & Kristensson, P. O. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.09610
“The identified objects are then listed on the text entry keyboard [15].”
Simulating the Human in Human-Centred Design: Increasing Efficiency in Design Activities
Yang, B. (2024) · Apollo (University of Cambridge) · doi:10.17863/cam.120556
A User Experience Evaluation of AAC Software
Frisch, B. W. M. (2020) · OhioLink ETD Center (Ohio Library and Information Network)
DESIGN PROBES FOR AI−DRIVEN AAC: ADDRESSING COMPLEX COMMUNICATION NEEDS IN APHASIA
Mao, L. (2025) · University Libraries (University of Maryland) · doi:10.13016/utqj-vqev
Beyond words : leveraging language models for incremental and context-aware text-to-speech synthesis
Stephenson, B. (2023) · theses.fr (ABES) · doi:10.70675/5b6c082czf0c2z4055zb230z5c5b56888ca5
It's Complicated: On the Design and Evaluation of AI-Powered AAC InterfacesPreprint
Frisch, B., Wade, W., Gaines, D. C., Kinsella, M., Peters, B., Broderick, T., & Vertanen, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.24854
“For example, AI might glean such clues from listening to the conversations around the user [1, 66], integrating suggestions from their communication partner [45], using a camera to infer things about a user’s surrounding environment [35], or identifying who the user is speaking with [34].”
da Silva, F. T. F., & Cunha, B. C. R. (2026) · Brazilian Symposium on Multimedia and the Web · doi:10.5753/webmedia.2026.30554
DIY Customized Speech Recognition for a Person with Speech Disabilities
Condado, P. A., & Lobo, F. G. (2026) · arXiv · doi:10.48550/arxiv.2611.11327
Metadata
- Authors
- Kane, S. K., & Morris, M. R.
- Affiliations
- University of Colorado Boulder (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the 2017 Conference on Designing Interactive Systems
- Publisher
- ACM
- Published
- 2017-06-10
- Pages
- 129-134
- OpenAlex
- W2606613250
- Cited by
- 20
- References
- 29
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 3 · 2023: 4 · 2022: 1 · 2021: 1 · 2020: 4 · 2019: 2
- FWCI
- 1.24
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Designing Interactive Systems Conference (publishedVersion)
- Funders
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Keywords
- augmentative and alternative communication, eye-gaze input, amyotrophic lateral sclerosis, context awareness, scene understanding, object recognition, word prediction, user interface design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.