Let's Talk About X: Combining Image Recognition and Eye Gaze to Support Conversation for People with ALS
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Communicating at a natural speed is a significant challenge for users of augmentative and alternative communication (AAC) devices, especially when input is provided by eye gaze, as is common for people with ALS and similar conditions. One way to improve AAC throughput is by drawing on contextual information from the outside world. Toward this goal, we present SceneTalk, a prototype gaze-based AAC system that uses computer vision to identify objects in the user's field of view and suggests words and phrases related to the current scene. We conducted a formative evaluation of SceneTalk with six people with ALS, in which we evaluated their preference for user interface modes and output preferences. Participants agreed that integrating contextual awareness into their AAC device could be helpful across a diverse range of situations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Prior research about AAC users has often identifed challenges with specifc forms of communication, such as when talking to a physician or telling a long story [18]”
From “The less I type, the better”: How AI Language Models can Enhance or Impede Communication for AAC Users (2023) · cites this paper as [18]
In the citing paper
“The social timing pressures that exist for AAC device users make it hard to add nuance and talk about other things beyond requests or short utterances [18].”
From “The less I type, the better”: How AI Language Models can Enhance or Impede Communication for AAC Users (2023) · cites this paper as [18]
Citations
Cites
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298754 · Cited by 1,287
Beukelman, D. R., & Mirenda, P. (2006) · Medical Entomology and Zoology · Cited by 757
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601 · cited 1×
“For example, if the user captured an image of a light switch, a crowd worker might suggest the phrase “Can you turn off the lights?” While crowd-based recognition is slower than automated recognition, crowd workers can be used to label images within minutes for a reasonable cost [6], which may still be faster than typ…”
BackgroundVizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Tatarowicz, A., White, B., White, S., & Yeh, T. (2010) · Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806020 · Cited by 536
Twenty years of eye typing: systems and design issues
Majaranta, P., & Räihä, K.-J. (2002) · Proceedings of the symposium on Eye tracking research & applications - ETRA '02 · doi:10.1145/507072.507076 · Cited by 483
Fast hands-free writing by gaze direction
Ward, D. J., & MacKay, D. J. C. (2002) · Nature · doi:10.1038/418838a · Cited by 329
Fast gaze typing with an adjustable dwell time
Majaranta, P., Ahola, U.-K., & Špakov, O. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518758 · Cited by 261
(2017) · doi:10.1145/2998181.2998364 · Cited by 258 · cited 1×
“Automatic image captioning has recently been used to provide alternative descriptions of images for blind users [27], but has not previously been used to support communication.”
MethodToward Everyday Gaze Input: Accuracy and Precision of Eye Tracking and Implications for Design
Feit, A. M., Williams, S., Toledo, A., Paradiso, A., Kulkarni, H., Kane, S., & Morris, M. R. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025599 · Cited by 234
Self-citationFast Hands-free Writing by Gaze Direction
Ward, D. J., & Mackay, D. (2002) · PubMed · doi:10.48550/arxiv.cs/0204030 · Cited by 214
Research works printed but not linked: 1
- ALS Association . Epidemiology of ALS and Suspected Clusters. ALSA.org . Retrieved May 25, 2016 from http://www.alsa.org/alscare/resources/publicationsvideos/factsheets/epidemiology.html ALS Association. Epidemiology of ALS and Suspected Clusters. ALSA.org. Retrieved May 25, 2016 from http://www.alsa.org/alscare/resources/publicationsvideos/factsheets/epidemiology.html
Cited by
Valencia, S., Cave, R., Kallarackal, K., Seaver, K., Terry, M., & Kane, S. K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581560 · Cited by 97
“Prior research about AAC users has often identifed challenges with specifc forms of communication, such as when talking to a physician or telling a long story [18], and that AAC users often conduct extensive work before a meeting to prepare what they wish to communicate [20].”
BackgroundInfluentialSelf-citationState of the Art in AAC: A Systematic Review and Taxonomy
Curtis, H., Neate, T., & Gonzalez, C. V. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3544810 · Cited by 60
Kane, S. K., Guo, A., & Morris, M. R. (2020) · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3373625.3416990 · Cited by 35
Self-citationConversational Agency in Augmentative and Alternative Communication
Valencia, S., Pavel, A., Santa Maria, J., Yu, S. (., Bigham, J. P., & Admoni, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376376 · Cited by 31
Who Are You Asking?: Qualitative Methods for Involving AAC Users as Primary Research Participants
Beneteau, E. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376883 · Cited by 28
Metadata
- Authors
- Kane, S. K., & Morris, M. R.
- Affiliations
- University of Colorado Boulder (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the 2017 Conference on Designing Interactive Systems
- Publisher
- ACM
- Published
- 2017-06-10
- Pages
- 129-134
- OpenAlex
- W2606613250
- Cited by
- 20
- References
- 29
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 3 · 2023: 4 · 2022: 1 · 2021: 1 · 2020: 4 · 2019: 2
- FWCI
- 1.24
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Designing Interactive Systems Conference (publishedVersion)
- Funders
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Keywords
- augmentative and alternative communication, eye-gaze input, amyotrophic lateral sclerosis, context awareness, scene understanding, object recognition, word prediction, user interface design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Let's Talk About X: Combining Image Recognition and Eye Gaze to Support Conversation for People with ALS. https://app.cochat.ai/papers/10.1145/3064663.3064762
- The paper
- Kane, S. K., & Morris, M. R. (2017). Let's Talk About X: Combining Image Recognition and Eye Gaze to Support Conversation for People with ALS. Proceedings of the 2017 Conference on Designing Interactive Systems. https://doi.org/10.1145/3064663.3064762
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.