Let's Talk About X: Combining Image Recognition and Eye Gaze to Support Conversation for People with ALS
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Communicating at a natural speed is a significant challenge for users of augmentative and alternative communication (AAC) devices, especially when input is provided by eye gaze, as is common for people with ALS and similar conditions. One way to improve AAC throughput is by drawing on contextual information from the outside world. Toward this goal, we present SceneTalk, a prototype gaze-based AAC system that uses computer vision to identify objects in the user's field of view and suggests words and phrases related to the current scene. We conducted a formative evaluation of SceneTalk with six people with ALS, in which we evaluated their preference for user interface modes and output preferences. Participants agreed that integrating contextual awareness into their AAC device could be helpful across a diverse range of situations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Prior research about AAC users has often identifed challenges with specifc forms of communication, such as when talking to a physician or telling a long story [18]”
From “The less I type, the better”: How AI Language Models can Enhance or Impede Communication for AAC Users (2023) · cites this paper as [18]
In the citing paper
“The social timing pressures that exist for AAC device users make it hard to add nuance and talk about other things beyond requests or short utterances [18].”
From “The less I type, the better”: How AI Language Models can Enhance or Impede Communication for AAC Users (2023) · cites this paper as [18]
Citations
Cites
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298754 · Cited by 1,287
Beukelman, D. R., & Mirenda, P. (2006) · Medical Entomology and Zoology · Cited by 757
VizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866080 · Cited by 601 · cited 1×
“For example, if the user captured an image of a light switch, a crowd worker might suggest the phrase “Can you turn off the lights?” While crowd-based recognition is slower than automated recognition, crowd workers can be used to label images within minutes for a reasonable cost [6], which may still be faster than typ…”
BackgroundVizWiz: nearly real-time answers to visual questions
Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Tatarowicz, A., White, B., White, S., & Yeh, T. (2010) · Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806020 · Cited by 536
Twenty years of eye typing: systems and design issues
Majaranta, P., & Räihä, K.-J. (2002) · Proceedings of the symposium on Eye tracking research & applications - ETRA '02 · doi:10.1145/507072.507076 · Cited by 483 · cited 1×
“Using gaze for real-time communication can be challenging, as eye-typing typically enables users to communicate at 10 to 20 words per minute (wpm) [18], while spoken conversation occurs at about 180 wpm [28].”
Fast hands-free writing by gaze direction
Ward, D. J., & MacKay, D. J. C. (2002) · Nature · doi:10.1038/418838a · Cited by 329 · cited 1×
“Alternative text entry methods such as Dasher [24,25] may offer improved performance, but learning an alternative text entry method requires extensive practice.”
MethodFast gaze typing with an adjustable dwell time
Majaranta, P., Ahola, U.-K., & Špakov, O. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518758 · Cited by 261 · cited 1×
“Prior approaches to speeding up AAC input include reducing dwell time [17,20], integrating word and phrase predictions [16], and supporting dwell-free eye-typing [15,22].”
Method(2017) · doi:10.1145/2998181.2998364 · Cited by 258 · cited 1×
“Automatic image captioning has recently been used to provide alternative descriptions of images for blind users [27], but has not previously been used to support communication.”
MethodToward Everyday Gaze Input: Accuracy and Precision of Eye Tracking and Implications for Design
Feit, A. M., Williams, S., Toledo, A., Paradiso, A., Kulkarni, H., Kane, S., & Morris, M. R. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025599 · Cited by 234 · cited 1×
“However, participants sometimes encountered difficulties using our prototype device due to calibration issues [9], positioning, and fatigue.”
BackgroundSelf-citationFast Hands-free Writing by Gaze Direction
Ward, D. J., & Mackay, D. (2002) · PubMed · doi:10.48550/arxiv.cs/0204030 · Cited by 214 · cited 1×
“Alternative text entry methods such as Dasher [24,25] may offer improved performance, but learning an alternative text entry method requires extensive practice.”
MethodYorkston, K. M., & Beukelman, D. R. (1981) · Journal of Speech and Hearing Disorders · doi:10.1044/jshd.4603.296 · Cited by 202 · cited 1×
“Using gaze for real-time communication can be challenging, as eye-typing typically enables users to communicate at 10 to 20 words per minute (wpm) [18], while spoken conversation occurs at about 180 wpm [28].”
Communication Support for People with ALS
Beukelman, D., Fager, S., & Nordness, A. (2011) · Neurology Research International · doi:10.1155/2011/714693 · Cited by 177 · cited 3×
“Given the rapid changes in ability that can occur for people with ALS [4] and the challenges in using gaze-based AAC in certain environments due to ambient light and other factors [13], there may be some benefit in supporting multiple modes of interaction, or adapting the interaction mode to the user’s context, such a…”
BackgroundThe potential of dwell-free eye-typing for fast assistive gaze communication
Kristensson, P. O., & Vertanen, K. (2012) · Proceedings of the Symposium on Eye Tracking Research and Applications · doi:10.1145/2168556.2168605 · Cited by 142 · cited 1×
“However, the theoretical maximum speed of any keyboardbased eye-typing method has been estimated to be only 46 wpm [14], well below the rate of spoken conversation.”
BackgroundImproving Dwell-Based Gaze Typing with Dynamic, Cascading Dwell Times
Mott, M. E., Williams, S., Wobbrock, J. O., & Morris, M. R. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025517 · Cited by 123 · cited 1×
“Prior approaches to speeding up AAC input include reducing dwell time [17,20], integrating word and phrase predictions [16], and supporting dwell-free eye-typing [15,22].”
MethodSelf-citationEyeSwipe: Dwell-free Text Entry Using Gaze Paths
Kurauchi, A., Feng, W., Joshi, A., Morimoto, C., & Betke, M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858335 · Cited by 114 · cited 1×
“Prior approaches to speeding up AAC input include reducing dwell time [17,20], integrating word and phrase predictions [16], and supporting dwell-free eye-typing [15,22].”
MethodMurphy, J. (2004) · Augmentative and Alternative Communication · doi:10.1080/07434610400005663 · Cited by 93 · cited 1×
“Slow input speed is one of the most significant barriers to AAC adoption and use for people with ALS [21].”
BackgroundWhat we talk about: designing a context-aware communication tool for people with aphasia
Kane, S. K., Linam-Church, B., Althoff, K., & McCall, D. (2012) · Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/2384916.2384926 · Cited by 87 · cited 1×
“Marco Polo [7], TalkAbout [12], and GLAAC [26] offered predictions relevant to the user’s location.”
BackgroundSelf-citationEye typing using word and letter prediction and a fixation algorithm
MacKenzie, I. S., & Zhang, X. (2008) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/1344471.1344484 · Cited by 84 · cited 1×
“Prior approaches to speeding up AAC input include reducing dwell time [17,20], integrating word and phrase predictions [16], and supporting dwell-free eye-typing [15,22].”
MethodFilteryedping: Design Challenges and User Performance of Dwell-Free Eye Typing
Pedrosa, D. D. C., Pimentel, M. D. G. C., Wright, A., & Truong, K. N. (2015) · ACM Transactions on Accessible Computing · doi:10.1145/2724728 · Cited by 82 · cited 1×
“Prior approaches to speeding up AAC input include reducing dwell time [17,20], integrating word and phrase predictions [16], and supporting dwell-free eye-typing [15,22].”
MethodNow Dasher! Dash away!: longitudinal study of fast text entry by Eye Gaze
Tuisku, O., Majaranta, P., Isokoski, P., & Räihä, K.-J. (2008) · Proceedings of the 2008 symposium on Eye tracking research & applications - ETRA '08 · doi:10.1145/1344471.1344476 · Cited by 79 · cited 1×
“Alternative text entry methods such as Dasher [24,25] may offer improved performance, but learning an alternative text entry method requires extensive practice.”
MethodKane, S. K., Morris, M. R., Paradiso, A., & Campbell, J. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998284 · Cited by 50 · cited 2×
“Given the rapid changes in ability that can occur for people with ALS [4] and the challenges in using gaze-based AAC in certain environments due to ambient light and other factors [13], there may be some benefit in supporting multiple modes of interaction, or adapting the interaction mode to the user’s context, such a…”
BackgroundSelf-citationFiannaca, A., Paradiso, A., Shah, M., & Morris, M. R. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998215 · Cited by 49 · cited 1×
“AACrobat [10] introduced a secondary mobile device application that enabled conversation partners to suggest words and phrases as the AAC user typed.”
BackgroundSelf-citationIntroducing Microsoft Cognitive Services
Sole, A. D. (2017) · Apress eBooks · doi:10.1007/978-1-4842-3342-9_1 · Cited by 47
The design and field evaluation of PhotoTalk: a digital image communication application for people
Allen, M., McGrenere, J., & Purves, B. (2007) · Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1296843.1296876 · Cited by 46 · cited 1×
“PhotoTalk [1] enabled individuals with aphasia to communicate by sharing photos rather than writing or speaking text.”
BackgroundThe effects of word prediction on communication rate for AAC
Trnka, K., Yarrington, D., McCaw, J., McCoy, K. F., & Pennington, C. (2007) · Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers on XX - NAACL '07 · doi:10.3115/1614108.1614152 · Cited by 41 · cited 2×
“A common approach to improving AAC speed is to predict likely words using a language model and to suggest them as the user types [23].”
BackgroundDesigning Conversation Cues on a Head-Worn Display to Support Persons with Aphasia
Williams, K., Moffatt, K., McCall, D., & Findlater, L. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702484 · Cited by 36 · cited 1×
“Marco Polo [7], TalkAbout [12], and GLAAC [26] offered predictions relevant to the user’s location.”
BackgroundIrani, L. C. (2017) · Wiley eBooks · doi:10.1002/9781405165518.wbeos0994 · Cited by 32
The effect of context priming and task type on augmentative communication performance
Higginbotham, D. J., Bisantz, A. M., Sunm, M., Adams, K., & Yik, F. (2009) · Augmentative and Alternative Communication · doi:10.1080/07434610802131869 · Cited by 30 · cited 1×
“[11] found that including task-specific predictions in an AAC reduced the number of key presses needed to discuss that subject.”
BackgroundTowards providing just-in-time vocabulary support for assistive and augmentative communication
Epp, C. D., Djordjevic, J., Wu, S., Moffatt, K., & Baecker, R. M. (2012) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/2166966.2166973 · Cited by 30 · cited 1×
“Marco Polo [7], TalkAbout [12], and GLAAC [26] offered predictions relevant to the user’s location.”
BackgroundKawakami, R. (2018) · The Journal of The Institute of Image Information and Television Engineers · doi:10.3169/itej.72.423 · Cited by 22
Research works printed but not linked: 1
- ALS Association . Epidemiology of ALS and Suspected Clusters. ALSA.org . Retrieved May 25, 2016 from http://www.alsa.org/alscare/resources/publicationsvideos/factsheets/epidemiology.html ALS Association. Epidemiology of ALS and Suspected Clusters. ALSA.org. Retrieved May 25, 2016 from http://www.alsa.org/alscare/resources/publicationsvideos/factsheets/epidemiology.html
Cited by
Valencia, S., Cave, R., Kallarackal, K., Seaver, K., Terry, M., & Kane, S. K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581560 · Cited by 97
“Prior research about AAC users has often identifed challenges with specifc forms of communication, such as when talking to a physician or telling a long story [18], and that AAC users often conduct extensive work before a meeting to prepare what they wish to communicate [20].”
BackgroundInfluentialSelf-citationState of the Art in AAC: A Systematic Review and Taxonomy
Curtis, H., Neate, T., & Gonzalez, C. V. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3544810 · Cited by 60
“Knowledge of the context i.e., environment, location or communication partner have been harnessed in high-tech AAC systems for: natural language generation (NLG) [152], synthesised vocabulary searches, discourse prediction, improved adaptation to topics and to capture experiences [90, 91].”
BackgroundKane, S. K., Guo, A., & Morris, M. R. (2020) · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3373625.3416990 · Cited by 35
“Related paradigms like context-aware computing [18] expand the vision of how sensing infrastructures can support myriad implicit and explicit interactions; context-awareness has shown promise for supporting accessibility applications such as predictive text suggestions for AAC systems [36, 37].”
BackgroundSelf-citationConversational Agency in Augmentative and Alternative Communication
Valencia, S., Pavel, A., Santa Maria, J., Yu, S. (., Bigham, J. P., & Admoni, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376376 · Cited by 31
“As a first step towards making communication possible and efficient, designers and researchers of AAC have focused on improving device throughput, for example via text prediction [34], and vocabulary sorting [2, 21], and improving Figure 1.”
BackgroundWho Are You Asking?: Qualitative Methods for Involving AAC Users as Primary Research Participants
Beneteau, E. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376883 · Cited by 28
“At times, studies incorporate two or more of these methods [11,29,45].”
Background
Metadata
- Authors
- Kane, S. K., & Morris, M. R.
- Affiliations
- University of Colorado Boulder (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the 2017 Conference on Designing Interactive Systems
- Publisher
- ACM
- Published
- 2017-06-10
- Pages
- 129-134
- OpenAlex
- W2606613250
- Cited by
- 20
- References
- 29
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 3 · 2023: 4 · 2022: 1 · 2021: 1 · 2020: 4 · 2019: 2
- FWCI
- 1.24
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Designing Interactive Systems Conference (publishedVersion)
- Funders
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Keywords
- augmentative and alternative communication, eye-gaze input, amyotrophic lateral sclerosis, context awareness, scene understanding, object recognition, word prediction, user interface design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.