CoChat
Sign Up
Cited by 30Tactile and Sensory InteractionsAssistive Technology in Communication and MobilityInteractive and Immersive Displays

Autonomous is Not Enough: Designing Multisensory Mid-Air Gestures for Vehicle Interactions Among People with Visual Impairments

Open access
Fink, Paul D. S., Dimitrov, Velin, Yasuda, Hiroshi, Chen, Tiffany L., Corey, Richard R., Giudice, Nicholas A., Sumner, Emily S.·2023·Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3544548.3580762
Open this paper in CoChat — free accountOpen PDF

Abstract

Should fully autonomous vehicles (FAVs) be designed inclusively and accessibly, independence will be transformed for millions of people experiencing transportation-limiting disabilities worldwide. Although FAVs hold promise to improve efficient transportation without intervention, a truly accessible experience must enable user input, for all people, in many driving scenarios (e.g., to alter a route or pull over during an emergency). Therefore, this paper explores desires for control in FAVs among (n=23) people who are blind and visually impaired. Results indicate strong support for control across a battery of driving tasks, as well as the need for multimodal information. These findings inspired the design and evaluation of a novel multisensory interface leveraging mid-air gestures, audio, and haptics. All participants successfully navigated driving scenarios using our gestural-audio interface, reporting high ease-of-use. Contributions include the first inclusively designed gesture set for FAV control and insight regarding supplemental haptic and audio cues.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3544548.3580762

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Perceptual cues often outperform verbal instructions for layout and orientation tasks.

    “This echoes evidence that perceptual cues often outperform verbal instructions for layout and orientation tasks [1, 10].”

    From Get In Touch with Your Seat: Accessible Seat Localization for Blind and Low Vision Travelers in Autonomous Shuttles (2026) · cites this paper as [10]

    Section 5.1 reports that spatialized audio improves environmental learning and spatial memory and reduces cognitive load compared with nonspatialized auditory descriptions. That supports a narrower comparison in nonvisual navigation, but not the broad claim that perceptual cues often outperform verbal instructions across layout and orientation tasks. The paper’s own Study 3 Results (section 5.3) found no statistically significant navigation-time benefit from adding haptics to gestural audio. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The majority of accessibility focused BLV projects have primarily concerned the in-vehicle stage of AV travel.

    “Although the majority of accessibility focused BLV projects have primarily concerned the in-vehicle stage of AV travel [2, 3, 7, 10, 12, 16, 17]”

    From Get In Touch with Your Seat: Accessible Seat Localization for Blind and Low Vision Travelers in Autonomous Shuttles (2026) · cites this paper as [10]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. Roger Bennett, Rohini Vijaygopal, and Rita Kottasz. 2020. Willingness of people who are blind to accept autonomous vehicles: An empirical investigation. Trans- portation Research Part F: Trafc Psychology and Behaviour 69 (Feb. 2020), 13-27. https://doi.org/10.1016/j.trf.2019.12.012 · DOI
  2. Rupert RA Bourne, Jaimie Adelson, Seth Flaxman, Paul Briant, Michele Bottone, Theo Vos, Kovin Naidoo, Tasanee Braithwaite, Maria Cicinelli, Jost Jonas, et al. 2020. Global Prevalence of Blindness and Distance and Near Vision Impairment in 2020: progress towards the Vision 2020 targets and what the future holds. Investigative Ophthalmology & Visual Science 61, 7 (2020), 2317-2317.
  3. Robin N Brewer and Vaishnav Kameswaran. 2018. Understanding the power of control in autonomous vehicles for people with vision impairment. In Pro- ceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility. 185-197.
  4. Julian Brinkley. 2021. Using personas with visual impairments to explore the design of an accessible self-driving vehicle human-machine interface. In Proceed- ings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 65. SAGE Publications Sage CA: Los Angeles, CA, 337-341.
  5. Julian Brinkley, Earl W Huf Jr, Briana Posadas, Julia Woodward, Shaundra B Daily, and Juan E Gilbert. 2020. Exploring the needs, preferences, and con- cerns of persons with visual impairments regarding autonomous vehicles. ACM Transactions on Accessible Computing (TACCESS) 13, 1 (2020), 1-34.
  6. Julian Brinkley, Brianna Posadas, Imani Sherman, Shaundra B Daily, and Juan E Gilbert. 2019. An open road evaluation of a self-driving vehicle human-machine interface designed for visually impaired users. International Journal of Human- Computer Interaction 35, 11 (2019), 1018-1032.
  7. Eddie Brown, David R. Large, Hannah Limerick, and Gary Burnett. 2020. Ultra- hapticons:"Haptifying" drivers' mental models to transform automotive mid-air haptic gesture infotainment interfaces. In 12th International Conference on Auto- motive User Interfaces and Interactive Vehicular Applications. 54-57.
  8. Henrik Detjen, Stefan Geisler, and Stefan Schneegass. 2020. Maneuver-based control interventions during automated driving: Comparing touch, voice, and mid-air gestures as input modalities. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 3268-3274.
  9. Brad E. Dicianno, Sivashankar Sivakanthan, S. Andrea Sundaram, Shantanu Satpute, Hailee Kulich, Elizabeth Powers, Nikitha Deepak, Rebecca Russell, Rose- marie Cooper, and Rory A. Cooper. 2021. Systematic review: Automated vehicles and services for people with disabilities. Neuroscience Letters 761 (Sept. 2021), 136103. https://doi.org/10.1016/j.neulet.2021.136103 · DOI
  10. Paul DS Fink, Jessica A Holz, and Nicholas A Giudice. 2021. Fully autonomous vehicles for people with visual impairment: Policy, accessibility, and future di- rections. ACM Transactions on Accessible Computing (TACCESS) 14, 3 (2021), 1-17.
  11. Nicholas A. Giudice, Benjamin A. Guenther, Toni M. Kaplan, Shane M. Anderson, Robert J. Knuesel, and Joseph F. Ciof. 2020. Use of an Indoor Navigation System by Sighted and Blind Travelers: Performance Similarities across Visual Status and Age. ACM Trans. Access. Comput. 13, 3, Article 11 (aug 2020), 27 pages. https://doi.org/10.1145/3407191 · DOI
  12. Nicholas A. Giudice and Jerome D. Tietz. 2008. Learning with Virtual Verbal Displays: Efects of Interface Fidelity on Cognitive Map Development. In Spatial Cognition VI. Learning, Reasoning, and Talking about Space, Christian Freksa, Nora S. Newcombe, Peter Gärdenfors, and Stefan Wölf (Eds.). Vol. 5248. Springer Berlin Heidelberg, Berlin, Heidelberg, 121-137. https://doi.org/10.1007/978- 3-540-87601-4_11 ISSN: 0302-9743, 1611-3349 Series Title: Lecture Notes in Computer Science. · DOI
  13. Nicholas A. Giudice, William E. Whalen, Timothy H. Riehle, Shane M. Ander- son, and Stacy A. Doore. 2019. Evaluation of an Accessible, Real-Time, and Infrastructure-Free Indoor Navigation System by Users Who Are Blind in the Mall of America. Journal of Visual Impairment & Blindness 113, 2 (March 2019), 140-155. https://doi.org/10.1177/0145482X19840918 Publisher: SAGE Publica- tions Inc. · DOI
  14. Mikaylah Gross, Joe Dara, Christopher Meyer, and Davide Bolchini. 2018. Explor- ing Aural Navigation by Screenless Access. In Proceedings of the 15th International Web for All Conference (W4A '18). Association for Computing Machinery, New York, NY, USA, 1-10. https://doi.org/10.1145/3192714.3192815 · DOI
  15. Kyle Harrington, David R. Large, Gary Burnett, and Orestis Georgiou. 2018. Ex- ploring the Use of Mid-Air Ultrasonic Feedback to Enhance Automotive User Interfaces. In Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (Toronto, ON, Canada) (Automo- tiveUI '18). Association for Computing Machinery, New York, NY, USA, 11-20. https://doi.org/10.1145/3239060.3239089 · DOI
  16. Kibum Kim, Xiangshi Ren, Seungmoon Choi, and Hong Z Tan. 2016. Assisting people with visual impairments in aiming at a target on a large wall-mounted display. International Journal of Human-Computer Studies 86 (2016), 109-120.
  17. Roberta L. Klatzky, James R. Marston, Nicholas A. Giudice, Reginald G. Golledge, and Jack M. Loomis. 2006. Cognitive load of navigating without vision when guided by virtual sound versus spatial language. Journal of Experimental Psychology. Applied 12, 4 (Dec. 2006), 223-232. https://doi.org/10.1037/1076- 898X.12.4.223 · DOI
  18. David R Large, Kyle Harrington, Gary Burnett, and Orestis Georgiou. 2019. Feel the noise: Mid-air ultrasound haptics as a novel human-vehicle interaction para- digm. Applied ergonomics 81 (2019), 102909. CHI '23, April 23-28, 2023, Hamburg, Germany
  19. Mathieu Nancel, Julie Wagner, Emmanuel Pietriga, Olivier Chapuis, and Wendy Mackay. 2011. Mid-Air Pan-and-Zoom on Wall-Sized Displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI '11). Association for Computing Machinery, New York, NY, USA, 177-186. https://doi.org/10.1145/1978942.1978969 · DOI
  20. Hari P. Palani, Paul D. S. Fink, and Nicholas A. Giudice. 2020. Design Guidelines for Schematizing and Rendering Haptically Perceivable Graphical Elements on Touchscreen Devices. International Journal of Human-Computer Interaction (April 2020), 1-22. https://doi.org/10.1080/10447318.2020.1752464 · DOI · Paper page
  21. Hari Prasath Palani, Paul D. S. Fink, and Nicholas A. Giudice. 2022. Compar- ing Map Learning between Touchscreen-Based Visual and Haptic Displays: A Behavioral Evaluation with Blind and Sighted Users. Multimodal Technologies and Interaction 6, 1 (Jan. 2022), 1. https://doi.org/10.3390/mti6010001 Number: 1 Publisher: Multidisciplinary Digital Publishing Institute. · DOI · Paper page
  22. Xiaosong Qian, Wendy Ju, and David Michael Sirkin. 2020. Aladdin's magic carpet: Navigation by in-air static hand gesture in autonomous vehicles. International Journal of Human-Computer Interaction 36, 20 (2020), 1912-1927.
  23. Parivash Ranjbar, Pournami Krishnan Krishnakumari, Jonas Andersson, and Maria Klingegård. 2022. Vibrotactile guidance for trips with autonomous vehicles for persons with blindness, deafblindness, and deafness. Transportation Research Interdisciplinary Perspectives 15 (2022), 100630.
  24. SAE. 2019. SAE Demo Days Survey. SAE. https://www.sae.org/news/press- room/2019/11/sae-international-survey-shows-public-enthusiasm-about-self- driving-cars
  25. Gözel Shakeri, John H Williamson, and Stephen Brewster. 2018. May the force be with you: Ultrasound haptic feedback for mid-air gesture interaction in cars. In Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. 1-10.
  26. Surak Son, YiNa Jeong, and Byungkwan Lee. 2019. An Audifcation and Visualiza- tion System (AVS) of an Autonomous Vehicle for Blind and Deaf People Based on Deep Learning. Sensors 19, 22 (Jan. 2019), 5035. https://doi.org/10.3390/s19225035 Number: 22 Publisher: Multidisciplinary Digital Publishing Institute. · DOI
  27. Robert Walter, Gilles Bailly, and Jörg Müller. 2013. StrikeAPose: Revealing Mid- Air Gestures on Public Displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI '13). Association for Computing Machinery, New York, NY, USA, 841-850. https://doi.org/10.1145/ 2470654.2470774 · DOI
  28. Gareth Young, Hamish Milne, Daniel Grifths, Elliot Padfeld, Robert Blenkinsopp, and Orestis Georgiou. 2020. Designing Mid-Air Haptic Gesture Controlled User Interfaces for Cars. Proceedings of the ACM on Human-Computer Interaction 4, EICS (June 2020), 81:1-81:23. https://doi.org/10.1145/3397869 · DOI

Cited by

Sorted by citations

Authors
Fink, P. D. S., Dimitrov, V., Yasuda, H., Chen, T. L., Corey, R. R., Giudice, N. A., & Sumner, E. S.
Affiliations
  • University of Maine (US)
  • Toyota Research Institute (US)
ORCID
Venue
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2023-04-19
Pages
1-13
DOI
10.1145/3544548.3580762
OpenAlex
W4366548252
Cited by
30
References
3
Citations by year
2026: 6 · 2025: 12 · 2024: 10 · 2023: 2
FWCI
15.44
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
CHI Conference on Human Factors in Computing Systems (publishedVersion)
Funders
National Science Foundation (CHS-1910603)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
  • Health Sciences › Health Professions › Occupational Therapy › Assistive Technology in Communication and Mobility
  • Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
Keywords
fully automated vehicles, visual impairment, blind users, mid-air gestures, multisensory interfaces, haptic feedback, audio cues, accessibility, inclusive design, user control, multimodal data, vehicle interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review