Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing
Abstract
Persons with hearing loss use visual signals such as gestures and lip movement to interpret speech. While hearing aids and cochlear implants can improve sound recognition, they generally do not help the wearer localize sound necessary to leverage these visual cues. In this paper, we design and evaluate visualizations for spatially locating sound on a head-mounted display (HMD). To investigate this design space, we developed eight high-level visual sound feedback dimensions. For each dimension, we created 3-12 example visualizations and evaluated these as a design probe with 24 deaf and hard of hearing participants (Study 1). We then implemented a real-time proof-of-concept HMD prototype and solicited feedback from 4 new participants (Study 2). Study 1 findings reaffirm past work on challenges faced by persons with hearing loss in group conversations, provide support for the general idea of sound awareness visualizations on HMDs, and reveal preferences for specific design options. Although preliminary, Study 2 further contextualizes the design probe and uncovers directions for future work.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Providing both graphical and textual forms of data has been shown to cater to different user preferences and needs, enhancing accessibility and usability.
“In the web and data visualization contexts, providing both graphical and textual forms of data, as well as enabling natural language querying of visualizations, has been shown to cater to different user preferences and needs, enhancing accessibility and usability [27].”
From Accessible IoT Dashboard Design with AI-Enhanced Descriptions for Visually Impaired Users (2025) · cites this paper as [27]
The gender of speakers, sound types, loudness, and sound directions are displayed as indicator icons in sound visualization systems for the hearing impaired.
“For example, the gender of speakers, sound types, loudness, sound directions are displayed as indicator icons [252], specifically designed for the hearing impaired;”
From Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things (2021) · cites this paper as [252]
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414 · see citing works
A Survey of Augmented Reality Technologies, Applications and Limitations
van Krevelen, D. W. F., & Poelman, R. (2010) · International Journal of Virtual Reality · doi:10.20870/ijvr.2010.9.2.2767 · Cited by 1,662 · see citing works
Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR
Zhou, F., Duh, H. B., & Billinghurst, M. (2008) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar.2008.4637362 · Cited by 1,165 · see citing works
Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research
Hruschka, D. J., Schwartz, D., St.John, D. C., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W. (2004) · Field Methods · doi:10.1177/1525822x04266540 · Cited by 829 · see citing works
Development of the Communication Profile for the Hearing Impaired
Demorest, M. E., & Erdman, S. A. (1987) · Journal of Speech and Hearing Disorders · doi:10.1044/jshd.5202.129 · Cited by 235 · see citing works
Mynatt, E. D., Back, M., Want, R., Baer, M. B., & Ellis, J. B. (1998) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/274644.274720 · Cited by 218 · see citing works
Horizontal localization with bilateral hearing aids: Without is better than with
den Bogaert, T. V., Klasen, T. J., Moonen, M., et al. (2006) · The Journal of the Acoustical Society of America · doi:10.1121/1.2139653 · Cited by 202 · see citing works
How Hearing Aids, Background Noise, and Visual Cues Influence Objective Listening Effort
Picou, E. M., Ricketts, T., & Hornsby, B. W. Y. (2013) · Ear and Hearing · doi:10.1097/aud.0b013e31827f0431 · Cited by 143 · see citing works
Augmented reality vs. street views
Medenica, Ž., Kun, A. L., Paek, T., & Palinko, O. (2011) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/2037373.2037414 · Cited by 128 · see citing works
Scale Composition and Item Analysis of the Communication Profile for the Hearing Impaired
Demorest, M. E., & Erdman, S. A. (1986) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.2904.535 · Cited by 113 · see citing works
Potential Benefits and Limitations of Three Types of Directional Processing in Hearing Aids
Picou, E. M., Aspell, E., & Ricketts, T. A. (2014) · Ear and Hearing · doi:10.1097/aud.0000000000000004 · Cited by 91 · see citing works
Ho-Ching, F. W.-L., Mankoff, J. C., & Landay, J. A. (2003) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/642611.642641 · Cited by 73 · see citing works
Plane-Wave Decomposition of Acoustical Scenes Via Spherical and Cylindrical Microphone Arrays
Zotkin, D. N., Duraiswami, R., & Gumerov, N. A. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2009.2022000 · Cited by 67 · see citing works
Self-citationMcCreery, R. W., Venediktov, R. A., Coleman, J. J., & Leech, H. M. (2012) · American Journal of Audiology · doi:10.1044/1059-0889(2012/12-0014 · Cited by 63 · see citing works
Visualizing non-speech sounds for the deaf
Matthews, T. L., Fong, J., & Mankoff, J. C. (2005) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1090785.1090797 · Cited by 62 · see citing works
Evaluating non-speech sound visualizations for the deaf
Matthews, T. L., Fong, J., Ho-Ching, F. W.-L., & Mankoff, J. C. (2006) · Behaviour and Information Technology · doi:10.1080/01449290600636488 · Cited by 60 · see citing works
Semi-supervised learning helps in sound event classification
Zhang, Z., & Schuller, B. W. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6287884 · Cited by 60 · see citing works
Spatial sound localization in an augmented reality environment
Sodnik, J., Tomažič, S., Grasset, R., et al. (2006) · doi:10.1145/1228175.1228197 · Cited by 59 · see citing works
Grieco‐Calub, T. M., & Litovsky, R. Y. (2012) · Ear and Hearing · doi:10.1097/aud.0b013e31824c7801 · Cited by 53 · see citing works
Captioning for Deaf and Hard of Hearing People by Editing Automatic Speech Recognition in Real Time
Wald, M. (2006) · Lecture notes in computer science · doi:10.1007/11788713_100 · Cited by 42 · see citing works
Automatic sound recognition for the hearing impaired
Yoo, I.-C., & Yook, D. (2008) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2008.4711269 · Cited by 42 · see citing works
Conversation tactics in persons with normal hearing and hearing-impairment
Hallam, R., & Corney, R. (2013) · International Journal of Audiology · doi:10.3109/14992027.2013.852256 · Cited by 32 · see citing works
The Attentive Hearing Aid: Eye Selection of Auditory Sources for Hearing Impaired Users
Hart, J., Onceanu, D., Sohn, C., et al. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-03655-2_4 · Cited by 31 · see citing works
Tactual display of consonant voicing as a supplement to lipreading
Yuan, H., Reed, C. M., & Durlach, N. I. (2005) · The Journal of the Acoustical Society of America · doi:10.1121/1.1945787 · Cited by 28 · see citing works
Head mounted displays and deaf children
Jones, M. D., Lawler, M., Hintz, E. G., et al. (2014) · ACM Conference on Interaction Design and Children (IDC) · doi:10.1145/2593968.2610481 · Cited by 27 · see citing works
Mueller, M. F., Meisenbacher, K., Lai, W., et al. (2013) · Cochlear Implants International · doi:10.1179/1754762813y.0000000040 · Cited by 27 · see citing works
Reported Use of Communication Strategies by SHHH Members
Tye‐Murray, N., Purdy, S. C., & Woodworth, G. G. (1992) · Journal of Speech Language and Hearing Research · doi:10.1044/jshr.3503.708 · Cited by 24 · see citing works
A Wearable Multichannel Tactile Display of Voice Fundamental Frequency
Young, E., Boothroyd, A., & Redmond, C. (1988) · Ear and Hearing · doi:10.1097/00003446-198812000-00011 · Cited by 19 · see citing works
A Comparison of a New Prototype Tickle Talker™ with the Tactaid 7
Galvin, K. L., Ginis, J., Cowan, R., Blamey, P. J., & Clark, G. M. (2001) · Australian and New Zealand Journal of Audiology · doi:10.1375/audi.23.1.18.31095 · Cited by 14 · see citing works
Kaneko, Y., Chung, I., & Suzuki, K. (2013) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2013.608 · Cited by 13 · see citing works
Design Implications for a Ubiquitous Ambient Sound Display for the Deaf.
Tomitsch, M., & Grechenig, T. (2007) · Cited by 8 · see citing works
Real-time captioning with the crowd
Lasecki, W. S., & Bigham, J. P. (2014) · interactions · doi:10.1145/2594459 · Cited by 8 · see citing works
Chain architecture: An efficient hardware solution for a large microphone array system
Zotkin, D. N., Adelman, R. N., O’Donovan, A. E., et al. (2013) · The Journal of the Acoustical Society of America · doi:10.1121/1.4805469 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
The AT Effect: How Disability Affects the Perceived Social Acceptability of Head-Mounted Display Use
Profita, H., Albaghli, R., Findlater, L., Jaeger, P., & Kane, S. K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858130 · Cited by 131 · see citing works
Self-citationToward fairness in AI for people with disabilities SBG@a research roadmap
Guo, A., Kamar, E., Vaughan, J. W., Wallach, H., & Morris, M. R. (2020) · ACM SIGACCESS Accessibility and Computing · doi:10.1145/3386296.3386298 · Cited by 128 · see citing works
McDonnell, E. J., Liu, P., Goodman, S. M., Kushalnagar, R., Froehlich, J. E., & Findlater, L. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479578 · Cited by 54 · see citing works
Self-citationUncovering Practical Security and Privacy Threats for Connected Glasses with Embedded Video Cameras
Opaschi, O., & Vatavu, R.-D. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3432700 · Cited by 10 · see citing works
Accessible IoT Dashboard Design with AI-Enhanced Descriptions for Visually Impaired Users
Stelea, G. A., Sangeorzan, L., & Enache-David, N. (2025) · Future Internet · doi:10.3390/fi17070274 · Cited by 5 · see citing works
- Authors
- Jain, D., Findlater, L., Gilkeson, J., Holland, B., Duraiswami, R., Zotkin, D., Vogler, C., & Froehlich, J. E.
- Affiliations
- Massachusetts Institute of Technology (US)
- University of Maryland, College Park (US)
- Montgomery Blair High School, College Park, MD, USA
- University of Maryland, College Park, College Park, MD, USA
- Gallaudet University (US)
- Venue
- Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2015-04-18
- Pages
- 241-250
- OpenAlex
- W2161071682
- Cited by
- 123
- References
- 33
- Citations by year
- 2026: 10 · 2025: 16 · 2024: 14 · 2023: 11 · 2022: 13 · 2021: 12 · 2020: 16 · 2019: 9 · 2018: 8 · 2017: 8 · 2016: 6
- FWCI
- 12.22
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Social Sciences › Psychology › Developmental and Educational Psychology › Hearing Impairment and Communication
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Keywords
- head-mounted displays, sound awareness, deaf and hard of hearing, hearing loss, sound localization, visualization design, lip reading, group conversation, assistive technology, design probes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
