Preferred Appearance of Captions Generated by Automatic Speech Recognition for Deaf and Hard-of-Hearing Viewers
Open accessAbstract
As the accuracy of Automatic Speech Recognition (ASR) nears human-level quality, it might become feasible as an accessibility tool for people who are Deaf and Hard of Hearing (DHH) to transcribe spoken language to text. We conducted a study using in-person laboratory methodologies, to investigate requirements and preferences for new ASR-based captioning services when used in a small group meeting context. The open-ended comments reveal an interesting dynamic between: caption readability (visibility of text) and occlusion (captions blocking the video contents). Our 105 DHH participants provided valuable feedback on a variety of caption-appearance parameters (strongly preferring familiar styles such as closed captions), and in this paper we start a discussion on how ASR captioning could be visually styled to improve text readability for DHH viewers.
AI snapshot
Written from the paper’s full text.
TL;DR
In an in-person study with 105 Deaf and Hard-of-Hearing participants, the authors surveyed preferences for how automatically generated (ASR) captions should look during small-group meetings, finding strong preferences for captions placed inside the video at the bottom, two lines of text, and Arial or Times New Roman fonts, while captioning type and appearance preferences were split. The authors recommend that developers of ASR captioning software give users extensive customization options.
Key findings
- All five caption-appearance questions had significant chi-square goodness-of-fit results: Q1 (type) χ²=55.333, df=4, p=2.766e-11; Q2 (appearance) χ²=68.829, df=3, p=7.604e-15; Q3 (location) χ²=165.24, df=4, p<2.2e-16; Q4 (number of lines) χ²=200.53, df=6, p<2.2e-16; Q5 (font) χ²=240.67, df=10, p<2.2e-16.
- Significantly preferred options: Q1 — Movie style (white text) and TV style (black box); Q2 — Movie subtitle style and TV CC style; Q3 — Inside the video (bottom); Q4 — 2 lines; Q5 — Arial and Times New Roman.
- Q1 and Q2 showed participants "almost divided" between TV CC Style and Movie Subtitle Style. The strong preference for captions inside the video (bottom) contrasted with Crabb et al.'s recommendation of outside-the-frame placement, and the 2-line preference contrasted with Kushalnagar et al.'s observed preference for 3+ lines; the authors speculate this reflects the small-group live-meeting context versus online video viewing.
- Thematic analysis of 116 open-ended comments (2069 words) surfaced themes of familiarity with specific styles (n=43), readability of captions (n=24), captions occluding video (n=21), and captions closer to speaker (n=6), revealing tension between readability and occlusion, plus a desire for customization.
Methods
- IRB-approved in-person controlled experiment at the authors' laboratory, part of a larger project on DHH users' use of ASR tools. Participants were recruited by e-mail and campus flyers, screened as identifying as Deaf or Hard of Hearing and using captions when viewing television, and paid $40 for a 60-minute session.
- 105 DHH participants (69 Deaf, 36 Hard-of-Hearing; 58 male, 47 female; ages 18–30, mean=22.105, median=22). Participants first viewed videos of ASR-captioned mock business meetings (average Word Error Rate 23.2%) to become familiar with ASR captioning, then answered five caption-appearance questions (Q1–Q5) presented with ASL videos, closed captions, and pictures of options, plus open-ended questions.
- Analysis: chi-square goodness-of-fit tests for each question, and a two-round consensus coding thematic analysis of open-ended comments conducted by the authors (two of whom are DHH).
Limitations
- Stated: the authors note they are conducting a second study with a larger pool of online participants to investigate their speculation about whether preferences differ between ASR use for small-group meetings versus online videos (e.g., YouTube); the placement and line-count differences from prior work are described as speculation.
- Evident from the text: participants were recruited on a university campus and ranged only from 18–30 years old, so older adults and non-campus populations are not represented; the goodness-of-fit tests establish that some options were preferred over others but do not provide pairwise comparisons among the preferred options; the abstract text provided is truncated and does not report results.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 17 cited sources checked · 0 supported · 0 flagged
- Findingown data
For Q1 (Captioning Type), the answer choices significantly preferred over the rest were Movie style (white text) and TV style (black box).
“Answer choices significantly preferred over the rest include: Q1 (Type): Movie style (white text) and TV style (black box)”
Quote in the paperSubstantial claim - Findingown data
For Q4 (Number of Lines), the answer choice significantly preferred over the rest was 2 lines.
“Q4 (Number of Lines): 2 lines”
Quote in the paperSubstantial claim - Findingown data
For Q2 (Caption Appearance), the answer choices significantly preferred over the rest were Movie subtitle style and TV CC style.
“Q2 (Appearance): Movie subtitle style and TV CC style”
Quote in the paperSubstantial claim - Findingown data
The thematic analysis of open-ended comments revealed a tension between caption readability and captions occluding the video frame.
“Our thematic analysis of the open-ended comments (see Sidebar C for some quotes from our participants) revealed a tension between caption readability and captions occluding the video frame (for example, the black box helps reading the text but hinders visibility of the video contents).”
Quote in the paperSubstantial claim - Findingown data
Both Q1 and Q2 revealed that participants were almost divided in their preference for traditional 'TV CC Style' or 'Movie Subtitle Style' captions.
“Both Q1 (Captioning Type) and Q2 (Caption Appearance) revealed that our participants were almost divided in their preference for traditional”
Quote in the paperNot a substantial claim
- Findingown data
Participants expressed a desire for familiarity in caption appearance (TV CC or Movie Subtitle) style.
“Participants expressed a desire for familiarity in caption appearance (TV CC or Movie Subtitle) style, yet they also expressed a desire for the ability to customize the captioning”
Quote in the paperNot a substantial claim - Findingown data
For Q3 (Location), the answer choice significantly preferred over the rest was Inside the video (bottom).
“Q3 (Location): Inside the video (bottom)”
Quote in the paperSubstantial claim - Findingown data
For Q5 (Font), the answer choices significantly preferred over the rest were Arial and Times New Roman.
“Q5 (Font): Arial and Times New Roman.”
Quote in the paperSubstantial claim - Interpretationown data
Participants' preference for 2 lines of captions contrasted with Kushalnagar et al. who observed a preference for 3 or more lines.
“In Q4 (Number of Lines) our participants strongly preferred to see 2 lines of captions, whereas Kushalnagar et al. [12] observed a preference for 3 or more lines.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [12]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationown data
Participants' preference for captions inside the video (bottom) contrasted with prior work from Crabb et al. which recommended captions be outside of the video frame for online videos.
“For Q3 (Location of Captions), our participants strongly preferred to see captions inside the video (bottom), which contrasted with prior work from Crabb et al. [3] which recommended the captions be outside of the video frame for online videos.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Classified as probably not a substantial claim. 25
- May combine two claims. 6
- The quote only partly states this claim. 1
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataParticipants' gender was 58 males and 47 females.
“A total of 105 DHH individuals participated, and they self-identified their hearing status as (69 Deaf, 36 Hard-of-Hearing), and gender as (58 males and 47 females).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataParticipants self-identified their hearing status as 69 Deaf and 36 Hard-of-Hearing.
“A total of 105 DHH individuals participated, and they self-identified their hearing status as (69 Deaf, 36 Hard-of-Hearing), and gender as (58 males and 47 females).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataParticipants' ages ranged from 18-30 years old (mean=22.105, median=22).
“Participants' ages ranged from 18-30 years old [mean=22.105,median=22].”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataAll five questions (Q1-Q5) had significant chi-square goodness-of-fit results.
“A chi-square test of goodness-of-fit was performed to determine whether some answer choices were preferred over others, and all questions had significant results: [Q1 𝜒 2 =55.333, df=4, p=2.766𝑒”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataA total of 105 DHH individuals participated in the study.
“A total of 105 DHH individuals participated, and they self-identified their hearing status as (69 Deaf, 36 Hard-of-Hearing), and gender as (58 males and 47 females).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe thematic analysis identified a theme of Captions Closer to Speaker (n=6).
“Captions Closer to Speaker (n=6)”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataAll questions (Q1-5) had participants who preferred each potential option, showing diverse perspectives in the DHH community.
“All of our questions (Q1-5) had participants whom preferred each potential option, which showed the diverse perspectives in the DHH community.”
Quote in the paperSubstantial claim- Rests on an assumption: The 105 participants are representative of the DHH community.
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe thematic analysis identified a theme of Readability of Captions (n=24).
“Readability of Captions (n=24)”
Quote in the paperNot a substantial claimMay combine two claims. The quote only partly states this claim.
Findingown dataSome users had a strong preference for 'traditional' captioning as seen on televisions, yet others preferred the movie subtitle style, and others wanted entirely novel combinations of options.
“Some users had a strong preference for”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe thematic analysis identified a theme of Captions Occluding Video (n=21).
“Captions Occluding Video (n=21)”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataParticipants expressed a desire for the ability to customize the captioning to suit the particular ambience and/or background of the video, to improve readability or to reduce occlusion.
“yet they also expressed a desire for the ability to customize the captioning, to suit the particular ambience and/or background of the video, to improve readability or to reduce occlusion.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationPeng et al. investigated placement of speech caption bubbles on Augmented Reality devices.
“Finally, Peng et al. [14] investigated placement of speech caption bubbles on Augmented Reality devices.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [14]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S4 (based).
InterpretationreasoningDHH users may have different preferences for the display of captions from ASR than when compared to captions from human transcribers.
“DHH users may have different preferences for the display of captions from ASR than when compared to captions from human transcribers.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationElliot et al. revealed in an exploratory survey of DHH users that they were agreeable to having ASR support their conversations at the workplace.
“Elliot et al. [4] revealed in an exploratory survey of DHH users that they were agreeable to having ASR support their conversations at the workplace.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [4]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationGower et al. looked into using speech pauses for automatic punctuation of ASR captions.
“Gower et al. [5] looked into using speech pauses for automatic punctuation of ASR captions.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationResearchers have considered whether ASR could provide text transcriptions of spoken language for DHH users.
“Researchers [15] have considered whether ASR could provide text transcriptions of spoken language for DHH users.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [15]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationCrabb et al. discussed the UX of subtitle position for online videos and made several recommendations on how to style the captions.
“Crabb et al. [3] discussed the UX of subtitle position for online videos and made several recommendations on how to style the captions (manually created by human transcriptionists).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationTracked captions were found by Kushalnagar et al. to enhance the perception of captions by DHH users in classrooms.
“Tracked captions (captioning projected above the speaker and following their movement) were found by Kushalnagar et al. [12] to enhance the perception of captions by DHH users in classrooms.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [12]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationDHH individuals who do not identify as culturally Deaf or older adults who have lost hearing later in life may prefer text-based accessibility tools rather than sign language interpretation.
“Furthermore, DHH individuals who do not identify as culturally Deaf or older adults who have lost hearing later in life may prefer text-based accessibility tools [7], rather than sign language interpretation.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [7]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationPeople who are Deaf and Hard of Hearing (DHH) use a wide variety of communication technologies and accommodations, such as real-time captioning services produced by professional transcriptionists or American Sign Language (ASL) interpreting.
“People who are Deaf and Hard of Hearing (DHH) make use of a wide variety of communication technologies and accommodations, e.g.real-time captioning services produced by professional transcriptionists (with text displayed on a screen for the user) or American Sign Language (ASL) interpreting [13].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [13]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
PremisecitationASR technologies are currently imperfect.
“Since it is known that ASR technologies are currently imperfect [1,4,10]”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [4]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationKawas et al. and Berke et al. discovered that the DHH community was receptive to ASR captioning of spoken information in classrooms and small group meetings.
“Likewise, Kawas et al. [10] and Berke et al. [1] discovered that the DHH community was receptive to ASR captioning of spoken information in classrooms and small group meetings.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [10]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationEye-tracking was used by Szarkowska et al. to monitor how DHH users read different styles of captioning.
“Eye-tracking was used by Szarkowska et al. [16] to monitor how DHH users read different styles of captioning (verbatim, standard, or edited captions).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationHong et al. looked at how captioning could be moved dynamically inside the video frame to improve the accessibility for DHH users.
“Hong et al. looked at how captioning could be moved dynamically inside the video frame to improve the accessibility for DHH users [6].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [6]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
Study design and conductnone givenParticipants were eligible if they answered 'yes' to both screening questions: Do you identify as Deaf or Hard of Hearing? Do you use captions when viewing television?
“Participants were eligible if they answered”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone givenThe authors (two are DHH) conducted a thematic analysis of 116 comments (2069 words) using a two-round consensus coding process.
“The authors (two are DHH) conducted a thematic analysis of 116 comments (2069 words) using a two-round consensus coding process.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataThe study was IRB-approved.
“We reached out to participants by e-mail and flyers on the university campus for this IRB-approved study.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
InterpretationreasoningCustomizing caption styling and appearance reduces the viewer's cognitive burden when reading captions.
“We strongly recommend programmers incorporating ASR captioning into their software empower their users with the ability to customize the styling and appearance of the captions as much as possible, in order to reduce the viewer's cognitive burden when reading the captions”
Quote in the paperSubstantial claim
The paper relies on this without support.
ASR is nearing human-level accuracy.
“Recent breakthroughs in Automatic Speech Recognition (ASR) wherein ASR is nearing humanlevel accuracy”
The paper relies on this without support.
DHH users might view live captions generated by ASR tools in small-group meetings with hearing colleagues.
“In small-group meetings with colleagues who are hearing, DHH users might view live captions generated by ASR tools, as seen in a prototype visualized in Figure 2.”
The paper relies on this without support.
Prior work has investigated a variety of caption/subtitle appearance options, but very few asked participants questions about how the captions should be styled.
“Prior work has investigated a variety of caption/subtitle appearance options, but very few asked participants questions about how the captions should be styled.”
The paper relies on this without support.
None of the prior work looked at the intersection of ASR and caption-appearance preferences for small-group meetings (in-person or remotely via technologies such as Skype) with DHH individuals.
“However, none of the prior work looked at the intersection of ASR and caption-appearance preferences for small-group meetings (in-person or remotely via technologies such as Skype) with DHH individuals.”
The paper relies on this without support.
The 105 participants are representative of the DHH community.
“which showed the diverse perspectives in the DHH community.”
Cites
Peng, Y.-H., Hsi, M.-W., Taele, P., Lin, T., Lai, P.-E., Hsu, L., Chen, T.-C., Wu, T.-Y., Chen, Y.-A., Tang, H.-H., et al. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173867 · Cited by 108 · see citing works
Hong, R., Wang, M., Xu, M., Yan, S., & Chua, T. (2010) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1873951.1874013 · Cited by 102 · see citing works
Technology Use Among Adults Who Are Deaf and Hard of Hearing: A National Survey
Maiorana-Basas, M., & Pagliaro, C. M. (2014) · The Journal of Deaf Studies and Deaf Education · doi:10.1093/deafed/enu005 · Cited by 100 · see citing works
Closed-Captioned Television Presentation Speed and Vocabulary
Jensema, C. J., McCann, R., & Ramsey, S. D. (1996) · American annals of the deaf · doi:10.1353/aad.2012.0377 · Cited by 83 · see citing works
Improving Real-Time Captioning Experiences for Deaf and Hard of Hearing Students
Kawas, S., Karalis, G., Wen, T., & Ladner, R. E. (2016) · Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/2982142.2982164 · Cited by 82 · see citing works
Szarkowska, A., Krejtz, I., Klyszejko, Z., & Wieczorek, A. (2011) · American annals of the deaf · doi:10.1353/aad.2011.0039 · Cited by 81 · see citing works
Berke, L., Caulfield, C., & Huenerfauth, M. (2017) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3132525.3132541 · Cited by 62 · see citing works
Self-citationKafle, S., & Huenerfauth, M. (2017) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3132525.3132542 · Cited by 60 · see citing works
Self-citationSubtitles on the moving image: an overview of eye tracking studies
Kruger, J., Szarkowska, A., & Krejtz, I. (2015) · Cited by 56 · see citing works
Berke, L., Kafle, S., & Huenerfauth, M. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173665 · Cited by 43 · see citing works
Self-citationCrabb, M., Jones, R., Armstrong, M., et al. (2015) · doi:10.1145/2700648.2809866 · Cited by 35 · see citing works
Tracked Speech-To-Text Display
Kushalnagar, R., Behm, G. W., Kelstone, A. W., & Ali, S. (2015) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2700648.2809843 · Cited by 34 · see citing works
Elliot, L. B., Stinson, M., Mallory, J., Easton, D., & Huenerfauth, M. (2016) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2982142.2982198 · Cited by 31 · see citing works
Self-citationLeveraging Pauses to Improve Video Captions
Gower, M., Shiver, B., Pandhi, C., & Trewin, S. M. (2018) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3234695.3241023 · Cited by 16 · see citing works
Prietch, S. S., de Souza, N. S., & Filgueiras, L. V. L. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07437-5_42 · Cited by 12 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Marylyn Howe and Bill Graham . 1990 . The importance of captioning for late-deafened adults . International Journal of Technology & Aging 3 , 2 (1990), 121 -- 131 . Marylyn Howe and Bill Graham. 1990. The importance of captioning for late-deafened adults. International Journal of Technology & Aging 3, 2 (1990), 121--131.
Cited by
Sorted by citations
McDonnell, E. J., Liu, P., Goodman, S. M., Kushalnagar, R., Froehlich, J. E., & Findlater, L. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479578 · Cited by 54 · see citing works
“, use 2 lines and common fonts [6]), their non-interactive nature does not allow for understanding how captions influence small-group social dynamics.”
BackgroundAlonzo, O., Shin, H. V., & Li, D. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3544808 · Cited by 40 · see citing works
“Prior work has examined the preferred appearances of captions among DHH adults, fnding great diversity in preferences towards various visual characteristics (e.g. font, background color) [4], and preferences towards the use of punctuation in automatically generated captions [20].”
BackgroundPataca, C. D. L., Watkins, M., Peiris, R. L., Lee, S., & Huenerfauth, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581511 · Cited by 38 · see citing works
“[9] saw a tension between caption styles that either favor legibility (e.”
BackgroundSelf-citationMcDonnell, E. J., Eagle, T., Sinlapanuntakul, P., Moon, S. H., Ringland, K. E., Froehlich, J. E., & Findlater, L. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642177 · Cited by 34 · see citing works
“…Prior HCI research on captioning design has identified strong preferences for online captioning styles that align with tele-vision standards (e.g., [13, 28, 60, 84]), but studying TikTok offers an opportunity to explore both how creative captioning practices organically emerge on a social media…”
BackgroundWang, Y., Li, Z., Chelladurai, P. K., Dannels, W. A., Oh, T. T., & Peiris, R. L. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581076 · Cited by 32 · see citing works
“Our study extended on existing literature on the visual captioning style preference in terms of comparing with the Haptic-Captioning modality [1, 3, 4].”
ResultMcDonnell, E. J., Moon, S. H., Jiang, L., Goodman, S. M., Kushalnagar, R., Froehlich, J. E., & Findlater, L. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580809 · Cited by 27 · see citing works
“, [5, 6, 18, 19, 23, 28, 48, 52, 56]), our participants’ agreed that reading captions requires too much cognitive load to make on-caption visualizations useful [6, 56].”
BackgroundIt is Okay to be Distracted: How Real-time Transcriptions Facilitate Online Meeting with Distraction
Son, S., Choi, J., Lee, S., Song, J. Y., & Shin, I. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580742 · Cited by 20 · see citing works
“To satisfy the preferences of DHH users, they suggested providing options to select the style of the captions desired by each person [12] or providing diferent types of captions according to the degree of hearing [55].”
MethodAmin, A. A., Hassan, S., Lee, S., & Huenerfauth, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517681 · Cited by 17 · see citing works
“[2, 6, 16, 21, 36, 46, 58, 60], there has been some research with DHH participants that has examined how their viewing experience is diminished when captions occlude other onscreen information [6, 10, 37, 62].”
BackgroundSelf-citationNevsky, A., Neate, T., Simperl, E. P. B., & Vatavu, R.-D. (2023) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3573381.3596159 · Cited by 18 · see citing works
Oomori, K., Shitara, A., Minagawa, T., et al. (2020) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3373625.3418032 · Cited by 17 · see citing works
“Following the literature [1], we used a movie-style, two-line, at the bottom captioning style method, which exhibited the best usability scores in the study.”
MethodFreire, A. P., Paiva, D. M. B., & Fortes, R. P. D. M. (2020) · Revista Brasileira de Informática na Educação · doi:10.5753/rbie.2020.28.0.956 · Cited by 18 · see citing works
“Apesar da melhoria na qualidade de traduções automáticas (Berke et al., 2019), ainda há diversos estudos recentes que mostram que ainda há diversas imprecisões nas legendas geradas automaticamente, que podem prejudicar a compreensão de pessoas com deficiência auditiva (Berke et al., 2018; Kafle &…”
BackgroundAmin, A. A., Hassan, S., & Huenerfauth, M. (2021) · International Web for All Conference (W4A) · doi:10.1145/3430263.3452429 · Cited by 16 · see citing works
“users were concerned about captions occluding graphical content in online videos [4], but this prior study did not specifcally focus on television content, nor live programming genres.”
BackgroundSelf-citationWhat Factors Motivate Culturally Deaf People to Want Assistive Technologies?
Vogel, A., & Korte, J. L. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650871 · Cited by 15 · see citing works
“Readability of displayed text can afect comprehension [3].”
Preferences of Deaf or Hard of Hearing Users for Live-TV Caption Appearance
Amin, A. A., Glasser, A., Kushalnagar, R., et al. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78095-1_15 · Cited by 13 · see citing works
Self-citationYamamoto, K., Suzuki, I., Shitara, A., et al. (2021) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3441852.3476551 · Cited by 12 · see citing works
“Previous studies have investigated the caption-appearance preferences [2] and designs based on the reliability of speech recognition [3].”
BackgroundTowards Inclusive Video Commenting: Introducing Signmaku for the Deaf and Hard-of-Hearing
Chen, S., Cheng, H., Situ, J., et al. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642287 · Cited by 11 · see citing works
“Beyond the accuracy of captions, the appearance of caption is also important to learners’ understanding [11].”
BackgroundToward Language Justice: Exploring Multilingual Captioning for Accessibility
Desai, A., Alharbi, R., Hsueh, S., Ladner, R. E., & Mankoff, J. C. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713622 · Cited by 13 · see citing works
Unspoken Sound: Identifying Trends in Non-Speech Audio Captioning on YouTube
May, L., Ohshiro, K., Dang, K., et al. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642162 · Cited by 10 · see citing works
“However, there is no clear consensus on the color and typeface of text, caption background color, and opacity, the number of lines of captioning, caption placement, as well as personal preference for genre-adaptive caption typeface and movement [8, 15].”
Modeling and Improving Text Stability in Live Captions
Liu, X., Zhang, J., Ferrer, L., et al. (2023) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585609 · Cited by 11 · see citing works
Comparing the Impact of Professional and Automatic Closed Captions on Video-Watching Experience
Kim, H., Tao, Y., Liu, C., Zhang, Y., & Li, Y. (2023) · CHI Extended Abstracts · doi:10.1145/3544549.3585634 · Cited by 12 · see citing works
“However, these studies have only focused on improving accessibility for deaf or hard-of-hearing individuals [2, 10, 18].”
BackgroundMcDonnell, E. J., & Findlater, L. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3675649 · Cited by 13 · see citing works
“For example, Berke et al. [20, 21] aimed to study how different algorithmic confidence visualizations could impact DHH people’s experiences of one-on-one automatically captioned conversations.”
BackgroundUnderstanding and Enhancing The Role of Speechreading in Online d/DHH Communication Accessibility
Desai, A., Mankoff, J. C., & Ladner, R. E. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580810 · Cited by 10 · see citing works
“When focused on captioning, research has worked to increase accuracy [24, 39], enhance caption display online [9, 36] and in augmented reality [29, 31, 51], support topic extraction [32], and improve richness by embedding emotions and punctuation in cap-tions [21, 50, 61].”
BackgroundUnderstanding How Deaf and Hard of Hearing Viewers Visually Explore Captioned Live TV News
Amin, A. A., Hassan, S., Lee, S., et al. (2023) · International Web for All Conference (W4A) · doi:10.1145/3587281.3587287 · Cited by 8 · see citing works
“While these metrics improve the quality caption transcriptions, several research have shown that captions blocking onscreen information may reduce DHH viewers’ ability to perceive vital information that appear on the screen [3, 4, 6, 9].”
BackgroundSelf-citationVRCaptions: Design Captions for DHH Users in Multiplayer Communication in VR
Xie, T., Zhang, X., Huang, F., Liu, D., An, P., & Je, S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714186 · Cited by 10 · see citing works
“Allowing users to adjust caption positions or adaptive background transparency could further enhance caption accessibility for DHH users [8, 63], all while preserving the immersive experience in VR multiplayer conversations.”
InfluentialLoaiza-Ramírez, I., & González-Beltrán, B. A. (2025) · Universal Access in the Information Society · doi:10.1007/s10209-025-01270-y · Cited by 8 · see citing works
Amin, A. A., Mendis, J., Kushalnagar, R., et al. (2023) · International Web for All Conference (W4A) · doi:10.1145/3587281.3587286 · Cited by 8 · see citing works
“In addition, prior research has revealed frustration among DHH viewers in seeking the current speaker among multiple speakers visible on the screen while watching live captioned TV programming [2, 10, 11, 22].”
BackgroundSelf-citationAccessibility of services in digital television for hearing impaired consumers
Samčović, A. B. (2020) · Assistive Technology · doi:10.1080/10400435.2020.1757786 · Cited by 8 · see citing works
Seita, M. (2020) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3375039 · Cited by 7 · see citing works
“We conducted an online survey study in which we asked about DHH participants’ preferences regarding the appearance of captions [2].”
BackgroundMethodSelf-citationMcDonnell, E. J. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3550417 · Cited by 7 · see citing works
Customization of Closed Captions via Large Language Models
Chavez, M. A., Thompson, B., Feanny, M., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-62849-8_7 · Cited by 6 · see citing works
Kim, J., & Hong, J.-H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713911 · Cited by 6 · see citing works
Modeling Accessibility: Characterizing What We Mean by “Accessible”
Mack, K., Martinez, J. J., Lewis, A., Mankoff, J. C., Fogarty, J. A., Findlater, L., Evans, H., Bennett, C. L., & McDonnell, E. J. (2025) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663547.3746344 · Cited by 11 · see citing works
“This example highlights how one model of accessibility does not necessarily characterize a person’s experience outside a single point in time; a person’s model can change drastically depending on the type of access barrier, their repertoire, and relevant contextual factors.”
Pataca, C. D. L., Ahn, S., Yoo, S., et al. (2025) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663547.3746400 · Cited by 5 · see citing works
“…boundaries of a shared cultural and geographic landscape these communities are not homogeneous—with individual dispositions playing a major role in shaping user preferences [7, 12, 26, 32, 46]—little is known about how cultural and linguistic factors could further influence these same preferences.”
Self-citationSharevski, F., Alonzo, O., & Hau, S. M. G. (2025) · International Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713114 · Cited by 6 · see citing works
“Research, for instance, has explored DHH viewers’ preferences for how closed captions should be styled (e.g. [ 10, 85]), how non-speech sounds and information should be included (e.g. [ 7, 23, 24, 58, 59]), how much error can be tolerated in automated captions [11], how to evaluate the quality of…”
Dementyev, A., Kanevsky, D., Yang, S., et al. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713631 · Cited by 3 · see citing works
“Research has shown various improvements to real-time ASR, such as ways to display transcription confidence [9] and customizing fonts appearance to improve readability [8].”
Amin, A. A., Hassan, S., Alm, C. O., & Huenerfauth, M. (2022) · Workshop on Language Technology for Equality, Diversity, Inclusion (LT-EDI) · doi:10.18653/v1/2022.ltedi-1.5 · Cited by 5 · see citing works
“alone, over 15% people are DHH, and regularly rely on captioning while watching videos to perceive salient auditory information (Berke et al., 2019).”
BackgroundSelf-citationAcceptance of AI-Based Meeting Tools: Psychological Safety as a Foundation for Smart CollaborationPreprint
Cardon, P. W., Fleischmann, C., Carradını, S., Getchell, K. M., Stapp, J., & Aritz, J. (2023) · doi:10.31235/osf.io/sgxyp · Cited by 3 · see citing works
Effect of Occlusion on Deaf and Hard of Hearing Users’ Perception of Captioned Video Quality
Amin, A. A., Hassan, S., & Huenerfauth, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78095-1_16 · Cited by 2 · see citing works
Self-citation"It's Not an Issue of Malice, but of Ignorance"
Davis, J. U., Wang, H., Chilana, P. K., & Yang, X.-D. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3610901 · Cited by 3 · see citing works
“Other techniques explore how users with DHH can effectively insert their ASL interpreter into the meeting [50] or allow human-in-the-loop approaches to improve ASR and captions [5] [6].”
BackgroundKang, J., Layton, E., Martin, D., & Starner, T. E. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650976 · Cited by 1 · see citing works
“Recently, we have seen ASR integrated into HWDs [3, 12, 13, 17, 19, 22], the classroom [4, 23] and video-conferencing [2, 16].”
BackgroundKuhn, K., Reuter, B., Egger, N., & Zimmermann, G. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3688495 · Cited by 1 · see citing works
“DHH individuals show diferent preferences for caption styles, highlighting a signifcant need for customisation options to increase accessibility for individual needs or specifc situations [3, 10, 37].”
McDonnell, E. J. (2023) · ACM SIGACCESS Accessibility and Computing · doi:10.1145/3584732.3584735 · Cited by 1 · see citing works
Yamamoto, K., Suzuki, I., Shitara, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11326 · Cited by 1 · see citing works
“Previous studies have investigated the caption-appearance preferences [2] and designs based on the reliability of speech recognition [3].”
BackgroundHeyko, D. (2021) · The Atrium (University of Guelph) · Cited by 1 · see citing works
An automated effective communication system in a VR based environment for hearing impaired
Tharun, S., Arunachalam, V., Bandaru, S. T., et al. (2023) · International Conference on Recent Trends in Advance Computing (ICRTAC) · doi:10.1109/icrtac59277.2023.10480761
“The desired position of captions generated by automatic speech recognition (ASR) systems on a screen is explored from the perspective of hearing-impaired and deaf viewers in the study [15].”
BackgroundAn Audio-Visual Speech Separation and Personalized Keyphrase Detection in Noisy Environments
Kumar, S., Tejas, B. N., Hithaish, et al. (2026) · Smart innovation, systems and technologies · doi:10.1007/978-981-96-8126-6_21
Iwamoto, Y., Raees, M., Oh, T., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29656-6_13
Ko, T., & Lee, K. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3817018
- Authors
- Berke, L., Albusays, K., Seita, M., & Huenerfauth, M.
- Affiliations
- Rochester Institute of Technology (US)
- Venue
- Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Pages
- 1-6
- OpenAlex
- W2943167538
- Semantic Scholar
- 29899205c4fe0b6371f6f529088e7e4e44064cdc
- Cited by
- 48
- References
- 15
- Citations by year
- 2026: 3 · 2025: 8 · 2024: 8 · 2023: 14 · 2022: 4 · 2021: 7 · 2020: 4
- FWCI
- 38.15
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA) (publishedVersion)
- ACM Proceedings
- Funders
- NSF (1746056)
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › Subtitles and Audiovisual Media
- Social Sciences › Psychology › Developmental and Educational Psychology › Hearing Impairment and Communication
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speech recognition, captions, deaf and hard of hearing, occlusion, closed captioning, accessibility, video captioning, group meetings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-20.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-20.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
