CoChat
Sign Up
Cited by 32Speech Recognition and SynthesisSpeech and Audio ProcessingHearing Impairment and Communication

Behavioral Changes in Speakers who are Automatically Captioned in Meetings with Deaf or Hard-of-Hearing Peers

Open access
Seita, Matthew, Albusays, Khaled, Kafle, Sushant, Stinson, Michael, Huenerfauth, Matt·2018·Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility
DOI 10.1145/3234695.3236355
Open this paper in CoChat — free accountOpen PDF

Abstract

Deaf and hard of hearing (DHH) individuals face barriers to communication in small-group meetings with hearing peers; we examine generation of captions on mobile devices by automatic speech recognition (ASR). While ASR output displays errors, we study whether such tools benefit users and influence conversational behaviors. An experiment was conducted where DHH and hearing individuals collaborated in discussions in three conditions (without an ASR-based application, with the application, and with a version indicating words for which the ASR has low confidence). An analysis of audio recordings, from each participant across conditions, revealed significant differences in speech features. When using the ASR-based automatic captioning application, hearing individuals spoke more loudly, with improved voice quality (harmonics-to-noise ratio), with a non-standard articulation (changes in F1 and F2 formants), and at a faster rate. Identifying non-standard speech in this setting has implications on the composition of data used for ASR training/testing, which should be representative of its usage context. Understanding these behavioral influences may also enable designers of ASR captioning systems to leverage these effects, to promote communication success.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3234695.3236355

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A prior study indicates that speakers adjust their behaviour in meetings with DHH peers when ASR is used.

    “A prior study indicates that speakers adjust their behaviour in meetings with DHH peers when ASR is used [Seita et al. 2018].”

    From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Seita et al. 2018

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. Keith Bain, Sara H. Basson, and Mike Wald. 2002. Speech recognition in university classrooms: liberated learning project. In Proceedings of the fifth international ACM conference on Assistive technologies (Assets '02). ACM, New York, NY, USA, 192-196. http://dx.doi.org/10.1145/638249.638284 · DOI
  2. Jon P. Barker, Ricard Marxer, Emmanuel Vincent, Shinji Watanabe. 2017. The CHiME challenges: Robust speech recognition in everyday environments. In: Watanabe S., Delcroix M., Metze F., Hershey J. (eds.), New Era for Robust Speech Recognition. Springer, Cham, 327-344. https://doi.org/10.1007/978-3-319-64680-0_14 · DOI
  3. Larwan Berke, Christopher Caulfield, and Matt Huenerfauth. 2017. Deaf and Hard-of-Hearing Perspectives on Imperfect Automatic Speech Recognition for Captioning One-on-One Meetings. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '17). ACM, NY, NY, USA, 155-164. https://doi.org/10.1145/3132525.3132541 · DOI
  4. Larwan Berke, Sushant Kafle, Matt Huenerfauth. 2018. Methods for Evaluation of Imperfect Captioning Tools by Deaf or Hard-of-Hearing Users at Different Reading Literacy Levels. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI'18). ACM, New York, NY, USA. https://doi.org/10.1145/3173574.3173665 · DOI
  5. Denis Burnham, Sebastian Joeffry, Lauren Rice. 2010. Computer-and Human-Directed Speech Before and After Correction. In Proceedings of the 9 th Speech Science and Technology Conference 2010, Melbourne, Australia. Australian Speech Science and Technology Association.
  6. Esteban Buz, Michael K. Tanenhaus, and T. Florian Jaeger. 2016. Dynamically adapted context-specific hyper-articulation: Feedback from interlocutors affects speakers' subsequent pronunciations. Journal of Memory and Language 89, Supplement C (2016), 68 -86. https://doi.org/10.1016/j.jml.2015.12.009 · DOI
  7. Lisa B. Elliot, Michael Stinson, James Mallory, Donna Easton, and Matt Huenerfauth. 2016. Deaf and Hard of Hearing Individuals' Perceptions of Communication with Hearing Colleagues in Small Groups. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '16). ACM, NY, NY, USA, 271-272. https://doi.org/10.1145/2982142.2982198 · DOI
  8. Lisa B. Elliot, Michael Stinson, Syed Ahmed, and Donna Easton. 2017. User Experiences When Testing a Messaging App for Communication Between Individuals Who Are Hearing and Deaf or Hard of Hearing. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '17). ACM, NY, NY, USA, 405-406. https://doi.org/10.1145/3132525.3134798 · DOI
  9. Maria Federico and Marco Furini. 2012. Enhancing Learning Accessibility Through Fully Automatic Captioning. In Proceedings of the International CrossDisciplinary Conference on Web Accessibility (W4A '12). ACM, New York, NY, USA, Article 40, 4 pages. https://doi.org/10.1145/2207016.2207053 · DOI
  10. Ira R. Forman, Ben Fletcher, John Hartley, Bill Rippon, and Allen Wilson. 2012. Blue Herd: Automated Captioning for Videoconferences. In Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '12). ACM, NY, NY, USA, 227-228. https://doi.org/10.1145/2384916.2384966 · DOI
  11. Jay Hall. 1983. The rejection of deviates as a function of threat. Doctoral Dissertation, University of Texas.
  12. Sushant Kafle and Matt Huenerfauth. 2017. Evaluating the Usability of Automatically Generated Captions for People Who Are Deaf or Hard of Hearing. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '17). ACM, NY, NY, USA, 165-174. https://doi.org/10.1145/3132525.3132542 · DOI
  13. Saba Kawas, George Karalis, Tzu Wen, and Richard E. Ladner. 2016. Improving Real-Time Captioning Experiences for Deaf and Hard of Hearing Students. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '16). ACM, NY, NY, USA, 15-23. https://doi.org/10.1145/2982142.2982164 Session 2: Supporting Speech ASSETS'18, October 22-24, 2018, Galway, Ireland · DOI · Paper page
  14. S. Koster. 2001. Acoustic-phonetic characteristics of hyperarticulated speech for different speaking styles. In Proceedings of the 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. (Cat. No.01CH37221). IEEE. https://doi.org/10.1109/icassp.2001.941054. · DOI
  15. Raja Kushalnagar, Walter Lasecki, Jeffrey Bigham. 2014. Accessibility evaluation of classroom captions. ACM Trans. Access. Comput. 5, 3, Article 7 (Jan. 2014), 24 pp. http://dx.doi.org/10.1145/2543578 · DOI
  16. Walter Lasecki, Christopher Miller, Adam Sadilek, Andrew Abumoussa, Donato Borrello, Raja Kushalnagar, and Jeffrey Bigham. 2012. Real-time captioning by groups of non-experts. In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12). ACM, New York, NY, USA, 23-34. https://doi.org/10.1145/2380116.2380122 · DOI
  17. Steven R. Livingstone, Deanna H. Choi, and Frank A. Russo. 2014. The influence of vocal training and acting experience on measures of voice quality and emotional genuineness. Frontiers in Psychology 5 (Mar 2014). https://doi.org/10.3389/fpsyg.2014.00156 · DOI
  18. Catherine L. Lortie, Mélanie Thibeault, Matthieu J. Guitton, and Pascale Tremblay. 2015. Effects of age on the amplitude, frequency and perceived quality of voice. AGE 37, 6 (Nov 2015). https://doi.org/10.1007/s11357-015-9854-1 · DOI
  19. James R. Mallory, Michael Stinson, Lisa Elliot, and Donna Easton. 2017. Personal Perspectives on Using Automatic Speech Recognition to Facilitate Communication Between Deaf Students and Hearing Customers. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '17). ACM, NY, NY, USA, 419-421. https://doi.org/10.1145/3132525.3134779 · DOI
  20. Sharon Oviatt, Gina-Anne Levow, Elliott Moreton, and Margaret MacEachern. 1998. Modeling Global and Focal Hyperarticulation during Human-Computer Error Resolution. J. Acoust. Soc. Amer. 104, 3080- 3098. https://doi.org/10.1121/1.423888 · DOI · Paper page
  21. Benjamin Picart, Thomas Drugman, and Thierry Dutoit. 2010. Analysis and synthesis of hypo-and hyperarticulated speech. In Proceedings of the Seventh ISCA Workshop on Speech Synthesis.
  22. Koenraad S. Rhebergena, Johannes Lyzenga, Wouter A. Dreschler, Joost M. Festen. 2010. Modeling speech intelligibility in quiet and noise in listeners with normal and impaired hearing. The Journal of the Acoustical Society of America 127(3), 1570-1583. Acoustical Society of America. https://doi.org/10.1121/1.3291000 · DOI
  23. Hagen Soltau and Alex Waibel. 1998. On the Influence of Hyperarticulated Speech on Recognition Performance. In Proceedings of the 5th International Conference on Spoken Language Processing, ICSLP 1998, Sydney, Australia.
  24. Amanda J. Stent, Marie K. Huffman, and Susan E. Brennan. 2008. Adapting Speaking After Evidence of Misrecognition: Local and Global Hyperarticulation. Speech Commun. 50, 3 (March 2008), 163-178. https://doi.org/10.1016/j.specom.2007.07.005 · DOI
  25. Michael Stinson, Carly Linneah, Jonathan MacDonald, and Chelsea Powers. 2014. Using technology to improve communication in small groups with deaf and hearing students. Presentation at the 2 nd Annual Effective Access Technology Conference. Rochester Institute of Technology, Rochester, NY, USA.
  26. Hironobu Takagi, Takashi Itoh, and Kaoru Shinkawa. 2015. Evaluation of Realtime Captioning by Machine Recognition with Human Support. In Proceedings of the 12th Web for All Conference (W4A '15). ACM, New York, NY, USA, Article 5, 4 pages. https://doi.org/10.1145/2745555.2746648 · DOI
  27. M. Wald. 2011. Crowdsourcing Correction of Speech Recognition Captioning Errors. In Proceedings of the Session 2: Supporting Speech ASSETS'18, October 22-24, 2018, Galway, Ireland International Cross-Disciplinary Conference on Web Accessibility (W4A '11). ACM, New York, NY, USA, Article 22, 2 pages. https://doi.org/10.1145/1969289.1969318 · DOI
  28. W. Xiong, J. Droppo, X. Huang, F. Seide, . Seltzer, A. Stolcke, D. Yu, G. Zweig. 2016. Achieving human parity in conversational speech recognition. Computing Research Repository (CoRR), http://arxiv.org/abs/1610.05256 · arXiv
  29. Jiahong Yuan, Mark Liberman, Christopher Cieri. 2007. Towards an Integrated Understanding of Speaking Rate in Conversation. In Proceedings of the 16th International Congress of Phonetic Sciences ICPhS XVI, 6-10 August 2007, Saarbrucken, Germany, 1337-1340, University of Saarbrüken. Session 2: Supporting Speech ASSETS'18, October 22-24, 2018, Galway, Ireland

Cited by

Sorted by citations

Authors
Seita, M., Albusays, K., Kafle, S., Stinson, M., & Huenerfauth, M.
Affiliations
Rochester Institute of Technology (US)
ORCID
Venue
Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility
Publisher
ACM
Published
2018-10-08
Pages
68-80
DOI
10.1145/3234695.3236355
OpenAlex
W2896199444
Cited by
32
References
1
Citations by year
2026: 2 · 2025: 2 · 2024: 3 · 2023: 7 · 2022: 7 · 2021: 6 · 2020: 5
FWCI
1.12
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) (publishedVersion)
Funders
National Science Foundation (1400802, 1462280, 1460894, 1746056)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Social Sciences › Psychology › Developmental and Educational Psychology › Hearing Impairment and Communication
Keywords
speech recognition, automatic captioning, deaf and hard of hearing, conversational behavior, speech quality, harmonic-to-noise ratio, formant shift, speech rate, small group meetings

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review