CoChat
Sign Up
Cited by 52Usability and User Interface DesignDigital Accessibility for DisabilitiesInnovative Human-Technology Interaction

Remotely Co-Designing Features for Communication Applications using Automatic Captioning with Deaf and Hearing Pairs

Open access
Seita, Matthew, Lee, Sooyeon, Andrew, Sarah, Shinohara, Kristen, Huenerfauth, Matt·2022·CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3491102.3501843
Open this paper in CoChat — free accountOpen PDF

Abstract

Deaf and Hard-of-Hearing (DHH) users face accessibility challenges during in-person and remote meetings. While emerging use of applications incorporating automatic speech recognition (ASR) is promising, more user-interface and user-experience research is needed. While co-design methods could elucidate designs for such applications, COVID-19 has interrupted in-person research. This study describes a novel methodology for conducting online co-design workshops with 18 DHH and hearing participant pairs to investigate ASR-supported mobile and videoconferencing technologies along two design dimensions: Correcting errors in ASR output and implementing notification systems for influencing speaker behaviors. Our methodological findings include an analysis of communication modalities and strategies participants used, use of an online collaborative whiteboarding tool, and how participants reconciled differences in ideas. Finally, we present guidelines for researchers interested in online DHH co-design methodologies, enabling greater geographically diversity among study participants even beyond the current pandemic.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3491102.3501843

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Human involvement, e.g. by correcting or monitoring ASR, can potentially increase the acceptability of ASR independent of its actual accuracy.

    “Human involvement, e.g. by correcting or monitoring ASR [McDonnell et al. 2021;Seita et al. 2022], can potentially increase the acceptability of ASR independent of its actual accuracy.”

    From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Seita et al. 2022

    Section 2.2 (page unavailable) assigns hearing partners responsibility for catching and correcting ASR errors, and Section 5.1 reports that participants proposed monitoring and correction features. The paper does not report that human involvement increases ASR acceptability independently of its actual accuracy. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Focus groups can provide ideas on how to improve the use of ASR or help assess the acceptance of new features like automatic content summarisation.

    “Focus groups can also provide ideas on how to improve the use of ASR [Seita et al. 2022], or help assess the acceptance of new features like automatic content summarisation [Alonzo et al. 2020].”

    From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Seita et al. 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 38

  1. Shirin D. Antia, Darrell L. Sabers, and Michael S. Stinson. 2006. Validity and Reliability of the Classroom Participation Questionnaire With Deaf and Hard of Hearing Students in Public Schools. The Journal of Deaf Studies and Deaf Education 12, 2 (11 2006), 158-171. https://doi.org/10.1093/deafed/enl028 arXiv:https://academic.oup.com/jdsde/article-pdf/12/2/158/1233927/enl028.pdf · DOI
  2. George Balch and Donna Mertens. 1999. Focus Group Design and Group Dynam- ics: Lessons from Deaf and Hard of Hearing Participants. The American Journal of Evaluation 20 (06 1999), 265-277. https://doi.org/10.1016/S1098-2140(99)00019-3 · DOI
  3. Mark S. Baldwin, Sen H. Hirano, Jennifer Mankof, and Gillian R. Hayes. 2019. Design in the Public Square: Supporting Assistive Technology Design Through Public Mixed-Ability Cooperation. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 155 (Nov. 2019), 22 pages. https://doi.org/10.1145/3359257 · DOI
  4. Steven Barnett, Jonathan Klein, Robert Pollard Jr, Vincent Samar, Deirdre Schle- hofer, Matthew Starr, Erika Sutter, Hongmei Yang, and Thomas Pearson. 2011. Community Participatory Research With Deaf Sign Language Users to Iden- tify Health Inequities. American journal of public health 101 (12 2011), 2235-8. https://doi.org/10.2105/AJPH.2011.300247 · DOI
  5. Cynthia L Bennett. 2020. Toward Centering Access in Professional Design. Ph.D. Dissertation. University of Washington.
  6. Larwan Berke, Christopher Caulfeld, and Matt Huenerfauth. 2017. Deaf and Hard-of-Hearing Perspectives on Imperfect Automatic Speech Recognition for Captioning One-on-One Meetings. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (Baltimore, Maryland, USA) (ASSETS '17). Association for Computing Machinery, New York, NY, USA, 155-164. https://doi.org/10.1145/3132525.3132541 · DOI
  7. Larwan Berke, Sushant Kafe, and Matt Huenerfauth. 2018. Methods for Evaluation of Imperfect Captioning Tools by Deaf or Hard-of-Hearing Users at Diferent Reading Literacy Levels. Association for Computing Machinery, New York, NY, USA, 1-12. https://doi.org/10.1145/3173574.3173665 · DOI
  8. Debra L Blackwell, Jacqueline W Lucas, and Tainya C Clarke. 2014. Summary Health Statistics for U.S. Adults: National Health Interview Survey. Vital Health Stat 10 260, 10 (2014), 1-161.
  9. Jordan L. Boyd-Graber, Sonya S. Nikolova, Karyn A. Mofatt, Kenrick C. Kin, Joshua Y. Lee, Lester W. Mackey, Marilyn M. Tremaine, and Maria M. Klawe. 2006. Participatory Design with Proxies: Developing a Desktop-PDA System to Support People with Aphasia. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada) (CHI '06). Association for Computing Machinery, New York, NY, USA, 151-160. https: //doi.org/10.1145/1124772.1124797 · DOI
  10. Robin N. Brewer. 2018. Facilitating Discussion and Shared Meaning: Rethinking Co-Design Sessions with People with Vision Impairments. In Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Health- care (New York, NY, USA) (PervasiveHealth '18). Association for Computing Ma- chinery, New York, NY, USA, 258-262. https://doi.org/10.1145/3240925.3240981 · DOI
  11. Clare Cullen and Oussama Metatla. 2019. Co-Designing Inclusive Multisensory Story Mapping with Children with Mixed Visual Abilities. In Proceedings of the 18th ACM International Conference on Interaction Design and Children (Boise, ID, USA) (IDC '19). Association for Computing Machinery, New York, NY, USA, 361-373. https://doi.org/10.1145/3311927.3323146 · DOI
  12. Judith Dörrenbächer and Marc Hassenzahl. 2019. Changing Perspective: A Co-Design Approach to Explore Future Possibilities of Divergent Hearing. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1-12. https://doi.org/10.1145/3290605.3300259 · DOI
  13. Lisa Elliot, Michael Stinson, James Mallory, Donna Easton, and Matt Huenerfauth. 2016. Deaf and Hard of Hearing Individuals' Perceptions of Communication with Hearing Colleagues in Small Groups. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (Reno, Nevada, USA) (ASSETS '16). Association for Computing Machinery, New York, NY, USA, 271-272. https://doi.org/10.1145/2982142.2982198 · DOI
  14. Lisa B. Elliot, Michael Stinson, Syed Ahmed, and Donna Easton. 2017. User Experiences When Testing a Messaging App for Communication Between In- dividuals Who Are Hearing and Deaf or Hard of Hearing. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (Baltimore, Maryland, USA) (ASSETS '17). Association for Computing Machinery, New York, NY, USA, 405-406. https://doi.org/10.1145/3132525.3134798 · DOI
  15. Abraham Glasser, Kesavan Kushalnagar, and Raja Kushalnagar. 2017. Deaf, Hard of Hearing, and Hearing Perspectives on Using Automatic Speech Recog- nition in Conversation. In Proceedings of the 19th International ACM SIGAC- CESS Conference on Computers and Accessibility (Baltimore, Maryland, USA) (AS- SETS '17). Association for Computing Machinery, New York, NY, USA, 427-432. https://doi.org/10.1145/3132525.3134781 · DOI
  16. Jan Gugenheimer, Katrin Plaumann, Florian Schaub, Patrizia Di Campli San Vito, Saskia Duck, Melanie Rabus, and Enrico Rukzio. 2017. The Impact of Assistive Technology on Communication Quality Between Deaf and Hearing Individuals. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW '17). Association for Computing Machinery, New York, NY, USA, 669-682. https://doi.org/10.1145/ 2998181.2998203 · DOI
  17. Sushant Kafe and Matt Huenerfauth. 2017. Evaluating the Usability of Automati- cally Generated Captions for People Who Are Deaf or Hard of Hearing. In Proceed- ings of the 19th International ACM SIGACCESS Conference on Computers and Acces- sibility (Baltimore, Maryland, USA) (ASSETS '17). Association for Computing Ma- chinery, New York, NY, USA, 165-174. https://doi.org/10.1145/3132525.3132542 · DOI
  18. Saba Kawas, George Karalis, Tzu Wen, and Richard E. Ladner. 2016. Improving Real-Time Captioning Experiences for Deaf and Hard of Hearing Students. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (Reno, Nevada, USA) (ASSETS '16). Association for Computing Machinery, New York, NY, USA, 15-23. https://doi.org/10.1145/2982142.2982164 · DOI · Paper page
  19. Charlotte Magnusson, Per-Olof Hedvall, and Héctor Caltenco. 2018. Co-designing together with Persons with Visual Impairments. In Mobility of Visually Impaired People: Fundamentals and ICT Assistive Technologies, Edwige Pissaloux and Ramiro Velazquez (Eds.). Springer International Publishing, Cham, 411-434. https: //doi.org/10.1007/978-3-319-54446-5_14 · DOI
  20. James R. Mallory, Michael Stinson, Lisa Elliot, and Donna Easton. 2017. Personal Perspectives on Using Automatic Speech Recognition to Facilitate Communi- cation between Deaf Students and Hearing Customers. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (Baltimore, Maryland, USA) (ASSETS '17). Association for Computing Machinery, New York, NY, USA, 419-421. https://doi.org/10.1145/3132525.3134779 · DOI
  21. Angela M. O'Donnell. 2006. The Role of Peers and Group Learning. Lawrence Erlbaum Associates Publishers, New Jersey, USA, Chapter 34, 781-802. https: //doi.org/10.4324/9780203874790.ch34 · DOI
  22. Yi-Hao Peng, Ming-Wei Hsi, Paul Taele, Ting-Yu Lin, Po-En Lai, Leon Hsu, Tzu- chuan Chen, Te-Yen Wu, Yu-An Chen, Hsien-Hui Tang, and Mike Y. Chen. 2018. SpeechBubbles: Enhancing Captioning Experiences for Deaf and Hard-of-Hearing People in Group Conversations. Association for Computing Machinery, New York, NY, USA, 1-10. https://doi.org/10.1145/3173574.3173867 · DOI
  23. Anthony Peruma and Yasmine N. El-Glaly. 2017. CollabAll: Inclusive Discussion Support System For Deaf and Hearing Students. In Proceedings of the 19th Inter- national ACM SIGACCESS Conference on Computers and Accessibility (Baltimore, Maryland, USA) (ASSETS '17). Association for Computing Machinery, New York, NY, USA, 315-316. https://doi.org/10.1145/3132525.3134800 · DOI
  24. Anne Marie Piper and James D. Hollan. 2008. Supporting Medical Conversations between Deaf and Hearing Individuals with Tabletop Displays. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (San Diego, CA, USA) (CSCW '08). Association for Computing Machinery, New York, NY, USA, 147-156. https://doi.org/10.1145/1460563.1460587 · DOI
  25. Soraia Silva Prietch, Napoliana Silva de Souza, and Lucia Villela Leite Filgueiras. 2014. A Speech-To-Text System's Acceptance Evaluation: Would Deaf Individuals Adopt This Technology in Their Lives?. In Universal Access in Human-Computer Interaction. Design and Development Methods for Universal Access, Constantine Stephanidis and Margherita Antona (Eds.). Springer International Publishing, Cham, 440-449. https://doi.org/10.1007/978-3-319-07437-5_42 · DOI
  26. Nuzhah Gooda Sahib, Tony Stockman, Anastasios Tombros, and Oussama Metatla. 2013. Participatory Design with Blind Users: A Scenario-Based Approach. In Human-Computer Interaction -INTERACT 2013, Paula Kotzé, Gary Marsden, Gitte Lindgaard, Janet Wesson, and Marco Winckler (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 685-701. https://doi.org/10.1007/978-3-642-40483-2_48 · DOI
  27. Angelina Sales, Yuska Paola Costa Aguiar, and Tiago Maritan U. de Araujo. 2020. Participatory Design Workshops for Adapting a Form for Deaf Respondents. In Universal Access in Human-Computer Interaction. Applications and Practice, Margherita Antona and Constantine Stephanidis (Eds.). Springer International Publishing, Cham, 120-131. https://doi.org/10.1007/978-3-030-49108-6_9 · DOI
  28. Ricardo Pezzotti Schefer and Luciana A. Martinez Zaina. 2016. Designing Social Networking Apps on Mobile Devices: A Participatory Design Experience with Deaf Students. In Proceedings of the 34th ACM International Conference on the Design of Communication (Silver Spring, MD, USA) (SIGDOC '16). Association for Computing Machinery, New York, NY, USA, Article 5, 9 pages. https://doi. org/10.1145/2987592.2987609 · DOI
  29. Matthew Seita, Khaled Albusays, Sushant Kafe, Michael Stinson, and Matt Huen- erfauth. 2018. Behavioral Changes in Speakers Who Are Automatically Captioned in Meetings with Deaf or Hard-of-Hearing Peers. In Proceedings of the 20th Inter- national ACM SIGACCESS Conference on Computers and Accessibility (Galway, Ireland) (ASSETS '18). Association for Computing Machinery, New York, NY, USA, 68-80. https://doi.org/10.1145/3234695.3236355 · DOI · Paper page
  30. Matthew Seita, Sarah Andrew, and Matt Huenerfauth. 2021. Deaf and Hard-of- Hearing Users' Preferences for Hearing Speakers' Behavior during Technology- Mediated in-Person and Remote Conversations. In Proceedings of the 18th In- ternational Web for All Conference (Ljubljana, Slovenia) (W4A '21). Associ- ation for Computing Machinery, New York, NY, USA, Article 25, 12 pages. https://doi.org/10.1145/3430263.3452430 · DOI
  31. Matthew Seita and Matt Huenerfauth. 2020. Deaf Individuals' Views on Speak- ing Behaviors of Hearing Peers When Using an Automatic Captioning App. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA '20). Association for Computing Machinery, New York, NY, USA, 1-8. https://doi.org/10.1145/3334480.3383083 · DOI
  32. Kristen Shinohara, Nayeri Jacobo, Wanda Pratt, and Jacob O. Wobbrock. 2020. Design for Social Accessibility Method Cards: Engaging Users and Refecting on Social Scenarios for Accessible Design. ACM Trans. Access. Comput. 12, 4, Article 17 (Jan. 2020), 33 pages. https://doi.org/10.1145/3369903 · DOI
  33. Kristen Shinohara, Jacob O. Wobbrock, and Wanda Pratt. 2018. Incorporating Social Factors in Accessible Design. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (Galway, Ireland) (ASSETS '18). Association for Computing Machinery, New York, NY, USA, 149-160. https: //doi.org/10.1145/3234695.3236346 · DOI
  34. Stephanie Valencia, Michal Luria, Amy Pavel, Jefrey P. Bigham, and Henny Admoni. 2021. Co-Designing Socially Assistive Sidekicks for Motion-Based AAC. In Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction (Boulder, CO, USA) (HRI '21). Association for Computing Machinery, New York, NY, USA, 24-33. https://doi.org/10.1145/3434073.3444646 · DOI
  35. Nina Wallerstein, John Oetzel, Bonnie Duran, Maya Magarati, Cynthia Pearson, Lorenda Belone, Joyce Davis, Lori Dewindt, Sarah Kastelic, Julie Lucero, Char- maine Ruddock, Erika Sutter, and Mohan Dutta. 2019. Culture-centeredness in community-based participatory research: contributions to health educa- tion intervention research. Health education research 34 (06 2019). https: //doi.org/10.1093/her/cyz021 · DOI
  36. Emily Q. Wang and Anne Marie Piper. 2018. Accessibility in Action: Co-Located Collaboration among Deaf and Hearing Professionals. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 180 (Nov. 2018), 25 pages. https://doi.org/10.1145/ 3274449 CHI '22, April 29-May 5, 2022, New Orleans, LA, USA · DOI · Paper page
  37. Michele A. Williams, Erin Buehler, Amy Hurst, and Shaun K. Kane. 2015. What Not to Wearable: Using Participatory Workshops to Explore Wearable Device Form Factors for Blind Users. In Proceedings of the 12th International Web for All Conference (Florence, Italy) (W4A '15). Association for Computing Machinery, New York, NY, USA, Article 31, 4 pages. https://doi.org/10.1145/2745555.2746664 · DOI
  38. Norziha Megat Mohd Zainuddin, Halimah Badioze Zaman, and Azlina Ahmad Ahmad. 2010. A Participatory Design in Developing Prototype an Augmented Reality Book for Deaf Students. In 2010 Second International Conference on Com- puter Research and Development. IEEE Computer Society, Washington, DC, United States, 400-404. https://doi.org/10.1109/ICCRD.2010.55 · DOI

Cited by

Sorted by citations

Authors
Seita, M., Lee, S., Andrew, S., Shinohara, K., & Huenerfauth, M.
Affiliations
Rochester Institute of Technology (US)
ORCID
Venue
CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2022-04-29
Pages
1-13
DOI
10.1145/3491102.3501843
OpenAlex
W4225011965
Cited by
52
References
3
Citations by year
2026: 6 · 2025: 18 · 2024: 11 · 2023: 13 · 2022: 4
FWCI
9.20
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
CHI Conference on Human Factors in Computing Systems (publishedVersion)
Funders
National Science Foundation (1954284, 1746056)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
deaf and hard of hearing users, speech recognition, captioning, mobile accessibility, ASR error correction, communication modality, COVID-19 pandemic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review