Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter

Probably not a substantial claim: may not pass S3 (load-bearing).

Studies of industry-scale data collection revealed that the pursuit of label precision is often driven by market-oriented values such as objectivity, standardization, and corporate interests.

“studies of industryscale data collection revealed that the pursuit of label precision is often driven by market-oriented values such as objectivity, standardization, and corporate interests [49, 51, 71, 112, 113].”

• cites [49] [49] Shivani Kapania, Alex S Taylor, and Ding Wang. 2023. A hunt for the Snark: Annotator Diversity in Data Practices. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23) . Association for Computing Machinery, New York, NY, USA, Article 133, 15 pages. doi:10.1145/3544548.3580645 - sourcePartly supported

  Sections 4.2.1 and 5 describe practitioners pursuing objective annotations and standardised metrics; section 4.3.3 documents cost and competitive pressures on annotation work. The paper does not establish that market-oriented values drive a pursuit of label precision specifically, so that causal framing goes beyond its findings. The available passages have no page numbers. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [51] [51] Gunay Kazimzade and Milagros Miceli. 2020. Biased Priorities, Biased Outcomes: Three Recommendations for Ethics-oriented Data Annotation Practices. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (New York, NY, USA) (AIES '20) . Association for Computing Machinery, New York, NY, USA, 71. doi:10.1145/3375627.3375809 - sourceNot checked — no full text of the source
• cites [71] [71] Milagros Miceli, Martin Schuessler, and Tianling Yang. 2020. Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 115 (Oct. 2020), 25 pages. doi:10.1145/3415186 - sourcePartly supported

  Section 4.4.1 (page unavailable) says annotation guidelines and quality assurance standardize labels to meet clients’ product and revenue plans; Section 4.4.4 says market logics and profit priorities become inscribed in labels. The paper does not establish objectivity as a market-oriented value driving label precision, and it questions the supposed neutrality of data-driven systems. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [112] [112] Ben Zefeng Zhang, Tianling Yang, Oliver Haimson, and Michaelanne Thomas. 2024. Conceptualizing Precision Labor in Artificial Intelligence Training. AoIR Selected Papers of Internet Research (2024). https://doi.org/10.5210/spir.v2024i0. 14079 - sourceSupported
• cites [113] [113] Ben Zefeng Zhang, Tianling Yang, Milagros Miceli, Oliver L. Haimson, and Michaelanne Thomas. 2025. The Making of Performative Accuracy in AI Training: Precision Labor and Its Consequences. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25) . Association for Computing Machinery, New York, NY, USA, Article 1203, 19 pages. doi:10.1145/3706598.3713112 - sourcePartly supported

  The Introduction links precision with objectivity and describes technical precision demanded by technology-company clients. The Findings describe standardized accuracy targets set by clients, supporting the role of corporate requirements. The paper does not establish that objectivity itself is a market-oriented value driving label precision, and its fieldwork concerns AI training in China rather than industry-scale data collection generally. (LLM, conf=0.80)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Similar label inconsistencies have been observed in datasets of sign languages and stuttered speech.

“Similar label inconsistencies have been observed in datasets of sign languages [12] and stuttered speech [58, 92].”

• cites [12] [12] Danielle Bragg, Naomi Caselli, Julie A. Hochgesang, Matt Huenerfauth, Leah Katz-Hernandez, Oscar Koller, Raja Kushalnagar, Christian Vogler, and Richard E. Ladner. 2021. The FATE Landscape of Sign Language AI Datasets: An Interdisciplinary Perspective. ACM Trans. Access. Comput. 14, 2, Article 7 (July 2021), 45 pages. doi:10.1145/3436996 - sourcePartly supported

  The citation to this paper applies to sign-language datasets, not stuttered speech. Section 8.3 discusses how annotators may produce systematically biased or inaccurate labels, and Sections 3.1 and 8.3 describe differences among annotation systems. These passages support concerns about label variation, but do not establish that label inconsistencies were observed in datasets. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [58] [58] Colin Lea, Vikramjit Mitra, Aparna Joshi, Sachin Kajarekar, and Jeffrey P. Bigham. 2021. SEP-28k: A Dataset for Stuttering Event Detection from Podcasts with People Who Stutter. In ICASSP 2021 -2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . 6798-6802. doi:10.1109/ICASSP39728.2021.9413520 - sourceSupported
• cites [92] [92] Charan Sridhar and Shaomei Wu. 2025. J-j-j-just Stutter: Benchmarking Whisper's Performance Disparities on Different Stuttering Patterns. In Interspeech 2025 . 3753-3757. doi:10.21437/Interspeech.2025-2700 - sourceNot checked — no full text of the source

May combine two claims.

Sep-28k consists of over 28k three-second audio clips curated from public podcasts featuring PWS, and the annotators' backgrounds remain unclear.

“Sep-28k consists of over 28k three-second audio clips curated from public podcasts featuring PWS; however, the annotators' backgrounds remain unclear.”

• cites [58] [58] Colin Lea, Vikramjit Mitra, Aparna Joshi, Sachin Kajarekar, and Jeffrey P. Bigham. 2021. SEP-28k: A Dataset for Stuttering Event Detection from Podcasts with People Who Stutter. In ICASSP 2021 -2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . 6798-6802. doi:10.1109/ICASSP39728.2021.9413520 - sourcePartly supported

  Section 2.1 reports 28,177 three-second clips from podcasts, and the Abstract says the audio came from public podcasts largely featuring people who stutter. Section 2.3 does give some information about the annotators: at least three people labeled each clip, received training, and were not clinicians. Their backgrounds are therefore not wholly unclear, although the paper gives no further qualifications. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Research has called for more reflective data practices, advocating for the treatment of bias or error as a site of negotiation rather than simply as model failures.

“Research has also called for more reflective data practices, advocating for the treatment of bias or error as a site of negotiation rather than simply as model failures [16, 65].”

• cites [16] [16] Scott Allen Cambo and Darren Gergle. 2022. Model Positionality and Computational Reflexivity: Promoting Reflexivity in Data Science. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22) . Association for Computing Machinery, New York, NY, USA, Article 572, 19 pages. doi:10.1145/3491102.3501998 - sourceSupported
• cites [65] [65] Cindy Kaiying Lin and Steven J Jackson. 2023. From bias to repair: Error as a site of collaboration and negotiation in applied data science work. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1-32. doi:10.1145/ 3579607 - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Other research has uncovered the ambiguity and uncertainty inherent in socially constructed labels, such as gender, accent, and content toxicity.

“Other research has uncovered the ambiguity and uncertainty inherent in socially constructed labels, such as gender [85], accent [81], and content toxicity [1, 16].”

• cites [85] [85] Morgan Klaus Scheuerman, Kandrea Wade, Caitlin Lustig, and Jed R. Brubaker. 2020. How We've Taught Algorithms to See Identity: Constructing Race and Gender in Image Databases for Facial Analysis. Proc. ACM Hum.-Comput. Interact. 4, CSCW1, Article 58 (May 2020), 35 pages. doi:10.1145/3392866 - sourceNot checked — no full text of the source
• cites [81] [81] Kerri Prinos, Neal Patwari, and Cathleen A. Power. 2024. Speaking of accent: A content analysis of accent misconceptions in ASR research. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (Rio de Janeiro, Brazil) (FAccT '24) . Association for Computing Machinery, New York, NY, USA, 1245-1254. doi:10.1145/3630106.3658969 - sourceNot checked — no full text of the source
• cites [1] [1] Nuredin Ali Abdelkadir, Tianling Yang, Shivani Kapania, Meron Estefanos, Fasica Berhane Gebrekidan, Zecharias Zelalem, Messai Ali, Rishan Berhe, Dylan Baker, Zeerak Talat, Milagros Miceli, Alex Hanna, and Timnit Gebru. 2025. The Role of Expertise in Effectively Moderating Harmful Social Media Content. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25) . Association for Computing Machinery, New York, NY, USA, Article 227, 21 pages. doi:10.1145/3706598.3714010 - sourceNot checked — no full text of the source
• cites [16] [16] Scott Allen Cambo and Darren Gergle. 2022. Model Positionality and Computational Reflexivity: Promoting Reflexivity in Data Science. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22) . Association for Computing Machinery, New York, NY, USA, Article 572, 19 pages. doi:10.1145/3491102.3501998 - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

A growing body of scholarship emphasizes the need to document the production processes and contextual factors involved in dataset creation.

“Consequently, a growing body of scholarship emphasizes the need to document the production processes and contextual factors involved in dataset creation [72, 73].”

• cites [72] [72] Milagros Miceli, Tianling Yang, Adriana Alvarado Garcia, Julian Posada, Sonja Mei Wang, Marc Pohl, and Alex Hanna. 2022. Documenting Data Production Processes: A Participatory Approach for Data Work. Proc. ACM Hum.Comput. Interact. 6, CSCW2, Article 510 (Nov. 2022), 34 pages. doi:10.1145/ 3555623 - sourceSupported
• cites [73] [73] Milagros Miceli, Tianling Yang, Laurens Naudts, Martin Schuessler, Diana Serbanescu, and Alex Hanna. 2021. Documenting Computer Vision Datasets: An Invitation to Reflexive Data Practices. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT '21) . Association for Computing Machinery, New York, NY, USA, 161-172. doi:10.1145/3442188.3445880 - sourceSupported

May combine two claims.

Sep-28k does not have transcriptions and provides only clip-level annotations rather than word-/syllable-level labels.

“as the largest publicly available English stuttering dataset, Sep-28k does not have transcriptions and provides only clip-level annotations rather than word-/syllablelevel labels [58].”

• cites [58] [58] Colin Lea, Vikramjit Mitra, Aparna Joshi, Sachin Kajarekar, and Jeffrey P. Bigham. 2021. SEP-28k: A Dataset for Stuttering Event Detection from Podcasts with People Who Stutter. In ICASSP 2021 -2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . 6798-6802. doi:10.1109/ICASSP39728.2021.9413520 - sourcePartly supported

  Section 2.3 (page not provided) says SEP-28k audio was divided into three-second clips with binary labels, supporting the claim that its annotations are clip-level rather than word- or syllable-level. Our copy does not explicitly say that SEP-28k has no transcriptions, so that part of the claim is not established. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

Sep-28k divides audio into strictly 3-second clips, making it often difficult to determine stuttering events without access to conversational flow and context.

“As Sep-28k divides audio into strictly 3-second clips [58], it is often difficult to determine stuttering events without access to the conversational flow and context.”

• cites [58] [58] Colin Lea, Vikramjit Mitra, Aparna Joshi, Sachin Kajarekar, and Jeffrey P. Bigham. 2021. SEP-28k: A Dataset for Stuttering Event Detection from Podcasts with People Who Stutter. In ICASSP 2021 -2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . 6798-6802. doi:10.1109/ICASSP39728.2021.9413520 - sourceSupported

May combine two claims.

We focused on Sep-28k due to its widespread use, its public availability of annotations produced by PWS and non-PWS, and the reported discrepancy between these two versions.

“We focused on Sep-28k due to its widespread use 4 , its public availability of annotations produced by PWS [92] and non-PWS [58], and the reported discrepancy between these two versions [92].”

• cites [92] [92] Charan Sridhar and Shaomei Wu. 2025. J-j-j-just Stutter: Benchmarking Whisper's Performance Disparities on Different Stuttering Patterns. In Interspeech 2025 . 3753-3757. doi:10.21437/Interspeech.2025-2700 - sourceNot checked — no full text of the source
• cites [58] [58] Colin Lea, Vikramjit Mitra, Aparna Joshi, Sachin Kajarekar, and Jeffrey P. Bigham. 2021. SEP-28k: A Dataset for Stuttering Event Detection from Podcasts with People Who Stutter. In ICASSP 2021 -2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . 6798-6802. doi:10.1109/ICASSP39728.2021.9413520 - sourcePartly supported

  The Introduction says SEP-28k’s annotations were released publicly, and §2.3 says at least three trained people annotated each clip. Section 2.3 describes them as non-clinicians but does not say whether they were non-PWS; that distinction is not established for the part attributed to [58]. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

May combine two claims.

Since ASR models have increasingly relied on language models to produce most probable transcripts, how speech is segmented would significantly influence the amount of context available to language models.

“Since ASR models have increasingly relied on language models to produce most probable transcripts [82], how the speech is segmented would significantly influence the amount of context available to language models.”

• cites [82] [82] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. Robust Speech Recognition via Large-Scale Weak Supervision. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202) , Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 28492-28518. https://proceedings.mlr.press/v202/radford23a.html - sourcePartly supported

  Section 2.3 (p. 3) describes Whisper’s decoder as an audio-conditional language model that uses preceding transcript text to resolve ambiguous audio. This supports the role of a language model in transcription, but our copy does not establish the cited clause’s broader claim that ASR models have increasingly relied on language models. The segmentation inference follows the citation and is not attributed to this source. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

May combine two claims.

Our guidelines differ from SLP frameworks developed to measure fluency and stuttering severity, as well as from earlier labeling frameworks designed primarily to detect stuttering events.

“Our guidelines differ from SLP frameworks developed to measure fluency and stuttering severity [83, 109], as well as from earlier labeling frameworks designed primarily to detect stuttering events [58].”

• cites [83] [83] Nan Bernstein Ratner and Brian MacWhinney. 2018. Fluency Bank: A new resource for fluency research and practice. Journal of fl uency disorders 56 (2018), 69-80. https://doi.org/10.1016/j.jfludis.2018.03.002 - sourcePartly supported

  Section 1.2 (TalkBank Features; page unavailable) describes FluencyBank’s standardized codes for marking stuttering and other disfluencies, supporting its identification as a fluency-labeling framework. Section 3.3 (Teaching with FluencyBank; page unavailable) discusses scoring stuttering severity with the separate SSI-4 instrument, rather than presenting FluencyBank as a framework developed to measure severity. The sentence’s comparison with the citing authors’ own guidelines is not established by this source. (LLM, conf=0.81)

  Experimental — accuracy not yet measured
• cites [109] [109] J Scott Yaruss. 1997. Clinical measurement of stuttering behaviors. Contemporary Issues in Communication Science and Disorders 24, Spring (1997), 27-38. https: //doi.org/10.1044/cicsd\_24\_S\_27 - sourceNot checked — no full text of the source
• cites [58] [58] Colin Lea, Vikramjit Mitra, Aparna Joshi, Sachin Kajarekar, and Jeffrey P. Bigham. 2021. SEP-28k: A Dataset for Stuttering Event Detection from Podcasts with People Who Stutter. In ICASSP 2021 -2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . 6798-6802. doi:10.1109/ICASSP39728.2021.9413520 - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In many tasks, disabled people might have limited access to sensory information that is important to the task, or their desired information might come into tension with other values such as privacy or making racial inference from photos.

“In many tasks, disabled people might have limited access to sensory information that is important to the task [33, 75], or their desired information might come into tension with other values such as privacy or making racial inference from photos [10, 37, 94].”

• cites [33] [33] Steven M Goodman, Ping Liu, Dhruv Jain, Emma J McDonnell, Jon E Froehlich, and Leah Findlater. 2021. Toward user-driven sound recognizer personalization with people who are d/deaf or hard of hearing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 2 (2021), 1-23. doi:10. 1145/3463501 - sourceNot checked — no full text of the source
• cites [75] [75] Annika Muehlbradt and Shaun K. Kane. 2022. What's in an ALT Tag? Exploring Caption Content Priorities through Collaborative Captioning. ACM Trans. Access. Comput. 15, 1, Article 6 (March 2022), 32 pages. doi:10.1145/3507659 - sourcePartly supported

  Section 2.3 (Crowdsourced Captioning; page unavailable) says captioning services can violate users’ privacy by exposing sensitive information to strangers. The example captions in Table 3 include racial descriptions of people in photos, but the paper does not identify racial inference as a value in tension with users’ desired information. The privacy concern supports part of the clause cited to this paper; the claimed tension around racial inference is not established. (LLM, conf=0.77)

  Experimental — accuracy not yet measured
• cites [10] [10] Cynthia L. Bennett, Cole Gleason, Morgan Klaus Scheuerman, Jeffrey P. Bigham, Anhong Guo, and Alexandra To. 2021. 'It's Complicated': Negotiating Accessibility and (Mis)Representation in Image Descriptions of Race, Gender, and Disability. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21) . Association for Computing Machinery, New York, NY, USA, Article 375, 19 pages. doi:10.1145/3411764.3445498 - sourceNot checked — no full text of the source
• cites [37] [37] Margot Hanley, Solon Barocas, Karen Levy, Shiri Azenkot, and Helen Nissenbaum. 2021. Computer vision and conflicting values: Describing people with automated alt text. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society . 543-554. doi:10.1145/3461702.3462620 - sourceSupported
• cites [94] [94] Xinru Tang, Ali Abdolrahmani, Darren Gergle, and Anne Marie Piper. 2025. Everyday Uncertainty: How Blind People Use GenAI Tools for Information Access. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25) . Association for Computing Machinery, New York, NY, USA, Article 63, 17 pages. doi:10.1145/3706598.3713433 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Garg et al. benchmarked vision-language models using a re-annotated subset of VizWiz, informed by a survey of blind participants.

“For example, Garg et al. benchmarked visionlanguage models using a re-annotated subset of VizWiz, informed by a survey of blind participants [31].”

• cites [31] [31] Kapil Garg, Xinru Tang, Jimin Heo, Dwayne R Morgan, Darren Gergle, Erik B Sudderth, and Anne Marie Piper. 2025. 'It's trained by non-disabled people': Evaluating How Image Quality Affects Product Captioning with VLMs. arXiv preprint arXiv:2511.08917 (2025). - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

These challenges echo recent work that treats data practices as reflexive practices and frames inconsistencies as sites for reflection.

“These challenges echo recent work that treats data practices as reflexive practices [16] and frames inconsistencies as sites for reflection [65].”

• cites [16] [16] Scott Allen Cambo and Darren Gergle. 2022. Model Positionality and Computational Reflexivity: Promoting Reflexivity in Data Science. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22) . Association for Computing Machinery, New York, NY, USA, Article 572, 19 pages. doi:10.1145/3491102.3501998 - sourceSupported
• cites [65] [65] Cindy Kaiying Lin and Steven J Jackson. 2023. From bias to repair: Error as a site of collaboration and negotiation in applied data science work. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1-32. doi:10.1145/ 3579607 - sourceSupported

Probably not a substantial claim: may not pass S2 (checkable).

Disability expertise is the particular knowledge that disabled people develop and enact about unorthodox configurations of agency, cultural norms, and relationships between selves, bodies, and the designed world.

“Such knowledge represents what Hartblay called disability expertise -' the particular knowledge that disabled people develop and enact about unorthodox configurations of agency, cultural norms, and relationships between selves, bodies, and the designed world ' [38].”

• cites [38] [38] Cassandra Hartblay. 2020. Disability expertise: claiming disability anthropology. Current Anthropology 61, S21 (2020), S26-S36. doi:10.1086/705781 - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Hearing nonsigners may overlook critical linguistic elements in sign language that deaf people are familiar with, such as facial expressions.

“For instance, hearing nonsigners may overlook critical linguistic elements in sign language that deaf people are familiar with, such as facial expressions [28, 97].”

• cites [28] [28] Michael Erard. 2017. Why Sign-Language Gloves Don't Help Deaf People. https://www.theatlantic.com/technology/archive/2017/11/why-signlanguage-gloves-dont-help-deaf-people/545441/. Retrieved June 3, 2025. -Not checked yet

  Not checked in the free run.

• cites [97] [97] Xinru Tang and Anne Marie Piper. 2026. Reimagining Sign Language Technologies: Analyzing Translation Work of Chinese Deaf Online Content Creators. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems - sourcePartly supported

  Section 2 (p. 2) identifies facial expressions as visual-spatial elements that convey meaning in sign language, and the Introduction (p. 1) says sign-language gloves have been criticized for neglecting them. Section 5.2 (p. 9) also describes pressure to simplify signing for hearing audiences. The paper does not specifically say that hearing nonsigners themselves overlook facial expressions that deaf people recognize. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

Labeling issues are common in accessibility datasets and can benefit from the involvement of PWD.

“Similar to stuttered speech datasets, labeling issues are common in accessibility datasets and can benefit from the involvement of PWD [12, 31].”

• cites [12] [12] Danielle Bragg, Naomi Caselli, Julie A. Hochgesang, Matt Huenerfauth, Leah Katz-Hernandez, Oscar Koller, Raja Kushalnagar, Christian Vogler, and Richard E. Ladner. 2021. The FATE Landscape of Sign Language AI Datasets: An Interdisciplinary Perspective. ACM Trans. Access. Comput. 14, 2, Article 7 (July 2021), 45 pages. doi:10.1145/3436996 - sourcePartly supported

  Section 8.3 describes potential labeling biases in sign-language datasets and discusses involving deaf signers in annotation. It does not establish that labeling issues are common across accessibility datasets generally; its discussion is specific to sign-language data. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [31] [31] Kapil Garg, Xinru Tang, Jimin Heo, Dwayne R Morgan, Darren Gergle, Erik B Sudderth, and Anne Marie Piper. 2025. 'It's trained by non-disabled people': Evaluating How Image Quality Affects Product Captioning with VLMs. arXiv preprint arXiv:2511.08917 (2025). - sourceSupported

Classified as probably not a substantial claim.

Fricker used the concept of epistemic injustice to describe the harm done to people in their capacity as a knower, excluding their perspective due to bias, prejudice, or structural inequality.

“Fricker used the concept to describe the harm done to people in their capacity as a knower, excluding their perspective due to bias, prejudice, or structural inequality [29].”

• cites [29] [29] Miranda Fricker. 2017. Evolving concepts of epistemic injustice. In The Routledge handbook of epistemic injustice . Routledge, 53-60. - sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Sighted people often do not understand what information blind people want in visual descriptions.

“Sighted people often do not understand what information blind people want in visual descriptions [31, 90].”

• cites [31] [31] Kapil Garg, Xinru Tang, Jimin Heo, Dwayne R Morgan, Darren Gergle, Erik B Sudderth, and Anne Marie Piper. 2025. 'It's trained by non-disabled people': Evaluating How Image Quality Affects Product Captioning with VLMs. arXiv preprint arXiv:2511.08917 (2025). - sourcePartly supported

  In Results (p. 7), blind and low-vision participants report that AI-generated product captions often omit details they need. Final Reflections (p. 8) calls on developers to understand those needs and records one participant’s concern about models trained by non-disabled people. These passages support a mismatch in product-captioning tools, but do not establish that sighted people generally or often misunderstand what blind people want in visual descriptions. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [90] [90] Rachel N. Simons, Danna Gurari, and Kenneth R. Fleischmann. 2020. 'I Hope This Is Helpful': Understanding Crowdworkers' Challenges and Motivations for an Image Description Task. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 105 (Oct. 2020), 26 pages. doi:10.1145/3415176 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The survey allowed Garg et al. to incorporate information that blind people want but that sighted crowdworkers had not included.

“The survey allowed them to incorporate information that blind people want but that sighted crowdworkers had not included [31].”

• cites [31] [31] Kapil Garg, Xinru Tang, Jimin Heo, Dwayne R Morgan, Darren Gergle, Erik B Sudderth, and Anne Marie Piper. 2025. 'It's trained by non-disabled people': Evaluating How Image Quality Affects Product Captioning with VLMs. arXiv preprint arXiv:2511.08917 (2025). - sourcePartly supported

  Section 5.1 (p. 15) says the survey surfaced blind and low-vision participants’ unmet needs for details in product images. Section 4.1 (p. 8) separately says crowdworker captions varied in detail and motivated a dataset with verified annotations. The paper does not establish that the survey let the authors incorporate specific information that sighted crowdworkers had omitted. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

Our perspectives may be skewed toward people with higher socio-economic backgrounds and greater technical literacy, and under-represent those with multiple disabilities.

“Despite our efforts in engaging multiple perspectives and community voices, our perspectives may be skewed toward people with higher socio-economic backgrounds and greater technical literacy, and under-represent those with multiple disabilities [102].”

• cites [102] [102] Ana Rita Valente, Rufael Marew, Hawau Olamide Toyin, Hamdan Al-Ali, Anelise Bohnen, Inma Becerra, Elsa Marta Soares, Goncalo Leal, and Hanan Aldarmaki. 2025. Clinical Annotations for Automatic Stuttering Severity Assessment. arXiv preprint arXiv:2506.00644 (2025). - sourceCould not verify

  Section 4.4 (p. 3) discusses non-native English speakers and possible comorbidities in speech samples, but not the socioeconomic background or technical literacy of the people whose perspectives were gathered. Searches for those terms and for under-representation of people with multiple disabilities found no supporting passage in our copy. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Recent work has highlighted reflexive practices drawn from interpretivist traditions, such as encouraging data workers to reflect on their positionality and drawing on qualitative research methods to aid in data interpretation.

“Recent work has highlighted reflexive practices drawn from interpretivist traditions, such as encouraging data workers to reflect on their positionality [16] and drawing on qualitative research methods to aid in data interpretation [103].”

• cites [16] [16] Scott Allen Cambo and Darren Gergle. 2022. Model Positionality and Computational Reflexivity: Promoting Reflexivity in Data Science. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22) . Association for Computing Machinery, New York, NY, USA, Article 572, 19 pages. doi:10.1145/3491102.3501998 - sourceSupported
• cites [103] [103] Ruyuan Wan, Haonan Wang, Ting-Hao Kenneth Huang, and Jie Gao. 2025. From Noise to Nuance: Enriching Subjective Data Annotation through Qualitative Analysis. In Proceedings of the Fourth Workshop on Bridging HumanComputer Interaction and Natural Language Processing (HCI+NLP) , Su Lin Blodgett, Amanda Cercas Curry, Sunipa Dev, Siyan Li, Michael Madaio, Jack Wang, Sherry Tongshuang Wu, Ziang Xiao, and Diyi Yang (Eds.). Association for Computational Linguistics, Suzhou, China, 240-254. doi:10.18653/v1/2025.hcinlp1.20 - sourceSupported

May combine two claims.

PWS participants showed strong intuition on which label 'best' captures the speech patterns, but they were also conscious about the space of interpretation and the fairness risk when deterministically assigning one label as the 'ground truth'.

“While PWS participants showed strong intuition on which label 'best' captures the speech patterns, they were also conscious about the space of interpretation and the fairness risk when deterministically assigning one label as the 'ground truth' [69].”

• cites [69] [69] Anna P. Meyer, Yea-Seul Kim, Loris D'Antoni, and Aws Albarghouthi. 2025. Perceptions of the Fairness Impacts of Multiplicity in Machine Learning. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25) . Association for Computing Machinery, New York, NY, USA, Article 524, 15 pages. doi:10.1145/3706598.3713524 - sourcePartly supported

  Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Charan observed that a significant portion of the errors in Sep-28k's labels stemmed from labeling clearly fluent speech as stuttered.

“Charan observed that a significant portion of the ' errors ' in Sep-28k's labels stemmed from labeling ' clearly fl uent speech ' as stuttered:”

May combine two claims.

Both participants noted the difficult case when PWS try to circumvent overt stuttering, yet they could better notice these 'tricks' for masking stutters.

“Moreover, both participants noted the difficult case when PWS try to circumvent overt stuttering, yet they could better notice these 'tricks' for masking stutters:”

May combine two claims.

Sep-28k consistently labels more stuttering events than Sep-28k-SW, resulting in more false positives than false negatives.

“While disagreements occur across all stuttering types, Sep-28k consistently labels more stuttering events than Sep-28k-SW -resulting in more false positives than false negatives in Table 4.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Charan mentioned accessory signals such as breathing, tone of speech, and speed of talking.

“Charan mentioned accessory signals such as 'breathing, tone of speech, and speed of talking' .”

Probably not a substantial claim: may not pass S3 (load-bearing).

Rong highlighted the challenges of converting and preserving speech disfluencies into text transcriptions.

“Rong highlighted the challenges of converting and preserving speech disfluencies into text transcriptions,”

Probably not a substantial claim: may not pass S3 (load-bearing).

Rong explained that prolongation should be continuous in the flow when pronouncing a sound, which is distinct from the choppy sounds associated with blocks.

“Rong explained that prolongation ' should be continuous in the fl ow when pronouncing a sound , ' which is distinct from the choppy sounds associated with blocks.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Rong found that annotations could be consistent in roughly 80% of cases, but achieving complete agreement across all instances is impossible.

“Drawing from his experience training non-PWS annotators, Rong found that while annotations could be consistent in 'roughly 80% of cases', achieving complete agreement across all instances is 'impossible'.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Rong explained the kinematic signals he used to identify blocks, derived from his own sensation of stuttering.

“Similarly, Rong explained the kinematic signals he used to identify blocks, derived from his own sensation of stuttering:”

May combine two claims.

In general, PWS participants agreed with the stuttering event categorizations in the preliminary guidelines and did not see the need to add or remove any labels.

“In general, the participants agreed with the stuttering event categorizations presented in the preliminary guidelines, and did not see the need to add or remove any labels from the framework.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Variability in speech segmentation can affect how much context is available to speech annotators and models to interpret stuttering.

“Building on this point, Jia noted the variability in speech segmentation can affect how much context available to speech annotators and models to interpret stuttering.”

Classified as probably not a substantial claim.

Participants explained that they have certain stuttering patterns, which are important for differentiating labels.

“Furthermore, participants explained that they have certain stuttering patterns, which are important for differentiating labels.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Sometimes stuttering events or cues can be cut out from auto segments.

“Software Use : examining the audio waveform rather than relying on the software's auto-segmentation, as sometimes stuttering events or cues can be cut out from auto segments.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants noted that the labels represent a big improvement over previous datasets.

“They noted that the labels capture a significant portion of stuttering (Adedotun, Tatianna), represent a big improvement over previous datasets (Benji), and emphasized the importance of verbatim transcription for reflecting a key aspect of their identity (Amina) and educating others about stuttering (Adedotun, Amina, Tatianna).”

Classified as probably not a substantial claim.

Participants noted that the labels capture a significant portion of stuttering.

“They noted that the labels capture a significant portion of stuttering (Adedotun, Tatianna), represent a big improvement over previous datasets (Benji), and emphasized the importance of verbatim transcription for reflecting a key aspect of their identity (Amina) and educating others about stuttering (Adedotun, Amina, Tatianna).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both SLP participants agreed that videos could be incorporated into the review process, as embodied cues such as lip movements are often a crucial aspect of stuttering and can aid interpretation.

“For instance, Jia and Kerrigan both agreed that videos could be incorporated into the review process, as embodied cues such as lip movements are often a crucial aspect of stuttering and can aid interpretation [99].”

• cites [99] [99] Seth E Tichenor and J Scott Yaruss. 2019. Stuttering as defined by adults who stutter. Journal of Speech, Language, and Hearing Research 62, 12 (2019), 43564369. https://doi.org/10.1044/2019\_JSLHR-19-00137 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Adedotun highlighted the value of labeling silences.

“For example, Adedotun highlighted the value of labeling silences:”

Probably not a substantial claim: may not pass S3 (load-bearing).

Amina expressed a strong preference for having professionals handle the annotations.

“Amina expressed a strong preference for having professionals handle the annotations:”

Probably not a substantial claim: may not pass S3 (load-bearing).

Adedotun valued inconsistent labels as a means to promote awareness and educate others about the diversity of human speech perception.

“Recognizing the inherent subjectivity in human perception, Adedotun valued the inconsistent labels, seeing them as a means to promote awareness and educate others about the diversity of human speech perception.”

May combine two claims.

These short, light events could be easily missed by listeners, but both participants were able to identify them due to familiarity with their own stuttering patterns.

“These short, light events could be easily missed by listeners but both participants were able to identity them due to their familiarity with their own stuttering patterns.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Tatianna attributed differences in annotations to the gap between how stuttering sounds and how it is experienced internally.

“Tatianna echoed similar points, attributing the differences in annotations to the gap between how stuttering sounds and how it is experienced internally:”

Probably not a substantial claim: may not pass S3 (load-bearing).

This challenge fostered an appreciation for context-dependent evaluations of label accuracy.

“This challenge fostered an appreciation for context-dependent evaluations of label accuracy.”

Classified as probably not a substantial claim. May combine two claims.

Both SLP participants agreed that there is no single correct way to represent the speech but different perspectives.

“As a result, both participants agreed that there is no single correct way to represent the speech but different perspectives.”

Classified as probably not a substantial claim.

How to represent mixed and nuanced types of stuttering presents a challenge when following a rigid guideline.

“To her, how to represent these mixed and nuanced types of stuttering present a challenge when following a rigid guideline.”

May combine two claims.

In scenarios with unfamiliar interlocutors, including stuttering labels in transcriptions was seen as important for conveying personal identity and promoting stuttering awareness.

“In such scenarios, including stuttering labels in transcriptions was seen as important for conveying personal identity and promoting stuttering awareness.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Amina explained that she often injects small interjections to get over blocks, which could blur the boundary between the two labels.

“For example, Amina explained that she often injects small interjections to get over blocks, which could blur the boundary between the two labels.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Adedotun acknowledged that for those who prioritize fluency, specific labels might be important because different ways of stuttering can use different techniques.

“However, Adedotun acknowledged that for those who prioritize fluency, the specific labels might be important ' because the different ways of stuttering, you can use different techniques, so I think it would be important to have those nuances. '”

May combine two claims.

SLP professionals' practice with the guidelines confirmed the challenges identified in earlier sessions and surfaced two additional ones: the influence of audio processing within annotation software, and the challenge of representing diverse stuttering patterns.

“Their practice with our guidelines confirmed the challenges identified in earlier sessions, while also surfacing two additional ones that enhanced our understanding of the subjectivity in annotations: (1) the influence of audio processing within annotation software, and (2) the challenge of representing diverse stuttering patterns.”

May combine two claims.

Although they annotated the speech differently, both SLP participants believed their approaches were grounded in deliberate considerations.

“Although they annotated the speech differently, both believed that their approaches are grounded in deliberate considerations.”

May combine two claims.

Charan did not label Example #2 as sound repetition or elongation, basing his judgment on his embodied experiences with stuttering.

“However, Charan did not label it as sound repetition or elongation, basing his judgment on his embodied experiences with stuttering.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

As part of SLP training, Jia learned multiple standards for speech segmentation and could apply them fluidly in different situations.

“Jia explained, as part of the SLP training, they have learned multiple standards for speech segmentation, and could apply them fluidly in different situations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Adedotun proposed ways to express ambiguity in speech perception by using mixed notations such as '/b::/p, /b:/p, and /b=/p.'

“Putting himself in the listener's position, he proposed ways to express ambiguity in speech perception by using mixed notations such as '/b::/p, /b:/p, and /b=/p.'”

Probably not a substantial claim: may not pass S3 (load-bearing).

Tatianna noted that speech perception is deeply shaped by listeners' personal language background, including potential differences between fluent and disfluent speakers, as well as between native and non-native English users.

“Echoing this point, Tatianna noted that speech perception is deeply shaped by listeners' personal language background, including potential differences between fluent and disfluent speakers, as well as between native and non-native English users.”

May combine two claims.

A key way participants resolved differences was by sharing their embodied experiences of stuttering, as many disagreements stemmed from the gap between living with stuttering and observing it from the outside.

“A key way participants resolved differences was by sharing their embodied experiences of stuttering, as many disagreements stemmed from the gap between living with stuttering and observing it from the outside.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Audio segmentation and the design of annotation tools can distort stuttered speech and omit key cues such as breathing or non-speech sounds.

“The socio-technical influences. Audio segmentation and the design of annotation tools can distort stuttered speech and omit key cues such as breathing or non-speech sounds.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Tatianna learned over years that her blocks are manifested mainly through repetitions and prolongations, while Benji described his stuttering as consisting primarily of repetitions.

“For example, Tatianna learned over years that her blocks are manifested mainly through repetitions and prolongations, while Benji described his stuttering as consisting primarily of repetitions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Benji highlighted the value of phonetic annotations, noting that people who stutter might 'transform sounds' during repetitions, making each repetition sound slightly different.

“Similarly, Benji highlighted the value of phonetic annotations, noting that people who stutter might ' transform sounds ' during repetitions, making each repetition sound slightly different.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Most heuristics were derived from the bodily experience of stuttering.

“Most heuristics were derived from the bodily experience of stuttering.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some participants placed greater value on specific stuttering subtypes because therapists need to learn from different ways of stuttering and to see the patient's whole self.

“Some participants placed greater value on specific stuttering subtypes, as therapists need to ' learn from those different ways of stuttering and better take care of patients who stutter ' (Adedotun), and to see their ' whole self ' (Amina).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The airflow is continuous in prolongation, but choppy in blocks.

“The airflow is continuous in prolongation, but choppy in blocks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It is hard to tell the words because of audio segmentation.

“Hard to tell the words because of audio segmentation, e.g., um vs. I'm , and vs. uh .”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A gush of air right before a sound can indicate a block, even when there is no clear pause in speech.

“a gush of air right before a sound (e.g. a strong airy 't' sound in 'turn') can indicate a block, even when there is no clear pause in speech.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Sometimes stuttering events or cues could be missed outside of the timestamped segments.

“Sometimes stuttering events or cues could be missed outside of the segments.”

Classified as probably not a substantial claim. May combine two claims.

Annotations might prioritize content accuracy in certain scenarios such as voice commands and speech-to-texting, while focusing on accurate stuttering event labels in therapy or educational contexts.

“For example, annotations might prioritize content accuracy in certain scenarios such as voice commands and speechto-texting, while focusing on accurate stuttering event labels in therapy or educational contexts.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

PWS might prioritize the accuracy of certain labels according to their speech patterns.

“Additionally, PWS might prioritize the accuracy of certain labels according to their speech patterns.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

It is easy to miss non-speech content because of how Praat renders segments.

“Easy to miss non-speech content because of how Pratt renders segments; annotators might click the segment while missing cues outside of the segments.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Sound repetition is involuntary but the repetition from block and backtracking is intentional.

“Sound repetition is involuntary but the repetition from block and backtracking is intentional.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

One word can contain multiple stuttering events.

“One word can contain multiple stuttering events, thus be annotated with more than one stuttering code.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It is hard to contextualize the sounds because of the audio segmentation.

“Hard to contextualize the sounds because of the audio segmentation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Our SLP participants proposed a stewardship model in which a more experienced member oversees the annotation process.

“Reflecting similar ideas, our SLP participants proposed a stewardship model in which a more experienced member oversees the annotation process.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The perception of elongation of vowels is inherently subjective, especially without listening to the speech flow, and for long vowels and compound vowels.

“The perception of elongation of vowels is inherently subjective especially without listening to the speech flow, and for long vowel and compound vowel: hear, been, language, oh, ends, two, so, okay, wow .”

Classified as probably not a substantial claim.

Prior research has shown that so-called 'ground truth' labels are shaped by specific contexts, such as annotators' backgrounds and expertise, as well as broader organizational control.

“Prior research has shown that so-called 'ground truth' labels are shaped by specific contexts, such as annotators' backgrounds and expertise, as well as broader organizational control [1, 76].”

• cites [1] [1] Nuredin Ali Abdelkadir, Tianling Yang, Shivani Kapania, Meron Estefanos, Fasica Berhane Gebrekidan, Zecharias Zelalem, Messai Ali, Rishan Berhe, Dylan Baker, Zeerak Talat, Milagros Miceli, Alex Hanna, and Timnit Gebru. 2025. The Role of Expertise in Effectively Moderating Harmful Social Media Content. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25) . Association for Computing Machinery, New York, NY, USA, Article 227, 21 pages. doi:10.1145/3706598.3714010 - sourceNot checked — no full text of the source
• cites [76] [76] Michael Muller, Christine T. Wolf, Josh Andres, Michael Desmond, Narendra Nath Joshi, Zahra Ashktorab, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Evelyn Duesterwald, and Casey Dugan. 2021. Designing Ground Truth and the Social Life of Labels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21) . Association for Computing Machinery, New York, NY, USA, Article 94, 16 pages. doi:10.1145/3411764.3445402 - sourceNot checked — no full text of the source

May combine two claims.

A critical gap remains in data annotations: disabled people are typically treated solely as the source of data during data collection and excluded from other crucial stages such as data annotation and model design.

“Despite recent disability-first data collection efforts, a critical gap remains in data annotations. Disabled people are typically treated solely as the source of data during data collection, excluded from other crucial stages of the AI development pipeline such as data annotation and model design [61].”

• cites [61] [61] Jingjin Li, Peiyao Liu, Rebecca Lietz, Ningjing Tang, Norman Makoto Su, and Shaomei Wu. 2025. Govern With, Not For: Understanding the Stuttering Community's Preferences and Goals for Speech AI Data Governance in the US and China. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , [Vol. 8. 1548-1560. https://doi.org/10.1609/aies.v8i2.36654](https://doi.org/10.1609/aies.v8i2.36654) -Not checked yet

  Not checked in the free run.

May combine two claims.

Label noise in accessibility datasets remains a persistent yet underexplored problem.

“Despite ongoing data collection efforts, label noise in accessibility datasets remains a persistent yet underexplored problem.”

Probably not a substantial claim: may not pass S4 (based).

The resulting guidelines from the co-design study represent the first effort to develop PWS-centered stuttered speech annotation guidelines for speech AI development.

“The resulting guidelines from our co-design study represent the first effort to develop PWS-centered stuttered speech annotation guidelines for speech AI development (see Appendix A).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Stuttering is a neurodevelopmental condition characterized by involuntary disfluencies in speech.

“Stuttering is a neurodevelopmental condition characterized by involuntary disfluencies in speech, affecting over 1% of the global population [77].”

• cites [77] [77] NIDCD Information Clearinghouse. [n. d.]. Stuttering. https://www.nidcd.nih. gov/health/stuttering. Retrieved June, 2025. -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

The guidelines caution against the common practice of trimming non-speech audio during automatic speech recognition model training.

“and caution against the common practice of trimming non-speech audio during automatic speech recognition (ASR) model training [56].”

• cites [56] [56] Lori Lamel, Jean-Luc Gauvain, and Gilles Adda. 2002. Lightly supervised and unsupervised acoustic model training. Comput. Speech Lang. 16, 1 (Jan. 2002), 115-129. doi:10.1006/csla.2001.0186 - sourceNot checked — no full text of the source

May combine two claims.

The field of SLP has historically framed stuttering as a problem to be fixed, relying on external observations and control.

“The field of SLP has historically framed stuttering as a problem to be fixed, relying on external observations and control [22, 99].”

• cites [22] [22] Christopher Dominick Constantino, Naomi Eichorn, Eugene H Buder, J Gayle Beck, and Walter H Manning. 2020. The speaker's experience of stuttering: Measuring spontaneity. Journal of Speech, Language, and Hearing Research 63, 4 (2020), 983-1001. https://doi.org/10.1044/2019\_JSLHR-19-00068 - sourceNot checked — no full text of the source
• cites [99] [99] Seth E Tichenor and J Scott Yaruss. 2019. Stuttering as defined by adults who stutter. Journal of Speech, Language, and Hearing Research 62, 12 (2019), 43564369. https://doi.org/10.1044/2019\_JSLHR-19-00137 - sourceNot checked — no full text of the source

May combine two claims.

A recent study by PWS researchers observed a significant number of mistakes in the original event labels of Sep-28k and adjusted the labeling for over 25% of the 2,621 clips in their sample.

“a recent study conducted by a team of PWS researchers (Sridhar and Wu) observed a significant number of 'mistakes' in the original event labels and adjusted the labeling for over 25% of the 2,621 clips in their sample for their benchmarking studies [92].”

• cites [92] [92] Charan Sridhar and Shaomei Wu. 2025. J-j-j-just Stutter: Benchmarking Whisper's Performance Disparities on Different Stuttering Patterns. In Interspeech 2025 . 3753-3757. doi:10.21437/Interspeech.2025-2700 - sourceNot checked — no full text of the source

May combine two claims.

PWS have frequently contributed as data sources, but their influence on annotation frameworks, labeling guidelines, and the interpretation of stuttering behaviors remains minimal.

“While PWS have frequently contributed as data sources, their influence on other critical aspects such as annotation frameworks, labeling guidelines, and the interpretation of stuttering behaviors remains minimal.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Existing stuttered speech datasets have been developed primarily for two purposes: corporate initiatives and academic research.

“Existing stuttered speech datasets have been developed primarily for two purposes: corporate initiatives and academic research.”

Probably not a substantial claim: may not pass S3 (load-bearing).

AS-70's quality control team included one PWS member who reviewed the annotations.

“An exception to these efforts is AS-70, where the quality control team included one PWS member who reviewed the annotations.”

• cites [32] [32] Rong Gong, Hongfei Xue, Lezhi Wang, Xin Xu, Qisheng Li, Lei Xie, Hui Bu, Shaomei Wu, Jiaming Zhou, Yong Qin, et al. 2024. AS-70: A mandarin stuttered speech dataset for automatic speech recognition and stuttering event detection. arXiv preprint arXiv:2406.07256 (2024). - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

LibriStutter did not collect speech from PWS but injected synthetic stuttering sounds into read speech from fluent speakers.

“LibriStutter, another influential English stuttering dataset, did not collect speech from PWS but injected synthetic stuttering sounds into read speech from fluent speakers [53].”

• cites [53] [53] Tedd Kourkounakis, Amirhossein Hajavi, and Ali Etemad. 2021. Fluentnet: Endto-end detection of stuttered speech disfluencies with deep learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021), 2986-2999. doi:10.1109/TASLP.2021.3110146 - sourceNot checked — no full text of the source

May combine two claims.

Datasets from the ML community tend to be more scalable but are still limited by annotation granularity and representativeness.

“In contrast, datasets from the ML community tend to be more scalable, but still limited by annotation granularity and representativeness.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

SLP datasets are typically limited in size.

“As training materials for speech pathologists, SLP datasets are typically limited in size.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Datasets collected by academia primarily come from two fields: SLP and machine learning.

“Datasets collected by academia primarily come from two fields: SLP and machine learning (see Table 1 for an overview).”

May combine two claims.

AS-70 focused on Mandarin and its original annotations were still generated by non-PWS crowdworkers.

“However, AS-70 focused on Mandarin and the original annotations were still generated by non-PWS crowdworkers.”

• cites [32] [32] Rong Gong, Hongfei Xue, Lezhi Wang, Xin Xu, Qisheng Li, Lei Xie, Hui Bu, Shaomei Wu, Jiaming Zhou, Yong Qin, et al. 2024. AS-70: A mandarin stuttered speech dataset for automatic speech recognition and stuttering event detection. arXiv preprint arXiv:2406.07256 (2024). - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

This sole-coder approach aligns with the interpretative nature of qualitative research, and senior researchers were involved at all stages to enhance the reliability of the findings.

“This sole-coder approach aligns with the interpretative nature of qualitative research, and senior researchers were involved at all stages to enhance the reliability of the findings [17].”

• cites [17] [17] John L Campbell, Charles Quincy, Jordan Osserman, and Ove K Pedersen. 2013. Coding in-depth semistructured interviews: Problems of unitization and intercoder reliability and agreement. Sociological methods & research 42, 3 (2013), 294-320. https://doi.org/10.1177/0049124113500475 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All study sessions were conducted over Zoom with participants' consent to record for analysis.

“All study sessions were conducted over Zoom with participants' consent to record for analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We ensured participants were fully informed and explicitly consented before playing the recordings of their speech during the evaluation phase.

“we ensured participants were fully informed and explicitly consented before playing the recordings of their speech during the evaluation phase.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We iterated on the coding process until reaching consensus over all the themes.

“We iterated on the coding process until reaching consensus over all the themes.”

Classified as probably not a substantial claim. May combine two claims.

The data used in analysis involved notes from dataset reviews, transcripts from interviews, and video recordings of co-design workshops, all collected with participants' informed consent.

“The data used in analysis involved notes from dataset reviews, transcripts from interviews, and video recordings of co-design workshops, all collected with participants' informed consent.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The lead author conducted open inductive coding of the cumulative data and regularly discussed the themes within the research team.

“The lead author conducted open inductive coding of the cumulative data and regularly discussed the themes within the research team.”

Classified as probably not a substantial claim. May combine two claims.

Aside from descriptive statistical analysis during the dataset review, we employed reflexive thematic analysis as our primary method.

“aside from the descriptive statistical analysis conducted during our review of stuttered speech datasets in the formative stage, we employed reflexive thematic analysis as our primary method [14].”

• cites [14] [14] Virginia Braun and Victoria Clarke. 2021. Can I use TA? Should I use TA? Should I not use TA? Comparing reflexive thematic analysis and other pattern-based qualitative analytic approaches. Counselling and Psychotherapy Research 21, 1 (2021), 37-47. https://doi.org/10.1002/capr.12360 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The evaluation sessions emphasized thematic saturation in recruitment and analysis.

“Accordingly, we emphasized thematic saturation in recruitment and analysis.”

May combine two claims.

Stuttering often blends tension and avoidance behaviors, making events easy to misclassify.

“A common instance is that stuttering often blends tension and avoidance behaviors, making events easy to misclassify.”

Classified as probably not a substantial claim.

A big takeaway from the PWS discussions is the value of embodied knowledge that PWS bring to data interpretation.

“A big takeaway from the discussions is the value of embodied knowledge that PWS bring to data interpretation.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

The perception of a pause as 'unnecessary' or 'unnatural' is inherently subjective and difficult to articulate, often influenced by listeners' exposure to the speaker, embodied experiences, and judgments.

“In this instance, the perception of a pause as 'unnecessary' or 'unnatural' is inherently subjective and difficult to articulate, often influenced by listeners' exposure to the speaker, embodied experiences, and judgments.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the context of stuttering, intentional repetition is known as backtracking, a technique to pause and repeat a sound, word, or syllable to avoid an experienced block or disfluency.

“In the context of stuttering, the former is known as backtracking [40], a technique to pause and repeat a sound, word, or syllable to avoid an experienced block or disfluency.”

• cites [40] [40] Peter A Heeman, Andy McMillin, and J Scott Yaruss. 2006. An annotation scheme for complex disfluencies. In INTERSPEECH 2006 and 9th International Conference on Spoken Language Processing, INTERSPEECH 2006-ICSLP . International Speech Communication Association, 1081-1084. doi:10.21437/Interspeech.2006-55 - sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

Discussion led to three core principles for attending to potential differences in annotation: centering PWS's embodied understanding; grounding decisions in community interests; and acknowledging subjectivity in speech perception.

“Around these challenges, our discussion has led to three core principles for attending to potential differences in annotation: (1) centering PWS's embodied understanding; (2) grounding decisions in community interests; and (3) acknowledging subjectivity in speech perception.”

• AssumptionRests on an assumption: PWS's embodied knowledge should be prioritized in stuttered speech annotation.

May combine two claims.

The utility of labels remains nuanced, as each individual may have their own perspective on what it means to speak 'better' and it varies depending on contexts.

“Nevertheless, the utility of labels remains nuanced, as each individual may have their own perspective on what it means to speak 'better' and it varies depending on contexts [23].”

• cites [23] [23] Christopher Dominick Constantino, Walter H Manning, and Susan Naomi Nordstrom. 2017. Rethinking covert stuttering. Journal of Fluency Disorders 53 (2017), 26-40. https://doi.org/10.1016/j.jfludis.2017.06.001 [(2022), 92-110. https://doi.org/10.1162/tacl\_a\_00449](https://doi.org/10.1162/tacl_a_00449) - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable).

Our study represents an important step in centering disabled people and their expertise throughout the AI development pipeline.

“While prior efforts of disabilityfirst datasets often focus on the data collection process [36, 47, 63, 68, 88, 98], our study represents an important step in centering disabled people and their expertise throughout the AI development pipeline.”

May combine two claims.

The conversation about breathing reveals stuttering as a complex mix of involuntary disruptions and intentional bodily control, which blurs the boundary between stuttered and non-stuttered speech.

“The conversation above reveals stuttering as a complex mix of involuntary disruptions and intentional bodily control, which blurs the boundary between stuttered and non-stuttered speech.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Experimental evidence shows that even subtle alterations to a stuffed toy presented to human subjects can shift their perception of vowel sounds.

“Experimental evidence shows that even subtle alterations to a stuffed toy presented to human subjects can shift their perception of vowel sounds [39].”

• cites [39] [39] Jennifer Hay and Katie Drager. 2010. Stuffed toys and speech perception. Linguistics 48, 4 (2010). doi:10.1515/LING.2010.027 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The strong preference for professionals highlights annotation as a process of encoding human knowledge and perception, rather than just a mechanical or objective task.

“This strong preference for professionals highlights annotation as a process of encoding human knowledge and perception, rather than just a mechanical or objective task.”

Probably not a substantial claim: may not pass S4 (based).

Our work offers the first attempt to bring disability-first dataset practices into AI dataset annotation.

“Our work offers the first attempt to bring disability-first dataset practices into AI dataset annotation.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The complex, organic, and often contested nature of stuttering requires embodied knowledge from PWS and, in some cases, insight from the speakers themselves to produce faithful annotations.

“The complex, organic, and often contested nature of stuttering. Producing faithful annotations thus requires embodied knowledge from PWS and, in some cases, insight from the speakers themselves.”

Probably not a substantial claim: may not pass S2 (checkable).

Annotation is better understood as a form of collective sense-making, an ongoing effort to approach stuttering not as a fixed phenomenon, but as a multifaceted process requiring continuous reflection, interpretation, and care.

“These perspectives suggest that annotation is better understood as a form of collective sense-making, an ongoing effort to approach stuttering not as a fixed phenomenon, but as a multifaceted process requiring continuous reflection, interpretation, and care.”

Probably not a substantial claim: may not pass S2 (checkable).

The annotation of stuttered speech requires an open-ended approach, allowing space for uncertainty and iterative refinements.

“This observation suggests that the annotation of stuttered speech requires an openended approach, allowing space for uncertainty and iterative refinements.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Our disability-first approach has potential to inform the development of accessibility datasets more broadly.

“While centered on stuttering, our disability-first approach has potential to inform the development of accessibility datasets more broadly.”

May combine two claims.

In our study, disability expertise is reflected in both the embodied experience of stuttering and the awareness of community interests; both are essential for capturing stuttering and addressing annotation challenges, such as the need for situated evaluation.

“In our study, this expertise is reflected in both the embodied experience of stuttering and the awareness of community interests; both are essential for capturing stuttering and addressing annotation challenges, such as the need for situated evaluation.”

Classified as probably not a substantial claim.

A key lesson learned is the value of involving stakeholders with dense and diverse expertise.

“A key lesson we learned is the value of involving stakeholders with dense and diverse expertise.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The stuttering community advocates for more diverse evaluation measures, such as spontaneity rather than fluency.

“Reflecting similar ideas, the stuttering community advocates for more diverse evaluation measures, such as spontaneity rather than fluency [22].”

• cites [22] [22] Christopher Dominick Constantino, Naomi Eichorn, Eugene H Buder, J Gayle Beck, and Walter H Manning. 2020. The speaker's experience of stuttering: Measuring spontaneity. Journal of Speech, Language, and Hearing Research 63, 4 (2020), 983-1001. https://doi.org/10.1044/2019\_JSLHR-19-00068 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The case of stuttering provides yet another example showing that label inconsistencies are not always problems to be 'settled'; rather, they reflect the diversity of human perception and can prompt ongoing, reflexive, and collaborative efforts to better understand human experiences.

“The case of stuttering provides yet another example showing that label inconsistencies are not always problems to be 'settled'; rather, they reflect the diversity of human perception and can prompt ongoing, reflexive, and collaborative efforts to better understand human experiences.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Sheehan's iceberg theory of stuttering holds that stuttering is like an iceberg, with only a small part above the waterline and a much bigger part below.

“In the context of stuttering, Sheehan proposed the iceberg theory of stuttering [89], noting that 'stuttering is like an iceberg, with only a small part above the waterline and a much bigger part below.'”

• cites [89] [89] Joseph G Sheehan. 1970. Stuttering: Research and therapy. (1970). -Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

We carefully recruited participants who possess a wide range of expertise including embodied stuttering experiences, experiences as organizers and advocates of the stuttering community, experiences with stuttered speech annotations, and technical proficiency.

“Faced with these challenges, we carefully recruited participants who possess a wide range of expertise including embodied stuttering experiences, experiences as the organizers and advocates of the stuttering community, experiences with stuttered speech annotations, and technical proficiency.”

Classified as probably not a substantial claim.

The gap between what the current set of labels can capture and the richness of stuttering experiences reflects a long-standing question in HCI: the distinction between subjective, internal human experience and external observation.

“The gap between what the current set of labels can capture and the richness of stuttering experiences reflects a long-standing question in HCI: the distinction between subjective, internal human experience and external observation [59].”

• cites [59] [59] Lucian Leahu, Steve Schwenk, and Phoebe Sengers. 2008. Subjective objectivity: negotiating emotional meaning. In Proceedings of the 7th ACM conference on Designing interactive systems . 425-434. doi:10.1145/1394445.1394491 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable).

The intimacy and complexity of embodied knowledge pose deeper challenges for AI models to capture the full richness of disability.

“Moreover, the intimacy and complexity of embodied knowledge pose deeper challenges for AI models to capture the full richness of disability.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The importance of disability expertise in disability data annotation is shared with many other disabilities, as the interpretation of data often relies on embodied knowledge unique to disabled bodies.

“While stuttering can be particularly internalized and subjective, the importance of disability expertise in disability data annotation is shared with many other disabilities, as the interpretation of data often relies on embodied knowledge unique to disabled bodies.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Prior research suggested recruiting disabled people is a key challenge in creating disability-first datasets.

“Prior research suggested recruiting disabled people is a key challenge in creating disability-first datasets [78].”

• cites [78] [78] Joon Sung Park, Danielle Bragg, Ece Kamar, and Meredith Ringel Morris. 2021. Designing an Online Infrastructure for Collecting AI Data From People With Disabilities. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT '21) . Association for Computing Machinery, New York, NY, USA, 52-63. doi:10.1145/3442188.3445870 - sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Our guidelines seek to capture the lived knowledge of stuttering from PWS.

“Instead, our guidelines seek to capture the lived knowledge of stuttering from PWS.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

While essential, the inclusion of disabled individuals is not a panacea, especially as the notion of 'meaningful access' remains contested and evolving.

“Crucially, while essential, the inclusion of disabled individuals is not a panacea [25, 91], especially as the notion of 'meaningful access' remains contested and evolving.”

• cites [25] [25] Maartje De Meulder. 2021. Is 'good enough' good enough? Ethical and responsible development of sign language technologies. In Proceedings of the 1st international workshop on automatic translation for signed and spoken languages (AT4SSL) . 12-22. https://aclanthology.org/2021.mtsummit-at4ssl.2/ -Not checked yet

  Not checked in the free run.

• cites [91] [91] Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano. 2022. Participation Is not a Design Fix for Machine Learning. In Proceedings of the 2nd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (Arlington, VA, USA) (EAAMO '22) . Association for Computing Machinery, New York, NY, USA, Article 1, 6 pages. doi:10.1145/3551624.3555285 - sourceNot checked — no full text of the source

May combine two claims.

Missing disability expertise affects not only the accuracy of specific labels but also risks embedding ableism within the underlying AI training paradigm.

“Missing disability expertise affects not only the accuracy of specific labels but also risks embedding ableism within the underlying AI training paradigm.”

Probably not a substantial claim: may not pass S4 (based).

Speakers' internal experience of their speech often differs from what listeners are able to perceive.

“Speakers' internal experience of their speech often differs from what listeners are able to perceive.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Additional challenges could surface when extending our approach to more diverse speech patterns.

“Additional challenges could surface when extending our approach to more diverse speech patterns.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Listening to non-verbal breathing sounds in front or end of sentences is particularly important for detecting small blocks.

“Listen to non-verbal breathing sounds in front or end of the sentences. This is particularly important for detecting small blocks.”

May combine two claims.

As PWS participants imagined the use of their annotated speech, the interpretation of labels is never final but continuously evolving; for example, inaccurate annotations of stuttering type might still be useful in helping conversation partners become aware of the existence of stuttering, which could enhance listeners' comprehension of stuttered speech.

“As PWS participants imagined the use of their annotated speech, the interpretation of labels is never final but continuously evolving; for example, inaccurate annotations of stuttering type might still be useful in helping conversation partners become aware of the existence of stuttering, which could enhance listeners' comprehension of stuttered speech [15].”

• cites [15] [15] C. T. Byrd, M. McGill, Z. Gkalitsiou, and C. Cappellini. 2017. The effects of selfdisclosure on male and female perceptions of individuals who stutter. American Journal of Speech-Language Pathology 26, 1 (2017), 69-80. https://doi.org/10. 1044/2016\_AJSLP-15-0164 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Our perspectives, including those of PWS, are inevitably influenced by pre-existing frameworks, such as those from SLPs.

“Second, although the stuttering community has been exploring new ways to understand stuttering, our perspectives, including those of PWS, are inevitably influenced by pre-existing frameworks, such as those from SLPs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Our study relies on a relatively small number of participants.

“First, our study relies on a relatively small number of participants.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Although VQA has traditionally been treated as a task with a single ground-truth label, recent work highlights the inherent subjectivity in visual information seeking, showing that blind users integrate multiple perspectives to shape their visual understanding, whether with AI tools or humans.

“Although VQA has traditionally been treated as a task with a single ground-truth label [4], recent work highlights the inherent subjectivity in visual information seeking, showing that blind users integrate multiple perspectives to shape their visual understanding, whether with AI tools or humans [94].”

• cites [4] [4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision . 2425-2433. - sourceNot checked — no full text of the source
• cites [94] [94] Xinru Tang, Ali Abdolrahmani, Darren Gergle, and Anne Marie Piper. 2025. Everyday Uncertainty: How Blind People Use GenAI Tools for Information Access. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25) . Association for Computing Machinery, New York, NY, USA, Article 63, 17 pages. doi:10.1145/3706598.3713433 - sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

Part of our guidelines was still built off existing labeling frameworks.

“Part of our guidelines was still built off existing labeling frameworks.”

Probably not a substantial claim: may not pass S2 (checkable).

Another takeaway from our study is that interpretation of speech labels should extend beyond the moment of annotation.

“Another takeaway from our study is that interpretation of speech labels should extend beyond the moment of annotation.”

Probably not a substantial claim: may not pass S2 (checkable).

A multiplicity-aware perspective offers a valuable paradigm shift in data practices related to human perception.

“Although task-specific research is needed, a multiplicity-aware perspective offers a valuable paradigm shift in data practices related to human perception.”

Probably not a substantial claim: may not pass S2 (checkable).

These varying priorities suggest that annotation should reflect the goals of the intended application rather than enforcing uniformity.

“These varying priorities suggest that annotation should reflect the goals of the intended application rather than enforcing uniformity.”