Cited by 339Speech Recognition and SynthesisLinguistic Variation and MorphologyEthics and Social Impacts of AI

Gender and Dialect Bias in YouTube’s Automatic Captions

Open access
Tatman, Rachael·2017·Proceedings of the First ACL Workshop on Ethics in Natural Language Processing

Abstract

This project evaluates the accuracy of YouTube's automatically-generated captions across two genders and five dialects of English.Speakers' dialect and gender was controlled for by using videos uploaded as part of the "accent tag challenge", where speakers explicitly identify their language background.The results show robust differences in accuracy across both gender and dialect, with lower accuracy for 1) women and 2) speakers from Scotland.This finding builds on earlier research finding that speaker's sociolinguistic identity may negatively impact their ability to use automatic speech recognition, and demonstrates the need for sociolinguistically-stratified validation of systems.

Integrity signals

Registry

Crossref · 10.18653/v1/w17-1606

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Models can perform well on average but poorly on some users.

    “These models can perform well on average but poorly on some users (Tatman, 2017; Caldas et al., 2018; Li et al., 2019b; Koenecke et al., 2020).”

    From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Tatman, 2017

  • Performance disparities across individuals have been observed in a wide range of tasks and applications, including natural language processing, automatic speech recognition, federated learning, and medical imaging.

    “Performance disparities across individuals have been observed in a wide range of tasks and applications, including in natural language processing (Geva et al., 2019), automatic speech recognition (Koenecke et al., 2020; Tatman, 2017), federated learning (Li et al., 2019b; Caldas et al., 2018), and medical imaging (Badgeley et al., 2019).”

    From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Tatman, 2017

  • Models can perform well on average but poorly on some individuals.

    “These models can perform well on average but poorly on some individuals (Tatman, 2017; Caldas et al., 2018; Li et al., 2019b; Koenecke et al., 2020).”

    From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Tatman, 2017

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Tatman, R.
Affiliations
University of Washington (US)
Corresponding author
Tatman, Rachael
Venue
Proceedings of the First ACL Workshop on Ethics in Natural Language Processing
Published
2017
Pages
53-59
OpenAlex
W2607719644
Cited by
338
References
36
Citations by year
2026: 34 · 2025: 33 · 2024: 35 · 2023: 50 · 2022: 56 · 2021: 62 · 2020: 33 · 2019: 21 · 2018: 11 · 2017: 3
FWCI
8.48
Type
conference paper
Language
en
Open access
Licence
cc-by
Funders
  • National Science Foundation (1256082)
  • University of Washington
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Social Sciences › Linguistics and Language › Linguistic Variation and Morphology
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
speech recognition, speech recognition accuracy, gender bias, Scottish English, accent, sociolinguistic identity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review