Cited by 13Speech Recognition and SynthesisSpeech and Audio ProcessingEthics and Social Impacts of AI

FairSSD: Understanding Bias in Synthetic Speech Detectors

Open access
Singh Yadav, Amit Kumar, Bhagtani, Kratika, Salvi, Davide, Bestagini, Paolo, Delp, Edward J.·2024·2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Abstract

Methods that can generate synthetic speech which is perceptually indistinguishable from speech recorded by a human speaker, are easily available. Several incidents report misuse of synthetic speech generated from these methods to commit fraud. To counter such misuse, many methods have been proposed to detect synthetic speech. Some of these detectors are more interpretable, can generalize to detect synthetic speech in the wild and are robust to noise. However, limited work has been done on understanding bias in these detectors. In this work, we examine bias in existing synthetic speech detectors to determine if they will unfairly target a particular gender, age and accent group. We also inspect whether these detectors will have a higher misclassification rate for bona fide speech from speech-impaired speakers w.r.t fluent speakers. Extensive experiments on 6 existing synthetic speech detectors using more than 0.9 million speech signals demonstrate that most detectors are gender, age and accent biased, and future work is needed to ensure fairness. To support future research, we release our evaluation dataset, models used in our study and source code at https://gitlab.com/viper-purdue/fairssd.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvprw63382.2024.00445

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SSD systems are biased, exhibiting higher false-positive rates for bona fide speech from male speakers, adolescents, the elderly, and speakers with South Asian and Australian English accents.

    “The results revealed that SSD systems are biased, exhibiting higher false-positive rates for bona fide speech from male speakers, adolescents, and the elderly, as well as speakers with South Asian and Australian English accents.”
  • SSD systems are particularly unfair to speech-impaired speakers.

    “Additionally, SSD systems were found to be particularly unfair to speech-impaired speakers.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. [7][7] H. Kim, S. Kim, J. Yeom, and S. Yoon, 'UnitSpeech: Speakeradaptive Speech Synthesis with Untranscribed Data,' Proceedings of the ISCA Interspeech , pp. 3038-3042, August 2023, Dublin, Ireland. 1 -
  2. [17][17] K. Li, X.-M. Zeng, J.-T. Zhang, and Y. Song, 'Convolutional recurrent neural network and multitask learning for manipulation region location,' Proceedings of IJCAI Workshop on Deepfake Audio Detection and Analysis , pp. 18-22, August 2023, Macao. 1, 2 -
  3. [52][52] Y. Gong, C.-I. Lai, Y.-A. Chung, and J. Glass, 'SSAST: SelfSupervised Audio Spectrogram Transformer,' Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 10, pp. 10 699-10 709, October 2022, Virtual. 2 -
  4. [77][77] A. Nautsch, X. Wang, N. Evans, T. H. Kinnunen, V. Vestman, M. Todisco, H. Delgado, M. Sahidullah, J. Yamagishi, and K. A. Lee, 'ASVspoof 2019: Spoofing Countermeasures for the Detection of Synthesized, Converted and Replayed Speech,' IEEE Transactions on Biometrics, Behavior, and Identity Science , vol. 3, no. 2, pp. 252-265, February 2021. 13, 14 -
  5. [79][79] I. Loshchilov and F. Hutter, 'Decoupled Weight Decay Regularization,' Proceedings of the International Conference on Learning Representations , May 2019, New Orleans, LA. 13 -
  6. [80][80] O. Tange, 'Gnu parallel - the command-line power tool,' ;login: The USENIX Magazine , vol. 36, no. 1, pp. 42-47, Feb 2011. [Online]. Available: http://www.gnu.org/s/parallel 14

Cited by

Sorted by citations

Authors
Singh Yadav, A. K., Bhagtani, K., Salvi, D., Bestagini, P., & Delp, E. J.
Affiliations
  • Purdue University,Video and Image Processing Lab (VIPER),West Lafayette,Indiana,USA
  • Purdue University West Lafayette (US)
  • Informazione e Bioingegneria,Dipartimento di Elettronica,Milano,Italy
Venue
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
Publisher
IEEE
Published
2024-06-17
Pages
4418-4428
OpenAlex
W4402917138
Cited by
12
References
67
Citations by year
2026: 3 · 2025: 7 · 2024: 2
FWCI
3.46
Type
conference paper
Language
en
Open access
Locations
  • IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Air Force Research Laboratory
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
synthetic speech detection, algorithmic bias, fairness, gender bias, age bias, accent bias, speech disorders, noise robustness, evaluation dataset

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review