CoChat

Dataset Artefacts in Anti-Spoofing Systems: A Case Study on the ASVspoof 2017 Benchmark

Open access
Chettri, Bhusan, Benetos, Emmanouil, Sturm, Bob L. T. · 2020 · IEEE/ACM Transactions on Audio, Speech, and Language Processing

The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.

Abstract

The Automatic Speaker Verification Spoofing and Countermeasures Challenges motivate research in protecting speech biometric systems against a variety of different access attacks. The 2017 edition focused on replay spoofing attacks, and involved participants building and training systems on a provided dataset (ASVspoof 2017). More than 60 research papers have so far been published with this dataset, but none have sought to answer why countermeasures appear successful in detecting spoofing attacks. This article shows how artefacts inherent to the dataset may be contributing to the apparent success of published systems. We first inspect the ASVspoof 2017 dataset and summarize various artefacts present in the dataset. Second, we demonstrate how countermeasure models can exploit these artefacts to appear successful in this dataset. Third, for reliable and robust performance estimates on this dataset we propose discarding nonspeech segments and silence before and after the speech utterance during training and inference. We create speech start and endpoint annotations in the dataset and demonstrate how using them helps countermeasure models become less vulnerable from being manipulated using artefacts found in the dataset. Finally, we provide several new benchmark results for both frame-level and utterance-level models that can serve as new baselines on this dataset.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/taslp.2020.3036777

Cites

65 of 65 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Block diagram: intervention pipeline towards understanding the influence of artefacts on the predictions of countermeasure models.
Fig. 1 Block diagram: intervention pipeline towards understanding the influence of artefacts on the predictions of countermeasure models.CC-BY-NC-SA · From the PDF
Fig. 2 Spectrogram (top) and raw audio (bottom) of 'Ok google'. Left represents a bonafide example and right its replayed version. The green rectangular box highlights the original audio and the red box shows signal replication to create a fixed duration (4 seconds) input representation. Two use cases are reflected: pattern difference and BCS. It shows how artefacts spread while creating an input representation with fixed duration.
Fig. 2 Spectrogram (top) and raw audio (bottom) of 'Ok google'. Left represents a bonafide example and right its replayed version. The green rectangular box highlights the original audio and the red box shows signal replication to create a fixed duration (4 seconds) input representation. Two use cases are reflected: pattern difference and BCS. It shows how artefacts spread while creating an input representation with fixed duration.CC-BY-NC-SA · From the PDF
Fig. 3 Score distributions of spoof files in the evaluation set that were originally detected correctly by CNN 2 and are now misclassified after adding 100 ms silence at random time locations.
Fig. 3 Score distributions of spoof files in the evaluation set that were originally detected correctly by CNN 2 and are now misclassified after adding 100 ms silence at random time locations.CC-BY-NC-SA · From the PDF
Fig. 4 Proposed countermeasure design for reliable performance estimates.
Fig. 4 Proposed countermeasure design for reliable performance estimates.CC-BY-NC-SA · From the PDF
Fig. 5 Performance (EER%) on the evaluation set using models trained with manual and automatic speech endpoint annotations.
Fig. 5 Performance (EER%) on the evaluation set using models trained with manual and automatic speech endpoint annotations.CC-BY-NC-SA · From the PDF
Image 6
CC-BY-NC-SA · From the PDF
Image 7
CC-BY-NC-SA · From the PDF
Image 8
CC-BY-NC-SA · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 2

  1. 8. K. A. Lee, B. Ma, and H. Li, "Speaker verification makes its debut in smartphone," IEEE Signal Processing Society SLTC Newsletter, 2013.
  2. 58. W. Cai, C. Danwei, W. Liu, G. Li, and M. Li, "Countermeasures for automatic speaker verification replay spoofing attack: On data augmentation, feature representation, classification and fusion," 2017, pp. 17-21. · DOI

Cited by

See all 46

Metadata

Authors
Chettri, B., Benetos, E., & Sturm, B. L. T.
Affiliations
  • University of London (GB)
  • KTH Royal Institute of Technology (SE)
Venue
IEEE/ACM Transactions on Audio, Speech, and Language Processing
Published
2020
Volume
28
Pages
3018-3028
ISSN
  • 2329-9290 (ISSN-L)
  • 2329-9304
OpenAlex
W3092742905
Cited by
35
References
61
Citations by year
2026: 3 · 2025: 11 · 2024: 6 · 2023: 6 · 2022: 6 · 2021: 3
FWCI
1.47
Type
journal article
Language
en
Open access
Locations
  • IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University) (submittedVersion)
  • Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Keywords
ASVspoof 2017, anti-spoofing countermeasures, speaker verification, anti-spoofing, benchmark results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.