CoChat

ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild

Open access
Liu, Xuechen, Wang, Xin, Sahidullah, Md, Patino, Jose, Delgado, Héctor, Kinnunen, Tomi, Todisco, Massimiliano, Yamagishi, Junichi, Evans, Nicholas, Nautsch, Andreas, Lee, Kong Aik · 2023 · IEEE/ACM Transactions on Audio, Speech, and Language Processing

The DOI matches its Crossref record; 19 of the 67 references it cites were found in a registry or an index, and 20 are not checked yet; 18 of the 41 sources its claims cite were checked for support.

Abstract

Benchmarking initiatives support the meaningful comparison of competing solutions to prominent problems in speech and language processing. Successive benchmarking evaluations typically reflect a progressive evolution from ideal lab conditions towards to those encountered in the wild. ASVspoof, the spoofing and deepfake detection initiative and challenge series, has followed the same trend. This article provides a summary of the ASVspoof 2021 challenge and the results of 54 participating teams that submitted to the evaluation phase. For the logical access (LA) task, results indicate that countermeasures are robust to newly introduced encoding and transmission effects. Results for the physical access (PA) task indicate the potential to detect replay attacks in real, as opposed to simulated physical spaces, but a lack of robustness to variations between simulated and real acoustic environments. The Deepfake (DF) task, new to the 2021 edition, targets solutions to the detection of manipulated, compressed speech data posted online. While detection solutions offer some resilience to compression effects, they lack generalization across different source datasets. In addition to a summary of the top-performing systems for each task, new analyses of influential data factors and results for hidden data subsets, the article includes a review of post-challenge results, an outline of the principal challenge limitations and a road-map for the future of ASVspoof.

Verified 2026-11-22 · 20 references not checked yet

Overview

CoChat Summary

## TL;DR

ASVspoof 2021 benchmarked spoofing and deepfake speech detection under more realistic “in the wild” conditions across three tasks: logical access, physical access, and deepfake. The main takeaway is that logical-access detection remained fairly robust under transmission distortions, while physical-access replay detection was hardest and deepfake detection suffered from poor generalization across source datasets.

## Key findings

  • The challenge drew submissions from 54 teams; 34 teams submitted the required system description.
  • Logical access (LA): many systems outperformed the best baseline B03; the best system T23 achieved 0.1815 min t-DCF / 0.89% EER on the progress set and 0.2177 min t-DCF / 1.32% EER on the evaluation set.
  • Physical access (PA): all systems were substantially worse than the ASV floor of 0.12; the best system T07 had 0.6736 min t-DCF / 23.60% EER on progress and 0.6824 / 24.25% on evaluation.
  • Deepfake (DF): performance dropped sharply from progress to evaluation; the best system T23 went from 0.24% EER on progress to 15.64% EER on evaluation, and 23 of 33 systems had progress-set EERs below 10%, while all evaluation-set EERs exceeded 15%.
  • Hidden-subset analyses showed that removing non-speech harmed performance: for example, in LA the baseline B04 rose from 0.426 min t-DCF with non-speech to 0.908 when non-speech at the two ends was removed, and to 0.961 on the hidden track without any non-speech.

## Methods

  • The paper reports the ASVspoof 2021 challenge as three independent sub-challenges: LA (spoofed speech over telephony), PA (replay attacks in real physical spaces), and DF (deepfake detection in compressed audio).
  • Data were distributed in FLAC, 16 kHz, 16-bit format; participants trained on ASVspoof 2019 partitions, with task-specific evaluation databases for 2021.
  • Main evaluation metrics were min t-DCF for LA and PA, and EER for DF; the paper analyzes challenge results, hidden subsets, influential data factors, and post-challenge baseline experiments on features, backends, losses, and augmentation.

## Limitations

  • The paper explicitly notes that ASVspoof 2021 is not without limitations and points to future work on more realistic audio conditions, including background noise for LA, human-talker recordings instead of loudspeakers for PA, and real social-media data for DF.
  • It also states that non-speech intervals can be a cue, which is a limitation because they may reflect database artifacts rather than robust spoofing cues.
  • Evident from the text: the challenge uses fixed training/data policies and largely English read speech from VCTK, which the paper itself links to limited generalization and over-fitting across source domains.

Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/taslp.2023.3285283

Cites

19 of 67 references verified0 citation errors · 28 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
From the PDF
Image 2
From the PDF
Image 3
From the PDF
Fig. 2 Illustration of talker-to-ASV distance D s ∈ { D1 , D2 , ⋯ , D6 } . The full set of ASV microphones { M1 , M2 , M3 } are used in all positions. Attackerto-ASV distance D ′ s can be plotted similarly by replacing the talker with the attacker replay device. Distances and angles are depicted for D1 and D2 only; those for D3-6 are listed in Table III.
Fig. 2 Illustration of talker-to-ASV distance D s ∈ { D1 , D2 , ⋯ , D6 } . The full set of ASV microphones { M1 , M2 , M3 } are used in all positions. Attackerto-ASV distance D ′ s can be plotted similarly by replacing the talker with the attacker replay device. Distances and angles are depicted for D1 and D2 only; those for D3-6 are listed in Table III.From the PDF
Fig. 3 ASVspoof 2021 results for LA, PA, DF tracks. Results shown in terms of pooled normalised minimum t-DCF and pooled EER (%). Blue line marks ASV floor. Only teams that submitted system description are marked with ID.
Fig. 3 ASVspoof 2021 results for LA, PA, DF tracks. Results shown in terms of pooled normalised minimum t-DCF and pooled EER (%). Blue line marks ASV floor. Only teams that submitted system description are marked with ID.From the PDF
Image 6
From the PDF
Fig. 4 Boxplots of evaluation set min t-DCF or EERs of top-10 submissions decomposed over different factors. Markers are top-1 submission ( ⋄ ), B01 (o), B02 ( ◁ ), B03 ( ⭐ ), and B04 ( × ).
Fig. 4 Boxplots of evaluation set min t-DCF or EERs of top-10 submissions decomposed over different factors. Markers are top-1 submission ( ⋄ ), B01 (o), B02 ( ◁ ), B03 ( ⭐ ), and B04 ( × ).From the PDF
Fig. 6 Break-down of the EER in the VCC 2018/2020 subset of the DF track, according to the codecs and the type of vocoder used by the VC system.
Fig. 6 Break-down of the EER in the VCC 2018/2020 subset of the DF track, according to the codecs and the type of vocoder used by the VC system.From the PDF
Fig. 7 Comparison of top-10 CMs performance on evaluation subset (with non-speech) and hidden subset (w/o non-speech). For LA and DF, min t-DCFs over VC- and TTS-based spoofing attacks are plotted separately. Markers are top-1 system ( ⋄ ), B01 (o), B02 ( ◁ ), B03 ( ⭐ ), and B04 ( × ).
Fig. 7 Comparison of top-10 CMs performance on evaluation subset (with non-speech) and hidden subset (w/o non-speech). For LA and DF, min t-DCFs over VC- and TTS-based spoofing attacks are plotted separately. Markers are top-1 system ( ⋄ ), B01 (o), B02 ( ◁ ), B03 ( ⭐ ), and B04 ( × ).From the PDF
Fig. 8 Score distributions of bonafide (top) and spoof (bottom) for top-5 DF subsystem and baseline B04, broken down according to three data sources used.
Fig. 8 Score distributions of bonafide (top) and spoof (bottom) for top-5 DF subsystem and baseline B04, broken down according to three data sources used.From the PDF
Fig. 9 Illustration of top-5 systems for the LA scenario. The descending order is based on performance via minimum t-DCF values on evaluation set. Numbers in brackets are the values. Dash lines indicate steps/modules which are only applied for training, while solid lines indicate ones applied for both training and testing.
Fig. 9 Illustration of top-5 systems for the LA scenario. The descending order is based on performance via minimum t-DCF values on evaluation set. Numbers in brackets are the values. Dash lines indicate steps/modules which are only applied for training, while solid lines indicate ones applied for both training and testing.From the PDF
Fig. 10 Illustration of top-5 systems for the PA scenario. The descending order is based on performance via minimum t-DCF values on evaluation set. Numbers in brackets are the values. Dash lines indicate steps/modules which are only applied for training, while solid lines indicate ones applied for both training and testing.
Fig. 10 Illustration of top-5 systems for the PA scenario. The descending order is based on performance via minimum t-DCF values on evaluation set. Numbers in brackets are the values. Dash lines indicate steps/modules which are only applied for training, while solid lines indicate ones applied for both training and testing.From the PDF
Fig. 11 Illustration of top-5 systems for the DF scenario. The descending order is based on performance via equal error rates on evaluation set. Numbers in brackets are the error rates. Dash lines indicate steps/modules which are only applied for training, while solid lines indicate ones applied for both training and testing.
Fig. 11 Illustration of top-5 systems for the DF scenario. The descending order is based on performance via equal error rates on evaluation set. Numbers in brackets are the error rates. Dash lines indicate steps/modules which are only applied for training, while solid lines indicate ones applied for both training and testing.From the PDF
Fig. 12 Comparison of CMs EERs on evaluation set and hidden tracks. Blue dots correspond to top 1-10 submissions. Red dots are other submissions. Best viewed in color.
Fig. 12 Comparison of CMs EERs on evaluation set and hidden tracks. Blue dots correspond to top 1-10 submissions. Red dots are other submissions. Best viewed in color.From the PDF
Fig. 13 Boxplots of min t-DCF of top-10 submissions decomposed over different factors on PA simulated subset (hidden track 1). Markers are top-1 submission ( ⋄ ), B01 (o), B02 ( ◁ ), B03 ( ⭐ ), and B04 ( × ). Note that data in PA hidden track 1 only covers D ′ s = d4 and D s = D4.
Fig. 13 Boxplots of min t-DCF of top-10 submissions decomposed over different factors on PA simulated subset (hidden track 1). Markers are top-1 submission ( ⋄ ), B01 (o), B02 ( ◁ ), B03 ( ⭐ ), and B04 ( × ). Note that data in PA hidden track 1 only covers D ′ s = d4 and D s = D4.From the PDF

Verification

Claims the paper makes

15 Valid · 5 Not valid · 8 Not checked · 129 Own results and interpretations

77 more claims · 4 with their cited sources checked

Create a free account to see all 157 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 42
  • Probably not a substantial claim. 41
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 1

  1. [44] J. M. Mart´ ın-Do˜ nas and A. ´ Alvarez, 'The Vicomtech audio deepfake detection system based on Wav2vec2 for the 2022 ADD challenge,' in Proc. ICASSP , 2022, pp. 9241-9245. -

Cited by

See all 312

Metadata

Authors
Liu, X., Wang, X., Sahidullah, M., Patino, J., Delgado, H., Kinnunen, T., Todisco, M., Yamagishi, J., Evans, N., Nautsch, A., & Lee, K. A.
Affiliations
  • Centre National de la Recherche Scientifique (FR)
  • National Institute of Informatics (JP)
  • EURECOM (FR)
  • Nuance Communications, Madrid, Spain
  • University of Eastern Finland (FI)
  • Agency for Science, Technology and Research (SG)
Venue
IEEE/ACM Transactions on Audio, Speech, and Language Processing
Published
2023
Volume
31
Pages
2507-2522
ISSN
  • 2329-9290 (ISSN-L)
  • 2329-9304
OpenAlex
W4381198892
Cited by
312
References
58
Citations by year
2026: 93 · 2025: 118 · 2024: 71 · 2023: 28
FWCI
29.14
Type
journal article
Language
en
Open access
Locations
  • IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Agence Nationale de la Recherche (ANR-18-JSTS-0001)
  • Japan Science and Technology Agency through Project VoicePersonae (JPMJCR18A6)
  • Academy of Finland (349605)
  • SPEECHFAKES
  • MEXT KAKENHI (21K17775, 21H04906)
  • Region Grand Est, France, joint doctoral funding from Inria and UEF
  • Science and Engineering Research Council, Agency for Science, Technology and Research (A*STAR), Singapore
  • Council Research Fund (CR-2021-005)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
Keywords
anti-spoofing, speech deepfake detection, replay attacks, speaker verification security

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.