CoChat

Audio deepfakes: A survey

Open access
Khanjani, Zahra, Watson, Gabrielle, Janeja, Vandana P. · 2023 · Frontiers in Big Data

The DOI matches its Crossref record; 127 of the 143 references it cites were found in a registry or an index; 32 of the 68 sources its claims cite were checked for support.

Abstract

A deepfake is content or material that is synthetically generated or manipulated using artificial intelligence (AI) methods, to be passed off as real and can include audio, video, image, and text synthesis. The key difference between manual editing and deepfakes is that deepfakes are AI generated or AI manipulated and closely resemble authentic artifacts. In some cases, deepfakes can be fabricated using AI-generated content in its entirety. Deepfakes have started to have a major impact on society with more generation mechanisms emerging everyday. This article makes a contribution in understanding the landscape of deepfakes, and their detection and generation methods. We evaluate various categories of deepfakes especially in audio. The purpose of this survey is to provide readers with a deeper understanding of (1) different deepfake categories; (2) how they could be created and detected; (3) more specifically, how audio deepfakes are created and detected in more detail, which is the main focus of this paper. We found that generative adversarial networks (GANs), convolutional neural networks (CNNs), and deep neural networks (DNNs) are common ways of creating and detecting deepfakes. In our evaluation of over 150 methods, we found that the majority of the focus is on video deepfakes, and, in particular, the generation of video deepfakes. We found that for text deepfakes, there are more generation methods but very few robust methods for detection, including fake news detection, which has become a controversial area of research because of the potential heavy overlaps with human generation of fake content. Our study reveals a clear need to research audio deepfakes and particularly detection of audio deepfakes. This survey has been conducted with a different perspective, compared to existing survey papers that mostly focus on just video and image deepfakes. This survey mainly focuses on audio deepfakes that are overlooked in most of the existing surveys. This article's most important contribution is to critically analyze and provide a unique source of audio deepfake research, mostly ranging from 2016 to 2021. To the best of our knowledge, this is the first survey focusing on audio deepfakes generation and detection in English.

Overview

CoChat Summary

## TL;DR

This 2023 survey argues that audio deepfakes are under-covered relative to video/image deepfakes and provides a detailed overview of how audio deepfakes are generated, detected, and benchmarked. It reports that across the surveyed literature, generation methods have advanced faster than detection, and that audio deepfake research needs better generalization and more robust defenses.

## Key findings

  • The authors surveyed over 150 methods and say that audio deepfakes are overlooked in existing surveys; they claim this is the first survey focusing on audio deepfake generation and detection in English.
  • In their review, they found that GANs, CNNs, and DNNs are common ways of creating and detecting deepfakes, and that most surveyed work focuses on video deepfakes, especially generation.
  • For text deepfakes, they report more generation methods but very few robust detection methods, including fake news detection.
  • For audio generation quality, they report that Mean Opinion Score (MOS) is better with single-speaker training data than with multi-speaker datasets, and that sampling rates below 16 kHz significantly reduce perceived quality.
  • For replay-attack detection, they note one deep convolutional network method achieved EER = 0% on the ASVspoof 2017 development and evaluation set, compared with prior bests of 12% development EER and 2.76% evaluation EER.

## Methods

  • The paper is a survey/review, not an empirical experiment: it synthesizes prior work across audio, text, video, and image deepfakes, with special focus on audio.
  • It states that the review covers more than 150 methods, with detailed discussion of audio generation, detection, datasets, and architectures from roughly 2016 to 2021.
  • The authors also summarize common datasets and models used in audio deepfake research, including ASVspoof 2015/2017/2019/2021, FoR, and WaveFake, plus architectures such as WaveNet, Tacotron, Tacotron 2, WaveGlow, Deep Voice 3, MelNet, ResNet, TCN, and biLSTM.

## Limitations

  • The paper explicitly says it does not discuss the details of the frameworks used for all deepfake categories; instead, it gives more detailed attention to audio and only recent trends for the other categories.
  • It also says that the survey of audio deepfakes is not exhaustive for every framework, relying on supplementary tables for summaries and guides.
  • Evident from the text: several sections are high-level and descriptive rather than systematic meta-analysis, so the paper does not report a unified quantitative comparison across all methods.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3389/fdata.2022.1001063

Cites

127 of 127 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 (A) Audio deepfake generation, text-to-speech. (B) Audio deepfake generation, voice conversion. (C) Audio fake generation frameworks.
Figure 1 (A) Audio deepfake generation, text-to-speech. (B) Audio deepfake generation, voice conversion. (C) Audio fake generation frameworks.CC BY · PubMed Central
Figure 2 Audio deepfake detection frameworks.
Figure 2 Audio deepfake detection frameworks.CC BY · PubMed Central
Figure 3 Replay attack end-to-end detection. The audio input comes to group delay grams which are novel time-frequency representations of an utterance. Also, the novel attention mechanism softly weights the GD grams. The ResNet-18 network and its GAP layer are used to provide attention maps for a second stage of discriminative training.
Figure 3 Replay attack end-to-end detection. The audio input comes to group delay grams which are novel time-frequency representations of an utterance. Also, the novel attention mechanism softly weights the GD grams. The ResNet-18 network and its GAP layer are used to provide attention maps for a second stage of discriminative training.CC BY · PubMed Central
Figure 4 Char2Wav. It gets text as input, then brings it to an encoder and then decoder that both of them are RNN based, the output of this phase is linguistic features. The vocoder takes the linguistic features and gives audio.
Figure 4 Char2Wav. It gets text as input, then brings it to an encoder and then decoder that both of them are RNN based, the output of this phase is linguistic features. The vocoder takes the linguistic features and gives audio.CC BY · PubMed Central
Figure 5 WaveNet. The input text is presented to causal convolutions, then the output comes to dilated convolutional layers, and then goes to gated activation units, and the activation functions. The Activation Function box includes two (Relu activation functions followed by 1 × 1 layers) plus a Softmax activation function. The figure above also shows the residual block used in WaveNet.
Figure 5 WaveNet. The input text is presented to causal convolutions, then the output comes to dilated convolutional layers, and then goes to gated activation units, and the activation functions. The Activation Function box includes two (Relu activation functions followed by 1 × 1 layers) plus a Softmax activation function. The figure above also shows the residual block used in WaveNet.CC BY · PubMed Central
Figure 6 WaveGlow. The text input goes to a single network which is CNN based, and also tries to maximize the likelihood of the training data, and produces the audio output. X is a group of 8 audio samples squeezed as vectors.
Figure 6 WaveGlow. The text input goes to a single network which is CNN based, and also tries to maximize the likelihood of the training data, and produces the audio output. X is a group of 8 audio samples squeezed as vectors.CC BY · PubMed Central
Figure 7 (A) Tacotron. The model takes input characters, then goes to some RNN based networks as well as a 1-D convolution bank + highway network + bidirectional GRU. Then, it gives the corresponding raw spectrogram as output, that is fed to the Griffin-Lim reconstruction algorithm for speech synthesizing. (B) Tacotron 2. Text characters are input then they go to embedding, PreNet, three convolutional layers, bidirectional LSTM, attention, 2 LSTM layers (recurrent sequence-to-sequence feature prediction network), Linear Projection, Modified WaveNet as vocoder. Then it outputs the wave. (C) Deep Voice3. Text characters are inputs, then they go for embedding, prenet, convolutional blocks and postnet that all of them are considered as the encoder. The output of this phase goes to the decoder which contains: prenet, attention blocks, causal convolutions, a fully-connected layer, and a binary final frame prediction. Then, it can use one of the existing vocoders for producing audio (WORLD, Griffin-lim and Wavenet).
Figure 7 (A) Tacotron. The model takes input characters, then goes to some RNN based networks as well as a 1-D convolution bank + highway network + bidirectional GRU. Then, it gives the corresponding raw spectrogram as output, that is fed to the Griffin-Lim reconstruction algorithm for speech synthesizing. (B) Tacotron 2. Text characters are input then they go to embedding, PreNet, three convolutional layers, bidirectional LSTM, attention, 2 LSTM layers (recurrent sequence-to-sequence feature prediction network), Linear Projection, Modified WaveNet as vocoder. Then it outputs the wave. (C) Deep Voice3. Text characters are inputs, then they go for embedding, prenet, convolutional blocks and postnet that all of them are considered as the encoder. The output of this phase goes to the decoder which contains: prenet, attention blocks, causal convolutions, a fully-connected layer, and a binary final frame prediction. Then, it can use one of the existing vocoders for producing audio (WORLD, Griffin-lim and Wavenet).CC BY · PubMed Central
Figure 8 MelNet. The model gives the input to the three aforementioned stacks. The stacks extract features from different input sections to collectively summarize the entire context.
Figure 8 MelNet. The model gives the input to the three aforementioned stacks. The stacks extract features from different input sections to collectively summarize the entire context.CC BY · PubMed Central
Figure 9 Impersonation using GAN. The GAN contains 6-layer CNN encoder and transposed6-layer CNN as its generative networks.The discriminative network contains 7-layer CNN with adaptive pooling.
Figure 9 Impersonation using GAN. The GAN contains 6-layer CNN encoder and transposed6-layer CNN as its generative networks.The discriminative network contains 7-layer CNN with adaptive pooling.CC BY · PubMed Central
Figure 10 DeepSonar architecture for detecting AI-synthesized fake voices.
Figure 10 DeepSonar architecture for detecting AI-synthesized fake voices.CC BY · PubMed Central

Verification

Claims the paper makes

25 Valid · 5 Not valid · 26 Not checked · 20 Own results and interpretations

14 more claims

Create a free account to see all 76 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 38
  • It may state two separate claims as one. 35

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Cited works that were retracted

Works this paper cites that were later retracted, from the retraction notices publishers deposit. Major when the notice came out before this paper was published.

1 of 1

Citations

Cites

1 retracted

Show all 127 references

Research works printed but not linked: 9

  1. Ajder, 2019, The State of Deepfakes: Landscape, Threats, and Impact, Deeptrace
  2. Blue, 2022, “Who are you (I Really Wanna Know)? Detecting audio DeepFakes through vocal tract reconstruction,”, 31st USENIX Security Symposium (USENIX Security 22), 2691
  3. Chauhan, 2020, Establishing environment setup for preventing deepfakes using blockchain technology, Mukt Shabd J, 9, 771
  4. 2021, Ctrl shift face. YouTube
  5. Goodfellow, 2016, Deep Learning
  6. Ito, 2017, The LJ Speech Dataset
  7. Rea, 2019, Artists Create a Sinister 'deepfake' of Mark Zuckerberg to Teach Facebook (and the rest of us) a Lesson About Digital Propaganda
  8. Saul, 2020, After biden plays ‘despacito' at an event, trump shares a doctored video replacing it with an anti-police song, The New York Times
  9. Stupp, 2019, Fraudsters used AI to mimic CEO's voice in unusual cybercrime case, Wall Street J

Cited by

See all 121

Metadata

Authors
Khanjani, Z., Watson, G., & Janeja, V. P.
Affiliations
University of Maryland, Baltimore County (US)
Corresponding author
Janeja, Vandana P.
Venue
Frontiers in Big Data
Published
2023-01-09
Volume
5
Pages
1001063-1001063
ISSN
2624-909X (ISSN-L)
OpenAlex
W4313984387
Cited by
121
References
143
Citations by year
2026: 36 · 2025: 49 · 2024: 31 · 2023: 5
FWCI
2.53
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Big Data (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Maryland Shared Open Access Repository (USMAI Consortium) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Science Foundation (2210011, 14203)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
Keywords
deepfake audio, deepfake detection, deepfake generation, generative adversarial networks, convolutional neural networks, deep neural network, fake news detection, synthetic media

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.