CoChat
Sign Up
PreprintCited by 18Speech Recognition and SynthesisSpeech and Audio ProcessingMusic and Audio Processing

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Real-time Detection of AI-Generated Speech for DeepFake Voice Conversion

Open access
Bird, Jordan J., Lotfi, Ahmad·2023·arXiv
DOI 10.48550/arxiv.2308.12734arXiv 2308.12734
Open this paper in CoChat — free accountOpen PDF

Abstract

There are growing implications surrounding generative AI in the speech domain that enable voice cloning and real-time voice conversion from one individual to another. This technology poses a significant ethical threat and could lead to breaches of privacy and misrepresentation, thus there is an urgent need for real-time detection of AI-generated speech for DeepFake Voice Conversion. To address the above emerging issues, the DEEP-VOICE dataset is generated in this study, comprised of real human speech from eight well-known figures and their speech converted to one another using Retrieval-based Voice Conversion. Presenting as a binary classification problem of whether the speech is real or AI-generated, statistical analysis of temporal audio features through t-testing reveals that there are significantly different distributions. Hyperparameter optimisation is implemented for machine learning models to identify the source of speech. Following the training of 208 individual machine learning models over 10-fold cross validation, it is found that the Extreme Gradient Boosting model can achieve an average classification accuracy of 99.3% and can classify speech in real-time, at around 0.004 milliseconds given one second of speech. All data generated for this study is released publicly for future research on AI speech detection.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2308.12734

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 32

  1. F. Juefei-Xu, R. Wang, Y. Huang, Q. Guo, L. Ma, and Y. Liu, "Countering malicious deepfakes: Survey, battleground, and horizon," International journal of computer vision, vol. 130, no. 7, pp. 1678-1734, 2022.
  2. A. Banks, "What are deepfakes & why the future of porn is terrifying," Highsnobiety. Retrieved, vol. 17, p. 2021, 2018.
  3. M. Waldrop, "Synthetic media: The real trouble with deepfakes," Knowable Magazine, vol. 3, 2020.
  4. B. Borel, "Clicks, lies and videotape," Scientific American, vol. 319, no. 4, pp. 38-43, 2018.
  5. J. Truby and R. Brown, "Human digital thought clones: the holy grail of artificial intelligence for big data," Information & Communications Technology Law, vol. 30, no. 2, pp. 140-168, 2021.
  6. J. J. Beard, "Clones, bones and twilight zones: protecting the digital persona of the quick, the dead and the imaginary," J. Copyright Soc'y USA, vol. 49, p. 441, 2001.
  7. J. Meese, B. Nansen, T. Kohn, M. Arnold, and M. Gibbs, "Posthumous personhood and the affordances of digital media," Mortality, vol. 20, no. 4, pp. 408-420, 2015.
  8. S. Agarwal, H. Farid, Y. Gu, M. He, K. Nagano, and H. Li, "Protecting world leaders against deep fakes.," in CVPR workshops, vol. 1, p. 38, 2019.
  9. B. Wells-Edwards, "What's in a voice? the legal implications of voice cloning," Ariz. L. Rev., vol. 64, p. 1213, 2022.
  10. C. Coburn, K. Williams, and S. R. Stroud, "Enhanced realism or ai-generated illusion? synthetic voice in the documentary film roadrunner," Journal of Media Ethics, vol. 37, no. 4, pp. 282-284, 2022.
  11. R. Dale, "The voice synthesis business: 2022 update," Natural language engineering, vol. 28, no. 3, pp. 401-408, 2022.
  12. R. Skerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang, D. Stanton, J. Shor, R. Weiss, R. Clark, and R. A. Saurous, "Towards end-to-end prosody transfer for expressive speech synthesis with tacotron," in international conference on machine learning, pp. 4693-4702, PMLR, 2018.
  13. Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor, Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, "Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis," in International conference on machine learning, pp. 5180-5189, PMLR, 2018.
  14. Y. Jia, Y. Zhang, R. Weiss, Q. Wang, J. Shen, F. Ren, P. Nguyen, R. Pang, I. Lopez Moreno, Y. Wu, et al., "Transfer learning from speaker verification to multispeaker text-to-speech synthesis," Advances in neural information processing systems, vol. 31, 2018.
  15. S.-Y. Lim, D.-K. Chae, and S.-C. Lee, "Detecting deepfake voice using explainable deep learning techniques," Applied Sciences, vol. 12, no. 8, p. 3926, 2022.
  16. T. Chen, A. Kumar, P. Nagarsheth, G. Sivaraman, and E. Khoury, "Generalization of audio deepfake detection.," in Odyssey, pp. 132-137, 2020.
  17. M. Mcuba, A. Singh, R. A. Ikuesan, and H. Venter, "The effect of deep learning methods on deepfake audio detection for digital investigation," Procedia Computer Science, vol. 219, pp. 211-219, 2023.
  18. E. Conti, D. Salvi, C. Borrelli, B. Hosler, P. Bestagini, F. Antonacci, A. Sarti, M. C. Stamm, and S. Tubaro, "Deepfake speech detection through emotion recognition: a semantic approach," in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8962-8966, IEEE, 2022.
  19. X. Wang, J. Yamagishi, M. Todisco, H. Delgado, A. Nautsch, N. Evans, M. Sahidullah, V. Vestman, T. Kinnunen, K. A. Lee, et al., "Asvspoof 2019: A large-scale public database of synthesized, converted and replayed speech," Computer Speech & Language, vol. 64, p. 101114, 2020.
  20. R. Hennequin, A. Khlif, F. Voituret, and M. Moussallam, "Spleeter: a fast and efficient music source separation tool with pre-trained models," Journal of Open Source Software, vol. 5, no. 50, p. 2154, 2020.
  21. B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto, "librosa: Audio and music signal analysis in python," in Proceedings of the 14th python in science conference, vol. 8, pp. 18-25, 2015.
  22. J. Kim, J. Kong, and J. Son, "Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech," in International Conference on Machine Learning, pp. 5530-5540, PMLR, 2021.
  23. J. W. Kim, J. Salamon, P. Li, and J. P. Bello, "Crepe: A convolutional representation for pitch estimation," in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 161-165, IEEE, 2018.
  24. T. Chen and C. Guestrin, "Xgboost: A scalable tree boosting system," in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
  25. L. Breiman, "Random forests," Machine learning, vol. 45, pp. 5-32, 2001.
  26. R. A. Fisher, "The use of multiple measurements in taxonomic problems," Annals of eugenics, vol. 7, no. 2, pp. 179-188, 1936.
  27. A. E. Hoerl and R. W. Kennard, "Ridge regression: Biased estimation for nonorthogonal problems," Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
  28. T. Bayes, "Lii. an essay towards solving a problem in the doctrine of chances. by the late rev. mr. bayes, frs communicated by mr. price, in a letter to john canton, amfr s," Philosophical transactions of the Royal Society of London, no. 53, pp. 370-418, 1763.
  29. M. A. Hearst, S. T. Dumais, E. Osuna, J. Platt, and B. Scholkopf, "Support vector machines," IEEE Intelligent Systems and their applications, vol. 13, no. 4, pp. 18-28, 1998.
  30. H. Robbins and S. Monro, "A stochastic approximation method," The annals of mathematical statistics, pp. 400- 407, 1951.
  31. C. Williams and C. Rasmussen, "Gaussian processes for regression," Advances in neural information processing systems, vol. 8, 1995.
  32. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, "Scikit-learn: Machine learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825-2830, 2011.

Cited by

Sorted by citations

See all 18

Metadata

Authors
Bird, J. J., & Lotfi, A.
Affiliations
Nottingham Trent University (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2308.12734
arXiv
2308.12734
OpenAlex
W4386185355
Cited by
18
References
1
Citations by year
2026: 1 · 2025: 10 · 2024: 7
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
Keywords
voice cloning, extreme gradient boosting, speech forensics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review