CoChat
Sign Up
Cited by 131Speech Recognition and SynthesisDigital Media Forensic DetectionSpeech and Audio Processing

DeepSonar: Towards Effective and Robust Detection of AI-Synthesized Fake Voices

Open access
Wang, Run, Juefei-Xu, Felix, Huang, Yihao, Guo, Qing, Xie, Xiaofei, Ma, Lei, Liu, Yang·2020·Proceedings of the 28th ACM International Conference on Multimedia
DOI 10.1145/3394171.3413716
Open this paper in CoChat — free accountOpen PDF

Abstract

With the recent advances in voice synthesis, AI-synthesized fake voices are indistinguishable to human ears and widely are applied to produce realistic and natural DeepFakes, exhibiting real threats to our society. However, effective and robust detectors for synthesized fake voices are still in their infancy and are not ready to fully tackle this emerging threat. In this paper, we devise a novel approach, named DeepSonar, based on monitoring neuron behaviors of speaker recognition (SR) system, i.e., a deep neural network (DNN), to discern AI-synthesized fake voices. Layer-wise neuron behaviors provide an important insight to meticulously catch the differences among inputs, which are widely employed for building safety, robust, and interpretable DNNs. In this work, we leverage the power of layer-wise neuron activation patterns with a conjecture that they can capture the subtle differences between real and AI-synthesized fake voices, in providing a cleaner signal to classifiers than raw inputs. Experiments are conducted on three datasets (including commercial products from Google, Baidu, etc) containing both English and Chinese languages to corroborate the high detection rates (98.1% average accuracy) and low false alarm rates (about 2% error rate) of DeepSonar in discerning fake voices. Furthermore, extensive experimental results also demonstrate its robustness against manipulation attacks (e.g., voice conversion and additive real-world noises). Our work further poses a new insight into adopting neuron behaviors for effective and robust AI aided multimedia fakes forensics as an inside-out approach instead of being motivated and swayed by various artifacts introduced in synthesizing fakes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3394171.3413716

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Analyses of three datasets show that DeepSonar is effective and robust and has real-world promise in a loud background.

    “Analyses of the three datasets show that it is effective and robust and that it has real-world promise in a loud background.”

    From Deepfake detection using deep learning methods: A systematic and comprehensive review (2024) · cites this paper as R. Wang et al. (2020)

    The Introduction (p. 2) and Detection Results (p. 6) report strong performance across three datasets, while the Robustness Evaluation (p. 7) finds that DeepSonar handles several types of added real-world noise. But the Discussion (p. 8) says high-magnitude mixed noise can reduce performance, so the claim of promise in a loud background needs that qualification. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Studies on three datasets encompassing Chinese and English languages confirm DeepSonar's high detection rates (98.1% average accuracy) and low false alarm rates (approximately 2% mistake rate) in detecting phony voices.

    “Studies on three datasets encompassing both Chinese and English languages were done to confirm DeepSonar's high detection rates (98.1% average accuracy) and low false alarm rates (approximately 2% mistake rate) in detecting phony voices.”

    From Deepfake detection using deep learning methods: A systematic and comprehensive review (2024) · cites this paper as R. Wang et al. (2020)

  • Extensive testing results demonstrate DeepSonar's resistance to manipulation attacks.

    “Furthermore, extensive testing results demonstrate its resistance to manipulation attacks.”

    From Deepfake detection using deep learning methods: A systematic and comprehensive review (2024) · cites this paper as R. Wang et al. (2020)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 38

  1. [1][1] Ehab A AlBadawy, Siwei Lyu, and Hany Farid. 2019. Detecting AI-Synthesized Speech Using Bispectral Analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops . 104-109. -
  2. [5][5] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. 2019. Everybody dance now. In Proceedings of the IEEE International Conference on Computer Vision . 5933-5942. -
  3. [4][4] Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. 2019. Activation Atlas. Distill (2019). https://doi.org/10.23915/distill.00015 https://distill.pub/2019/activation-atlas. - · DOI · Paper page
  4. [13][13] Yihao Huang, Felix Juefei-Xu, Run Wang, Qing Guo, Lei Ma, Xiaofei Xie, Jianwen Li, Weikai Miao, Yang Liu, and Geguang Pu. 2020. FakePolisher: Making DeepFakes More Detection-Evasive by Shallow Reconstruction. ACM International Conference on Multimedia (ACM MM) (2020). -
  5. [12][12] Yu Gu and Yongguo Kang. 2018. Multi-task WaveNet: A Multi-task Generative Model for Statistical Parametric Speech Synthesis without Fundamental Frequency Conditions. Proc. Interspeech 2018 (2018), 2007-2011. -
  6. [14][14] Yihao Huang, Felix Juefei-Xu, Run Wang, Qing Guo, Xiaofei Xie, Lei Ma, Jianwen Li, Weikai Miao, Yang Liu, and Geguang Pu. 2020. FakeLocator: Robust Localization of GAN-Based Face Manipulations. arXiv preprint arXiv:2001.09598 (2020). - · arXiv
  7. [16][16] Anne Johnson and Emily Grumbling. 2019. Implications of Artificial Intelligence for Cybersecurity: Proceedings of a Workshop . The National Academies Press, Washington, DC. https://doi.org/10.17226/25488 - · DOI
  8. [19][19] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2019. Analyzing and improving the image quality of stylegan. arXiv preprint arXiv:1912.04958 (2019). - · arXiv
  9. [18][18] Shiyin Kang, Xiaojun Qian, and Helen Meng. 2013. Multi-distribution deep belief network for speech synthesis. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing . IEEE, 8012-8016. -
  10. [22][22] Runnan Li, Zhiyong Wu, Xunying Liu, Helen Meng, and Lianhong Cai. 2017. Multi-task learning of structured output layer bidirectional LSTMs for speech synthesis. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 5510-5514. -
  11. [21][21] Bruce E Koenig and Douglas S Lacey. 2012. Forensic authenticity analyses of the header data in re-encoded WMA files from small Olympus audio recorders. Journal of the Audio Engineering Society 60, 4 (2012), 255-265. -
  12. [23][23] Yuezun Li and Siwei Lyu. 2018. Exposing deepfake videos by detecting face warping artifacts. arXiv preprint arXiv:1811.00656 (2018). - · arXiv
  13. [25][25] Jaime Lorenzo-Trueba, Junichi Yamagishi, Tomoki Toda, Daisuke Saito, Fernando Villavicencio, Tomi Kinnunen, and Zhenhua Ling. 2018. The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods. arXiv preprint arXiv:1804.04262 (2018). - · arXiv
  14. [24][24] Zhen-Hua Ling, Li Deng, and Dong Yu. 2013. Modeling spectral envelopes using restricted Boltzmann machines for statistical parametric speech synthesis. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing . IEEE, 7825-7829. -
  15. [26][26] Jorge C Lucero, Jean Schoentgen, and Mara Behlau. 2013. Physics-based synthesis of disordered voices.. In Interspeech . 587-591. -
  16. [28][28] Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, et al. 2018. DeepGauge: Multi-granularity testing criteria for deep learning systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering . ACM, 120-131. -
  17. [31][31] Shiqing Ma, Yingqi Liu, Wen-Chuan Lee, Xiangyu Zhang, and Ananth Grama. 2018. MODE: automated neural network model debugging via state differential analysis and input selection. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 175-186. -
  18. [30][30] Shiqing Ma and Yingqi Liu. 2019. NIC: Detecting adversarial samples with neural network invariant checking. In Proceedings of the 26th Network and Distributed System Security Symposium (NDSS 2019) . -
  19. [32][32] Aravindh Mahendran and Andrea Vedaldi. 2015. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition . 5188-5196. -
  20. [35][35] Augustus Odena, Catherine Olsson, David Andersen, and Ian Goodfellow. 2019. TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing. In International Conference on Machine Learning . 4901-4911. -
  21. [36][36] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. 2017. Feature Visualization. Distill (2017). https://doi.org/10.23915/distill.00007 https://distill.pub/2017/feature-visualization. - · DOI · Paper page
  22. [44][44] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016). - · arXiv
  23. [45][45] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759 (2016). - · arXiv
  24. [47][47] Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia (2015-10-13). ACM Press, 1015-1018. https://doi.org/10.1145/2733373.2806390 - · DOI
  25. [46][46] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. Deepxplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles . ACM, 1-18. -
  26. [50][50] Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. 2017. Synthesizing obama: learning lip sync from audio. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1-13. -
  27. [49][49] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. 2018. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 4779-4783. -
  28. [51][51] Guanhong Tao, Shiqing Ma, Yingqi Liu, and Xiangyu Zhang. 2018. Attacks meet interpretability: Attribute-steered detection of adversarial samples. In Advances in Neural Information Processing Systems . 7717-7728. -
  29. [52][52] Paul Taylor. 2009. Text-to-speech synthesis . Cambridge university press. -
  30. [56][56] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. 2016. Conditional image generation with pixelcnn decoders. In Advances in neural information processing systems . 4790-4798. -
  31. [57][57] Run Wang, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Yihao Huang, Jian Wang, and Yang Liu. 2020. FakeSpotter: A Simple yet Robust Baseline for Spotting AISynthesized Fake Faces. International Joint Conference on Artificial Intelligence (IJCAI) (2020). -
  32. [59][59] W. Xie, A. Nagrani, J. S. Chung, and A. Zisserman. 2019. Utterance-level Aggregation For Speaker Recognition In The Wild. In International Conference on Acoustics, Speech, and Signal Processing (Oral) . -
  33. [58][58] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. 2017. Tacotron: Towards End-to-End Speech Synthesis. Proc. Interspeech 2017 (2017), 4006-4010. -
  34. [60][60] Xiaofei Xie, Lei Ma, Felix Juefei-Xu, Minhui Xue, Hongxu Chen, Yang Liu, Jianjun Zhao, Bo Li, Jianxiong Yin, and Simon See. 2019. DeepHunter: a coverage-guided fuzz testing framework for deep neural networks. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis . 146-157. -
  35. [62][62] Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. 2019. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems 30, 9 (2019), 2805-2824. -
  36. [61][61] Xin Yang, Yuezun Li, and Siwei Lyu. 2019. Exposing deep fakes using inconsistent head poses. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 8261-8265. -
  37. [63][63] Mohammed Zakariah, Muhammad Khurram Khan, and Hafiz Malik. 2018. Digital multimedia audio forensics: past, present and future. Multimedia tools and applications 77, 1 (2018), 1009-1040. -
  38. [65][65] Hong Zhao and Hafiz Malik. 2013. Audio recording location identification using acoustic environment signature. IEEE Transactions on Information Forensics and Security 8, 11 (2013), 1746-1759.

Cited by

Sorted by citations

Authors
Wang, R., Juefei-Xu, F., Huang, Y., Guo, Q., Xie, X., Ma, L., & Liu, Y.
Affiliations
  • Nanyang Technological University (SG)
  • Alibaba Group (United States) (US)
  • East China Normal University (CN)
  • Kyushu University (JP)
ORCID
Venue
Proceedings of the 28th ACM International Conference on Multimedia
Publisher
ACM
Published
2020-10-12
Pages
1207-1216
DOI
10.1145/3394171.3413716
OpenAlex
W3093077017
Cited by
131
References
7
Citations by year
2026: 12 · 2025: 28 · 2024: 27 · 2023: 28 · 2022: 23 · 2021: 10 · 2020: 3
FWCI
6.10
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM International Conference on Multimedia (ACM MM) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • Singapore National Cybersecurity R&D Program (NRF2018NCR-NCR005-0001)
  • National Satellite of Excellence in Trustworthy Software System (NRF2018NCR-NSOE003-0001)
  • NRF Investigatorship (NRFI06-2020-0022)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Keywords
speaker recognition, deep neural network interpretability, deepfakes, multimedia forensics, voice conversion attacks, noise robustness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review