The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index; none of the 4 sources its claims cite is checked for support yet.
Benchmarking Audio Deepfake Detection Robustness in Real-World Communication Scenarios
Open accessAbstract
Existing Audio Deepfake Detection (ADD) systems often struggle to generalise effectively due to the significantly degraded audio quality caused by audio codec compression and channel transmission effects in real-world communication scenarios. To address this challenge, we developed a rigorous benchmark to evaluate the performance of the ADD system under such scenarios. We introduced ADD-C, a new test dataset to evaluate the robustness of ADD systems under diverse communication conditions, including different combinations of audio codecs for compression and packet loss rates. Benchmarking three baseline ADD models on the ADD-C dataset demonstrated a significant decline in robustness under such conditions. A novel Data Augmentation (DA) strategy was proposed to improve the robustness of ADD systems. Experimental results demonstrated that the proposed approach significantly enhances the performance of ADD systems on the proposed ADD-C dataset. Our benchmark can assist future efforts towards building practical and robustly generalisable ADD systems.
Overview
CoChat Summary
TL;DR
The paper introduces ADD-C, a benchmark and test dataset for evaluating audio deepfake detection (ADD) robustness under simulated real-world communication conditions involving codec compression and packet loss. It shows that baseline ADD models degrade significantly under these conditions, and proposes a data augmentation strategy that improves robustness.
Key findings
- Benchmarking three baseline ADD models (GMM, LCNN, AASIST) on ADD-C showed an average degradation from clean condition C0 to communication condition C1 of 5.30% in EER, 3.16% in AUC, and 3.34% in F1-score.
- Proposed models trained on the Original dataset also degraded from C0 to C1, with average drops of 2.33% (EER), 0.49% (AUC), and 1.40% (F1-score).
- Proposed models trained on the Augmented dataset remained stable across conditions: EER was unchanged with a slight improvement of 0.003% from C0 to C1, while AUC and F1-score showed a negligible decrease of 0.1%.
- The ADD-C test dataset contains six conditions: one clean condition and five communication conditions combining six speech codecs with five packet loss rates (0%, 1%, 5%, 10%, 20%).
Methods
- ADD-C was built from six public speech datasets: Fake-or-Real (FoR), Wavefake, LJSpeech, MLAAD, M-AILABS, and ASVspoof2021 Logical Access (ASV), totaling 130,041 real and 240,373 fake utterances across 36 deepfake algorithms.
- Real-world communication was simulated using six codecs (AMR-WB, EVS, IVAS, OPUS, Speex WB, SILK) and five packet loss rates; ADD-C has 4,000 utterances in C0 and 24,000 utterances in each of C1–C5.
- The proposed data augmentation partitioned the Original dataset into six equal subsets, applied codec compression and packet loss simulation, and merged the five PLR-generated datasets into an Augmented dataset of 1,832,070 utterances (640,205 real, 1,191,865 fake). Models used 4-second mono audio, 80%:20% train/validation split, five epochs, batch size 256, Adam, early stopping with patience 3, and cross-entropy loss.
Limitations
- The paper does not state an explicit limitations section.
- Evident from the text: evaluation is based on simulated codec compression and packet loss rather than live real-world communication deployments.
- Evident from the text: the selected datasets are all English-language, and the benchmark uses a fixed set of six codecs, five PLRs, and three baseline models, which may limit generalization to other languages, codecs, or network conditions.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 4 Not checked · 24 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Benchmarking three baseline ADD models on the ADD-C dataset demonstrated a significant decline in robustness under such conditions.”
- Also stated in: § I. INTRODUCTION · plain
“Benchmarking three baseline ADD models reveals a significant decline in performance, highlighting the need for improving the robustness of ADDs.”
§ Benchmarking Audio Deepfake Detection Robustness in Real-world Communication Scenarios > Abstract
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § I. INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AUC and F1-score also remain highly stable, with a negligible decrease of 0.1%.”
§ Benchmarking Audio Deepfake Detection Robustness in Real-world Communication Scenarios > B. Results and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, unlike the models trained on the Original dataset that suffer from a significant performance degradation, those trained on the Augmented dataset exhibit consistent stability across all evaluation metrics from C 0 to C 5 , without noticeable fluctuations or drops.”
§ Benchmarking Audio Deepfake Detection Robustness in Real-world Communication Scenarios > B. Results and Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results demonstrated that the proposed approach significantly enhances the performance of ADD systems on the proposed ADD-C dataset.”
- Also stated in: § I. INTRODUCTION · boosted
“Experimental results demonstrate that our approach significantly enhances the robustness of ADD systems in real-world communication scenarios.”
§ Benchmarking Audio Deepfake Detection Robustness in Real-world Communication Scenarios > Abstract
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § I. INTRODUCTION · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Specifically, comparing C 0 to C 1 yields an average degradation of 2.33% (EER), 0.49% (AUC), and 1.40% (F1-score).”
§ Benchmarking Audio Deepfake Detection Robustness in Real-world Communication Scenarios > B. Results and Discussion
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
9 more claims
Create a free account to see all 28 claims and every source verdict.
- The quote only partly states this claim. 1
- It may state two separate claims as one. 7
- Probably not a substantial claim. 8
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Benchmarking Audio Deepfake Detection Robustness in Real-world Communication Scenarios > B. Data Augmentation (DA) Strategy
“This produced six augmented datasets, each corresponding to a different PLR.”
§ Benchmarking Audio Deepfake Detection Robustness in Real-world Communication Scenarios > III. BENCHMARK DESIGN AND BASELINE EVALUATION A. Real-world Communication Simulation
“Five PLRs, 0%, 1%, 5%, 10%, and 20%, were selected for this study”
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Optimization Methods for Large-Scale Machine Learning
Bottou, L., Curtis, F. E., & Nocedal, J. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,128 · cited 1×
“Early Stopping [43] with a patience of three is employed to prevent overfitting.”
Scheduled denoising autoencodersPreprint
Geras, K. J., & Sutton, C. A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.3269 · Cited by 2,019
LTE – The UMTS Long Term Evolution: From Theory to Practice
(2011) · Wiley eBooks · doi:10.1002/9780470978504 · Cited by 1,344
Voice over Internet protocol (VoIP)
Goode, B. (2002) · Proceedings of the IEEE · doi:10.1109/jproc.2002.802005 · Cited by 480
Constant Q Cepstral Coefficients: A Spoofing Countermeasure for Automatic Speaker Verification
Todisco, M., Delgado, H., & Evans, N. (2017) · Computer Speech & Language · doi:10.1016/j.csl.2017.01.001 · Cited by 424 · cited 1×
“Prior studies indicate that channel-induced data loss degrades the performance of audio-based feature systems [28], while codec-induced compression artifacts result in diminished audio quality as high-frequency information is lost [29], which reduces the robustness of the ADD system.”
End-to-End anti-spoofing with RawNet2
Tak, H., Patino, J., Todisco, M., Nautsch, A., Evans, N., & Larcher, A. (2021) · ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp39728.2021.9414234 · Cited by 402 · cited 1×
“Additionally, [24] modified the original RawNet2 architecture to classify audio authenticity from raw waveform inputs directly.”
AASIST: Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention Networks
Jung, J.-W., Heo, H.-S., Tak, H., Shim, H.-J., Chung, J. S., Lee, B.-J., Yu, H.-J., & Evans, N. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747766 · Cited by 399 · cited 1×
“In this section, three models and a DA strategy are proposed to address the performance decline of ADD systems caused by real-world communication scenarios.”
The adaptive multirate wideband speech codec (AMR-WB)
Bessette, B., Salami, R., Lefebvre, R., Jelinek, M., Rotola-Pukkila, J., Vainio, J., Mikkola, H., & Jarvinen, K. (2002) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2002.804299 · Cited by 315 · cited 1×
“To systematically evaluate the impact of real-world communication scenarios on ADD systems, we propose the ADD-C test dataset, which is based on six publicly available speech datasets, Fake-or-Real (FoR) [37], Wavefake [38], LJSpeech [39], MLAAD [40], M-AILABS [41] and ASVspoof2021 Logical Access…”
ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild
Liu, X., Wang, X., Sahidullah, M., Patino, J., Delgado, H., Kinnunen, T., Todisco, M., Yamagishi, J., Evans, N., Nautsch, A., et al. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3285283 · Cited by 312 · cited 4×
“…communication scenarios on ADD systems, particularly considering audio codec compression and channel transmission effects, three baseline ADD models were selected for evaluation: GMM [4], LCNN [4], and AASIST [36], utilising CQCC, LFCC, and raw waveform as acoustic features, respectively.”
Influential
Research works printed but not linked: 8
- Brewster, 2021, Fraudsters cloned company director’s voice in $ 35 million heist, police find, Forbes
- 10.1145/3658664.3659647 · DOI
- Blue, Who are you(i really wanna know)? detecting audio deepfakes through vocal tract reconstruction, 2691
- Molisch, 2012, Wireless communications, 34
- 2024, Lte; 5g; codec for immersive voice and audio services - detailed algorithmic description incl. rtp payload format and sdp parameter definitions
- Astrom, 2009, Rtp payload format and file storage format for silk speech and audio codec
- Ito, 2017, The lj speech dataset
- 2024, The m-ailabs speech dataset
Cited by
Deepfake Audio Detection in Voice Authentication: A Spectral and CNN-Based Comprehensive Review
Salih, A. O. M., Emam, A. H. M., Ahmed, A. B. G. E., Khalifa, M., Suliman, A., & Babiker, N. B. M. (2025) · Engineering Technology & Applied Science Research · doi:10.48084/etasr.13400 · Cited by 17
“These methods allow for the detection of inconsistencies indicative of artificial manipulation [2, 3].”
FakeVoiceFinder: An Open-Source Framework for Synthetic and Deepfake Audio Detection
Pachón, C. G., & Ballesteros, D. M. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10010025 · Cited by 3
“Beyond these metrics, recent studies highlight the importance of robustness evaluation, examining model generalization across datasets [21], languages [22], and real-world communication conditions such as compression and channel noise [23].”
Audio Deepfake Detection at the First Greeting: "Hi!"
Shi, H., Shi, X., Dogan, S., Huang, T., & Zhang, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463587 · Cited by 1
“These include LCNN [3], RawNet2 [18], AASIST [8], AASIST-L [8], RawGAT-ST [10], FC-LFCC [6], and MGAA [7].”
InfluentialSelf-citationAudio Deepfake Detectors vs. Real Fraud - the Fall of Benchmarks
Gajewska, J., Martinek, A., & Bartuzi-Trokielewicz, E. (2026) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw68408.2026.00031 · Cited by 2
“Shi et al. [24] introduced ADD-C, a benchmark simulating communication artifacts such as codec compression and packet loss, revealing significant performance drops in existing detectors.”
Fairness and Explainability in Portuguese Audio Deepfake Detection: The Effect of Audio Codecs
Nagpal, R., Rinaldo, G., Simons, K., Silva, C. O., Pereira, L. V. A. S., Xing, L., & Gupta, A. D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39892-5_9
Metadata
- Authors
- Shi, H., Shi, X., Dogan, S., Alzubi, S., Huang, T., & Zhang, Y.
- Affiliations
- Loughborough University (GB)
- University of Exeter (GB)
- Venue
- 2025 33rd European Signal Processing Conference (EUSIPCO)
- Publisher
- IEEE
- Published
- 2025-09-08
- Pages
- 566-570
- OpenAlex
- W4417052665
- Cited by
- 8
- References
- 33
- Citations by year
- 2026: 7 · 2025: 1
- FWCI
- 7.63
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- European Signal Processing Conference (EUSIPCO) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Loughborough University (GS1016)
- China Scholarship Council (202208060237)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- audio deepfake detection, benchmarking, packet loss, data augmentation, robustness, generalization, audio quality degradation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Benchmarking Audio Deepfake Detection Robustness in Real-World Communication Scenarios. https://app.cochat.ai/papers/10.23919/eusipco63237.2025.11226601
- The paper
- Shi, H., Shi, X., Dogan, S., Alzubi, S., Huang, T., & Zhang, Y. (2025). Benchmarking Audio Deepfake Detection Robustness in Real-World Communication Scenarios. 2025 33rd European Signal Processing Conference (EUSIPCO). https://doi.org/10.23919/eusipco63237.2025.11226601
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.