t-EER: Parameter-free tandem evaluation of countermeasures and biometric comparators
Open accessAbstract
Presentation attack (spoofing) detection (PAD) typically operates alongside biometric verification to improve reliablity in the face of spoofing attacks. Even though the two sub-systems operate in tandem to solve the single task of reliable biometric verification, they address different detection tasks and are hence typically evaluated separately. Evidence shows that this approach is suboptimal. We introduce a new metric for the joint evaluation of PAD solutions operating in situ with biometric verification. In contrast to the tandem detection cost function proposed recently, the new tandem equal error rate (t-EER) is parameter free. The combination of two classifiers nonetheless leads to a set of operating points at which false alarm and miss rates are equal and also dependent upon the prevalence of attacks. We therefore introduce the concurrent t-EER, a unique operating point which is invariable to the prevalence of attacks. Using both modality (and even application) agnostic simulated scores, as well as real scores for a voice biometrics application, we demonstrate application of the t-EER to a wide range of biometric system evaluations under attack. The proposed approach is a strong candidate metric for the tandem evaluation of PAD systems and biometric comparators.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The tandem detection cost function and tandem equal error rate are complementary metrics for Track 2.
“with the tandem detection cost function (t-DCF) [10] and tandem equal error rate (t-EER) [11] being complementary.”
From ASVspoof 5: crowdsourced speech data, deepfakes, and adversarial attacks at scale (2024) · cites this paper as [11]
The tandem equal error rate provides a complementary parameter-free measure of class discrimination.
“the latter provides a complementary parameter-free measure of class discrimination.”
From ASVspoof 5: crowdsourced speech data, deepfakes, and adversarial attacks at scale (2024) · cites this paper as [11]
For submissions with clearly identified ASV and CM sub-systems, the ASV-constrained minimum tandem detection cost function and tandem equal error rate are also reported.
“For submissions which provide clearly-identified ASV and CM sub-systems, the ASV-constrained minimum tandem detection cost function (t-DCF) [10] and the tandem equal error rate (t-EER) [11] are also reported.”
From ASVspoof 5: crowdsourced speech data, deepfakes, and adversarial attacks at scale (2024) · cites this paper as [11]
In ASVspoof 5, t-DCF is used concurrently with t-EER and a-DCF.
“In ASVspoof 5, it is used concurrently with the recently proposed t-EER [94] and a-DCF [95]”
From Audio Deepfake Detection: What Has Been Achieved and What Lies Ahead (2025) · cites this paper as [94]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879 · see citing works
Detection theory: A user's guide
Lories, G. (1993) · Behavioural Processes · doi:10.1016/0376-6357(93)90083-4 · Cited by 2,927 · see citing works
X-Vectors: Robust DNN Embeddings for Speaker Recognition
Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., & Khudanpur, S. P. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461375 · Cited by 2,698 · see citing works
Classification assessment methods
Tharwat, A. (2021) · Applied Computing and Informatics · doi:10.1016/j.aci.2018.08.003 · Cited by 2,417 · see citing works
Enhancing security and privacy in biometrics-based authentication systems
Ratha, N., Connell, J. H., & Bolle, R. M. (2001) · IBM Systems Journal · doi:10.1147/sj.403.0614 · Cited by 1,688 · see citing works
Desplanques, B., Thienpondt, J., & Demuynck, K. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2650 · Cited by 1,568 · see citing works
The DET curve in assessment of detection task performance
Martín, A. F., Doddington, G. R., Kamm, T., Ordowski, M., & Przybocki, M. A. (1997) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.1997-504 · Cited by 1,393 · see citing works
Pattern Recognition and Machine Learning
Fu, K. (1971) · Springer eBooks · doi:10.1007/978-1-4615-7566-5 · Cited by 1,134 · see citing works
Probabilistic Linear Discriminant Analysis for Inferences About Identity
Prince, S. J. D., & Elder, J. H. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4409052 · Cited by 988 · see citing works
Media Forensics and DeepFakes: An Overview
Verdoliva, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3002101 · Cited by 979 · see citing works
ROC Curves for Continuous Data
Krzanowski, W. J., & Hand, D. J. (2009) · Chapman & Hall/CRC eBooks · doi:10.1201/9781439800225 · Cited by 669 · see citing works
Spoofing and countermeasures for speaker verification: A survey
Wu, Z., Evans, N., Kinnunen, T., Yamagishi, J., Alegre, F., & Li, H. (2015) · Speech Communication · doi:10.1016/j.specom.2014.10.005 · Cited by 601 · see citing works
Self-citationBolle, R. M., Connell, J. H., Pankanti, S., et al. (2004) · Springer eBooks · doi:10.1007/978-1-4757-4036-3 · Cited by 592 · see citing works
End-to-End anti-spoofing with RawNet2
Tak, H., Patino, J., Todisco, M., Nautsch, A., Evans, N., & Larcher, A. (2021) · ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp39728.2021.9414234 · Cited by 402 · see citing works
Self-citationAASIST: Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention Networks
Jung, J.-W., Heo, H.-S., Tak, H., Shim, H.-J., Chung, J. S., Lee, B.-J., Yu, H.-J., & Evans, N. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747766 · Cited by 399 · see citing works
Self-citationASVspoof 2021: accelerating progress in spoofed and deepfake speech detection
Yamagishi, J., Wang, X., Todisco, M., Sahidullah, M., Patino, J., Nautsch, A., Liu, X., Lee, K. A., Kinnunen, T., Evans, N., et al. (2021) · 2021 Edition of the Automatic Speaker Verification and Spoofing Countermeasures Challenge · doi:10.21437/asvspoof.2021-8 · Cited by 380 · see citing works
Self-citationThe NIST speaker recognition evaluation – Overview, methodology, systems, results, perspective
Doddington, G. R., Przybocki, M. A., Martín, A. F., & Reynolds, D. A. (2000) · Speech Communication · doi:10.1016/s0167-6393(99)00080-1 · Cited by 366 · see citing works
Biometric Systems Technology Design And Performance Evaluation
Wayman, J. L., Jain, A. K., Maltoni, D., et al. (2011) · CERN Document Server (European Organization for Nuclear Research) · doi:10.1007/b138151 · Cited by 360 · see citing works
ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild
Liu, X., Wang, X., Sahidullah, M., Patino, J., Delgado, H., Kinnunen, T., Todisco, M., Yamagishi, J., Evans, N., Nautsch, A., et al. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3285283 · Cited by 312 · cited 4× · see citing works
“This property has been exploited in the organisation of the ASVspoof challenge series [9] for which an ASV system is implemented by the organizers, allowing participants to foucs upon the design of CMs.”
BackgroundMethodInfluentialSelf-citationTak, H., Todisco, M., Wang, X., Jung, J.-W., Yamagishi, J., & Evans, N. (2022) · The Speaker and Language Recognition Workshop (Odyssey 2022) · doi:10.21437/odyssey.2022-16 · Cited by 237 · see citing works
Self-citationAssessing the Performance of Classification Methods
Hand, D. J. (2012) · International Statistical Review · doi:10.1111/j.1751-5823.2012.00183.x · Cited by 177 · see citing works
The BOSARIS Toolkit: Theory, Algorithms and Code for Surviving the New DCFPreprint
Brümmer, N., & de Villiers, E. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1304.2865 · Cited by 165 · see citing works
MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification
Zhang, Y., Lv, Z., Wu, H., Zhang, S., Hu, P., Wu, Z., Lee, H.-Y., & Meng, H. M. L. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-563 · Cited by 145 · see citing works
Investigating Self-Supervised Front Ends for Speech Spoofing Countermeasures
Wang, X., & Yamagishi, J. (2022) · The Speaker and Language Recognition Workshop (Odyssey 2022) · doi:10.21437/odyssey.2022-14 · Cited by 138 · see citing works
A Comparative Study on Recent Neural Spoofing Countermeasures for Synthetic Speech Detection
Wang, X., & Yamagishi, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-702 · Cited by 137 · see citing works
ASVspoof 2021: Automatic Speaker Verification Spoofing and Countermeasures Challenge Evaluation Plan
Delgado, H., Evans, N., Kinnunen, T., Lee, K. A., Liu, X., Nautsch, A., Patino, J., Sahidullah, M., Todisco, M., Wang, X., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.00535 · Cited by 129 · see citing works
Self-citationTandem Assessment of Spoofing Countermeasures and Automatic Speaker Verification: Fundamentals
Kinnunen, T., Delgado, H., Evans, N., Lee, K. A., Vestman, V., Nautsch, A., Todisco, M., Wang, X., Sahidullah, M., Yamagishi, J., et al. (2020) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3009494 · Cited by 108 · see citing works
Self-citationBiometrics Evaluation Under Spoofing Attacks
Chingovska, I., dos Anjos, A. R., & Marcel, S. (2014) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2014.2349158 · Cited by 96 · see citing works
Application-independent evaluation of speaker detection
Brümmer, N., & du Preez, J. (2006) · Computer Speech and Language · doi:10.1016/j.csl.2005.08.001 · Cited by 71 · see citing works
The distribution of calibrated likelihood-ratios in speaker recognition
van Leeuwen, D. A., & Brümmer, N. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-406 · Cited by 68 · see citing works
Anti-spoofing in Action: Joint Operation with a Verification System
Chingovska, I., Anjos, A., & Marcel, S. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2013.22 · Cited by 62 · see citing works
Introduction to Face Presentation Attack Detection
Hernandez‐Ortega, J., Fiérrez, J., Morales, A., & Javier Galbally, A. (2019) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-319-92627-8_9 · Cited by 56 · see citing works
SASV 2022: The First Spoofing-Aware Speaker Verification Challenge
Jung, J.-W., Tak, H., Shim, H.-J., Heo, H.-S., Lee, B.-J., Chung, S.-W., Yu, H.-J., Evans, N., & Kinnunen, T. (2022) · Interspeech 2022 · doi:10.21437/interspeech.2022-11270 · Cited by 56 · see citing works
Self-citationSahidullah, M., Delgado, H., Todisco, M., Yu, H. T., Kinnunen, T., Evans, N., & Tan, Z. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-1280 · Cited by 54 · see citing works
Self-citationTodisco, M., Delgado, H., Lee, K. A., Sahidullah, M., Evans, N., Kinnunen, T., & Yamagishi, J. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2289 · Cited by 54 · see citing works
Self-citationThe ins and outs of speaker recognition: lessons from VoxSRC 2020
Kwon, Y., Heo, H.-S., Lee, B., & Chung, J. S. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413948 · Cited by 46 · see citing works
Combining match scores with liveness values in a fingerprint verification system
Marasco, E., Ding, Y., & Ross, A. A. (2012) · IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS) · doi:10.1109/btas.2012.6374609 · Cited by 42 · see citing works
Korshunov, P., & Marcel, S. (2017) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2017.2692389 · Cited by 35 · see citing works
Poh, N., & Bengio, S. (2004) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2004.1327255 · Cited by 31 · see citing works
Frequency and Multi-Scale Selective Kernel Attention for Speaker Verification
Mun, S. H., Jung, J.-W., Han, M. H., & Kim, N. S. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023305 · Cited by 28 · see citing works
Evaluation Methodologies for Biometric Presentation Attack Detection
Chingovska, I., Mohammadi, A., Anjos, A., & Marcel, S. (2019) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-319-92627-8_20 · Cited by 24 · see citing works
Review of the Fingerprint Liveness Detection (LivDet) Competition Series: From 2009 to 2021
Micheletto, M., Orrù, G., Casula, R., et al. (2023) · Advances in computer vision and pattern recognition · doi:10.1007/978-981-19-5288-3_3 · Cited by 23 · see citing works
A speaker verification backend with robust performance across conditions
Ferrer, L., McLaren, M., & Brümmer, N. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101258 · Cited by 20 · see citing works
Notes on the H-measure of classifier performance
Hand, D. J., & Anagnostopoulos, C. (2022) · Advances in Data Analysis and Classification · doi:10.1007/s11634-021-00490-3 · Cited by 16 · see citing works
SASV Challenge 2022: A Spoofing Aware Speaker Verification Challenge Evaluation PlanPreprint
Jung, J.-W., Tak, H., Shim, H.-J., Heo, H.-S., Lee, B., Chung, S.-W., Kang, H.-G., Yu, H.-J., Evans, N., & Kinnunen, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.10283 · Cited by 14 · see citing works
Self-citationAn Introduction to Fingerprint Presentation Attack Detection
Javier Galbally, A., Fiérrez, J., & Cappelli, R. (2019) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-319-92627-8_1 · Cited by 10 · see citing works
Shim, H.-J., Tak, H., Liu, X., Heo, H.-S., Jung, J.-W., Chung, J. S., Chung, S.-W., Yu, H.-J., Lee, B., Todisco, M., et al. (2022) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2022-46 · Cited by 9 · see citing works
Self-citationOut of a Hundred Trials, How Many Errors Does Your Speaker Verifier Make?
Brümmer, N., Ferrer, L., & Swart, A. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-541 · Cited by 5 · see citing works
Review of the Fingerprint Liveness Detection (LivDet) competition series: from 2009 to 2021Preprint
Micheletto, M., Orrù, G., Casula, R., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.07259 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde- Farley, S. Ozair, A. C. Courville, and Y. Bengio, "Generative Adversarial Nets," in Proc. Neural Information Processing Systems (NIPS), 2014, pp. 2672-2680.
- A. Nautsch, "Speaker recognition in unconstrained environ- ments," Ph.D. dissertation, Technische Universität Darmstadt, 2019.
- S. Bengio and J. Mariéthoz, "The expected performance curve: a new assessment measure for person authentication," in Proc. the Speaker and Language Recognition Workshop (Speaker Odyssey), 2004, pp. 279-284.
Cited by
Sorted by citations
ASVspoof 5: crowdsourced speech data, deepfakes, and adversarial attacks at scale
Wang, X., Delgado, H., Tak, H., Jung, J.-W., Shim, H.-J., Todisco, M., Kukanov, I., Liu, X., Sahidullah, M., Kinnunen, T. H., et al. (2024) · The Automatic Speaker Verification Spoofing Countermeasures Workshop (ASVspoof 2024) · doi:10.21437/asvspoof.2024-1 · Cited by 101 · see citing works
Self-citationAudio Deepfake Detection: What Has Been Achieved and What Lies Ahead
Zhang, B., Cui, H., Nguyen, V., & Whitty, M. (2025) · Sensors · doi:10.3390/s25071989 · Cited by 72 · see citing works
- Authors
- Kinnunen, T. H., Lee, K. A., Tak, H., Evans, N., & Nautsch, A.
- Affiliations
- University of Eastern Finland (FI)
- Agency for Science, Technology and Research (SG)
- EURECOM (FR)
- goSmart GmbH, Nürnberg, Germany
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2024-05
- Volume
- 46
- Issue
- 5
- Pages
- 2622-2637
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4386609452
- Cited by
- 4
- References
- 49
- Citations by year
- 2026: 1 · 2025: 2 · 2023: 1
- FWCI
- 0.49
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- UEF eRepo (University of Eastern Finland) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Academy of Finland (349605)
- Agency of Science, Technology and Research
- CRF Core Project Scheme (CR-2021-005)
- Agence Nationale de la Recherche (ANR-18-JSTS-0001)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Information Systems › User Authentication and Security Systems
- Keywords
- anti-spoofing, biometric authentication, voice biometrics, biometric security
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
