Leveraging Self-Supervised Learning for Speaker Diarization
Open accessAbstract
End-to-end neural diarization has evolved considerably over the past few years, but data scarcity is still a major obstacle for further improvements. Self-supervised learning methods such as WavLM have shown promising performance on several downstream tasks, but their application on speaker diarization is somehow limited. In this work, we explore using WavLM to alleviate the problem of data scarcity for neural diarization training. We use the same pipeline as Pyannote and improve the local end-to-end neural diarization with WavLM and Conformer. Experiments on far-field AMI, AISHELL-4, and AliMeeting datasets show that our method substantially outperforms the Pyannote baseline and achieves new state-of-the-art results on AMI and AISHELL4, respectively. In addition, by analyzing the system performance under different data quantity scenarios, we show that WavLM representations are much more robust against data scarcity than filterbank features, enabling less data hungry training strategies. Furthermore, we found that simulated data, usually used to train end-to-end diarization models, does not help when using WavLM in our experiments. Additionally, we also evaluate our model on the recent CHiME8 NOTSOFAR-1 task where it achieves better performance than the Pyannote baseline. Our source code is publicly available at https://github.com/BUTSpeechFIT/DiariZen.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The speech-caption branch routes segments to annotation when either predicted speech score or singing score is ≥ 0.5, and uses DiariZen for speaker-aware segmentation.
“A segment is routed to this branch when either its predicted speech score or singing score is greater than or equal to 0.5. For each selected segment, we apply DiariZen [33] to obtain speaker-aware segmentation, where each diarized region is associated with a speaker ID and a time interval.”
From MOSS-Audio Technical Report (2026) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · see citing works
Conformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3015 · Cited by 3,150 · see citing works
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983 · see citing works
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256 · see citing works
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3188113 · Cited by 1,926 · see citing works
The AMI Meeting Corpus: A Pre-announcement
Carletta, J., Ashby, S., Bourban, S., Flynn, M., Guillemot, M., Hain, T., Kadlec, J., Vasilis Karaiskos, B., Kraaij, W., Kronenthal, M., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_3 · Cited by 939 · see citing works
Speaker Recognition from Raw Waveform with SincNet
Ravanelli, M., & Bengio, Y. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639585 · Cited by 896 · see citing works
SUPERB: Speech Processing Universal PERformance Benchmark
Yang, S.-W., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., et al. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1775 · Cited by 681 · see citing works
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372 · cited 1× · see citing works
“Another possible way to make diarization training less data hungry is to use self-supervised learning (SSL) methods, which attempt to make a single universal model applicable to a wide variety of tasks and domains [13]–[16].”
Wang, Q., Downey, C., Wan, L., Mansfield, P. A., & Moreno, I. L. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462628 · Cited by 324 · see citing works
End-to-End Neural Speaker Diarization with Self-Attention
Fujita, Y., Kanda, N., Horiguchi, S., Xue, Y., Nagamatsu, K., & Watanabe, S. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003959 · Cited by 223 · see citing works
Horiguchi, S., Fujita, Y., Watanabe, S., Xue, Y., & Nagamatsu, K. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1022 · Cited by 182 · see citing works
Landini, F., Profant, J., Díez, M., & Burget, L. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101254 · Cited by 181 · see citing works
Self-citationpyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe
Bredin, H. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-105 · Cited by 157 · see citing works
Auto-Tuning Spectral Clustering for Speaker Diarization Using Normalized Maximum Eigengap
Park, T. J., Han, K. J., Kumar, M., & Narayanan, S. (2019) · IEEE Signal Processing Letters · doi:10.1109/lsp.2019.2961071 · Cited by 127 · see citing works
Powerset multi-class cross entropy loss for neural speaker diarization
Plaquet, A., & Bredin, H. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-205 · Cited by 123 · see citing works
M2Met: The Icassp 2022 Multi-Channel Multi-Party Meeting Transcription Challenge
Yu, F., Zhang, S., Fu, Y., Xie, L., Zheng, S., Du, Z., Huang, W., Guo, P., Yan, Z., Ma, B., et al. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746465 · Cited by 96 · see citing works
Fu, Y., Cheng, L., Lv, S., Jv, Y., Kong, Y., Chen, Z., Hu, Y., Xie, L., Wu, J., Bu, H., et al. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1397 · Cited by 93 · see citing works
Integrating End-to-End Neural and Clustering-Based Diarization: Getting the Best of Both Worlds
Kinoshita, K., Delcroix, M., & Tawara, N. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414333 · Cited by 79 · see citing works
Kinoshita, K., Delcroix, M., & Tawara, N. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1004 · Cited by 58 · see citing works
Autoclip: Adaptive Gradient Clipping for Source Separation Networks
Seetharaman, P., Wichern, G., Pardo, B., & Roux, J. L. (2020) · Machine learning for signal processing ... · doi:10.1109/mlsp49062.2020.9231926 · Cited by 40 · see citing works
Peng, J., Plchot, O., Stafylakis, T., Mošner, L., Burget, L., & Černocký, J. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022775 · Cited by 38 · see citing works
Self-citationLandini, F., Lozano-Díez, A., Díez, M., & Burget, L. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10451 · Cited by 38 · see citing works
Self-citationAttention-Based Encoder-Decoder End-to-End Neural Diarization With Embedding Enhancer
Chen, Z., Han, B., Wang, S., & Qian, Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3366756 · Cited by 27 · see citing works
Advancing speaker embedding learning: Wespeaker toolkit for research and production
Wang, S., Chen, Z., Han, B., Wang, H., Liang, C., Zhang, B., Xiang, X., Ding, W., Rohdin, J., Silnova, A., et al. (2024) · Speech Communication · doi:10.1016/j.specom.2024.103104 · Cited by 21 · see citing works
Self-citationDiaPer: End-to-End Neural Diarization With Perceiver-Based Attractors
Landini, F., Díez, M., Stafylakis, T., & Burget, L. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3422818 · Cited by 20 · see citing works
Self-citationNOTSOFAR-1 Challenge: New Datasets, Baseline, and Tasks for Distant Meeting Transcription
Vinnikov, A., Ivry, A., Hurvitz, A., Abramovski, I., Koubi, S., Gurvich, I., Peer, S., Xiao, X., Elizalde, B., Kanda, N., et al. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-1788 · Cited by 20 · see citing works
EEND-M2F: Masked-attention mask transformers for speaker diarization
Härkönen, M., Broughton, S. J., & Samarakoon, L. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-668 · Cited by 20 · see citing works
Improving the Naturalness of Simulated Conversations for End-to-End Neural Diarization
Yamashita, N., Horiguchi, S., & Homma, T. (2022) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2022-19 · Cited by 19 · see citing works
Kalda, J., Pagès, C., Marxer, R., Alumäe, T., & Bredin, H. (2024) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2024-17 · Cited by 17 · see citing works
Landini, F., Díez, M., Lozano-Díez, A., & Burget, L. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097049 · Cited by 17 · see citing works
Self-citationDelcroix, M., Tawara, N., Díez, M., Landini, F., Silnova, A., Ogawa, A., Nakatani, T., Burget, L., & Araki, S. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-628 · Cited by 11 · see citing works
Self-citationTawara, N., Delcroix, M., Ando, A., & Ogawa, A. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447817 · Cited by 8 · see citing works
pyannote.audio speaker diarization pipeline at VoxSRC 2023
Baroudi, S., Bredin, H., Plaquet, A., & Pellegrini, T. (2023) · HAL · Cited by 7 · see citing works
TalTech-IRIT-LIS Speaker and Language Diarization Systems for DISPLACE 2024
Kalda, J., Alumäe, T., Lebourdais, M., Bredin, H., Baroudi, S., & Marxer, R. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-2462 · Cited by 4 · see citing works
Flagged first, then by citations
1 of the 37 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations," Advances in neural information processing systems, vol. 33, pp. 12 449- 12 460, 2020.
- W. Kraaij, T. Hain, M. Lincoln, and W. Post, "The AMI meeting corpus," in Proc. International Conference on Methods and Techniques in Behavioral Research, 2005, pp. 1-4.
Cited by
Sorted by citations
DiCoW: Diarization-conditioned Whisper for target speaker automatic speech recognition
Polok, A., Klement, D., Kocour, M., Han, J., Landini, F., Yusuf, B., Wiesner, M., Khudanpur, S. P., Černocký, J., & Burget, L. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101841 · Cited by 11 · see citing works
Self-citationTarget Speaker ASR with Whisper
Polok, A., Klement, D., Wiesner, M., Khudanpur, S. P., Černocký, J., & Burget, L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887683 · Cited by 9 · see citing works
Self-citationEfficient and Robust Speaker Diarization via Structured Pruning of Self-Supervised Models
Han, J., Pálka, P., Delcroix, M., Landini, F., Rohdin, J., Černocký, J., & Burget, L. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3675801 · Cited by 11 · see citing works
Self-citationOn the Use of Self-Supervised Representation Learning for Speaker Diarization and Separation
Baroudi, S., Bredin, H., Razik, J., & Marxer, R. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434629 · Cited by 2 · see citing works
Design choices for PixIT-based speaker-attributed ASR: Team ToTaTo at the NOTSOFAR-1 challenge
Kalda, J., Baroudi, S., Lebourdais, M., Pagès, C., Marxer, R., Alumäe, T., & Bredin, H. (2025) · Computer Speech & Language · doi:10.1016/j.csl.2025.101824 · Cited by 1 · see citing works
Empowering Speaker Segmentation With Self‐Supervised Learning
Yi, J., Gu, Y., & Xu, Y. (2025) · Electronics Letters · doi:10.1049/ell2.70321 · Cited by 1 · see citing works
Dicow: Diarization-Conditioned Whisper for Target Speaker Automatic Speech RecognitionPreprint
Polok, A., Klement, D., Kocour, M., Han, J., Landini, F., Yusuf, B., Wiesner, M., Khudanpur, S. P., Černocký, J., & Burget, L. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5123534 · Cited by 1 · see citing works
Self-citationYang, C., Yu, C., Chen, H., Zhu, J., Chen, J., Chen, K., Wang, W., Wang, Y., Jiang, Y., Jiang, Y., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.01802
“For each selected segment, we apply DiariZen [33] to obtain speaker-aware segmentation, where each diarized region is associated with a speaker ID and a time interval.”
Adapting WavLM for Vietnamese Speaker Diarization in Real-world Conversations
Thang, T.-D., Nguyen, V., Do, T.-N., Nguyen, Q.-K., Phan, T.-K., & Mạc, Đ.-K. (2025) · International Conference on Multimedia Analysis and Pattern Recognition (MAPR) · doi:10.1109/mapr67746.2025.11133848
Can We Really Repurpose Multi-Speaker ASR Corpus for Speaker Diarization?
Horiguchi, S., Tawara, N., Ashihara, T., Ando, A., & Delcroix, M. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434778
Utilizing Kolmogorov-Arnold Network in Self-Supervised Learning for Speaker Diarization
Vu, M., Dat, P. T., Teh, K. K., Nguyen, V. T., & Dat, T. H. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434699
AAM-SA-ASR: Compact Architecture for Speaker-Attributed Transcription
Alexandr, A., Novoselov, S., & Matveev, Y. (2026) · International Conference on Digital Signal Processing and its Applications (DSPA) · doi:10.1109/dspa69176.2026.11476748
Ramoji, S., Ravindranath, V. K. T., & Hain, T. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464748
Chen, Z., Wu, S., Duan, Y., Xu, X., & Hu, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11465000
On the Role of Spatial Features in Foundation-Model-Based Speaker Diarization
Deegen, M., Gburrek, T., Cord-Landwehr, T., von Neumann, T., Han, J., Burget, L., & Haeb‐Umbach, R. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462871
Self-citationDual-Strategy-Enhanced Conbimamba for Neural Speaker Diarization
Liao, Z., Dai, G., Chen, M., Cheng, W., & Xu, W. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461708
Meise, A., Cord-Landwehr, T., Boeddeker, C., Delcroix, M., Nakatani, T., & Haeb‐Umbach, R. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463540
Song, Z., Tan, T., Wang, Y., Wang, Z., & Lu, J. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464003
Ishizuka, K., Zeng, C., Ono, M., & Hashimoto, T. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463892
Spatially Aware Self-Supervised Models for Multi-Channel Neural Speaker Diarization
Han, J., Wang, R., Masuyama, Y., Delcroix, M., Rohdin, J., Du, J., & Burget, L. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463023
Self-citationHothi, V., & Narendrasinh, B. G. (2026) · INTERNATIONAL JOURNAL OF NEXT-GENERATION COMPUTING · doi:10.47164/ijngc.v17i1.2007
Uteuliev, N., Kudaybergenov, J., & Kudaybergenov, T. (2026) · doi:10.70413/si-v04-i76-s59128
Speaker Diarization in Static and Egocentric Videos via Speech-Face Alignment
Zhang, Z., Yin, X. O., Zheng, Z., Akash, K., Misu, T., & Busso, C. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831222
Sequential Speaker Diarization with Dual-Model Identifier Buffers and Overlap-Aware Assignment
Poghosyan, V., Malajyan, A., Avetisyan, K., Arutunian, M., & Ghukasyan, T. (2026) · Programming and Computer Software · doi:10.1134/s0361768826700556
- Authors
- Han, J., Landini, F., Rohdin, J., Silnova, A., Diez, M., & Burget, L.
- Affiliations
- Brno University of Technology (CZ)
- Brno University of Technology, Speech@FIT,Czechia
- Venue
- ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2025-04-06
- Pages
- 1-5
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W4408352169
- Cited by
- 23
- References
- 37
- Citations by year
- 2026: 14 · 2025: 9
- FWCI
- 21.93
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Funders
- Ministry of Education
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- speaker diarization, self-supervised learning, WavLM, end-to-end neural diarization, data scarcity, conformational isomers, far-field speech, synthetic training data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
