The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Joint Speech Recognition and Speaker Diarization via Sequence Transduction
Open accessAbstract
Speech applications dealing with conversations require not only recognizing the spoken words, but also determining who spoke when. The task of assigning words to speakers is typically addressed by merging the outputs of two separate systems, namely, an automatic speech recognition (ASR) system and a speaker diarization (SD) system. The two systems are trained independently with different objective functions. Often the SD systems operate directly on the acoustics and are not constrained to respect word boundaries and this deficiency is overcome in an ad hoc manner. Motivated by recent advances in sequence to sequence learning, we propose a novel approach to tackle the two tasks by a joint ASR and SD system using a recurrent neural network transducer. Our approach utilizes both linguistic and acoustic cues to infer speaker roles, as opposed to typical SD systems, which only use acoustic cues. We evaluated the performance of our approach on a large corpus of medical conversations between physicians and patients. Compared to a competitive conventional baseline, our approach improves word-level diarization error rate from 15.8% to 2.2%.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The position of the recording device also has a strong impact on the captured audio.18,20”
From Challenges of developing a digital scribe to reduce clinical documentation burden (2019) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537
X-Vectors: Robust DNN Embeddings for Speaker Recognition
Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., & Khudanpur, S. P. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461375 · Cited by 2,794
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
Sequence Transduction with Recurrent Neural NetworksPreprint
Graves, A. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1211.3711 · Cited by 1,842
Speaker Diarization: A Review of Recent Research
Anguera, X., Bozonnet, S., Evans, N., Fredouille, C., Friedland, G. H., & Vinyals, O. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2125954 · Cited by 704
An overview of automatic speaker diarization systems
Tranter, S. E., & Reynolds, D. A. (2006) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2006.878256 · Cited by 605
Streaming End-to-end Speech Recognition for Mobile Devices
He, Y., Sainath, T. N., Prabhavalkar, R., McGraw, I., Alvarez, R., Zhao, D., Rybach, D., Kannan, A., Wu, Y., Pang, R., et al. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682336 · Cited by 601
End-to-end text-dependent speaker verification
Heigold, G., Moreno, I. L., Samy Bengio, A., & Shazeer, N. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472652 · Cited by 537
Wang, Q., Downey, C., Wan, L., Mansfield, P. A., & Moreno, I. L. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462628 · Cited by 338
Phoneme Recognition Using Time-Delay Neural Networks
Waibel, A. H., Hanazawa, T., Hinton, G. E., Shikano, K., & LANG, K. J. (1990) · Elsevier eBooks · doi:10.1016/b978-0-08-051584-7.50037-1 · Cited by 313
Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition
Soltau, H., Liao, H., & Sak, H. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1566 · Cited by 304
Self-citationSpeaker diarization using deep neural network embeddings
Garcia-Romero, D., Snyder, D., Sell, G., Povey, D., & McCree, A. V. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953094 · Cited by 244
A robust speaker clustering algorithm
Ajmera, J., & Wooters, C. (2004) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2003.1318476 · Cited by 243
Fully Supervised Speaker Diarization
Zhang, A., Wang, Q., Zhu, Z., Paisley, J. W., & Wang, C. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683892 · Cited by 235
Speaker diarization with plda i-vector scoring and unsupervised calibration
Sell, G., & Garcia-Romero, D. (2014) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2014.7078610 · Cited by 213
Sell, G., Snyder, D., McCree, A. V., Garcia-Romero, D., Villalba, J., Maciejewski, M., Manohar, V., Dehak, N., Povey, D., Watanabe, S., et al. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1893 · Cited by 211
TristouNet: Triplet loss for speaker turn embedding
Bredin, H. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953194 · Cited by 183
Multistage speaker diarization of broadcast news
Barras, C., Zhu, X., Meignier, S., & Gauvain, J.-L. (2006) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2006.878261 · Cited by 180
Morfessor 2.0: Python Implementation and Extensions for Morfessor Baseline
Virpioja, S., Smit, P., Grönroos, S.-A., & Kurimo, M. (2013) · Aaltodoc (Aalto University) · Cited by 156
The Use of Recurrent Neural Networks in Continuous Speech Recognition
Robinson, T., Hochberg, M. M., & Renals, S. J. (1996) · Kluwer international series in engineering and computer science · doi:10.1007/978-1-4613-1367-0_10 · Cited by 110
Feature Learning with Raw-Waveform CLDNNs for Voice Activity Detection
Zazo, R., Sainath, T. N., Simko, G., & Parada, C. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-268 · Cited by 108
Speaker diarization from speech transcripts
Lamel, L. F., Gauvain, J., & Canseco-Rodriguez, L. (2004) · Interspeech (USB) · doi:10.21437/interspeech.2004-250 · Cited by 46
Park, T., & Georgiou, P. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1364 · Cited by 35
Efficient Implementation of Recurrent Neural Network Transducer in Tensorflow
Bagby, T., Rao, K., & Sim, K. C. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639690 · Cited by 29
Streaming End-to-end Speech Recognition For Mobile DevicesPreprint
He, Y., Sainath, T. N., Prabhavalkar, R., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.06621 · Cited by 23
Reducing the computational complexity for whole word models
Soltau, H., Liao, H., & Sak, H. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8268917 · Cited by 14
Self-citationImproving the efficiency of forward-backward algorithm using batched computation in TensorFlow
Sim, K. C., Narayanan, A., Bagby, T., Sainath, T. N., & Bacchiani, M. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8268944 · Cited by 10
Fully Supervised Speaker DiarizationPreprint
Zhang, A., Wang, Q., Zhu, Z., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.04719 · Cited by 8
Flagged first, then by citations
Cited by
Sorted by citations
Recent Advances in End-to-End Automatic Speech Recognition
Li, J. (2022) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000050 · Cited by 451
Speaker recognition based on deep learning: An overview
Bai, Z., & Zhang, X.-L. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.03.004 · Cited by 386
A review of speaker diarization: Recent advances with deep learning
Park, T., Kanda, N., Dimitriadis, D., Han, K. J., Watanabe, S., & Narayanan, S. S. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101317 · Cited by 405
Challenges of developing a digital scribe to reduce clinical documentation burden
Quiroz, J. C., Laranjo, L., Kocaballi, A. B., Berkovsky, S., Rezazadegan, D., & Coiera, E. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0190-1 · Cited by 162
The digital scribe in clinical practice: a scoping review and research agenda
van Buchem, M. M., Boosman, H., Bauer, M. P., Kant, I. M. J., Cammel, S. A., & Steyerberg, E. W. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00432-5 · Cited by 152
Metadata
- Authors
- Shafey, L. E., Soltau, H., & Shafran, I.
- Affiliations
- Google (United States) (US)
- Venue
- Interspeech 2019
- Publisher
- ISCA
- Published
- 2019-09-15
- Pages
- 396-400
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2957946919
- Cited by
- 104
- References
- 30
- Citations by year
- 2026: 9 · 2025: 8 · 2024: 14 · 2023: 5 · 2022: 21 · 2021: 29 · 2020: 16 · 2019: 2
- FWCI
- 6.50
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speaker diarization, speech recognition, RNN-T, medical conversation, physician-patient interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
