The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
WhisperX: Time-Accurate Speech Transcription of Long-Form Audio
Open accessAbstract
Large-scale, weakly-supervised speech recognition models, such as Whisper, have demonstrated impressive results on speech recognition across domains and languages. However, their application to long audio transcription via buffered or sliding window approaches is prone to drifting, hallucination & repetition; and prohibits batched transcription due to their sequential nature. Further, timestamps corresponding each utterance are prone to inaccuracies and word-level timestamps are not available out-of-the-box. To overcome these challenges, we present WhisperX, a time-accurate speech recognition system with word-level timestamps utilising voice activity detection and forced phoneme alignment. In doing so, we demonstrate state-of-the-art performance on long-form transcription and word segmentation benchmarks. Additionally, we show that pre-segmenting audio with our proposed VAD Cut & Merge strategy improves transcription quality and enables a twelve-fold transcription speedup via batched inference.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“They also tend to be inaccurate in predicting word-level timestamps [Bain et al. 2023].”
From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Bain et al. 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“A related study reported a speed increase of up to twelve times without sacriicing transcription accuracy [Bain et al. 2023].”
From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Bain et al. 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369
Robust Speech Recognition via Large-Scale Weak Supervision
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022) · arXiv · doi:10.48550/arxiv.2212.04356 · Cited by 1,953
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3188113 · Cited by 1,926
Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi
McAuliffe, M., Socolof, M., Mihuc, S., Wagner, M., & Sonderegger, M. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1386 · Cited by 1,169
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., & Sivic, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00272 · Cited by 986
The AMI Meeting Corpus: A Pre-announcement
Carletta, J., Ashby, S., Bourban, S., Flynn, M., Guillemot, M., Hain, T., Kadlec, J., Vasilis Karaiskos, B., Kraaij, W., Kronenthal, M., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_3 · Cited by 939
Unsupervised Cross-Lingual Representation Learning for Speech Recognition
Conneau, A., Baevski, A., Collobert, R., Mohamed, A., & Auli, M. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-329 · Cited by 637
Wang, C., Rivière, M., Lee, A., Wu, A., Talnikar, C., Haziza, D., Williamson, M., Pino, J., & Emmanuel Dupoux, A. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.80 · Cited by 383
Wang, Q., Downey, C., Wan, L., Mansfield, P. A., & Moreno, I. L. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462628 · Cited by 338
Ssast: Self-supervised audio spectrogram transformer
Gong, Y., Lai, C.-I., Chung, Y.-A., & Glass, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21315 · Cited by 294
TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation
Hernandez, F., Nguyen, V., Ghannay, S., Tomashenko, N., & Estève, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99579-3_21 · Cited by 280
Fully Supervised Speaker Diarization
Zhang, A., Wang, Q., Zhu, Z., Paisley, J. W., & Wang, C. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683892 · Cited by 235
Prosodylab-aligner: A tool for forced alignment of laboratory speech
Gorman, K., Howell, J., & Wagner, M. (2011) · Canadian acoustics · Cited by 223
Automatic segmentation and labeling of speech based on Hidden Markov Models
Brugnara, F., Falavigna, D., & Omologo, M. (1993) · Speech Communication · doi:10.1016/0167-6393(93)90083-w · Cited by 167
Leveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation
Ye, J., Johnson, M., Macherey, W., Weiss, R. J., Cao, Y. J., Chiu, C., Ari, N., Laurenzo, S., & Wu, Y. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683343 · Cited by 153
Sub-word Level Lip Reading With Visual Attention
R, P. K., Afouras, T., & Zisserman, A. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00510 · Cited by 137
Self-citationA Comparison of End-to-End Models for Long-Form Speech Recognition
Chiu, C., Han, W., Zhang, Y., Pang, R., Kishchenko, S., Nguyen, P., Narayanan, A., Liao, H., Zhang, S., Kannan, A., et al. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003854 · Cited by 83
CTC-Segmentation of Large Corpora for German End-to-End Speech Recognition
Kürzinger, L., Winkelbauer, D., Li, L., Watzel, T., & Rigoll, G. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60276-5_27 · Cited by 80
Automatic segmentation combining an HMM-based approach and spectral boundary correction
Kim, Y.-J., & Conkie, A. D. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2002-88 · Cited by 61
Automatic phonetic segmentation using boundary models
Yuan, J., Ryant, N., Liberman, M., Stolcke, A., Mitra, V., & Wang, W. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-540 · Cited by 59
Highly accurate phonetic segmentation using boundary correction models and system fusion
Stolcke, A., Ryant, N., Mitra, V., Yuan, J., Wang, W., & Liberman, M. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854665 · Cited by 42
Unsupervised Sound Separation Using Mixtures of MixturesPreprint
Wisdom, S., Tzinis, E., Erdogan, H., Weiss, R. J., Wilson, K., & Hershey, J. (2020) · arXiv.org · doi:10.48550/arxiv.2006.12701 · Cited by 27
Neufa: Neural Network Based End-to-End Forced Alignment with Bidirectional Attention Mechanism
Li, J., Meng, Y., Wu, Z., Meng, H. M. L., Tian, Q., Wang, Y., & Wang, Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747085 · Cited by 21
Augmentation adversarial training for self-supervised speaker recognitionPreprint
Huh, J., Heo, H. S., Kang, J., Watanabe, S., & Chung, J. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.12085 · Cited by 19
Self-citationAugmentation Adversarial Training for Self-Supervised Speaker Representation Learning
Kang, J., Huh, J., Heo, H. S., & Chung, J. S. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3200915 · Cited by 16
Self-citationVisual Keyword Spotting with AttentionPreprint
Prajwal, K. R., Momeni, L., Afouras, T., & Zisserman, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15957 · Cited by 8
Self-citation
Flagged first, then by citations
Research works printed but not linked: 5
- H. Chen, H. Zhang, L. Wang, K. A. Lee, M. Liu, and J. Dang, "Self-supervised audio-visual speaker representation with co- meta learning," in ICASSP, 2023.
- S. Wisdom, E. Tzinis, H. Erdogan, R. Weiss, K. Wilson, and J. Hershey, "Unsupervised sound separation using mixture invari- ant training," NeurIPS, vol. 33, pp. 3846-3857, 2020.
- G. Gelly and J.-L. Gauvain, "Optimization of rnn-based speech activity detection," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 3, pp. 646-656, 2018.
- H. Bredin, R. Yin, J. M. Coria, G. Gelly, P. Korshunov, M. Lavechin, D. Fustes, H. Titeux, W. Bouaziz, and M.-P. Gill, "pyannote.audio: neural building blocks for speaker diarization," in ICASSP, 2020.
- Y.-Y. Yang, M. Hira, Z. Ni, A. Astafurov, C. Chen, C. Puhrsch, D. Pollack, D. Genzel, D. Greenberg, E. Z. Yang et al., "Torchau- dio: Building blocks for audio and speech processing," in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6982-6986.
Cited by
Sorted by citations
Automatic Speech Recognition: A survey of deep learning techniques and approaches
Ahlawat, H., Aggarwal, N., & Gupta, D. (2025) · International Journal of Cognitive Computing in Engineering · doi:10.1016/j.ijcce.2024.12.007 · Cited by 86
Emilia: An Extensive, Multilingual, and Diverse Speech Dataset For Large-Scale Speech Generation
He, H., Shang, Z., Wang, C., Li, X., Gu, Y., Hua, H., Liu, L., Yang, C., Li, J., Shi, P., et al. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832365 · Cited by 141
Measuring the Accuracy of Automatic Speech Recognition Solutions
Kuhn, K., Kersken, V., Reuter, B., Egger, N., & Zimmermann, G. (2023) · ACM Transactions on Accessible Computing · doi:10.1145/3636513 · Cited by 44
Take the aTrain. Introducing an interface for the Accessible Transcription of Interviews
Haberl, A., Fleiß, J., Kowald, D., & Thalmann, S. (2024) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2024.100891 · Cited by 44
AutoAD: Movie Description in Context
Han, T., Bain, M., Nagrani, A., Varol, G., Xie, W., & Zisserman, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01815 · Cited by 43
Self-citation
Metadata
- Authors
- Bain, M., Huh, J., Han, T., & Zisserman, A.
- Affiliations
- University of Oxford (GB)
- Venue
- INTERSPEECH 2023
- Publisher
- ISCA
- Published
- 2023-08-20
- Pages
- 4489-4493
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W4385822293
- Cited by
- 337
- References
- 29
- Citations by year
- 2026: 134 · 2025: 123 · 2024: 65 · 2023: 15
- FWCI
- 59.71
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Engineering and Physical Sciences Research Council (EP/T028572/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Keywords
- speech transcription, voice activity detection, forced alignment, phone recognition, timestamping
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
