CoChat
Sign Up
Cited by 15Speech Recognition and SynthesisSpeech and dialogue systemsSpeech and Audio Processing

A Comprehensive Evaluation of Incremental Speech Recognition and Diarization for Conversational AI

Open access
Addlesee, Angus, Yu, Yanchao, Eshghi, Arash·2020·Proceedings of the 28th International Conference on Computational Linguistics
DOI 10.18653/v1/2020.coling-main.312
Open this paper in CoChat — free accountOpen PDF

Abstract

Automatic Speech Recognition (ASR) systems are increasingly powerful and more accurate, but also more numerous with several options existing currently as a service (e.g.Google, IBM, and Microsoft).Currently the most stringent standards for such systems are set within the context of their use in, and for, Conversational AI technology.These systems are expected to operate incrementally in real-time, be responsive, stable, and robust to the pervasive yet peculiar characteristics of conversational speech such as disfluencies and overlaps.In this paper we evaluate the most popular of such systems with metrics and experiments designed with these standards in mind.We also evaluate the speaker diarization (SD) capabilities of the same systems which will be particularly important for dialogue systems designed to handle multi-party interaction.We found that Microsoft has the leading incremental ASR system which preserves disfluent materials and IBM has the leading incremental SD system in addition to the ASR that is most robust to speech overlaps.Google strikes a balance between the two but none of these systems are yet suitable to reliably handle natural spontaneous conversations in real-time.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2020.coling-main.312

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. Angus Addlesee, Arash Eshghi, and Ioannis Konstas. 2019. Current challenges in spoken dialogue systems and why they are critical for those living with dementia. In Proceedings of the Dialogue for Good (DiGo) workshop on Speech and Language Technology Serving Society.
  2. G.S. Aist, J. Allen, E. Campana, L. Galescu, C.A. Gomez Gallo, S. Stoness, M. Swift, and M. Tanenhaus. 2006. Software architectures for incremental understanding of human speech. In Proceedings of the International Conference on Spoken Language Processing (ICSLP), Pittsburgh.
  3. G. Aist, J. Allen, E. Campana, C.A. Gomez Gallo, S. Stoness, M. Swift, and M.K. Tanenhaus. 2007. Incremental dialogue system faster than and preferred to its nonincremental counterpart. In Proceedings of the 29th Annual Conference of the Cognitive Science Society.
  4. Timo Baumann and David Schlangen. 2012. The inprotk 2012 release. In NAACL-HLT Workshop on Future Directions and Needs in the Spoken Dialog Community: Tools and Data, pages 29-32. Association for Compu- tational Linguistics.
  5. T. Baumann, O. Buß, and D. Schlangen. 2011. Evaluation and optimisation of incremental processors. Dialogue & Discourse, 2(1):113-141.
  6. Timo Baumann, Casey Kennington, Julian Hough, and David Schlangen. 2016. Recognising conversational speech: What an incremental asr should do for a dialogue system and how to get there. In International Workshop on Dialogue Systems Technology (IWSDS) 2016. Universität Hamburg.
  7. S.E. Brennan and M.F. Schober. 2001. How listeners compensate for disfluencies in spontaneous speech. Journal of Memory and Language, 44(2):274-296.
  8. Dimitrios Dimitriadis and Petr Fousek. 2017. Developing on-line speaker diarization system. In INTERSPEECH.
  9. Arash Eshghi, Igor Shalyminov, and Oliver Lemon. 2017. Bootstrapping incremental dialogue systems from minimal data: the generalisation power of dialogue grammars. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP). 14 https://wallscope.co.uk/ 15 https://www.thedatalab.com/ 16 https://spring-h2020.eu/
  10. Arash Eshghi. 2015. DS-TTR: An incremental, semantic, contextual parser for dialogue. In Proceedings of Semdial 2015 (goDial), the 19th workshop on the semantics and pragmatics of dialogue.
  11. Jonathan G. Fiscus, Jerome Ajot, Martial Michel, and John S. Garofolo. 2006. The rich transcription 2006 spring meeting recognition evaluation. In Steve Renals, Samy Bengio, and Jonathan G. Fiscus, editors, Machine Learning for Multimodal Interaction, Third International Workshop, MLMI 2006, Bethesda, MD, USA, May 1- 4, 2006, Revised Selected Papers, volume 4299 of Lecture Notes in Computer Science, pages 309-322. Springer.
  12. Daniel Garcia-Romero, David Snyder, Gregory Sell, Daniel Povey, and Alan McCree. 2017. Speaker diarization using deep neural network embeddings. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017, pages 4930-4934. IEEE.
  13. Israel D. Gebru, Silèye Ba, Xiaofei Li, and Radu Horaud. 2017. Audio-visual speaker diarization based on spatiotemporal bayesian fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39.
  14. John J. Godfrey, Edward Holliman, and J. McDaniel. 1992b. SWITCHBOARD: Telephone speech corpus for research and development. In Proceedings of IEEE ICASSP-92, pages 517-520, San Francisco, CA.
  15. P. G. T. Healey, Arash Eshghi, Christine Howes, and Matthew Purver. 2011. Making a contribution: Process- ing clarification requests in dialogue. In Proceedings of the 21st Annual Meeting of the Society for Text and Discourse, Poitiers, July.
  16. Julian Hough and David Schlangen. 2015. Recurrent neural networks for incremental disfluency detection. In IN- TERSPEECH 2015, 16th Annual Conference of the International Speech Communication Association, Dresden, Germany, September 6-10, 2015, pages 849-853.
  17. Julian Hough. 2015. Modelling Incremental Self-Repair Processing in Dialogue. Ph.D. thesis, Queen Mary University of London.
  18. Christine Howes and Arash Eshghi. 2017. Feedback relevance spaces: The organisation of increments in conver- sation. In IWCS 2017 -12th International Conference on Computational Semantics.
  19. Ruth Kempson, Ronnie Cann, Eleni Gregoromichelaki, and Stergios Chatzikiriakidis. 2016. Language as mecha- nisms for interaction. Theoretical Linguistics, 42(3-4):203-275.
  20. Ian McGraw and Alexander Gruenstein. 2012. Estimating word-stability during incremental speech recognition. In Proceedings of INTERSPEECH'12.
  21. Alfonso Ortega, Ignacio Vinals, Antonio Miguel, and Eduardo Lleida. 2016. The albayzin 2016 speaker diariza- tion evaluation. In Proc. IberSPEECH 2016, Lisbon, Portugal.
  22. Matthew Purver, Christine Howes, Eleni Gregoromichelaki, and Patrick G. T. Healey. 2009. Split utterances in dialogue: A corpus study. In Proceedings of the 10th Annual SIGDIAL Meeting on Discourse and Dialogue (SIGDIAL 2009 Conference), pages 262-271, London, UK, September. Association for Computational Linguis- tics.
  23. George Saon, Tom Sercu, Steven J. Rennie, and Hong-Kwang Jeff Kuo. 2016. The ibm 2016 english conversa- tional telephone speech recognition system. ArXiv, abs/1604.08242. · arXiv
  24. George Saon, Gakuto Kurata, Tom Sercu, Kartik Audhkhasi, Samuel Thomas, Dimitrios Dimitriadis, Xiaodong Cui, Bhuvana Ramabhadran, Michael Picheny, Lynn-Li Lim, Bergul Roomi, and Phil Hall. 2017. English conversational telephone speech recognition by humans and machines. In Proc. Interspeech 2017, pages 132- 136.
  25. E.A. Schegloff, Gail Jefferson, and Harvey Sacks. 1977. The preference for self-correction in the organization of repair in conversation. Language, 53(2):361-382.
  26. David Schlangen and Gabriel Skantze. 2009. A general, abstract model of incremental dialogue processing. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 710-718, Athens, Greece, March. Association for Computational Linguistics.
  27. David Schlangen, Timo Baumann, Hendrik Buschmeier, Okko Buß, Stefan Kopp, Gabriel Skantze, and Ramin Yaghoubzadeh. 2010. Middleware for incremental processing in conversational agents. In Proceedings of the SIGDIAL 2010 Conference, pages 51-54, Tokyo, Japan, September. Association for Computational Linguistics.
  28. F. Seide, G. Li, X. Chen, and D. Yu. 2011. Feature engineering in context-dependent deep neural networks for conversational speech transcription. In 2011 IEEE Workshop on Automatic Speech Recognition Understanding, pages 24-29.
  29. Ethan Selfridge, Iker Arizmendi, Peter Heeman, and Jason Williams. 2011. Stability and accuracy in incremental speech recognition. In Proceedings of the SIGDIAL 2011 Conference, pages 110-119, Portland, Oregon, June. Association for Computational Linguistics.
  30. Igor Shalyminov, Arash Eshghi, and Oliver Lemon. 2018. Multi-task learning for domain-general spoken disflu- ency detection in dialogue systems. In Proceedings of the 22nd Workshop on the Semantics and Pragmatics of Dialogue -Full Papers, Aix-en-Provence, France, November. SEMDIAL.
  31. Elizabeth Shriberg. 1996. Disfluencies in switchboard. In In Proceedings of the International Conference on Spoken Language Processing, volume 96, pages 3-6. Citeseer.
  32. Gabriel Skantze and Anna Hjalmarsson. 2010. Towards incremental speech generation in dialogue systems. In Proceedings of the SIGDIAL 2010 Conference, pages 1-8, Tokyo, Japan, September. Association for Computa- tional Linguistics.
  33. Quan Wang, Carlton Downey, Li Wan, Philip Andrew Mansfield, and Ignacio Lopez-Moreno. 2018. Speaker diarization with LSTM. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada, April 15-20, 2018, pages 5239-5243. IEEE.
  34. Aonan Zhang, Quan Wang, Zhenyao Zhu, John W. Paisley, and Chong Wang. 2018. Fully supervised speaker diarization. CoRR, abs/1810.04719. · arXiv
  35. Aonan Zhang, Quan Wang, Zhenyao Zhu, John W. Paisley, and Chong Wang. 2019. Fully supervised speaker diarization. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, pages 6301-6305. IEEE.

Cited by

Sorted by citations

Authors
Addlesee, A., Yu, Y., & Eshghi, A.
Affiliations
Heriot-Watt University (GB)
ORCID
Venue
Proceedings of the 28th International Conference on Computational Linguistics
Publisher
International Committee on Computational Linguistics
Published
2020
Pages
3492-3503
ISSN
1525-2477 (ISSN-L)
DOI
10.18653/v1/2020.coling-main.312
OpenAlex
W3115628053
Cited by
15
References
5
Citations by year
2026: 2 · 2025: 2 · 2024: 2 · 2023: 3 · 2022: 3 · 2021: 3
FWCI
0.86
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • International Conference on Computational Linguistics (publishedVersion, cc-by)
  • Edinburgh Napier Research Repository (Edinburgh Napier University) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Keywords
speaker diarization, conversational AI, real-time speech recognition, IBM, Google

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review