CoChat
Sign Up
PreprintCited by 47Speech Recognition and SynthesisSpeech and Audio ProcessingNatural Language Processing Techniques

Seamless: Multilingual Expressive and Streaming Speech Translation

Open access
Communication, Seamless, Barrault, Loïc, Chung, Yu-An, Meglioli, Mariano Coria, Dale, David, Dong, Ning, Duppenthaler, Mark, Duquenne, Paul-Ambroise +57 more·2023·arXiv
DOI 10.48550/arxiv.2312.05187arXiv 2312.05187
Open this paper in CoChat — free accountOpen PDF

Abstract

Large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model-SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. SeamlessM4T v2 provides the foundation on which our next two models are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one's voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. The contributions to this work are publicly released and accessible at https://github.com/facebookresearch/seamless_communication

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2312.05187

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LID based filtering and Blaser2.0 were used to remove low quality data.

    “We use LID based filtering and Blaser2.0 (Seamless Communication et al., 2023) to remove low quality data.”

    From The Llama 3 Herd of Models (2024) · cites this paper as Seamless Communication et al., 2023

    The “LID filtering” section says the authors discarded pairs whose target sentences did not appear to be in the expected language. The Introduction describes Blaser 2.0 as a metric for evaluating translation quality, but the passages found do not say it was used to remove low-quality data. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 107

  1. P. Aguero, J. Adell, and A. Bonafonte. Prosody generation for speech-to-speech translation. In 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, volume 1, pages I-I. IEEE, 2006.
  2. P. Andrews, G. Wenzek, K. Heffernan, O. Çelebi, A. Sun, A. Kamran, Y. Guo, A. Mourachko, H. Schwenk, and A. Fan. stopes -modular machine translation pipelines. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 258-265, Abu Dhabi, UAE, Dec. 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.emnlp-demos.
  3. G. K. Anumanchipalli, L. C. Oliveira, and A. W. Black. Intent transfer in speech-to-speech machine translation. In 2012 IEEE Spoken Language Technology Workshop (SLT), pages 153-158. IEEE, 2012.
  4. L. J. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. CoRR, abs/1607.06450, 2016. · arXiv
  5. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12449-12460.
  6. Curran Associates, Inc., 2020. URL https: //proceedings.neurips.cc/paper_files/paper/2020/file/92d1e1eb1cd6f9fba3227870bb6d7f07-Paper.pdf.
  7. L. Bentivogli, M. Cettolo, M. Gaido, A. Karakanta, A. Martinelli, M. Negri, and M. Turchi. Cascade versus direct speech translation: Do the differences still make a difference? arXiv preprint arXiv:2106.01045, 2021. · arXiv
  8. E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. Gölge, and M. A. Ponti. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning, pages 2709-2720.
  9. PMLR, 2022.
  10. G. Chen, Y. Wu, S. Liu, T. Liu, X. Du, and F. Wei. Wavmark: Watermarking for audio generation. arXiv preprint arXiv:2308.12770, 2023a. · arXiv
  11. M. Chen, P.-A. Duquenne, P. Andrews, J. Kao, A. Mourachko, H. Schwenk, and M. R. Costa-jussà. BLASER: A text-free speech-to-speech translation evaluation metric. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9064-9079, Toronto, Canada, July 2023b. Association for Computational Linguistics. URL https://aclanthology.org/2023.acl-long.504.
  12. P.-J. Chen, K. Tran, Y. Yang, J. Du, J. Kao, Y.-A. Chung, P. Tomasello, P.-A. Duquenne, H. Schwenk, H. Gong, H. Inaguma, S. Popuri, C. Wang, J. Pino, W.-N. Hsu, and A. Lee. Speech-to-speech translation for a real-world unwritten language. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4969-4983, Toronto, Canada, July 2023c. Association for Computational Linguistics. URL https://aclanthology.org/2023. findings-acl.307.
  13. S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE J. Sel. Top. Signal Process., 16(6):1505-1518, 2022.
  14. C.-C. Chiu* and C. Raffel*. Monotonic chunkwise attention. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=Hko85plCW.
  15. C.-C. Chiu, J. Qin, Y. Zhang, J. Yu, and Y. Wu. Self-supervised learning with random-projection quantizer for speech recognition. In International Conference on Machine Learning, pages 3915-3924. PMLR, 2022.
  16. K. Cho and M. Esipova. Can neural machine translation do simultaneous translation? arXiv preprint arXiv:1606.02012, 2016. · arXiv
  17. D. Cokely. The effects of lag time on interpreter errors. Sign Language Studies, pages 341-375, 1986.
  18. A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006.13979, 2020. · arXiv
  19. A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. arXiv preprint arXiv:2205.12446, 2022. URL https://arxiv.org/abs/2205.12446. · arXiv
  20. M. R. Costa-jussà, P. Andrews, E. Smith, P. Hansanti, C. Ropers, E. Kalbassi, C. Gao, D. Licht, and C. Wood. Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale. In EMNLP, 2023.
  21. M. Costa-jussà. An analysis of gender bias studies in natural language processing. Nature Machine Intelligence, pages 495--496, 2019. doi: 10.1038/s42256-019-0105-5. · DOI
  22. M. R. Costa-jussà, D. Dale, M. Elbayad, and B. Yu. Added toxicity mitigation at inference time for massively multilingual and multimodal translation. CoRR, abs/2311.06532, 2023a. URL https://arxiv.org/abs/2311.06532. · arXiv
  23. M. R. Costa-jussà, E. Smith, C. Ropers, D. Licht, J. Maillard, J. Ferrando, and C. Escolano. Toxicity in multilingual machine translation at scale. In EMNLP, 2023b.
  24. J. Costello. Aac intervention in the intensive care unit: The children's hospital boston model. Augmentative and alternative communication, 16(3):137-153, 2000.
  25. A. Défossez, J. Copet, G. Synnaeve, and Y. Adi. High fidelity neural audio compression. ArXiv, abs/2210.13438, 2022. URL https://api.semanticscholar.org/CorpusID:253097788. · arXiv
  26. L. Delander, M. Hammarstedt, J. M Ånsson, and E. Nyberg. Integration of immigrants: The role of language proficiency and experience. Evaluation review, 29(1):24-41, 2005. doi: 10.1177/0193841X04270230. URL https: //doi.org/10.1177/0193841X04270230. · DOI
  27. H. Ding and L. Hargraves. Stress-associated poor health among adult immigrants with a language barrier in the united states. Journal of immigrant and minority health, 11:446-452, 2009.
  28. Q. T. Do, S. Sakti, and S. Nakamura. Toward expressive speech translation: A unified sequence-to-sequence lstms approach for translating words and emphasis. In INTERSPEECH, pages 2640-2644, 2017.
  29. Z.-Y. Dou and G. Neubig. Word alignment by fine-tuning embeddings on parallel corpora. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2021.
  30. Z. Du, B. Sisman, K. Zhou, and H. Li. Expressive voice conversion: A joint framework for speaker identity and emotional style transfer. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 594-601. IEEE, 2021.
  31. P.-A. Duquenne, H. Gong, and H. Schwenk. Multimodal and multilingual embeddings for large-scale speech mining. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 15748-15761.
  32. Curran Associates, Inc., 2021. URL https://proceedings. neurips.cc/paper_files/paper/2021/file/8466f9ace6a9acbe71f75762ffc890f1-Paper.pdf.
  33. P.-A. Duquenne, K. Heffernan, A. Mourachko, B. Sagot, and H. Schwenk. Expressivity-aware SONAR: speech decoder, 2023a. P.-A. Duquenne, H. Schwenk, and B. Sagot. SONAR: sentence-level multimodal and language-agnostic representations, 2023b. URL https://arxiv.org/abs/2308.11466. · arXiv
  34. F. Eyben, M. Wöllmer, and S. Schuller, Björn. "opensmile -the munich versatile and fast open-source audio feature extractor". In Proceedings of the 2010 ACM Multimedia (MM), ACM, Florence, Italy, pages 1459-1462, 2010.
  35. P. Fernandez, G. Couairon, H. Jégou, M. Douze, and T. Furon. The stable signature: Rooting watermarks in latent diffusion models. ICCV, 2023.
  36. S. Ghosh, S. Lepcha, S. Sakshi, R. R. Shah, and S. Umesh. Detoxy: A large-scale multimodal dataset for toxicity classification in spoken utterances. In Interspeech, 2021. URL https://api.semanticscholar.org/CorpusID: 247940269.
  37. H. Gong, N. Dong, S. Popuri, V. Goswami, A. Lee, and J. Pino. Multilingual speech-to-speech translation into multiple target languages. arXiv preprint arXiv:2307.08655, 2023a. · arXiv
  38. S. Gong, M. Li, J. Feng, Z. Wu, and L. Kong. DiffuSeq: Sequence to sequence text generation with diffusion models. In International Conference on Learning Representations, ICLR, 2023b.
  39. J. Gu, J. Bradbury, C. Xiong, V. O. Li, and R. Socher. Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02281, 2017a. · arXiv
  40. J. Gu, G. Neubig, K. Cho, and V. O. Li. Learning to Translate in Real-time with Neural Machine Translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1053-1062, Valencia, Spain, Apr. 2017b. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/E17-1099.
  41. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.
  42. W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
  43. W.-C. Huang, B. Peloquin, J. Kao, C. Wang, H. Gong, E. Salesky, Y. Adi, A. Lee, and P.-J. Chen. A holistic cascade system, benchmark, and human evaluation protocol for expressive speech-to-speech translation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
  44. J. Hutchins. Multiple uses of machine translation and computerised translation tools. Machine Translation, pages 13-20, 2009.
  45. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 448-456. PMLR, 2015.
  46. Y. Jia, M. Johnson, W. Macherey, R. J. Weiss, Y. Cao, C.-C. Chiu, N. Ari, S. Laurenzo, and Y. Wu. Leveraging weakly supervised data to improve end-to-end speech-to-text translation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7180-7184. IEEE, 2019.
  47. Y. Jia, M. T. Ramanovich, T. Remez, and R. Pomerantz. Translatotron 2: High-quality direct speech-to-speech translation with voice preservation. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 10120-10134. PMLR, 17-23 Jul 2022a. URL https://proceedings.mlr.press/v162/jia22b.html.
  48. Y. Jia, M. Tadmor Ramanovich, Q. Wang, and H. Zen. CVSS corpus and massively multilingual speech-to-speech translation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6691-6703, Marseille, France, June 2022b. European Language Resources Association. URL https://aclanthology.org/2022. lrec-1.720.
  49. J. Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3): 535-547, 2019.
  50. L. Juvela and X. Wang. Collaborative watermarking for adversarial speech synthesis. arXiv preprint arXiv:2309.15224, 2023. · arXiv
  51. E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. ArXiv, abs/2302.03540, 2023. URL https://api.semanticscholar.org/CorpusID:256627687. · arXiv
  52. J. Kim, S. Kim, J. Kong, and S. Yoon. Glow-TTS: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33:8067-8077, 2020.
  53. J. Kim, J. Kong, and J. Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pages 5530-5540. PMLR, 2021.
  54. J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein. A watermark for large language models. arXiv preprint arXiv:2301.10226, 2023. · arXiv
  55. A. Koenecke, A. Nam, E. Lake, J. Nudell, M. Quartey, Z. Mengesha, C. Toups, J. R. Rickford, D. Jurafsky, and S. Goel. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117 (14):7684-7689, 2020.
  56. J. Kong, J. Kim, and J. Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-17033, 2020.
  57. R. Kraut, J. Galegher, R. Fish, and B. Chalfonte. Task requirements and media choice in collaborative writing. Human-Computer Interaction, 7(4):375-407, 1992.
  58. M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. arXiv preprint arXiv:2306.15687, 2023. · arXiv
  59. K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini. Deduplicating training data makes language models better. CoRR, 2021. URL https://arxiv.org/abs/2107.06499. · arXiv
  60. S. C. Levinson. Turn-taking in human communication-origins and implications for language processing. Trends in cognitive sciences, 20(1):6-14, 2016.
  61. M. P. Lewis, editor. Ethnologue: Languages of the World. SIL International, Dallas, TX, USA, sixteenth edition, 2009.
  62. Z. Liu and B. Mak. Cross-lingual multi-speaker text-to-speech synthesis for voice cloning without using parallel corpus for unseen speakers. arXiv preprint arXiv:1911.11601, 2019. · arXiv
  63. K. Lueck and M. Wilson. Acculturative stress in latino immigrants: The impact of social, socio-psychological and migration-related factors. International Journal of Intercultural Relations, 35(2):186-195, 2011.
  64. X. Ma, J. Pino, and P. Koehn. SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 582-587, Suzhou, China, Dec. 2020b. Association for Computational Linguistics. URL https://www.aclweb.org/ anthology/2020.aacl-main.
  65. X. Ma, J. M. Pino, J. Cross, L. Puzon, and J. Gu. Monotonic multihead attention. In International Conference on Learning Representations, 2020c. URL https://openreview.net/forum?id=Hyg96gBKPS.
  66. M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pages 498-502, 2017.
  67. M. Morise, F. Yokomori, and K. Ozawa. World: A vocoder-based high-quality speech synthesis system for real-time applications. IEICE Trans. Inf. Syst., 99-D:1877-1884, 2016.
  68. G. Neubig, S. Sakti, T. Toda, S. Nakamura, et al. Collection and analysis of a japanese-english emphasized speech corpora. In 2014 17th Oriental Chapter of the International Committee for the Co-ordination and Standardization of Speech Databases and Assessment Techniques (COCOSDA), pages 1-5. IEEE, 2014.
  69. M. K. Ngueajio and G. Washington. Hey asr system! why aren't you more inclusive? automatic speech recognition systems' bias and proposed bias mitigation techniques. a literature review. In International Conference on Human- Computer Interaction, pages 421-440. Springer, 2022.
  70. T. Q. Nguyen and D. Chiang. Transfer learning across low-resource, related languages for neural machine translation. arXiv preprint arXiv:1708.09803, 2017. · arXiv
  71. NLLB Team, M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Barrault, G. Mejia-Gonzalez, P. Hansanti, J. Hoffman, S. Jarrett, K. R. Sadagopan, D. Rowe, S. Spruit, C. Tran, P. Andrews, N. F. Ayan, S. Bhosale, S. Edunov, A. Fan, C. Gao, V. Goswami, F. Guzmán, P. Koehn, A. Mourachko, C. Ropers, S. Saleem, H. Schwenk, and J. Wang. No language left behind: Scaling human-centered machine translation, 2022. URL https://arxiv.org/abs/2207.04672. · arXiv
  72. M. F. Orellana, L. Dorner, and L. Pulido. Accessing assets: Immigrant youth's work as family translators or" para-phrasers". Social problems, 50(4):505-524, 2003.
  73. B. N. Oreshkin, P. Rodriguez, and A. Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, page 719-729, Red Hook, NY, USA, 2018.
  74. Curran Associates Inc. S. Papi, M. Gaido, M. Negri, and M. Turchi. Over-generation cannot be rewarded: Length-adaptive average lagging for simultaneous speech translation. In J. Ive and R. Zhang, editors, Proceedings of the Third Workshop on Automatic Simultaneous Translation, pages 12-17, Online, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.autosimtrans-1.2. URL https://aclanthology.org/2022.autosimtrans-1. · DOI
  75. D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. In Proc. Interspeech 2019, pages 2613-2617, 2019.
  76. J. Parry, D. Palaz, G. Clarke, P. Lecomte, R. Mead, M. Berger, and G. Hofer. Analysis of deep learning architectures for cross-corpus speech emotion recognition. In Interspeech, pages 1656-1660, 2019.
  77. E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. C. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
  78. E. Perez, S. Huang, H. F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. CoRR, abs/2202.03286, 2022. URL https://arxiv.org/abs/2202.03286. · arXiv
  79. A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux. Speech Resynthesis from Discrete Disentangled Self-Supervised Representations. In Proc. Interspeech 2021, 2021.
  80. K. Prajwal, R. Mukhopadhyay, V. P. Namboodiri, and C. Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pages 484-492, 2020.
  81. L. Qu, T. Li, C. Weber, T. Pekarek-Rosin, F. Ren, and S. Wermter. Disentangling prosody representations with unsupervised speech reconstruction. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:39-54, 2022.
  82. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356, 2022. · arXiv
  83. C. Raffel, M.-T. Luong, P. J. Liu, R. J. Weiss, and D. Eck. Online and linear-time attention by enforcing monotonic alignments. In Proceedings of the 34th International Conference on Machine Learning -Volume 70, ICML'17, pages 2837-2846, Sydney, NSW, Australia, Aug. 2017. JMLR.org.
  84. H. Rashkin, E. M. Smith, M. Li, and Y.-L. Boureau. Towards empathetic open-domain conversation models: a new benchmark and dataset. In ACL, 2019.
  85. M. Reid, V. J. Hellendoorn, and G. Neubig. Diffuser: Discrete diffusion via edit-based reconstruction. arXiv preprint arXiv:2210.16886, 2022. · arXiv
  86. Y. Ren, C. Hu, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu. FastSpeech 2: Fast and high-quality end-to-end text-to-speech. In Proc. ICLR, 2021a.
  87. Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu. FastSpeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations, 2021b. URL https://openreview.net/ forum?id=piLPYqxtWuA.
  88. S. Rouard, F. Massa, and A. Défossez. Hybrid transformers for music source separation. In ICASSP 23, 2023.
  89. P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. de Chaumont Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov, H. Muckenhirn, D. R. Padfield, J. Qin, D. Rozenberg, T. N. Sainath, J. Schalkwyk, M. Sharifi, M. D. Tadmor, Ramanovich, M. Tagliasacchi, A. Tudor, M. Velimirovi'c, D. Vincent, J. Yu, Y. Wang, V. Zayats, N. Zeghidour, Y. Zhang, Z. Zhang, L. Zilka, and C. H. Frank. Audiopalm: A large language model that can speak and listen. ArXiv, abs/2306.12925, 2023b. URL https://api.semanticscholar.org/CorpusID:259224345. · arXiv
  90. B. Schuller, S. Steidl, A. Batliner, F. Burkhardt, L. Devillers, C. MüLler, and S. Narayanan. Paralinguistics in speech and language-state-of-the-art and the challenge. Computer Speech & Language, 27(1):4-39, 2013.
  91. Seamless Communication, L. Barrault, Y.-A. Chung, M. C. Meglioli, D. Dale, N. Dong, P.-A. Duquenne, H. Elsahar, H. Gong, K. Heffernan, J. Hoffman, C. Klaiber, P. Li, D. Licht, J. Maillard, A. Rakotoarison, K. R. Sadagopan, G. Wenzek, E. Ye, B. Akula, P.-J. Chen, N. E. Hachem, B. Ellis, G. M. Gonzalez, J. Haaheim, P. Hansanti, R. Howes, B. Huang, M.-J. Hwang, H. Inaguma, S. Jain, E. Kalbassi, A. Kallet, I. Kulikov, J. Lam, D. Li, X. Ma, R. Mavlyutov, B. Peloquin, M. Ramadan, A. Ramakrishnan, A. Sun, K. Tran, T. Tran, I. Tufanov, V. Vogeti, C. Wood, Y. Yang, B. Yu, P. Andrews, C. Balioglu, M. R. Costa-jussà, O. Celebi, M. Elbayad, C. Gao, F. Guzmán, J. Kao, A. Lee, A. Mourachko, J. Pino, S. Popuri, C. Ropers, S. Saleem, H. Schwenk, P. Tomasello, C. Wang, J. Wang, and S. Wang. Seamlessm4t-massively multilingual & multimodal machine translation, 2023. URL https://arxiv.org/abs/2308.11596. · arXiv
  92. J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan, R. A. Saurous, Y. Agiomvrgiannakis, and Y. Wu. Natural TTS synthesis by conditioning WaveNet on Mel spectrogram predictions. In Proc. ICASSP, pages 4779-4783, 2018.
  93. J. Shen, Y. Jia, M. Chrzanowski, Y. Zhang, I. Elias, H. Zen, and Y. Wu. Non-attentive tacotron: Robust and controllable neural tts synthesis including unsupervised duration modeling, 2020. URL https://arxiv.org/abs/2010.04301. · arXiv
  94. K. Shen, Z. Ju, X. Tan, Y. Liu, Y. Leng, L. He, T. Qin, S. Zhao, and J. Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. ArXiv, abs/2304.09116, 2023. URL https: //api.semanticscholar.org/CorpusID:258187322. · arXiv
  95. K. J. Shih, R. Valle, R. Badlani, A. Lancucki, W. Ping, and B. Catanzaro. RAD-TTS: Parallel flow-based tts with robust alignment learning and diverse synthesis. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
  96. R. Skerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang, D. Stanton, J. Shor, R. Weiss, R. Clark, and R. A. Saurous. Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4693-4702. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/skerry-ryan18a.html.
  97. D. Snyder, G. Chen, and D. Povey. Musan: A music, speech, and noise corpus. arXiv preprint arXiv:1510.08484, 2015. · arXiv
  98. L. Specia, F. Blain, M. Fomicheva, C. Zerva, Z. Li, V. Chaudhary, and A. F. T. Martins. Findings of the WMT 2021 shared task on quality estimation. In Proceedings of the Sixth Conference on Machine Translation, pages 684-725, Online, Nov. 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.wmt-1.
  99. I. G. Sánchez and M. F. Orellana. The construction of moral and social identity in immigrant children's narratives-in- translation. Linguistics and education, 17(3):209-239, 2006.
  100. Tjandra, S. Sakti, and S. Nakamura. Speech-to-speech translation between untranscribed unknown languages. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 593-600. IEEE, 2019.
  101. C. Wang, J. Pino, A. Wu, and J. Gu. CoVoST: A diverse multilingual speech-to-text translation corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4197-4203, Marseille, France, May 2020a. European Language Resources Association. ISBN 979-10-95546-34-4. URL https://aclanthology.org/2020.lrec-1.517.
  102. S. Wang, N. Cooper, M. Eby, and E. S. Jo. From human-centered to social-centered artificial intelligence: Assessing chatgpt's impact through disruptive events. arXiv preprint arXiv:2306.00227, 2023b. · arXiv
  103. Y. Wang, J. Bai, R. Huang, R. Li, Z. Hong, and Z. Zhao. Speech-to-speech translation with discrete-unit-based style transfer. CoRR, abs/2309.07566, 2023c. · arXiv
  104. A. Waytz, J. Heafner, and N. Epley. The mind in the machine: Anthropomorphism increases trust in an autonomous vehicle. Journal of experimental social psychology, 52:113-117, 2014.
  105. Y. Wen, J. Kirchenbauer, J. Geiping, and T. Goldstein. Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust. arXiv preprint arXiv:2305.20030, 2023. · arXiv
  106. N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2022. URL https: //api.semanticscholar.org/CorpusID:236149944.
  107. Z. Zhang, L. Zhou, J. Ao, S. Liu, L. Dai, J. Li, and F. Wei. SpeechUT: Bridging speech and text with hidden-unit for encoder-decoder based speech-text pre-training. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1663-1676, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.emnlp-main.108.

Cited by

Sorted by citations

Related

Authors
Communication, S., Barrault, L., Chung, Y.-A., Meglioli, M. C., Dale, D., Dong, N., Duppenthaler, M., Duquenne, P.-A., Ellis, B., Elsahar, H., Haaheim, J., Hoffman, J., Hwang, M.-J., Inaguma, H., Klaiber, C., Kulikov, I., Li, P., Licht, D., Maillard, J., Mavlyutov, R., Rakotoarison, A., Sadagopan, K. R., Ramakrishnan, A., Tran, T., Wenzek, G., Yang, Y., Ye, E., Evtimov, I., Fernandez, P., Gao, C., Hansanti, P., Kalbassi, E., Kallet, A., Kozhevnikov, A., Gonzalez, G. M., Roman, R. S., Touret, C., Wong, C., Wood, C., Yu, B., Andrews, P., Balioglu, C., Chen, P.-J., Costa-jussà, M. R., Elbayad, M., Gong, H., Guzmán, F., Heffernan, K., Jain, S., Kao, J., Lee, A., Ma, X., Mourachko, A., Peloquin, B., Pino, J., Popuri, S., Ropers, C., Saleem, S., Schwenk, H., Sun, A., Tomasello, P., Wang, C., Wang, J., Wang, S., & Williamson, M.
Affiliations
  • University of California, Berkeley (US)
  • Institut national de recherche en sciences et technologies du numérique (FR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2312.05187
arXiv
2312.05187
OpenAlex
W4389600306
Cited by
45
References
0
Citations by year
2026: 2 · 2025: 24 · 2024: 19
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
speech translation, multilingual speech translation, expressive speech synthesis, prosody preservation, low-latency translation, multimodal machine translation, red teaming, toxicity detection, gender bias, watermarking, interlingual communication

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review