CoChat

Moshi: a speech-text foundation model for real-time dialogue

Open access
Défossez, Alexandre, Mazaré, Laurent, Orsini, Manu, Royer, Amélie, Pérez, Patrick, Jégou, Hervé, Grave, Edouard, Zeghidour, Neil · 2024 · arXiv

The DOI matches its DataCite record; 103 of the 103 references it cites were found in a registry or an index; none of the 66 sources its claims cite is checked for support yet.

Abstract

We introduce Moshi, a speech-text foundation model and full-duplex spoken dialogue framework. Current systems for spoken dialogue rely on pipelines of independent components, namely voice activity detection, speech recognition, textual dialogue and text-to-speech. Such frameworks cannot emulate the experience of real conversations. First, their complexity induces a latency of several seconds between interactions. Second, text being the intermediate modality for dialogue, non-linguistic information that modifies meaning -- such as emotion or non-speech sounds -- is lost in the interaction. Finally, they rely on a segmentation into speaker turns, which does not take into account overlapping speech, interruptions and interjections. Moshi solves these independent issues altogether by casting spoken dialogue as speech-to-speech generation. Starting from a text language model backbone, Moshi generates speech as tokens from the residual quantizer of a neural audio codec, while modeling separately its own speech and that of the user into parallel streams. This allows for the removal of explicit speaker turns, and the modeling of arbitrary conversational dynamics. We moreover extend the hierarchical semantic-to-acoustic token generation of previous work to first predict time-aligned text tokens as a prefix to audio tokens. Not only this "Inner Monologue" method significantly improves the linguistic quality of generated speech, but we also illustrate how it can provide streaming speech recognition and text-to-speech. Our resulting model is the first real-time full-duplex spoken large language model, with a theoretical latency of 160ms, 200ms in practice, and is available at https://github.com/kyutai-labs/moshi.

Overview

CoChat Summary

TL;DR

Moshi is a speech-text foundation model that casts spoken dialogue as full-duplex speech-to-speech generation, combining a 7B text LLM (Helium), a streaming semantic-acoustic codec (Mimi), multi-stream modeling, and Inner Monologue. It reports the first real-time full-duplex spoken LLM with 160 ms theoretical latency (200 ms practical), state-of-the-art spoken QA among compared speech-text models, and streaming ASR/TTS derived by changing text-audio delay.

Key findings

  • Moshi achieves 160 ms theoretical latency and 200 ms in practice, models 5 min of context, and is full-duplex; Inner Monologue nearly triples spoken QA accuracy: Web Questions 26.6 vs 9.2 without it, LlaMA Questions 62.3 vs 21.0, and Trivia QA 22.8 vs 7.3.
  • Mimi is a 24 kHz, 12.5 Hz, 1.1 kbps causal codec; adversarial-only training reaches MUSHRA 81.0 vs 58.8 for an Encodec-style loss mix, and split RVQ improves MUSHRA from 57.8 to 64.0 while ABX degrades from 6.5% to 8.1%.
  • Helium, a 7B text LLM pretrained on 2.1T tokens, reaches MMLU 54.3; Moshi reaches MMLU 49.7 after audio training. Safety results include ALERT overall safety score 83.05, 98.7% voice consistency toward the intended speaker, and zero regurgitation of the most frequent training segment after deduplication/fine-tuning.
  • Changing the text-audio delay yields streaming ASR at 5.7% WER and streaming TTS at 4.7% WER on LibriSpeech test-clean.

Methods

  • Architecture: Helium text LLM backbone; Mimi neural audio codec with split RVQ and WavLM distillation; RQ-Transformer with Temporal and Depth Transformers; multi-stream modeling of user and Moshi; Inner Monologue predicts time-aligned text tokens as a prefix to semantic/acoustic audio tokens.
  • Training: Helium pretrained on 2.1T tokens; Moshi trained in four phases—unsupervised audio pretraining on 7M hours, diarization-based simulated multi-stream post-training, Fisher 2000h finetuning, and instruction finetuning on >20k hours of synthetic speech; TTS trained on 170h supervised multi-stream data.
  • Evaluation: text benchmarks, codec ABX/VisQOL/MOSNet/MUSHRA, textless NLP metrics, spoken QA (Web Questions, LlaMA Questions, TriviaQA), dialogue turn-taking statistics, streaming ASR/TTS WER, quantization, and safety/toxicity/regurgitation/watermarking analyses.

Limitations

  • Stated: ASR/TTS experiments are limited and not intended to compete with state-of-the-art systems; TTS evaluation on LibriSpeech test-clean does not showcase multistream or expressive dialogue, and thorough streaming TTS evaluation is left for future work.
  • Stated: post-training quantization causes significant MMLU drops below 6 bits; quantized-aware finetuning may help but requires further investigation. Audio quality remains close to floating point down to 4 bits, but artifacts appear at 3 and 2 bits.
  • Stated/evident: signal-based Audioseal watermarking is not robust to compression, and generative watermarking was a negative result due to codec non-idempotence; the paper states there is no way to fully prevent regurgitation and fine-tuning could be over-ridden. Moshi is weaker than Helium on spoken QA, especially TriviaQA, attributed to oral-style finetuning. Evident: full-duplex dialogue evaluation uses generated dialogues and automatic metrics (DialoGPT perplexity, turn-taking statistics), not human ratings of conversational naturalness.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.00037

Cites

103 of 103 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Overview of Moshi . Moshi is a speech-text foundation model which enables real-time spoken dialogue. The main components of Moshi's architecture are: a bespoke text language model backbone (Helium, see Section 3.2); a neural audio codec with residual vector quantization and with semantic knowledge distilled from a self-supervised speech model (Mimi, Section 3.3); the streaming, hierarchical generation of semantic and acoustic tokens for both the user and Moshi, along with time-aligned text tokens for Moshi when using Inner Monologue (Section 3.4).
Figure 1 Overview of Moshi . Moshi is a speech-text foundation model which enables real-time spoken dialogue. The main components of Moshi's architecture are: a bespoke text language model backbone (Helium, see Section 3.2); a neural audio codec with residual vector quantization and with semantic knowledge distilled from a self-supervised speech model (Mimi, Section 3.3); the streaming, hierarchical generation of semantic and acoustic tokens for both the user and Moshi, along with time-aligned text tokens for Moshi when using Inner Monologue (Section 3.4).CC-BY-NC-SA · From the PDF
Figure 2 Architecture and training of Mimi, our neural audio codec, with its split residual vector quantization . During training (blue part, top), we distill noncausal embeddings from WavLM (Chen et al., 2022) into a single vector quantizer which produces semantic tokens, and is combined with separate acoustic tokens for reconstruction.
Figure 2 Architecture and training of Mimi, our neural audio codec, with its split residual vector quantization . During training (blue part, top), we distill noncausal embeddings from WavLM (Chen et al., 2022) into a single vector quantizer which produces semantic tokens, and is combined with separate acoustic tokens for reconstruction.CC-BY-NC-SA · From the PDF
Figure 3 Architecture of the RQ-Transformer. The RQ-Transformer breaks down a flattened sequence of length K · S into S timesteps for a large Temporal Transformer which produces a context embedding used to condition a smaller Depth Transformer over K steps. This allows scaling to longer sequences by increasing S -or to a higher depth by increasing K -than modeling the flattened sequence with a single model. In this figure, we use K = 4 for the sake of illustration.
Figure 3 Architecture of the RQ-Transformer. The RQ-Transformer breaks down a flattened sequence of length K · S into S timesteps for a large Temporal Transformer which produces a context embedding used to condition a smaller Depth Transformer over K steps. This allows scaling to longer sequences by increasing S -or to a higher depth by increasing K -than modeling the flattened sequence with a single model. In this figure, we use K = 4 for the sake of illustration.CC-BY-NC-SA · From the PDF
Figure 4 Representation of the joint sequence modeled by Moshi . Each column represents the tokens for a given step in the joint sequence ( V s,k ) described in Equation 6 with an acoustic delay τ = 1, e.g. the input of the Temporal Transformer for this step. Tokens are predicted from bottom to top in the Depth Transformer. At inference time, tokens under the dashed line (corresponding to Moshi) are sampled, while those above are fed from the user. This design allows for our model to handle overlapping speech turns.
Figure 4 Representation of the joint sequence modeled by Moshi . Each column represents the tokens for a given step in the joint sequence ( V s,k ) described in Equation 6 with an acoustic delay τ = 1, e.g. the input of the Temporal Transformer for this step. Tokens are predicted from bottom to top in the Depth Transformer. At inference time, tokens under the dashed line (corresponding to Moshi) are sampled, while those above are fed from the user. This design allows for our model to handle overlapping speech turns.CC-BY-NC-SA · From the PDF
Figure 5 Acoustic impact of model compression on Moshi . MOSNet evaluation of samples generated by models compressed for different bitwidths. We evaluate the MOSNet scores across non overlapping windows of 5s, and report the distribution of these scores over 500 samples for each model.
Figure 5 Acoustic impact of model compression on Moshi . MOSNet evaluation of samples generated by models compressed for different bitwidths. We evaluate the MOSNet scores across non overlapping windows of 5s, and report the distribution of these scores over 500 samples for each model.CC-BY-NC-SA · From the PDF
Image 6
CC-BY-NC-SA · From the PDF
Image 7
CC-BY-NC-SA · From the PDF
Figure 8 Representation of the joint sequence modeled by Moshi when used for ASR. Each column represents the tokens for a given step in the joint sequence ( V s,k ), similar to the one described in Equation 6, but adapted for ASR. The text is delayed by 2 seconds, and we use an acoustic token delay τ = 2. Tokens are predicted from bottom to top in the depth Transformer. The audio tokens are kept to match those of the input audio, while text tokens are sampled freely. This also provides fine word timestamps.
Figure 8 Representation of the joint sequence modeled by Moshi when used for ASR. Each column represents the tokens for a given step in the joint sequence ( V s,k ), similar to the one described in Equation 6, but adapted for ASR. The text is delayed by 2 seconds, and we use an acoustic token delay τ = 2. Tokens are predicted from bottom to top in the depth Transformer. The audio tokens are kept to match those of the input audio, while text tokens are sampled freely. This also provides fine word timestamps.CC-BY-NC-SA · From the PDF
Figure 9 Representation of the joint sequence modeled by Moshi when used in TTS mode . Each column represents the tokens for a given step in the joint sequence ( V s,k ), similar to the one described in Equation 6, but adapted for TTS. The audio is delayed by 2 seconds, and we use an acoustic token delay τ = 2. Tokens are predicted from bottom to top in the depth Transformer. Text predictions are usually ignored, and the tokens from the text to generate are used instead. However, this text input lacks padding token. At the end of each word, we allow the model to sample freely PAD and EPAD tokens. If the model tries to sample another token, we instead use the tokens from the next word. The semantic and acoustic audio tokens are sampled normally, being implicitly conditioned on the text due to the delay used. This method also provides a fine alignment of the words in the generated audio, by noting the time at which a given word is consumed by the model.
Figure 9 Representation of the joint sequence modeled by Moshi when used in TTS mode . Each column represents the tokens for a given step in the joint sequence ( V s,k ), similar to the one described in Equation 6, but adapted for TTS. The audio is delayed by 2 seconds, and we use an acoustic token delay τ = 2. Tokens are predicted from bottom to top in the depth Transformer. Text predictions are usually ignored, and the tokens from the text to generate are used instead. However, this text input lacks padding token. At the end of each word, we allow the model to sample freely PAD and EPAD tokens. If the model tries to sample another token, we instead use the tokens from the next word. The semantic and acoustic audio tokens are sampled normally, being implicitly conditioned on the text due to the delay used. This method also provides a fine alignment of the words in the generated audio, by noting the time at which a given word is consumed by the model.CC-BY-NC-SA · From the PDF
Image 10
CC-BY-NC-SA · From the PDF
Image 11
CC-BY-NC-SA · From the PDF
Image 12
CC-BY-NC-SA · From the PDF
Figure 10 Example of typical entropy spectrums capturing specific audio artifacts caused by model quantization . For each timestep, we compute the entropy over the past 128 tokens, independently for the text and audio codebooks tokens. Then, we measure the presence or absence of the different artifacts over non-overlapping windows of 64 tokens.
Figure 10 Example of typical entropy spectrums capturing specific audio artifacts caused by model quantization . For each timestep, we compute the entropy over the past 128 tokens, independently for the text and audio codebooks tokens. Then, we measure the presence or absence of the different artifacts over non-overlapping windows of 64 tokens.CC-BY-NC-SA · From the PDF
Image 14
CC-BY-NC-SA · From the PDF

Verification

Claims the paper makes

0 Valid · 11 Not valid · 51 Not checked · 231 Own results and interpretations

173 more claims

Create a free account to see all 293 claims and to verify the 37 references still open.

See all claims — free account
  • It may state two separate claims as one. 65
  • The quote only partly states this claim. 7
  • Probably not a substantial claim. 66

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 5

Citations

Cites

Show all 103 references

Research works printed but not linked: 21

  1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  2. Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In Yoshua Bengio and Yann Le- Cun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
  3. Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 7750-7774. PMLR, 23-29 Jul 2023.
  4. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate post- training quantization for generative pre-trained transformers. In ICLR, 2023.
  5. Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016b. · arXiv
  6. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in neural information processing systems, 2020.
  7. Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Trans. Assoc. Comput. Linguistics, 11:1703-1718, 2023. doi: 10.1162/TACL\ A\ 00618. · doi:10.1162/tacl\a\00618
  8. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Rep- resentations, 2015. 11
  9. Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved rvqgan. In Advances in Neural Informa- tion Processing Systems, 2024. 23, 24
  10. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th Inter- national Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. 11, 20
  11. Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. Spoken question answering and speech continuation using spectrogram-powered LLM. In The Twelfth International Conference on Learning Representations, 2024. 5, 29
  12. Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdelrahman Mohamed, and Emmanuel Dupoux. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11:250-266, 2023. doi: 10.1162/tacl a 00545. 6, 30, 31 · doi:10.1162/tacla00545.6,30
  13. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 28492-28518. PMLR, 2023. 4, 16, 18, 25, 26
  14. Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29, 2016. 10
  15. Robin San Roman, Pierre Fernandez, Antoine Deleforge, Yossi Adi, and Romain Serizel. La- tent watermarking of audio generative models. arXiv e-prints, pages arXiv-2409, 2024a.
  16. Marco Tagliasacchi, Yunpeng Li, Karolis Misiunas, and Dominik Roblek. SEANet: A multi-modal speech enhancement network. In Interspeech, 2020. 10, 11
  17. Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, and Lukasz Kaiser and. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NeurIPS), pages 5998-6008, 2017. 3, 5, 7
  19. Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers. Ad- vances in Neural Information Processing Systems, 2024. 4, 13
  20. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE ACM Trans. Audio Speech Lang. Process., 30, 2022. 3, 4, 9, 10, 11
  21. Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech language models. In The Twelfth International Conference on Learning Representations, 2024b. 3, 9, 12, 23, 24

Cited by

See all 1,027

Metadata

Authors
Défossez, A., Mazaré, L., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., & Zeghidour, N.
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
OpenAlex
W4403883071
Cited by
1,027
References
81
Citations by year
2026: 696 · 2025: 307 · 2024: 24
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University) (cc-by-nc-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Keywords
spoken dialogue systems, neural audio codecs, streaming speech recognition, text-to-speech, real-time latency, overlapping speech, conversational dynamics, inner speech, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-22, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-22 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Moshi: a speech-text foundation model for real-time dialogue. https://app.cochat.ai/papers/10.48550/arxiv.2410.00037
The paper
Défossez, A., Mazaré, L., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., & Zeghidour, N. (2024). Moshi: a speech-text foundation model for real-time dialogue. arXiv. https://doi.org/10.48550/arxiv.2410.00037

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.