CoChat

An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue

Open access

The DOI matches its DataCite record; 16 of the 16 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.

Abstract

Full-duplex spoken dialogue, in which a system listens while it speaks and can backchannel, yield the floor, and be interrupted, is usually obtained by training on hundreds of thousands to millions of hours of speech. This is costly, rewrites the language model that supplies the system's knowledge, and still leaves turn-taking either implicit or decided coarsely rather than word by word. We show that a frozen, general-purpose multimodal LLM can instead be adapted into a word-level full-duplex system with little data. Our system is a thinker-talker cascade on an 80 ms frame clock. The thinker keeps the LLM frozen and trains only lightweight adapters and a few new token embeddings (97M parameters); at every frame it emits one token of agent text, with pauses, yields, and onsets represented as tokens of the same vocabulary and kept legal by a finite-state machine. A four-stage curriculum, each stage anchored to the model's own behavior under the previous format, performs the conversion in about 30 GPU-hours. The talker reads only the thinker's discrete tokens, so the two halves are trained independently. On Full-Duplex-Bench and InteractionBench, the system is competitive with open-source full-duplex systems built through large-scale speech training: among them, it is the least likely to take the floor when the user merely pauses, backchannels most often, and leads in spoken-task accuracy, groundedness, and coherence over long conversations.

Overview

CoChat Summary

TL;DR

The paper adapts a frozen Qwen3-Omni-30B-A3B multimodal LLM into a word-level full-duplex spoken dialogue system by training only 97.4M parameters in a four-stage curriculum (~30 GPU-hours), using an 80 ms frame clock and a finite-state turn-taking mask. On Full-Duplex-Bench and InteractionBench it is competitive with open-source full-duplex systems, especially in pause handling, backchannel frequency, and groundedness, while remaining weaker in responsiveness and whisper/volume control.

Key findings

  • On Full-Duplex-Bench v1.0, the system is the least likely open-source system to take the floor when the user merely pauses: turn-over rate 0.170 on the synthetic subset and 0.267 on Candor, against 0.584/0.662 for the next-best system (PersonaPlex). It also has backchannel TOR 0.145, backchannel frequency 0.073, JSD 0.753, smooth turn-taking TOR 0.997 with latency 0.441 s, and user-interruption TOR 0.998 with relevance 2.681 and latency 0.598 s.
  • On Full-Duplex-Bench v1.5, it has the lowest response latency in every overlap scenario; after user interruption its stop/response latencies are 0.75/0.64 s, response rate 0.42, and resumption 0.10, while after a user backchannel it resumes its earlier utterance 0.24 of the time versus 0.80 for Freeze-Omni. It responds less often after interruption than Freeze-Omni and Moshi.
  • On InteractionBench, among open-source systems it is strongest on groundedness (53%, 27/51, versus 35% for Freeze-Omni), coherence (47), and spoken-task accuracy (26%, 13/50); it is second on conversational timing (63%, 88/140) behind PersonaPlex and weakest on whisper/volume (5%, 1/20).
  • In analysis, a clean-trained model’s question-answering score falls from 4.31 on clean audio to 1.10 with a room-tone floor; making channel fusion trainable adds +0.65 [+0.31, +1.04] on noisy audio, and training both channel fusion and the audio adapter adds +1.00 [+0.46, +1.62] over the frozen model. The evaluated talker reaches word error rate 0.088 versus 0.128 for an earlier talker in its lineage; without a text lead, WER is almost four times that at a lead of four frames (+0.199 [+0.186, +0.212]).

Methods

  • Thinker: frozen Qwen3-Omni-30B-A3B; trainable parts are rank-16 LoRA on attention projections (13.4M), audio adapter (75.5M), channel-fusion layer (8.4M), and 17 new token embeddings, totaling 97.4M. It runs at 12.5 Hz (80 ms per frame), emits one agent-text token per frame, and uses a finite-state machine to mask illegal tokens before sampling. Four-stage curriculum: Stage 1 audio alignment (17,210 windows, ~270 h; distillation on 3,911 questions, 6.3 h), Stage 2 pause timing (760 questions), Stage 3 dual-channel self-distillation (760 examples) plus multi-turn retrain (12,146 dialogues, 406 h), Stage 4 barge-in (36,621 dialogues, 1,058 h) and backchannel (42,229 sequences, 1,150 h); total 30.5 GPU-hours.
  • Talker: Moshi architecture at reduced scale, trained in a single supervised stage on 36,996 filtered examples for 4 epochs (35,504 steps), taking 2.65 h on one NVIDIA B200; text leads audio by 3 frames, and training uses synthesized speech with word-level transcripts.
  • Evaluation: Full-Duplex-Bench v1.0/v1.5 and InteractionBench, with open-source baselines PersonaPlex, dGSLM, Moshi, and Freeze-Omni, and closed-source reference rows Gemini Live and GPT-Realtime; Full-Duplex-Bench results are means over three decoding seeds, InteractionBench is a single decoding seed, and no external voice-activity detector is used.

Limitations

  • Paper-stated: the data-efficiency claim is about adaptation given a starting model that already understands speech; Qwen3-Omni was itself pretrained on large amounts of audio and the talker starts from a pretrained speech-text checkpoint, so the paper does not claim full-duplex dialogue can be learned from small data by a model that has never heard speech. The comparison with Moshi and SyncLLM also crosses units (training examples and GPU-hours versus hours of audio), with no conversion.
  • Paper-stated: benchmark per-behavior sets are small (55–353 items on Full-Duplex-Bench, 10–140 on InteractionBench), so few-point differences are within run-to-run variation; the internal question-answering probe is likewise small. The talker’s architecture takes user audio, but in the checkpoint used the user-audio embeddings are initialized at zero and frozen, so it speaks from agent text alone; it is trained entirely on synthesized speech and has no loudness or whisper control. The backchannel step trains on 701 unique backchannel dialogues repeated eight times, and generalization beyond this set is not established.
  • Evident/paper-stated: the noise-floor ablation uses single training runs per cell and 13 probe items, so effects below about half a point are within noise; the talker comparison to an earlier lineage differs in three ways at once, not an ablation; reference transcripts join consecutive utterances without a space, inflating absolute WER; and the InteractionBench row is a single decoding seed. Results also show slower response than PersonaPlex and Moshi, lower interruption relevance, less frequent response after interruption, and weaker whisper/volume performance.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2611.14774

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Illustration of the two behaviors on a shared timeline (user above, agent below; schematic: timings and text are illustrative). (a) A free-running conversation: the agent backchannels ('mmhm', 'yeah', in the backchannel color) while the user is still talking and takes the turn when the user finishes, with no external turn detector. (b) A user interruption: the agent yields the floor when the user cuts in (silence-start token, SS ) and takes it back to answer the new request (silence-end token, SE ). SE / SS markers are drawn at the start and end of each agent turn.
Figure 1 Illustration of the two behaviors on a shared timeline (user above, agent below; schematic: timings and text are illustrative). (a) A free-running conversation: the agent backchannels ('mmhm', 'yeah', in the backchannel color) while the user is still talking and takes the turn when the user finishes, with no external turn detector. (b) A user interruption: the agent yields the floor when the user cuts in (silence-start token, SS ) and takes it back to answer the new request (silence-end token, SE ). SE / SS markers are drawn at the start and end of each agent turn.CC-BY · From the PDF
Figure 2 Overview. A frozen, general-purpose multimodal LLM (Qwen3-Omni), which only talks in turns, becomes the thinker of a full-duplex system through four training stages. Each stage card shows what the model reads and emits at that stage as one cell per frame (user codec tokens, agent words, pause, SE / SS , and PAD cells), which modules it trains, and which model anchors its KL term; the frozen core is never changed. The resulting thinker emits agent text and timing tokens frame by frame, and the talker turns them into speech, so that the agent listens and speaks at the same time.
Figure 2 Overview. A frozen, general-purpose multimodal LLM (Qwen3-Omni), which only talks in turns, becomes the thinker of a full-duplex system through four training stages. Each stage card shows what the model reads and emits at that stage as one cell per frame (user codec tokens, agent words, pause, SE / SS , and PAD cells), which modules it trains, and which model anchors its KL term; the frozen core is never changed. The resulting thinker emits agent text and timing tokens frame by frame, and the talker turns them into speech, so that the agent listens and speaks at the same time.CC-BY · From the PDF
Figure 3 Data flow and frame-level token structure. A frozen codec encoder turns the user's waveform into per-frame audio tokens. The thinker reads the user-audio stream together with its own agent-text stream (pad, text, and control tokens on one lane per frame) and extends the text stream by one token per frame. The talker reads the agent-text stream and its own previous audio tokens and extends the agent-audio stream by one frame; a frozen codec decoder renders it as a waveform. The control tokens SS and SE open and close the agent's silences, and pause tokens mark short gaps; the example includes a short backchannel ('Mhm') produced while the user is still speaking.
Figure 3 Data flow and frame-level token structure. A frozen codec encoder turns the user's waveform into per-frame audio tokens. The thinker reads the user-audio stream together with its own agent-text stream (pad, text, and control tokens on one lane per frame) and extends the text stream by one token per frame. The talker reads the agent-text stream and its own previous audio tokens and extends the agent-audio stream by one frame; a frozen codec decoder renders it as a waveform. The control tokens SS and SE open and close the agent's silences, and pause tokens mark short gaps; the example includes a short backchannel ('Mhm') produced while the user is still speaking.CC-BY · From the PDF
Figure 4 The thinker's turn-taking state machine. In SILENCE the legal tokens are PAD (stay silent) and SE (take the floor, entering CONTENT after one automatically fed EPAD frame). In CONTENT they are word pieces and pause 0 (stay), pause m with m> 0 (enter PAUSE), and SS (yield, entering SILENCE). PAUSE samples nothing: it feeds its queued PAD and EPAD frames and returns to CONTENT when the queue is empty.
Figure 4 The thinker's turn-taking state machine. In SILENCE the legal tokens are PAD (stay silent) and SE (take the floor, entering CONTENT after one automatically fed EPAD frame). In CONTENT they are word pieces and pause 0 (stay), pause m with m> 0 (enter PAUSE), and SS (yield, entering SILENCE). PAUSE samples nothing: it feeds its queued PAD and EPAD frames and returns to CONTENT when the queue is empty.CC-BY · From the PDF
Figure 5 Adapting under a room-tone floor. (a) Question answering after the self-distillation run of Stage 3 (13-item probe, judge score 1 to 5, mean and standard deviation over three seeds) when the whole pipeline is trained with a floor, as a function of which input modules are trainable; dotted lines are the clean-trained model on clean audio and with the floor; bar labels are the means. (b) Mean L2 norm of the user-audio branch entering the channel-fusion layer on 40 held-out noisy examples, for frames with user speech and for answer-phase frames that contain only room tone.
Figure 5 Adapting under a room-tone floor. (a) Question answering after the self-distillation run of Stage 3 (13-item probe, judge score 1 to 5, mean and standard deviation over three seeds) when the whole pipeline is trained with a floor, as a function of which input modules are trainable; dotted lines are the clean-trained model on clean audio and with the floor; bar labels are the means. (b) Mean L2 norm of the user-audio branch entering the channel-fusion layer on 40 held-out noisy examples, for frames with user speech and for answer-phase frames that contain only room tone.CC-BY · From the PDF
Image 6
CC-BY · From the PDF
Figure 6 Text lead and talker intelligibility: word error rate on the 150 held-out dialogues, split into deletions, substitutions, and insertions (mean of three decoding seeds; the standard deviation of the total across seeds is at most 0.010). The talkers at L = 0 , 2, 4, and 6 differ only in the text lead and are trained from PersonaPlex on the same data for the same number of steps; the talker at L = 3 is trained with the recipe of Section 4.3, a separate lineage.
Figure 6 Text lead and talker intelligibility: word error rate on the 150 held-out dialogues, split into deletions, substitutions, and insertions (mean of three decoding seeds; the standard deviation of the total across seeds is at most 0.010). The talkers at L = 0 , 2, 4, and 6 differ only in the text lead and are trained from PersonaPlex on the same data for the same number of steps; the talker at L = 3 is trained with the recipe of Section 4.3, a separate lineage.CC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 2 Not valid · 7 Not checked · 62 Own results and interpretations

20 more claims

Create a free account to see all 71 claims and to verify the 3 references still open.

See all claims — free account
  • It may state two separate claims as one. 37
  • Probably not a substantial claim. 21
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 16 references

Research works printed but not linked: 5

  1. Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and Controllable Music Generation, 2023. URL https://arxiv.org/ abs/2306.05284. NeurIPS 2023. · arXiv
  2. Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, and Xie Chen. Language Model Can Listen While Speaking, 2024. URL https://arxiv.org/ abs/2408.02622. · arXiv
  3. Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, and Emmanuel Dupoux. Generative Spoken Dialogue Language Modeling, 2022. URL https://arxiv.org/abs/ 2203.16502.
  4. Peng Wang, Songshuo Lu, Yaohua Tang, Sijie Yan, Wei Xia, and Yuanjun Xiong. A Full-duplex Speech Dialogue Scheme Based On Large Language Models, 2024a. URL https://arxiv. org/abs/2405.19487. NeurIPS 2024.
  5. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- Stream: An End-to-End Neural Audio Codec, 2021. URL https://arxiv.org/abs/2107. 03312.

Cited by

Still being checked.

Metadata

Authors
Chen, T., He, R. Y., Liu, Y., Cao, B., & Xu, C.
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
OpenAlex
W7226014191
Cited by
0
References
16
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Keywords
backchanneling, data-efficient fine-tuning, lightweight adapters, token embeddings, curriculum learning, conversational grounding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-22, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-22 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue. https://app.cochat.ai/papers/10.48550/arxiv.2611.14774
The paper
Chen, T., He, R. Y., Liu, Y., Cao, B., & Xu, C. (2026). An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue. arXiv. https://doi.org/10.48550/arxiv.2611.14774

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.