An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue
Open accessThe DOI matches its DataCite record; 16 of the 16 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.
Abstract
Full-duplex spoken dialogue, in which a system listens while it speaks and can backchannel, yield the floor, and be interrupted, is usually obtained by training on hundreds of thousands to millions of hours of speech. This is costly, rewrites the language model that supplies the system's knowledge, and still leaves turn-taking either implicit or decided coarsely rather than word by word. We show that a frozen, general-purpose multimodal LLM can instead be adapted into a word-level full-duplex system with little data. Our system is a thinker-talker cascade on an 80 ms frame clock. The thinker keeps the LLM frozen and trains only lightweight adapters and a few new token embeddings (97M parameters); at every frame it emits one token of agent text, with pauses, yields, and onsets represented as tokens of the same vocabulary and kept legal by a finite-state machine. A four-stage curriculum, each stage anchored to the model's own behavior under the previous format, performs the conversion in about 30 GPU-hours. The talker reads only the thinker's discrete tokens, so the two halves are trained independently. On Full-Duplex-Bench and InteractionBench, the system is competitive with open-source full-duplex systems built through large-scale speech training: among them, it is the least likely to take the floor when the user merely pauses, backchannels most often, and leads in spoken-task accuracy, groundedness, and coherence over long conversations.
Overview
CoChat Summary
TL;DR
The paper adapts a frozen Qwen3-Omni-30B-A3B multimodal LLM into a word-level full-duplex spoken dialogue system by training only 97.4M parameters in a four-stage curriculum (~30 GPU-hours), using an 80 ms frame clock and a finite-state turn-taking mask. On Full-Duplex-Bench and InteractionBench it is competitive with open-source full-duplex systems, especially in pause handling, backchannel frequency, and groundedness, while remaining weaker in responsiveness and whisper/volume control.
Key findings
- On Full-Duplex-Bench v1.0, the system is the least likely open-source system to take the floor when the user merely pauses: turn-over rate 0.170 on the synthetic subset and 0.267 on Candor, against 0.584/0.662 for the next-best system (PersonaPlex). It also has backchannel TOR 0.145, backchannel frequency 0.073, JSD 0.753, smooth turn-taking TOR 0.997 with latency 0.441 s, and user-interruption TOR 0.998 with relevance 2.681 and latency 0.598 s.
- On Full-Duplex-Bench v1.5, it has the lowest response latency in every overlap scenario; after user interruption its stop/response latencies are 0.75/0.64 s, response rate 0.42, and resumption 0.10, while after a user backchannel it resumes its earlier utterance 0.24 of the time versus 0.80 for Freeze-Omni. It responds less often after interruption than Freeze-Omni and Moshi.
- On InteractionBench, among open-source systems it is strongest on groundedness (53%, 27/51, versus 35% for Freeze-Omni), coherence (47), and spoken-task accuracy (26%, 13/50); it is second on conversational timing (63%, 88/140) behind PersonaPlex and weakest on whisper/volume (5%, 1/20).
- In analysis, a clean-trained model’s question-answering score falls from 4.31 on clean audio to 1.10 with a room-tone floor; making channel fusion trainable adds +0.65 [+0.31, +1.04] on noisy audio, and training both channel fusion and the audio adapter adds +1.00 [+0.46, +1.62] over the frozen model. The evaluated talker reaches word error rate 0.088 versus 0.128 for an earlier talker in its lineage; without a text lead, WER is almost four times that at a lead of four frames (+0.199 [+0.186, +0.212]).
Methods
- Thinker: frozen Qwen3-Omni-30B-A3B; trainable parts are rank-16 LoRA on attention projections (13.4M), audio adapter (75.5M), channel-fusion layer (8.4M), and 17 new token embeddings, totaling 97.4M. It runs at 12.5 Hz (80 ms per frame), emits one agent-text token per frame, and uses a finite-state machine to mask illegal tokens before sampling. Four-stage curriculum: Stage 1 audio alignment (17,210 windows, ~270 h; distillation on 3,911 questions, 6.3 h), Stage 2 pause timing (760 questions), Stage 3 dual-channel self-distillation (760 examples) plus multi-turn retrain (12,146 dialogues, 406 h), Stage 4 barge-in (36,621 dialogues, 1,058 h) and backchannel (42,229 sequences, 1,150 h); total 30.5 GPU-hours.
- Talker: Moshi architecture at reduced scale, trained in a single supervised stage on 36,996 filtered examples for 4 epochs (35,504 steps), taking 2.65 h on one NVIDIA B200; text leads audio by 3 frames, and training uses synthesized speech with word-level transcripts.
- Evaluation: Full-Duplex-Bench v1.0/v1.5 and InteractionBench, with open-source baselines PersonaPlex, dGSLM, Moshi, and Freeze-Omni, and closed-source reference rows Gemini Live and GPT-Realtime; Full-Duplex-Bench results are means over three decoding seeds, InteractionBench is a single decoding seed, and no external voice-activity detector is used.
Limitations
- Paper-stated: the data-efficiency claim is about adaptation given a starting model that already understands speech; Qwen3-Omni was itself pretrained on large amounts of audio and the talker starts from a pretrained speech-text checkpoint, so the paper does not claim full-duplex dialogue can be learned from small data by a model that has never heard speech. The comparison with Moshi and SyncLLM also crosses units (training examples and GPU-hours versus hours of audio), with no conversion.
- Paper-stated: benchmark per-behavior sets are small (55–353 items on Full-Duplex-Bench, 10–140 on InteractionBench), so few-point differences are within run-to-run variation; the internal question-answering probe is likewise small. The talker’s architecture takes user audio, but in the checkpoint used the user-audio embeddings are initialized at zero and frozen, so it speaks from agent text alone; it is trained entirely on synthesized speech and has no loudness or whisper control. The backchannel step trains on 701 unique backchannel dialogues repeated eight times, and generalization beyond this set is not established.
- Evident/paper-stated: the noise-floor ablation uses single training runs per cell and 13 probe items, so effects below about half a point are within noise; the talker comparison to an earlier lineage differs in three ways at once, not an ablation; reference transcripts join consecutive utterances without a space, inflating absolute WER; and the InteractionBench row is a single decoding seed. Results also show slower response than PersonaPlex and Moshi, lower interruption relevance, less frequent response after interruption, and weaker whisper/volume performance.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
0 Valid · 2 Not valid · 7 Not checked · 62 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Energy before the first word, present in 3.6% of the earlier talker's outputs, does not occur.”
§ An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue > 7.2 Talker intelligibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is the weakest on whisper and volume, which our talker does not control (Section 8).”
§ An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue > 6.3 InteractionBench
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We have described a full-duplex spoken dialogue system built by adapting a frozen, general-purpose multimodal LLM rather than by pretraining a new speech model.”
§ An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue > 9 Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With both modules frozen, the floor costs 1.33 points.”
§ An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue > 7 Analysis 7.1 Adapting under a noise floor
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“our system is the least likely of the open-source systems to take the floor when the user merely pauses, on both subsets and by a wide margin (turn-over rate 0.170 on the synthetic subset against 0.584 for the next-best system).”
§ An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue > 6.1 Full-Duplex-Bench v1.0
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 71 claims and to verify the 3 references still open.
- It may state two separate claims as one. 37
- Probably not a substantial claim. 21
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue > 4.3 Talker training
“The training set has 36,996 examples, kept from 60,926 by four filters”
§ An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue > 4.3 Talker training
“Training runs for four epochs (35,504 steps at batch size 4)”
Citations
Cites
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
Moshi: a speech-text foundation model for real-time dialogue
Défossez, A., Mazaré, L., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., & Zeghidour, N. (2024) · arXiv · doi:10.48550/arxiv.2410.00037 · Cited by 1,027
Autoregressive Image Generation using Residual Quantization
Lee, D., Kim, C., Kim, S., Cho, M., & Han, W.-S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01123 · Cited by 445
High Fidelity Neural Audio Compression
Défossez, A., Copet, J., Synnaeve, G., & Adi, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.13438 · Cited by 282
Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMPreprint
Wang, X., Li, Y., Fu, C., Shen, Y., Xie, L., Li, K., Sun, X., & Ma, L. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.00774 · Cited by 176
Lin, G.-T., Lian, J., Li, T., Wang, Q., Anumanchipalli, G. K., Liu, A. H., & Lee, H.-Y. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11433838 · Cited by 171
Voice Activity Projection: Self-supervised Learning of Turn-taking Events
Ekstedt, E., & Skantze, G. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10955 · Cited by 147
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation
Zhang, Q., Cheng, L., Deng, C., Chen, Q., Wang, W., Zheng, S., Liu, J., Yu, H., Tan, C.-H., Du, Z., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.709 · Cited by 112
PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models
Roy, R., Raiman, J., Lee, S.-G., Ene, T.-D., Kirby, R., Kim, S., Kim, J., & Catanzaro, B. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463413 · Cited by 104
Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models
Lin, G.-T., Kuan, S.-Y. S., Wang, Q., Lian, J., Li, T., Watanabe, S., & Lee, H.-Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463576 · Cited by 89
Research works printed but not linked: 5
- Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and Controllable Music Generation, 2023. URL https://arxiv.org/ abs/2306.05284. NeurIPS 2023. · arXiv
- Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, and Xie Chen. Language Model Can Listen While Speaking, 2024. URL https://arxiv.org/ abs/2408.02622. · arXiv
- Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, and Emmanuel Dupoux. Generative Spoken Dialogue Language Modeling, 2022. URL https://arxiv.org/abs/ 2203.16502.
- Peng Wang, Songshuo Lu, Yaohua Tang, Sijie Yan, Wei Xia, and Yuanjun Xiong. A Full-duplex Speech Dialogue Scheme Based On Large Language Models, 2024a. URL https://arxiv. org/abs/2405.19487. NeurIPS 2024.
- Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- Stream: An End-to-End Neural Audio Codec, 2021. URL https://arxiv.org/abs/2107. 03312.
Cited by
Still being checked.
Metadata
- Authors
- Chen, T., He, R. Y., Liu, Y., Cao, B., & Xu, C.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2611.14774
- OpenAlex
- W7226014191
- Cited by
- 0
- References
- 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- backchanneling, data-efficient fine-tuning, lightweight adapters, token embeddings, curriculum learning, conversational grounding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-22, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-22 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue. https://app.cochat.ai/papers/10.48550/arxiv.2611.14774
- The paper
- Chen, T., He, R. Y., Liu, Y., Cao, B., & Xu, C. (2026). An LLM Is All You Need: Data-Efficient Adaptation to Full-Duplex, Word-Level Spoken Dialogue. arXiv. https://doi.org/10.48550/arxiv.2611.14774
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.