CoChat

SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents

Open access

The DOI matches its DataCite record; 34 of the 34 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.

Abstract

Full-duplex speech large language models (LLMs) enable low-latency, natural voice interaction. However, real-world agents must also use tools and perform deliberative reasoning-operations whose variable latency and computational cost conflict with the stringent timing requirements of real-time conversation. To reconcile these demands, we propose SALMONN-duo, an adaptive dual-system voice agent inspired by dual-process theories of cognition. SALMONN-duo separates real-time interaction from deliberative computation by pairing an always-on, fast-thinking full-duplex speech LLM (system 1) with a powerful asynchronous slow-thinking LLM agent (system 2). Beyond handling real-time interaction, system 1 learns when to answer directly and when to delegate, remaining responsive during backend execution and seamlessly integrating returned information into the ongoing dialogue without exposing tool traces or losing conversational context. Evaluations on single-turn spoken question answering (QA) and multi-turn conversations demonstrate that adaptive delegation substantially improves accuracy on knowledge-intensive and multi-hop reasoning questions, while knowledge-boundary-aware training avoids unnecessary system 2 invocations. On a customized version of $τ$-Voice, SALMONN-duo further demonstrates its ability to complete environment-grounded, policy-constrained tasks through multi-turn interactions in realistic business scenarios. Finally, cost-aware reinforcement learning further enhances the trade-off between task performance and backend usage across the QA and conversation tasks, while improving task success and response safety on $τ$-Voice with an acceptable increase in the delegation rate.

Overview

CoChat Summary

TL;DR

SALMONN-duo is a dual-system full-duplex voice agent that pairs an always-on speech LLM frontend with an asynchronous backend agent, learning when to delegate via knowledge-boundary-aware SFT and cost-aware GRPO. It improves knowledge-intensive and multi-hop QA and τ-Voice task performance while controlling backend invocation cost.

Key findings

  • The delegation-token mechanism outperformed delegation heads: on TriviaQA it reached 68.5% accuracy, 0.745 AUROC, and 98.6% answer-reference agreement; on HotpotQA 23.6% accuracy, 0.691 AUROC, 94.5% agreement; on HaluEval 31.1% accuracy, 0.732 AUROC, 93.6% agreement.
  • On spoken QA, SALMONN-duo (8B) reported accuracy/delegation rate of 79.7%/16.3% on Llama Questions, 71.5%/26.7% on Web Questions, 80.2%/25.0% on TriviaQA, 46.1%/61.3% on HotpotQA, and 43.9%/59.0% on HaluEval. On MultiTurn, it averaged 56.2% accuracy with 61.9% delegation, versus MoshiRAG’s 54.3% accuracy with 95.6% delegation.
  • Cost-aware GRPO improved routing and agreement: TriviaQA AUROC rose from 0.745 (SFT) to 0.757 at \(r_{del,cor}=0.80\), HotpotQA from 0.691 to 0.714, HaluEval from 0.732 to 0.748, and MultiTurn from 0.730 to 0.754.
  • On customized τ-Voice, adding safety reward reduced hallucination rate from 0.83 to 0.62 and raised policy compliance from 0.56 to 0.75; adding cost reward gave 64.0 overall Pass@1, 62.7% delegation, and 85.4% effective delegation. For interruptions, SFT achieved 81.9% accuracy and 77.2% order correctness; cost-aware GRPO reached 87.8% accuracy and 74.7% order correctness.

Methods

  • SALMONN-duo uses system 1 as a full-duplex speech LLM based on SALMONN-omni, with SPEAR as streaming speech encoder, Llama-3.1-8B-Instruct as LLM backbone, and CosyVoice2-0.5B adapted into a streaming speech synthesizer. System 2 is an asynchronous backend agent, instantiated as a single LLM or multi-agent coordinator; QA used gpt-4o-2024-11-20, τ-Voice used gpt-5.2-2025-12-11, and gpt-realtime-whisper was used for streaming ASR.
  • Training combined knowledge-boundary-aware SFT, which labels delegation based on correctness of N rollouts, with cost-aware GRPO using rewards for direct-correct > delegate-correct > incorrect, plus safety and cost penalties. Data included 309k single-turn QA samples, 36k multi-turn daily conversations of 10–15 turns, and 32k τ-Voice episodes; training used 64 A100-80G GPUs, LoRA rank 32, and GRPO group size 4.
  • Evaluation covered spoken QA on Llama Questions, Web Questions, TriviaQA, HotpotQA, and HaluEval; a 923-dialogue MultiTurn set of 3–5 turns; and a customized τ-Voice test split with interruptions/backchannels disabled. GPT-4o judged QA accuracy, GPT-5.2 judged hallucination and policy compliance, and backend invocation latency was simulated.

Limitations

  • Stated: the ASR module is not the focus, and evaluation assumes the backend has access to ground-truth dialogue history unless otherwise specified; ASR transcription accuracy does affect overall performance, and a more systematic investigation is left to future work. The paper also notes it does not currently consider system 1 correcting errors.
  • Stated: the customized τ-Voice evaluation disables user-simulator interruption and backchannel checks, removes background noise and speaker variation, and restricts evaluation to the test split; backend latency is simulated rather than measured in all settings.
  • Evident: results rely on GPT-based judges and simulated latency, and the paper assumes system 2 is more capable in real-world settings; it also notes that the small GRPO group size of \(G=4\) made episode-level rewards sparse.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2609.34247

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Illustration of SALMONN-duo completing a user task through adaptive tool use, realtime handling of complex conversational dynamics (e.g., user barge-in), and seamless delivery of backend results to the user.
Figure 1 Illustration of SALMONN-duo completing a user task through adaptive tool use, realtime handling of complex conversational dynamics (e.g., user barge-in), and seamless delivery of backend results to the user.CC-BY-SA · From the PDF
Figure 2 SALMONN-duo comprises two collaborative systems: System 1 interacts with the user in a full-duplex manner, adaptively invokes an asynchronous system 2, and naturally incorporates the information provided by system 2 into the ongoing conversation to fulfill the user's goals.
Figure 2 SALMONN-duo comprises two collaborative systems: System 1 interacts with the user in a full-duplex manner, adaptively invokes an asynchronous system 2, and naturally incorporates the information provided by system 2 into the ongoing conversation to fulfill the user's goals.CC-BY-SA · From the PDF
Figure 3 The impact of cost-aware GRPO on accuracy-delegation trade-offs of SALMONN-duo on single- and multi-turn knowledge-based question answering tasks.
Figure 3 The impact of cost-aware GRPO on accuracy-delegation trade-offs of SALMONN-duo on single- and multi-turn knowledge-based question answering tasks.CC-BY-SA · From the PDF
Figure 4 Knowledge boundary aware multi-turn conversation generation pipeline.
Figure 4 Knowledge boundary aware multi-turn conversation generation pipeline.CC-BY-SA · From the PDF
Figure 5 SALMONN-duo can use the Codex backend to launch tasks that gather up-to-date information and compile it into documents.
Figure 5 SALMONN-duo can use the Codex backend to launch tasks that gather up-to-date information and compile it into documents.CC-BY-SA · From the PDF
Figure 6 SALMONN-duo can use the Codex backend to review and modify code.
Figure 6 SALMONN-duo can use the Codex backend to review and modify code.CC-BY-SA · From the PDF

Verification

Claims the paper makes

0 Valid · 6 Not valid · 7 Not checked · 57 Own results and interpretations

20 more claims

Create a free account to see all 70 claims and to verify the 8 references still open.

See all claims — free account
  • It may state two separate claims as one. 39
  • Probably not a substantial claim. 15

Create a free account to see these claims.

Citations

Cites

Show all 34 references

Research works printed but not linked: 12

  1. Lihu Chen, Gerard de Melo, Fabian M. Suchanek, and Gaël Varoquaux. Query-level uncertainty in large language models. In Proc. ICLR, Rio de Janeiro, 2026.
  2. Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research, 2024.
  3. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In Proc. ICLR, Virtual, 2022.
  4. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. HaluEval: A large- scale hallucination evaluation benchmark for large language models. In Proc. EMNLP, Singapore, 2023.
  5. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. In Proc. ICLR, Singapore, 2025.
  6. Yi-Jen Shih, Desh Raj, Chunyang Wu, Wei Zhou, SK Bong, Yashesh Gaur, Jay Mahadeokar, Ozlem Kalinli, and Mike Seltzer. Can speech llms think while listening? In Proc. ICLR, Rio de Janeiro, 2026.
  7. Hongjin SU, Shizhe Diao, Ximing Lu, Mingjie Liu, Jiacheng Xu, Xin Dong, Yonggan Fu, Peter Belcak, Hanrong Ye, Hongxu Yin, Yi Dong, Evelina Bakhturina, Tao Yu, Yejin Choi, Jan Kautz, and Pavlo Molchanov. ToolOrchestra: Elevating intelligence via efficient model and tool orches- tration. In Proc. ICML, Seoul, 2026.
  8. Xiong Wang, Yangze Li, Chaoyou Fu, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-Omni: A smart and low latency speech-to-speech dialogue model with frozen LLM. In Proc. ICML, Vancouver, 2025.
  9. Donghang Wu, Tianyu Zhang, Yuxin Li, Hexin Liu, Chen Chen, Eng Siong Chng, and Yoshua Bengio. The silent thought: Modeling internal cognition in full-duplex spoken dialogue models via latent reasoning. In Proc. ICML, Seoul, 2026.
  10. Xiaoyu Yang, Yifan Yang, Zengrui Jin, Ziyun Cui, Wen Wu, Baoxiangli, Chao Zhang, and Phil Woodland. SPEAR: A unified ssl framework for learning speech and audio representations. In Proc. ICML, Seoul, 2026b.
  11. Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, and Chao Zhang. SALMONN-omni: A standalone speech LLM without codec injection for full-duplex conversation. In Proc. NeurIPS, San Diego, 2025.
  12. Hang Zheng, Hongshen Xu, Yongkai.lin, Shuai Fan, Lu Chen, and Kai Yu. DiSRouter: Distributed self-routing for LLM selections. In ICLR, Rio de Janeiro, 2026.

Cited by

See all 1

Metadata

Authors
Yu, W., Wang, S., Chiba, T., Chen, X., Tian, X., Zhang, J., Lu, L., & Zhang, C.
Affiliations
  • Tsinghua University (CN)
  • ByteDance (China) (CN)
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
OpenAlex
W7214705598
Cited by
1
References
30
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
dual-process theory, System 1 and System 2, deliberative reasoning, tool use, spoken question answering, multi-turn dialogue

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-22, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-22 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents. https://app.cochat.ai/papers/10.48550/arxiv.2609.34247
The paper
Yu, W., Wang, S., Chiba, T., Chen, X., Tian, X., Zhang, J., Lu, L., & Zhang, C. (2026). SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents. arXiv. https://doi.org/10.48550/arxiv.2609.34247

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.