SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents
Open accessThe DOI matches its DataCite record; 34 of the 34 references it cites were found in a registry or an index; none of the 16 sources its claims cite is checked for support yet.
Abstract
Full-duplex speech large language models (LLMs) enable low-latency, natural voice interaction. However, real-world agents must also use tools and perform deliberative reasoning-operations whose variable latency and computational cost conflict with the stringent timing requirements of real-time conversation. To reconcile these demands, we propose SALMONN-duo, an adaptive dual-system voice agent inspired by dual-process theories of cognition. SALMONN-duo separates real-time interaction from deliberative computation by pairing an always-on, fast-thinking full-duplex speech LLM (system 1) with a powerful asynchronous slow-thinking LLM agent (system 2). Beyond handling real-time interaction, system 1 learns when to answer directly and when to delegate, remaining responsive during backend execution and seamlessly integrating returned information into the ongoing dialogue without exposing tool traces or losing conversational context. Evaluations on single-turn spoken question answering (QA) and multi-turn conversations demonstrate that adaptive delegation substantially improves accuracy on knowledge-intensive and multi-hop reasoning questions, while knowledge-boundary-aware training avoids unnecessary system 2 invocations. On a customized version of $τ$-Voice, SALMONN-duo further demonstrates its ability to complete environment-grounded, policy-constrained tasks through multi-turn interactions in realistic business scenarios. Finally, cost-aware reinforcement learning further enhances the trade-off between task performance and backend usage across the QA and conversation tasks, while improving task success and response safety on $τ$-Voice with an acceptable increase in the delegation rate.
Overview
CoChat Summary
TL;DR
SALMONN-duo is a dual-system full-duplex voice agent that pairs an always-on speech LLM frontend with an asynchronous backend agent, learning when to delegate via knowledge-boundary-aware SFT and cost-aware GRPO. It improves knowledge-intensive and multi-hop QA and τ-Voice task performance while controlling backend invocation cost.
Key findings
- The delegation-token mechanism outperformed delegation heads: on TriviaQA it reached 68.5% accuracy, 0.745 AUROC, and 98.6% answer-reference agreement; on HotpotQA 23.6% accuracy, 0.691 AUROC, 94.5% agreement; on HaluEval 31.1% accuracy, 0.732 AUROC, 93.6% agreement.
- On spoken QA, SALMONN-duo (8B) reported accuracy/delegation rate of 79.7%/16.3% on Llama Questions, 71.5%/26.7% on Web Questions, 80.2%/25.0% on TriviaQA, 46.1%/61.3% on HotpotQA, and 43.9%/59.0% on HaluEval. On MultiTurn, it averaged 56.2% accuracy with 61.9% delegation, versus MoshiRAG’s 54.3% accuracy with 95.6% delegation.
- Cost-aware GRPO improved routing and agreement: TriviaQA AUROC rose from 0.745 (SFT) to 0.757 at \(r_{del,cor}=0.80\), HotpotQA from 0.691 to 0.714, HaluEval from 0.732 to 0.748, and MultiTurn from 0.730 to 0.754.
- On customized τ-Voice, adding safety reward reduced hallucination rate from 0.83 to 0.62 and raised policy compliance from 0.56 to 0.75; adding cost reward gave 64.0 overall Pass@1, 62.7% delegation, and 85.4% effective delegation. For interruptions, SFT achieved 81.9% accuracy and 77.2% order correctness; cost-aware GRPO reached 87.8% accuracy and 74.7% order correctness.
Methods
- SALMONN-duo uses system 1 as a full-duplex speech LLM based on SALMONN-omni, with SPEAR as streaming speech encoder, Llama-3.1-8B-Instruct as LLM backbone, and CosyVoice2-0.5B adapted into a streaming speech synthesizer. System 2 is an asynchronous backend agent, instantiated as a single LLM or multi-agent coordinator; QA used gpt-4o-2024-11-20, τ-Voice used gpt-5.2-2025-12-11, and gpt-realtime-whisper was used for streaming ASR.
- Training combined knowledge-boundary-aware SFT, which labels delegation based on correctness of N rollouts, with cost-aware GRPO using rewards for direct-correct > delegate-correct > incorrect, plus safety and cost penalties. Data included 309k single-turn QA samples, 36k multi-turn daily conversations of 10–15 turns, and 32k τ-Voice episodes; training used 64 A100-80G GPUs, LoRA rank 32, and GRPO group size 4.
- Evaluation covered spoken QA on Llama Questions, Web Questions, TriviaQA, HotpotQA, and HaluEval; a 923-dialogue MultiTurn set of 3–5 turns; and a customized τ-Voice test split with interruptions/backchannels disabled. GPT-4o judged QA accuracy, GPT-5.2 judged hallucination and policy compliance, and backend invocation latency was simulated.
Limitations
- Stated: the ASR module is not the focus, and evaluation assumes the backend has access to ground-truth dialogue history unless otherwise specified; ASR transcription accuracy does affect overall performance, and a more systematic investigation is left to future work. The paper also notes it does not currently consider system 1 correcting errors.
- Stated: the customized τ-Voice evaluation disables user-simulator interruption and backchannel checks, removes background noise and speaker variation, and restricts evaluation to the test split; backend latency is simulated rather than measured in all settings.
- Evident: results rely on GPT-based judges and simulated latency, and the paper assumes system 2 is more capable in real-world settings; it also notes that the small GRPO group size of \(G=4\) made episode-level rewards sparse.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
0 Valid · 6 Not valid · 7 Not checked · 57 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“delegation token | 68.5 0.745 | 98.6 23.6 0.691 | 94.5 31.1 0.732 | 93.6”
§ SALMONN-DUO: ADAPTIVE DUAL-SYSTEM COOR-DINATION FOR FULL-DUPLEX VOICE AGENTS > Evidence records -`M...`records are chronological public messages.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It also frequently violates domain-specific policies.”
§ SALMONN-DUO: ADAPTIVE DUAL-SYSTEM COOR-DINATION FOR FULL-DUPLEX VOICE AGENTS > 5.3 ENVIRONMENT-GROUNDED TASK-ORIENTED DIALOGUE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 5, after SFT, the model demonstrates the capability to use tools to fulfill user requests through multi-turn interactions.”
§ SALMONN-DUO: ADAPTIVE DUAL-SYSTEM COOR-DINATION FOR FULL-DUPLEX VOICE AGENTS > 5.3 ENVIRONMENT-GROUNDED TASK-ORIENTED DIALOGUE
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“By accounting for the model's knowledge boundaries during SFT and incorporating cost-aware GRPO, SALMONN-duo achieves a better trade-off between cost and performance.”
§ SALMONN-DUO: ADAPTIVE DUAL-SYSTEM COOR-DINATION FOR FULL-DUPLEX VOICE AGENTS > 2.2 TOOL USE AND DELEGATION IN VOICE AGENTS
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Since VoiceChat rarely requests reference answers, variations in retrieval latency have little impact on performance.”
§ SALMONN-DUO: ADAPTIVE DUAL-SYSTEM COOR-DINATION FOR FULL-DUPLEX VOICE AGENTS > D.2 EXTENDED RESULTS OF COST-AWARE GRPO ON KNOWLEDGE-BASED QA
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
20 more claims
Create a free account to see all 70 claims and to verify the 8 references still open.
- It may state two separate claims as one. 39
- Probably not a substantial claim. 15
Citations
Cites
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 17,283
Self-citationA perspective on judgment and choice: mapping bounded rationality.
Kahneman, D. (2003) · American Psychologist · doi:10.1037/0003-066x.58.9.697 · Cited by 5,540
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690
Moshi: a speech-text foundation model for real-time dialogue
Défossez, A., Mazaré, L., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., & Zeghidour, N. (2024) · arXiv · doi:10.48550/arxiv.2410.00037 · Cited by 1,027
Kimi-Audio Technical ReportPreprint
KimiTeam, Ding, D., Ju, Z., Leng, Y., Liu, S., Liu, T., Shang, Z., Shen, K., Song, W., Xu, T., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.18425 · Cited by 519
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Self-citationStep-Audio 2 Technical ReportPreprint
Wu, B., Yan, C., Chen, H., Yi, C., Feng, C., Tian, F., Shen, F., Yu, G., Zhang, H., Li, J., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.16632 · Cited by 237
Self-citationLibriheavy: A 50,000 Hours ASR Corpus with Punctuation Casing and Context
Kang, W., Yang, X., Yao, Z., Kuang, F., Yang, Y., Guo, L., Lin, L., & Povey, D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447120 · Cited by 152
Research works printed but not linked: 12
- Lihu Chen, Gerard de Melo, Fabian M. Suchanek, and Gaël Varoquaux. Query-level uncertainty in large language models. In Proc. ICLR, Rio de Janeiro, 2026.
- Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research, 2024.
- Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In Proc. ICLR, Virtual, 2022.
- Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. HaluEval: A large- scale hallucination evaluation benchmark for large language models. In Proc. EMNLP, Singapore, 2023.
- Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. In Proc. ICLR, Singapore, 2025.
- Yi-Jen Shih, Desh Raj, Chunyang Wu, Wei Zhou, SK Bong, Yashesh Gaur, Jay Mahadeokar, Ozlem Kalinli, and Mike Seltzer. Can speech llms think while listening? In Proc. ICLR, Rio de Janeiro, 2026.
- Hongjin SU, Shizhe Diao, Ximing Lu, Mingjie Liu, Jiacheng Xu, Xin Dong, Yonggan Fu, Peter Belcak, Hanrong Ye, Hongxu Yin, Yi Dong, Evelina Bakhturina, Tao Yu, Yejin Choi, Jan Kautz, and Pavlo Molchanov. ToolOrchestra: Elevating intelligence via efficient model and tool orches- tration. In Proc. ICML, Seoul, 2026.
- Xiong Wang, Yangze Li, Chaoyou Fu, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-Omni: A smart and low latency speech-to-speech dialogue model with frozen LLM. In Proc. ICML, Vancouver, 2025.
- Donghang Wu, Tianyu Zhang, Yuxin Li, Hexin Liu, Chen Chen, Eng Siong Chng, and Yoshua Bengio. The silent thought: Modeling internal cognition in full-duplex spoken dialogue models via latent reasoning. In Proc. ICML, Seoul, 2026.
- Xiaoyu Yang, Yifan Yang, Zengrui Jin, Ziyun Cui, Wen Wu, Baoxiangli, Chao Zhang, and Phil Woodland. SPEAR: A unified ssl framework for learning speech and audio representations. In Proc. ICML, Seoul, 2026b.
- Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, and Chao Zhang. SALMONN-omni: A standalone speech LLM without codec injection for full-duplex conversation. In Proc. NeurIPS, San Diego, 2025.
- Hang Zheng, Hongshen Xu, Yongkai.lin, Shuai Fan, Lu Chen, and Kai Yu. DiSRouter: Distributed self-routing for LLM selections. In ICLR, Rio de Janeiro, 2026.
Cited by
Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?Preprint
Xie, Y., Zhu, M., Liu, T., Zhang, Y., Xiong, H., & Liu, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.19856
Metadata
- Authors
- Yu, W., Wang, S., Chiba, T., Chen, X., Tian, X., Zhang, J., Lu, L., & Zhang, C.
- Affiliations
- Tsinghua University (CN)
- ByteDance (China) (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2609.34247
- OpenAlex
- W7214705598
- Cited by
- 1
- References
- 30
- Citations by year
- 2026: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- dual-process theory, System 1 and System 2, deliberative reasoning, tool use, spoken question answering, multi-turn dialogue
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-22, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-22 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents. https://app.cochat.ai/papers/10.48550/arxiv.2609.34247
- The paper
- Yu, W., Wang, S., Chiba, T., Chen, X., Tian, X., Zhang, J., Lu, L., & Zhang, C. (2026). SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents. arXiv. https://doi.org/10.48550/arxiv.2609.34247
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.