The DOI matches its Crossref record; 42 of the 49 references it cites were found in a registry or an index.
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
Open accessAbstract
Large Language Models (LLMs) still struggle with natural language reasoning tasks.Motivated by the society of minds (Minsky, 1988), we propose RECONCILE, a multi-model multiagent framework designed as a round table conference among diverse LLM agents.RECON-CILE enhances collaborative reasoning between LLM agents via multiple rounds of discussion, learning to convince other agents to improve their answers, and employing a confidenceweighted voting mechanism that leads to a better consensus.In each round, RECONCILE initiates discussion between agents via a 'discussion prompt' that consists of (a) grouped answers and explanations generated by each agent in the previous round, (b) their confidence scores, and (c) demonstrations of answerrectifying human explanations, used for convincing other agents.Experiments on seven benchmarks demonstrate that RECONCILE significantly improves LLMs' reasoning -both individually and as a team -surpassing prior single-agent and multi-agent baselines by up to 11.4% and even outperforming GPT-4 on three datasets.RECONCILE also flexibly incorporates different combinations of agents, including API-based, open-source, and domainspecific models, leading to an 8% improvement on MATH.Finally, we analyze the individual components of RECONCILE, demonstrating that the diversity originating from different models is critical to its superior performance.1 Self-Refine MAD+Judge Multi-Agent Debate (MAD) ReConcile (Group-Discuss-and-Convince) Yes, with 95% confidence No, with 50% confidence No, with 40% confidence yes no no yes no no yes no no Question (Q): Is an ammonia fighting cleaner good for pet owners?Human Explanation (Exp): Ammonia is a component in pet urine.It has an unpleasant odor.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Other studies have demonstrated that structured contrast and deliberation among agents can enhance argumentative coherence and mitigate bias (Chen et al., 2024;Liu et al., 2026).”
From Multi-Agent debate system based on large language models: structured deliberation and validation in satellite communications (2026) · cites this paper as Chen et al., 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Some studies show that discussion, often run as multi-agent debate, outperforms standalone reasoning or voting (Du et al., 2024; Liang et al., 2024; Chen et al., 2024), whereas others report no effect (Smit et al., 2024; Wang et al., 2024b; Kaesberg et al., 2025).”
From Multi-agent discussion gains less when dissent is withheld (2026) · cites this paper as Chen et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“In contrast, other studies report that discussion can outperform individual reasoning or voting (Du et al., 2024; Liang et al., 2024; Chen et al., 2024).”
From Multi-agent discussion gains less when dissent is withheld (2026) · cites this paper as Chen et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“The consensus-building process generally takes 2 to 3 rounds (Chen et al., 2024b;Du et al., 2024;Yang et al., 2025;Xiong et al., 2023) to be effective.”
From Free-MAD: Consensus-Free Multi-Agent Debate (2026) · cites this paper as Chen et al., 2024b
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This framework explains conflicting findings in prior work (Du et al., 2024; Liang et al., 2024; Chen et al., 2024; Smit et al., 2024; Wang et al., 2024b; Kaesberg et al., 2025) as arising from tasks operating in different regimes.”
From Multi-agent discussion gains less when dissent is withheld (2026) · cites this paper as Chen et al., 2024
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 12
- BIG-bench collaboration. 2023. Beyond the imitation game: Quantifying and extrapolating the capabili- ties of language models. Transactions on Machine Learning Research.
- Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547-5569. PMLR.
- Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller language models towards multi-step reasoning. In ICML.
- Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International conference on machine learn- ing, pages 1321-1330. PMLR.
- Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS.
- Shuang Li, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, and Igor Mordatch. 2022a. Composing en- sembles of pre-trained models via iterative consensus. International Conference on Learning Representa- tions (ICLR).
- Yanhong Li, Gang Kou, Guangxu Li, and Yi Peng. 2022b. Consensus reaching process in large-scale group decision making based on bounded confidence and social network. European Journal of Opera- tional Research, 303(2):790-802. · DOI
- John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized like- lihood methods. Advances in large margin classifiers, 10(3):61-74.
- Swarnadeep Saha, Peter Hase, and Mohit Bansal. 2023. Can language models teach weaker agents? teacher explanations improve students via theory of mind. In NeurIPS.
- Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural net- works: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representa- tions.
- Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
- Andy Zeng, Maria Attarian, brian ichter, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, Vin- cent Vanhoucke, and Pete Florence. 2023. Socratic models: Composing zero-shot multimodal reasoning with language. In The Eleventh International Conference on Learning Representations.
Cited by
Sorted by citations
Items 101–134 of 134
Who is the most suitable one? Compliance review method based on multi-agent routing
Yu, C., Jiangqian, H., Chen, X., Gao, M., Zhang, L., Yuan, X., Sun, J., Bao, D., & Yang, Y. (2026) · AI Open · doi:10.1016/j.aiopen.2026.03.001
Kim, J., Suzara, M., & Mitchell, J. C. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790456
CARE-Agent: Multi-Agent Collaboration with Conflict-Aware Routing Mechanism for Diagnosis Prediction
Zhan, P., Cai, B., Zhang, J., Lv, H., Tan, Y., Lin, Z., Chen, X., & Yang, C. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464147
MARE: Multi-Agent Role Embedding for Role-Consistent Generation in Multi-Agent Systems
Gu, W., Han, J., Zhao, J., Chen, Y., Wang, H., & Cheng, B. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464394
Sun, S. Y., Su, D., Chen, W., & Cheng, X. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792159
Multi-Agent Systems have Some More to Learn from Human Multi-Agent SystemsPreprint
Riordan, B., Bosch, P., Salarian, A., Saeks, M., & Pandey, V. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6736398
Beyond Assistance: Orchestrating the Developmental Triad for Agentic AI Mentorship
Park, W., Oh, C., & Kim, D. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2687899
The Consensus Trap: Dissecting Subjectivity and the “Ground Truth” Illusion in Data Annotation
Munir, S., Mah, B., Kalsi, K., Kapania, S., Posada, J., Law, E., Wang, D., & Ahmed, S. I. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806510 · Cited by 1
Table-Aware Group Relative Policy Optimization: Reward Design for Large Vision–Language Models
Lee, J., Park, H., & Oh, H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1947-6_12
(2026) · International Research Journal of Modernization in Engineering Technology and Science · doi:10.56726/irjmets101057
A Systematic Evaluation of Decision Mechanisms in Multi-agent LLMs for Misinformation Detection
Lee, Y., Huang, Y., & Chu, T.-H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30044-7_6
D$$^3$$: Dispute-Driven Debate for Stabilizing Multi-agent Reasoning in Large Language Models
Liu, Y., Sun, S. Y., Wang, Y., Zhang, S., & Li, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2852-2_47
MAP: GPU Resource Planning for Multi-Agent Systems With Stochastic and Verbose Outputs
Zhang, Q., Li, M., Li, Q., Hua, X., Chen, F., Zeng, D., & Li, P. L. (2026) · IEEE Transactions on Cloud Computing · doi:10.1109/tcc.2026.3712315
ReaCR: Reasoning-Conditioned Representation Learning for Fake News Detection
Feng, Y., Xing, Z., Liu, E., Xu, B., & Yang, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3441-7_43
Zhou, J., & Zhou, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33865-5_22
Hao, J., Bai, H., Dong, X., Dong, Z., Ji, W., Xie, D., & Guo, S. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-1884-4_15 · Cited by 1
Large Language Model Agents and Workflow
Xue, X., & Wang, F.-Y. (2026) · Wiley eBooks · doi:10.1002/9781394377480.ch11
LLM ‐Based Agents and Social Simulation
(2026) · Wiley eBooks · doi:10.1002/9781394377480.ch10
Blockchain-enabled explainable multi-agent AI for secure digital identity verification: A review
., K., Turkani, R., Siwach, M. K., Chaubey, P. K., Kumar, R., Malik, A., Mathur, S., & Tripathi, V. (2026) · International Journal of Computing and Artificial Intelligence · doi:10.33545/27076571.2026.v7.i8a.368
Event Level Incremental State Modeling for Long Context Meeting Understanding
Huang, T.-C., Yang, W.-C., Chang, Y. J., Tao, H.-S., & Chang, C. (2026) · IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan) · doi:10.1109/icce-taiwan71481.2026.11652797
Petrovič, P., Tran, C. V., Trung, H. N., & Babič, F. (2026) · IEEE International Conference on Intelligent Engineering Systems (INES) · doi:10.1109/ines69513.2026.11661226
Semantic Raft: A Fault-Tolerant Multi-Agent Inference Framework for Reliable LLM Services
Su, Y., Wang, Y., Dong, Y., & Xu, Z. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661187
Evaluating automated refereeing with heterogeneous LLM panels
Douven, I. (2026) · Scientometrics · doi:10.1007/s11192-026-05783-y · Cited by 1
Uncertainty-Aware Adaptive Debate for Robust and Efficient Large Language Model Reasoning
Liu, R., Guo, W., Liao, T., & Fang, X. (2026) · International Conference on Computer Vision and Data Mining (ICCVDM) · doi:10.1109/iccvdm71812.2026.11709092
Sadjadi, S. M., Bai, Q., Yang, W., & Liu, Q. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7593345
Wang, Y., Han, Y., Gao, J., Zhang, W., Wang, J., & Zhang, W. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728474
Can Dialogue and Role Improve LLM Sidewalk Image Descriptions Across Mobility Groups?
Miyata, A., Go, K., Matsubayashi, R., & Ichikawa, Y. (2026) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3797867.3829022
AgendaFlow: Task-Adaptive Meeting Protocols for Multi-Agent LLM Reasoning
Sun, M., Wang, Y., & Zhang, Y. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.322
Finansal Tahminde Büyük Dil Modelleri ve Çoklu-Ajan Yaklaşımı: Bir Kavram Kanıtı Çalışması
Özel, S., & Aydin, M. (2026) · Bilişim Teknolojileri Dergisi · doi:10.17671/gazibtd.1963739
When Is Debateability a Model Property? Cross-Interlocutor Validation in Multi-Agent Debate
Ju, C.-Y., Kim, J.-Y., & Lee, D. (2026) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3799682.3840079
Sun, L.-S., Zhou, H.-Y., Chen, M.-F., Song, M.-F., Wang, Q., Guo, H.-X., Yang, J.-Y., Zhao, Q.-H., & Yan, R.-A. (2026) · Journal of Big Data and Computing · doi:10.62517/jbdc.202601318
At Equal Inference Cost, Multi-Agent Structure Does Not Beat a Single Frozen Agent
Dylan, D., Brennan, A., Murphy, C., O'Sullivan, N., Kelly, C., & Walsh, S. (2026) · arXiv · doi:10.48550/arxiv.2609.04217 · Cited by 3
Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models
Ferreira, L., Liu, G., & Zheng, K. (2026) · arXiv · doi:10.48550/arxiv.2609.35875
Metadata
- Authors
- Chen, J., Saha, S., & Bansal, M.
- Affiliations
- University of North Carolina at Chapel Hill (US)
- Venue
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 7066-7085
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4402671543
- Cited by
- 64
- References
- 0
- Citations by year
- 2026: 37 · 2025: 20 · 2024: 6
- FWCI
- 18.59
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1846185, 2112635, CAREER)
- Defense Advanced Research Projects Agency (N66001-19-2-4031, N66001-19)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- multi-agent LLM systems, collaborative reasoning, Society of Mind, discussion prompts, model diversity, natural language reasoning, multi-agent debate, MATH benchmark
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
