Enhancing LLM Reasoning Capabilities Through Brokered Multi-Expert Reflection
Open accessAbstract
Large Language Models (LLMs) have found increasing application in tasks requiring multi-step reasoning, yet challenges such as hallucinations and inconsistencies in the generated responses persist. This study presents an innovative methodology to enhance the reasoning capabilities of LLMs by brokering and integrating multiple expert LLMs within a reflection layer to provide targeted feedback on the reasoning trajectories of the base LLM. The approach employs a foundational pre-trained LLM as the base model, which is further supported by agents to promote cognitive assistance for specific task types. In instances where conclusions are deemed incorrect or reasoning is interrupted, these instances are forwarded to the expert LLM layer, which includes systems such as Claude-3 haiku for intricate contexts and MedAlpaca for medical reasoning, to deliver feedback on the base model’s reasoning paths. This feedback forms a ’reflection pool,’ enabling the base LLM to amend and enhance its reasoning trajectories in subsequent iterations. The experiments conducted across diverse datasets, including HotPotQA, SimpleQA, and PubmedQA, underscore the proposed architecture’s efficacy in augmenting success signals, Rouge-L scores (indicative of quality and precision), and CTRLEval Consistency Scores (indicative of coherence and consistency). The architecture effectively addresses the issues of hallucinations and inconsistencies that frequently occur in multi-step reasoning. Importantly, the approach exhibits considerable potential in tackling domain-specific tasks, underscoring the importance of achieving correct and reliable conclusions. To facilitate further investigation and validation of our proposed brokered multi-expert reflection framework, we are open-sourcing the source code of our system, available at https://github.com/WiZY936/Brokered-Multi-Expert-Reflection.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Current LLM-based multi-agent systems frequently rely on static communication protocols and lack dynamic retraining or adaptation procedures to handle evolving tasks and contexts.
“These systems frequently rely on static communication protocols and lack dynamic retraining or adaptation procedures to handle evolving tasks and contexts (Zhang et al., 2025;Sheokand et al., 2025).”
From Multi-Agent debate system based on large language models: structured deliberation and validation in satellite communications (2026) · cites this paper as Sheokand et al., 2025
Cites
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · see citing works
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · see citing works
FEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343 · see citing works
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · see citing works
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · see citing works
PMC-LLaMA: toward building open-source language models for medicine
Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., & Wang, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae045 · Cited by 300 · see citing works
LLMs Will Always Hallucinate, and We Need to Live with This
Banerjee, S., Agarwal, A., & Singla, S. (2025) · Lecture Notes in Networks and Systems · doi:10.1007/978-3-031-99965-9_39 · Cited by 93 · see citing works
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., & Hausknecht, M. (2020) · arXiv · doi:10.48550/arxiv.2010.03768 · Cited by 80 · see citing works
AgentBench: Evaluating LLMs as Agents
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.03688 · Cited by 79 · see citing works
Understanding the planning of LLM agents: A survey
Huang, X., Liu, W., Chen, X., Wang, X., Wang, H., Lian, D., Wang, Y., Tang, R., & Chen, E. (2024) · arXiv · doi:10.48550/arxiv.2402.02716 · Cited by 47 · see citing works
ScienceWorld: Is your Agent Smarter than a 5th Grader?
Wang, R., Jansen, P., Côté, M.-A., & Ammanabrolu, P. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.775 · Cited by 45 · see citing works
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation
Ke, P., Zhou, H., Lin, Y., Li, P., Zhou, J., Zhu, X., & Huang, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.164 · Cited by 27 · see citing works
Measuring short-form factuality in large language models
Wei, J., Karina, N., Chung, H. W., Jiao, Y. J., Papay, S., Glaese, A., Schulman, J., & Fedus, W. (2024) · arXiv · doi:10.48550/arxiv.2411.04368 · Cited by 14 · cited 1× · see citing works
“As illustrated in Figure 14, the approach employing Ministral 8B and Claude 3-haiku demonstrates a higher number of correct conclusions (69) and fewer incorrect conclusions (16), indicating that generalized experts are proficient at effectively capturing reasoning patterns.”
Flagged first, then by citations
Cited by
Sorted by citations
Gómez Álvarez, S., Mozo Quesada, A., Navarro, T., Gálvez Rojas, S., & López Valverde, F. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01086-z
- Authors
- Sheokand, T., Jain, G., Bahga, A., & Madisetti, V. K.
- Affiliations
- Bennett University (IN)
- Chandigarh University Uttar Pradesh (IN)
- Georgia Institute of Technology (US)
- Venue
- IEEE Access
- Published
- 2025
- Volume
- 13
- Pages
- 67993-68019
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4409494283
- Cited by
- 7
- References
- 14
- Citations by year
- 2026: 6 · 2025: 1
- FWCI
- 2.83
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, multi-step reasoning, hallucination mitigation, reasoning consistency, medical question answering, question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
