CoChat
Sign Up
PreprintCited by 0Topic ModelingMultimodal Machine Learning ApplicationsNatural Language Processing Techniques

The DOI matches its DataCite record; the references it cites are not checked yet.

Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents

Open access
Jaiswal, Raj, Jain, Dhruv, Popat, Harsh Parimal, Anand, Avinash, Dharmadhikari, Abhishek, Marathe, Atharva, Shah, Rajiv Ratn·2024·arXiv
DOI 10.48550/arxiv.2412.00821arXiv 2412.00821
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) demonstrate remarkable capabilities in various reasoning tasks. However, they encounter significant challenges when it comes to scientific reasoning, particularly in physics, which requires not only mathematical reasoning but also factual and conceptual understanding. When addressing complex physics problems, LLMs typically face three key issues: problem miscomprehension, incorrect concept application, and computational errors. While each of these problems can be addressed individually, there is a need for a generalized approach that can tackle all three issues simultaneously. To address this, we introduce Mixture of Refinement Agents (MoRA), a novel agentic refinement framework that iteratively refines the LLM generated base solution by correcting the aforementioned errors, resulting in a significant performance improvement for open-source LLMs. Our approach aims to bridge the gap between opensource LLMs and GPT-4o by utilizing the latter as error identifier to guide these refinement agents. We evaluate our approach on the SciEval and MMLU subsets along with our own physics dataset (PhysicsQA). MoRA significantly improves the performance of Llama-3-70B and Gemma-2-27B on these datasets, achieving up to a 16% increase in final answer accuracy.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2412.00821

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Even when LLMs possess scientific skills, they often fail to apply them effectively in complex problems and real-world scientific discovery.

    “Notably, even when LLMs possess these scientific skills, they often fail to apply them effectively in complex problems and real-world scientific discovery (Jaiswal et al., 2024; Ouyang et al., 2023; Chen et al., 2025).”

    From Large Language Model Reasoning Failures (2026) · cites this paper as Jaiswal et al., 2024

    The Motivation and Introduction sections report that LLMs struggle to apply appropriate concepts and formulas when solving complex physics problems. The paper does not establish the broader claim about real-world scientific discovery; its evidence concerns physics problem-solving. Page numbers were not provided in our copy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

Research works printed but not linked: 39

  1. Anand, A.; Addala, K.; Baghel, K.; Goel, A.; Hira, M.; Gupta, R.; and Shah, R. R. 2023a. Revolutionizing High School Physics Education: A Novel Dataset. In Goyal, V.; Kumar, N.; Bhowmick, S. S.; Goyal, P.; Goyal, N.; and Kumar, D., eds., Big Data and Artificial Intelligence, 64- 79. Cham: Springer Nature Switzerland. ISBN 978-3-031- 49601-1.
  2. Anand, A.; Goel, A.; Hira, M.; Buldeo, S.; Kumar, J.; Verma, A.; Gupta, R.; and Shah, R. R. 2023b. Sciphyrag-retrieval augmentation to improve llms on physics q &a. In Interna- tional Conference on Big Data Analytics, 50-63. Springer.
  3. Anand, A.; Gupta, M.; Prasad, K.; Goel, U.; Lal, N.; Verma, A.; and Shah, R. R. 2023c. KG-CTG: citation generation through knowledge graph-guided large language models. In International Conference on Big Data Analytics, 37-49.
  4. Anand, A.; Gupta, M.; Prasad, K.; Singla, N.; Sanjeev, S.; Kumar, J.; Shivam, A. R.; and Shah, R. R. 2024a. Mathify: Evaluating Large Language Models on Mathematical Prob- lem Solving Tasks. arXiv preprint arXiv:2404.13099. · arXiv
  5. Anand, A.; Jairath, A.; Lal, N.; Bangar, S.; Sikka, J.; Verma, A.; Shah, R. R.; and Satoh, S. 2023d. GEC-DCL: Grammat- ical Error Correction Model with Dynamic Context Learn- ing for Paragraphs and Scholarly Papers. In International Conference on Big Data Analytics, 95-110. Springer.
  6. Anand, A.; Jaiswal, R.; Dharmadhikari, A.; Marathe, A.; Popat, H.; Mital, H.; Nair, A. R.; Prasad, K.; Kumar, S.; Verma, A.; et al. 2024b. GeoVQA: A Comprehensive Multi- modal Geometry Dataset for Secondary Education. In 2024 IEEE 7th International Conference on Multimedia Informa- tion Processing and Retrieval (MIPR), 102-108. IEEE.
  7. Anand, A.; Kapuriya, J.; Kirtani, C.; Singh, A.; Saraf, J.; Lal, N.; Kumar, J.; Shivam, A. R.; Verma, A.; Shah, R. R.; et al. 2024c. MM-PhyRLHF: Reinforcement Learning Frame- work for Multimodal Physics Question-Answering. arXiv preprint arXiv:2404.12926. · arXiv
  8. Anand, A.; Kapuriya, J.; Singh, A.; Saraf, J.; Lal, N.; Verma, A.; Gupta, R.; and Shah, R. 2024d. MM-PhyQA: Multi- modal Physics Question-Answering with Multi-image CoT Prompting. In Pacific-Asia Conference on Knowledge Dis- covery and Data Mining, 53-64. Springer.
  9. Anand, A.; Prasad, K.; Goel, U.; Gupta, M.; Lal, N.; Verma, A.; and Shah, R. R. 2023e. Context-enhanced language models for generating multi-paper citations. In International Conference on Big Data Analytics, 80-94. Springer.
  10. Arora, D.; Singh, H. G.; et al. 2023. Have llms advanced enough? a challenging problem solving benchmark for large language models. arXiv preprint arXiv:2305.15074. · arXiv
  11. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Ad- vances in neural information processing systems, 33: 1877- 1901.
  12. Chen, W.; Ma, X.; Wang, X.; and Cohen, W. W. 2022. Program of thoughts prompting: Disentangling computa- tion from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588. · arXiv
  13. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. · arXiv
  14. Edge, D.; Trinh, H.; Cheng, N.; Bradley, J.; Chao, A.; Mody, A.; Truitt, S.; and Larson, J. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130. · arXiv
  15. Fu, Y.; Peng, H.; Sabharwal, A.; Clark, P.; and Khot, T. 2022. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Repre- sentations.
  16. Gao, L.; Madaan, A.; Zhou, S.; Alon, U.; Liu, P.; Yang, Y.; Callan, J.; and Neubig, G. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, 10764-10799.
  17. PMLR.
  18. He, C.; Luo, R.; Bai, Y.; Hu, S.; Thai, Z. L.; Shen, J.; Hu, J.; Han, X.; Huang, Y.; Zhang, Y.; et al. 2024. Olympiad- bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008. · arXiv
  19. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. · arXiv
  20. Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; Yih, W.-t.; Rocktäschel, T.; et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Infor- mation Processing Systems, 33: 9459-9474.
  21. Li, X.; Wang, W.; Li, M.; Guo, J.; Zhang, Y.; and Feng, F. 2024. Evaluating Mathematical Reasoning of Large Lan- guage Models: A Focus on Error Identification and Correc- tion. arXiv preprint arXiv:2406.00755. · arXiv
  22. Ling, Z.; Fang, Y.; Li, X.; Huang, Z.; Lee, M.; Memisevic, R.; and Su, H. 2024. Deductive verification of chain-of- thought reasoning. Advances in Neural Information Pro- cessing Systems, 36.
  23. Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.- C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022. Learn to explain: Multimodal reasoning via thought chains for sci- ence question answering. Advances in Neural Information Processing Systems, 35: 2507-2521.
  24. Luo, H.; Sun, Q.; Xu, C.; Zhao, P.; Lou, J.; Tao, C.; Geng, X.; Lin, Q.; Chen, S.; and Zhang, D. 2023. Wizard- math: Empowering mathematical reasoning for large lan- guage models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583. · arXiv
  25. Mruthyunjaya, V.; Pezeshkpour, P.; Hruschka, E.; and Bhutani, N. 2023. Rethinking language models as symbolic knowledge graphs. arXiv preprint arXiv:2308.13676. · arXiv
  26. Ouyang, S.; Zhang, Z.; Yan, B.; Liu, X.; Han, J.; and Qin, L. 2023. Structured chemistry reasoning with large language models. arXiv preprint arXiv:2311.09656. · arXiv
  27. Petroni, F.; Piktus, A.; Fan, A.; Lewis, P.; Yazdani, M.; De Cao, N.; Thorne, J.; Jernite, Y.; Karpukhin, V.; Maillard, J.; et al. 2020. KILT: a benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009.02252. · arXiv
  28. Sun, J.; Luo, Y.; Gong, Y.; Lin, C.; Shen, Y.; Guo, J.; and Duan, N. 2023a. Enhancing chain-of-thoughts prompting with iterative bootstrapping in large language models. arXiv preprint arXiv:2304.11657. · arXiv
  29. Sun, L.; Han, Y.; Zhao, Z.; Ma, D.; Shen, Z.; Chen, B.; Chen, L.; and Yu, K. 2023b. SciEval: A Multi-Level Large Lan- guage Model Evaluation Benchmark for Scientific Research. arXiv preprint arXiv:2308.13149. · arXiv
  30. Sun, L.; Han, Y.; Zhao, Z.; Ma, D.; Shen, Z.; Chen, B.; Chen, L.; and Yu, K. 2024. Scieval: A multi-level large language model evaluation benchmark for scientific research. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 38, 19053-19061.
  31. Tyen, G.; Mansoor, H.; Carbune, V.; Chen, P.; and Mak, T. 2024. LLMs cannot find reasoning errors, but can correct them given the error location. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computa- tional Linguistics ACL 2024, 13894-13908. Bangkok, Thai- land and virtual meeting: Association for Computational Linguistics.
  32. Wang, K.; Ren, H.; Zhou, A.; Lu, Z.; Luo, S.; Shi, W.; Zhang, R.; Song, L.; Zhan, M.; and Li, H. 2023. Mathcoder: Seamless code integration in llms for enhanced mathemati- cal reasoning. arXiv preprint arXiv:2310.03731. · arXiv
  33. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of- thought prompting elicits reasoning in large language mod- els. Advances in neural information processing systems, 35: 24824-24837.
  34. Xu, B.; Yang, A.; Lin, J.; Wang, Q.; Zhou, C.; Zhang, Y.; and Mao, Z. 2023. Expertprompting: Instructing large lan- guage models to be distinguished experts. arXiv preprint arXiv:2305.14688. · arXiv
  35. Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2024. Tree of thoughts: Deliber- ate problem solving with large language models. Advances in Neural Information Processing Systems, 36.
  36. Yasunaga, M.; Chen, X.; Li, Y.; Pasupat, P.; Leskovec, J.; Liang, P.; Chi, E. H.; and Zhou, D. 2023. Large language models as analogical reasoners. arXiv preprint arXiv:2310.01714. · arXiv
  37. Yu, L.; Jiang, W.; Shi, H.; Yu, J.; Liu, Z.; Zhang, Y.; Kwok, J. T.; Li, Z.; Weller, A.; and Liu, W. 2023. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284. · arXiv
  38. Zhang, Z.; Zhang, A.; Li, M.; and Smola, A. 2022. Auto- matic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493. · arXiv
  39. Zhou, A.; Wang, K.; Lu, Z.; Shi, W.; Luo, S.; Qin, Z.; Lu, S.; Jia, A.; Song, L.; Zhan, M.; et al. 2023. Solving challenging math word problems using gpt-4 code interpreter with code- based self-verification. arXiv preprint arXiv:2308.07921. · arXiv

Metadata

Authors
Jaiswal, R., Jain, D., Popat, H. P., Anand, A., Dharmadhikari, A., Marathe, A., & Shah, R. R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2412.00821
arXiv
2412.00821
OpenAlex
W4405033595
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
physics reasoning, large language models, scientific reasoning, computational errors, open-source LLMs, Llama-3-70B, GPT-4o

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review