CoChat
Sign Up
PreprintCited by 10Machine Learning in Materials ScienceMultimodal Machine Learning ApplicationsTopic Modeling

Structured Chemistry Reasoning with Large Language Models

Open access
Ouyang, Siru, Zhang, Zhuosheng, Yan, Bing, Liu, Xuan, Choi, Yejin, Han, Jiawei, Qin, Lianhui·2023·arXiv
DOI 10.48550/arxiv.2311.09656arXiv 2311.09656
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) excel in diverse areas, yet struggle with complex scientific reasoning, especially in the field of chemistry. Different from the simple chemistry tasks (e.g., molecule classification) addressed in previous studies, complex chemistry problems require not only vast knowledge and precise calculation, but also compositional reasoning about rich dynamic interactions of different concepts (e.g., temperature changes). Our study shows that even advanced LLMs, like GPT-4, can fail easily in different ways. Interestingly, the errors often stem not from a lack of domain knowledge within the LLMs, but rather from the absence of an effective reasoning structure that guides the LLMs to elicit the right knowledge, incorporate the knowledge in step-by-step reasoning, and iteratively refine results for further improved quality. On this basis, we introduce StructChem, a simple yet effective prompting strategy that offers the desired guidance and substantially boosts the LLMs' chemical reasoning capability. Testing across four chemistry areas -- quantum chemistry, mechanics, physical chemistry, and kinetics -- StructChem substantially enhances GPT-4's performance, with up to 30\% peak improvement. Our analysis also underscores the unique difficulties of precise grounded reasoning in science with LLMs, highlighting a need for more research in this area. Code is available at \url{https://github.com/ozyyshr/StructChem}.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2311.09656

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Even when LLMs possess scientific skills, they often fail to apply them effectively in complex problems and real-world scientific discovery.

    “Notably, even when LLMs possess these scientific skills, they often fail to apply them effectively in complex problems and real-world scientific discovery (Jaiswal et al., 2024; Ouyang et al., 2023; Chen et al., 2025).”

    From Large Language Model Reasoning Failures (2026) · cites this paper as Ouyang et al., 2023

    The Introduction (Section 1; page unavailable) says LLMs often encode the knowledge needed for a chemistry problem but lack a reasoning structure to apply it effectively, leading to errors on complex problems. The paper studies chemistry reasoning tasks and discusses possible scientific applications; it does not establish that LLMs often fail in real-world scientific discovery. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 33

  1. Atkins, P., De Paula, J., and Friedman, R. Physical chemistry: quanta, matter, and change. Oxford University Press, USA, 2014.
  2. Atkins, P., De Paula, J., and Keeler, J. Atkins' physical chemistry. Oxford university press, 2023.
  3. Baum, Z. J., Yu, X., Ayala, P. Y., Zhao, Y., Watkins, S. P., and Zhou, Q. Artificial intelligence in chemistry: current trends and future directions. Journal of Chemical Information and Modeling, 61(7):3197- 3212, 2021.
  4. Bran, A. M., Cox, S., White, A. D., and Schwaller, P. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376, 2023. · arXiv
  5. Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023a. ISSN 2835-8856. URL https:// openreview.net/forum?id=YfZ4ZPt8zd.
  6. Chen, W., Ming, Y., Max, K., Elaine, W., Ma, X., Xu, J., Xia, T., Wang, X., and Lu, P. Theoremqa: A theorem-driven question answering dataset. arXiv preprint arXiv:2305.12524, 2023b. · arXiv
  7. Edwards, C. N., Naik, A., Khot, T., Burke, M. D., Ji, H., and Hope, T. Synergpt: In-context learning for personalized drug synergy prediction and drug design. bioRxiv, pp. 2023-07, 2023.
  8. Fang, Y., Liang, X., Zhang, N., Liu, K., Huang, R., Chen, Z., Fan, X., and Chen, H. Mol-instructions: A large-scale biomolecular instruction dataset for large language models. arXiv preprint arXiv:2306.08018, 2023. · arXiv
  9. Feinberg, E. N., Sur, D., Wu, Z., Husic, B. E., Mai, H., Li, Y., Sun, S., Yang, J., Ramsundar, B., and Pande, V. S. Potentialnet for molecular property prediction. ACS central science, 4(11):1520-1530, 2018.
  10. Fu, Y., Peng, H., Sabharwal, A., Clark, P., and Khot, T. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, 2023. URL https: //openreview.net/forum?id=yf1icZHC-l9.
  11. Graulich, N., Rost, M., Schultz, M., and Gallardo- Williams, M. To read is the challenge -insights from 100 days, 100 papers reading challenge in chemistry education research. Journal of Chemical Education, 99(10):3364-3369, 2022.
  12. Hair, J., Black, W., Babin, B., Anderson, R., and Tatham, R. Pearson prentice-hall; upper saddle river, nj. Multivariate data analysis.[Google Scholar], 2009.
  13. Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview. net/forum?id=nZeVKeeFYf9.
  14. Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023. · arXiv
  15. Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., and Sabharwal, A. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id= _nGgzQjzaRy.
  16. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero- shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
  17. Liao, C., Yu, Y., Mei, Y., and Wei, Y. From words to molecules: A survey of large language models in chemistry. arXiv preprint arXiv:2402.01439, 2024. · arXiv
  18. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
  19. Öztürk, H., Özgür, A., Schwaller, P., Laino, T., and Ozkirimli, E. Exploring chemical space using natural language processing methodologies for drug discovery. Drug Discovery Today, 25(4):689-705, 2020.
  20. Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole- Lewis, H., Pfohl, S., et al. Large language models encode clinical knowledge. Nature, 620(7972):172- 180, 2023.
  21. Stechly, K., Marquez, M., and Kambhampati, S. Gpt- 4 doesn't know it's wrong: An analysis of iterative prompting for reasoning problems. arXiv preprint arXiv:2310.12397, 2023. · arXiv
  22. Sun, Z., Wang, X., Tay, Y., Yang, Y., and Zhou, D. Recitation-augmented language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview. net/forum?id=-cqvvvb-NkI.
  23. Tang, X., Jin, Q. J., Zhu, K., Yuan, T., Zhang, Y., Zhou, W., Qu, M., Zhao, Y., Tang, J., Zhang, Z., Cohan, A., Lu, Z., and Gerstein, M. Prioritizing safeguarding over autonomy: Risks of llm agents for science. arXiv preprint arXiv:2402.04247, 2024. · arXiv
  24. Taskin, V. and Bernholt, S. Students' understanding of chemical formulae: A review of empirical research. International Journal of Science Education, 36(1): 157-185, 2014.
  25. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
  26. Wang, X., Hu, Z., Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., and Wang, W. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint arXiv:2307.10635, 2023a. · arXiv
  27. Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In The Eleventh Inter- national Conference on Learning Representations, 2023b. URL https://openreview.net/forum? id=1PL1NIMMrw.
  28. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. URL https: //openreview.net/forum?id=_VjQlMeSB_J.
  29. Weng, Y., Zhu, M., He, S., Liu, K., and Zhao, J. Large language models are reasoners with self-verification. arXiv preprint arXiv:2212.09561, 2022. · arXiv
  30. Yang, K., Swanson, K., Jin, W., Coley, C., Eiden, P., Gao, H., Guzman-Perez, A., Hopper, T., Kelley, B., Mathea, M., et al. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 59(8):3370- 3388, 2019.
  31. Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations (ICLR 2023), 2023.
  32. Zhong, M., Ouyang, S., Jiang, M., Hu, V., Jiao, Y., Wang, X., and Han, J. ReactIE: Enhancing chemical reaction extraction with weak supervision. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 12120-12130, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl. 767. URL https://aclanthology.org/2023. findings-acl.767. · DOI
  33. Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q. V., and Chi, E. H. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview. net/forum?id=WZH7099tgfM. Please provide a clear and step-by-step solution for a scientific problem in the categories of Chemistry, Physics, or Mathematics. The problem will specify the unit of measurement, which should not be included in the answer. Express the final answer as a decimal number with three digits after the decimal point. Conclude the answer by stating "The answer is therefore \\boxed{[ANSWER]}.

Cited by

Sorted by citations

Related

Authors
Ouyang, S., Zhang, Z., Yan, B., Liu, X., Choi, Y., Han, J., & Qin, L.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Shanghai Jiao Tong University (CN)
  • New York University (US)
  • University of Washington (US)
  • University of California San Diego (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2311.09656
arXiv
2311.09656
OpenAlex
W4388787940
Cited by
9
References
0
Citations by year
2026: 2 · 2025: 4 · 2024: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, compositional reasoning, prompting strategies, GPT-4, quantum chemistry, physical chemistry, reaction kinetics, mechanics, grounded reasoning, iterative refinement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review