CoChat
Sign Up
PreprintCited by 19Machine Learning in Materials ScienceTopic ModelingMultimodal Machine Learning Applications

LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation

Open access
Chiang, Yuan, Hsieh, Elvis, Chou, Chia-Hong, Riebesell, Janosh·2024·arXiv
DOI 10.48550/arxiv.2401.17244arXiv 2401.17244
Open this paper in CoChat — free accountOpen PDF

Abstract

Reducing hallucination of Large Language Models (LLMs) is imperative for use in the sciences, where reliability and reproducibility are crucial. However, LLMs inherently lack long-term memory, making it a nontrivial, ad hoc, and inevitably biased task to fine-tune them on domain-specific literature and data. Here we introduce LLaMP, a multimodal retrieval-augmented generation (RAG) framework of hierarchical reasoning-and-acting (ReAct) agents that can dynamically and recursively interact with computational and experimental data on Materials Project (MP) and run atomistic simulations via high-throughput workflow interface. Without fine-tuning, LLaMP demonstrates strong tool usage ability to comprehend and integrate various modalities of materials science concepts, fetch relevant data stores on the fly, process higher-order data (such as crystal structure and elastic tensor), and streamline complex tasks in computational materials and chemistry. We propose a simple metric combining uncertainty and confidence estimates to evaluate the self-consistency of responses by LLaMP and vanilla LLMs. Our benchmark shows that LLaMP effectively mitigates the intrinsic bias in LLMs, counteracting the errors on bulk moduli, electronic bandgaps, and formation energies that seem to derive from mixed data sources. We also demonstrate LLaMP's capability to edit crystal structures and run annealing molecular dynamics simulations using pre-trained machine-learning force fields. The framework offers an intuitive and nearly hallucination-free approach to exploring and scaling materials informatics, and establishes a pathway for knowledge distillation and fine-tuning other language models. Code and live demo are available at https://github.com/chiang-yuan/llamp

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2401.17244

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLaMP significantly reduces hallucination in material informatics by grounding predictions in high-fidelity datasets from the Materials Project and running atomistic simulations.

    “It significantly reduces hallucination in material informatics by grounding predictions in high-fidelity datasets from the Materials Project (MP) and running atomistic simulations.”

    From Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions (2025) · cites this paper as Chiang et al., 2024

    The Abstract and §5.2 Performance Benchmarks describe LLaMP as reducing hallucinations and errors by grounding responses in Materials Project data. The Abstract and the language-driven simulation discussion also show that LLaMP can run atomistic simulations, but they do not establish that running those simulations reduces hallucination; the paper presents this as a separate capability. Page numbers are unavailable in our copy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • LLaMP successfully retrieves and predicts key material properties such as bulk modulus, formation energy, and electronic bandgap, outperforming standard LLMs.

    “LLaMP successfully retrieves and predicts key material properties such as bulk modulus, formation energy, and electronic bandgap, outperforming standard LLMs.”

    From Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions (2025) · cites this paper as Chiang et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 43

  1. Y. Bang et al. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reason- ing, Hallucination, and Interactivity, Nov. 2023. URL http://arxiv.org/abs/2302.04023. arXiv:2302.04023 [cs]. · arXiv
  2. I. Batatia et al. A foundation model for atomistic materials chemistry, Dec. 2023. URL http: //arxiv.org/abs/2401.00096. arXiv:2401.00096 [cond-mat, physics:physics]. · arXiv
  3. D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes. Autonomous chemical research with large language models. Nature, 624(7992):570-578, Dec. 2023. ISSN 1476- 4687. doi: 10.1038/s41586-023-06792-0. URL https://www.nature.com/articles/ s41586-023-06792-0. Number: 7992 Publisher: Nature Publishing Group. · DOI · Paper page
  4. P. Borlido et al. Exchange-correlation functionals for band gaps of solids: benchmark, reparametrization and machine learning. npj Computational Materials, 6(1):1-17, July 2020. ISSN 2057-3960. doi: 10.1038/s41524-020-00360-0. URL https://www.nature.com/ articles/s41524-020-00360-0. Number: 1 Publisher: Nature Publishing Group. · DOI
  5. K. Cobbe et al. Training Verifiers to Solve Math Word Problems, Nov. 2021. URL http: //arxiv.org/abs/2110.14168. arXiv:2110.14168 [cs]. · arXiv
  6. J. Dagdelen et al. Structured information extraction from scientific text with large language models. Nature Communications, 15(1):1418, Feb. 2024. ISSN 2041-1723. doi: 10.1038/ s41467-024-45563-x. URL https://www.nature.com/articles/s41467-024-45563-x. Publisher: Nature Publishing Group. · DOI · Paper page
  7. N. Frey et al. Neural Scaling of Deep Chemical Models, May 2022. URL https://chemrxiv. org/engage/chemrxiv/article-details/627bddd544bdd532395fb4b5.
  8. A. M. Ganose and A. Jain. Robocrystallographer: automated crystal structure text descriptions and analysis. MRS Communications, 9(3):874-881, Sept. 2019. ISSN 2159-6867. doi: 10. 1557/mrc.2019.94. URL https://doi.org/10.1557/mrc.2019.94. · DOI
  9. N. Gruver et al. Fine-Tuned Language Models Generate Stable Inorganic Materials as Text. Nov. 2023. URL https://openreview.net/forum?id=0r5DE2ZSwJ.
  10. T. Gupta, M. Zaki, N. M. A. Krishnan, and Mausam. MatSciBERT: A materials domain language model for text mining and information extraction. npj Computational Materi- als, 8(1):1-11, May 2022. ISSN 2057-3960. doi: 10.1038/s41524-022-00784-w. URL https://www.nature.com/articles/s41524-022-00784-w. Number: 1 Publisher: Nature Publishing Group. · DOI
  11. D. Hendrycks et al. Measuring Massive Multitask Language Understanding, Jan. 2021. URL http://arxiv.org/abs/2009.03300. arXiv:2009.03300 [cs]. · arXiv
  12. K. M. Jablonka, P. Schwaller, A. Ortega-Guerrero, and B. Smit. Leveraging large language models for predictive chemistry. Nature Machine Intelligence, 6(2):161-169, Feb. 2024. ISSN 2522-5839. doi: 10.1038/s42256-023-00788-1. URL https://www.nature.com/articles/ s42256-023-00788-1. Publisher: Nature Publishing Group. · DOI · Paper page
  13. A. Jain et al. Commentary: The Materials Project: A materials genome approach to acceler- ating materials innovation. APL Materials, 1(1):011002, July 2013. ISSN 2166-532X. doi: 10.1063/1.4812323. URL https://doi.org/10.1063/1.4812323. · DOI · Paper page
  14. O. Kononova et al. Text-mined dataset of inorganic materials synthesis recipes. Scientific Data, 6(1):203, Oct. 2019. ISSN 2052-4463. doi: 10.1038/s41597-019-0224-1. URL https:// www.nature.com/articles/s41597-019-0224-1. Number: 1 Publisher: Nature Publishing Group. · DOI
  15. G. Kresse and J. Furthmüller. Efficient iterative schemes for ab initio total-energy calcula- tions using a plane-wave basis set. Physical Review B, 54(16):11169-11186, Oct. 1996. doi: 10.1103/PhysRevB.54.11169. URL https://link.aps.org/doi/10.1103/PhysRevB.54. 11169. Publisher: American Physical Society. · DOI · Paper page
  16. P. Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459-9474.
  17. P. Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Apr. 2021. URL http://arxiv.org/abs/2005.11401. arXiv:2005.11401 [cs] version: 4. · arXiv
  18. J. Li et al. True Composition and Structure of Hexagonal "YAlO3", Actually Y3Al3O8CO3. Inorganic Chemistry, 54(3):837-844, Feb. 2015. ISSN 0020-1669. doi: 10.1021/ic502027k. URL https://doi.org/10.1021/ic502027k. Publisher: American Chemical Society. · DOI
  19. P. Liang et al. Holistic Evaluation of Language Models, Oct. 2023. URL http://arxiv.org/ abs/2211.09110. arXiv:2211.09110 [cs]. · arXiv
  20. Y. Luo, C. Liu, and S. Ji. Towards Symmetry-Aware Generation of Periodic Materials, July 2023. URL https://arxiv.org/abs/2307.02707v2. · arXiv
  21. A. M. Bran et al. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6(5):525-535, May 2024. ISSN 2522-5839. doi: 10.1038/s42256-024-00832-8. URL https://www.nature.com/articles/s42256-024-00832-8. Publisher: Nature Pub- lishing Group. · DOI · Paper page
  22. A. Mallen et al. When Not to Trust Language Models: Investigating Effectiveness of Paramet- ric and Non-Parametric Memories, July 2023. URL http://arxiv.org/abs/2212.10511. arXiv:2212.10511 [cs]. · arXiv
  23. A. L. Maximenko and E. A. Olevsky. Effective diffusion coefficients in solid-state sinter- ing. Acta Materialia, 52(10):2953-2963, June 2004. ISSN 1359-6454. doi: 10.1016/ j.actamat.2004.02.042. URL https://www.sciencedirect.com/science/article/pii/ S1359645404001326. · DOI
  24. A. Merchant et al. Scaling deep learning for materials discovery. Nature, pages 1-6, Nov. 2023. ISSN 1476-4687. doi: 10.1038/s41586-023-06735-9. URL https://www.nature. com/articles/s41586-023-06735-9. Publisher: Nature Publishing Group. · DOI · Paper page
  25. S. Miret and N. M. A. Krishnan. Are LLMs Ready for Real-World Materials Discovery?, Sept. 2024. URL http://arxiv.org/abs/2402.05200. arXiv:2402.05200 [cond-mat]. · arXiv
  26. S. Ouyang et al. Structured Chemistry Reasoning with Large Language Models, Feb. 2024. URL http://arxiv.org/abs/2311.09656. arXiv:2311.09656 [cs]. · arXiv
  27. G. Pilania, J. E. Gubernatis, and T. Lookman. Multi-fidelity machine learning models for accurate bandgap predictions of solids. Computational Materials Science, 129:156- 163, Mar. 2017. ISSN 0927-0256. doi: 10.1016/j.commatsci.2016.12.004. URL https: //www.sciencedirect.com/science/article/pii/S0927025616306188. · DOI
  28. M. P. Polak and D. Morgan. Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering, June 2023. URL http://arxiv. org/abs/2303.05352. arXiv:2303.05352 [cond-mat]. · arXiv
  29. A. Radford et al. Learning Transferable Visual Models From Natural Language Supervision, Feb. 2021. URL http://arxiv.org/abs/2103.00020. arXiv:2103.00020 [cs]. · arXiv
  30. Y. Shao et al. Advances in molecular quantum chemistry contained in the Q-Chem 4 pro- gram package. Molecular Physics, 113(2):184-215, Jan. 2015. ISSN 0026-8976. doi: 10. 1080/00268976.2014.952696. URL https://www.tandfonline.com/doi/full/10.1080/ 00268976.2014.952696. Publisher: Taylor & Francis. · DOI
  31. E. W. C. Spotte-Smith et al. A database of molecular properties integrated in the Materials Project, Oct. 2023. URL https://chemrxiv.org/engage/chemrxiv/article-details/ 651a3302ade1178b247ee6d1.
  32. N. J. Szymanski et al. An autonomous laboratory for the accelerated synthesis of novel materials. Nature, 624(7990):86-91, Dec. 2023. ISSN 1476-4687. doi: 10.1038/ s41586-023-06734-w. URL https://www.nature.com/articles/s41586-023-06734-w. Number: 7990 Publisher: Nature Publishing Group. · DOI · Paper page
  33. P. K. Todd et al. Selectivity in Yttrium Manganese Oxide Synthesis via Local Chemical Potentials in Hyperdimensional Phase Space. Journal of the American Chemical Society, 143(37):15185-15194, Sept. 2021. ISSN 0002-7863. doi: 10.1021/jacs.1c06229. URL https://doi.org/10.1021/jacs.1c06229. Publisher: American Chemical Society. · DOI
  34. V. Tshitoyan et al. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature, 571(7763):95-98, July 2019. ISSN 1476-4687. doi: 10.1038/ s41586-019-1335-8. URL https://www.nature.com/articles/s41586-019-1335-8. Number: 7763 Publisher: Nature Publishing Group. · DOI · Paper page
  35. W. Wang et al. Augmenting Language Models with Long-Term Memory, June 2023. URL http://arxiv.org/abs/2306.07174. arXiv:2306.07174 [cs]. · arXiv
  36. M. Wen et al. Chemical reaction networks and opportunities for machine learning. Na- ture Computational Science, 3(1):12-24, Jan. 2023. ISSN 2662-8457. doi: 10.1038/ s43588-022-00369-z. URL https://www.nature.com/articles/s43588-022-00369-z. Publisher: Nature Publishing Group. · DOI
  37. M. Wen et al. An equivariant graph neural network for the elasticity tensors of all seven crystal systems. Digital Discovery, 3(5):869-882, May 2024. ISSN 2635-098X. doi: 10. 1039/D3DD00233K. URL https://pubs.rsc.org/en/content/articlelanding/2024/ dd/d3dd00233k. Publisher: RSC. · DOI
  38. T. Xie et al. DARWIN Series: Domain Specific Large Language Models for Natural Sci- ence, Aug. 2023. URL http://arxiv.org/abs/2308.13565. arXiv:2308.13565 [cond-mat, physics:physics]. · arXiv
  39. Z. Xu, S. Jain, and M. Kankanhalli. Hallucination is Inevitable: An Innate Limitation of Large Language Models, Jan. 2024. URL http://arxiv.org/abs/2401.11817. arXiv:2401.11817 [cs]. · arXiv
  40. S. Yang et al. Accurate Prediction of Experimental Band Gaps from Large Language Model- Based Data Extraction. Nov. 2023. URL https://openreview.net/forum?id=oRKWhmtUG6.
  41. S. Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models, Mar. 2023. URL http://arxiv.org/abs/2210.03629. arXiv:2210.03629 [cs]. · arXiv
  42. D. Zagorac et al. Recent developments in the Inorganic Crystal Structure Database: theo- retical crystal structure data and related features. Journal of Applied Crystallography, 52 (5):918-925, Oct. 2019. ISSN 1600-5767. doi: 10.1107/S160057671900997X. URL https://journals.iucr.org/j/issues/2019/05/00/in5024/. Publisher: International Union of Crystallography. · DOI
  43. C. Zeni et al. MatterGen: a generative model for inorganic materials design, Jan. 2024. URL http://arxiv.org/abs/2312.03687. arXiv:2312.03687 [cond-mat]. · arXiv

Cited by

Sorted by citations

Authors
Chiang, Y., Hsieh, E., Chou, C.-H., & Riebesell, J.
Affiliations
  • Lawrence Berkeley National Laboratory (US)
  • University of California, Berkeley (US)
  • Foothill College (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2401.17244
arXiv
2401.17244
OpenAlex
W4391421491
Cited by
19
References
0
Citations by year
2026: 4 · 2025: 10 · 2024: 5
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
retrieval-augmented generation, ReAct agents, hallucination mitigation, knowledge distillation, Materials Project, atomistic simulation, high-throughput workflow, multimodal large language models, machine learning interatomic potentials, materials informatics, LLM fine-tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review