Do Large Language Models Understand Chemistry? A Conversation with ChatGPT
Abstract
Large language models (LLMs) have promised a revolution in answering complex questions using the ChatGPT model. Its application in chemistry is still in its infancy. This viewpoint addresses the question of how well ChatGPT understands chemistry by posing five simple tasks in different subareas of chemistry.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT's accuracy on five straightforward chemistry tasks ranged from 25% to 100%.
“Castro Nascimento and Pimentel [18] presented five straightforward tasks from various subareas of chemistry to assess ChatGPT's comprehension of the subject, with accuracy ranging from 25% to 100%.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [18]
Cites
Weininger, D. (1988) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci00057a005 · Cited by 7,436 · see citing works
ChatGPT and other large language models are double-edged swords
Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., & Moy, L. (2023) · Radiology · doi:10.1148/radiol.230163 · Cited by 1,008 · see citing works
Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation
Krenn, M., Häse, F., Nigam, A., Friederich, P., & Aspuru‐Guzik, A. (2020) · Machine Learning Science and Technology · doi:10.1088/2632-2153/aba947 · Cited by 917 · see citing works
Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation
Krenn, M., Häse, F., Nigam, A., Friederich, P., & Aspuru‐Guzik, A. (2020) · Repository KITopen (Karlsruhe Institute of Technology) · doi:10.5445/ir/1000128111 · Cited by 616 · see citing works
Water Soluble Polymers for Pharmaceutical Applications
Kadajji, V. G., & Betageri, G. V. (2011) · Polymers · doi:10.3390/polym3041972 · Cited by 611 · see citing works
Large-scale chemical language representations capture molecular structure and properties
Ross, J., Belgodere, B., Chenthamarakshan, V., Padhi, I., Mroueh, Y., & Das, P. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00580-7 · Cited by 590 · see citing works
Wang, S., Guo, Y., Wang, Y., Sun, H., & Huang, J. (2019) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3307339.3342186 · Cited by 412 · see citing works
Griffin, S. G., Wyllie, S. G., & Markham, J. L. (1999) · Journal of Chromatography A · doi:10.1016/s0021-9673(99)01009-2 · Cited by 190 · see citing works
Language models can learn complex molecular distributions
Flam-Shepherd, D., Zhu, K., & Aspuru‐Guzik, A. (2022) · Nature Communications · doi:10.1038/s41467-022-30839-x · Cited by 167 · see citing works
Assessment of chemistry knowledge in large language models that generate code
White, A. D., Hocky, G. M., Gandhi, H. A., Ansari, M., Cox, S., Wellawatte, G. P., Sasmal, S., Yang, Z., Liu, K., Singh, Y., et al. (2023) · Digital Discovery · doi:10.1039/d2dd00087c · Cited by 119 · see citing works
A fingerprints based molecular property prediction method using the BERT model
Wen, N., Liu, G., Zhang, J., Zhang, R., Fu, Y., & Han, X. (2022) · Journal of Cheminformatics · doi:10.1186/s13321-022-00650-3 · Cited by 74 · see citing works
Hocky, G. M., & White, A. D. (2022) · Digital Discovery · doi:10.1039/d1dd00009h · Cited by 60 · see citing works
Vilas-Boas, S. M., da Costa, M. C., Coutinho, J. A. P., Ferreira, O., & Pinho, S. P. (2022) · Industrial & Engineering Chemistry Research · doi:10.1021/acs.iecr.1c04196 · Cited by 32 · see citing works
Large language model for molecular chemistry
Pan, J. (2023) · Nature Computational Science · doi:10.1038/s43588-023-00399-1 · Cited by 31 · see citing works
Language models for the prediction of SARS-CoV-2 inhibitors
Blanchard, A. E., Gounley, J., Bhowmik, D., Chandra Shekar, M., Lyngaas, I., Gao, S., Yin, J., Tsaris, A., Wang, F., & Glaser, J. (2022) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420221121804 · Cited by 30 · see citing works
Lipophilicity of Coarse-Grained Cholesterol Models
Fornasier, F., Souza, L. M. P., de Souza, F. R., Reynaud, F., & Pimentel, A. S. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00830 · Cited by 27 · see citing works
Self-citationMolformer: Large Scale Chemical Language Representations Capture Molecular Structure and PropertiesPreprint
Ross, J., Belgodere, B. M., Chenthamarakshan, V., et al. (2022) · Research Square · doi:10.21203/rs.3.rs-1570270/v1 · Cited by 24 · see citing works
Neural Scaling of Deep Chemical Models
Frey, N., Soklaski, R., Axelrod, S., Samsi, S., Gomez-Bombarelli, R., Coley, C., & Gadepally, V. (2022) · doi:10.26434/chemrxiv-2022-3s512 · Cited by 20 · see citing works
Horawalavithana, S., Ayton, E., Sharma, S. K., Howland, S., Subramanian, M., Vasquez, S., Cosbey, R., Glenski, M., & Volkova, S. (2022) · BIGSCIENCE · doi:10.18653/v1/2022.bigscience-1.12 · Cited by 19 · see citing works
Souza, L. M. P., de Souza, F. R., Reynaud, F., & Pimentel, A. S. (2020) · Journal of Molecular Liquids · doi:10.1016/j.molliq.2020.114132 · Cited by 18 · see citing works
Self-citationChemical Language Models for Applications in Medicinal Chemistry
Yoshimori, A., Chen, H., & Bajorath, J. (2023) · Future Medicinal Chemistry · doi:10.4155/fmc-2022-0315 · Cited by 9 · see citing works
Costa, R. K. M., Souza, L. M. P., Silva, R. S., et al. (2023) · Journal of Biomolecular Structure and Dynamics · doi:10.1080/07391102.2023.2173298 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Augmenting large language models with chemistry tools
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00832-8 · Cited by 794 · see citing works
A review of large language models and autonomous agents in chemistry
Ramos, M. C., Collison, C. J., & White, A. D. (2025) · Chemical Science · doi:10.1039/d4sc03921a · Cited by 274 · see citing works
West, J. K., Franz, J. L., Hein, S. M., Leverentz-Culp, H. R., Mauser, J. F., Ruff, E. F., & Zemke, J. M. (2023) · Journal of Chemical Education · doi:10.1021/acs.jchemed.3c00581 · Cited by 79 · see citing works
Towards Automatic Boundary Detection for Human-AI Collaborative Hybrid Essay in Education
Zeng, Z., Sha, L., Li, Y., Yang, K., Gašević, D., & Chen, G. (2023) · arXiv · doi:10.48550/arxiv.2307.12267 · Cited by 25 · see citing works
- Authors
- Castro Nascimento, C. M., & Pimentel, A. S.
- Affiliations
- Departamento de Química, Pontifícia Universidade Católica do Rio de Janeiro, P.O. Box 38097, Rio de Janeiro, RJ 22451-900, Brazil
- Pontifícia Universidade Católica do Rio de Janeiro (BR)
- Corresponding author
- Pimentel, André Silva
- Venue
- Journal of Chemical Information and Modeling
- Publisher
- American Chemical Society (ACS)
- Published
- 2023-03-16
- Volume
- 63
- Issue
- 6
- Pages
- 1649-1655
- ISSN
- 1549-9596 (ISSN-L)
- 1549-960X
- OpenAlex
- W4327564965
- Cited by
- 147
- References
- 22
- Citations by year
- 2026: 21 · 2025: 49 · 2024: 43 · 2023: 34
- FWCI
- 10.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Journal of Chemical Information and Modeling (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Conselho Nacional de Desenvolvimento Cient?fico e Tecnol?gico (302554/2017-3, 310166/2020-9, 465259/2014-6, 573560/2008-0)
- Funda??o de Amparo ? Pesquisa do Estado de S?o Paulo (2014/50983-3)
- Funda??o Carlos Chagas Filho de Amparo ? Pesquisa do Estado do Rio de Janeiro (201.186/2022, 210.104/2020, E-26/010.000983/2019)
- Coordena??o de Aperfei?oamento de Pessoal de N?vel Superior (001)
- Topics
- Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- large language models, ChatGPT, chemical knowledge
- MeSH terms
- Communication, Language
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
