CoChat
Sign Up
Cited by 126Machine Learning in Materials ScienceComputational Drug Discovery MethodsAdvanced Chemical Physics Studies

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.

Neural scaling of deep chemical models

Open access
Frey, Nathan C., Soklaski, Ryan, Axelrod, Simon, Samsi, Siddharth, Gómez-Bombarelli, Rafael, Coley, Connor W., Gadepally, Vijay·2023·Nature Machine Intelligence
DOI 10.1038/s42256-023-00740-3

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Massive scale, in terms of both data availability and computation, enables important breakthroughs in key application areas of deep learning such as natural language processing and computer vision. There is emerging evidence that scale may be a key ingredient in scientific deep learning, but the importance of physical priors in scientific domains makes the strategies and benefits of scaling uncertain. Here we investigate neural-scaling behaviour in large chemical models by varying model and dataset sizes over many orders of magnitude, studying models with over one billion parameters, pre-trained on datasets of up to ten million datapoints. We consider large language models for generative chemistry and graph neural networks for machine-learned interatomic potentials. We investigate the interplay between physical priors and scale and discover empirical neural-scaling relations for language models in chemistry with a scaling exponent of 0.17 for the largest dataset size considered, and a scaling exponent of 0.26 for equivariant graph neural network interatomic potentials.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s42256-023-00740-3

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 48 references

Research works printed but not linked: 21

  1. Sriram, A., Das, A., Wood, B. M., Goyal, S. & Zitnick, C. L. Towards training billion parameter graph neural networks for atomic simulations. Preprint at https://arxiv.org/abs/2203.09697 (2022). · arXiv
  2. Chithrananda, S., Grand, G. & Ramsundar, B. Chemberta: large-scale self-supervised pretraining for molecular property prediction. Preprint at https://arxiv.org/abs/2010.09885 (2020). · arXiv
  3. Kalinin, S. V., Ziatdinov, M., Sumpter, B. G. & White, A. D. Physics is the new data. Preprint at https://arxiv.org/abs/2204.05095 (2022). · arXiv
  4. Pappu, A. & Paige, B. Making graph neural networks worth it for low-data molecular machine learning. Preprint at https://arxiv.org/abs/2011.12203 (2020). · arXiv
  5. Gasteiger, J. et al. How do graph networks generalize to large and diverse molecular systems? Preprint at https://arxiv.org/abs/2204.02782 (2022). · arXiv
  6. Ahmad, W., Simon, E., Chithrananda, S., Grand, G. & Ramsundar, B. ChemBERTa-2: towards chemical foundation models. Preprint at https://arxiv.org/abs/2209.01712 (2022). · arXiv
  7. Musaelian, A., Johansson, A., Batzner, S. & Kozinsky, B. Scaling the leading accuracy of deep equivariant models to biomolecular simulations of realistic size. Preprint at https://arxiv.org/abs/2304.10061 (2023) · arXiv
  8. McCandlish, S., Kaplan, J., Amodei, D. & OpenAI Dota Team An empirical model of large-batch training. Preprint at https://arxiv.org/abs/1812.06162 (2018). · arXiv
  9. Bahri, Y., Dyer, E., Kaplan, J., Lee, J. & Sharma, U. Explaining neural scaling laws. Preprint at https://arxiv.org/abs/2102.06701 (2021). · arXiv
  10. Caballero, E., Gupta, K., Rish, I. & Krueger, D. Broken neural scaling laws. Preprint at https://arxiv.org/abs/2210.14891 (2022). · arXiv
  11. Loukas, A. What graph neural networks cannot learn: depth vs width. Preprint at https://arxiv.org/abs/1907.03199 (2019). · arXiv
  12. Batatia, I., Kovács, D. P., Simm, G. N. C., Ortner, C. & Csányi, G. MACE: higher order equivariant message passing neural networks for fast and accurate force fields. In Advances in Neural Information Processing Systems Vol. 35 (eds Koyejo, S. et al.) 11423–11436 (Curran Associates, 2022).
  13. Rackers, J. A. & Rao, P. Hierarchical learning in Euclidean neural networks. Preprint at https://arxiv.org/abs/2210.0476 (2022).
  14. Gruver, N., Finzi, M., Goldblum, M. & Wilson, A. G. The lie derivative for measuring learned equivariance. Preprint at https://arxiv.org/abs/2210.02984 (2022). · arXiv
  15. Radford, A. et al. Language models are unsupervised multitask learners. OpenAI Blog 1, 9 (2019).
  16. Gao, L. et al. The Pile: an 800gb dataset of diverse text for language modeling. Preprint at https://arxiv.org/abs/2101.00027 (2020). · arXiv
  17. Bengio, Y., Ducharme, R. & Vincent, P. A neural probabilistic language model. In Advances in Neural Information Processing Systems Vol 13 (eds Leen, T. et al.) (MIT Press, 2000).
  18. Schütt, K., Unke, O. & Gastegger, M. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning 9377–9388 (PMLR, 2021).
  19. Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A. & Talwalkar, A. Hyperband: a novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res. 18, 6765–6816 (2017).
  20. Loshchilov, I., Hutter, F. Decoupled weight decay regularization. Preprint at https://arxiv.org/abs/1711.05101 (2017). · arXiv
  21. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization Preprint at https://arxiv.org/abs/1412.6980 (2014). · arXiv

Cited by

Sorted by citations

See all 126

Related

Metadata

Authors
Frey, N. C., Soklaski, R., Axelrod, S., Samsi, S., Gómez-Bombarelli, R., Coley, C. W., & Gadepally, V.
Affiliations
  • MIT Lincoln Laboratory (US)
  • Harvard University (US)
  • Massachusetts Institute of Technology (US)
Corresponding author
Frey, Nathan C.
ORCID
Venue
Nature Machine Intelligence
Publisher
Springer Science and Business Media LLC
Published
2023-10-23
Volume
5
Issue
11
Pages
1297-1305
ISSN
2522-5839 (ISSN-L)
DOI
10.1038/s42256-023-00740-3
OpenAlex
W4387874094
Cited by
125
References
48
Citations by year
2026: 41 · 2025: 43 · 2024: 36 · 2023: 4
FWCI
9.16
Type
journal article
Language
en
Open access
Licence
Locations
Nature Machine Intelligence (publishedVersion, cc-by)
Funders
United States Department of Defense | U.S. Air Force (FA8702-15-D-0001)
Topics
  • Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
  • Physical Sciences › Physics and Astronomy › Atomic and Molecular Physics, and Optics › Advanced Chemical Physics Studies
Keywords
physical priors, generative chemistry, large language models, graph neural networks, interatomic potentials, equivariant graph neural networks, scaling exponents, model size, dataset size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review