CoChat
Sign Up
Cited by 635Computational Drug Discovery MethodsMachine Learning in Materials ScienceAdvanced Chemical Physics Studies

The DOI matches its Crossref record; 44 of the 46 references it cites were found in a registry or an index.

Large-scale chemical language representations capture molecular structure and properties

Open access
Ross, Jerret, Belgodere, Brian, Chenthamarakshan, Vijil, Padhi, Inkit, Mroueh, Youssef, Das, Payel·2022·Nature Machine Intelligence
DOI 10.1038/s42256-022-00580-7

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s42256-022-00580-7

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Overview of MoLFormer pipeline. The transformer neural network based model is trained on the SMILES sequences corresponding to a large collection of chemical molecules from PubChem and Zinc, two public chemical databases in a self-supervised fashion. MOLFORMER was designed with an efficient linear attention mechanism and relative positional embeddings, with the goal of learning a meaningful and compressed representation of chemical molecules. This foundation model was then adopted to different downstream molecular property prediction tasks via fine-tuning on task-specific data. The representative power was further tested by recovering molecular similarity using the MOLFORMER encodings, as well as by analyzing the correspondence between the interatomic spatial distance and attention value for a given molecule.
Figure 1 Overview of MoLFormer pipeline. The transformer neural network based model is trained on the SMILES sequences corresponding to a large collection of chemical molecules from PubChem and Zinc, two public chemical databases in a self-supervised fashion. MOLFORMER was designed with an efficient linear attention mechanism and relative positional embeddings, with the goal of learning a meaningful and compressed representation of chemical molecules. This foundation model was then adopted to different downstream molecular property prediction tasks via fine-tuning on task-specific data. The representative power was further tested by recovering molecular similarity using the MOLFORMER encodings, as well as by analyzing the correspondence between the interatomic spatial distance and attention value for a given molecule.CC-BY · From the PDF
Image 2
CC-BY · From the PDF
Image 3
CC-BY · From the PDF
Figure 4 Visualization of the learned attention map (using either full or linear attention) under rotary embedding and corresponding molecular structure (bond connectivity and 3D distance) for two random molecules: 'CC1(C)C(C)(O)C1(C)O' (top) and (bottom). The attention map (only tokens that map to
Figure 4 Visualization of the learned attention map (using either full or linear attention) under rotary embedding and corresponding molecular structure (bond connectivity and 3D distance) for two random molecules: 'CC1(C)C(C)(O)C1(C)O' (top) and (bottom). The attention map (only tokens that map toCC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 46 references

Research works printed but not linked: 20

  1. Goh, G. B., Hodas, N. O., Siegel, C. & Vishnu, A. SMILES2Vec: an interpretable general-purpose deep neural network for predicting chemical properties. Preprint at https://arxiv.org/abs/1712.02034 (2017). · arXiv
  2. Shin, B., Park, S., Kang, K. & Ho, J. C. Self-attention based molecule representation for predicting drug–target interaction. Proc. Mach. Learn. Res. 106, 230–248 (2019).
  3. Gao, W., Fu, T., Sun, J. & Coley, C. W. Sample efficiency matters: a benchmark for practical molecular optimization. In Thirty-Sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2022).
  4. Defferrard, M., Bresson, X. & Vandergheynst, P. Convolutional neural networks on graphs with fast localized spectral filtering. Adv. Neural Inf. Process. Syst. 29 (2016).
  5. Kipf, T. N. & Welling, M. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations (OpenReview.net, 2017).
  6. Li, Y., Tarlow, D., Brockschmidt, M. & Zemel, R. Gated graph sequence neural networks. In 4th International Conference on Learning Representations (eds Bengio, Y. & LeCun, Y.) (OpenReview.net, 2016).
  7. Veličković, P. et al. Graph attention networks. In International Conference on Learning Representations (2018).
  8. Hamilton, W., Ying, Z. & Leskovec, J. Inductive representation learning on large graphs. Adv. Neural Inf. Process. Syst. 30, 1025–1035 (Curran Associates Inc., 2017).
  9. Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O. & Dahl, G. E. Neural message passing for quantum chemistry. Proc. Mach. Learn Res. 70, 1263–1272 (2017).
  10. Liao, R., Zhao, Z., Urtasun, R. & Zemel, R. S. LanczosNet: multi-scale deep graph convolutional networks. In 7th International Conference on Learning Representations (OpenReview.net, 2019).
  11. Chen, P., Liu, W., Hsieh, C.-Y., Chen, G. & Zhang, S. Utilizing edge features in graph neural networks via variational information maximization. Preprint at https://arxiv.org/abs/1906.05488 (2019). · arXiv
  12. Chithrananda, S., Grand, G. & Ramsundar, B. ChemBERTa: large-scale self-supervised pretraining for molecular property prediction. Preprint at https://arxiv.org/abs/2010.09885 (2020). · arXiv
  13. Katharopoulos, A., Vyas, A., Pappas, N. & Fleuret, F. Transformers are RNNs: fast autoregressive transformers with linear attention. Proc. Mach. Learn. Res. 119, 5156–5165 (2020).
  14. Liu, S., Demirel, M. F. & Liang, Y. N-gram graph: simple unsupervised representation for graphs, with applications to molecules. In NIPS’19: Proc. 33rd International Conference on Neural Information Processing Systems (eds Wallach, H. et al.) 8464–8476 (Curran Associates, Inc., 2019).
  15. Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. Proc. Mach. Learn. Res. 119, 1597–1607 (2020).
  16. Oord, A. V. D., Li, Y. & Vinyals, O. Representation learning with contrastive predictive coding. Preprint at https://arxiv.org/abs/1807.03748 (2018). · arXiv
  17. Klicpera, J., Groß, J. & Günnemann, S. Directional message passing for molecular graphs. In 8th International Conference on Learning Representations (OpenReview.net, 2020).
  18. Ke, G., He, D. & Liu, T.-Y. Rethinking positional encoding in language pre-training. In 9th International Conference on Learning Representations (OpenReview.net, 2021).
  19. Beltagy, I., Peters, M. E. & Cohan, A. Longformer: the long-document transformer. Preprint at https://arxiv.org/abs/2004.05150 (2020). · arXiv
  20. Kitaev, N., Kaiser, L. & Levskaya, A. Reformer: the efficient transformer. In 8th International Conference on Learning Representations (OpenReview.net, 2020).

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Ross, J., Belgodere, B., Chenthamarakshan, V., Padhi, I., Mroueh, Y., & Das, P.
Affiliations
IBM (United States) (US)
Corresponding author
  • Ross, Jerret
  • Das, Payel
ORCID
Venue
Nature Machine Intelligence
Publisher
Springer Science and Business Media LLC
Published
2022-12-21
Volume
4
Issue
12
Pages
1256-1264
ISSN
2522-5839 (ISSN-L)
DOI
10.1038/s42256-022-00580-7
OpenAlex
W4313485929
Cited by
590
References
46
Citations by year
2026: 222 · 2025: 212 · 2024: 112 · 2023: 42 · 2022: 2
FWCI
49.91
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion)
  • Research Square (acceptedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
  • Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
  • Physical Sciences › Physics and Astronomy › Atomic and Molecular Physics, and Optics › Advanced Chemical Physics Studies
Keywords
molecular embeddings, Transformer encoder, rotary positional embeddings, linear attention, SMILES, molecular property prediction, quantum chemical properties, attention analysis, drug discovery, material design, self-supervised learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review