CoChat
Sign Up
PreprintCited by 3,280Machine Learning in BioinformaticsProtein Structure and DynamicsRNA and protein synthesis mechanisms

The DOI matches its Crossref record; 148 of the 148 references it cites were found in a registry or an index.

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Open access
Rives, Alexander, Meier, Joshua, Sercu, Tom, Goyal, Siddharth, Lin, Zeming, Liu, Jason, Guo, Demi, Ott, Myle, Zitnick, C. Lawrence, Ma, Jerry, Fergus, Rob·2019·Proceedings of the National Academy of Sciences
DOI 10.1101/622803PMC PMC8053943

Preprint — published as 10.1073/pnas.2016239118 · Paper page

Open this paper in CoChat — free account

Abstract

In the field of artificial intelligence, a combination of scale in data and model capacity enabled by un-supervised learning has led to major advances in representation learning and statistical generation. In the life sciences, the anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To this end we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250 million protein sequences spanning evolutionary diversity. The resulting model contains information about biological properties in its representations. The representations are learned from sequence data alone. The learned representation space has a multi-scale organization reflecting structure from the level of biochemical properties of amino acids to remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and can be identified by linear projections. Representation learning produces features that generalize across a range of applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure, and improving state-of-the-art features for long-range contact prediction.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/622803

Cites

148 of 148 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. Biochemical properties of amino acids are represented in the Transformer model’s output embeddings, visualized here with t-SNE. Through unsupervised learning, residues are clustered into hydrophobic, polar, and aromatic groups and reflect overall organization by molecular weight and charge. Visualization of 36-layer Transformer trained on UniParc.
Fig. 1. Biochemical properties of amino acids are represented in the Transformer model’s output embeddings, visualized here with t-SNE. Through unsupervised learning, residues are clustered into hydrophobic, polar, and aromatic groups and reflect overall organization by molecular weight and charge. Visualization of 36-layer Transformer trained on UniParc.CC BY-NC-ND · PubMed Central
Fig. 2. Protein sequence representations encode and organize biological variations. (A) Each point represents a gene, and each gene is colored by the orthologous group it belongs to (dimensionality is reduced by t-SNE). Orthologous groups of genes are densely clustered in the trained representation space. By contrast, the untrained representation space and unigram representations do not reflect strong organization by evolutionary relationships. (B) Genes corresponding to a common biological variation are related linearly in the trained representation space. Genes are colored by their orthologous group, and their species are indicated by a character label. PCA recovers a species axis (horizontal) and orthology axis (vertical) in the trained representation space but not in the untrained or unigram spaces. Representations are from the 36-layer Transformer model trained on UniParc.
Fig. 2. Protein sequence representations encode and organize biological variations. (A) Each point represents a gene, and each gene is colored by the orthologous group it belongs to (dimensionality is reduced by t-SNE). Orthologous groups of genes are densely clustered in the trained representation space. By contrast, the untrained representation space and unigram representations do not reflect strong organization by evolutionary relationships. (B) Genes corresponding to a common biological variation are related linearly in the trained representation space. Genes are colored by their orthologous group, and their species are indicated by a character label. PCA recovers a species axis (horizontal) and orthology axis (vertical) in the trained representation space but not in the untrained or unigram spaces. Representations are from the 36-layer Transformer model trained on UniParc.CC BY-NC-ND · PubMed Central
Fig. 3. Final representations from trained models implicitly align sequences. Cosine similarity distributions are depicted for the final representations of residues from sequences within Pfam family PF01010. The differences between the aligned (dark blue) and unaligned (light blue) distributions imply that the trained Transformer representations are a powerful discriminator between aligned and unaligned positions in the sequences. In contrast, representations prior to training do not separate the aligned (dark red) and unaligned positions (light red). (A) Overall distribution; distribution under constraint that residue pairs have (B) same amino acid identity; or (C) different amino acid identities. AUCs across 128 Pfam families are reported in SI Appendix, Table S1.
Fig. 3. Final representations from trained models implicitly align sequences. Cosine similarity distributions are depicted for the final representations of residues from sequences within Pfam family PF01010. The differences between the aligned (dark blue) and unaligned (light blue) distributions imply that the trained Transformer representations are a powerful discriminator between aligned and unaligned positions in the sequences. In contrast, representations prior to training do not separate the aligned (dark red) and unaligned positions (light red). (A) Overall distribution; distribution under constraint that residue pairs have (B) same amino acid identity; or (C) different amino acid identities. AUCs across 128 Pfam families are reported in SI Appendix, Table S1.CC BY-NC-ND · PubMed Central
Fig. 4. Secondary structure (linear projections). Example predictions for held-out folds. Unsupervised pretraining encodes secondary structure into representations. Following pretraining, linear projections recover secondary structure (left column). Without pretraining, little information is recovered (right column). (A) d1nt4a_ Phosphoglycerate mutase-like fold; (B) d3wr7a_ Acyl-CoA N-acyltransferases fold. Colors indicate secondary structure class identified by the projection: helix (red), strand (green), and coil (blue). Color intensities indicate confidence. Representations from ESM-1b Transformer are used.
Fig. 4. Secondary structure (linear projections). Example predictions for held-out folds. Unsupervised pretraining encodes secondary structure into representations. Following pretraining, linear projections recover secondary structure (left column). Without pretraining, little information is recovered (right column). (A) d1nt4a_ Phosphoglycerate mutase-like fold; (B) d3wr7a_ Acyl-CoA N-acyltransferases fold. Colors indicate secondary structure class identified by the projection: helix (red), strand (green), and coil (blue). Color intensities indicate confidence. Representations from ESM-1b Transformer are used.CC BY-NC-ND · PubMed Central
Fig. 5. Residue–residue contacts (linear projections). (Left) Top-L predictions for fold level held-out example d1n3ya_, with vWA-like fold. True positives in blue, false positives in red, superimposed on ground truth contact map in gray. ESM-1b Transformer projections below the diagonal, CCMpred predictions above the diagonal. (Right) Precision distribution (top-L long-range) comparing ESM-1b projections with CCMpred across all domains in the five test partitions with structural holdout at the fold level. Visualized domain marked by ×.
Fig. 5. Residue–residue contacts (linear projections). (Left) Top-L predictions for fold level held-out example d1n3ya_, with vWA-like fold. True positives in blue, false positives in red, superimposed on ground truth contact map in gray. ESM-1b Transformer projections below the diagonal, CCMpred predictions above the diagonal. (Right) Precision distribution (top-L long-range) comparing ESM-1b projections with CCMpred across all domains in the five test partitions with structural holdout at the fold level. Visualized domain marked by ×.CC BY-NC-ND · PubMed Central
Fig. 6. Relationship between the language modeling objective and structure learning. Eight-class secondary structure prediction accuracy (Left) and contact prediction top-L long-range precision (Right) both as a function of pretraining ECE. Performance is evaluated on held-out folds. Linear projections are fit using model checkpoints over the course of pretraining on UR50/S. The linear relationship for each model indicates that for a given model and pretraining dataset, the language modeling ECE is a good proxy for the structural content of the representations. Improvement of the model’s ECE leads to an increase in information about structure. This establishes a link between the language modeling objective and unsupervised structure learning.
Fig. 6. Relationship between the language modeling objective and structure learning. Eight-class secondary structure prediction accuracy (Left) and contact prediction top-L long-range precision (Right) both as a function of pretraining ECE. Performance is evaluated on held-out folds. Linear projections are fit using model checkpoints over the course of pretraining on UR50/S. The linear relationship for each model indicates that for a given model and pretraining dataset, the language modeling ECE is a good proxy for the structural content of the representations. Improvement of the model’s ECE leads to an increase in information about structure. This establishes a link between the language modeling objective and unsupervised structure learning.CC BY-NC-ND · PubMed Central
Fig. 7. Representation learning enables state-of-the-art supervised prediction of the quantitative effect of mutations. (Left) Envision dataset (65). (Right) DeepSequence dataset (26). Transformer representations (34-layer, UR50/S) are compared to the LSTM bidirectional language model (large model, UR50/S). The result of fivefold cross validation is reported for each protein. For each partition, supervised fine-tuning is performed on 80% of the mutational data for the protein, and results are evaluated on the remaining 20%. Transformer representations outperform baseline LSTM representations on both datasets. State-of-the-art methods are also shown for each dataset. Gray et al. (65) is a supervised method using structural, evolutionary, and biochemical features, trained with the same protocol as used for the Transformer. Riesselman et al. (26) is an unsupervised method trained on the MSA of each protein. Mean and SD across the five partitions for Transformer model and LSTM baseline.
Fig. 7. Representation learning enables state-of-the-art supervised prediction of the quantitative effect of mutations. (Left) Envision dataset (65). (Right) DeepSequence dataset (26). Transformer representations (34-layer, UR50/S) are compared to the LSTM bidirectional language model (large model, UR50/S). The result of fivefold cross validation is reported for each protein. For each partition, supervised fine-tuning is performed on 80% of the mutational data for the protein, and results are evaluated on the remaining 20%. Transformer representations outperform baseline LSTM representations on both datasets. State-of-the-art methods are also shown for each dataset. Gray et al. (65) is a supervised method using structural, evolutionary, and biochemical features, trained with the same protocol as used for the Transformer. Riesselman et al. (26) is an unsupervised method trained on the MSA of each protein. Mean and SD across the five partitions for Transformer model and LSTM baseline.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

  • Attention Is All You Need

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×

    “We investigate the Transformer (Vaswani et al., 2017), which has emerged as a powerful general-purpose model architecture for representation learning and generative modeling of textual data, and outperforms more traditionally employed recurrent and convolutional architectures, such as LSTM and GConv networks (Hochreit…”
    Method

  • Long Short-Term Memory

    Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318

  • Basic local alignment search tool

    Altschul, S. F., Gish, W., Miller, W., Myers, E. W., & Lipman, D. J. (1990) · Journal of Molecular Biology · doi:10.1006/jmbi.1990.9999 · Cited by 98,669

  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 5×

    “…a form of unsupervised learning in which context within the text is used to predict missing words, have been shown to materialize representations of word meaning that can generalize across problems (Collobert & Weston, 2008; Dai & Le, 2015; Peters et al., 2018; Devlin et al., 2018).”
    BackgroundMethodInfluential

  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 5×

    “…a form of unsupervised learning in which context within the text is used to predict missing words, have been shown to materialize representations of word meaning that can generalize across problems (Collobert & Weston, 2008; Dai & Le, 2015; Peters et al., 2018; Devlin et al., 2018).”
    BackgroundMethodInfluential

  • Glove: Global Vectors for Word Representation

    Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×

    “A.1 B ACKGROUND ON LANGUAGE MODELS AND EMBEDDINGS Deep language models are parametric functions that map sequences into distributed word representations in a learned vector space (Bengio et al., 2003; Mikolov et al., 2013a;b; Pennington et al., 2014; Lample et al., 2017).”
    Background

  • Untitled work

    (2008) · Journal of Machine Learning Research · Cited by 33,051

  • Efficient Estimation of Word Representations in Vector Space

    Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×

    “A.1 B ACKGROUND ON LANGUAGE MODELS AND EMBEDDINGS Deep language models are parametric functions that map sequences into distributed word representations in a learned vector space (Bengio et al., 2003; Mikolov et al., 2013a;b; Pennington et al., 2014; Lample et al., 2017).”
    Background

  • Distributed Representations of Words and Phrases and their Compositionality

    Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×

    “A.1 B ACKGROUND ON LANGUAGE MODELS AND EMBEDDINGS Deep language models are parametric functions that map sequences into distributed word representations in a learned vector space (Bengio et al., 2003; Mikolov et al., 2013a;b; Pennington et al., 2014; Lample et al., 2017).”
    Background

  • Reducing the dimensionality of data with neural networks

    Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928

Show all 100 references

Research works printed but not linked: 24

  1. 10.1103/PhysRevE.87 · DOI
  2. Mikolov, T. , Sutskever, I. , Deoras, A. , Le, H.-S. , Kom-brink, S. , and Cernocky, J. Subword language modeling with neural networks. preprint (http://www.fit.vutbr.cz/imikolov/rnnlm/char.pdf), 8, 2012.
  3. Riesselman, A. J. , Shin, J.-E. , Kollasch, A. W. , McMahon, C. , Simon, E. , Sander, C. , Manglik, A. , Kruse, A. C. , and Marks, D. S. Accelerating protein design using autoregressive generative models. bioRxiv, pp. 757252, 2019.
  4. 2011, Learning generative models for protein fold families, Proteins: Structure, Function, and Bioinformatics, 79, 1061, 10.1002/prot.22934 · DOI · Paper page
  5. 10.1093/nar/gkt1223 · DOI · Paper page
  6. Devlin, J. , Chang, M. , Lee, K. , and Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018b. URL http://arxiv.org/abs/1810.04805. · arXiv
  7. 2013, Improved contact prediction in proteins: using pseu-dolikelihoods to infer potts models, Physical Review E, 87, 012707, 10.1103/PhysRevE.87.012707 · DOI · Paper page
  8. 10.1093/nar/gkt1240 · DOI · Paper page
  9. 2018, Quantitative missense variant effect pre-diction using large-scale mutagenesis data, Cell systems, 6, 116, 10.1016/j.cels.2017.11.003 · DOI
  10. 10.1093/nar/gky1085 · DOI
  11. Johnson, J. , Douze, M. , and Jégou, H. Billion-scale similarity search with gpus. CoRR, abs/1702.08734, 2017. URL http://arxiv.org/abs/1702.08734. · arXiv
  12. 2019, NetSurfP-2.0: Improved prediction of protein structural features by integrated deep learning, Proteins, 87, 520, 10.1002/prot.25674 · DOI
  13. Liu, Y. , Ott, M. , Goyal, N. , Du, J. , Joshi, M. , Chen, D. , Levy, O. , Lewis, M. , Zettlemoyer, L. , and Stoyanov, V. RoBERTa: A Robustly Optimized BERT Pretraining Approach. 7 2019. URL http://arxiv.org/abs/1907.11692. · arXiv
  14. 10.1093/nar/gkw1081 · DOI
  15. 10.1038/nmeth.1818 · DOI · Paper page
  16. 10.1038/s41592-018-0138-4 · DOI · Paper page
  17. 10.1093/bioinformatics/btu500 · DOI
  18. 10.1016/j.sbi.2011.03.005 · DOI
  19. 10.1093/bioinformatics/btu739 · DOI · Paper page
  20. 10.1093/nar/gkm895 · DOI
  21. 2008, Graphical models of residue coupling in protein families, IEEE/ACM Transactions on Computational Biology and Bioinformatics, 5, 183, 10.1109/TCBB.2007.70225 · DOI · Paper page
  22. Vaswani, A. , Shazeer, N. , Parmar, N. , Uszkoreit, J. , Jones, L. , Gomez, A. N. , Kaiser, Ł. , and Polosukhin, I. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017b.
  23. 2017, Accurate de novo prediction of protein contact map by ultra-deep learning model, PLoS computational biology, 13, e1005324, 10.1371/journal.pcbi.1005324 · DOI · Paper page
  24. 10.1073/pnas.0805923106 · DOI · Paper page

Cited by

Sorted by citations

See all 3,280

Related

Metadata

Authors
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., & Fergus, R.
Affiliations
  • Meta (United States) (US)
  • New York University (US)
  • Harvard University (US)
  • Yale University (US)
Corresponding author
Rives, Alexander
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
openRxiv
Published
2019-04-29
Volume
118
Issue
15
ISSN
0027-8424 (ISSN-L)
DOI
10.1101/622803
PMC
PMC8053943
OpenAlex
W3146944767
Cited by
3,280
References
100
Citations by year
2026: 780 · 2025: 816 · 2024: 697 · 2023: 448 · 2022: 305 · 2021: 161 · 2020: 51 · 2019: 18
FWCI
126.76
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Knowledge@UChicago (University of Chicago) (submittedVersion, cc-by-nc-nd)
  • University of Chicago
Funders
National Science Foundation (1339362)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
Keywords
protein language models, unsupervised learning, protein evolution, protein structure prediction, secondary structure prediction, protein tertiary structure, remote homology
MeSH terms
Unsupervised Machine Learning, Amino Acids, Protein Conformation, Sequence Homology, Amino Acid, Sequence Analysis, Protein

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review