The DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning
Open accessPreprint — published as 10.1109/tpami.2021.3095381 · Paper page
Abstract
Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models taken from NLP. These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive models (Transformer-XL, XLNet) and four auto-encoder models (BERT, Albert, Electra, T5) on data from UniRef and BFD containing up to 393 billion amino acids. The LMs were trained on the Summit supercomputer using 5616 GPUs and TPU Pod up-to 1024 cores. Dimensionality reduction revealed that the raw protein LM- embeddings from unlabeled data captured some biophysical features of protein sequences. We validated the advantage of using the embeddings as exclusive input for several subsequent tasks. The first was a per-residue prediction of protein secondary structure (3-state accuracy Q3=81%-87%); the second were per-protein predictions of protein sub-cellular localization (ten-state accuracy: Q10=81%) and membrane vs. water-soluble (2-state accuracy Q2=91%). For the per-residue predictions the transfer of the most informative embeddings (ProtT5) for the first time outperformed the state-of-the-art without using evolutionary information thereby bypassing expensive database searches. Taken together, the results implied that protein LMs learned some of the grammar of the language of life . To facilitate future work, we released our models at https://github.com/agemagician/ProtTrans .
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
UniProt: a worldwide hub of protein knowledge
The UniProt Consortium (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1049 · Cited by 7,698
Amino acid substitution matrices from protein blocks.
Henikoff, S., & Henikoff, J. G. (1992) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.89.22.10915 · Cited by 6,218
MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Steinegger, M., & Söding, J. (2017) · Nature Biotechnology · doi:10.1038/nbt.3988 · Cited by 4,840
Self-citationIn-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537
Approximate nearest neighbors: towards removing the curse of dimensionality
Indyk, P., & Motwani, R. (1998) · Proceedings of the thirtieth annual ACM symposium on Theory of computing - STOC '98 · doi:10.1145/276698.276876 · Cited by 4,188
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
Prediction of Protein Secondary Structure at Better than 70% Accuracy
Rost, B., & Sander, C. (1993) · Journal of Molecular Biology · doi:10.1006/jmbi.1993.1413 · Cited by 2,910
Self-citationWhat is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153
UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches
Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B., Wu, C. H., & the UniProt Consortium (2015) · Bioinformatics · doi:10.1093/bioinformatics/btu739 · Cited by 1,928
Self-citationELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.10555 · Cited by 1,844
JPred4: a protein secondary structure prediction server
Drozdetskiy, A., Cole, C., Procter, J., & Barton, G. J. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv332 · Cited by 1,744
PISCES: a protein sequence culling server
Wang, G., & Dunbrack, R. L. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btg224 · Cited by 1,662
Combining evolutionary information and neural networks to predict protein secondary structure
Rost, B., & Sander, C. (1994) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.340190108 · Cited by 1,452
Self-citationReformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435
Improved protein structure prediction using predicted interresidue orientations
Yang, J., Anishchenko, I., Park, H., Peng, Z., Ovchinnikov, S., & Baker, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914677117 · Cited by 1,421
Big Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349
Protein 3D Structure Computed from Evolutionary Sequence Variation
Marks, D. S., Colwell, L. J., Sheridan, R., Hopf, T. A., Pagnani, A., Zecchina, R., & Sander, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028766 · Cited by 1,144
DeepLoc: prediction of protein subcellular localization using deep learning
Almagro Armenteros, J. J., Sønderby, C. K., Sønderby, S. K., Nielsen, H., & Winther, O. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx431 · Cited by 1,143
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131
Bairoch, A. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.304 · Cited by 1,129
FastText.zip: Compressing text classification models
Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., & Mikolov, T. (2016) · arXiv · doi:10.48550/arxiv.1612.03651 · Cited by 887
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719
Evaluation and improvement of multiple sequence methods for protein secondary structure prediction
Cuff, J. A., & Barton, G. J. (1999) · Proteins: Structure, Function, and Genetics · doi:10.1002/(sici)1097-0134(19990301)34:4<508::aid-prot10>3.0.co;2-4 · Cited by 667
Modeling the language of life – Deep Learning Protein Sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · doi:10.1101/614313 · Cited by 606
Self-citationModeling aspects of the language of life through transfer-learning protein sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3220-8 · Cited by 606
Self-citationA Multiscale Visualization of Attention in the Transformer Model
Vig, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3007 · Cited by 600
Studies on the Reduction and Re-formation of Protein Disulfide Bonds
Anfinsen, C. B., & Haber, E. (1961) · Journal of Biological Chemistry · doi:10.1016/s0021-9258(18)64177-8 · Cited by 595
NetSurfP‐2.0: Improved prediction of protein structural features by integrated deep learning
Klausen, M. S., Jespersen, M. C., Nielsen, H., Jensen, K. K., Jurtz, V. I., Sønderby, C. K., Sommer, M. O. A., Winther, O., Nielsen, M., Petersen, B., et al. (2019) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25674 · Cited by 574
RaptorX-Property: a web server for protein structure property prediction
Wang, S., Li, W., Liu, S., & Xu, J. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw306 · Cited by 565
Protein Secondary Structure Prediction Using Deep Convolutional Neural Fields
Wang, S., Peng, J., Ma, J., & Xu, J. (2016) · Scientific Reports · doi:10.1038/srep18962 · Cited by 563
Du, D., Tan, L., Wang, Y., Peng, B., Weinstein, J. N., Wondisford, F. E., Su, X., & Lorenzi, P. L. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-2669-9 · Cited by 549
Self-citationImproved prediction of protein secondary structure by use of sequence profiles and neural networks.
Rost, B., & Sander, C. (1993) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.90.16.7558 · Cited by 515
Self-citationThree-Dimensional Structures of Membrane Proteins from Genomic Sequencing
Hopf, T. A., Colwell, L. J., Sheridan, R., Rost, B., Sander, C., & Marks, D. S. (2012) · Cell · doi:10.1016/j.cell.2012.04.012 · Cited by 512
Self-citationProtein-level assembly increases protein sequence recovery from metagenomic samples manyfold
Steinegger, M., Mirdita, M., & Söding, J. (2019) · Nature Methods · doi:10.1038/s41592-019-0437-4 · Cited by 445
Self-citationNVIDIA cuda software and gpu parallel computing architecture
Kirk, D. (2007) · Proceedings of the 6th international symposium on Memory management · doi:10.1145/1296907.1296909 · Cited by 435
End-to-End Differentiable Learning of Protein Structure
AlQuraishi, M. (2019) · Cell Systems · doi:10.1016/j.cels.2019.03.006 · Cited by 371
Heffernan, R., Yang, Y., Paliwal, K., & Zhou, Y. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx218 · Cited by 357
The SWISS-PROT protein sequence data bank
Bairoch, A., & Boeckmann, B. (1991) · Nucleic Acids Research · doi:10.1093/nar/19.suppl.2247 · Cited by 351
Rao, R., Liu, J., Verkuil, R., Meier, J., Canny, J. F., Abbeel, P., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.02.12.430858 · Cited by 350
Protein flexibility and intrinsic disorder
Radivojac, P., Obradovic, Z., Smith, D. K., Zhu, G., Vucetic, S., Brown, C. J., Lawson, J. D., & Dunker, A. K. (2004) · Protein Science · doi:10.1110/ps.03128904 · Cited by 345
Transformer protein language models are unsupervised structure learners
Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A. (2020) · doi:10.1101/2020.12.15.422761 · Cited by 341
Attention Interpretability Across NLP TasksPreprint
Vashishth, S., Upadhyay, S., Tomar, G. S., & Faruqui, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11218 · Cited by 243
Sixty-five years of the long march in protein secondary structure prediction: the final stretch?
Yang, Y., Gao, J., Wang, J., Heffernan, R., Hanson, J., Paliwal, K., & Zhou, Y. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbw129 · Cited by 237
PredictProtein - Predicting Protein Structure and Function for 29 Years
Bernhofer, M., Dallago, C., Karl, T., Satagopam, V., Heinzinger, M., Littmann, M., Olenyi, T., Qiu, J., Schütze, K., Yachdav, G., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab354 · Cited by 233
Self-citationEpitope Mapping by Label-Free Biomolecular Interaction Analysis
Malmqvist, M. (1996) · Methods · doi:10.1006/meth.1996.0060 · Cited by 222
ProGen: Language Modeling for Protein Generation
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P.-S., & Socher, R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.07.982272 · Cited by 212
Unexpected features of the dark proteome
Perdigão, N., Heinrich, J., Stolte, C., Sabir, K. S., Buckley, M. J., Tabor, B., Signal, B., Gloss, B. S., Hammang, C. J., Rost, B., et al. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1508380112 · Cited by 208
Self-citationElrod-Erickson, M., Benson, T. E., & Pabo, C. O. (1998) · Structure · doi:10.1016/s0969-2126(98)00047-1 · Cited by 202
Embeddings from deep learning transfer GO annotations beyond homology
Littmann, M., Heinzinger, M., Dallago, C., Olenyi, T., & Rost, B. (2021) · Scientific Reports · doi:10.1038/s41598-020-80786-0 · Cited by 175
Self-citationBridging the Protein Sequence--Structure Gap by Structure Predictions
Rost, B., & Sander, C. (1996) · Annual Review of Biophysics and Biomolecular Structure · doi:10.1146/annurev.bb.25.060196.000553 · Cited by 172
Self-citationChandonia, J.-M., Fox, N. K., & Brenner, S. E. (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1134 · Cited by 165
PSI-2: Structural Genomics to Cover Protein Domain Family Space
Dessailly, B. H., Nair, R. P., Jaroszewski, L., Fajardo, J. E., Kouranov, A., Lee, D., Fiser, A., Godzik, A., Rost, B., & Orengo, C. A. (2009) · Structure · doi:10.1016/j.str.2009.03.015 · Cited by 133
Self-citationLight attention predicts protein location from the language of life
Stärk, H., Dallago, C., Heinzinger, M., & Rost, B. (2021) · Bioinformatics Advances · doi:10.1093/bioadv/vbab035 · Cited by 130
Self-citationLocTree2 predicts localization for all domains of life
Goldberg, T., Hamp, T., & Rost, B. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts390 · Cited by 125
Self-citationPromotion of virus assembly and organization by the measles virus matrix protein
Ke, Z., Strauss, J. D., Hampton, C. M., Brindley, M. A., Dillard, R. S., Leon, F., Lamb, K. M., Plemper, R. K., & Wright, E. (2018) · Nature Communications · doi:10.1038/s41467-018-04058-2 · Cited by 118
Abriata, L. A., Tamò, G. E., Monastyrskyy, B., Kryshtafovych, A., & Peraro, M. D. (2017) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25423 · Cited by 101
TMSEG: Novel prediction of transmembrane helices
Bernhofer, M., Kloppmann, E., Reeb, J., & Rost, B. (2016) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25155 · Cited by 62
Self-citationEnhanced protein domain discovery by using language modeling techniques from speech recognition
Coin, L., Bateman, A. G., & Durbin, R. M. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0737502100 · Cited by 60
Quantitative firing pattern phenotyping of hippocampal neuron types
Komendantov, A. O., Venkadesh, S., Rees, C. L., Wheeler, D. W., Hamilton, D. J., & Ascoli, G. A. (2019) · Scientific Reports · doi:10.1038/s41598-019-52611-w · Cited by 57
Accurate contact predictions using covariation techniques and machine learning
Kościółek, T., & Jones, D. T. (2015) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.24863 · Cited by 55
Transforming the Language of Life: Transformer Neural Networks for Protein Prediction TasksPreprint
Nambiar, A., Liu, S., Hopkins, M., Heflin, M., Maslov, S., & Ritz, A. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.06.15.153643 · Cited by 54
Pitfalls of protein sequence analysis
Rost, B., & Valencia, A. (1996) · Current Opinion in Biotechnology · doi:10.1016/s0958-1669(96)80124-8 · Cited by 44
Self-citationHydrophobic forces and the length limit of foldable protein domains
Lin, M. M., & Zewail, A. H. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1207382109 · Cited by 40
Pred‐MutHTP: Prediction of disease‐causing and neutral mutations in human transmembrane proteins
Kulandaisamy, A., Zaucha, J. M., Sakthivel, R., Frishman, D., & Gromiha, M. M. (2019) · Human Mutation · doi:10.1002/humu.23961 · Cited by 30
Evolutionary couplings and sequence variation effect predict protein binding sites
Littmann, M., Hopf, T. A., & Rost, B. (2018) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25585 · Cited by 27
Self-citationLanguage modelling for biological sequences – curated datasets and baselinesPreprint
Armenteros, J. J. A., Johansen, A. R., Winther, O., & Nielsen, H. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.09.983585 · Cited by 19
Dark Proteins Important for Cellular Function
Schafferhans, A., O'Donoghue, S., Heinzinger, M., & Rost, B. (2018) · Proteomics · doi:10.1002/pmic.201800227 · Cited by 10
Self-citationLarge Batch Optimization for Deep Learning: Training BERT in 76 minutes
You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., & Hsieh, C.-J. (2019) · arXiv · doi:10.48550/arxiv.1904.00962
Cited by
SignalP 6.0 predicts all five types of signal peptides using protein language models
Teufel, F., Almagro Armenteros, J. J., Johansen, A. R., Gíslason, M. H., Pihl, S. I., Tsirigos, K. D., Winther, O., Brunak, S., von Heijne, G., & Nielsen, H. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01156-3 · Cited by 2,867
A guide to machine learning for biologists
Greener, J. G., Kandathil, S. M., Moffat, L., & Jones, D. T. (2022) · Nature Reviews Molecular Cell Biology · doi:10.1038/s41580-021-00407-0 · Cited by 2,127
Scientific discovery in the age of artificial intelligence.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06221-2 · Cited by 1,823
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Self-citationWang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
Self-citation
Metadata
- Authors
- Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., Bhowmik, D., & Rost, B.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2020-07-12
- OpenAlex
- W3040739508
- Cited by
- 354
- References
- 75
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.