Transformer-based deep learning for predicting protein properties in the life sciences
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Recent developments in deep learning, coupled with an increasing number of sequenced proteins, have led to a breakthrough in life science applications, in particular in protein property prediction. There is hope that deep learning can close the gap between the number of sequenced proteins and proteins with known properties based on lab experiments. Language models from the field of natural language processing have gained popularity for protein property predictions and have led to a new computational revolution in biology, where old prediction results are being improved regularly. Such models can learn useful multipurpose representations of proteins from large open repositories of protein sequences and can be used, for instance, to predict protein properties. The field of natural language processing is growing quickly because of developments in a class of models based on a particular model—the Transformer model. We review recent developments and the use of large-scale Transformer models in applications for predicting protein characteristics and how such models can be used to predict, for example, post-translational modifications. We review shortcomings of other deep learning models and explain how the Transformer models have quickly proven to be a very promising way to unravel information hidden in the sequences of amino acids.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Common RNN models include the long short-term memory (LSTM) model ( Hochreiter and Schmidhuber, 1997 ) and the gated recurrent units (GRUs) ( Chung et al., 2014 ).”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Principal component analysis: A review and recent developments
Jolliffe, I. T., & Cadima, J. (2016) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2015.0202 · Cited by 10,117
ColabFold - Making protein folding accessible to all
Mirdita, M., Schütze, K., Moriwaki, Y., Heo, L., Ovchinnikov, S., & Steinegger, M. (2021) · doi:10.1101/2021.08.15.456425 · Cited by 9,391
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542 · cited 2×
“They also suffer from some fundamental problems (called the vanishing and exploding gradient problems) ( Bengio et al., 1994 ; Pascanu et al., 2013 ; Hanin, 2018 ), which makes them difficult to train ( Dai et al., 2018 ).”
BackgroundVaradi, M., Anyango, S., Deshpande, M., Nair, S., Natassia, C., Yordanova, G., Yuan, D., Stroe, O., Wood, G., Laydon, A., et al. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkab1061 · Cited by 7,850
UniProt: a worldwide hub of protein knowledge
The UniProt Consortium (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1049 · Cited by 7,698
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
UniProt: the universal protein knowledgebase in 2021
The UniProt Consortium, Bateman, A., Martin, M.-J., Orchard, S., Magrane, M., Agivetova, R., Ahmad, S., Alpi, E., Bowler-Barnett, E. H., Britto, R., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkaa1100 · Cited by 6,372 · cited 1×
“The largest open sources of protein sequence information and data are the Universal Protein Resource (UniProt) ( UniProt, 2021 ), Pfam ( Mistry et al., 2021 ), and the Big Fantastic Data-base (BFD) ( BFD, 2022 ).”
BackgroundHMMER web server: interactive sequence similarity searching
Finn, R. D., Clements, J., & Eddy, S. R. (2011) · Nucleic Acids Research · doi:10.1093/nar/gkr367 · Cited by 5,952
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“To generate the input embeddings to the model, the authors used two schemes: encoding the sentences using byte-pair encoding ( Britz et al., 2017 ) or splitting tokens into word-piece vocabulary ( Wu et al., 2016 ), based on the training dataset.”
MethodMMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Steinegger, M., & Söding, J. (2017) · Nature Biotechnology · doi:10.1038/nbt.3988 · Cited by 4,840 · cited 1×
“The conventional approach for homology prediction has been to use MSAs, where computational tools such as MMseqs2 ( Steinegger and Söding, 2017 ), Pfam profile ( ElGebali et al., 2019 ), and PSI-BLAST ( Altschul et al., 1997 ) align evolutionary related protein positions by deducing conserved…”
The Pfam protein families database in 2019
El-Gebali, S., Mistry, J., Bateman, A., Eddy, S. R., Luciani, A., Potter, S. C., Qureshi, M., Richardson, L. J., Salazar, G. A., Smart, A., et al. (2019) · Nucleic Acids Research · doi:10.1093/nar/gky995 · Cited by 4,602 · cited 1×
“…approach for homology prediction has been to use MSAs, where computational tools such as MMseqs2 ( Steinegger and Söding, 2017 ), Pfam profile ( ElGebali et al., 2019 ), and PSI-BLAST ( Altschul et al., 1997 ) align evolutionary related protein positions by deducing conserved sequence patterns…”
MethodChEMBL: a large-scale bioactivity database for drug discovery
Gaulton, A., Bellis, L. J., Bento, A. P., Chambers, J., Davies, M., Hersey, A., Light, Y., McGlinchey, S., Michalovich, D., Al-Lazikani, B., et al. (2012) · Nucleic Acids Research · doi:10.1093/nar/gkr777 · Cited by 4,382
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
DrugBank: a comprehensive resource for in silico drug discovery and exploration
Wishart, D. S. (2006) · Nucleic Acids Research · doi:10.1093/nar/gkj067 · Cited by 3,892
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 9×
“Mutation prediction was also among one of the tasks in Rives et al., 2021 and Rao et al., 2019 to verify the potential of their new pre-trained Transformer models.”
BackgroundMethodResultInfluentialTransformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 1×
“Recent developments have attempted to overcome the fixed-length issue, where, for instance, some variants allow hidden states from previous fragments to be used as inputs for the current fragment ( Elnaggar et al., 2020a ; Dai et al., 2019 ).”
BackgroundRecent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
Comprehensive analysis of kinase inhibitor selectivity
Davis, M. I., Hunt, J. P., Herrgard, S., Ciceri, P., Wodicka, L. M., Pallares, G., Hocker, M., Treiber, D. K., & Zarrinkar, P. P. (2011) · Nature Biotechnology · doi:10.1038/nbt.1990 · Cited by 2,409 · cited 1×
“They utilized multiple datasets ( Tang et al., 2014 ; Davis et al., 2011 ; Liu et al., 2007 ) to evaluate their method.”
MethodProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · cited 11×
“…tasks of secondary structure prediction ( Elnaggar et al., 2020a ; Rives et al., 2021 ; Brandes et al., 2021 ; Rao et al., 2021 ; Rao et al., 2019 ; Elnaggar et al., 2020b ; Sturmfels et al., 2020 ) and contact prediction Rives et al., 2021 ; Rao et al., 2021 ; He et al., 2021 ; Sturmfels et al.,…”
BackgroundMethodInfluentialReformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435 · cited 1×
“Moreover, the memory requirements and the complexity of Transformer models was also addressed by Kitaev et al., 2020 , who introduced two techniques: (1) they replaced the dot-product attention with a locality-sensitive hashing which deals with a subset of nearest neighbours in high-dimensional…”
MethodProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009
Transformer Interpretability Beyond Attention Visualization
Chefer, H., Gur, S., & Wolf, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00084 · Cited by 903
Advances in protein structure prediction and design
Kuhlman, B., & Bradley, P. (2019) · Nature Reviews Molecular Cell Biology · doi:10.1038/s41580-019-0163-x · Cited by 806 · cited 1×
“…of proteins, such as their 3D structure, the mapping from a sequence of amino acids to the corresponding 3D structure is quite challenging ( Kuhlman and Bradley, 2019 ; Jiang et al., 2017 ), but there is typically an abundance of sequenced proteins openly available that a DL model can make…”
BackgroundProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · cited 4×
“Mutation prediction was also among one of the tasks in Rives et al., 2021 and Rao et al., 2019 to verify the potential of their new pre-trained Transformer models.”
BackgroundMethodResultInfluentialEfficient large-scale language model training on GPU clusters using megatron-LM
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. (2021) · Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis · doi:10.1145/3458817.3476209 · Cited by 639 · cited 1×
“…GPT-3 model with 175 billion model parameters ( Brown et al., 2020 ) or Microsoft and Nvidia’s Megatron-Turing NLG model with 530 billion model parameters ( Smith et al., 2022 ), but the very latest models have over a trillion model parameters ( Fedus et al., 2021 ; Narayanan et al., 2021 ).”
MethodModeling the language of life – Deep Learning Protein Sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · doi:10.1101/614313 · Cited by 606
A Multiscale Visualization of Attention in the Transformer Model
Vig, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3007 · Cited by 600 · cited 1×
“Moreover, the intricacy in the existing ways of interpreting Transformer models, especially its multihead attention, is also being improved so that model’s internal learnings can be made more easy for analysis ( Hao et al., 2021 ; Vig, 2019b ).”
BackgroundCritical assessment of methods of protein structure prediction (CASP)—Round XIII
Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., & Moult, J. (2019) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25823 · Cited by 506
Massive Exploration of Neural Machine Translation Architectures
Britz, D., Goldie, A., Luong, M.-T., & Le, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.03906 · Cited by 487
Learning the protein language: Evolution, structure, and function
Bepler, T., & Berger, B. (2021) · Cell Systems · doi:10.1016/j.cels.2021.05.017 · Cited by 483 · cited 1×
“…model representations are better in general compared, for instance, to MSAs but does show that this is a promising avenue to be explored further since there are recurrent evolutionary relations captured in the representations from such type of language models ( Bepler and Berger, 2021 ).”
BackgroundSingle-sequence protein structure prediction using language models from deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Rochereau, C., Church, G. M., Sorger, P. K., & AlQuraishi, M. (2021) · doi:10.1101/2021.08.02.454840 · Cited by 455
Single-sequence protein structure prediction using a language model and deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01432-w · Cited by 455
Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold
Steinegger, M., Mirdita, M., & Söding, J. (2019) · Nature Methods · doi:10.1038/s41592-019-0437-4 · Cited by 445 · cited 1×
“It comprises 65.9M families covering more than 2B protein sequences and was built using UniProt, and a reference protein catalogue ( Steinegger et al., 2019 ), clustered to 30% sequence identity.”
MethodA comparison of word embeddings for the biomedical natural language processing
Wang, Y., Liu, S., Afzal, N., Rastegar-Mojarad, M., Wang, L., Shen, F., Kingsbury, P., & Liu, H. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.09.008 · Cited by 370
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338
Using deep learning to annotate the protein universe
Bileschi, M. L., Belanger, D., Bryant, D. H., Sanderson, T., Carter, B., Sculley, D., Bateman, A., DePristo, M. A., & Colwell, L. J. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01179-w · Cited by 332 · cited 1×
“Since recent large-scale DL models have played a crucial role in computational protein property prediction ( Bileschi et al., 2022 ), we describe in this review the most common DL architectures in use today.”
BackgroundProtein-Protein Docking: From Interaction to Interactome
Vakser, I. A. (2014) · Biophysical Journal · doi:10.1016/j.bpj.2014.08.033 · Cited by 318
Theoretical Limitations of Self-Attention in Neural Sequence Models
Hahn, M. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00306 · Cited by 165 · cited 1×
“Work by Hahn, 2020 also showed that Transformers had problems to accurately evaluate logical formulas and to model hierarchical structures.”
BackgroundLanguage Models with Transformers
Wang, C., Li, M., & Smola, A. J. (2019) · arXiv · doi:10.48550/arxiv.1904.09408 · Cited by 161 · cited 3×
“Also, since the data is processed one token at a time, it is not possible to parallelize the computations, making the training slow ( Wang et al., 2019 ).”
BackgroundMethodInfluentialConvolutions are competitive with transformers for protein sequence pretraining
Yang, K. K., Fusi, N., & Lu, A. X. (2024) · Cell Systems · doi:10.1016/j.cels.2024.01.008 · Cited by 137
Can AlphaFold2 predict protein-peptide complex structures accurately?
Ko, J., & Lee, J. (2021) · doi:10.1101/2021.07.27.453972 · Cited by 83
Research works printed but not linked: 112
- Alanis-Lobato, 2016, HIPPIE v2. 0: enhancing meaningfulness and reliability of protein–protein interaction networks, Nucleic Acids Research, 45, D408, 10.1093/nar/gkw985 · DOI · Paper page
- Albawi S, Mohammed TA, Al-Zawi S. 2017. Understanding of a convolutional neural network. 2017 International Conference on Engineering and Technology (ICET. · DOI · Paper page
- Altschul, 1997, Gapped blast and PSI-BLAST: a new generation of protein database search programs, Nucleic Acids Research, 25, 3389, 10.1093/nar/25.17.3389 · DOI · Paper page
- Ammari MG, Gresham CR, McCarthy FM, Nanduri B. 2016. HPIDB 2.0: a curated database for host–pathogen interactions. Database.
- Behjati, 2022, Protein sequence profile prediction using protalbert transformer, Computational Biology and Chemistry, 99, 10.1016/j.compbiolchem.2022.107717 · DOI
- Bergstra J, Yamins D, Cox D. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. International conference on machine learning; 2013: PMLR.
- Bulcun, 2012, Disorders of glucose metabolism and insulin resistance in patients with obstructive sleep apnoea syndrome, International Journal of Clinical Practice, 66, 91, 10.1111/j.1742-1241.2011.02795.x · DOI
- Cai, 2021, MSA-regularized protein sequence transformer toward predicting genome-wide chemical-protein interactions: application to gpcrome deorphanization, Journal of Chemical Information and Modeling, 61, 1570, 10.1021/acs.jcim.0c01285 · DOI
- Chan, 2015, Glass: a comprehensive database for experimentally validated GPCR-ligand associations, Bioinformatics, 31, 3035, 10.1093/bioinformatics/btv302 · DOI · Paper page
- Chandra, 2019, Bigram-PGK: phosphoglycerylation prediction using the technique of bigram probabilities of position specific scoring matrix, BMC Molecular and Cell Biology, 20, 10.1186/s12860-019-0240-1 · DOI
- Chandra, 2020, RAM-PGK: prediction of lysine phosphoglycerylation based on residue adjacency matrix, Genes, 11, 10.3390/genes11121524 · DOI
- Charte, 2019, A snapshot on nonstandard supervised learning problems: taxonomy, relationships, problem transformations and algorithm adaptations, Progress in Artificial Intelligence, 8, 1, 10.1007/s13748-018-00167-7 · DOI
- Chen, 2002, The binding database: data management and interface design, Bioinformatics, 18, 130, 10.1093/bioinformatics/18.1.130 · DOI · Paper page
- Chen, 2020, IStable 2.0: predicting protein thermal stability changes by integrating various characteristic modules, Computational and Structural Biotechnology Journal, 18, 622, 10.1016/j.csbj.2020.02.021 · DOI
- Cheng, 2021, BERTMHC: improved MHC-peptide class II interaction prediction with transformer and multiple instance learning, Bioinformatics, 37, 4172, 10.1093/bioinformatics/btab422 · DOI · Paper page
- Choromanski K, Likhosherstov V, Dohan D, Song X, Gane A, Sarlos T. 2020. Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers. arXiv. https://arxiv.org/abs/2006.03555. · arXiv
- Chou, 1995, Prediction of protein structural classes, Critical Reviews in Biochemistry and Molecular Biology, 30, 275, 10.3109/10409239509083488 · DOI · Paper page
- Chou, 2020, Progresses in predicting post-translational modification, International Journal of Peptide Research and Therapeutics, 26, 873, 10.1007/s10989-019-09893-5 · DOI
- Clark K, Luong MT, Le QV. 2020. ELECTRA: Pre-Training Text Encoders as Discriminators Rather Than Generators. arXiv. https://arxiv.org/abs/2003.10555. · arXiv
- Cortés, 2011, Predicting protein distance maps according to physicochemical properties, Journal of Integrative Bioinformatics, 8, 158, 10.1515/jib-2011-181 · doi:10.1515/jib-2011-181
- Dai Z, Yang Z, Yang Y, Cohen WW, Carbonell J, Le QV. 2018. Transformer-xl: language modeling with longer-term dependency. Transformer-Xl.
- Dehghani M, Gouws S, Vinyals O, Uszkoreit J, Kaiser Ł. 2018. Universal Transformers. arXiv. https://arxiv.org/abs/1807.03819. · arXiv
- Dick, 2018, Reciprocal perspective for improved protein-protein interaction prediction, Scientific Reports, 8, 1, 10.1038/s41598-018-30044-1 · DOI
- Dodge, 1998, The HSSP database of protein structure-sequence alignments and family profiles, Nucleic Acids Research, 26, 313, 10.1093/nar/26.1.313 · DOI · Paper page
- Du, 2021, The trrosetta server for fast and accurate protein structure prediction, Nature Protocols, 16, 5634, 10.1038/s41596-021-00628-9 · DOI · Paper page
- Ehrenberger, 2015, Computational prediction of protein-protein interactions, Methods in Molecular Biology, 1278, 57, 10.1007/978-1-4939-2425-7_4 · DOI
- ElAbd, 2020, Amino acid encoding for deep learning applications, BMC Bioinformatics, 21, 10.1186/s12859-020-03546-x · DOI · Paper page
- Elman, 1990, Finding structure in time, Cognitive Science, 14, 179, 10.1207/s15516709cog1402_1 · DOI · Paper page
- Elnaggar A, Heinzinger M, Dallago C, Rehawi G, Wang Y, Jones L, Gibbs T, Feher T, Angerer C, Steinegger M, Bhowmik D, Rost B. 2020b. ProtTrans: Towards Cracking the Language of Life’s Code through Self-Supervised Deep Learning and High Performance Computing. arXiv. https://arxiv.org/abs/2007.06225. · arXiv
- Fang, 2020, A critical review of five machine learning-based algorithms for predicting protein stability changes upon mutation, Briefings in Bioinformatics, 21, 1285, 10.1093/bib/bbz071 · DOI · Paper page
- Fedus W, Zoph B, Shazeer N. 2021. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv. https://arxiv.org/abs/2101.03961. · arXiv
- Golubchik, 2007, Mind the gaps: evidence of bias in estimates of multiple sequence alignments, Molecular Biology and Evolution, 24, 2433, 10.1093/molbev/msm176 · DOI
- Google Scholar. 2022. Google Scholar. https://scholar.google.com/scholar?q=transformer+language+model+transformer+language+model&hl=en&as_sdt=0,5 [Accessed July 1, 2022].
- Gromiha MM, Nagarajan R, Selvaraj S. 2019. Protein structural bioinformatics: an overview. Protein Structural Bioinformatics.
- Guerler, 2013, Mapping monomeric threading to protein-protein structure prediction, Journal of Chemical Information and Modeling, 53, 717, 10.1021/ci300579r · DOI · Paper page
- Hanin B. 2018. Which neural net architectures give rise to exploding and vanishing gradients?. Advances in Neural Information Processing Systems.
- Hao Y, Dong L, Wei F, Xu K. 2021. Self-attention attribution: interpreting information interactions inside transformer. Proceedings of the AAAI Conference on Artificial Intelligence. · DOI · Paper page
- He L, Zhang S, Wu L, Xia H, Ju F, Zhang H. 2021. Pre-Training Co-Evolutionary Protein Representation via A Pairwise Masked Language Model. arXiv. https://arxiv.org/abs/2110.15527. · arXiv
- Heinzinger, 2021, Contrastive Learning on Protein Embeddings Enlightens Midnight Zone at Lightning Speed, 10.1101/2021.11.14.468528 · DOI · Paper page
- Heinzinger, 2022, Contrastive learning on protein embeddings enlightens midnight zone, NAR Genomics and Bioinformatics, 4, 10.1093/nargab/lqac043 · doi:10.1093/nargab/lqac043
- Hong, 2021, FastMSA: Accelerating Multiple Sequence Alignment with Dense Retrieval on Protein Language, 10.1101/2021.12.20.473431 · DOI
- Huang Z, Xu W, Yu K. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv. https://arxiv.org/abs/1508.01991. · arXiv
- Jia, 2016, ISuc-pseopt: identifying lysine succinylation sites in proteins by incorporating sequence-coupling effects into pseudo components and optimizing imbalanced training dataset, Analytical Biochemistry, 497, 48, 10.1016/j.ab.2015.12.009 · DOI · Paper page
- Jiang, 2017, Protein secondary structure prediction: A survey of the state of the art, Journal of Molecular Graphics & Modelling, 76, 379, 10.1016/j.jmgm.2017.07.015 · DOI
- Jiang T, Fang L, Wang K. 2021. MutFormer: A Context-Dependent Transformer-Based Model to Predict Pathogenic Missense Mutations. arXiv. https://arxiv.org/abs/2110.14746. · arXiv
- Jones, 1999, Protein secondary structure prediction based on position-specific scoring matrices, Journal of Molecular Biology, 292, 195, 10.1006/jmbi.1999.3091 · DOI · Paper page
- Katchalski-Katzir, 1992, Molecular surface recognition: determination of geometric fit between proteins and their ligands by correlation techniques, PNAS, 89, 2195, 10.1073/pnas.89.6.2195 · DOI · Paper page
- Khalili, 2022, Predicting protein phosphorylation sites in soybean using interpretable deep tabular learning network, Briefings in Bioinformatics, 23, 10.1093/bib/bbac015 · DOI
- Khan M, Jan B, Farman H. 2019. Deep learning: convergence to big data analytics. Deep Learning Methods and Applications. · DOI · Paper page
- Koumakis, 2020, Deep learning models in genomics; are we there yet?, Computational and Structural Biotechnology Journal, 18, 1466, 10.1016/j.csbj.2020.06.017 · DOI · Paper page
- Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R. 2019. Albert: A Lite Bert for Self-Supervised Learning of Language Representations. arXiv. https://arxiv.org/abs/1909.11942. · arXiv
- Lanchantin J, Weingarten T, Sekhon A, Miller C, Qi Y. 2021. Transfer learning for predicting virus-host protein interactions for novel virus sequences. BCB ’21: Proceedings of the 12th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics. · DOI
- Laskar MTR, Huang X, Hoque E. 2020. Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task. Proceedings of The 12th Language Resources and Evaluation Conference.
- Levy O, Goldberg Y. 2014. Dependency-based word embeddings. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics.
- Li, 2022, Positive-unlabeled learning in bioinformatics and computational biology: a brief review, Briefings in Bioinformatics, 23, 10.1093/bib/bbab461 · DOI
- Liu, 2007, BindingDB: a web-accessible database of experimentally determined protein-ligand binding affinities, Nucleic Acids Research, 35, D198, 10.1093/nar/gkl999 · DOI · Paper page
- Liu, 2012, Using amino acid physicochemical distance transformation for fast protein remote homology detection, PLOS ONE, 7, 10.1371/journal.pone.0046633 · doi:10.1371/journal.pone.0046633
- Liu, 2020, DeepSVM-fold: protein fold recognition by combining support vector machines and pairwise sequence similarity scores generated by deep learning networks, Briefings in Bioinformatics, 21, 1733, 10.1093/bib/bbz098 · DOI
- López, 2017, SucStruct: prediction of succinylated lysine residues by using structural properties of amino acids, Analytical Biochemistry, 527, 24, 10.1016/j.ab.2017.03.021 · DOI
- Lopez MJ, Mohiuddin SS. 2020. Biochemistry, essential amino acids. Biochemistry.
- Lu T, Lu AX, Moses AM. 2021. Random Embeddings and Linear Regression Can Predict Protein Function. arXiv. https://arxiv.org/abs/2104.14661. · arXiv
- McDowall, 2009, Pips: human protein-protein interaction prediction database, Nucleic Acids Research, 37, D651, 10.1093/nar/gkn870 · DOI · Paper page
- Mikolov T, Kombrink S, Burget L, Černocký J. 2011. Extensions of recurrent neural network language model. 2011 IEEE international conference on acoustics, speech and signal processing (ICASSP); 2011: IEEE. · DOI · Paper page
- Mikolov T, Chen K, Corrado GS. 2013a. International Conference on Learning Representations. Efficient Estimation of Word Representations in Vector Space.
- Mistry, 2021, Pfam: the protein families database in 2021, Nucleic Acids Research, 49, D412, 10.1093/nar/gkaa913 · DOI · Paper page
- Nambiar A, Heflin M, Liu S, Maslov S, Hopkins M, Ritz A. 2020. Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks. BCB ’20: Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics. · DOI · Paper page
- Nauman, 2019, Beyond homology transfer: deep learning for automated annotation of proteins, Journal of Grid Computing, 17, 225, 10.1007/s10723-018-9450-6 · DOI
- Ofer, 2021, The language of proteins: Nlp, machine learning & protein sequences, Computational and Structural Biotechnology Journal, 19, 1750, 10.1016/j.csbj.2021.03.022 · DOI · Paper page
- Öztürk, 2018, DeepDTA: deep drug-target binding affinity prediction, Bioinformatics, 34, i821, 10.1093/bioinformatics/bty593 · DOI · Paper page
- Pan, 2022, Systematic evaluation of computational tools to predict the effects of mutations on protein stability in the absence of experimental structures, Briefings in Bioinformatics, 23, 10.1093/bib/bbac025 · DOI
- Phuong, 2006, Multiple alignment of protein sequences with repeats and rearrangements, Nucleic Acids Research, 34, 5932, 10.1093/nar/gkl511 · DOI
- Qiao, 2022, BERT-kcr: prediction of lysine crotonylation sites by a transfer learning method with pre-trained BERT models, Bioinformatics, 38, 648, 10.1093/bioinformatics/btab712 · DOI
- Raghu M, Unterthiner T, Kornblith S, Zhang C, Dosovitskiy A. 2021. Do vision transformers see like convolutional neural networks. Advances in Neural Information Processing Systems.
- Ramazi, 2021, Posttranslational modifications in proteins: resources, tools and prediction methods, Database, 2021, 10.1093/database/baab012 · DOI · Paper page
- Rao RM, Liu J, Verkuil R, Meier J, Canny J. 2021. MSA transformer. International Conference on Machine Learning; 2021: PMLR.
- Saethang, 2016, A machine learning strategy for predicting localization of post-translational modification sites in protein-protein interacting regions, BMC Bioinformatics, 17, 10.1186/s12859-016-1165-8 · DOI
- Schmiedel, 2019, Determining protein structures using deep mutagenesis, Nature Genetics, 51, 1177, 10.1038/s41588-019-0431-x · DOI · Paper page
- Sharma, 2013, A feature extraction technique using bi-gram probabilities of position specific scoring matrix for protein fold recognition, Journal of Theoretical Biology, 320, 41, 10.1016/j.jtbi.2012.12.008 · DOI
- Shi, 2021, Deep learning for mining protein data, Briefings in Bioinformatics, 22, 194, 10.1093/bib/bbz156 · DOI · Paper page
- Singh, 2021, SPOT-1D-single: improving the single-sequence-based prediction of protein secondary structure, backbone angles, solvent accessibility and half-sphere exposures using a large training set and ensembled deep learning, Bioinformatics, 37, 3464, 10.1093/bioinformatics/btab316 · DOI · Paper page
- Singh, 2022, SPOT-contact-LM: improving single-sequence-based prediction of protein contact MAP using a transformer language model, Bioinformatics, 38, 1888, 10.1093/bioinformatics/btac053 · DOI · Paper page
- Smith S, Patwary M, Norick B, LeGresley P, Rajbhandari S, Casper J. 2022. Using Deepspeed and Megatron to Train Megatron-Turing Nlg 530b, a Large-Scale Generative Language Model. arXiv. https://arxiv.org/abs/2201.11990. · arXiv
- Sourkov V. 2018. Igloo: Slicing the Features Space to Represent Sequences. arXiv. https://arxiv.org/abs/1807.03402. · arXiv
- Sturmfels P, Vig J, Madani A, Rajani NF. 2020. Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models. arXiv. https://arxiv.org/abs/2012.00195. · arXiv
- Sułkowska, 2012, Genomics-aided structure prediction, PNAS, 109, 10340, 10.1073/pnas.1207864109 · DOI · Paper page
- Tang, 2014, Making sense of large-scale kinase inhibitor bioactivity data sets: a comparative and integrative analysis, Journal of Chemical Information and Modeling, 54, 735, 10.1021/ci400709d · DOI · Paper page
- Tavares, 2013, Strategies and molecular tools to fight antimicrobial resistance: resistome, transcriptome, and antimicrobial peptides, Frontiers in Microbiology, 4, 10.3389/fmicb.2013.00412 · DOI · Paper page
- Tran K, Bisazza A, Monz C. 2018. The Importance of Being Recurrent for Modeling Hierarchical Structure. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. · DOI · Paper page
- Trost, 2013, Computational phosphorylation site prediction in plants using random forests and organism-specific instance weights, Bioinformatics, 29, 686, 10.1093/bioinformatics/btt031 · DOI
- Turian J, Ratinov L. 2010. Word representations: a simple and general method for semi-supervised learning. Proceedings of the 48th annual meeting of the association for computational linguistics.
- Van der Maaten, 2008, Visualizing data using t-SNE, Journal of Machine Learning Research, 9
- Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN. 2017. Attention is all you need. Advances in Neural Information Processing Systems.
- Väth, 2022, PROVAL: A framework for comparison of protein sequence embeddings, Journal of Computational Mathematics and Data Science, 2022, 10.1016/j.jcmds.2022.100044 · DOI
- Vig J. 2019a. BertViz: A tool for visualizing multihead self-attention in the BERT model. ICLR Workshop: Debugging Machine Learning Models.
- Walls, 2017, Protein Stability: Enhancement and Measurement, 101, 10.1007/978-1-4939-6412-3 · doi:10.1007/978-1-4939-6412-3
- Wang, 2017, Prediction of phosphorylation sites based on krawtchouk image moments, Proteins, 85, 2231, 10.1002/prot.25388 · DOI
- Wang, 2020, MusiteDeep: a deep-learning based Webserver for protein post-translational modification site prediction and visualization, Nucleic Acids Research, 48, W140, 10.1093/nar/gkaa275 · DOI · Paper page
- wang, 2021, ELECTRA-DTA: A new compound-protein binding affinity prediction model based on the contextualized sequence encoding, Journal of Cheminformatics, 14, 10.1186/s13321-022-00591-x · DOI
- Wang, 2021, There once was a really bad poet, it was automated but you did ’'t know it, Transactions of the Association for Computational Linguistics, 9, 605, 10.1162/tacl_a_00387 · DOI
- Wang, 2022, Prior knowledge facilitates low homologous protein secondary structure prediction with DSM distillation, Bioinformatics, 38, 3574, 10.1093/bioinformatics/btac351 · DOI
- Wettig A, Gao T, Zhong Z, Mask C. 2022. Should You Mask 15% in Masked Language Modeling?. arXiv. https://arxiv.org/abs/2202.08005. · arXiv
- Wilburn, 2020, Remote homology search with hidden Potts models, PLOS Computational Biology, 16, 10.1371/journal.pcbi.1008085 · DOI · Paper page
- Xu, 2018, PhosContext2vec: a distributed representation of residue-level sequence contexts and its application to general and kinase-specific phosphorylation site prediction, Scientific Reports, 8, 10.1038/s41598-018-26392-7 · DOI
- Xue Y, Liu Z, Fang X. 2022. Multimodal Pre-Training Model for Sequence-based Prediction of Protein-Protein Interaction. Machine Learning in Computational Biology; 2022: PMLR.
- Yamaguchi, 2021, Evotuning protocols for transformer-based variant effect prediction on multi-domain proteins, Briefings in Bioinformatics, 22, 10.1093/bib/bbab234 · DOI
- Yin W, Kann K, Yu M, Schütze H. 2017. Comparative Study of CNN and RNN for Natural Language Processing. arXiv. https://arxiv.org/abs/1702.01923. · arXiv
- Yu, 2017, Protein sequence comparison based on physicochemical properties and the position-feature energy matrix, Scientific Reports, 7, 10.1038/srep46237 · DOI
- Zaheer M, Guruganesh G, Dubey KA, Ainslie J, Alberti C, Ontanon S. 2020. Big bird: transformers for longer sequences. Advances in Neural Information Processing Systems.
- Zare-Mirakabad, 2021, Protein Sequence Profile Prediction Using Protalbert Transformer1, 10.1101/2021.09.23.461475 · DOI
- Zeng, 2016, Convolutional neural network architectures for predicting DNA-protein binding, Bioinformatics, 32, i121, 10.1093/bioinformatics/btw255 · DOI · Paper page
- Zhang H, Ju F, Zhu J, He L, Shao B, Zheng N. 2021. Co-evolution Transformer for Protein Contact Prediction. Advances in Neural Information Processing Systems.
- Zhao Q, Ma J, Wang Y, Xie F, Lv Z, Xu Y. 2021. Mul-SNO: A novel prediction tool for S-nitrosylation sites based on deep learning methods. IEEE Journal of Biomedical and Health Informatics. · DOI
Cited by
Protein codes promote selective subcellular compartmentalization
Kilgore, H. R., Chinn, I., Mikhael, P. G., Mitnikov, I., Van Dongen, C., Zylberberg, G., Afeyan, L., Banani, S. F., Wilson-Hawken, S., Lee, T. I., et al. (2025) · Science · doi:10.1126/science.adq2634 · Cited by 99
Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis
Liu, J., Cen, X., Yi, C., Wang, F.-A., Ding, J., Cheng, J., Wu, Q., Gai, B., Zhou, Y., He, R., et al. (2025) · Genomics, Proteomics & Bioinformatics · doi:10.1093/gpbjnl/qzaf011 · Cited by 75
Medium-sized protein language models perform well at transfer learning on realistic datasets
Vieira, L. C., Handojo, M. L., & Wilke, C. O. (2025) · Scientific Reports · doi:10.1038/s41598-025-05674-x · Cited by 41
ESM ‐scan—A tool to guide amino acid substitutions
Totaro, M. G., Vide, U., Zausinger, R., Winkler, A., & Oberdorfer, G. (2024) · Protein Science · doi:10.1002/pro.5221 · Cited by 19
Kong, S. W., Lee, I.-H., Collen, L. V., Field, M., Manrai, A. K., Snapper, S. B., & Mandl, K. D. (2025) · npj Genomic Medicine · doi:10.1038/s41525-025-00480-w · Cited by 18
Metadata
- Authors
- Chandra, A., Tünnermann, L., Löfstedt, T., & Gratz, R.
- Venue
- eLife
- Publisher
- eLife Sciences Publications, Ltd
- Published
- 2023-01-18
- Volume
- 12
- ISSN
- 2050-084X (ISSN-L)
- OpenAlex
- W4317212783
- Cited by
- 184
- References
- 51
- FWCI
- 13.75
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/4.0/ (vor)
- creativecommons.org/licenses/by/4.0/ (am)
- creativecommons.org/licenses/by/4.0/ (tdm)
- Funders
- Kempestiftelserna (JCK-2144, JCK-2015.1)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.