Transformer-based deep learning for predicting protein properties in the life sciences
Open accessThe DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Recent developments in deep learning, coupled with an increasing number of sequenced proteins, have led to a breakthrough in life science applications, in particular in protein property prediction. There is hope that deep learning can close the gap between the number of sequenced proteins and proteins with known properties based on lab experiments. Language models from the field of natural language processing have gained popularity for protein property predictions and have led to a new computational revolution in biology, where old prediction results are being improved regularly. Such models can learn useful multipurpose representations of proteins from large open repositories of protein sequences and can be used, for instance, to predict protein properties. The field of natural language processing is growing quickly because of developments in a class of models based on a particular model—the Transformer model. We review recent developments and the use of large-scale Transformer models in applications for predicting protein characteristics and how such models can be used to predict, for example, post-translational modifications. We review shortcomings of other deep learning models and explain how the Transformer models have quickly proven to be a very promising way to unravel information hidden in the sequences of amino acids.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Common RNN models include the long short-term memory (LSTM) model ( Hochreiter and Schmidhuber, 1997 ) and the gated recurrent units (GRUs) ( Chung et al., 2014 ).”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Principal component analysis: A review and recent developments
Jolliffe, I. T., & Cadima, J. (2016) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2015.0202 · Cited by 10,117
ColabFold - Making protein folding accessible to all
Mirdita, M., Schütze, K., Moriwaki, Y., Heo, L., Ovchinnikov, S., & Steinegger, M. (2021) · doi:10.1101/2021.08.15.456425 · Cited by 9,391
Research works printed but not linked: 112
- Alanis-Lobato, 2016, HIPPIE v2. 0: enhancing meaningfulness and reliability of protein–protein interaction networks, Nucleic Acids Research, 45, D408, 10.1093/nar/gkw985 · DOI · Paper page
- Albawi S, Mohammed TA, Al-Zawi S. 2017. Understanding of a convolutional neural network. 2017 International Conference on Engineering and Technology (ICET. · DOI · Paper page
- Altschul, 1997, Gapped blast and PSI-BLAST: a new generation of protein database search programs, Nucleic Acids Research, 25, 3389, 10.1093/nar/25.17.3389 · DOI · Paper page
- Ammari MG, Gresham CR, McCarthy FM, Nanduri B. 2016. HPIDB 2.0: a curated database for host–pathogen interactions. Database.
- Behjati, 2022, Protein sequence profile prediction using protalbert transformer, Computational Biology and Chemistry, 99, 10.1016/j.compbiolchem.2022.107717 · DOI
- Bergstra J, Yamins D, Cox D. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. International conference on machine learning; 2013: PMLR.
- Bulcun, 2012, Disorders of glucose metabolism and insulin resistance in patients with obstructive sleep apnoea syndrome, International Journal of Clinical Practice, 66, 91, 10.1111/j.1742-1241.2011.02795.x · DOI
- Cai, 2021, MSA-regularized protein sequence transformer toward predicting genome-wide chemical-protein interactions: application to gpcrome deorphanization, Journal of Chemical Information and Modeling, 61, 1570, 10.1021/acs.jcim.0c01285 · DOI
- Chan, 2015, Glass: a comprehensive database for experimentally validated GPCR-ligand associations, Bioinformatics, 31, 3035, 10.1093/bioinformatics/btv302 · DOI · Paper page
- Chandra, 2019, Bigram-PGK: phosphoglycerylation prediction using the technique of bigram probabilities of position specific scoring matrix, BMC Molecular and Cell Biology, 20, 10.1186/s12860-019-0240-1 · DOI
- Chandra, 2020, RAM-PGK: prediction of lysine phosphoglycerylation based on residue adjacency matrix, Genes, 11, 10.3390/genes11121524 · DOI
- Charte, 2019, A snapshot on nonstandard supervised learning problems: taxonomy, relationships, problem transformations and algorithm adaptations, Progress in Artificial Intelligence, 8, 1, 10.1007/s13748-018-00167-7 · DOI
- Chen, 2002, The binding database: data management and interface design, Bioinformatics, 18, 130, 10.1093/bioinformatics/18.1.130 · DOI · Paper page
- Chen, 2020, IStable 2.0: predicting protein thermal stability changes by integrating various characteristic modules, Computational and Structural Biotechnology Journal, 18, 622, 10.1016/j.csbj.2020.02.021 · DOI
- Cheng, 2021, BERTMHC: improved MHC-peptide class II interaction prediction with transformer and multiple instance learning, Bioinformatics, 37, 4172, 10.1093/bioinformatics/btab422 · DOI · Paper page
- Choromanski K, Likhosherstov V, Dohan D, Song X, Gane A, Sarlos T. 2020. Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers. arXiv. https://arxiv.org/abs/2006.03555. · arXiv
- Chou, 1995, Prediction of protein structural classes, Critical Reviews in Biochemistry and Molecular Biology, 30, 275, 10.3109/10409239509083488 · DOI · Paper page
- Chou, 2020, Progresses in predicting post-translational modification, International Journal of Peptide Research and Therapeutics, 26, 873, 10.1007/s10989-019-09893-5 · DOI
- Clark K, Luong MT, Le QV. 2020. ELECTRA: Pre-Training Text Encoders as Discriminators Rather Than Generators. arXiv. https://arxiv.org/abs/2003.10555. · arXiv
- Cortés, 2011, Predicting protein distance maps according to physicochemical properties, Journal of Integrative Bioinformatics, 8, 158, 10.1515/jib-2011-181 · doi:10.1515/jib-2011-181
- Dai Z, Yang Z, Yang Y, Cohen WW, Carbonell J, Le QV. 2018. Transformer-xl: language modeling with longer-term dependency. Transformer-Xl.
- Dehghani M, Gouws S, Vinyals O, Uszkoreit J, Kaiser Ł. 2018. Universal Transformers. arXiv. https://arxiv.org/abs/1807.03819. · arXiv
- Dick, 2018, Reciprocal perspective for improved protein-protein interaction prediction, Scientific Reports, 8, 1, 10.1038/s41598-018-30044-1 · DOI
- Dodge, 1998, The HSSP database of protein structure-sequence alignments and family profiles, Nucleic Acids Research, 26, 313, 10.1093/nar/26.1.313 · DOI · Paper page
- Du, 2021, The trrosetta server for fast and accurate protein structure prediction, Nature Protocols, 16, 5634, 10.1038/s41596-021-00628-9 · DOI · Paper page
- Ehrenberger, 2015, Computational prediction of protein-protein interactions, Methods in Molecular Biology, 1278, 57, 10.1007/978-1-4939-2425-7_4 · DOI
- ElAbd, 2020, Amino acid encoding for deep learning applications, BMC Bioinformatics, 21, 10.1186/s12859-020-03546-x · DOI · Paper page
- Elman, 1990, Finding structure in time, Cognitive Science, 14, 179, 10.1207/s15516709cog1402_1 · DOI · Paper page
- Elnaggar A, Heinzinger M, Dallago C, Rehawi G, Wang Y, Jones L, Gibbs T, Feher T, Angerer C, Steinegger M, Bhowmik D, Rost B. 2020b. ProtTrans: Towards Cracking the Language of Life’s Code through Self-Supervised Deep Learning and High Performance Computing. arXiv. https://arxiv.org/abs/2007.06225. · arXiv
- Fang, 2020, A critical review of five machine learning-based algorithms for predicting protein stability changes upon mutation, Briefings in Bioinformatics, 21, 1285, 10.1093/bib/bbz071 · DOI · Paper page
- Fedus W, Zoph B, Shazeer N. 2021. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv. https://arxiv.org/abs/2101.03961. · arXiv
- Golubchik, 2007, Mind the gaps: evidence of bias in estimates of multiple sequence alignments, Molecular Biology and Evolution, 24, 2433, 10.1093/molbev/msm176 · DOI
- Google Scholar. 2022. Google Scholar. https://scholar.google.com/scholar?q=transformer+language+model+transformer+language+model&hl=en&as_sdt=0,5 [Accessed July 1, 2022].
- Gromiha MM, Nagarajan R, Selvaraj S. 2019. Protein structural bioinformatics: an overview. Protein Structural Bioinformatics.
- Guerler, 2013, Mapping monomeric threading to protein-protein structure prediction, Journal of Chemical Information and Modeling, 53, 717, 10.1021/ci300579r · DOI · Paper page
- Hanin B. 2018. Which neural net architectures give rise to exploding and vanishing gradients?. Advances in Neural Information Processing Systems.
- Hao Y, Dong L, Wei F, Xu K. 2021. Self-attention attribution: interpreting information interactions inside transformer. Proceedings of the AAAI Conference on Artificial Intelligence. · DOI · Paper page
- He L, Zhang S, Wu L, Xia H, Ju F, Zhang H. 2021. Pre-Training Co-Evolutionary Protein Representation via A Pairwise Masked Language Model. arXiv. https://arxiv.org/abs/2110.15527. · arXiv
- Heinzinger, 2021, Contrastive Learning on Protein Embeddings Enlightens Midnight Zone at Lightning Speed, 10.1101/2021.11.14.468528 · DOI · Paper page
- Heinzinger, 2022, Contrastive learning on protein embeddings enlightens midnight zone, NAR Genomics and Bioinformatics, 4, 10.1093/nargab/lqac043 · doi:10.1093/nargab/lqac043
- Hong, 2021, FastMSA: Accelerating Multiple Sequence Alignment with Dense Retrieval on Protein Language, 10.1101/2021.12.20.473431 · DOI
- Huang Z, Xu W, Yu K. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv. https://arxiv.org/abs/1508.01991. · arXiv
- Jia, 2016, ISuc-pseopt: identifying lysine succinylation sites in proteins by incorporating sequence-coupling effects into pseudo components and optimizing imbalanced training dataset, Analytical Biochemistry, 497, 48, 10.1016/j.ab.2015.12.009 · DOI · Paper page
- Jiang, 2017, Protein secondary structure prediction: A survey of the state of the art, Journal of Molecular Graphics & Modelling, 76, 379, 10.1016/j.jmgm.2017.07.015 · DOI
- Jiang T, Fang L, Wang K. 2021. MutFormer: A Context-Dependent Transformer-Based Model to Predict Pathogenic Missense Mutations. arXiv. https://arxiv.org/abs/2110.14746. · arXiv
- Jones, 1999, Protein secondary structure prediction based on position-specific scoring matrices, Journal of Molecular Biology, 292, 195, 10.1006/jmbi.1999.3091 · DOI · Paper page
- Katchalski-Katzir, 1992, Molecular surface recognition: determination of geometric fit between proteins and their ligands by correlation techniques, PNAS, 89, 2195, 10.1073/pnas.89.6.2195 · DOI · Paper page
- Khalili, 2022, Predicting protein phosphorylation sites in soybean using interpretable deep tabular learning network, Briefings in Bioinformatics, 23, 10.1093/bib/bbac015 · DOI
- Khan M, Jan B, Farman H. 2019. Deep learning: convergence to big data analytics. Deep Learning Methods and Applications. · DOI · Paper page
- Koumakis, 2020, Deep learning models in genomics; are we there yet?, Computational and Structural Biotechnology Journal, 18, 1466, 10.1016/j.csbj.2020.06.017 · DOI · Paper page
- Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R. 2019. Albert: A Lite Bert for Self-Supervised Learning of Language Representations. arXiv. https://arxiv.org/abs/1909.11942. · arXiv
- Lanchantin J, Weingarten T, Sekhon A, Miller C, Qi Y. 2021. Transfer learning for predicting virus-host protein interactions for novel virus sequences. BCB ’21: Proceedings of the 12th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics. · DOI
- Laskar MTR, Huang X, Hoque E. 2020. Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task. Proceedings of The 12th Language Resources and Evaluation Conference.
- Levy O, Goldberg Y. 2014. Dependency-based word embeddings. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics.
- Li, 2022, Positive-unlabeled learning in bioinformatics and computational biology: a brief review, Briefings in Bioinformatics, 23, 10.1093/bib/bbab461 · DOI
- Liu, 2007, BindingDB: a web-accessible database of experimentally determined protein-ligand binding affinities, Nucleic Acids Research, 35, D198, 10.1093/nar/gkl999 · DOI · Paper page
- Liu, 2012, Using amino acid physicochemical distance transformation for fast protein remote homology detection, PLOS ONE, 7, 10.1371/journal.pone.0046633 · doi:10.1371/journal.pone.0046633
- Liu, 2020, DeepSVM-fold: protein fold recognition by combining support vector machines and pairwise sequence similarity scores generated by deep learning networks, Briefings in Bioinformatics, 21, 1733, 10.1093/bib/bbz098 · DOI
- López, 2017, SucStruct: prediction of succinylated lysine residues by using structural properties of amino acids, Analytical Biochemistry, 527, 24, 10.1016/j.ab.2017.03.021 · DOI
- Lopez MJ, Mohiuddin SS. 2020. Biochemistry, essential amino acids. Biochemistry.
- Lu T, Lu AX, Moses AM. 2021. Random Embeddings and Linear Regression Can Predict Protein Function. arXiv. https://arxiv.org/abs/2104.14661. · arXiv
- McDowall, 2009, Pips: human protein-protein interaction prediction database, Nucleic Acids Research, 37, D651, 10.1093/nar/gkn870 · DOI · Paper page
- Mikolov T, Kombrink S, Burget L, Černocký J. 2011. Extensions of recurrent neural network language model. 2011 IEEE international conference on acoustics, speech and signal processing (ICASSP); 2011: IEEE. · DOI · Paper page
- Mikolov T, Chen K, Corrado GS. 2013a. International Conference on Learning Representations. Efficient Estimation of Word Representations in Vector Space.
- Mistry, 2021, Pfam: the protein families database in 2021, Nucleic Acids Research, 49, D412, 10.1093/nar/gkaa913 · DOI · Paper page
- Nambiar A, Heflin M, Liu S, Maslov S, Hopkins M, Ritz A. 2020. Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks. BCB ’20: Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics. · DOI · Paper page
- Nauman, 2019, Beyond homology transfer: deep learning for automated annotation of proteins, Journal of Grid Computing, 17, 225, 10.1007/s10723-018-9450-6 · DOI
- Ofer, 2021, The language of proteins: Nlp, machine learning & protein sequences, Computational and Structural Biotechnology Journal, 19, 1750, 10.1016/j.csbj.2021.03.022 · DOI · Paper page
- Öztürk, 2018, DeepDTA: deep drug-target binding affinity prediction, Bioinformatics, 34, i821, 10.1093/bioinformatics/bty593 · DOI · Paper page
- Pan, 2022, Systematic evaluation of computational tools to predict the effects of mutations on protein stability in the absence of experimental structures, Briefings in Bioinformatics, 23, 10.1093/bib/bbac025 · DOI
- Phuong, 2006, Multiple alignment of protein sequences with repeats and rearrangements, Nucleic Acids Research, 34, 5932, 10.1093/nar/gkl511 · DOI
- Qiao, 2022, BERT-kcr: prediction of lysine crotonylation sites by a transfer learning method with pre-trained BERT models, Bioinformatics, 38, 648, 10.1093/bioinformatics/btab712 · DOI
- Raghu M, Unterthiner T, Kornblith S, Zhang C, Dosovitskiy A. 2021. Do vision transformers see like convolutional neural networks. Advances in Neural Information Processing Systems.
- Ramazi, 2021, Posttranslational modifications in proteins: resources, tools and prediction methods, Database, 2021, 10.1093/database/baab012 · DOI · Paper page
- Rao RM, Liu J, Verkuil R, Meier J, Canny J. 2021. MSA transformer. International Conference on Machine Learning; 2021: PMLR.
- Saethang, 2016, A machine learning strategy for predicting localization of post-translational modification sites in protein-protein interacting regions, BMC Bioinformatics, 17, 10.1186/s12859-016-1165-8 · DOI
- Schmiedel, 2019, Determining protein structures using deep mutagenesis, Nature Genetics, 51, 1177, 10.1038/s41588-019-0431-x · DOI · Paper page
- Sharma, 2013, A feature extraction technique using bi-gram probabilities of position specific scoring matrix for protein fold recognition, Journal of Theoretical Biology, 320, 41, 10.1016/j.jtbi.2012.12.008 · DOI
- Shi, 2021, Deep learning for mining protein data, Briefings in Bioinformatics, 22, 194, 10.1093/bib/bbz156 · DOI · Paper page
- Singh, 2021, SPOT-1D-single: improving the single-sequence-based prediction of protein secondary structure, backbone angles, solvent accessibility and half-sphere exposures using a large training set and ensembled deep learning, Bioinformatics, 37, 3464, 10.1093/bioinformatics/btab316 · DOI · Paper page
- Singh, 2022, SPOT-contact-LM: improving single-sequence-based prediction of protein contact MAP using a transformer language model, Bioinformatics, 38, 1888, 10.1093/bioinformatics/btac053 · DOI · Paper page
- Smith S, Patwary M, Norick B, LeGresley P, Rajbhandari S, Casper J. 2022. Using Deepspeed and Megatron to Train Megatron-Turing Nlg 530b, a Large-Scale Generative Language Model. arXiv. https://arxiv.org/abs/2201.11990. · arXiv
- Sourkov V. 2018. Igloo: Slicing the Features Space to Represent Sequences. arXiv. https://arxiv.org/abs/1807.03402. · arXiv
- Sturmfels P, Vig J, Madani A, Rajani NF. 2020. Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models. arXiv. https://arxiv.org/abs/2012.00195. · arXiv
- Sułkowska, 2012, Genomics-aided structure prediction, PNAS, 109, 10340, 10.1073/pnas.1207864109 · DOI · Paper page
- Tang, 2014, Making sense of large-scale kinase inhibitor bioactivity data sets: a comparative and integrative analysis, Journal of Chemical Information and Modeling, 54, 735, 10.1021/ci400709d · DOI · Paper page
- Tavares, 2013, Strategies and molecular tools to fight antimicrobial resistance: resistome, transcriptome, and antimicrobial peptides, Frontiers in Microbiology, 4, 10.3389/fmicb.2013.00412 · DOI · Paper page
- Tran K, Bisazza A, Monz C. 2018. The Importance of Being Recurrent for Modeling Hierarchical Structure. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. · DOI · Paper page
- Trost, 2013, Computational phosphorylation site prediction in plants using random forests and organism-specific instance weights, Bioinformatics, 29, 686, 10.1093/bioinformatics/btt031 · DOI
- Turian J, Ratinov L. 2010. Word representations: a simple and general method for semi-supervised learning. Proceedings of the 48th annual meeting of the association for computational linguistics.
- Van der Maaten, 2008, Visualizing data using t-SNE, Journal of Machine Learning Research, 9
- Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN. 2017. Attention is all you need. Advances in Neural Information Processing Systems.
- Väth, 2022, PROVAL: A framework for comparison of protein sequence embeddings, Journal of Computational Mathematics and Data Science, 2022, 10.1016/j.jcmds.2022.100044 · DOI
- Vig J. 2019a. BertViz: A tool for visualizing multihead self-attention in the BERT model. ICLR Workshop: Debugging Machine Learning Models.
- Walls, 2017, Protein Stability: Enhancement and Measurement, 101, 10.1007/978-1-4939-6412-3 · doi:10.1007/978-1-4939-6412-3
- Wang, 2017, Prediction of phosphorylation sites based on krawtchouk image moments, Proteins, 85, 2231, 10.1002/prot.25388 · DOI
- Wang, 2020, MusiteDeep: a deep-learning based Webserver for protein post-translational modification site prediction and visualization, Nucleic Acids Research, 48, W140, 10.1093/nar/gkaa275 · DOI · Paper page
- wang, 2021, ELECTRA-DTA: A new compound-protein binding affinity prediction model based on the contextualized sequence encoding, Journal of Cheminformatics, 14, 10.1186/s13321-022-00591-x · DOI
- Wang, 2021, There once was a really bad poet, it was automated but you did ’'t know it, Transactions of the Association for Computational Linguistics, 9, 605, 10.1162/tacl_a_00387 · DOI
- Wang, 2022, Prior knowledge facilitates low homologous protein secondary structure prediction with DSM distillation, Bioinformatics, 38, 3574, 10.1093/bioinformatics/btac351 · DOI
- Wettig A, Gao T, Zhong Z, Mask C. 2022. Should You Mask 15% in Masked Language Modeling?. arXiv. https://arxiv.org/abs/2202.08005. · arXiv
- Wilburn, 2020, Remote homology search with hidden Potts models, PLOS Computational Biology, 16, 10.1371/journal.pcbi.1008085 · DOI · Paper page
- Xu, 2018, PhosContext2vec: a distributed representation of residue-level sequence contexts and its application to general and kinase-specific phosphorylation site prediction, Scientific Reports, 8, 10.1038/s41598-018-26392-7 · DOI
- Xue Y, Liu Z, Fang X. 2022. Multimodal Pre-Training Model for Sequence-based Prediction of Protein-Protein Interaction. Machine Learning in Computational Biology; 2022: PMLR.
- Yamaguchi, 2021, Evotuning protocols for transformer-based variant effect prediction on multi-domain proteins, Briefings in Bioinformatics, 22, 10.1093/bib/bbab234 · DOI
- Yin W, Kann K, Yu M, Schütze H. 2017. Comparative Study of CNN and RNN for Natural Language Processing. arXiv. https://arxiv.org/abs/1702.01923. · arXiv
- Yu, 2017, Protein sequence comparison based on physicochemical properties and the position-feature energy matrix, Scientific Reports, 7, 10.1038/srep46237 · DOI
- Zaheer M, Guruganesh G, Dubey KA, Ainslie J, Alberti C, Ontanon S. 2020. Big bird: transformers for longer sequences. Advances in Neural Information Processing Systems.
- Zare-Mirakabad, 2021, Protein Sequence Profile Prediction Using Protalbert Transformer1, 10.1101/2021.09.23.461475 · DOI
- Zeng, 2016, Convolutional neural network architectures for predicting DNA-protein binding, Bioinformatics, 32, i121, 10.1093/bioinformatics/btw255 · DOI · Paper page
- Zhang H, Ju F, Zhu J, He L, Shao B, Zheng N. 2021. Co-evolution Transformer for Protein Contact Prediction. Advances in Neural Information Processing Systems.
- Zhao Q, Ma J, Wang Y, Xie F, Lv Z, Xu Y. 2021. Mul-SNO: A novel prediction tool for S-nitrosylation sites based on deep learning methods. IEEE Journal of Biomedical and Health Informatics. · DOI
Cited by
Protein codes promote selective subcellular compartmentalization
Kilgore, H. R., Chinn, I., Mikhael, P. G., Mitnikov, I., Van Dongen, C., Zylberberg, G., Afeyan, L., Banani, S. F., Wilson-Hawken, S., Lee, T. I., et al. (2025) · Science · doi:10.1126/science.adq2634 · Cited by 99
Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis
Liu, J., Cen, X., Yi, C., Wang, F.-A., Ding, J., Cheng, J., Wu, Q., Gai, B., Zhou, Y., He, R., et al. (2025) · Genomics, Proteomics & Bioinformatics · doi:10.1093/gpbjnl/qzaf011 · Cited by 75
Medium-sized protein language models perform well at transfer learning on realistic datasets
Vieira, L. C., Handojo, M. L., & Wilke, C. O. (2025) · Scientific Reports · doi:10.1038/s41598-025-05674-x · Cited by 41
ESM ‐scan—A tool to guide amino acid substitutions
Totaro, M. G., Vide, U., Zausinger, R., Winkler, A., & Oberdorfer, G. (2024) · Protein Science · doi:10.1002/pro.5221 · Cited by 19
Kong, S. W., Lee, I.-H., Collen, L. V., Field, M., Manrai, A. K., Snapper, S. B., & Mandl, K. D. (2025) · npj Genomic Medicine · doi:10.1038/s41525-025-00480-w · Cited by 18
Large language models transform biological research: from architecture to utilization
Wang, T., & Luo, Z. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4466-3 · Cited by 14
Efficient inference, training, and fine-tuning of protein language models
Çelik, M. H., & Xie, X. (2025) · iScience · doi:10.1016/j.isci.2025.113495 · Cited by 7
Metadata
- Authors
- Chandra, A., Tünnermann, L., Löfstedt, T., & Gratz, R.
- Venue
- eLife
- Publisher
- eLife Sciences Publications, Ltd
- Published
- 2023-01-18
- Volume
- 12
- ISSN
- 2050-084X (ISSN-L)
- OpenAlex
- W4317212783
- Cited by
- 184
- References
- 51
- FWCI
- 13.75
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/4.0/ (vor)
- creativecommons.org/licenses/by/4.0/ (am)
- creativecommons.org/licenses/by/4.0/ (tdm)
- Funders
- Kempestiftelserna (JCK-2144, JCK-2015.1)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.