The DOI matches its Crossref record; 113 of the 113 references it cites were found in a registry or an index.
The language of proteins: NLP, machine learning & protein sequences
Open accessAbstract
Natural language processing (NLP) is a field of computer science concerned with automated text and language analysis. In recent years, following a series of breakthroughs in deep and machine learning, NLP methods have shown overwhelming progress. Here, we review the success, promise and pitfalls of applying NLP algorithms to the study of proteins. Proteins, which can be represented as strings of amino-acid letters, are a natural fit to many NLP methods. We explore the conceptual similarities and differences between proteins and language, and review a range of protein-related tasks amenable to machine learning. We present methods for encoding the information of proteins as text and analyzing it with NLP methods, reviewing classic concepts such as bag-of-words, k-mers/n-grams and text search, as well as modern techniques such as word embedding, contextualized embedding, deep learning and neural language models. In particular, we focus on recent innovations such as masked language modeling, self-supervised learning and attention-based models. Finally, we discuss trends and challenges in the intersection of NLP and protein research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“The method has been popularized with efficient algorithms such as word2vec [59,70,42,16].”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Structural basis for the recognition of SARS-CoV-2 by full-length human ACE2
Yan, R., Zhang, Y., Li, Y., Xia, L., Guo, Y., & Zhou, Q. (2020) · Science · doi:10.1126/science.abb2762 · Cited by 5,040
MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Steinegger, M., & Söding, J. (2017) · Nature Biotechnology · doi:10.1038/nbt.3988 · Cited by 4,840
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2068 · Cited by 4,545
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2016) · arXiv · doi:10.48550/arxiv.1607.01759 · Cited by 4,542
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · cited 1×
“Arguably, the field of NLP has recently reached this milestone, thanks to deep language models such as ULMFiT, BERT, XLNet and a range of other BERT variations (e.g. ALBERT, RoBERTA) [30,27,114,77,50,55,38].”
BackgroundGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
Improved protein structure prediction using potentials from deep learning
Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1923-7 · Cited by 3,378
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 4×
“As of the present day, fine-tuning of pretrained deep models have shown considerable promise for improving our ability to solve problems, even on small data [79,83,19].”
BackgroundInfluentialUnderstanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838 · cited 1×
“Deep-learning models also easily overfit (even on random noise), and may not necessarily generalize to new, unseen data [120,118].”
BackgroundCodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Prediction and Entropy of Printed English
Shannon, C. E. (1951) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1951.tb01366.x · Cited by 2,686
HHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignment
Remmert, M., Biegert, A., Hauser, A., & Söding, J. (2012) · Nature Methods · doi:10.1038/nmeth.1818 · Cited by 2,288
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.10555 · Cited by 1,844
Yu, F., Koltun, V., & Funkhouser, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.75 · Cited by 1,804
The Unreasonable Effectiveness of Data
Halevy, A., Norvig, P., & Pereira, F. (2009) · IEEE Intelligent Systems · doi:10.1109/mis.2009.36 · Cited by 1,767
OpenNMT: Open-Source Toolkit for Neural Machine Translation
Klein, G., Kim, Y., Deng, Y., Senellart, J., & Rush, A. (2017) · Proceedings of ACL 2017, System Demonstrations · doi:10.18653/v1/p17-4012 · Cited by 1,749
(2018) · International Conference on Learning Representations · Cited by 1,621
Deep learning for computational biology
Angermueller, C., Pärnamaa, T., Parts, L., & Stegle, O. (2016) · Molecular Systems Biology · doi:10.15252/msb.20156651 · Cited by 1,473 · cited 1×
“Past decades have seen a continuous trickle of statistical and machinelearning algorithms from the field of NLP into bioinformatics [67,117,66,6,34,60,88,109,117,105].”
BackgroundDeep learning in bioinformatics
Min, S., Lee, B., & Yoon, S. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbw068 · Cited by 1,465
Ji, Y., Zhou, Z., Liu, H., & Davuluri, R. V. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab083 · Cited by 1,303
word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method
Goldberg, Y., & Levy, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.3722 · Cited by 1,245
DeepLoc: prediction of protein subcellular localization using deep learning
Almagro Armenteros, J. J., Sønderby, C. K., Sønderby, S. K., Nielsen, H., & Winther, O. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx431 · Cited by 1,143
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131 · cited 4×
“lutional networks pretrained on 3D-structure contact prediction [13,116], LSTM language models [3], as well as non-deep-”
BackgroundInfluential(2017) · International Conference on Learning Representations · Cited by 1,075
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates
Kudo, T. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1007 · Cited by 1,021 · cited 1×
“Common examples are WordPiece, SentencePiece and Byte Pair Encoding (BPE) [92,48], where a vocabulary is initialized to individual characters, and the most frequent combinations of symbols in the vocabulary are iteratively merged into the vocabulary.”
BackgroundNeural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
THE SPECTRUM KERNEL: A STRING KERNEL FOR SVM PROTEIN CLASSIFICATION
LESLIE, C., ESKIN, E., & NOBLE, W. S. (2001) · Biocomputing 2002 · doi:10.1142/9789812799623_0053 · Cited by 927
Continuous Distributed Representation of Biological Sequences for Deep Proteomics and Genomics
Asgari, E., & Mofrad, M. R. K. (2015) · PLOS ONE · doi:10.1371/journal.pone.0141287 · Cited by 821
Boutet, E., Lieberherr, D., Tognolli, M., Schneider, M., & Bairoch, A. (2007) · Plant Bioinformatics · doi:10.1007/978-1-59745-535-0_4 · Cited by 763
CTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · cited 6×
“They are slow to train, and can under-perform simpler methods (such as BLAST-based nearest-neighbour search or logistic regression over BoW representation [79,58,2].”
BackgroundMethodInfluentialCAPRI: A Critical Assessment of PRedicted Interactions
Janin, J., Henrick, K., Moult, J., Eyck, L. T., Sternberg, M. J. E., Vajda, S., Vakser, I., & Wodak, S. J. (2003) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.10381 · Cited by 703
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 1×
“1755 actions between proteins) challenging [23].”
Modeling the language of life – Deep Learning Protein Sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · doi:10.1101/614313 · Cited by 606
Modeling aspects of the language of life through transfer-learning protein sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3220-8 · Cited by 606 · cited 4×
“There have been works using contextualized embedding models (e.g. ELMO) on proteins for supervised-learning tasks (such as GO annotation, subcellular localization or structure prediction), as well as clustering sequences based on the learned representations [15,29,54,34].”
BackgroundMethodInfluentialLearned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · cited 5×
“A CoVe model pretrained on translation was then used on other NLP tasks.”
BackgroundMethodInfluentialMismatch string kernels for discriminative protein classification
Leslie, C. S., Eskin, E., Cohen, A., Weston, J., & Noble, W. S. (2004) · Bioinformatics · doi:10.1093/bioinformatics/btg431 · Cited by 561
Critical assessment of methods of protein structure prediction (CASP)—Round XIII
Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., & Moult, J. (2019) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25823 · Cited by 506
Global analysis of protein folding using massively parallel design, synthesis, and testing
Rocklin, G. J., Chidyausiku, T. M., Goreshnik, I., Ford, A., Houliston, S., Lemak, A., Carter, L., Ravichandran, R., Mulligan, V. K., Chevalier, A., et al. (2017) · Science · doi:10.1126/science.aan0693 · Cited by 501 · cited 1×
“gene ontology, GO) such as antiviral activity [79,64,41,84,5,25].”
BackgroundBig Self-Supervised Models are Strong Semi-Supervised Learners
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., & Hinton, G. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10029 · Cited by 421
An expanded evaluation of protein function prediction methods shows an improvement in accuracy
Jiang, Y., Oron, T. R., Clark, W. T., Bankapur, A. R., D’Andrea, D., Lepore, R., Funk, C. S., Kahanda, I., Verspoor, K. M., Ben-Hur, A., et al. (2016) · Genome Biology · doi:10.1186/s13059-016-1037-6 · Cited by 396
Self-citationLearning the language of viral evolution and escape
Hie, B., Zhong, E. D., Berger, B., & Bryson, B. (2021) · Science · doi:10.1126/science.abd7331 · Cited by 361 · cited 1×
“Language models have been used to understand and predict viral mutations that evade neutralizing antibodies [35].”
BackgroundProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.07.12.199554 · Cited by 354
Rao, R., Liu, J., Verkuil, R., Meier, J., Canny, J. F., Abbeel, P., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.02.12.430858 · Cited by 350
Unsupervised learning of natural languages
Solan, Z., Horn, D., Ruppin, E., & Edelman, S. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0409746102 · Cited by 346
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338
Using deep learning to annotate the protein universe
Bileschi, M. L., Belanger, D., Bryant, D. H., Sanderson, T., Carter, B., Sculley, D., Bateman, A., DePristo, M. A., & Colwell, L. J. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01179-w · Cited by 332 · cited 1×
“There have been works using contextualized embedding models (e.g. ELMO) on proteins for supervised-learning tasks (such as GO annotation, subcellular localization or structure prediction), as well as clustering sequences based on the learned representations [15,29,54,34].”
MethodHöglund, A., Dönnes, P., Blum, T., Adolph, H.-W., & Kohlbacher, O. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl002 · Cited by 325
Learned protein embeddings for machine learning
Yang, K. K., Wu, Z., Bedbrook, C. N., & Arnold, F. H. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty178 · Cited by 311
Simplified amino acid alphabets for protein fold recognition and implications for folding
Murphy, L. R., Wallqvist, A., & Levy, R. M. (2000) · Protein Engineering, Design and Selection · doi:10.1093/protein/13.3.149 · Cited by 235
Learning protein sequence embeddings using information from structure
Bepler, T., & Berger, B. (2019) · arXiv · doi:10.48550/arxiv.1902.08661 · Cited by 219
ProGen: Language Modeling for Protein Generation
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P.-S., & Socher, R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.07.982272 · Cited by 212
The Shannon information entropy of protein sequences
Strait, B. J., & Dewey, T. G. (1996) · Biophysical Journal · doi:10.1016/s0006-3495(96)79210-x · Cited by 191
UDSMProt: universal deep sequence models for protein classification
Strodthoff, N., Wagner, P., Wenzel, M., & Samek, W. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa003 · Cited by 189
Wen, B., Zeng, W., Liao, Y., Shi, Z., Savage, S. R., Jiang, W., & Zhang, B. (2020) · PROTEOMICS · doi:10.1002/pmic.201900335 · Cited by 184
Deep learning models in genomics; are we there yet?
Koumakis, L. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.06.017 · Cited by 176
Embeddings from deep learning transfer GO annotations beyond homology
Littmann, M., Heinzinger, M., Dallago, C., Olenyi, T., & Rost, B. (2021) · Scientific Reports · doi:10.1038/s41598-020-80786-0 · Cited by 175
DeepSig: deep learning improves signal peptide detection in proteins
Savojardo, C., Martelli, P. L., Fariselli, P., & Casadio, R. (2018) · Bioinformatics · doi:10.1093/bioinformatics/btx818 · Cited by 160
Proteomic signatures: Amino acid and oligopeptide compositions differentiate among phyla
Pe’er, I., Felder, C. E., Man, O., Silman, I., Sussman, J. L., & Beckmann, J. S. (2003) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.10559 · Cited by 154
DeepType: Multilingual Entity Linking by Neural Type System Evolution
Raiman, J., & Raiman, O. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12008 · Cited by 146
Reduced amino acid alphabet is sufficient to accurately recognize intrinsically disordered protein
Weathers, E. A., Paulaitis, M. E., Woolf, T. B., & Hoh, J. H. (2004) · FEBS Letters · doi:10.1016/j.febslet.2004.09.036 · Cited by 140
Schnoes, A. M., Ream, D. C., Thorman, A. W., Babbitt, P. C., & Friedberg, I. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003063 · Cited by 129 · cited 1×
“training on protein–protein interactions is more problematic due to the far smaller, sparser and more biased data [94,89].”
BackgroundFFPred 3: feature-based function prediction for all Gene Ontology domains
Cozzetto, D., Minneci, F., Currant, H., & Jones, D. T. (2016) · Scientific Reports · doi:10.1038/srep31865 · Cited by 127
Cooperativity within proximal phosphorylation sites is revealed from large-scale proteomics data
Schweiger, R., & Linial, M. (2010) · Biology Direct · doi:10.1186/1745-6150-5-6 · Cited by 126
Self-citationBudowski-Tal, I., Nov, Y., & Kolodny, R. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0914097107 · Cited by 120
ProFET: Feature engineering captures high-level protein functions
Ofer, D., & Linial, M. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv345 · Cited by 120
Self-citationHow does protein synthesis give rise to the 3D‐structure?
Ptitsyn, O. B. (1991) · FEBS Letters · doi:10.1016/0014-5793(91)80799-9 · Cited by 110
A High Efficient Biological Language Model for Predicting Protein–Protein Interactions
Wang, Y., You, Z., Yang, S., Xiao, L., Jiang, T.-H., & Zhou, X. (2019) · Cells · doi:10.3390/cells8020122 · Cited by 109
Protein–protein interaction predictions using text mining methods
Papanikolaou, N., Pavlopoulos, G. A., Θεοδοσίου, Θ., & Iliopoulos, I. (2014) · Methods · doi:10.1016/j.ymeth.2014.10.026 · Cited by 101
Deep Nets: What have They Ever Done for Vision?
Yuille, A. L., & Liu, C. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-020-01405-z · Cited by 98 · cited 1×
“Deep-learning models also easily overfit (even on random noise), and may not necessarily generalize to new, unseen data [120,118].”
BackgroundClanTox: a classifier of short animal toxins
Naamati, G., Askenazi, M., & Michal Linial, B. (2009) · Nucleic Acids Research · doi:10.1093/nar/gkp299 · Cited by 94
Neural networks versus Logistic regression for 30 days all-cause readmission prediction
Allam, A., Nagy, M. L., Thoma, G. R., & Krauthammer, M. O. (2019) · Scientific Reports · doi:10.1038/s41598-019-45685-z · Cited by 94
One neuron versus deep learning in aftershock prediction
Mignan, A., & Broccardo, M. (2019) · Nature · doi:10.1038/s41586-019-1582-8 · Cited by 93
Signal Peptides Generated by Attention-Based Neural Networks
Wu, Z., Yang, K., Liszka, M. J., Lee, A., Batzilla, A., Wernick, D. G., Weiner, D. P., & Arnold, F. H. (2020) · ACS Synthetic Biology · doi:10.1021/acssynbio.0c00219 · Cited by 91
Asgari, E., McHardy, A. C., & Mofrad, M. R. K. (2019) · Scientific Reports · doi:10.1038/s41598-019-38746-w · Cited by 89
Grammar of protein domain architectures
Yu, L., Tanwar, D. K., Penha, E. D. D. S., Wolf, Y. I., Koonin, E. V., & Basu, M. K. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1814684116 · Cited by 82
Machine learning methods for predictive proteomics
Barla, A., Jurman, G., Riccadonna, S., Merler, S., Chierici, M., & Furlanello, C. (2007) · Briefings in Bioinformatics · doi:10.1093/bib/bbn008 · Cited by 71
Reduced amino acid alphabets exhibit an improved sensitivity and selectivity in fold assignment
Peterson, E. L., Kondev, J., Theriot, J. A., & Phillips, R. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp164 · Cited by 67
Gillis, J., & Pavlidis, P. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-s3-s15 · Cited by 61
The origin of the genetic code and of the earliest oligopeptides
Trifonov, E. N. (2009) · Research in Microbiology · doi:10.1016/j.resmic.2009.05.004 · Cited by 51
Nematzadeh, A., Meylan, S. C., & Griffiths, T. L. (2017) · eScholarship (California Digital Library) · Cited by 49
Dutta, D., & Chen, T. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btl645 · Cited by 48
Neural Attentive Bag-of-Entities Model for Text Classification
Yamada, I., & Shindo, H. (2019) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k19-1052 · Cited by 44
NeuroPID: a predictor for identifying neuropeptide precursors from metazoan proteomes
Ofer, D., & Michal Linial, B. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt725 · Cited by 43
Self-citationContextual Word Representations: A Contextual IntroductionPreprint
Smith, N. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06006 · Cited by 41
Akhtar, M. N., Southey, B. R., Andrén, P. E., Sweedler, J. V., & Rodriguez‐Zas, S. L. (2012) · Journal of Proteome Research · doi:10.1021/pr3007123 · Cited by 30
ASAP: a machine learning framework for local protein properties
Brandes, N., Ofer, D., & Linial, M. (2016) · Database · doi:10.1093/database/baw133 · Cited by 25
Self-citationThe complete peptide dictionary – A meta‐proteomics resource
Askenazi, M., Marto, J. A., & Linial, M. (2010) · PROTEOMICS · doi:10.1002/pmic.201000270 · Cited by 21
Self-citationDeep learning embedder method and tool for mass spectra similarity search
Qin, C., Luo, X., Deng, C., Shu, K., Zhu, W., Griss, J., Hermjakob, H., Bai, M., & Pérez‐Riverol, Y. (2020) · Journal of Proteomics · doi:10.1016/j.jprot.2020.104070 · Cited by 21
Language modelling for biological sequences – curated datasets and baselinesPreprint
Armenteros, J. J. A., Johansen, A. R., Winther, O., & Nielsen, H. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.09.983585 · Cited by 19
Sadka, T., & Linial, M. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti1035 · Cited by 12
Self-citationOn biases of attention in scientific discovery
Singer, U., Radinsky, K., & Horvitz, E. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa1036 · Cited by 12
Aligning the Pretraining and Finetuning Objectives of Language ModelsPreprint
Pierse, N. W., & Lü, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02000 · Cited by 2
Cited by
ProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009
Self-citationGenome-wide prediction of disease variant effects with a deep protein language model
Brandes, N., Goldman, G., Wang, C. H., Ye, C. J., & Ntranos, V. (2023) · Nature Genetics · doi:10.1038/s41588-023-01465-0 · Cited by 421
Self-citationRhea, the reaction knowledgebase in 2022
Bansal, P., Morgat, A., Axelsen, K. B., Muthukrishnan, V., Coudert, E., Aimo, L., Hyka-Nouspikel, N., Gasteiger, E., Kerhornou, A., Neto, T. B., et al. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkab1016 · Cited by 253
Applications of transformer-based language models in bioinformatics: a survey
Zhang, S., Fan, R., Liu, Y., Chen, S., Liu, Q., & Zeng, W. (2023) · Bioinformatics Advances · doi:10.1093/bioadv/vbad001 · Cited by 176
Embeddings from protein language models predict conservation and variant effects
(2021) · doi:10.1007/s00439-021-02411-y · Cited by 174
Metadata
- Authors
- Ofer, D., Brandes, N., & Linial, M.
- Venue
- Computational and Structural Biotechnology Journal
- Published
- 2021
- Volume
- 19
- Pages
- 1750-1758
- ISSN
- 2001-0370 (ISSN-L)
- OpenAlex
- W3144701084
- Cited by
- 339
- References
- 113
- FWCI
- 15.50
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.