The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
Evaluating Protein Transfer Learning with TAPE
Open accessAbstract
Protein modeling is an increasingly popular area of machine learning research. Semi-supervised learning has emerged as an important paradigm in protein modeling due to the high cost of acquiring supervised protein labels, but the current literature is fragmented when it comes to datasets and standardized evaluation techniques. To facilitate progress in this field, we introduce the Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology. We curate tasks into specific training, validation, and test splits to ensure that each task tests biologically relevant generalization that transfers to real-life scenarios. We bench-mark a range of approaches to semi-supervised protein representation learning, which span recent work as well as canonical sequence learning techniques. We find that self-supervised pretraining is helpful for almost all models on all tasks, more than doubling performance in some cases. Despite this increase, in several cases features learned by self-supervised pretraining still lag behind features extracted by state-of-the-art non-neural techniques. This gap in performance suggests a huge opportunity for innovative architecture design and improved modeling paradigms that better capture the signal in biological sequences. TAPE will help the machine learning community focus effort on scientifically relevant problems. Toward this end, all data and code used to run these experiments are available at https://github.com/songlab-cal/tape .
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“They also learn summary representations, powerful features that can be used to better capture sequence relationships and link sequence to function via transfer learning 27,28,32-35.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [33]
In the citing paper
“Since then, various language model-based protein embedding methods have been applied to these and other protein prediction problems through transfer learning, including protein phenotype prediction 28,32-34, residue-residue contact prediction 32,70, fold recognition 33, protein-protein 71,72 and protein-drug interaction prediction 35,73.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Basic local alignment search tool
Altschul, S. F., Gish, W., Miller, W., Myers, E. W., & Lipman, D. J. (1990) · Journal of Molecular Biology · doi:10.1006/jmbi.1990.9999 · Cited by 98,669
Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
Altschul, S. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.17.3389 · Cited by 73,302
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Identification of common molecular subsequences
Smith, T. F., & Waterman, M. S. (1981) · Journal of Molecular Biology · doi:10.1016/0022-2836(81)90087-5 · Cited by 9,962
UniProt: a worldwide hub of protein knowledge
The UniProt Consortium (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1049 · Cited by 7,698
Research works printed but not linked: 4
- 2018, Association for Computational Linguistics, 1, 2227
- 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 8
- 2017, Advances in Neural Information Processing Systems, 30, 5998
- Tianqi Chen , Bing Xu , Chiyuan Zhang , and Carlos Guestrin . Training Deep Nets with Sublinear Memory Cost. arXiv, 2016.
Cited by
Sorted by citations
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
Self-citationRives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329
A guide to machine learning for biologists
Greener, J. G., Kandathil, S. M., Moffat, L., & Jones, D. T. (2022) · Nature Reviews Molecular Cell Biology · doi:10.1038/s41580-021-00407-0 · Cited by 2,127
Large language models generate functional protein sequences across diverse families
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., et al. (2023) · Nature Biotechnology · doi:10.1038/s41587-022-01618-2 · Cited by 1,078
ProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009
ProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795
Language models enable zero-shot prediction of the effects of mutations on protein function
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.07.09.450648 · Cited by 635
Self-citationLearning the protein language: Evolution, structure, and function
Bepler, T., & Berger, B. (2021) · Cell Systems · doi:10.1016/j.cels.2021.05.017 · Cited by 483
ProGen2: Exploring the boundaries of protein language models
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.002 · Cited by 480
Accelerated antimicrobial discovery via deep generative models and molecular dynamics simulations
Das, P., Sercu, T., Wadhawan, K., Padhi, I., Gehrmann, S., Cipcigan, F. S., Chenthamarakshan, V., Strobelt, H., dos Santos, C., Chen, P.-Y., et al. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-021-00689-x · Cited by 506
Efficient evolution of human antibodies from general protein language models
Hie, B., Shanker, V. R., Xu, D., Bruun, T. U. J., Weidenbacher, P. A. B., Tang, S., Wu, W. C. H., Pak, J. E., & Kim, P. S. (2023) · Nature Biotechnology · doi:10.1038/s41587-023-01763-2 · Cited by 472
Learning inverse folding from millions of predicted structuresPreprint
Hsu, C., Verkuil, R., Liu, J. J., Lin, Z., Hie, B., Sercu, T., Lerer, A. K., & Rives, A. W. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.10.487779 · Cited by 492
Genome-wide prediction of disease variant effects with a deep protein language model
Brandes, N., Goldman, G., Wang, C. H., Ye, C., & Ntranos, V. (2023) · Nature Genetics · doi:10.1038/s41588-023-01465-0 · Cited by 489
Machine Learning in Enzyme Engineering
Mazurenko, S., Prokop, Z., & Damborský, J. (2019) · ACS Catalysis · doi:10.1021/acscatal.9b04321 · Cited by 417
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
“We use a deep bidirectional Transformer encoder model (Devlin et al., 2018; Vaswani et al., 2017) and process data at the character-level, corresponding to individual amino-acids in our application.”
MethodProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised LearningPreprint
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.07.12.199554 · Cited by 424
Rao, R., Liu, J., Verkuil, R., Meier, J., Canny, J. F., Abbeel, P., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.02.12.430858 · Cited by 350
Self-citationLearning the language of viral evolution and escape
Hie, B., Zhong, E. D., Berger, B., & Bryson, B. (2021) · Science · doi:10.1126/science.abd7331 · Cited by 361
Deep Learning applications for COVID-19
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal of Big Data · doi:10.1186/s40537-020-00392-9 · Cited by 350
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338
The language of proteins: NLP, machine learning & protein sequences
Ofer, D., Brandes, N., & Linial, M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.03.022 · Cited by 365
Transformer protein language models are unsupervised structure learners
Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A. (2020) · doi:10.1101/2020.12.15.422761 · Cited by 341
Self-citationProtein design and variant prediction using autoregressive generative models
Shin, J.-E., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A. C., & Marks, D. S. (2021) · Nature Communications · doi:10.1038/s41467-021-22732-w · Cited by 320
Xie, J., Zhang, J., Sun, J., Ma, Z., Qin, L., Li, G., Zhou, H., & Zhan, Y. (2022) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2022.3194600 · Cited by 335
Utilizing graph machine learning within drug discovery and development
Gaudelet, T., Day, B., Jamasb, A. R., Soman, J., Regep, C., Liu, G., Hayter, J. B. R., Vickers, R., Roberts, C., Tang, J., et al. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab159 · Cited by 313
flDPnn: Accurate intrinsic disorder prediction with putative propensities of disorder functions
Hu, G., Katuwawala, A., Wang, K., Wu, Z., Ghadermarzi, S., Gao, J., & Kurgan, L. (2021) · Nature Communications · doi:10.1038/s41467-021-24773-7 · Cited by 292
Opportunities and Challenges for Machine Learning-Assisted Enzyme Engineering
Yang, J., Li, F.-Z., & Arnold, F. H. (2024) · ACS Central Science · doi:10.1021/acscentsci.3c01275 · Cited by 268
SaProt: Protein Language Modeling with Structure-aware VocabularyPreprint
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., & Yuan, F. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.01.560349 · Cited by 264
PredictProtein - Predicting Protein Structure and Function for 29 Years
Bernhofer, M., Dallago, C., Karl, T., Satagopam, V., Heinzinger, M., Littmann, M., Olenyi, T., Qiu, J., Schütze, K., Yachdav, G., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab354 · Cited by 246
Fine-tuning protein language models boosts predictions across diverse tasks
Schmirler, R., Heinzinger, M., & Rost, B. (2024) · Nature Communications · doi:10.1038/s41467-024-51844-2 · Cited by 229
Deep Learning in Protein Structural Modeling and Design
Gao, W., Mahajan, S. P., Sulam, J., & Gray, J. J. (2020) · Patterns · doi:10.1016/j.patter.2020.100142 · Cited by 227
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.07.982272 · Cited by 383
Informed training set design enables efficient machine learning-assisted directed protein evolution
Wittmann, B. J., Yue, Y., & Arnold, F. H. (2021) · Cell Systems · doi:10.1016/j.cels.2021.07.008 · Cited by 215
Generating functional protein variants with variational autoencoders
Hawkins‐Hooker, A., Depardieu, F., Baur, S., Couairon, G., Chen, A., & Bikard, D. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008736 · Cited by 217
Learning protein fitness models from evolutionary and assay-labeled data
Hsu, C., Nisonoff, H. M., Fannjiang, C., & Listgarten, J. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01146-5 · Cited by 240
Learning functional properties of proteins with language models
Ünsal, S., Ataş, H., Albayrak, M., Turhan, K., Acar, A. C., & Doğan, T. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00457-9 · Cited by 206
Controllable protein design with language models
Ferruz, N., & Höcker, B. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00499-z · Cited by 209
ProteInfer, deep neural networks for protein functional inference
Sanderson, T., Bileschi, M. L., Belanger, D., & Colwell, L. J. (2023) · eLife · doi:10.7554/elife.80942 · Cited by 197
Turnover number predictions for kinetically uncharacterized enzymes using machine and deep learning
Kroll, A., Rousset, Y., Hu, X.-P., Liebrand, N. A., & Lercher, M. J. (2023) · Nature Communications · doi:10.1038/s41467-023-39840-4 · Cited by 186
Transformer-based deep learning for predicting protein properties in the life sciences
Chandra, A. A., Tünnermann, L., Löfstedt, T., & Gratz, R. (2023) · eLife · doi:10.7554/elife.82819 · Cited by 202
Jhong, J.-H., Yao, L., Pang, Y., Li, Z., Chung, C., Wang, R., Li, S., Li, W., Luo, M., Ma, R., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab1080 · Cited by 189
Ankh ☥: Optimized Protein Language Model Unlocks General-Purpose ModellingPreprint
Elnaggar, A., Essam, H., Salah-Eldin, W., Moustafa, W., Elkerdawy, M., Rochereau, C., & Rost, B. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.16.524265 · Cited by 243
TANKBind: Trigonometry-Aware Neural NetworKs for Drug-Protein Binding Structure Prediction
Lu, W., Wu, Q., Zhang, J., Rao, J., Li, C., & Zheng, S. (2022) · neural information processing systems · doi:10.52202/068431-0525 · Cited by 181
Pre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2023) · ACM Computing Surveys · doi:10.1145/3611651 · Cited by 238
ECNet is an evolutionary context-integrated deep learning framework for protein engineering
Luo, Y., Jiang, G., Yu, T., Liu, Y., Vo, L. T., Ding, H., Su, Y., Qian, W. W., Zhao, H., & Peng, J. (2021) · Nature Communications · doi:10.1038/s41467-021-25976-8 · Cited by 174
Learning meaningful representations of protein sequences
Detlefsen, N. S., Hauberg, S., & Boomsma, W. (2022) · Nature Communications · doi:10.1038/s41467-022-29443-w · Cited by 171
Blockwise Self-Attention for Long Document Understanding
Qiu, J., Ma, H., Levy, O., Yih, W.-T., Wang, S., & Tang, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.232 · Cited by 247
Advances in machine learning for directed evolution
Wittmann, B. J., Johnston, K. E., Wu, Z., & Arnold, F. H. (2021) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2021.01.008 · Cited by 166
AMPGAN v2: Machine Learning-Guided Design of Antimicrobial Peptides
Oort, C. M. V., Ferrell, J. B., Remington, J. M., Wshah, S., & Li, J. (2021) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c01441 · Cited by 183
Deciphering the language of antibodies using self-supervised learning
Leem, J., Mitchell, L., Farmery, J. H. R., Barton, J., & Galson, J. D. (2022) · Patterns · doi:10.1016/j.patter.2022.100513 · Cited by 169
Hie, B., Yang, K., & Kim, P. S. (2022) · Cell Systems · doi:10.1016/j.cels.2022.01.003 · Cited by 151
Single-sequence protein structure prediction using supervised transformer protein language models
Wang, W., Peng, Z., & Yang, J. (2022) · Nature Computational Science · doi:10.1038/s43588-022-00373-3 · Cited by 150
Narayanan, H., Dingfelder, F., Butté, A., Lorenzen, N., Sokolov, M., & Arosio, P. (2021) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2020.12.004 · Cited by 151
Anticancer peptides prediction with deep representation learning features
Lv, Z., Cui, F., Zou, Q., Zhang, L., & Xu, L. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab008 · Cited by 145
Protein remote homology detection and structural alignment using deep learning
Hamamsy, T., Morton, J. T., Blackwell, R., Berenberg, D., Carriero, N. J., Gligorijević, V., Strauss, C. E. M., Leman, J. K., Cho, K., & Bonneau, R. (2023) · Nature Biotechnology · doi:10.1038/s41587-023-01917-2 · Cited by 157
Protein sequence design with deep generative models
Wu, Z., Johnston, K. E., Arnold, F. H., & Yang, K. (2021) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2021.04.004 · Cited by 152
Active learning-assisted directed evolution
Yang, J., Lal, R., Bowden, J. C., Astudillo, R., Hameedi, M. A., Kaur, S., Hill, M., Yue, Y., & Arnold, F. H. (2025) · Nature Communications · doi:10.1038/s41467-025-55987-8 · Cited by 144
Protein language model embeddings for fast, accurate, alignment-free protein structure prediction
Weissenow, K., Heinzinger, M., & Rost, B. (2021) · doi:10.1101/2021.07.31.454572 · Cited by 139
Convolutions are competitive with transformers for protein sequence pretraining
Yang, K. K., Fusi, N., & Lu, A. X. (2024) · Cell Systems · doi:10.1016/j.cels.2024.01.008 · Cited by 137
Light attention predicts protein location from the language of life
Stärk, H., Dallago, C., Heinzinger, M., & Rost, B. (2021) · Bioinformatics Advances · doi:10.1093/bioadv/vbab035 · Cited by 130
Fang, X., Wang, F., Liu, L., He, J., Lin, D., Xiang, Y., Zhu, K., Zhang, X., Wu, H., Li, H., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00721-6 · Cited by 132
FLIP: Benchmark tasks in fitness landscape inference for proteinsPreprint
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.11.09.467890 · Cited by 142
Self-citationScientific Large Language Models: A Survey on Biological & Chemical Domains
Zhang, Q., Ding, K., Lv, T., Wang, X., Yin, Q., Zhang, Y., Jing, Y., Wang, Y., Li, X., Xiang, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3715318 · Cited by 216
Transformer models in biomedicine
Madan, S., Lentzen, M., Brandt, J., Rueckert, D., Hofmann‐Apitius, M., & Fröhlich, H. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02600-5 · Cited by 146
Functional annotation of enzyme-encoding genes using deep learning with transformer layers
Kim, G. B., Kim, J. Y., Lee, J. A., Norsigian, C. J., Palsson, B. Ø., & Lee, S. Y. (2023) · Nature Communications · doi:10.1038/s41467-023-43216-z · Cited by 122
Machine learning modeling of family wide enzyme-substrate specificity screens
Goldman, S., Das, R., Yang, K. K., & Coley, C. W. (2022) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1009853 · Cited by 120
Sparks of function by de novo protein design
Chu, A. E., Lu, T., & Huang, P. (2024) · Nature Biotechnology · doi:10.1038/s41587-024-02133-2 · Cited by 131
Foundation models in bioinformatics
Guo, F., Guan, R., Li, Y., Liu, Q., Wang, X., Yang, C., & Wang, J. (2025) · National Science Review · doi:10.1093/nsr/nwaf028 · Cited by 127
A foundational large language model for edible plant genomes
Mendoza‐Revilla, J., Trop, E., Gonzalez, L., Roller, M., Dalla-Torre, H., de Almeida, B. P., Richard, G., Caton, J., Carranza, N. L., Skwark, M. J., et al. (2024) · Communications Biology · doi:10.1038/s42003-024-06465-2 · Cited by 129
A new age in protein design empowered by deep learning
Khakzad, H., Igashov, I., Schneuing, A., Goverde, C. A., Bronstein, M. M., & Correia, B. E. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.006 · Cited by 121
Frozen Pretrained Transformers as Universal Computation Engines
Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20729 · Cited by 113
“We use the datasets provided by TAPE (Rao et al. 2019; Fox, Brenner, and Chandonia 2013; Hou, Adhikari, and Cheng 2018), where the train/test split is generated by holding out certain evolutionary groups.”
BackgroundMethodSelf-citationxTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of ProteinPreprint
Chen, B., Cheng, X., Li, P., Geng, Y.-A., Gong, J., Li, S., Bei, Z.-L., Tan, X., Wang, B., Zeng, X., et al. (2024) · bioRxiv · doi:10.1101/2023.07.05.547496 · Cited by 157
Machine learning to navigate fitness landscapes for protein engineering
Freschlin, C. R., Fahlberg, S. A., & Romero, P. A. (2022) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2022.102713 · Cited by 111
Multiple sequence alignment-based RNA language model and its application to structural inference
Zhang, Y., Lang, M., Jiang, J., Gao, Z., Xu, F., Litfin, T., Chen, K., Singh, J., Huang, X., Song, G., et al. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkad1031 · Cited by 112
Learned Embeddings from Deep Learning to Visualize and Predict Protein Sets
Dallago, C., Schütze, K., Heinzinger, M., Olenyi, T., Littmann, M., Lu, A. X., Yang, K., Min, S., Yoon, S., Morton, J. T., et al. (2021) · Current Protocols · doi:10.1002/cpz1.113 · Cited by 109
Mol‐BERT: An Effective Molecular Representation with BERT for Molecular Property Prediction
Li, J., & Jiang, X. (2021) · Wireless Communications and Mobile Computing · doi:10.1155/2021/7181815 · Cited by 112
Biological Sequence Classification: A Review on Data and General Methods
Ao, C., Jiao, S., Wang, Y., Yu, L., & Zou, Q. (2022) · Research · doi:10.34133/research.0011 · Cited by 103
PepNN: a deep attention model for the identification of peptide binding sites
Abdin, O., Nim, S., Wen, H., & Kim, P. M. (2022) · Communications Biology · doi:10.1038/s42003-022-03445-2 · Cited by 104
PLMSearch: Protein language model powers accurate and fast sequence search for remote homology
Liu, W., Wang, Z., You, R., Xie, C., Wei, H., Xiong, Y., Yang, J., & Zhu, S. (2024) · Nature Communications · doi:10.1038/s41467-024-46808-5 · Cited by 96
Li, L., Gupta, E., Spaeth, J., Shing, L., Jaimes, R., Engelhart, E., Lopez, R., Caceres, R. S., Bepler, T., & Walsh, M. E. (2023) · Nature Communications · doi:10.1038/s41467-023-39022-2 · Cited by 96
EEG Classification with Transformer-Based Models
Sun, J., Xie, J., & Zhou, H. (2021) · IEEE Global Conference on Life Sciences and Technologies (LifeTech) · doi:10.1109/lifetech52111.2021.9391844 · Cited by 97
A survey of drug-target interaction and affinity prediction methods via graph neural networks
Zhang, Y., Hu, Y., Han, N., Yang, A., Liu, X., & Cai, H. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107136 · Cited by 102
GEFA: Early Fusion Approach in Drug-Target Affinity Prediction
Nguyen, T. M., Nguyen, T., Le, T. M., & Tran, T. (2021) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2021.3094217 · Cited by 94
Machine learning approaches to predict drug efficacy and toxicity in oncology
Badwan, B. A., Liaropoulos, G., Kyrodimos, E., Skaltsas, D., Tsirigos, A., & Gorgoulis, V. G. (2023) · Cell Reports Methods · doi:10.1016/j.crmeth.2023.100413 · Cited by 92
NetGO 3.0: Protein Language Model Improves Large-Scale Functional Annotations
Wang, S., You, R., Liu, Y., Xiong, Y., & Zhu, S. (2023) · Genomics Proteomics & Bioinformatics · doi:10.1016/j.gpb.2023.04.001 · Cited by 91
Qiu, Y., & Wei, G. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad289 · Cited by 90
McInnes, G., Sharo, A. G., Koleske, M. L., Brown, J. E. H., Norstad, M., Adhikari, A. N., Wang, S., Brenner, S. E., Halpern, J., Koenig, B. A., et al. (2021) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2021.03.003 · Cited by 86
“Transformer-based approaches, frequently used in natural language processing, have been applied to functional predictions of variants in proteins.(91,92) Graph-CNNs have been used to make drug-binding predictions with protein structure data after being pre-trained with an unsupervised learning step.”
MethodLindorff-Larsen, K., & Kragelund, B. B. (2021) · Journal of Molecular Biology · doi:10.1016/j.jmb.2021.167196 · Cited by 85
Wang, Z., Combs, S. A., Brand, R. M., Calvo, M. R., Xu, P., Price, G., Golovach, N., Salawu, E. O., Wise, C., Ponnapalli, S. P., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10775-y · Cited by 89
Cheng, J., Bendjama, K., Rittner, K., & Malone, B. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab422 · Cited by 87
Improving protein succinylation sites prediction using embeddings from protein language model
Pokharel, S., Pratyush, P., Heinzinger, M., Newman, R. H., & KC, D. B. (2022) · Scientific Reports · doi:10.1038/s41598-022-21366-2 · Cited by 90
Representation learning applications in biological sequence analysis
Iuchi, H., Matsutani, T., Yamada, K., Iwano, N., Sumi, S., Hosoda, S., Zhao, S., Fukunaga, T., & Hamada, M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.05.039 · Cited by 88
Perceiver CPI: a nested cross-attention network for compound–protein interaction prediction
Nguyen, N.-Q., Jang, G., Kim, H., & Kang, J. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btac731 · Cited by 83
Enhancing scientific discoveries in molecular biology with deep generative models
Lopez, R., Gayoso, A., & Yosef, N. (2020) · Molecular Systems Biology · doi:10.15252/msb.20199198 · Cited by 82
Ultrafast homomorphic encryption models enable secure outsourcing of genotype imputation
Kim, M., Harmanci, A. O., Bossuat, J.-P., Carpov, S., Cheon, J. H., Chillotti, I., Cho, W., Froelicher, D., Gama, N., Georgieva, M., et al. (2021) · Cell Systems · doi:10.1016/j.cels.2021.07.010 · Cited by 80
Self-citation
Metadata
- Authors
- Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S.
- Affiliations
- University of California, Berkeley (US)
- Google (United States) (US)
- Corresponding author
- Rao, Roshan
- Bhattacharya, Nicholas
- Thomas, Neil
- Song, Yun S.
- ORCID
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2019-06-20
- OpenAlex
- W2951433247
- Cited by
- 719
- References
- 62
- Citations by year
- 2026: 55 · 2025: 114 · 2024: 150 · 2023: 148 · 2022: 118 · 2021: 100 · 2020: 31 · 2019: 3
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
- Funders
- Open Philanthropy Project
- National Institutes of Health
- Defense Advanced Research Projects Agency
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- protein representation learning, semi-supervised learning, protein embeddings, transfer learning, self-supervised pre-training, benchmark dataset, protein structure prediction, protein function prediction, sequence learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
