Transformer protein language models are unsupervised structure learners
Open accessAbstract
A bstract: Unsupervised contact prediction is central to uncovering physical, structural, and functional constraints for protein structure determination and design. For decades, the predominant approach has been to infer evolutionary constraints from a set of related sequences. In the past year, protein language models have emerged as a potential alternative, but performance has fallen short of state-of-the-art approaches in bioinformatics. In this paper we demonstrate that Transformer attention maps learn contacts from the unsupervised language modeling objective. We find the highest capacity models that have been trained to date already outperform a state-of-the-art unsupervised contact prediction pipeline, suggesting these pipelines can be replaced with a single forward pass of an end-to-end model. 1
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Attention-based networks are extremely powerful for modeling complicated interactions between entities, making them suitable for modeling proteins whose functions are largely determined by interactions between amino acids.
“Among many successful neural network architectures, attention-based networks are extremely powerful for modeling complicated interactions between entities, making them suitable for modeling proteins whose functions are largely determined by interactions between amino acids (34–37).”
From Deep learning guided optimization of human antibody against SARS-CoV-2 variants with broad neutralization (2022) · cites this paper as [37]
Various language model-based protein embedding methods have been applied to protein prediction problems through transfer learning, including protein phenotype prediction, residue-residue contact prediction, fold recognition, protein-protein and protein-drug interaction prediction.
“Since then, various language model-based protein embedding methods have been applied to these and other protein prediction problems through transfer learning, including protein phenotype prediction 28,32-34, residue-residue contact prediction 32,70, fold recognition 33, protein-protein 71,72 and protein-drug interaction prediction 35,73.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [70]
Increasing language model scale leads to continued improvements in downstream applications, such as residue-residue contact prediction.
“Recent works have shown that increasing language model scale leads to continued improvements in downstream applications, such as residue-residue contact prediction 70.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [70]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
mixOmics: An R package for ‘omics feature selection and multiple data integration
Rohart, F., Gautier, B., Singh, A., & Lê Cao, K.-A. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005752 · Cited by 4,056 · see citing works
Improved protein structure prediction using potentials from deep learning
Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1923-7 · Cited by 3,378 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286 · see citing works
Self-citationRives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · see citing works
Self-citationHHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignment
Remmert, M., Biegert, A., Hauser, A., & Söding, J. (2012) · Nature Methods · doi:10.1038/nmeth.1818 · Cited by 2,288 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“A similar scaling law has been observed previously for supervised secondary structure prediction (Rives et al., 2019), and parallels observations in the NLP community (Kaplan et al., 2020; Brown et al., 2020).”
BackgroundUniRef: comprehensive and non-redundant UniProt reference clusters
Suzek, B. E., Huang, H., McGarvey, P., Mazumder, R., & Wu, C. H. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm098 · Cited by 1,613 · see citing works
Morcos, F., Pagnani, A., Lunt, B., Bertolino, A., Marks, D. S., Sander, C., Zecchina, R., Onuchic, J. N., Hwa, T., & Weigt, M. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1111471108 · Cited by 1,434 · see citing works
Improved protein structure prediction using predicted interresidue orientations
Yang, J., Anishchenko, I., Park, H., Peng, Z., Ovchinnikov, S., & Baker, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914677117 · Cited by 1,421 · see citing works
Self-citationA Selected Core Microbiome Drives the Early Stages of Three Popular Italian Cheese Manufactures
De Filippis, F., La Storia, A., Stellato, G., Gatti, M., & Ercolini, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0089680 · Cited by 1,398 · see citing works
Hidden Markov model speed heuristic and iterative HMM search procedure
Johnson, L. S., Eddy, S. R., & Portugaly, E. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-431 · Cited by 1,332 · see citing works
Protein 3D Structure Computed from Evolutionary Sequence Variation
Marks, D. S., Colwell, L. J., Sheridan, R., Hopf, T. A., Pagnani, A., Zecchina, R., & Sander, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028766 · Cited by 1,144 · see citing works
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131 · cited 5× · see citing works
“Lastly, Alley et al. (2019) presented a method of fine-tuning where a pretrained language model is further trained on the MSA of the sequence of interest (‘evotuning’).”
BackgroundMethodInfluentialIdentification of direct residue contacts in protein–protein interaction by message passing
Weigt, M., White, R. A., Szurmant, H., Hoch, J. A., & Hwa, T. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0805923106 · Cited by 1,025 · see citing works
Simulation physiologischer Regelkreise mit der objektorientierten Modellbibliothek “HumanLib”
Brunberg, A., Spillner, J. W., Autschbach, R., & Abel, D. (2011) · at - Automatisierungstechnik · doi:10.1524/auto.2011.0951 · Cited by 1,008 · see citing works
Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model
Wang, S., Sun, S., Li, Z., Zhang, R., & Xu, J. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005324 · Cited by 931 · see citing works
E2F1 Regulates Cellular Growth by mTORC1 Signaling
Real, S., Meo-Evoli, N., Espada, L., & Tauler, A. (2011) · PLoS ONE · doi:10.1371/journal.pone.0016163 · Cited by 816 · see citing works
Jones, D. T., Buchan, D. W. A., Cozzetto, D., & Pontil, M. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr638 · Cited by 760 · see citing works
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · see citing works
Self-citationKamisetty, H., Ovchinnikov, S., & Baker, D. A. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1314045110 · Cited by 672 · see citing works
Self-citationImproved contact prediction in proteins: using pseudolikelihoods to infer Potts models
Ekeberg, M., Lövkvist, C., Lan, Y., Weigt, M., & Aurell, E. (2013) · Physical Review E · doi:10.1103/physreve.87.012707 · Cited by 661 · see citing works
Evaluation and improvement of multiple sequence methods for protein secondary structure prediction
Cuff, J. A., & Barton, G. J. (1999) · Proteins Structure Function and Bioinformatics · doi:10.1002/(sici)1097-0134(19990301)34:4 · Cited by 660 · see citing works
Modeling the language of life – Deep Learning Protein Sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · doi:10.1101/614313 · Cited by 606 · see citing works
Modeling aspects of the language of life through transfer-learning protein sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3220-8 · Cited by 606 · see citing works
Deep generative models of genetic variation capture the effects of mutations
Riesselman, A. J., Ingraham, J. B., & Marks, D. S. (2018) · Nature Methods · doi:10.1038/s41592-018-0138-4 · Cited by 601 · see citing works
NetSurfP-2.0: improved prediction of protein structural features by integrated deep learning
Klausen, M., Jespersen, M. C., Nielsen, H., Jensen, K. K., Jurtz, V., Sønderby, C., Sommer, M., Winther, O., Nielsen, M., Petersen, B., et al. (2018) · bioRxiv · doi:10.1002/prot.25674 · Cited by 564 · see citing works
Dunn, S. D., Wahl, L. M., & Gloor, G. B. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm604 · Cited by 490 · see citing works
Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold
Steinegger, M., Mirdita, M., & Söding, J. (2019) · Nature Methods · doi:10.1038/s41592-019-0437-4 · Cited by 445 · see citing works
CCMpred—fast and precise prediction of protein residue–residue contacts from correlated mutations
Seemayer, S., Gruber, M., & Söding, J. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu500 · Cited by 416 · see citing works
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised LearningPreprint
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.07.12.199554 · Cited by 409 · see citing works
End-to-End Differentiable Learning of Protein Structure
AlQuraishi, M. (2019) · Cell Systems · doi:10.1016/j.cels.2019.03.006 · Cited by 371 · see citing works
Predicting protein functions using incomplete hierarchical labels
Yu, G., Zhu, H., & Domeniconi, C. (2015) · BMC Bioinformatics · doi:10.1186/s12859-014-0430-y · Cited by 360 · see citing works
Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., et al. (2019) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25834 · Cited by 347 · see citing works
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338 · see citing works
Learning generative models for protein fold families
Balakrishnan, S., Kamisetty, H., Carbonell, J. G., Lee, S., & Langmead, C. J. (2011) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.22934 · Cited by 316 · see citing works
An evolution-based model for designing chorismate mutase enzymes
Russ, W. P., Figliuzzi, M., Stocker, C., Barrat-Charlaix, P., Socolich, M., Kast, P., Hilvert, D., Monasson, R., Cocco, S., Weigt, M., et al. (2020) · Science · doi:10.1126/science.aba3304 · Cited by 313 · see citing works
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.07.982272 · Cited by 282 · see citing works
Genomics-aided structure prediction
Sulkowska, J. I., Morcos, F., Weigt, M., Terence Hwa, B., & Onuchic, J. N. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1207864109 · Cited by 226 · see citing works
Learning protein sequence embeddings using information from structure
Bepler, T., & Berger, B. (2019) · arXiv · doi:10.48550/arxiv.1902.08661 · Cited by 219 · see citing works
Origins of coevolution between residues distant in protein 3D structures
Anishchenko, I., Ovchinnikov, S., Kamisetty, H., & Baker, D. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1702664114 · Cited by 185 · see citing works
Self-citationJones, D. T., & Kandathil, S. M. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty341 · Cited by 184 · see citing works
Zhang, C., Zheng, W. H., Mortuza, S. M., Li, Y., & Zhang, Y. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz863 · Cited by 182 · see citing works
BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language ModelPreprint
Wang, A., & Cho, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04094 · Cited by 174 · see citing works
Correlated mutations in models of protein sequences: phylogenetic and structural effects
Lapedes, A. S., Giraud, B., Liu, L., & Stormo, G. D. (1999) · Institute of Mathematical Statistics Lecture Notes - Monograph Series · doi:10.1214/lnms/1215455556 · Cited by 124 · see citing works
Enhancing Evolutionary Couplings with Deep Convolutional Neural Networks
Liu, Y., Palmedo, P., Ye, Q., Berger, B., & Peng, J. (2018) · Cell Systems · doi:10.1016/j.cels.2017.11.014 · Cited by 114 · see citing works
Ovchinnikov, S., Kim, D. E., Wang, R. Y., Liu, Y., DiMaio, F., & Baker, D. A. (2015) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.24974 · Cited by 111 · see citing works
Self-citationImproved protein structure prediction using predicted inter-residue orientationsPreprint
Yang, J., Anishchenko, I. V., Park, H., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/846279 · Cited by 109 · see citing works
De Novo Structure Prediction of Globular Proteins Aided by Sequence Variation-Derived Contacts
Kościółek, T., & Jones, D. T. (2014) · PLoS ONE · doi:10.1371/journal.pone.0092197 · Cited by 108 · see citing works
(2018) · International Conference on Learning Representations · Cited by 92 · see citing works
Synthetic sequence entanglement augments stability and containment of genetic information in cells
Blazejewski, T., Ho, H., & Wang, H. H. (2019) · Science · doi:10.1126/science.aav5477 · Cited by 82 · see citing works
Graphical Models of Residue Coupling in Protein Families
Thomas, J., Ramakrishnan, N., & Bailey-Kellogg, C. (2008) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2007.70225 · Cited by 77 · see citing works
Co‐Evolutionary Fitness Landscapes for Sequence Design
Tian, P., Louis, J. M., Baber, J. L., Aniana, A., & Best, R. B. (2018) · Angewandte Chemie International Edition · doi:10.1002/anie.201713220 · Cited by 76 · see citing works
Unified rational protein engineering with sequence-only deep representation learningPreprint
Alley, E. C., Khimulya, G., Biswas, S., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/589333 · Cited by 50 · see citing works
DEEPCON: protein contact prediction using dilated convolutional neural networks with dropout
Adhikari, B. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz593 · Cited by 49 · see citing works
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03497 · Cited by 42 · see citing works
Protein Residue Contacts and Prediction Methods
Adhikari, B., & Cheng, J. J. (2016) · Methods in molecular biology · doi:10.1007/978-1-4939-3572-7_24 · Cited by 40 · see citing works
End-to-end differentiable learning of protein structurePreprint
AlQuraishi, M. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/265231 · Cited by 33 · see citing works
Capturing coevolutionary signals inrepeat proteins
Espada, R., Parra, R. G., Mora, T., Walczak, A. M., & Ferreiro, D. U. (2015) · BMC Bioinformatics · doi:10.1186/s12859-015-0648-3 · Cited by 27 · see citing works
Assessment of CASP10 contact‐assisted predictions
Taylor, T. J., Bai, H., Tai, C., & Lee, B. (2013) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.24367 · Cited by 22 · see citing works
Sequence statistics of tertiary structural motifs reflect protein stability
Zheng, F., & Grigoryan, G. (2017) · PLoS ONE · doi:10.1371/journal.pone.0178272 · Cited by 15 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Robert D. Finn, Alex Bateman, Jody Clements, Penelope Coggill, Ruth Y. Eberhardt, Sean R. Eddy, Andreas Heger, Kirstie Hetherington, Liisa Holm, Jaina Mistry, Erik L.L. Sonnhammer, John Tate, and Marco Punta. Pfam: The protein families database, 1 2014. ISSN 03051048. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3965110/.
- pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC4201158https: //academic.oup.com/bioinformatics/article-lookup/doi/10.1093/ bioinformatics/btu500. · DOI
Cited by
Sorted by citations
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,523 · see citing works
Self-citationProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · see citing works
Simulating 500 million years of evolution with a language model
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., et al. (2025) · Science · doi:10.1126/science.ads0018 · Cited by 937 · see citing works
Self-citationDeepLoc 2.0: multi-label subcellular localization prediction using protein language models
Thumuluri, V., Armenteros, J. J. A., Johansen, A. R., Nielsen, H., & Winther, O. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac278 · Cited by 723 · see citing works
Language models enable zero-shot prediction of the effects of mutations on protein function
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.07.09.450648 · Cited by 635 · see citing works
Self-citationLearning the protein language: Evolution, structure, and function
Bepler, T., & Berger, B. (2021) · Cell Systems · doi:10.1016/j.cels.2021.05.017 · Cited by 483 · see citing works
Single-sequence protein structure prediction using language models from deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Rochereau, C., Church, G. M., Sorger, P. K., & AlQuraishi, M. (2021) · doi:10.1101/2021.08.02.454840 · Cited by 455 · see citing works
“We speculate that the latent space of the language model also captures recurrent evolutionary relationships 43 .”
BackgroundSingle-sequence protein structure prediction using a language model and deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01432-w · Cited by 455 · see citing works
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised LearningPreprint
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.07.12.199554 · Cited by 409 · see citing works
High-resolution <i>de novo</i> structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392 · see citing works
Rao, R., Liu, J., Verkuil, R., Meier, J., Canny, J. F., Abbeel, P., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.02.12.430858 · Cited by 350 · see citing works
Self-citationFast, accurate antibody structure prediction from deep learning on massive set of natural antibodies
Ruffolo, J. A., Chu, L., Mahajan, S. P., & Gray, J. J. (2023) · Nature Communications · doi:10.1038/s41467-023-38063-x · Cited by 339 · see citing works
Elnaggar, A., Heinzinger, M., Dallago, C., Rihawi, G., Wang, Y., Jones, L., Gibbs, T., Fehér, T., Angerer, C. M., Steinegger, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06225 · Cited by 305 · see citing works
Machine learning in protein structure prediction
AlQuraishi, M. (2021) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2021.04.005 · Cited by 252 · see citing works
SaProt: Protein Language Modeling with Structure-aware VocabularyPreprint
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., & Yuan, F. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.01.560349 · Cited by 251 · see citing works
PredictProtein - Predicting Protein Structure and Function for 29 Years
Bernhofer, M., Dallago, C., Karl, T., Satagopam, V., Heinzinger, M., Littmann, M., Olenyi, T., Qiu, J., Schütze, K., Yachdav, G., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab354 · Cited by 239 · see citing works
DeepLoc 2.1: multi-label membrane protein type prediction using protein language models
Ødum, M. T., Teufel, F., Thumuluri, V., Armenteros, J. J. A., Johansen, A. R., Winther, O., & Nielsen, H. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae237 · Cited by 230 · see citing works
Antibody structure prediction using interpretable deep learning
Ruffolo, J. A., Sulam, J., & Gray, J. J. (2022) · Patterns · doi:10.1016/j.patter.2021.100406 · Cited by 216 · see citing works
Modeling conformational states of proteins with AlphaFold
Sala, D., Engelberger, F., Mchaourab, H. S., & Meiler, J. (2023) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2023.102645 · Cited by 182 · see citing works
Ankh ☥: Optimized Protein Language Model Unlocks General-Purpose ModellingPreprint
Elnaggar, A., Essam, H., Salah-Eldin, W., Moustafa, W., Elkerdawy, M., Rochereau, C., & Rost, B. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.16.524265 · Cited by 201 · see citing works
Language models generalize beyond natural proteins
Verkuil, R., Kabeli, O., Du, Y., Wicky, B. I. M., Milles, L. F., Dauparas, J., Baker, D., Ovchinnikov, S., Sercu, T., & Rives, A. (2022) · doi:10.1101/2022.12.21.521521 · Cited by 181 · see citing works
Self-citationLearning meaningful representations of protein sequences
Detlefsen, N. S., Hauberg, S., & Boomsma, W. (2022) · Nature Communications · doi:10.1038/s41467-022-29443-w · Cited by 163 · see citing works
Shan, S., Luo, S., Yang, Z., Hong, J., Su, Y., Ding, F., Fu, L., Li, C., Chen, P., Ma, J., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2122954119 · Cited by 157 · see citing works
Single-sequence protein structure prediction using supervised transformer protein language models
Wang, W., Peng, Z., & Yang, J. (2022) · Nature Computational Science · doi:10.1038/s43588-022-00373-3 · Cited by 150 · see citing works
A 5′ UTR language model for decoding untranslated regions of mRNA and function predictions
Chu, Y., Yu, D., Li, Y., Huang, K., Shen, Y., Cong, L., Zhang, J. Y., & Wang, M. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00823-9 · Cited by 143 · see citing works
Protein language model embeddings for fast, accurate, alignment-free protein structure prediction
Weissenow, K., Heinzinger, M., & Rost, B. (2021) · doi:10.1101/2021.07.31.454572 · Cited by 139 · see citing works
Du, Z., Ding, X., Xu, Y., & Li, Y. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad135 · Cited by 137 · see citing works
Light attention predicts protein location from the language of life
Stärk, H., Dallago, C., Heinzinger, M., & Rost, B. (2021) · Bioinformatics Advances · doi:10.1093/bioadv/vbab035 · Cited by 130 · see citing works
FLIP: Benchmark tasks in fitness landscape inference for proteinsPreprint
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.11.09.467890 · Cited by 127 · see citing works
LAnoBERT: System log anomaly detection based on BERT masked language model
Lee, Y., Kim, J., & Kang, P. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110689 · Cited by 139 · see citing works
Machine learning modeling of family wide enzyme-substrate specificity screens
Goldman, S., Das, R., Yang, K. K., & Coley, C. W. (2022) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1009853 · Cited by 120 · see citing works
Fang, X., Wang, F., Liu, L., He, J., Lin, D., Xiang, Y., Zhu, K., Zhang, X., Wu, H., Li, H., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00721-6 · Cited by 132 · see citing works
Deep generative modeling for protein design
Strokach, A. V., & Kim, P. M. (2021) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2021.11.008 · Cited by 114 · see citing works
NetSolP: predicting protein solubility in Escherichia coli using language models
Thumuluri, V., Martiny, H.-M., Armenteros, J. J. A., Salomon, J., Nielsen, H., & Johansen, A. R. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab801 · Cited by 116 · see citing works
Markus, B., Gruber Christian, C., Andreas, K., Kummer, A., Stefan, L., Oberdorfer, G., Elina, S., & Šnajdrová, R. (2023) · ACS Catalysis · doi:10.1021/acscatal.3c03417 · Cited by 116 · see citing works
A new age in protein design empowered by deep learning
Khakzad, H., Igashov, I., Schneuing, A., Goverde, C. A., Bronstein, M. M., & Correia, B. E. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.006 · Cited by 113 · see citing works
Convolutions are competitive with transformers for protein sequence pretraining
Yang, K. K., Fusi, N., & Lu, A. X. (2024) · Cell Systems · doi:10.1016/j.cels.2024.01.008 · Cited by 137 · see citing works
Multiple sequence alignment-based RNA language model and its application to structural inference
Zhang, Y., Lang, M., Jiang, J., Gao, Z., Xu, F., Litfin, T., Chen, K., Singh, J., Huang, X., Song, G., et al. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkad1031 · Cited by 112 · see citing works
Learned Embeddings from Deep Learning to Visualize and Predict Protein Sets
Dallago, C., Schütze, K., Heinzinger, M., Olenyi, T., Littmann, M., Lu, A. X., Yang, K., Min, S., Yoon, S., Morton, J. T., et al. (2021) · Current Protocols · doi:10.1002/cpz1.113 · Cited by 101 · see citing works
Predicting functional effect of missense variants using graph attention neural networks
Zhang, H., Xu, M. S., Fan, X., Chung, W. K., & Shen, Y. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00561-w · Cited by 98 · see citing works
pLM4ACE: A protein language model based predictor for antihypertensive peptide screening
Du, Z., Ding, X., Hsu, W. W. Y., Munir, A., Xu, Y., & Li, Y. (2023) · Food Chemistry · doi:10.1016/j.foodchem.2023.137162 · Cited by 98 · see citing works
Structure-based protein design with deep learning
Ovchinnikov, S., & Huang, P.-S. (2021) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2021.08.004 · Cited by 93 · see citing works
Self-citationA survey of drug-target interaction and affinity prediction methods via graph neural networks
Zhang, Y., Hu, Y., Han, N., Yang, A., Liu, X., & Cai, H. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107136 · Cited by 98 · see citing works
Wang, Z., Combs, S. A., Brand, R. M., Calvo, M. R., Xu, P., Price, G., Golovach, N., Salawu, E. O., Wise, C., Ponnapalli, S. P., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10775-y · Cited by 84 · see citing works
Representation learning applications in biological sequence analysis
Iuchi, H., Matsutani, T., Yamada, K., Iwano, N., Sumi, S., Hosoda, S., Zhao, S., Fukunaga, T., & Hamada, M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.05.039 · Cited by 83 · see citing works
Sequence-based drug-target affinity prediction using weighted graph neural networks
Jiang, M., Wang, S., Zhang, S., Zhou, W. G., Zhang, Y., & Li, Z. (2022) · BMC Genomics · doi:10.1186/s12864-022-08648-9 · Cited by 84 · see citing works
InterPLM: discovering interpretable features in protein language models via sparse autoencoders
Simon, E. P., & Zou, J. (2025) · Nature Methods · doi:10.1038/s41592-025-02836-7 · Cited by 89 · see citing works
AI in drug development: a multidisciplinary perspective
Gallego, V., Naveiro, R., Roca, C., Insua, D. R., & Campillo, N. E. (2021) · Molecular Diversity · doi:10.1007/s11030-021-10266-8 · Cited by 86 · see citing works
Protein language models trained on multiple sequence alignments learn phylogenetic relationships
Lupo, U., Sgarbossa, D., & Bitbol, A.-F. (2022) · Nature Communications · doi:10.1038/s41467-022-34032-y · Cited by 67 · see citing works
Porebski, B. T., Balmforth, M. R., Browne, G. J., Riley, A., Jamali, K., Fürst, M. J. L. J., Velic, M., Buchanan, A. G., Minter, R., Vaughan, T. J., et al. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01093-3 · Cited by 72 · see citing works
Cheng, P., Mao, C., Tang, J., Yang, S., Cheng, Y., Wang, W., Gu, Q., Han, W., Chen, H., Li, S., et al. (2024) · Cell Research · doi:10.1038/s41422-024-00989-2 · Cited by 76 · see citing works
Prediction of RNA–protein interactions using a nucleotide language model
Yamada, K., & Hamada, M. (2022) · Bioinformatics Advances · doi:10.1093/bioadv/vbac023 · Cited by 69 · see citing works
Transformer-based protein generation with regularized latent space optimization
Castro, E., Godavarthi, A., Rubinfien, J., Givechian, K. B., Bhaskar, D., & Krishnaswamy, S. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00532-1 · Cited by 66 · see citing works
Singh, J., Paliwal, K. K., Litfin, T., Singh, J., & Zhou, Y. (2022) · Scientific Reports · doi:10.1038/s41598-022-11684-w · Cited by 57 · see citing works
Shi, Z., Deng, R., Yuan, Q., Mao, Z., Wang, R., Li, H., Liao, X., & Ma, H. (2023) · Research · doi:10.34133/research.0153 · Cited by 53 · see citing works
scPRINT: pre-training on 50 million cells allows robust gene network predictions
Kalfon, J., Samaran, J., Gabriel Peyré, A., & Cantini, L. (2025) · Nature Communications · doi:10.1038/s41467-025-58699-1 · Cited by 60 · see citing works
Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models
Li, F.-Z., Amini, A. P., Yue, Y., Yang, K. K., & Lu, A. X. (2024) · doi:10.1101/2024.02.05.578959 · Cited by 51 · see citing works
Are protein language models the new universal key?
Weißenow, K., & Rost, B. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.102997 · Cited by 51 · see citing works
End-to-end learning of multiple sequence alignments with differentiable Smith–Waterman
Petti, S., Bhattacharya, N., Rao, R., Dauparas, J., Thomas, N., Zhou, J., Rush, A. M., Koo, P., & Ovchinnikov, S. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btac724 · Cited by 54 · see citing works
Self-citationDeep self-supervised learning for biosynthetic gene cluster detection and product classification
Rios-Martinez, C., Bhattacharya, N., Amini, A. P., Crawford, L. A., & Yang, K. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011162 · Cited by 47 · see citing works
Multi-modal deep learning enables efficient and accurate annotation of enzymatic active sites
Wang, X., Yin, X., Jiang, D., Zhao, H., Wu, Z., Zhang, O., Wang, J., Li, Y., Deng, Y., Liu, H., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-51511-6 · Cited by 47 · see citing works
Drug-target interaction/affinity prediction: Deep learning models and advances review
Vefghi, A., Rahmati, Z., & Akbari, M. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110438 · Cited by 46 · see citing works
Protein sequence‐to‐structure learning: Is this the end(‐to‐end revolution)?
Laine, E., Eismann, S., Elofsson, A., & Grudinin, S. (2021) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26235 · Cited by 44 · see citing works
“works have also shown that the residue-residue dependencies extracted by certain attention heads in transformers trained on large amounts of sequences can be directly interpreted as 3D contacts or distances.(48,133,134)”
BackgroundFast, sensitive detection of protein homologs using deep dense retrieval
Hong, L., Hu, Z., Sun, S., Tang, X., Wang, J., Tan, Q., Zheng, L., Wang, S., Xu, S. Y., King, I., et al. (2024) · Nature Biotechnology · doi:10.1038/s41587-024-02353-6 · Cited by 43 · see citing works
Pakhrin, S. C., Pokharel, S., Aoki-Kinoshita, K. F., Beck, M. R., Dam, T. K., Caragea, D., & KC, D. B. (2023) · Glycobiology · doi:10.1093/glycob/cwad033 · Cited by 39 · see citing works
Protein structure prediction in the deep learning era
Peng, Z., Wang, W., Han, R., Zhang, F., & Yang, J. (2022) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2022.102495 · Cited by 37 · see citing works
Rapid directed evolution guided by protein language models and epistatic interactions
Tran, V. Q., Nemeth, M., Bartie, L. J., Chandrasekaran, S. S., Fanton, A., Moon, H. C., Hie, B., Konermann, S., & Hsu, P. D. (2026) · Science · doi:10.1126/science.aea1820 · Cited by 41 · see citing works
Generative AI: driving productivity and scientific breakthroughs in pharmaceutical R&D
Doron, G., Genway, S., Roberts, M., & Jasti, S. (2024) · Drug Discovery Today · doi:10.1016/j.drudis.2024.104272 · Cited by 36 · see citing works
Phosformer: an explainable transformer model for protein kinase-specific phosphorylation predictions
Zhou, Z., Yeung, W., Gravel, N., Salcedo, M. V., Soleymani, S., Li, S., & Kannan, N. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad046 · Cited by 34 · see citing works
AutoPeptideML: a study on how to build more trustworthy peptide bioactivity predictors
Fernández-Díaz, R., Cossio-Pérez, R., Agoni, C., Lam, H. T., López, V., & Shields, D. C. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae555 · Cited by 35 · see citing works
RAPPPID: towards generalizable protein interaction prediction with AWD-LSTM twin networks
Szymborski, J., & Emad, A. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac429 · Cited by 34 · see citing works
Identification of a covert evolutionary pathway between two protein folds
Chakravarty, D., Sreenivasan, S., Swint-Kruse, L., & Porter, L. L. (2023) · Nature Communications · doi:10.1038/s41467-023-38519-0 · Cited by 29 · see citing works
Protein representations: Encoding biological information for machine learning in biocatalysis
Harding-Larsen, D., Funk, J., Madsen, N. G., Gharabli, H., Acevedo-Rocha, C. G., Mazurenko, S., & Welner, D. H. (2024) · Biotechnology Advances · doi:10.1016/j.biotechadv.2024.108459 · Cited by 38 · see citing works
Zhang, Y., Zhang, Y., Xiong, Y., Wang, H., Deng, Z., Song, J., & Ou, H. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab420 · Cited by 31 · see citing works
Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodiesPreprint
Ruffolo, J. A., Chu, L., Mahajan, S. P., & Gray, J. J. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.20.488972 · Cited by 33 · see citing works
BEND: Benchmarking DNA Language Models on biologically meaningful tasksPreprint
Marin, F. I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., & Boomsma, W. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2311.12570 · Cited by 101 · see citing works
Pairing interacting protein sequences using masked language modeling
Lupo, U., Sgarbossa, D., & Bitbol, A.-F. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311887121 · Cited by 31 · see citing works
Evaluation of residue-residue contact prediction methods: From retrospective to prospective
Zhang, H., Bei, Z., Xi, W., Hao, M., Ju, Z., Saravanan, K. M., Zhang, H., Guo, N., & Wei, Y. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009027 · Cited by 30 · see citing works
DG-Affinity: predicting antigen–antibody affinity with language models from sequences
Yuan, Y., Chen, Q., Mao, J., Li, G., & Pan, X. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05562-z · Cited by 29 · see citing works
Prediction of drug-target binding affinity based on deep learning models
Zhang, H., Liu, X., Cheng, W., Wang, T., & Chen, Y. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108435 · Cited by 31 · see citing works
Drug Discovery in the Age of Artificial Intelligence: Transformative Target-Based Approaches
Patne, A., Dhulipala, S. M., Lawless, W., Prakash, S., Mohapatra, S., & Mohapatra, S. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms252212233 · Cited by 25 · see citing works
Ultra-fast protein structure prediction to capture effects of sequence variation in mutation moviesPreprint
Weißenow, K., Heinzinger, M., Steinegger, M., & Rost, B. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.11.14.516473 · Cited by 28 · see citing works
Protein Language Models: Is Scaling Necessary?Preprint
Fournier, Q., Vernon, R. M., van der Sloot, A. M., Schulz, B., Chandar, S., & LANGMEAD, C. J. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.23.614603 · Cited by 31 · see citing works
Yan, Z., Ge, F., Liu, Y., Zhang, Y., Li, F., Song, J., & Yu, D. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c02019 · Cited by 25 · see citing works
tFold-Ab: Fast and Accurate Antibody Structure Prediction without Sequence HomologsPreprint
Wu, J., Wu, F., Jiang, B., Liu, W., & Zhao, P. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.11.10.515918 · Cited by 26 · see citing works
DeepSS2GO: protein function prediction from secondary structure
Song, F. V., Su, J., Huang, S., Zhang, N., Li, K., Ni, M., & Liao, M. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae196 · Cited by 26 · see citing works
Xu, S., & Onoda, A. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c01202 · Cited by 26 · see citing works
Heinzinger, M., & Rost, B. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.102986 · Cited by 25 · see citing works
A Structure-Based B-cell Epitope Prediction Model Through Combing Local and Global Features
Lu, S., Li, Y., Ma, Q., Nan, X., & Zhang, S. (2022) · Frontiers in Immunology · doi:10.3389/fimmu.2022.890943 · Cited by 23 · see citing works
Machine learning modeling of RNA structures: methods, challenges and future perspectives
Wu, K., Zou, J., & Chang, H. Y. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad210 · Cited by 25 · see citing works
Improved prediction of MHC-peptide binding using protein language models
Hashemi, N., Hao, B., Ignatov, M., Paschalidis, I. C., Vakili, P., Vajda, S., & Kozakov, D. (2023) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2023.1207380 · Cited by 25 · see citing works
Yue, P., Wu, J., Sun, Y., Zhang, Y., Wang, Q., & Shao, S. (2025) · Nature Communications · doi:10.1038/s41467-025-56526-1 · Cited by 26 · see citing works
- Authors
- Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A.
- Affiliations
- University of California, Berkeley (US)
- Meta (United States) (US)
- Harvard University (US)
- Corresponding author
- Rao, Roshan
- Meier, Joshua
- Sercu, Tom
- Ovchinnikov, Sergey
- Rives, Alexander
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2020-12-15
- OpenAlex
- W3111174583
- Cited by
- 341
- References
- 63
- Citations by year
- 2026: 51 · 2025: 67 · 2024: 79 · 2023: 56 · 2022: 49 · 2021: 37 · 2020: 2
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- Funders
- National Institutes of Health (DP5OD026389)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
- Keywords
- protein language models, protein structure prediction, protein structure design, evolutionary constraints
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
