MSA Transformer
Open accessAbstract
Abstract Unsupervised protein language models trained across millions of diverse sequences learn structure and function of proteins. Protein language models studied to date have been trained to perform inference from individual sequences. The longstanding approach in computational biology has been to make inferences from a family of evo lutionarily related sequences by fitting a model to each family independently. In this work we combine the two paradigms. We introduce a protein language model which takes as input a set of sequences in the form of a multiple sequence alignment. The model interleaves row and column attention across the input sequences and is trained with a variant of the masked language modeling objective across many protein families. The performance of the model surpasses current state-of-the-art unsupervised structure learning methods by a wide margin, with far greater parameter efficiency than prior state-of-the-art protein language models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The MSA clusters contained evolutionary couplings for both states, and the score based on contact maps correlated with the r.m.s.d. in AF2.
“We used the same set of clusters to make predictions using the unsupervised deep learning model MSA Transformer36 and found that these clusters contained evolutionary couplings for both states, and the score based on contact maps correlated with the root mean squared deviation (r.m.s.d.) in AF2 (Methods and Extended Data Fig. 3).”
From Predicting multiple conformations via sequence clustering and AlphaFold2 (2024) · cites this paper as [36]
MSA Transformer is an unsupervised learning method; its contact predictions purely reflect evolutionary couplings learned in sequences, rather than being supervised on structure as is the case AF2.
“MSA transformer is an unsupervised learning method, which signifies that its contact predictions purely reflect evolutionary couplings learned in sequences, rather than being supervised on structure as is the case AF2.”
From Predicting multiple conformations via sequence clustering and AlphaFold2 (2024) · cites this paper as [36]
Attention-based networks are extremely powerful for modeling complicated interactions between entities, making them suitable for modeling proteins whose functions are largely determined by interactions between amino acids.
“Among many successful neural network architectures, attention-based networks are extremely powerful for modeling complicated interactions between entities, making them suitable for modeling proteins whose functions are largely determined by interactions between amino acids (34–37).”
From Deep learning guided optimization of human antibody against SARS-CoV-2 variants with broad neutralization (2022) · cites this paper as [36]
The MSA Transformer derives embeddings via a tied row attention framework that exploits the equivalence of aligned positions in the MSA.
“In the MSA Transformer (16), the neural network model exploits the equivalence of aligned positions in the MSA in order to derive embeddings via a tied row attention framework.”
From Ultrafast end-to-end protein structure prediction enables high-throughput exploration of uncharacterised proteins (2020) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286 · see citing works
Self-citationRives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · see citing works
Self-citationRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
CCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00069 · Cited by 3,045 · see citing works
Morcos, F., Pagnani, A., Lunt, B., Bertolino, A., Marks, D. S., Sander, C., Zecchina, R., Onuchic, J. N., Hwa, T., & Weigt, M. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1111471108 · Cited by 1,434 · see citing works
Improved protein structure prediction using predicted interresidue orientations
Yang, J., Anishchenko, I., Park, H., Peng, Z., Ovchinnikov, S., & Baker, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914677117 · Cited by 1,421 · see citing works
HH-suite3 for fast remote homology detection and deep protein annotation
Steinegger, M., Meier, M., Mirdita, M., Vöhringer, H., Haunsberger, S. J., & Söding, J. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3019-7 · Cited by 1,145 · see citing works
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131 · see citing works
Generating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063 · see citing works
Identification of direct residue contacts in protein–protein interaction by message passing
Weigt, M., White, R. A., Szurmant, H., Hoch, J. A., & Hwa, T. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0805923106 · Cited by 1,025 · see citing works
Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model
Wang, S., Sun, S., Li, Z., Zhang, R., & Xu, J. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005324 · Cited by 931 · see citing works
Correlated mutations and residue contacts in proteins
Göbel, U., Sander, C., Schneider, R., & Valencia, A. (1994) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.340180402 · Cited by 863 · see citing works
Disease variant prediction with deep generative models of evolutionary data
Frazer, J., Notin, P., Dias, M., Gomez, A., Min, J. K., Brock, K., Gal, Y., & Marks, D. S. (2021) · Nature · doi:10.1038/s41586-021-04043-8 · Cited by 791 · see citing works
Jones, D. T., Buchan, D. W. A., Cozzetto, D., & Pontil, M. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr638 · Cited by 760 · see citing works
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · see citing works
Self-citationUniclust databases of clustered and deeply annotated protein sequences and alignments
Mirdita, M., von den Driesch, L., Galiez, C., Martin, M. J., Söding, J., & Steinegger, M. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkw1081 · Cited by 710 · see citing works
Kamisetty, H., Ovchinnikov, S., & Baker, D. A. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1314045110 · Cited by 672 · see citing works
Improved contact prediction in proteins: using pseudolikelihoods to infer Potts models
Ekeberg, M., Lövkvist, C., Lan, Y., Weigt, M., & Aurell, E. (2013) · Physical Review E · doi:10.1103/physreve.87.012707 · Cited by 661 · see citing works
Modeling the language of life – Deep Learning Protein Sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · doi:10.1101/614313 · Cited by 606 · see citing works
Modeling aspects of the language of life through transfer-learning protein sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3220-8 · Cited by 606 · see citing works
Deep generative models of genetic variation capture the effects of mutations
Riesselman, A. J., Ingraham, J. B., & Marks, D. S. (2018) · Nature Methods · doi:10.1038/s41592-018-0138-4 · Cited by 601 · see citing works
NetSurfP‐2.0: Improved prediction of protein structural features by integrated deep learning
Klausen, M. S., Jespersen, M. C., Nielsen, H., Jensen, K. K., Jurtz, V. I., Sønderby, C. K., Sommer, M. O. A., Winther, O., Nielsen, M., Petersen, B., et al. (2019) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25674 · Cited by 562 · see citing works
Dunn, S. D., Wahl, L. M., & Gloor, G. B. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm604 · Cited by 490 · see citing works
Predicting the clinical impact of human mutation with deep neural networks
Sundaram, L., Gao, H., Padigepati, S. R., McRae, J. F., Li, Y., Kosmicki, J. A., Fritzilas, N., Hakenberg, J., Dutta, A., Shon, J. C., et al. (2018) · Nature Genetics · doi:10.1038/s41588-018-0167-z · Cited by 482 · see citing works
CCMpred—fast and precise prediction of protein residue–residue contacts from correlated mutations
Seemayer, S., Gruber, M., & Söding, J. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu500 · Cited by 416 · see citing works
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised LearningPreprint
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.07.12.199554 · Cited by 409 · see citing works
Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., et al. (2019) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25834 · Cited by 347 · see citing works
Transformer protein language models are unsupervised structure learners
Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A. (2020) · doi:10.1101/2020.12.15.422761 · Cited by 341 · see citing works
Self-citationLearning generative models for protein fold families
Balakrishnan, S., Kamisetty, H., Carbonell, J. G., Lee, S., & Langmead, C. J. (2011) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.22934 · Cited by 316 · see citing works
An evolution-based model for designing chorismate mutase enzymes
Russ, W. P., Figliuzzi, M., Stocker, C., Barrat-Charlaix, P., Socolich, M., Kast, P., Hilvert, D., Monasson, R., Cocco, S., Weigt, M., et al. (2020) · Science · doi:10.1126/science.aba3304 · Cited by 313 · see citing works
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.07.982272 · Cited by 282 · see citing works
DeepSF: deep convolutional neural network for mapping protein sequences to folds
Hou, J., Adhikari, B., & Cheng, J. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx780 · Cited by 180 · see citing works
Haas, J., Barbato, A., Behringer, D., Studer, G., Roth, S., Bertoni, M., Mostaguir, K., Gumienny, R., & Schwede, T. (2018) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25431 · Cited by 168 · see citing works
Search Engine Guided Neural Machine Translation
Gu, J., Wang, Y., Cho, K., & Li, V. O. K. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12013 · Cited by 147 · see citing works
Correlated mutations in models of protein sequences: phylogenetic and structural effects
Lapedes, A. S., Giraud, B., Liu, L., & Stormo, G. D. (1999) · Institute of Mathematical Statistics Lecture Notes - Monograph Series · doi:10.1214/lnms/1215455556 · Cited by 124 · see citing works
Enhancing Evolutionary Couplings with Deep Convolutional Neural Networks
Liu, Y., Palmedo, P., Ye, Q., Berger, B., & Peng, J. (2018) · Cell Systems · doi:10.1016/j.cels.2017.11.014 · Cited by 114 · see citing works
Improved protein structure prediction using predicted inter-residue orientationsPreprint
Yang, J., Anishchenko, I. V., Park, H., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/846279 · Cited by 109 · see citing works
rawMSA: End-to-end Deep Learning using raw Multiple Sequence Alignments
Mirabello, C., & Wallner, B. (2019) · PLOS ONE · doi:10.1371/journal.pone.0220182 · Cited by 105 · see citing works
Protein Structure Relationships Revealed by Mutational Analysis
Yanofsky, C., Horn, V., & Thorpe, D. (1964) · Science · doi:10.1126/science.146.3651.1593 · Cited by 98 · see citing works
Coordinated amino acid changes in homologous protein families
Altschuh, D., Vernet, T., Berti, P. J., Moras, D., & Nagai, K. (1988) · Protein Engineering Design and Selection · doi:10.1093/protein/2.3.193 · Cited by 96 · see citing works
Self-Supervised Contrastive Learning of Protein Representations By Mutual Information MaximizationPreprint
Lu, A. X., Zhang, H., Ghassemi, M., & Moses, A. M. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.09.04.283929 · Cited by 70 · see citing works
Assessing the accuracy of contact predictions in CASP13
Shrestha, R., Fajardo, E. J., Gil, N., Fidelis, K. A., Kryshtafovych, A., Monastyrskyy, B., & Fiser, A. (2019) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25819 · Cited by 60 · see citing works
Unified rational protein engineering with sequence-only deep representation learningPreprint
Alley, E. C., Khimulya, G., Biswas, S., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/589333 · Cited by 50 · see citing works
DEEPCON: protein contact prediction using dilated convolutional neural networks with dropout
Adhikari, B. (2019) · Bioinformatics · doi:10.1093/bioinformatics/btz593 · Cited by 49 · see citing works
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03497 · Cited by 42 · see citing works
Protein Design and Variant Prediction Using Autoregressive Generative ModelsPreprint
Shin, J.-E., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E. P., Sander, C., Manglik, A., Kruse, A. C., & Marks, D. S. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/757252 · Cited by 39 · see citing works
Single Layers of Attention Suffice to Predict Protein ContactsPreprint
Bhattacharya, N., Thomas, N., Rao, R., Dauparas, J., Koo, P. K., Baker, D. A., Song, Y. S., & Ovchinnikov, S. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.12.21.423882 · Cited by 31 · see citing works
Self-citationLarge-scale clinical interpretation of genetic variants using evolutionary data and deep learningPreprint
Frazer, J., Notin, P., Dias, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.12.21.423785 · Cited by 13 · see citing works
Neural Potts ModelPreprint
Sercu, T., Verkuil, R., Meier, J., Amos, B., Lin, Z., Chen, C., Liu, J. J., LeCun, Y., & Rives, A. W. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.04.08.439084 · Cited by 10 · see citing works
Self-citationKandathil, S. M., Greener, J. G., Lau, A. M., & Jones, D. T. (2020) · doi:10.1101/2020.11.27.401232 · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- Cuff, J. A. and Barton, G. J. Evaluation and improve- ment of multiple sequence methods for protein sec- ondary structure prediction. Proteins: Structure, Func- tion and Genetics, 34(4):508-519, 3 1999. ISSN 08873585. doi: 10.1002/(SICI)1097-0134(19990301) 34:4 508::AID-PROT10 3.0.CO;2-4. URL https:// pubmed.ncbi.nlm.nih.gov/10081963/. · DOI
- Ho, J., Kalchbrenner, N., Weissenborn, D., and Salimans, T. Axial Attention in Multidimensional Transformers. arXiv, 12 2019. URL http://arxiv.org/abs/ 1912.12180.
- Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Tunyasuvunakool, K., Ronneberger, O., Bates, R., Žídek, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Potapenko, A., Ballard, A. J., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Steinegger, M., Pacholska, M., Silver, D., Vinyals, O., Senior, A. W., Kavukcuoglu, K., Kohli, P., and Hassabis, D. High Accuracy Protein Structure Prediction Using Deep Learning. In Fourteenth Critical Assessment of Techniques for Protein Structure Prediction (Abstract Book), pp. 22. 2020.
- Riesselman, A., Shin, J.-E., Kollasch, A., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A., and Marks, D. Accelerating Protein Design Using Autoregressive Generative Models. bioRxiv, pp. 757252, 2019. doi: 10. 1101/757252. URL https://www.biorxiv.org/ content/10.1101/757252v1. · DOI
- Sturmfels, P., Vig, J., Madani, A., and Rajani, N. F. Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models. bioRxiv, 11 2020. URL http: //arxiv.org/abs/2012.00195.
- Thomas, J., Ramakrishnan, N., and Bailey-Kellogg, C. Graphical models of residue coupling in protein families, 4 2008. ISSN 15455963. URL https://pubmed. ncbi.nlm.nih.gov/18451428/.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polo- sukhin, I. Attention is All you Need. Advances in Neural Information Processing Systems 30, pp. 5998- 6008, 2017. URL http://papers.nips.cc/ paper/7181-attention-is-all-you-need. pdfhttps://papers.nips.cc/paper/ 7181-attention-is-all-you-need.pdf.
Cited by
Sorted by citations
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764 · see citing works
Accurate prediction of protein structures and interactions using a 3-track network
Baek, M., DiMaio, F., Anishchenko, I., Dauparas, J., Ovchinnikov, S., Lee, G. R., Wang, J., Cong, Q., Kinch, L. N., Schaeffer, R. D., et al. (2021) · doi:10.1101/2021.06.14.448402 · Cited by 5,542 · see citing works
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,523 · see citing works
Self-citationLanguage models enable zero-shot prediction of the effects of mutations on protein function
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.07.09.450648 · Cited by 635 · see citing works
Self-citationPredicting multiple conformations via sequence clustering and AlphaFold2
Wayment-Steele, H. K., Ojoawo, A., Otten, R., Apitz, J. M., Pitsawong, W., Hömberger, M., Ovchinnikov, S., Colwell, L., & Kern, D. (2024) · Nature · doi:10.1038/s41586-023-06832-9 · Cited by 545 · see citing works
Scaffolding protein functional sites using deep learning
Wang, J., Lisanza, S., Juergens, D., Tischer, D., Watson, J. L., Castro, K. M., Ragotte, R., Saragovi, A., Milles, L. F., Baek, M., et al. (2022) · Science · doi:10.1126/science.abn2100 · Cited by 487 · see citing works
Single-sequence protein structure prediction using language models from deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Rochereau, C., Church, G. M., Sorger, P. K., & AlQuraishi, M. (2021) · doi:10.1101/2021.08.02.454840 · Cited by 455 · see citing works
Single-sequence protein structure prediction using a language model and deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01432-w · Cited by 455 · see citing works
Learning inverse folding from millions of predicted structuresPreprint
Hsu, C., Verkuil, R., Liu, J. J., Lin, Z., Hie, B., Sercu, T., Lerer, A. K., & Rives, A. W. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.10.487779 · Cited by 434 · see citing works
Self-citationProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised LearningPreprint
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.07.12.199554 · Cited by 409 · see citing works
The language of proteins: NLP, machine learning & protein sequences
Ofer, D., Brandes, N., & Linial, M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.03.022 · Cited by 345 · see citing works
Machine learning for functional protein design
Notin, P., Rollins, N. J., Gal, Y., Sander, C., & Marks, D. S. (2024) · Nature Biotechnology · doi:10.1038/s41587-024-02127-0 · Cited by 278 · see citing works
Opportunities and Challenges for Machine Learning-Assisted Enzyme Engineering
Yang, J., Li, F.-Z., & Arnold, F. H. (2024) · ACS Central Science · doi:10.1021/acscentsci.3c01275 · Cited by 257 · see citing works
Elnaggar, A., Heinzinger, M., Dallago, C., Rihawi, G., Wang, Y., Jones, L., Gibbs, T., Fehér, T., Angerer, C. M., Steinegger, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06225 · Cited by 305 · see citing works
Machine learning in protein structure prediction
AlQuraishi, M. (2021) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2021.04.005 · Cited by 251 · see citing works
SaProt: Protein Language Modeling with Structure-aware VocabularyPreprint
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., & Yuan, F. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.01.560349 · Cited by 251 · see citing works
Antibody structure prediction using interpretable deep learning
Ruffolo, J. A., Sulam, J., & Gray, J. J. (2022) · Patterns · doi:10.1016/j.patter.2021.100406 · Cited by 216 · see citing works
Informed training set design enables efficient machine learning-assisted directed protein evolution
Wittmann, B. J., Yue, Y., & Arnold, F. H. (2021) · Cell Systems · doi:10.1016/j.cels.2021.07.008 · Cited by 208 · see citing works
(2023) · doi:10.1101/2023.12.07.570727 · Cited by 207 · see citing works
Prihoda, D., Maamary, J., Waight, A., Juan, V., Fayadat-Dilman, L., Svozil, D., & Bitton, D. A. (2022) · mAbs · doi:10.1080/19420862.2021.2020203 · Cited by 190 · see citing works
Goshisht, M. K. (2024) · ACS Omega · doi:10.1021/acsomega.3c05913 · Cited by 169 · see citing works
Shan, S., Luo, S., Yang, Z., Hong, J., Su, Y., Ding, F., Fu, L., Li, C., Chen, P., Ma, J., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2122954119 · Cited by 157 · see citing works
DNA language models are powerful predictors of genome-wide variant effects
Benegas, G., Batra, S. S., & Song, Y. S. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311219120 · Cited by 157 · see citing works
DeepConsensus improves the accuracy of sequences with a gap-aware sequence transformer
Baid, G., Cook, D. E., Shafin, K., Yun, T., Llinares-López, F., Berthet, Q., Belyaeva, A., Töpfer, A., Wenger, A. M., Rowell, W. J., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01435-7 · Cited by 147 · see citing works
Protein language model embeddings for fast, accurate, alignment-free protein structure prediction
Weissenow, K., Heinzinger, M., & Rost, B. (2021) · doi:10.1101/2021.07.31.454572 · Cited by 139 · see citing works
Toward the solution of the protein structure prediction problem
Pearce, R., & Zhang, Y. (2021) · Journal of Biological Chemistry · doi:10.1016/j.jbc.2021.100870 · Cited by 120 · see citing works
Transformer models in biomedicine
Madan, S., Lentzen, M., Brandt, J., Rueckert, D., Hofmann‐Apitius, M., & Fröhlich, H. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02600-5 · Cited by 133 · see citing works
Fang, X., Wang, F., Liu, L., He, J., Lin, D., Xiang, Y., Zhu, K., Zhang, X., Wu, H., Li, H., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00721-6 · Cited by 132 · see citing works
A DNA language model based on multispecies alignment predicts the effects of genome-wide variants
Benegas, G., Albors, C., Aw, A. J., Ye, C., & Song, Y. S. (2025) · Nature Biotechnology · doi:10.1038/s41587-024-02511-w · Cited by 118 · see citing works
Deep generative modeling for protein design
Strokach, A. V., & Kim, P. M. (2021) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2021.11.008 · Cited by 114 · see citing works
NetSolP: predicting protein solubility in Escherichia coli using language models
Thumuluri, V., Martiny, H.-M., Armenteros, J. J. A., Salomon, J., Nielsen, H., & Johansen, A. R. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab801 · Cited by 116 · see citing works
Deep-learning-based single-domain and multidomain protein structure prediction with D-I-TASSER
Zheng, W. H., Wuyun, Q., Li, Y., Liu, Q., Zhou, X., Peng, C., Zhu, Y., Freddolino, L., & Zhang, Y. (2025) · Nature Biotechnology · doi:10.1038/s41587-025-02654-4 · Cited by 115 · see citing works
Predicting functional effect of missense variants using graph attention neural networks
Zhang, H., Xu, M. S., Fan, X., Chung, W. K., & Shen, Y. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00561-w · Cited by 98 · see citing works
AlQuraishi, M., & Sorger, P. K. (2021) · Nature Methods · doi:10.1038/s41592-021-01283-4 · Cited by 96 · see citing works
Sato, K., & Hamada, M. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad186 · Cited by 101 · see citing works
In silico proof of principle of machine learning-based antibody design at unconstrained scale
Akbar, R., Robert, P. A., Weber, C. R., Widrich, M., Frank, R., Pavlović, M., Scheffer, L., Chernigovskaya, M., Snapkov, I., Slabodkin, A., et al. (2022) · mAbs · doi:10.1080/19420862.2022.2031482 · Cited by 94 · see citing works
Tao, J., Chen, Z., Sun, Z., Guo, H., Leng, B., Yu, Z., Wang, Y., He, Z., Lei, X., & Yang, J. (2023) · Remote Sensing · doi:10.3390/rs15061602 · Cited by 85 · see citing works
Ding, K., Chin, M., Zhao, Y., Huang, W., Mai, B. K., Wang, H., Liu, P., Yang, Y., & Luo, Y. (2024) · Nature Communications · doi:10.1038/s41467-024-50698-y · Cited by 86 · see citing works
Enhancing efficiency of protein language models with minimal wet-lab data through few-shot learning
Zhou, Z., Zhang, L., Yu, Y., Wu, B., Li, M., Hong, L., & Tan, P. (2024) · Nature Communications · doi:10.1038/s41467-024-49798-6 · Cited by 89 · see citing works
Representation learning applications in biological sequence analysis
Iuchi, H., Matsutani, T., Yamada, K., Iwano, N., Sumi, S., Hosoda, S., Zhao, S., Fukunaga, T., & Hamada, M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.05.039 · Cited by 83 · see citing works
Boadu, F., Cao, H., & Cheng, J. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad208 · Cited by 81 · see citing works
Deep graph learning of inter-protein contacts
Xie, Z., & Xu, J. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btab761 · Cited by 67 · see citing works
Application of Transformers in Cheminformatics
Luong, K.-D., & Singh, A. K. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c02070 · Cited by 74 · see citing works
Self-play reinforcement learning guides protein engineering
Wang, Y., Tang, H., Huang, L., Pan, L., Yang, L., Yang, H., Mu, F., & Yang, M. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00691-9 · Cited by 69 · see citing works
A general temperature-guided language model to design proteins of enhanced stability and activity
Jiang, F., Li, M.-C., Dong, J., Yu, Y.-X., Sun, X., Wu, B., Huang, J., Kang, L.-Q., Pei, Y., Zhang, L., et al. (2024) · Science Advances · doi:10.1126/sciadv.adr2641 · Cited by 64 · see citing works
Self-citationDiaz, D. J., Gong, C., Ouyang-Zhang, J., Loy, J. M., Wells, J., Yang, D. K., Ellington, A. D., Dimakis, A. G., & Klivans, A. R. (2024) · Nature Communications · doi:10.1038/s41467-024-49780-2 · Cited by 66 · see citing works
Buess, L., Keicher, M., Navab, N., Maier, A., & Arasteh, S. T. (2025) · Biomedical Engineering Letters · doi:10.1007/s13534-025-00497-1 · Cited by 71 · see citing works
Notin, P., van Niekerk, L., Kollasch, A. W., Ritter, D., Gal, Y., & Marks, D. S. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.12.07.519495 · Cited by 57 · see citing works
Fast and effective protein model refinement using deep graph neural networks
Jing, X., & Xu, J. (2021) · Nature Computational Science · doi:10.1038/s43588-021-00098-9 · Cited by 51 · see citing works
A text-guided protein design framework
Liu, S., Li, Y., Li, Z., Gitter, A., Zhu, Y., Lu, J., Xu, Z., Nie, W., Ramanathan, A., Xiao, C., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01011-z · Cited by 66 · see citing works
Lu, A. X., Lu, A. X., Pritišanac, I., Zarin, T., Forman-Kay, J. D., & Moses, A. M. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010238 · Cited by 51 · see citing works
Zeng, W., Dou, Y., Pan, L., Xu, L., & Peng, S. (2024) · Nature Communications · doi:10.1038/s41467-024-52293-7 · Cited by 49 · see citing works
Effect of tokenization on transformers for biological sequences
Dotan, E., Jaschek, G., Pupko, T., & Belinkov, Y. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae196 · Cited by 49 · see citing works
What does AlphaFold3 learn about antibody and nanobody docking, and what remains unsolved?
Hitawala, F. N., & Gray, J. J. (2025) · mAbs · doi:10.1080/19420862.2025.2545601 · Cited by 86 · see citing works
Xia, Y., Zhao, K., Liu, D., Zhou, X., & Zhang, G. (2023) · Communications Biology · doi:10.1038/s42003-023-05610-7 · Cited by 47 · see citing works
An end-to-end framework for the prediction of protein structure and fitness from single sequence
Chen, Y., Xu, Y., Liu, D., Xing, Y., & Gong, H. (2024) · Nature Communications · doi:10.1038/s41467-024-51776-x · Cited by 45 · see citing works
Protein sequence‐to‐structure learning: Is this the end(‐to‐end revolution)?
Laine, E., Eismann, S., Elofsson, A., & Grudinin, S. (2021) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26235 · Cited by 44 · see citing works
“Recent works have also shown that the residue-residue dependencies extracted by certain attention heads in transformers trained on large amounts of sequences can be directly interpreted as 3D contacts or distances [133, 134, 62].”
BackgroundMethodHe, H., Chen, G., Tang, Z., & Chen, C. Y. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01464-x · Cited by 43 · see citing works
ADOPT: intrinsic protein disorder prediction through deep bidirectional transformers
Redl, I., Fisicaro, C., Dutton, O., Hoffmann, F., Henderson, L. D., Owens, B. M. J., Heberling, M. M., Paci, E., & Tamiola, K. (2023) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqad041 · Cited by 41 · see citing works
What does AlphaFold3 learn about antigen and nanobody docking, and what remains unsolved?Preprint
Hitawala, F. N., & Gray, J. J. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.21.614257 · Cited by 41 · see citing works
Lin, P., Yan, Y., & Huang, S. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac499 · Cited by 39 · see citing works
Zero‐shot mutation effect prediction on protein stability and function using RoseTTAFold
Mansoor, S., Baek, M., Juergens, D., Watson, J. L., & Baker, D. A. (2023) · Protein Science · doi:10.1002/pro.4780 · Cited by 40 · see citing works
Understanding the brain with attention: A survey of transformers in brain sciences
Chen, C., Wang, H., Chen, Y., Yin, Z., Yang, X., Ning, H., Zhang, Q., Li, W., Xiao, R., & Zhao, J. (2023) · Brain‐X · doi:10.1002/brx2.29 · Cited by 43 · see citing works
Protein Engineering with Lightweight Graph Denoising Neural Networks
Zhong, B., Zheng, L., Wu, B., Tan, Y., Lv, O., Yi, K., Fan, G., & Hong, L. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00036 · Cited by 39 · see citing works
Xu, Y., Liu, D., & Gong, H. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00716-2 · Cited by 41 · see citing works
Engineering highly active nuclease enzymes with machine learning and high-throughput screening
Thomas, N., Belanger, D., Xu, C. A., Lee, H., Hirano, K., Iwai, K., Polic, V., Nyberg, K. D., Hoff, K. G., Frenz, L., et al. (2025) · Cell Systems · doi:10.1016/j.cels.2025.101236 · Cited by 39 · see citing works
CpG Transformer for imputation of single-cell methylomes
De Waele, G., Clauwaert, J., Menschaert, G., & Waegeman, W. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btab746 · Cited by 37 · see citing works
Chandra, A. A., Sharma, A. K., Dehzangi, A., Tsunoda, T., & Sattar, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-47624-5 · Cited by 36 · see citing works
Rapid directed evolution guided by protein language models and epistatic interactions
Tran, V. Q., Nemeth, M., Bartie, L. J., Chandrasekaran, S. S., Fanton, A., Moon, H. C., Hie, B., Konermann, S., & Hsu, P. D. (2026) · Science · doi:10.1126/science.aea1820 · Cited by 41 · see citing works
Meta learning addresses noisy and under-labeled data in machine learning-guided antibody engineering
Minot, M., & Reddy, S. T. (2024) · Cell Systems · doi:10.1016/j.cels.2023.12.003 · Cited by 35 · see citing works
Zhang, Y., Zhang, Y., Xiong, Y., Wang, H., Deng, Z., Song, J., & Ou, H. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab420 · Cited by 31 · see citing works
PerturbNet predicts single-cell responses to unseen chemical and genetic perturbations
Yu, H., Qian, W., Song, Y., & Welch, J. D. (2025) · Molecular Systems Biology · doi:10.1038/s44320-025-00131-3 · Cited by 57 · see citing works
Accurate and robust protein sequence design with CarbonDesign
Ren, M., Yu, C., Bu, D., & Zhang, H. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00838-2 · Cited by 30 · see citing works
Chen, X., Morehead, A., Liu, J., & Cheng, J. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad203 · Cited by 28 · see citing works
Self-citationSeq-InSite: sequence supersedes structure for protein interaction site prediction
Hosseini, S., Golding, G. B., & Ilie, L. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btad738 · Cited by 26 · see citing works
Engineering a New Generation of Gene Editors: Integrating Synthetic Biology and AI Innovations
Chia, B. S., Seah, S., Wang, B., Shen, K., Srivastava, D., & Chew, W. L. (2025) · ACS Synthetic Biology · doi:10.1021/acssynbio.4c00686 · Cited by 29 · see citing works
Lab-in-the-loop therapeutic antibody design with deep learningPreprint
Frey, N. C., Hötzel, I., Stanton, S., Kelly, R. L., Alberstein, R., Makowski, E. K., Martinkus, K., Berenberg, D., Bevers, J., Bryson, T., et al. (2025) · bioRxiv · doi:10.1101/2025.02.19.639050 · Cited by 52 · see citing works
tFold-Ab: Fast and Accurate Antibody Structure Prediction without Sequence HomologsPreprint
Wu, J., Wu, F., Jiang, B., Liu, W., & Zhao, P. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.11.10.515918 · Cited by 26 · see citing works
Identification of bacteriophage genome sequences with representation learning
Bai, Z., Zhang, Y., Miyano, S., Yamaguchi, R., Fujimoto, K., Uematsu, S., & Imoto, S. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac509 · Cited by 26 · see citing works
Si, Y., & Yan, C. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad039 · Cited by 25 · see citing works
Improved prediction of MHC-peptide binding using protein language models
Hashemi, N., Hao, B., Ignatov, M., Paschalidis, I. C., Vakili, P., Vajda, S., & Kozakov, D. (2023) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2023.1207380 · Cited by 25 · see citing works
CLADE 2.0: Evolution-Driven Cluster Learning-Assisted Directed Evolution
Qiu, Y., & Wei, G. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c01046 · Cited by 24 · see citing works
SSEmb: A joint embedding of protein sequence and structure enables robust variant effect predictions
Blaabjerg, L. M., Jonsson, N., Boomsma, W., Stein, A., & Lindorff‐Larsen, K. (2024) · Nature Communications · doi:10.1038/s41467-024-53982-z · Cited by 28 · see citing works
Zhang, C., Wang, Q., Li, Y., Teng, A., Hu, G., Wuyun, Q., & Zheng, W. H. (2024) · Biomolecules · doi:10.3390/biom14121531 · Cited by 26 · see citing works
Li, Y., Zhang, C., Zheng, W. H., Zhou, X., Bell, E. W., Yu, D., & Zhang, Y. (2021) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.26211 · Cited by 23 · see citing works
Protein language model-embedded geometric graphs power inter-protein contact prediction
Si, Y., & Yan, C. (2024) · eLife · doi:10.7554/elife.92184 · Cited by 20 · see citing works
pLM4CPPs: Protein Language Model-Based Predictor for Cell Penetrating Peptides
Kumar, N., Du, Z., & Li, Y. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01338 · Cited by 22 · see citing works
PRECOGx: e x ploring GPCR signaling mechanisms with deep protein representations
Matic, M., Singh, G., Carli, F., Rosa, N. D. O., Miglionico, P., Magni, L., Gutkind, J. S., Russell, R. B., Inoue, A., & Raimondi, F. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac426 · Cited by 22 · see citing works
An integrative approach to protein sequence design through multiobjective optimization
Hong, L., & Kortemme, T. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011953 · Cited by 21 · see citing works
Machine Learning for Protein Engineering
Johnston, K. E., Fannjiang, C., Wittmann, B. J., Hie, B., Yang, K., & Wu, Z. (2023) · Challenges and advances in computational chemistry and physics · doi:10.1007/978-3-031-37196-7_9 · Cited by 23 · see citing works
Abbasi, K., Razzaghi, P., Gharizadeh, A., Ghareyazi, A., Dehnad, A., Rabiee, H. R., & Mofrad, M. R. K. (2025) · Pharmacological Reviews · doi:10.1016/j.pharmr.2025.100095 · Cited by 23 · see citing works
Boosting single-cell gene regulatory network reconstruction via bulk-cell transcriptomic data
Shu, H., Ding, F., Zhou, J., Xue, Y., Zhao, D. Y., Zeng, J., & Ma, J. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac389 · Cited by 20 · see citing works
PITHIA: Protein Interaction Site Prediction Using Multiple Sequence Alignments and Attention
Hosseini, S., & Ilie, L. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms232112814 · Cited by 20 · see citing works
Tan, Y., Li, M., Zhou, Z., Tan, P., Yu, H., Fan, G., & Liang, H. (2024) · Journal of Cheminformatics · doi:10.1186/s13321-024-00884-3 · Cited by 21 · see citing works
Non-Identifiability and the Blessings of Misspecification in Models of Molecular Fitness
Weinstein, E. N., Amin, A., Frazer, J., & Marks, D. S. (2022) · neural information processing systems · doi:10.52202/068431-0396 · Cited by 19 · see citing works
Phyloformer: towards fast and accurate phylogeny estimation with self-attention networksPreprint
Nesterenko, L., Boussau, B., & Jacob, L. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.06.24.496975 · Cited by 20 · see citing works
- Authors
- Rao, R., Liu, J., Verkuil, R., Meier, J., Canny, J. F., Abbeel, P., Sercu, T., & Rives, A.
- Affiliations
- Facility for Antiproton and Ion Research (DE)
- Meta (United States) (US)
- University of California, Berkeley (US)
- Corresponding author
- Rao, Roshan
- Rives, Alexander
- ORCID
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2021-02-13
- OpenAlex
- W3133458480
- Cited by
- 350
- References
- 47
- Citations by year
- 2026: 63 · 2025: 90 · 2024: 74 · 2023: 45 · 2022: 47 · 2021: 28 · 2020: 3
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- Funders
- Defense Advanced Research Projects Agency
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- protein language models, multiple sequence alignment, masked language modeling, protein structure prediction, parameter efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
