The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
MSA Transformer
Open accessAbstract
Abstract Unsupervised protein language models trained across millions of diverse sequences learn structure and function of proteins. Protein language models studied to date have been trained to perform inference from individual sequences. The longstanding approach in computational biology has been to make inferences from a family of evo lutionarily related sequences by fitting a model to each family independently. In this work we combine the two paradigms. We introduce a protein language model which takes as input a set of sequences in the form of a multiple sequence alignment. The model interleaves row and column attention across the input sequences and is trained with a variant of the masked language modeling objective across many protein families. The performance of the model surpasses current state-of-the-art unsupervised structure learning methods by a wide margin, with far greater parameter efficiency than prior state-of-the-art protein language models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We used the same set of clusters to make predictions using the unsupervised deep learning model MSA Transformer36 and found that these clusters contained evolutionary couplings for both states, and the score based on contact maps correlated with the root mean squared deviation (r.m.s.d.) in AF2 (Methods and Extended Data Fig. 3).”
From Predicting multiple conformations via sequence clustering and AlphaFold2 (2024) · cites this paper as [36]
In the citing paper
“MSA transformer is an unsupervised learning method, which signifies that its contact predictions purely reflect evolutionary couplings learned in sequences, rather than being supervised on structure as is the case AF2.”
From Predicting multiple conformations via sequence clustering and AlphaFold2 (2024) · cites this paper as [36]
In the citing paper
“Among many successful neural network architectures, attention-based networks are extremely powerful for modeling complicated interactions between entities, making them suitable for modeling proteins whose functions are largely determined by interactions between amino acids (34–37).”
From Deep learning guided optimization of human antibody against SARS-CoV-2 variants with broad neutralization (2022) · cites this paper as [36]
In the citing paper
“In the MSA Transformer (16), the neural network model exploits the equivalence of aligned positions in the MSA in order to derive embeddings via a tied row attention framework.”
From Ultrafast end-to-end protein structure prediction enables high-throughput exploration of uncharacterised proteins (2020) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Self-citationRives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
Self-citationRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
CCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00069 · Cited by 3,045
Morcos, F., Pagnani, A., Lunt, B., Bertolino, A., Marks, D. S., Sander, C., Zecchina, R., Onuchic, J. N., Hwa, T., & Weigt, M. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1111471108 · Cited by 1,434
Improved protein structure prediction using predicted interresidue orientations
Yang, J., Anishchenko, I., Park, H., Peng, Z., Ovchinnikov, S., & Baker, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914677117 · Cited by 1,421
HH-suite3 for fast remote homology detection and deep protein annotation
Steinegger, M., Meier, M., Mirdita, M., Vöhringer, H., Haunsberger, S. J., & Söding, J. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3019-7 · Cited by 1,145
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131
Flagged first, then by citations
Research works printed but not linked: 5
- 10.1002/(SICI)1097-0134(19990301)34:4508::AID-PROT103.0.CO;2-4 · DOI
- Ho, J. , Kalchbrenner, N. , Weissenborn, D. , and Salimans, T. Axial Attention in Multidimensional Transformers. arXiv, 12 2019. URL http://arxiv.org/abs/1912.12180. · arXiv
- Sturmfels, P. , Vig, J. , Madani, A. , and Rajani, N. F. Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models. bioRxiv, 11 2020. URL http://arxiv.org/abs/2012.00195. · arXiv
- Thomas, J. , Ramakrishnan, N. , and Bailey-Kellogg, C. Graphical models of residue coupling in protein families, 4 2008. ISSN 15455963. URL https://pubmed.ncbi.nlm.nih.gov/18451428/.
- 2017, Attention is All you Need, Advances in Neural Information Processing Systems, 30, 5998
Cited by
Sorted by citations
Items 101–200 of 355
Yan, C., Zhang, Z., Xu, J., Yu, M., Yan, S., Wei, L., Zou, Q., Zhang, Q., & Cui, F. (2025) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2025.142309 · Cited by 21
Review and Comparative Analysis of Methods and Advancements in Predicting Protein Complex Structure
Zhao, N., Wu, T., Wang, W., Zhang, L., & Gong, X. (2024) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-024-00626-x · Cited by 20
Zero-shot transfer of protein sequence likelihood models to thermostability prediction
Reeves, S., & Kalyaanamoorthy, S. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00887-7 · Cited by 22
Design of Peptide-Based Protein Degraders via Contrastive Deep LearningPreprint
Palepu, K., Ponnapati, M. R., Bhat, S., Tysinger, E., Stan, T., Brixi, G., Koseki, S., & Chatterjee, P. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.05.23.493169 · Cited by 18
Zhang, W., Meng, Q., Wang, J., & Guo, F. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac169 · Cited by 18
DeCOIL: Optimization of Degenerate Codon Libraries for Machine Learning-Assisted Protein Engineering
Yang, J., Ducharme, J., Johnston, K. E., Li, F.-Z., Yue, Y., & Arnold, F. H. (2023) · ACS Synthetic Biology · doi:10.1021/acssynbio.3c00301 · Cited by 20
From machine learning to multimodal models: The AI revolution in enzyme engineering
Shi, Z., Xu, S., Xue, S., Chen, K., Lu, Y., Wang, F., Long, S., Tian, Y., Zhang, P., Wang, J., et al. (2025) · BioDesign Research · doi:10.1016/j.bidere.2025.100044 · Cited by 19
Predicting pathogen evolution and immune evasion in the age of artificial intelligence
Hamelin, D. J., Scicluna, M., Saadie, I., Mostefai, F., Grenier, J., Baron, C., Caron, É., & Hussin, J. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.03.044 · Cited by 21
How artificial intelligence is reengineering protein engineering
Listgarten, J., & Jiang, H. (2026) · Science · doi:10.1126/science.aec8444 · Cited by 20
Protein engineering in the deep learning era
Zhou, B., Tan, Y., Hu, Y., Zheng, L., Zhong, B., & Hong, L. (2024) · mLife · doi:10.1002/mlf2.12157 · Cited by 18
Ertelt, M., Moretti, R., Meiler, J., & Schoeder, C. T. (2025) · Science Advances · doi:10.1126/sciadv.adr7338 · Cited by 15
Gurusinghe, S. N. S., Wu, Y., DeGrado, W. F., & Shifman, J. M. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf270 · Cited by 15
Zhong, G., Zhao, Y., Zhuang, D., Chung, W. K., & Shen, Y. (2025) · Nature Communications · doi:10.1038/s41467-025-62318-4 · Cited by 15
Mehdiabadi, M., Conte, A. D., Nugnes, M. V., Aspromonte, M. C., Tosatto, S. C. E., & Piovesan, D. (2025) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.70045 · Cited by 20
Acoustic Hologram Reconstruction With Unsupervised Neural Network
Li, B., Lu, M., Liu, C., Liu, X., & Ta, D. (2022) · Frontiers in Materials · doi:10.3389/fmats.2022.916527 · Cited by 15
Zhang, M., Gong, C., Ge, F., & Yu, D. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c02025 · Cited by 15
Chassis engineering for high light tolerance in microalgae and cyanobacteria
Dou, B., Li, Y., Wang, F., Chen, L., & Zhang, W. (2024) · Critical Reviews in Biotechnology · doi:10.1080/07388551.2024.2357368 · Cited by 16
Improving Throughput-oriented LLM Inference with CPU Computations
Park, D., & Egger, B. (2024) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1145/3656019.3676949 · Cited by 17
Exploring the Protein Sequence Space with Global Generative ModelsPreprint
Romero‐Romero, S., Lindner, S., & Ferruz, N. (2023) · Cold Spring Harbor Perspectives in Biology · doi:10.1101/cshperspect.a041471 · Cited by 18
Jin, W., Chen, X., Vetticaden, A., Sarzikova, S., Raychowdhury, R., Uhler, C., & Hacohen, N. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.12.10.570461 · Cited by 20
Malbranke, C., Rostain, W., Depardieu, F., Cocco, S., Monasson, R., & Bikard, D. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011621 · Cited by 15
Scalable and cost-efficient custom gene library assembly from oligopools
Freschlin, C. R., Yang, K. K., & Romero, P. A. (2026) · Science Advances · doi:10.1126/sciadv.ady2279 · Cited by 16
Alsaedi, S. B., Gao, X., & Gojobori, T. (2025) · FEBS Open Bio · doi:10.1002/2211-5463.70003 · Cited by 19
Understanding Language Model Scaling on Protein Fitness PredictionPreprint
Hou, C., Liu, D., Zafar, A., & Shen, Y. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.04.25.650688 · Cited by 14
ProteinGLUE multi-task benchmark suite for self-supervised protein modeling
Capel, H., Weiler, R., Dijkstra, M., Vleugels, R., Bloem, P., & Feenstra, K. A. (2022) · Scientific Reports · doi:10.1038/s41598-022-19608-4 · Cited by 11
Ertelt, M., Meiler, J., & Schoeder, C. T. (2024) · ACS Synthetic Biology · doi:10.1021/acssynbio.3c00753 · Cited by 15
Machine learning in biological physics: From biomolecular prediction to design
Martin, J., Mateos, M., Onuchic, J. N., Coluzza, I., & Morcos, F. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311807121 · Cited by 14
ParaAntiProt provides paratope prediction using antibody and protein language models
Kalemati, M., Noroozi, A., Shahbakhsh, A., & Koohi, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-80940-y · Cited by 15
SPRoBERTa: protein embedding learning with local fragment modeling
Wu, L., Yin, C., Zhu, J., Wu, Z., He, L., Xia, Y., Xie, S., Qin, T., & Liu, T. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac401 · Cited by 12
Natural language processing approach to model the secretion signal of type III effectors
Wagner, N., Alburquerque, M., Ecker, N., Dotan, E., Zerah, B., Pena, M. M., Potnis, N., & Pupko, T. (2022) · Frontiers in Plant Science · doi:10.3389/fpls.2022.1024405 · Cited by 12
Paradigm Shift: The Promise of Deep Learning in Molecular Systems Engineering and Design
Alshehri, A. S., & You, F. (2021) · Frontiers in Chemical Engineering · doi:10.3389/fceng.2021.700717 · Cited by 14
ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular ModelingPreprint
Zheng, K., Long, S., Lu, T., Yang, J., Dai, X., Zhang, M., Nie, Z., Ma, W., & Zhou, H. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.03.04.583284 · Cited by 20
Decoding the Molecular Language of Proteins with EvollaPreprint
Zhou, X., Han, C., Zhang, Y., Du, H., Tian, J., Su, J., Liu, R., Zhuang, K., Jiang, S., Gitter, A., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.01.05.630192 · Cited by 16
Advances in the Modification of Enzymatic Properties Based on Protein Engineering Strategies
Huang, Y., Wen, L., & Yang, B. K. (2026) · Journal of Agricultural and Food Chemistry · doi:10.1021/acs.jafc.5c13434 · Cited by 11
FastFold: Optimizing AlphaFold Training and Inference on GPU Clusters
Cheng, S., Zhao, X., Lu, G., Fang, J., Zheng, T., Wu, R., Zhang, X., Peng, J., & You, Y. (2024) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3627535.3638465 · Cited by 14
Zhang, L., Pang, H., Zhang, C., Li, S., Tan, Y., Jiang, F., Li, M., Yu, Y., Zhou, Z., Wu, B., et al. (2025) · Acta Pharmaceutica Sinica B · doi:10.1016/j.apsb.2025.03.028 · Cited by 14
Does protein pretrained language model facilitate the prediction of protein–ligand interaction?
Zhang, W., Hu, F., Wang, L., & Yin, P. (2023) · Methods · doi:10.1016/j.ymeth.2023.08.016 · Cited by 11
Fan, X., Pan, H., Tian, A., Chung, W. K., & Shen, Y. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbac584 · Cited by 11
PRESCOTT: a population aware, epistatic, and structural model accurately predicts missense effects
Tekpinar, M., David, L., Henry, T., & Carbone, A. (2025) · Genome Biology · doi:10.1186/s13059-025-03581-y · Cited by 18
A survey on large language models in biology and chemistry
Ashyrmamatov, I., Gwak, S. J., Jin, S.-Y., Jun, I., Ucak, U. V., Lee, J. Y., & Lee, J. (2025) · Experimental & Molecular Medicine · doi:10.1038/s12276-025-01583-1 · Cited by 15
Luo, Y., & Cai, J. (2025) · Current Proteomics · doi:10.2174/0115701646374018250227075247 · Cited by 14
Intelligent Protein Design and Molecular Characterization Techniques: A Comprehensive Review
Wang, J., Chen, C., Yao, G., Ding, J., Wang, L., & Jiang, H. (2023) · Molecules · doi:10.3390/molecules28237865 · Cited by 10
Ghazikhani, H., & Butler, G. (2024) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.26694 · Cited by 10
Construction of a Deep Neural Network Energy Function for Protein Physics
Yang, H., Xiong, Z., & Zonta, F. (2022) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.2c00069 · Cited by 13
Scoring alignments by embedding vector similarity
(2024) · doi:10.1093/bib/bbae178 · Cited by 8
Semantical and geometrical protein encoding toward enhanced bioactivity and thermostability
Tan, Y., Zhou, B., Zheng, L., Fan, G., & Hong, L. (2025) · eLife · doi:10.7554/elife.98033.4 · Cited by 20
Yin, M., Zhou, H., Wu, J., Zhu, Y., Zhan, Y., Kong, Z., Xu, H., Hsieh, C., Chen, J., Hou, T., et al. (2025) · Research · doi:10.34133/research.0721 · Cited by 12
Flöge, K., Udayakumar, S., Sommer, J., Piraud, M., Kesselheim, S., Fortuin, V., Günnemann, S., van der Weg, K. J., Gohlke, H., Merdivan, E., et al. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013679 · Cited by 13
Connecting the Dots in Self-Supervised Learning: A Brief Survey for Beginners
Fang, P., Li, X., Yang, Y., Zhang, S., Kang, Q.-Y., Li, X., & Lan, Z. (2022) · Journal of Computer Science and Technology · doi:10.1007/s11390-022-2158-x · Cited by 9
Impact of phylogeny on structural contact inference from protein sequence data
Dietler, N., Lupo, U., & Bitbol, A.-F. (2023) · Journal of The Royal Society Interface · doi:10.1098/rsif.2022.0707 · Cited by 6
Kandathil, S. M., Greener, J. G., Lau, A. M., & Jones, D. T. (2020) · doi:10.1101/2020.11.27.401232 · Cited by 8
Zhang, M., Wang, X., Xu, S., Ge, F., Paixao, I. C., Song, J., & Yu, D. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00739 · Cited by 8
Cao, Z., Zhang, H., & Zou, Z. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2418254122 · Cited by 9
S-Pred: protein structural property prediction using MSA transformer
Hong, Y., Song, J., Ko, J., Lee, J., & Shin, W. (2022) · Scientific Reports · doi:10.1038/s41598-022-18205-9 · Cited by 7
Hidden neural networks for transmembrane protein topology prediction
Tamposis, I., Sarantopoulou, D., Theodoropoulou, M. C., Stasi, E. A., Kontou, P. I., Tsirigos, K. D., & Bagos, P. G. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.11.006 · Cited by 8
DeepConsensus: Gap-Aware Sequence Transformers for Sequence CorrectionPreprint
Baid, G., Cook, D. E., Shafin, K., Yun, T., Llinares-López, F., Berthet, Q., Wenger, A., Rowell, W., Nattestad, M., Yang, H., et al. (2021) · bioRxiv · doi:10.1101/2021.08.31.458403 · Cited by 13
Accelerating Antibody Design with Active LearningPreprint
Seo, S.-W., Kwak, M. W., Kang, E., Kim, C., Park, E. Y., Kang, T. H., & Kim, J. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.09.12.507690 · Cited by 8
Improved the Protein Complex Prediction with Protein Language ModelsPreprint
Chen, B., Xie, Z., Qiu, J., Ye, Z., Xu, J., & Tang, J. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.09.15.508065 · Cited by 8
Scoring alignments by embedding vector similarity
Ashrafzadeh, S., Golding, G. B., Ilie, S., & Ilie, L. (2023) · doi:10.1101/2023.08.30.555602 · Cited by 8
“Many protein embeddings have been proposed, including ProtVec [4], SeqVec [10], PRoBERTa [22], MSA-transformer [27], ESM-1b and ESM2 [28], and the models in the ProtTrans project: ProtTXL, ProtBert, ProtXLNet, ProtAlbert, ProtElectra, and ProtT5 [8].”
MethodACP-ESM2: The prediction of anticancer peptides based on pre-trained classifier
Song, H., Lin, X., Zhang, H., & Yin, H. (2024) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2024.108091 · Cited by 7
Piomponi, V., Cazzaniga, A., & Cuturello, F. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c01090 · Cited by 7
Probing the dark energy in the functional protein universe
Galpern, E. A., Bueno, C., Sanchez, I. E., Wolynes, P. G., & Ferreiro, D. U. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2531111123 · Cited by 7
Exploring Pair-Aware Triangular Attention for Biomedical Relation Extraction
Chen, L., Su, J., Lam, T., & Luo, R. (2023) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3584371.3612994 · Cited by 8
Wang, B., & Li, W. (2024) · Genes · doi:10.3390/genes15081090 · Cited by 7
Inter-domain distance prediction based on deep learning for domain assembly
Ge, F., Peng, C., Cui, X., Xia, Y., & Zhang, G. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad100 · Cited by 7
Zhu, X., Qin, C., Wang, F., Yang, F., He, B., Zhao, Y., & Yao, J. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.24.600337 · Cited by 8
Protein language model embedded geometric graphs power inter-protein contact prediction
Si, Y., & Yan, C. (2023) · doi:10.1101/2023.01.07.523121 · Cited by 5
From Traditional Methods to Deep Learning Approaches: Advances in Protein–Protein Docking
Jiang, L., Zhang, K., Zhu, K., Zhang, H., Shen, C., & Hou, T. (2025) · Wiley Interdisciplinary Reviews Computational Molecular Science · doi:10.1002/wcms.70016 · Cited by 6
Tremmel, R., Honoré, A., Park, Y., Zhou, Y.-T., Xiao, M., & Lauschke, V. (2025) · Pharmacogenomics (London) · doi:10.1080/14622416.2025.2504863 · Cited by 6
Learning the language of phylogeny with MSA Transformer
Chen, R., Foley, G., & Mikael Bodén, A. (2025) · Cell Systems · doi:10.1016/j.cels.2025.101445 · Cited by 7
Learning the shape of protein micro-environments with a holographic convolutional neural networkPreprint
Pun, M. N., Ivanov, A., Bellamy, Q., Montague, Z., LaMont, C., Bradley, P. H., Otwinowski, J., & Nourmohammad, A. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.10.31.514614 · Cited by 7
Funk, J., Machado, L., Bradley, S. A., Napiorkowska, M., Gallegos-Dextre, R., Pashkova, L., Madsen, N. G., Webel, H. E., Phaneuf, P. V., Jenkins, T. P., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.01.616114 · Cited by 7
Intrinsic dataset features drive mutational effect prediction by protein language modelsPreprint
Vieira, L. C., Lin, S., & Wilke, C. O. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.03.08.710389 · Cited by 12
Direct coupling analysis and the attention mechanism
(2025) · doi:10.1186/s12859-025-06062-y · Cited by 5
Pratyush, P., Pokharel, S., Schulze, S., Bramer, L., Newman, R. H., & KC, D. B. (2025) · Methods in molecular biology · doi:10.1007/978-1-0716-4623-6_19 · Cited by 6
Protein embeddings and local alignments
Malec, J., Golding, G. B., & Ilie, L. (2026) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.12.002 · Cited by 5
Wang, B., Wang, C., Chen, J., Liu, D., Sun, C., Zhang, J., Zhang, K., & Li, H. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01198-9 · Cited by 5
Toward life with a 19–amino acid alphabet through generative artificial intelligence design
Liu, L., Rochereau, C., Kozlov, S., Urtecho, G., Liu, X., Zhao, J., Wang, J., Huang, Y., Qu, Y., Zhang, Z., et al. (2026) · Science · doi:10.1126/science.aeb5171 · Cited by 7
MetaEnzyme: Meta Pan-Enzyme Learning for Task-Adaptive Redesign
Zheng, J., Zhang, H., Xu, Q., Zeng, A., & Li, S. Z. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680883 · Cited by 5
Zhang, Y., Qin, Y., Pourmirzaei, M., Shao, Q., Wang, D., & Xu, D. H. (2025) · Methods in molecular biology · doi:10.1007/978-1-0716-4623-6_2 · Cited by 5
Data-driven design of orthogonal protein-protein interactions
Malinverni, D., & Babu, M. M. (2023) · Science Signaling · doi:10.1126/scisignal.abm4484 · Cited by 7
Foundation models in molecular biology
Si, Y., Zou, J., Gao, Y., Chuai, G., Liu, Q., & Chen, L. (2024) · Biophysics Reports · doi:10.52601/bpr.2024.240006 · Cited by 6
Generating interacting protein sequences using domain-to-domain translation
Meynard-Piganeau, B., Fabbri, C., Weigt, M., Pagnani, A., & Feinauer, C. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad401 · Cited by 6
Distillation of MSA Embeddings to Folded Protein Structures with Graph TransformersPreprint
Costa, A. D. S., Ponnapati, M., Jacobson, J. M., & Chatterjee, P. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.06.02.446809 · Cited by 6
PeTriBERT : Augmenting BERT with tridimensional encoding for inverse protein folding and designPreprint
Dumortier, B., Antoine Liutkus, A., Carré, C., & Krouk, G. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.08.10.503344 · Cited by 6
Retrieved Sequence Augmentation for Protein Representation LearningPreprint
Ma, C., Zhao, H., Zheng, L., Xin, J., Li, Q., Wu, L., Deng, Z., Yang, L., Liu, Q., & Kong, L. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.02.22.529597 · Cited by 7
Improving Inverse Folding models at Protein Stability Prediction without additional Training or DataPreprint
Dutton, O., Bottaro, S., Invernizzi, M., Redl, I., Chung, A., Hoffmann, F., Henderson, L. D., Ruschetta, S., Airoldi, F., Owens, B. M. J., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.15.599145 · Cited by 6
Ma, J., Wang, Z., Tong, C., Yang, Q., Zhang, L., & Liu, H. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf321 · Cited by 5
A Benchmarking Platform for Assessing Protein Language Models on Function-Related Prediction Tasks
Çevrim, E., Yiğit, M. G., Ulusoy, E., Yılmaz, A., & Doğan, T. (2025) · Methods in molecular biology · doi:10.1007/978-1-0716-4662-5_14 · Cited by 7
Zero-shot benchmarking of RNA language models in structural, functional, and evolutionary learning
Wang, H., Zhang, Y., Chen, J., Zhan, J., & Zhou, Y. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag098 · Cited by 5
A Novel COVID-19 Image Classification Method Based on the Improved Residual Network
Chen, H., Zhang, T., Chen, R., Zhu, Z., & Wang, X. (2022) · Electronics · doi:10.3390/electronics12010080 · Cited by 4
Wu, D., & Wang, M. (2023) · Internet Technology Letters · doi:10.1002/itl2.462 · Cited by 4
CFOM: Lead Optimization For Drug Discovery With Limited Data
Kaminsky, N., Singer, U., & Radinsky, K. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3614807 · Cited by 4
Anton, B., Besalú, M., Fornés, O., Bonet, J., Molina, A., Molina-Fernández, R., Cuevas, G. D. L., Fernández‐Fuentes, N., & Oliva, B. (2021) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqab027 · Cited by 4
Chu, H. Y., & Wong, A. S. L. (2021) · Advanced Genetics · doi:10.1002/ggn2.202100038 · Cited by 4
Entropy and Variability: A Second Opinion by Deep Learning
Rademaker, D. T., Xue, L. C., Hoen, P. A. C. ’., & Vriend, G. (2022) · Biomolecules · doi:10.3390/biom12121740 · Cited by 4
A Transformer Architecture for Online Gesture Recognition of Mathematical Expressions
Ramo, M., & Silvestre, G. C. M. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-26438-2_5 · Cited by 5
FLIGHTED: Inferring Fitness Landscapes from Noisy High-Throughput Experimental DataPreprint
Sundar, V., Tu, B., Guan, L., & Esvelt, K. M. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.03.26.586797 · Cited by 4
Artificial Intelligence Learns Protein PredictionPreprint
Heinzinger, M., & Rost, B. (2024) · Cold Spring Harbor Perspectives in Biology · doi:10.1101/cshperspect.a041458 · Cited by 4
Metadata
- Authors
- Rao, R., Liu, J., Verkuil, R., Meier, J., Canny, J. F., Abbeel, P., Sercu, T., & Rives, A.
- Affiliations
- Facility for Antiproton and Ion Research (DE)
- Meta (United States) (US)
- University of California, Berkeley (US)
- Corresponding author
- Rao, Roshan
- Rives, Alexander
- ORCID
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2021-02-13
- OpenAlex
- W3133458480
- Cited by
- 350
- References
- 47
- Citations by year
- 2026: 63 · 2025: 90 · 2024: 74 · 2023: 45 · 2022: 47 · 2021: 28
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- Funders
- Defense Advanced Research Projects Agency
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- protein language models, multiple sequence alignment, masked language modeling, protein structure prediction, parameter efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
