The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
MSA Transformer
Open accessAbstract
Abstract Unsupervised protein language models trained across millions of diverse sequences learn structure and function of proteins. Protein language models studied to date have been trained to perform inference from individual sequences. The longstanding approach in computational biology has been to make inferences from a family of evo lutionarily related sequences by fitting a model to each family independently. In this work we combine the two paradigms. We introduce a protein language model which takes as input a set of sequences in the form of a multiple sequence alignment. The model interleaves row and column attention across the input sequences and is trained with a variant of the masked language modeling objective across many protein families. The performance of the model surpasses current state-of-the-art unsupervised structure learning methods by a wide margin, with far greater parameter efficiency than prior state-of-the-art protein language models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We used the same set of clusters to make predictions using the unsupervised deep learning model MSA Transformer36 and found that these clusters contained evolutionary couplings for both states, and the score based on contact maps correlated with the root mean squared deviation (r.m.s.d.) in AF2 (Methods and Extended Data Fig. 3).”
From Predicting multiple conformations via sequence clustering and AlphaFold2 (2024) · cites this paper as [36]
In the citing paper
“MSA transformer is an unsupervised learning method, which signifies that its contact predictions purely reflect evolutionary couplings learned in sequences, rather than being supervised on structure as is the case AF2.”
From Predicting multiple conformations via sequence clustering and AlphaFold2 (2024) · cites this paper as [36]
In the citing paper
“Among many successful neural network architectures, attention-based networks are extremely powerful for modeling complicated interactions between entities, making them suitable for modeling proteins whose functions are largely determined by interactions between amino acids (34–37).”
From Deep learning guided optimization of human antibody against SARS-CoV-2 variants with broad neutralization (2022) · cites this paper as [36]
In the citing paper
“In the MSA Transformer (16), the neural network model exploits the equivalence of aligned positions in the MSA in order to derive embeddings via a tied row attention framework.”
From Ultrafast end-to-end protein structure prediction enables high-throughput exploration of uncharacterised proteins (2020) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Self-citationRives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
Self-citationRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
CCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00069 · Cited by 3,045
Morcos, F., Pagnani, A., Lunt, B., Bertolino, A., Marks, D. S., Sander, C., Zecchina, R., Onuchic, J. N., Hwa, T., & Weigt, M. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1111471108 · Cited by 1,434
Improved protein structure prediction using predicted interresidue orientations
Yang, J., Anishchenko, I., Park, H., Peng, Z., Ovchinnikov, S., & Baker, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914677117 · Cited by 1,421
HH-suite3 for fast remote homology detection and deep protein annotation
Steinegger, M., Meier, M., Mirdita, M., Vöhringer, H., Haunsberger, S. J., & Söding, J. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3019-7 · Cited by 1,145
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131
Research works printed but not linked: 5
- 10.1002/(SICI)1097-0134(19990301)34:4508::AID-PROT103.0.CO;2-4 · DOI
- Ho, J. , Kalchbrenner, N. , Weissenborn, D. , and Salimans, T. Axial Attention in Multidimensional Transformers. arXiv, 12 2019. URL http://arxiv.org/abs/1912.12180. · arXiv
- Sturmfels, P. , Vig, J. , Madani, A. , and Rajani, N. F. Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models. bioRxiv, 11 2020. URL http://arxiv.org/abs/2012.00195. · arXiv
- Thomas, J. , Ramakrishnan, N. , and Bailey-Kellogg, C. Graphical models of residue coupling in protein families, 4 2008. ISSN 15455963. URL https://pubmed.ncbi.nlm.nih.gov/18451428/.
- 2017, Attention is All you Need, Advances in Neural Information Processing Systems, 30, 5998
Cited by
Sorted by citations
Items 301–355 of 355
Cui, X., Ge, L., Yang, X., Li, X., & Zhang, G. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.10.23.684122
Desai, D., Modi, S., & Kumar, L. (2025) · Advances in 21st century human settlements · doi:10.1007/978-981-96-9760-1_9
Meng, D., Alvarez, H. M. G., Glavina, J., Leonetti, C. O., Pollastri, G., & Chemes, L. B. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.10.29.685455 · Cited by 1
Elreify, H. M., El‐Samie, F. E. A., Dessouky, M. I., Torkey, H., El‐Khamy, S. E., & Shalaby, W. A. (2025) · Scientific Reports · doi:10.1038/s41598-025-15883-z
Patjoshi, S., Madan, S., & Fröhlich, H. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.11.05.686884
Full-Atom Protein-Protein Interaction Prediction via Atomic Equivariant Attention Network
Wang, C., Yang, C. H., Yang, W., Song, L., & Shi, C. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761352
Elreify, H. M., El‐Samie, F. E. A., Dessouky, M. I., Torkey, H., El‐Khamy, S. E., & Shalaby, W. A. (2025) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-025-00572-8
Ma, R., He, C., Zhang, Z., Zheng, H., & Duan, L. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.11.11.687944
Barethiya, S., Huang, J. W., Liu, X., Guan, H., & Chen, J. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.11.09.687530
Mazza, F., Gastaldello, F., Dalfovo, D., Lattanzi, G., & Romanel, A. (2025) · Nucleic Acids Research · doi:10.1093/nar/gkaf1184
Sarachakov, A., Yudina, A., Svekolkin, V., Parfenenkova, A., Spektor, M., Oshchepkova, P., Pak, M. A., Antysheva, Z., Balabanian, L., Brown, J. H., et al. (2025) · Human Genetics · doi:10.1007/s00439-025-02775-5
Benchmarking and Consensus Ranking of Inverse Folding Models for Protein-Ligand Interface Design
Wei, Y., Guerrini, U., & Eberini, I. (2025) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3768322.3769031
Wynne, J. H., Azbijari, N., Thurber, A. R., & David, M. M. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2025.12.17.695047
AI-Driven Insights into Protein Sequence Analysis Landscape
Asim, M. N., Ahmed, S., & Dengel, A. (2025) · Springer eBooks · doi:10.1007/978-3-031-90454-7_5
A Hybrid TCN and BiGRU-Based Deep Learning Model for Protein Function Prediction
Sharma, L., Deepak, A., Ranjan, A., & Kumar, P. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-7707-8_24
Synthetic Genomic and Omics Data Applications in Precision Medicine and Biomedical Research
Matchanova, B., Niyazova, N., Salayeva, L., Ahmad, M., & Shojonov, M. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-5641-9.ch002
Progress of Deep Learning Prediction of CD8+ T‐Cell Epitopes
Cheng, X., Wu, H., Chen, P., Liu, R., Lei, Y., Mei, H., & Wang, P. (2025) · PROTEOMICS · doi:10.1002/pmic.70101 · Cited by 1
Separating selection from mutation in antibody language modelsPreprint
Matsen, F. A., IV, Dumm, W., Sung, K., Johnson, M. M., Rich, D., Starr, T. N., Song, Y. S., Fukuyama, J., & Haddox, H. K. (2026) · eLife · doi:10.7554/elife.109644.1 · Cited by 1
Correia, C., Ung, C. Y., Grenko, C. M., Zhu, S., Billadeau, D. D., & Li, H. (2026) · Challenges and advances in computational chemistry and physics · doi:10.1007/978-3-032-07880-3_1
Rousset, Y., Kroll, A., & Lercher, M. J. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.01.23.701068
Separating selection from mutation in antibody language modelsPreprint
Matsen, F. A., IV, Dumm, W., Sung, K., Johnson, M. M., Rich, D., Starr, T. N., Song, Y. S., Fukuyama, J., & Haddox, H. K. (2026) · eLife · doi:10.7554/elife.109644.2
Orlov, A. V., Makus, Y. V., Ashniev, G. A., Orlova, N. N., & Nikitin, P. I. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.03.04.709491 · Cited by 2
From sequences to knowledge, improving and learning from sequence alignments
Blassel, L. (2022) · theses.fr (ABES) · doi:10.70675/edb2db56z9b8ez4f44zb76az28681bf8bc51
Structural prediction of protein assemblies with deep learning
Bret, H. (2023) · theses.fr (ABES) · doi:10.70675/b3849091zce01z4ab0z91efz2b71d9cc754d
A Graph Machine Learning Journey : from Explainability to Climate Action
Duval, A. (2024) · theses.fr (ABES) · doi:10.70675/53194ee1z2ba4z4903zaf99ze71420af953b
QOMD Simulator and Protein Atomic Coordinate Prediction
Ma, T., & Meng, Z. (2025) · International Conference on Computational Modeling, Simulation and Data Analysis (CMSDA) · doi:10.1145/3796731.3796791
Rahimzadeh, F., Khanli, L. M., Salehpour, P., & Golabi, F. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132559 · Cited by 1
Wu, H., Nie, Z., Zhang, H., & Ren, Z. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c03204
Generative Protein Design: From Deep Learning Algorithms to Translational Applications
Luo, S., & Zhou, B. (2026) · International Journal of Molecular Sciences · doi:10.3390/ijms27093917 · Cited by 1
Jiang, Y., Wang, D., Ihoeghian, N., Imam, I. A., Chen, J., Xu, D., & Shao, Q. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.04.29.721692
CPU-GPU Workload Distribution during Throughput-Oriented LLM Inference on Single-GPU Systems
Park, D., & Egger, B. (2026) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3816433
Qurat-ul-ain, Outeiral, C., Cagiada, M., Teh, Y. W., & Deane, C. M. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.28.728196 · Cited by 1
Species phylogenetics using intraprotein residue dependence inferred with protein foundation modelsPreprint
Alibutud, R. F., Mowlaei, M. E., & Kumar, S. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.26.727927
Interpreting embeddings from genome and protein language models
May, L. E., White, J. B., & Roell, G. W. (2026) · Biotechnology Advances · doi:10.1016/j.biotechadv.2026.108945
ProMiSE: Protein Multi-State Evaluation Benchmark in Biological ContextsPreprint
Ku, B., Kim, S., Kim, Y. W., Park, H., & Seok, C. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.11.731790
Differences between protein fitness models can be used to design variants of altered specificityPreprint
Berry, S. P., Gaudet, R., & Marks, D. S. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.10.731299
Phylogenetic tree inference using generative modelsPreprint
Dotan, E., Schers, A., Wygoda, E., Pupko, T., & Belinkov, Y. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.14.732140
Fei, X., Gu, J., Zhou, C., Liu, Y., Li, Z., & Kang, Y. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6c00969
Popformer: Learning general signatures of positive selection with a self-supervised transformer
Zong, L., Friedler, S. A., & Mathieson, S. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014492
Li, K., Peng, X., Qian, X., Li, S., & Zhang, X. (2026) · Microbiome · doi:10.1186/s40168-026-02467-w
Thai, K., Duong, C. Q., Nguyen, T. T. T., Tran, V.-T., Mai, Q.-M., & Le, M.-T. (2026) · Royal Society of Chemistry eBooks · doi:10.1039/9781837070206-00001
Machine Learning in Protein Structure Prediction
Thai, K., Nguyen, D.-N., Vo, T.-P., & Le, M.-T. (2026) · Royal Society of Chemistry eBooks · doi:10.1039/9781837070206-00064
Endoh, T., Camer, G. A., Kayama, K., Endoh, D., & Teraoka, H. (2026) · Proteomes · doi:10.3390/proteomes14030036
Kendiukhov, I. (2026) · BMC Bioinformatics · doi:10.1186/s12859-026-06538-5
PAC-Net: A physics-guided multimodal hybrid network for antibody-antigen interaction prediction
Wei, S., Tang, C., Chen, Y., Zhang, J., Tan, J., & Lv, L. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105097
Elucidating enzyme–substrate specificity through co-folding foundation modelPreprint
Cheng, X., Seo, S., Huh, C., Chen, J., Jiang, S., Guo, P., Weng, J., Kim, W. Y., & Jin, W. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.07.30.741672
Structure-enhanced cross-modal biomedical prediction
Wang, M., Zhang, T., Li, Y., & Chang, Y. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116835
Tzavella, K., Olsen, C., & Vranken, W. (2026) · Computational and Structural Biotechnology Journal · doi:10.34133/csbj.0226
Deep Learning in Enzyme Function Prediction and Novel Enzyme Discovery
Gao, R., Liu, Y., Duan, C., Qin, N., Liu, K., Tsuboi, T., & Chen, Z. (2026) · ACS Synthetic Biology · doi:10.1021/acssynbio.6c00633
Hierarchical Organization of Evolutionary Constraint in Duplicated EF-Hand Motifs
Basit, A. (2026) · Journal of Molecular Evolution · doi:10.1007/s00239-026-10353-0
Machine Learning Empowers Enzyme Discovery, Structure Prediction, Engineering, and Design
王木强, Jiang, L., Yu, H., Yu, J., Zheng, W., & Wu, J. (2026) · ACS Catalysis · doi:10.1021/acscatal.6c07060
Bulloni, M., Vescio, M., Bajolin, F. D. M., & Pattini, L. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag552
Introduction to the principles of protein engineering
Sharp, J. F., Nelson, M. K., Rustembek, M., Samanta, R., & Stern, L. A. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-44536-1.00010-9
GPN-MSA: an alignment-based DNA language model for genome-wide variant effect prediction
Benegas, G., Albors, C., Aw, A. J., Ye, C., & Song, Y. S. (2023) · doi:10.1101/2023.10.10.561776 · Cited by 118
“It draws inspiration from the MSA Transformer [19], a protein language model trained on MSAs of diverse protein families; it was originally designed for structure prediction but was later shown to achieve excellent missense variant effect prediction performance [5].”
PerturbNet predicts single-cell responses to unseen chemical and genetic perturbations
Yu, H., & Welch, J. D. (2022) · doi:10.1101/2022.07.20.500854 · Cited by 57
“711 As with other transformer models [46], the ESM model was pre-trained on large protein sequence 712 datasets [47].”
Method
Metadata
- Authors
- Rao, R., Liu, J., Verkuil, R., Meier, J., Canny, J. F., Abbeel, P., Sercu, T., & Rives, A.
- Affiliations
- Facility for Antiproton and Ion Research (DE)
- Meta (United States) (US)
- University of California, Berkeley (US)
- Corresponding author
- Rao, Roshan
- Rives, Alexander
- ORCID
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2021-02-13
- OpenAlex
- W3133458480
- Cited by
- 350
- References
- 47
- Citations by year
- 2026: 63 · 2025: 90 · 2024: 74 · 2023: 45 · 2022: 47 · 2021: 28
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- Funders
- Defense Advanced Research Projects Agency
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- protein language models, multiple sequence alignment, masked language modeling, protein structure prediction, parameter efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
