PreprintCited by 342Protein Structure and DynamicsMachine Learning in BioinformaticsRNA and protein synthesis mechanisms

The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.

Transformer protein language models are unsupervised structure learners

Open access

Rao, Roshan, Meier, Joshua, Sercu, Tom, Ovchinnikov, Sergey, Rives, Alexander·2020·bioRxiv (Cold Spring Harbor Laboratory)

DOI 10.1101/2020.12.15.422761

Open this paper in CoChat — free account Open PDF

Abstract

A bstract: Unsupervised contact prediction is central to uncovering physical, structural, and functional constraints for protein structure determination and design. For decades, the predominant approach has been to infer evolutionary constraints from a set of related sequences. In the past year, protein language models have emerged as a potential alternative, but performance has fallen short of state-of-the-art approaches in bioinformatics. In this paper we demonstrate that Transformer attention maps learn contacts from the unsupervised language modeling objective. We find the highest capacity models that have been trained to date already outperform a state-of-the-art unsupervised contact prediction pipeline, suggesting these pipelines can be replaced with a single forward pass of an end-to-end model. 1

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1101/2020.12.15.422761

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“A similar scaling law has been observed previously for supervised secondary structure prediction (Rives et al., 2019), and parallels observations in the NLP community (Kaplan et al., 2020; Brown et al., 2020).”
Background
mixOmics: An R package for ‘omics feature selection and multiple data integration
Rohart, F., Gautier, B., Singh, A., & Lê Cao, K.-A. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005752 · Cited by 4,056
Improved protein structure prediction using potentials from deep learning
Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1923-7 · Cited by 3,378
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Self-citation
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
Self-citation

Show all 63 references

Research works printed but not linked: 4

10.1002/(SICI)1097-0134(19990301)34:4(508::AID-PROT10)3.0.CO;2-4 · DOI
Robert D. Finn , Alex Bateman , Jody Clements , Penelope Coggill , Ruth Y. Eberhardt , Sean R. Eddy , Andreas Heger , Kirstie Hetherington , Liisa Holm , Jaina Mistry , Erik L.L. Sonnhammer , John Tate , and Marco Punta . Pfam: The protein families database, 1 2014. ISSN 03051048. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3965110/.
10.1101/622803 · DOI · Paper page
10.1101/2020.10.12.336859 · DOI

Sorted by citations

Items 201–300 of 342

PeTriBERT : Augmenting BERT with tridimensional encoding for inverse protein folding and designPreprint
Dumortier, B., Antoine Liutkus, A., Carré, C., & Krouk, G. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.08.10.503344 · Cited by 6
Tuned Fitness Landscapes for Benchmarking Model-Guided Protein DesignPreprint
Thomas, N., Agarwala, A., Belanger, D., Song, Y. S., & Colwell, L. J. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.10.28.514293 · Cited by 5
Retrieved Sequence Augmentation for Protein Representation LearningPreprint
Ma, C., Zhao, H., Zheng, L., Xin, J., Li, Q., Wu, L., Deng, Z., Yang, L., Liu, Q., & Kong, L. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.02.22.529597 · Cited by 7
ProtHyena: A fast and efficient foundation protein language model at single amino acid ResolutionPreprint
Zhang, Y., & Okumura, M. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.01.18.576206 · Cited by 5
A Generative Foundation Model for Antibody DesignPreprint
Wang, R., Wu, F., Shi, J., Song, Y., Kong, Y., Ma, J., He, B., Yan, Q., Ying, T., Zhao, P., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.09.12.675771 · Cited by 7
Evaluating data partitioning strategies for accurate prediction of protein-ligand binding free energy changes in mutated proteins
Xie, L., Bao, G., Zhang, D., Xu, L., Xu, X., & Chang, S. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.10.020 · Cited by 5
Stable de novo protein design via joint conformational landscape and sequence optimization
Cho, Y., Dauparas, J., Tsuboyama, K., Rocklin, G. J., & Ovchinnikov, S. (2025) · Nature Communications · doi:10.1038/s41467-025-66526-w · Cited by 6
Self-citation
Effective Tool Augmented Multi-Agent Framework for Data Analysis
Zhang, X., Mao, Z., Chen, Z., & Gao, S. (2024) · Data Intelligence · doi:10.3724/2096-7004.di.2024.0013 · Cited by 7
CCPL: Cross-Modal Contrastive Protein Learning
Zheng, J., & Li, S. Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78398-2_2 · Cited by 5
SPOT-Contact-Single: Improving Single-Sequence-Based Prediction of Protein Contact Map using a Transformer Language ModelPreprint
Singh, J., Litfin, T., Singh, J., Paliwal, K. K., & Zhou, Y. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.06.19.449089 · Cited by 4
SPOT-1D-LM: Reaching Alignment-profile-based Accuracy in Predicting Protein Secondary and Tertiary Structural Properties without AlignmentPreprint
Singh, J., Paliwal, K. K., Singh, J., & Zhou, Y. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.10.16.464622 · Cited by 4
The Impact of Protein Dynamics on Residue-Residue Coevolution and Contact PredictionPreprint
Fung, A., Koehl, A., Jagota, M., & Song, Y. S. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.10.16.512436 · Cited by 3
Artificial Intelligence Learns Protein PredictionPreprint
Heinzinger, M., & Rost, B. (2024) · Cold Spring Harbor Perspectives in Biology · doi:10.1101/cshperspect.a041458 · Cited by 4
Directed Evolution of Proteins via Bayesian Optimization in Embedding Space
Soldát, M., & Kléma, J. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822356 · Cited by 3
Efficiently Predicting Mutational Effect on Homologous Proteins by Evolution Encoding
Zhong, Z., & Mottin, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70368-3_24 · Cited by 3
HAIRpred : Prediction of human antibody interacting residues in an antigen from its primary structure
Sahni, R., Kumar, N., & Raghava, G. P. S. (2025) · Protein Science · doi:10.1002/pro.70212 · Cited by 3
A Self-Supervised Framework for Learning Biological Entities Representation by Fusing Class Information
Li, N., Yang, Z., Wang, J., & Lin, H. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3273333 · Cited by 3
MetaLLM: Residue-Wise Metal Ion Prediction Using Deep Transformer Model
Shishir, F. S., Sarker, B., Rahman, F., & Shomaji, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-34960-7_4 · Cited by 3
PCP-GC-LM: single-sequence-based protein contact prediction using dual graph convolutional neural network and convolutional neural network
Ouyang, J., Gao, Y., & Yang, Y. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05914-3 · Cited by 3
ILMCNet: A Deep Neural Network Model That Uses PLM to Process Features and Employs CRF to Predict Protein Secondary Structure
Dong, B., Su, H., Xu, D., Hou, C., Liu, Z., Niu, N., & Wang, G. (2024) · Genes · doi:10.3390/genes15101350 · Cited by 3
Knowledge from Large-Scale Protein Contact Prediction Models Can Be Transferred to the Data-Scarce RNA Contact Prediction Task
Jian, Y., Gao, C., Zeng, C., Zhao, Y., & Vosoughi, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78192-6_27 · Cited by 3
Leveraging pretrained deep protein language model to predict peptide collision cross section
Nakai-Kasai, A., Ogata, K., Ishihama, Y., & Tanaka, T. (2024) · bioRxiv · doi:10.1038/s42004-025-01540-z · Cited by 4
Infer global, predict local: Quantity-relevance trade-off in protein fitness predictions from sequence data
Posani, L., Rizzato, F., Monasson, R., & Cocco, S. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011521 · Cited by 4
LM-GVP: A Generalizable Deep Learning Framework for Protein Property Prediction from Sequence and StructurePreprint
Wang, Z., Combs, S. A., Brand, R. M., Calvo, M. R., Xu, P., Price, G., Golovach, N., Salawu, E. O., Wise, C. J., Ponnapalli, S. P., et al. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.09.21.460852 · Cited by 3
Adversarial Attacks on Protein Language ModelsPreprint
Carbone, G., Cuturello, F., Bortolussi, L., & Cazzaniga, A. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.10.24.513465 · Cited by 4
Annotation Vocabulary (Might Be) All You NeedPreprint
Hallee, L., Rafailidis, N., Horger, C., Hong, D. K., & Gleghorn, J. P. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.07.30.605924 · Cited by 7
Predicting RNA Structure Utilizing Attention from Pretrained Language Models
Papazoglou, I., Chatzigoulas, A., Tsekenis, G., & Cournia, Z. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c02094 · Cited by 4
BEST: Basic Embedding Search Tool Enhancing Discovery of Novel Enzyme
Wu, Y., Yi, X., Tan, Y., Yu, H., Fan, G., & Zheng, G. (2025) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-025-00753-z · Cited by 2
Inferred global dense residue transition graphs from primary structure sequences enable protein interaction prediction via directed graph convolutional neural networks
Ebeid, I. A., Tang, H., & Gu, P. (2025) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2025.1651623 · Cited by 2
S 2 ‐ PepAnalyst : A Web Tool for Predicting Plant Small Signalling Peptides
Vomo-Donfack, K. L., Abaach, M., Luna, A. M., GINOT, G., Doblas, V. G., & Morilla, I. (2026) · Plant Biotechnology Journal · doi:10.1111/pbi.70536 · Cited by 4
Active Finetuning Protein Language Model: A Budget-Friendly Method for Directed Evolution
Qin, M., Ding, K., Wu, B., Li, Z., Yang, H., Wang, Z., Ye, H., Yu, H., Chen, H., & Zhang, Q. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230481 · Cited by 3
Chunk-BERT: Boosted keyword extraction for long scientific literature via BERT with chunking capabilities
Yuan, Z. P., Cai, R., Maimaiti, M., & Abiderexiti, K. (2023) · IEEE International Conference on Pattern Recognition and Machine Learning (PRML) · doi:10.1109/prml59573.2023.10348182 · Cited by 2
Multimodal-AIR-BERT: A Multimodal Pre-trained Model for Antigen Specificity Prediction in Adaptive Immune Receptors
Xiao, Y., Huang, Y., Zhao, Y. S., Xu, F., Ren, Q., He, B., Yao, J., & Liu, X. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385479 · Cited by 2
Predicting Phosphoglycerylation with Transformer Features and Deep Learning
Chandra, A. A., Sharma, A. K., Dehzangi, A., Tsunoda, T., & Sattar, A. (2023) · IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) · doi:10.1109/csde59766.2023.10487711 · Cited by 2
Range-limited Heaps' law for functional DNA words in the human genome.
Li, W.-T., Almirantis, Y., & Provata, A. (2024) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2024.111878 · Cited by 4
Drug Molecule Generation Method Based on Fusion of Protein Sequence Features
Wang, M., Zhang, X., Liu, H., Lin, X., Hu, J., & Pang, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5692-6_11 · Cited by 2
FvFold: A model to predict antibody Fv structure using protein language model with residual network and Rosetta minimization
Sherpa, P., Chong, K. T., & Tayara, H. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109128 · Cited by 2
Improving Protein Secondary Structure Prediction by Deep Language Models and Transformer Networks
Wu, T., Cheng, W., & Cheng, J. J. (2024) · Methods in molecular biology · doi:10.1007/978-1-0716-4196-5_3 · Cited by 2
PLM-ATG: Identification of Autophagy Proteins by Integrating Protein Language Model Embeddings with PSSM-Based Features
Wang, Y., & Wang, C. (2025) · Molecules · doi:10.3390/molecules30081704 · Cited by 2
Inferring context-specific site variation with evotuned protein language models
Lytras, S., Strange, A. P., Ito, J., & Sato, K. (2026) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqag018 · Cited by 3
O impacto da inteligência artificial nas ciências da vida através da bioinformática
dos Santos, L. M., Mariano, D., & de Melo-Minardi, R. C. (2023) · Revista da Universidade Federal de Minas Gerais · doi:10.35699/2965-6931.2023.47996 · Cited by 4
Towards foundation models that learn across biological scales
Kalfon, J., Cantini, L., & Gabriel Peyré, A. (2026) · Machine Learning Health · doi:10.1088/3049-477x/aea764 · Cited by 4
A study on experimental bias in post-translational modification predictorsPreprint
Zuallaert, J., Ramasamy, P., Bouwmeester, R., Callewaert, N., & Degroeve, S. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.11.28.518163 · Cited by 2
Infer global, predict local: quantity-quality trade-off in protein fitness predictions from sequence dataPreprint
Posani, L., Rizzato, F., Monasson, R., & Cocco, S. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.12.12.520004 · Cited by 2
Ultra-Accurate Classification and Discovery of Functional Protein-Coding Genes from Microbiomes Using FunGeneTyper: An Expandable Deep Learning-Based FrameworkPreprint
Zhang, G., Wang, H., Zhang, Z., Zhang, L., Guo, G., Yang, J., Yuan, F., & Ju, F. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.12.28.522150 · Cited by 2
Zero-shot prediction of mutation effects on protein function with multimodal deep representation learningPreprint
Shu, W., Cheng, P., Mao, C., Tang, J., Yang, S., Gu, Q., Han, W., Yaofeng, C., Zhou, J., Li, W., et al. (2023) · Research Square · doi:10.21203/rs.3.rs-3358917/v1 · Cited by 2
Semantical and Geometrical Protein Encoding Toward Enhanced Bioactivity and ThermostabilityPreprint
Tan, Y., Zhong, B., Zheng, L., Fan, G., & Hong, L. (2024) · eLife · doi:10.7554/elife.98033.1 · Cited by 2
Semantical and geometrical protein encoding toward enhanced bioactivity and thermostability
Tan, Y., Zhou, B., Zheng, L., Fan, G., & Hong, L. (2025) · eLife · doi:10.7554/elife.98033 · Cited by 2
The end of protein structure prediction: Improving prediction accuracy in chimeric proteins by windowed multiple sequence alignmentPreprint
Vedula, S., Bronstein, A. M., & Marx, A. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.06.616858 · Cited by 2
Implicit modeling of the conformational landscape and sequence allows scoring and generation of stable proteinsPreprint
Cho, Y., Dauparas, J., Tsuboyama, K., Rocklin, G., & Ovchinnikov, S. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.12.20.629706 · Cited by 2
Self-citation
Mechanistic evidence that motif-gated domain recognition drives contact prediction in protein language modelsPreprint
Nainani, J., Reimer, B. M., Watts, C., Jensen, D., & Green, A. G. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.22.671739 · Cited by 2
Evolutionary discovery and characterization of fungal transcriptional activators using active learningPreprint
Waldburger, L. M., Nisonoff, H. M., Zintel, M. A., Kirkpatrick, L. D., Lam, A., Lanclos, N., Keasling, J. D., Staller, M. V., & Shih, P. M. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.09.12.675635 · Cited by 2
Squidly: Enzyme Catalytic Residue Prediction Harnessing a Biology-Informed Contrastive Learning FrameworkPreprint
Rieger, W. J., Mikael Bodén, A., Arnold, F. H., & Mora, A. (2025) · eLife · doi:10.7554/elife.108186.1 · Cited by 2
Predicting SARS-CoV-2 evolution dynamics with spatiotemporal resolution by DMS-empowered protein language modelPreprint
Yang, S., Luo, X., Luo, J., Jian, F., & Cao, Y. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.10.17.683094 · Cited by 2
Hit or Miss: Understanding Emergence and Absence of Homo-oligomeric Contacts in Protein Language ModelsPreprint
Zhang, Z., Akiyama, Y., Cho, Y., Jajoo, S., & Ovchinnikov, S. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.11.16.688745 · Cited by 2
Self-citation
Deep models of protein evolution in time generate realistic evolutionary trajectories and functional proteinsPreprint
Koehl, A., Prillo, S., Liu, M., Xiong, J. T., Weng, L., Savage, D. F., & Song, Y. S. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.19.706898 · Cited by 5
Coevolutionary mining of prokaryotic non-coding elements with a genome language modelPreprint
Li, D., Brixi, G., Kim, A., Fiamenghi, M. B., Driscoll, C. L., Evans, S. A., Gao, L., Ivanova, N., Kyrpides, N. C., Deisseroth, K. A., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.09.22.753630 · Cited by 5
Validating the Distinctiveness of the Omicron Lineage within the SARS-CoV-2 based on Protein Language Models
Dong, K., & Gao, J. (2024) · Current Bioinformatics · doi:10.2174/0115748936291075240409080924 · Cited by 1
MFFDTA: A Multimodal Feature Fusion Framework for Drug-Target Affinity Prediction
Wang, W., Su, Z., Liu, D., Zhang, H., Shang, J., Zhou, Y., & Wang, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5692-6_22 · Cited by 1
Prediction and Analysis of Protein 3D Structures Using Protein Language Model and Streamlit
Srushti, C., Prathibhavani, P. M., Veena, A., & Venugopal, K. R. (2024) · International Conference on Information Management and Technology (ICIMTech) · doi:10.1109/icimtech63123.2024.10780906 · Cited by 1
Pan-protein Design Learning Enables Task-adaptive Generalization for Low-resource Enzyme Design
Zheng, J., Wang, G., Zhang, H., & Li, S. Z. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889962 · Cited by 1
Bioinformatics-Driven Multi-Factorial Insight into α-Galactosidase Mutations
Mele, B. H., Rossetti, F., Andreotti, G., Cubellis, M. V., Guerriero, S., & Monticelli, M. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms26125802 · Cited by 1
A Survey of Pretrained Protein Language Models
Pokharel, S., Pratyush, P., Chaudhari, M., Heinzinger, M., Caragea, D., Saigo, H., & KC, D. B. (2025) · Methods in molecular biology · doi:10.1007/978-1-0716-4623-6_1 · Cited by 1
Controllable protein design via autoregressive direct coupling analysis conditioned on principal components
(2026) · doi:10.1371/journal.pcbi.1013996 · Cited by 1
Protein Language Model Embeddings Distinguish Catalytic from Structural Zinc-Binding Sites with Interpretable Attention Signatures
Sargsyan, K. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c03142 · Cited by 1
Protein Language Models in Virology: A Review of Advances and Applications
Luo, L., Li, Y., & Huang, T. (2026) · Methods in molecular biology · doi:10.1007/978-1-0716-5264-0_18 · Cited by 1
Machine Learning-Guided Enzyme Engineering Approaches for Enhanced Biocatalytic Efficiency: Concepts, Mechanisms, and Future Directions
Ahsan, W. (2026) · Catalysts · doi:10.3390/catal16070598 · Cited by 1
OTalign: optimal transport alignment for remote protein homologs using protein language model embeddings
Kim, M., Bae, H., Jo, G., Kunwoo, K., Yoo, J., & Joo, K. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag476 · Cited by 1
Deep Learning and Artificial Intelligence in the Design of Angiotensin-Converting Enzyme (ACE) Inhibitory Peptides
Qian, Z., Zhou, Y., Lin, R., Hong, Y., Xu, Y., Li, S., Gao, J., Zhu, Q., & Zheng, H. (2026) · Journal of Artificial Intelligence Research and Innovation · doi:10.29328/journal.jairi.1001022 · Cited by 1
Application of Protein Language Models to low-N Engineering of metalloproteinase inhibitors
Buxton, E. K., Patel, R., Toumaian, M. R., & Raeeszadeh‐Sarmazdeh, M. (2021) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci54926.2021.00048 · Cited by 1
Exploring Post-Training Quantization of Protein Language Models
Peng, S., Yang, F., Sun, N., Chen, S., Jiang, Y., & Pan, A. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385775 · Cited by 2
Freeprotmap: waiting-free prediction method for protein distance map
Huang, J., Li, J., Chen, Q., Wang, X., Chen, G., & Tang, J. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05771-0 · Cited by 1
Extending protein language models to a viral genomic scale using biologically induced sparse attention
Dejean, T., Ferrell, B. D., Harrigan, W., Schreiber, Z. D., Sawhney, R., Wommack, K. E., Polson, S. W., & Belcaid, M. (2026) · GigaScience · doi:10.1093/gigascience/giag081 · Cited by 1
Boost Protein Language Model with Injected Structure Information Through Parameter Efficient Fine-tuning
Zhang, Z., Zhou, Y., Zheng, J. X., Feng, C.-M., Cui, S. R., Wang, S., & Li, Z. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110607 · Cited by 2
MoCETSE: A mixture-of-convolutional experts and transformer-based model for predicting Gram-negative bacterial secreted effectors
Shi, H., Lin, Y., Liu, D., & Zou, Q. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013397 · Cited by 2
A Review of Protein Structure Prediction using Deep Learning
Susanty, M., Rajab, T. L. E., & Hertadi, R. (2021) · BIO Web of Conferences · doi:10.1051/bioconf/20214104003 · Cited by 1
An explainable unsupervised framework for alignment-free protein classification using sequence embeddingsPreprint
Yeung, W., Zhou, Z., Mathew, L., Gravel, N., Taujale, R., Venkat, A., Lanzilotta, W. N., Li, S., & Kannan, N. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.02.08.478871 · Cited by 1
Disentanglement of Entropy and Coevolution using Spectral RegularizationPreprint
Wang, H., Feng, S., Liu, S., & Ovchinnikov, S. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.03.04.483009 · Cited by 1
Self-citation
Integrating Self-Attention Transformer with Triplet Neural Networks for Protein Gene Ontology PredictionPreprint
Zhu, Y., Zhang, C., Yu, D., & Zhang, Y. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.07.07.499156 · Cited by 1
Fast and Accurate Prediction of Intrinsically Disordered Protein by Protein Language ModelPreprint
Xu, S., & Onoda, A. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.10.15.512345 · Cited by 1
Convolutional ProteinUnetLM competitive with LSTM-based protein secondary structure predictorsPreprint
Stąpor, K., Kotowski, K., Fabian, P., & Roterman, I. (2022) · doi:10.22541/au.166636090.03718226/v1 · Cited by 1
Opinion piece/Book chapter: Artificial Intelligence learns protein predictionPreprint
Heinzinger, M., & Rost, B. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/n7wfp · Cited by 1
From Context to Code: Rational De Novo DNA Design and Predicting Cross-Species DNA Functionality Using Deep Learning Transformer ModelsPreprint
Dahiya, G. S., Bakken, T. I., Fages‐Lartaud, M., & Lale, R. R. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.15.562386 · Cited by 2
Protein sequence domain annotation using a language modelPreprint
Sarkar, A., Krishnan, K., & Eddy, S. R. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.04.596712 · Cited by 2
Semantical and Geometrical Protein Encoding Toward Enhanced Bioactivity and ThermostabilityPreprint
Tan, Y., Zhong, B., Zheng, L., Fan, G., & Hong, L. (2024) · eLife · doi:10.7554/elife.98033.2 · Cited by 1
Squidly harnesses enzyme functional hierarchy and contrastive learning to efficiently predict catalytic residues from sequence
Rieger, W. J., Mikael Bodén, A., Arnold, F. H., & Mora, A. (2025) · eLife · doi:10.1101/2025.06.13.659624 · Cited by 2
Squidly: Enzyme Catalytic Residue Prediction Harnessing a Biology-Informed Contrastive Learning FrameworkPreprint
Rieger, W. J., Mikael Bodén, A., Arnold, F. H., & Mora, A. (2026) · eLife · doi:10.7554/elife.108186.2 · Cited by 1
Life Trajectories as Symbolic Language
Savcisens, G. (2024) · doi:10.5281/zenodo.14706678 · Cited by 1
ProteomeLM: A proteome-scale language model enables accurate and rapid prediction of protein–protein interactions and gene essentiality across taxa
(2026) · doi:10.1073/pnas.2524201123 · Cited by 2
Protein-Protein Interaction Site Prediction Based on Attention Mechanism and Convolutional Neural NetworksPreprint
Lu, S., Li, Y., Ma, Q., Nan, X., & Zhang, S. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.07.10.451856
Protein tertiary structure prediction and refinement using deep learning
Wu, T. (2022) · MOspace Institutional Repository (University of Missouri) · doi:10.32469/10355/94100
Prediction of Sumoylation Sites in Proteins From Language Model Representations
Sidorov, E. (2023) · doi:10.37099/mtu.dc.etdr/1607
Aplicação de redes neurais recorrentes ao projeto de anticorpos a partir da sequência de aminoácidos de epítopos
Filho, A. M. (2021) · doi:10.47749/t/unicamp.2021.1260506
Rendering protein mutation movies with MutAmore
Weissenow, K., & Rost, B. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05610-8
Discovering nuclear localization signal universe through a novel deep language learning network with attention as the neuronsPreprint
Shen, H., Li, Y., & Pan, X. (2024) · Research Square · doi:10.21203/rs.3.rs-4204156/v1
Reviewer #1 (Public Review): Semantical and Geometrical Protein Encoding Toward Enhanced Bioactivity and Thermostability
Tan, Y., Zhou, B., Zheng, L., et al. (2024) · eLife · doi:10.7554/elife.98033.1.sa1
Reviewer #2 (Public Review): Semantical and Geometrical Protein Encoding Toward Enhanced Bioactivity and Thermostability
Tan, Y., Zhou, B., Zheng, L., et al. (2024) · eLife · doi:10.7554/elife.98033.1.sa0
eLife Assessment: Semantical and Geometrical Protein Encoding Toward Enhanced Bioactivity and Thermostability
Koo, P. K. (2024) · eLife · doi:10.7554/elife.98033.1.sa2
Boosting Protein Language Models with Negative Sample Mining
Xu, Y., Zhao, X., Song, X., Wang, B., & Yu, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70381-2_13
Recursive Cleaning for Large-scale Protein Data via Multimodal LearningPreprint
Jiang, Z., Zhang, S., Cao, J., Zhang, Q., Liu, S., Fang, Y., Zhang, L., Qing, R., & Xu, R. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.08.617190

Authors: Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A.
Affiliations: University of California, Berkeley (US)
Meta (United States) (US)
Harvard University (US)
Corresponding author: Rao, Roshan
Meier, Joshua
Sercu, Tom
Ovchinnikov, Sergey
Rives, Alexander
ORCID: Rao, Roshan
Meier, Joshua
Sercu, Tom
Ovchinnikov, Sergey
Rives, Alexander
Venue: bioRxiv (Cold Spring Harbor Laboratory)
Publisher: openRxiv
Published: 2020-12-15
DOI: 10.1101/2020.12.15.422761
OpenAlex: W3111174583
Cited by: 341
References: 63
Citations by year: 2026: 51 · 2025: 67 · 2024: 79 · 2023: 56 · 2022: 49 · 2021: 37 · 2020: 2
Type: posted content
Language: en
Open access: green
doi.org
Licence: cc-by-nc-nd
creativecommons.org/licenses/by-nc-nd/4.0/ (vor)
Locations: bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
Funders: National Institutes of Health (DP5OD026389)
Topics: Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
Keywords: protein language models, protein structure prediction, protein structure design, evolutionary constraints

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Transformer protein language models are unsupervised structure learners

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks