The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Transformer protein language models are unsupervised structure learners
Open accessAbstract
A bstract: Unsupervised contact prediction is central to uncovering physical, structural, and functional constraints for protein structure determination and design. For decades, the predominant approach has been to infer evolutionary constraints from a set of related sequences. In the past year, protein language models have emerged as a potential alternative, but performance has fallen short of state-of-the-art approaches in bioinformatics. In this paper we demonstrate that Transformer attention maps learn contacts from the unsupervised language modeling objective. We find the highest capacity models that have been trained to date already outperform a state-of-the-art unsupervised contact prediction pipeline, suggesting these pipelines can be replaced with a single forward pass of an end-to-end model. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Among many successful neural network architectures, attention-based networks are extremely powerful for modeling complicated interactions between entities, making them suitable for modeling proteins whose functions are largely determined by interactions between amino acids (34–37).”
From Deep learning guided optimization of human antibody against SARS-CoV-2 variants with broad neutralization (2022) · cites this paper as [37]
In the citing paper
“Since then, various language model-based protein embedding methods have been applied to these and other protein prediction problems through transfer learning, including protein phenotype prediction 28,32-34, residue-residue contact prediction 32,70, fold recognition 33, protein-protein 71,72 and protein-drug interaction prediction 35,73.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [70]
In the citing paper
“Recent works have shown that increasing language model scale leads to continued improvements in downstream applications, such as residue-residue contact prediction 70.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [70]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“A similar scaling law has been observed previously for supervised secondary structure prediction (Rives et al., 2019), and parallels observations in the NLP community (Kaplan et al., 2020; Brown et al., 2020).”
BackgroundmixOmics: An R package for ‘omics feature selection and multiple data integration
Rohart, F., Gautier, B., Singh, A., & Lê Cao, K.-A. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005752 · Cited by 4,056
Improved protein structure prediction using potentials from deep learning
Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1923-7 · Cited by 3,378
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Self-citationRives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
Self-citation
Research works printed but not linked: 4
- 10.1002/(SICI)1097-0134(19990301)34:4(508::AID-PROT10)3.0.CO;2-4 · DOI
- Robert D. Finn , Alex Bateman , Jody Clements , Penelope Coggill , Ruth Y. Eberhardt , Sean R. Eddy , Andreas Heger , Kirstie Hetherington , Liisa Holm , Jaina Mistry , Erik L.L. Sonnhammer , John Tate , and Marco Punta . Pfam: The protein families database, 1 2014. ISSN 03051048. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3965110/.
- 10.1101/622803 · DOI · Paper page
- 10.1101/2020.10.12.336859 · DOI
Cited by
Sorted by citations
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
Self-citationProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329
Simulating 500 million years of evolution with a language model
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., et al. (2025) · Science · doi:10.1126/science.ads0018 · Cited by 937
Self-citationDeepLoc 2.0: multi-label subcellular localization prediction using protein language models
Thumuluri, V., Armenteros, J. J. A., Johansen, A. R., Nielsen, H., & Winther, O. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac278 · Cited by 736
Language models enable zero-shot prediction of the effects of mutations on protein function
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.07.09.450648 · Cited by 635
Self-citationLearning the protein language: Evolution, structure, and function
Bepler, T., & Berger, B. (2021) · Cell Systems · doi:10.1016/j.cels.2021.05.017 · Cited by 483
Single-sequence protein structure prediction using language models from deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Rochereau, C., Church, G. M., Sorger, P. K., & AlQuraishi, M. (2021) · doi:10.1101/2021.08.02.454840 · Cited by 455
“We speculate that the latent space of the language model also captures recurrent evolutionary relationships 43 .”
BackgroundSingle-sequence protein structure prediction using a language model and deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01432-w · Cited by 455
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised LearningPreprint
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.07.12.199554 · Cited by 424
High-resolution de novo structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392
Rao, R., Liu, J., Verkuil, R., Meier, J., Canny, J. F., Abbeel, P., Sercu, T., & Rives, A. (2021) · doi:10.1101/2021.02.12.430858 · Cited by 350
Self-citationFast, accurate antibody structure prediction from deep learning on massive set of natural antibodies
Ruffolo, J. A., Chu, L., Mahajan, S. P., & Gray, J. J. (2023) · Nature Communications · doi:10.1038/s41467-023-38063-x · Cited by 364
Elnaggar, A., Heinzinger, M., Dallago, C., Rihawi, G., Wang, Y., Jones, L., Gibbs, T., Fehér, T., Angerer, C. M., Steinegger, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06225 · Cited by 364
Machine learning in protein structure prediction
AlQuraishi, M. (2021) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2021.04.005 · Cited by 263
SaProt: Protein Language Modeling with Structure-aware VocabularyPreprint
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., & Yuan, F. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.01.560349 · Cited by 264
PredictProtein - Predicting Protein Structure and Function for 29 Years
Bernhofer, M., Dallago, C., Karl, T., Satagopam, V., Heinzinger, M., Littmann, M., Olenyi, T., Qiu, J., Schütze, K., Yachdav, G., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab354 · Cited by 246
DeepLoc 2.1: multi-label membrane protein type prediction using protein language models
Ødum, M. T., Teufel, F., Thumuluri, V., Armenteros, J. J. A., Johansen, A. R., Winther, O., & Nielsen, H. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae237 · Cited by 230
Antibody structure prediction using interpretable deep learning
Ruffolo, J. A., Sulam, J., & Gray, J. J. (2022) · Patterns · doi:10.1016/j.patter.2021.100406 · Cited by 216
Modeling conformational states of proteins with AlphaFold
Sala, D., Engelberger, F., Mchaourab, H. S., & Meiler, J. (2023) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2023.102645 · Cited by 182
Ankh ☥: Optimized Protein Language Model Unlocks General-Purpose ModellingPreprint
Elnaggar, A., Essam, H., Salah-Eldin, W., Moustafa, W., Elkerdawy, M., Rochereau, C., & Rost, B. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.16.524265 · Cited by 243
Language models generalize beyond natural proteins
Verkuil, R., Kabeli, O., Du, Y., Wicky, B. I. M., Milles, L. F., Dauparas, J., Baker, D., Ovchinnikov, S., Sercu, T., & Rives, A. (2022) · doi:10.1101/2022.12.21.521521 · Cited by 181
Self-citationLearning meaningful representations of protein sequences
Detlefsen, N. S., Hauberg, S., & Boomsma, W. (2022) · Nature Communications · doi:10.1038/s41467-022-29443-w · Cited by 171
Shan, S., Luo, S., Yang, Z., Hong, J., Su, Y., Ding, F., Fu, L., Li, C., Chen, P., Ma, J., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2122954119 · Cited by 157
Single-sequence protein structure prediction using supervised transformer protein language models
Wang, W., Peng, Z., & Yang, J. (2022) · Nature Computational Science · doi:10.1038/s43588-022-00373-3 · Cited by 150
A 5′ UTR language model for decoding untranslated regions of mRNA and function predictions
Chu, Y., Yu, D., Li, Y., Huang, K., Shen, Y., Cong, L., Zhang, J. Y., & Wang, M. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00823-9 · Cited by 148
Protein language model embeddings for fast, accurate, alignment-free protein structure prediction
Weissenow, K., Heinzinger, M., & Rost, B. (2021) · doi:10.1101/2021.07.31.454572 · Cited by 139
Du, Z., Ding, X., Xu, Y., & Li, Y. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad135 · Cited by 137
Light attention predicts protein location from the language of life
Stärk, H., Dallago, C., Heinzinger, M., & Rost, B. (2021) · Bioinformatics Advances · doi:10.1093/bioadv/vbab035 · Cited by 130
FLIP: Benchmark tasks in fitness landscape inference for proteinsPreprint
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.11.09.467890 · Cited by 142
LAnoBERT: System log anomaly detection based on BERT masked language model
Lee, Y., Kim, J., & Kang, P. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110689 · Cited by 159
Machine learning modeling of family wide enzyme-substrate specificity screens
Goldman, S., Das, R., Yang, K. K., & Coley, C. W. (2022) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1009853 · Cited by 120
Fang, X., Wang, F., Liu, L., He, J., Lin, D., Xiang, Y., Zhu, K., Zhang, X., Wu, H., Li, H., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00721-6 · Cited by 132
Deep generative modeling for protein design
Strokach, A. V., & Kim, P. M. (2021) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2021.11.008 · Cited by 122
NetSolP: predicting protein solubility in Escherichia coli using language models
Thumuluri, V., Martiny, H.-M., Armenteros, J. J. A., Salomon, J., Nielsen, H., & Johansen, A. R. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab801 · Cited by 118
Markus, B., Gruber Christian, C., Andreas, K., Kummer, A., Stefan, L., Oberdorfer, G., Elina, S., & Šnajdrová, R. (2023) · ACS Catalysis · doi:10.1021/acscatal.3c03417 · Cited by 121
A new age in protein design empowered by deep learning
Khakzad, H., Igashov, I., Schneuing, A., Goverde, C. A., Bronstein, M. M., & Correia, B. E. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.006 · Cited by 121
Contrastive learning on protein embeddings enlightens midnight zone
(2022) · doi:10.1093/nargab/lqac043 · Cited by 105
Convolutions are competitive with transformers for protein sequence pretraining
Yang, K. K., Fusi, N., & Lu, A. X. (2024) · Cell Systems · doi:10.1016/j.cels.2024.01.008 · Cited by 137
Multiple sequence alignment-based RNA language model and its application to structural inference
Zhang, Y., Lang, M., Jiang, J., Gao, Z., Xu, F., Litfin, T., Chen, K., Singh, J., Huang, X., Song, G., et al. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkad1031 · Cited by 112
Learned Embeddings from Deep Learning to Visualize and Predict Protein Sets
Dallago, C., Schütze, K., Heinzinger, M., Olenyi, T., Littmann, M., Lu, A. X., Yang, K., Min, S., Yoon, S., Morton, J. T., et al. (2021) · Current Protocols · doi:10.1002/cpz1.113 · Cited by 109
Predicting functional effect of missense variants using graph attention neural networks
Zhang, H., Xu, M. S., Fan, X., Chung, W. K., & Shen, Y. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00561-w · Cited by 98
pLM4ACE: A protein language model based predictor for antihypertensive peptide screening
Du, Z., Ding, X., Hsu, W. W. Y., Munir, A., Xu, Y., & Li, Y. (2023) · Food Chemistry · doi:10.1016/j.foodchem.2023.137162 · Cited by 98
Structure-based protein design with deep learning
Ovchinnikov, S., & Huang, P.-S. (2021) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2021.08.004 · Cited by 93
Self-citationA survey of drug-target interaction and affinity prediction methods via graph neural networks
Zhang, Y., Hu, Y., Han, N., Yang, A., Liu, X., & Cai, H. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107136 · Cited by 102
Wang, Z., Combs, S. A., Brand, R. M., Calvo, M. R., Xu, P., Price, G., Golovach, N., Salawu, E. O., Wise, C., Ponnapalli, S. P., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10775-y · Cited by 89
Representation learning applications in biological sequence analysis
Iuchi, H., Matsutani, T., Yamada, K., Iwano, N., Sumi, S., Hosoda, S., Zhao, S., Fukunaga, T., & Hamada, M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.05.039 · Cited by 88
Sequence-based drug-target affinity prediction using weighted graph neural networks
Jiang, M., Wang, S., Zhang, S., Zhou, W. G., Zhang, Y., & Li, Z. (2022) · BMC Genomics · doi:10.1186/s12864-022-08648-9 · Cited by 86
InterPLM: discovering interpretable features in protein language models via sparse autoencoders
Simon, E. P., & Zou, J. (2025) · Nature Methods · doi:10.1038/s41592-025-02836-7 · Cited by 89
AI in drug development: a multidisciplinary perspective
Gallego, V., Naveiro, R., Roca, C., Insua, D. R., & Campillo, N. E. (2021) · Molecular Diversity · doi:10.1007/s11030-021-10266-8 · Cited by 85
Protein language models trained on multiple sequence alignments learn phylogenetic relationships
Lupo, U., Sgarbossa, D., & Bitbol, A.-F. (2022) · Nature Communications · doi:10.1038/s41467-022-34032-y · Cited by 67
Porebski, B. T., Balmforth, M. R., Browne, G. J., Riley, A., Jamali, K., Fürst, M. J. L. J., Velic, M., Buchanan, A. G., Minter, R., Vaughan, T. J., et al. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01093-3 · Cited by 74
Cheng, P., Mao, C., Tang, J., Yang, S., Cheng, Y., Wang, W., Gu, Q., Han, W., Chen, H., Li, S., et al. (2024) · Cell Research · doi:10.1038/s41422-024-00989-2 · Cited by 76
Prediction of RNA–protein interactions using a nucleotide language model
Yamada, K., & Hamada, M. (2022) · Bioinformatics Advances · doi:10.1093/bioadv/vbac023 · Cited by 70
Transformer-based protein generation with regularized latent space optimization
Castro, E., Godavarthi, A., Rubinfien, J., Givechian, K. B., Bhaskar, D., & Krishnaswamy, S. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00532-1 · Cited by 76
Singh, J., Paliwal, K. K., Litfin, T., Singh, J., & Zhou, Y. (2022) · Scientific Reports · doi:10.1038/s41598-022-11684-w · Cited by 59
Shi, Z., Deng, R., Yuan, Q., Mao, Z., Wang, R., Li, H., Liao, X., & Ma, H. (2023) · Research · doi:10.34133/research.0153 · Cited by 53
scPRINT: pre-training on 50 million cells allows robust gene network predictions
Kalfon, J., Samaran, J., Gabriel Peyré, A., & Cantini, L. (2025) · Nature Communications · doi:10.1038/s41467-025-58699-1 · Cited by 76
Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models
Li, F.-Z., Amini, A. P., Yue, Y., Yang, K. K., & Lu, A. X. (2024) · doi:10.1101/2024.02.05.578959 · Cited by 51
Are protein language models the new universal key?
Weißenow, K., & Rost, B. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.102997 · Cited by 54
End-to-end learning of multiple sequence alignments with differentiable Smith–Waterman
Petti, S., Bhattacharya, N., Rao, R., Dauparas, J., Thomas, N., Zhou, J., Rush, A. M., Koo, P., & Ovchinnikov, S. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btac724 · Cited by 54
Self-citationDeep self-supervised learning for biosynthetic gene cluster detection and product classification
Rios-Martinez, C., Bhattacharya, N., Amini, A. P., Crawford, L. A., & Yang, K. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011162 · Cited by 50
Multi-modal deep learning enables efficient and accurate annotation of enzymatic active sites
Wang, X., Yin, X., Jiang, D., Zhao, H., Wu, Z., Zhang, O., Wang, J., Li, Y., Deng, Y., Liu, H., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-51511-6 · Cited by 47
Drug-target interaction/affinity prediction: Deep learning models and advances review
Vefghi, A., Rahmati, Z., & Akbari, M. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110438 · Cited by 47
Protein sequence‐to‐structure learning: Is this the end(‐to‐end revolution)?
Laine, E., Eismann, S., Elofsson, A., & Grudinin, S. (2021) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26235 · Cited by 44
“works have also shown that the residue-residue dependencies extracted by certain attention heads in transformers trained on large amounts of sequences can be directly interpreted as 3D contacts or distances.(48,133,134)”
BackgroundFast, sensitive detection of protein homologs using deep dense retrieval
Hong, L., Hu, Z., Sun, S., Tang, X., Wang, J., Tan, Q., Zheng, L., Wang, S., Xu, S. Y., King, I., et al. (2024) · Nature Biotechnology · doi:10.1038/s41587-024-02353-6 · Cited by 46
Pakhrin, S. C., Pokharel, S., Aoki-Kinoshita, K. F., Beck, M. R., Dam, T. K., Caragea, D., & KC, D. B. (2023) · Glycobiology · doi:10.1093/glycob/cwad033 · Cited by 39
Protein representations: Encoding biological information for machine learning in biocatalysis
Harding-Larsen, D., Funk, J., Madsen, N. G., Gharabli, H., Acevedo-Rocha, C. G., Mazurenko, S., & Welner, D. H. (2024) · Biotechnology Advances · doi:10.1016/j.biotechadv.2024.108459 · Cited by 38
Protein structure prediction in the deep learning era
Peng, Z., Wang, W., Han, R., Zhang, F., & Yang, J. (2022) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2022.102495 · Cited by 40
Rapid directed evolution guided by protein language models and epistatic interactions
Tran, V. Q., Nemeth, M., Bartie, L. J., Chandrasekaran, S. S., Fanton, A., Moon, H. C., Hie, B., Konermann, S., & Hsu, P. D. (2026) · Science · doi:10.1126/science.aea1820 · Cited by 41
Generative AI: driving productivity and scientific breakthroughs in pharmaceutical R&D
Doron, G., Genway, S., Roberts, M., & Jasti, S. (2024) · Drug Discovery Today · doi:10.1016/j.drudis.2024.104272 · Cited by 39
Phosformer: an explainable transformer model for protein kinase-specific phosphorylation predictions
Zhou, Z., Yeung, W., Gravel, N., Salcedo, M. V., Soleymani, S., Li, S., & Kannan, N. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad046 · Cited by 34
AutoPeptideML: a study on how to build more trustworthy peptide bioactivity predictors
Fernández-Díaz, R., Cossio-Pérez, R., Agoni, C., Lam, H. T., López, V., & Shields, D. C. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae555 · Cited by 35
RAPPPID: towards generalizable protein interaction prediction with AWD-LSTM twin networks
Szymborski, J., & Emad, A. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac429 · Cited by 34
Identification of a covert evolutionary pathway between two protein folds
Chakravarty, D., Sreenivasan, S., Swint-Kruse, L., & Porter, L. L. (2023) · Nature Communications · doi:10.1038/s41467-023-38519-0 · Cited by 29
Zhang, Y., Zhang, Y., Xiong, Y., Wang, H., Deng, Z., Song, J., & Ou, H. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab420 · Cited by 31
Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodiesPreprint
Ruffolo, J. A., Chu, L., Mahajan, S. P., & Gray, J. J. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.20.488972 · Cited by 33
BEND: Benchmarking DNA Language Models on biologically meaningful tasksPreprint
Marin, F. I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., & Boomsma, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.12570 · Cited by 54
Pairing interacting protein sequences using masked language modeling
Lupo, U., Sgarbossa, D., & Bitbol, A.-F. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311887121 · Cited by 31
Evaluation of residue-residue contact prediction methods: From retrospective to prospective
Zhang, H., Bei, Z., Xi, W., Hao, M., Ju, Z., Saravanan, K. M., Zhang, H., Guo, N., & Wei, Y. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009027 · Cited by 33
DG-Affinity: predicting antigen–antibody affinity with language models from sequences
Yuan, Y., Chen, Q., Mao, J., Li, G., & Pan, X. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05562-z · Cited by 30
Prediction of drug-target binding affinity based on deep learning models
Zhang, H., Liu, X., Cheng, W., Wang, T., & Chen, Y. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108435 · Cited by 32
Drug Discovery in the Age of Artificial Intelligence: Transformative Target-Based Approaches
Patne, A. Y., Dhulipala, S. M., Lawless, W., Prakash, S., Mohapatra, S. S., & Mohapatra, S. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms252212233 · Cited by 32
Ultra-fast protein structure prediction to capture effects of sequence variation in mutation moviesPreprint
Weißenow, K., Heinzinger, M., Steinegger, M., & Rost, B. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.11.14.516473 · Cited by 29
Protein Language Models: Is Scaling Necessary?Preprint
Fournier, Q., Vernon, R. M., van der Sloot, A. M., Schulz, B., Chandar, S., & LANGMEAD, C. J. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.23.614603 · Cited by 42
Yan, Z., Ge, F., Liu, Y., Zhang, Y., Li, F., Song, J., & Yu, D. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c02019 · Cited by 25
tFold-Ab: Fast and Accurate Antibody Structure Prediction without Sequence HomologsPreprint
Wu, J., Wu, F., Jiang, B., Liu, W., & Zhao, P. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.11.10.515918 · Cited by 26
DeepSS2GO: protein function prediction from secondary structure
Song, F. V., Su, J., Huang, S., Zhang, N., Li, K., Ni, M., & Liao, M. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae196 · Cited by 26
Xu, S., & Onoda, A. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c01202 · Cited by 26
Heinzinger, M., & Rost, B. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.102986 · Cited by 27
A Structure-Based B-cell Epitope Prediction Model Through Combing Local and Global Features
Lu, S., Li, Y., Ma, Q., Nan, X., & Zhang, S. (2022) · Frontiers in Immunology · doi:10.3389/fimmu.2022.890943 · Cited by 23
Machine learning modeling of RNA structures: methods, challenges and future perspectives
Wu, K., Zou, J., & Chang, H. Y. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad210 · Cited by 24
Improved prediction of MHC-peptide binding using protein language models
Hashemi, N., Hao, B., Ignatov, M., Paschalidis, I. C., Vakili, P., Vajda, S., & Kozakov, D. (2023) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2023.1207380 · Cited by 25
Yue, P., Wu, J., Sun, Y., Zhang, Y., Wang, Q., & Shao, S. (2025) · Nature Communications · doi:10.1038/s41467-025-56526-1 · Cited by 28
Metadata
- Authors
- Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A.
- Affiliations
- University of California, Berkeley (US)
- Meta (United States) (US)
- Harvard University (US)
- Corresponding author
- Rao, Roshan
- Meier, Joshua
- Sercu, Tom
- Ovchinnikov, Sergey
- Rives, Alexander
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2020-12-15
- OpenAlex
- W3111174583
- Cited by
- 341
- References
- 63
- Citations by year
- 2026: 51 · 2025: 67 · 2024: 79 · 2023: 56 · 2022: 49 · 2021: 37 · 2020: 2
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- Funders
- National Institutes of Health (DP5OD026389)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
- Keywords
- protein language models, protein structure prediction, protein structure design, evolutionary constraints
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
