Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
Open accessPublished version of a preprint
The DOI matches its Crossref record; 123 of the 123 references it cites were found in a registry or an index.
Abstract
Significance: Learning biological properties from sequence data is a logical step toward generative and predictive artificial intelligence for biology. Here, we propose scaling a deep contextual language model with unsupervised learning to sequences spanning evolutionary diversity. We find that without prior knowledge, information emerges in the learned representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity. We show the learned representations are useful across benchmarks for remote homology detection, prediction of secondary structure, long-range residue–residue contacts, and mutational effect. Unsupervised representation learning enables state-of-the-art supervised prediction of mutational effect and secondary structure and improves state-of-the-art features for long-range contact prediction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The current state-of-the-art in sequence-based modeling is the use of AI/ML embeddings pretrained on multimillion-scale protein collections.118”
From Advancing Targeted Protein Degradation via Multiomics Profiling and Artificial Intelligence (2023) · cites this paper as [118]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since then, various language model-based protein embedding methods have been applied to these and other protein prediction problems through transfer learning, including protein phenotype prediction 28,32-34, residue-residue contact prediction 32,70, fold recognition 33, protein-protein 71,72 and protein-drug interaction prediction 35,73.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has been shown that the resulting representations contain rich information about the structure and the function of the proteins 17 , 32 , 33 .”
From Turnover number predictions for kinetically uncharacterized enzymes using machine and deep learning (2023) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the evolutionary scale modeling (ESM) family (Rives et al. 2021), comprising transformer protein language models such as ESM-2 and ESMFold, surpasses earlier single-sequence protein language models, demonstrating enhanced capability in generating accurate structure predictions based on protein sequences.”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Rives et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“New work has since shown that the representations learned by language models are also powerful features for learning of variant fitness as a subsequent supervised learning task 32,34, building on earlier observations that language models can improve protein property prediction through transfer learning 27.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [32]
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Basic local alignment search tool
Altschul, S. F., Gish, W., Miller, W., Myers, E. W., & Lipman, D. J. (1990) · Journal of Molecular Biology · doi:10.1006/jmbi.1990.9999 · Cited by 98,669
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
(2008) · Journal of Machine Learning Research · Cited by 33,051
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Kabsch, W., & Sander, C. (1983) · Biopolymers · doi:10.1002/bip.360221211 · Cited by 15,320
Research works printed but not linked: 4
- A. Radford . Language models are unsupervised multitask learners. OpenAI Blog [Preprint] (2019). https://openai.com/blog/better-language-models (Accessed 6 August 2020).
- 10.1016/S0022-2836(05)80360-2 · doi:10.1016/s0022-2836(05)80360-2 · Paper page
- A. Radford K. Narasimhan T. Salimans I. Sutskever Improving language understanding by generative pre-training. OpenAI Blog [Preprint] (2018). https://openai.com/blog/language-unsupervised (Accessed 6 August 2020).
- T. Mikolov . Subword language modeling with neural networks. The website of T. Mikolov [Preprint] (2012). http://www.fit.vutbr.cz/∼imikolov/rnnlm/char.pdf (Accessed 14 March 2021).
Cited by
Items 201–300 of 1,102
A Comprehensive Survey of Foundation Models in Medicine
Khan, W., Leem, S., See, K. B., Wong, J. K., Zhang, S., & Fang, R. (2025) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2025.3531360 · Cited by 214
Evolutionary origin and functional diversification of aminotransferases
Koper, K., Han, S., Casas‐Pastor, D., Yoshikuni, Y., & Maeda, H. A. (2022) · Journal of Biological Chemistry · doi:10.1016/j.jbc.2022.102122 · Cited by 101
Deep learning for the PSIPRED Protein Analysis Workbench
Buchan, D. W. A., Moffat, L., Lau, A. M., Kandathil, S. M., & Jones, D. T. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae328 · Cited by 125
Masked inverse folding with sequence transfer for protein representation learning
(2022) · doi:10.1093/protein/gzad015 · Cited by 110
Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks
Nambiar, A., Heflin, M., Liu, S., Maslov, S., Hopkins, M., & Ritz, A. (2020) · Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics · doi:10.1145/3388440.3412467 · Cited by 92
John van der, O. O. S. T., & Patinios, C. (2023) · Trends in biotechnology · doi:10.1016/j.tibtech.2022.12.022 · Cited by 117
Large language models improve annotation of prokaryotic viral proteins
Flamholz, Z. N., Biller, S. J., & Kelly, L. (2024) · Nature Microbiology · doi:10.1038/s41564-023-01584-8 · Cited by 92
Large language models and their applications in bioinformatics
Sarumi, O. A., & Heider, D. (2024) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.09.031 · Cited by 119
NetGO 3.0: Protein Language Model Improves Large-Scale Functional Annotations
Wang, S., You, R., Liu, Y., Xiong, Y., & Zhu, S. (2023) · Genomics, Proteomics & Bioinformatics · doi:10.1016/j.gpb.2023.04.001 · Cited by 90
Signal Peptides Generated by Attention-Based Neural Networks
Wu, Z., Yang, K. K., Liszka, M. J., Lee, A., Batzilla, A., Wernick, D., Weiner, D. P., & Arnold, F. H. (2020) · ACS Synthetic Biology · doi:10.1021/acssynbio.0c00219 · Cited by 88
Koh, H. Y., Zheng, Y., Yang, M., Arora, R., Webb, G. I., Pan, S., Li, L., & Church, G. M. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00349-8 · Cited by 147
Democratizing protein language models with parameter-efficient fine-tuning
Sledzieski, S., Kshirsagar, M., Baek, M., Dodhia, R., Lavista Ferres, J., & Berger, B. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2405840121 · Cited by 86
Qiu, Y., & Wei, G.-W. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad289 · Cited by 88
Machine learning: its challenges and opportunities in plant system biology
Hesami, M., Alizadeh, M., Jones, A. M. P., & Torkamaneh, D. (2022) · Applied Microbiology and Biotechnology · doi:10.1007/s00253-022-11963-6 · Cited by 102
Protein-Based Hydrogels and Their Biomedical Applications
Lee, K. Z., Jeon, J., Jiang, B., Subramani, S. V., Li, J., & Zhang, F. (2023) · Molecules · doi:10.3390/molecules28134988 · Cited by 98
Hierarchical graph transformer with contrastive learning for protein function prediction
Gu, Z., Luo, X., Chen, J., Deng, M., & Lai, L. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad410 · Cited by 110
Wang, Z., Combs, S. A., Brand, R., Calvo, M. R., Xu, P., Price, G., Golovach, N., Salawu, E. O., Wise, C. J., Ponnapalli, S. P., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10775-y · Cited by 85
Computational scoring and experimental evaluation of enzymes generated by neural networks
Johnson, S. R., Fu, X., Viknander, S., Goldin, C., Monaco, S., Zelezniak, A., & Yang, K. K. (2025) · Nature Biotechnology · doi:10.1038/s41587-024-02214-2 · Cited by 99
Improving protein succinylation sites prediction using embeddings from protein language model
Pokharel, S., Pratyush, P., Heinzinger, M., Newman, R. H., & KC, D. B. (2022) · Scientific Reports · doi:10.1038/s41598-022-21366-2 · Cited by 103
Roche, R., Moussad, B., Shuvo, M. H., Tarafder, S., & Bhattacharya, D. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae039 · Cited by 85
Enhancing efficiency of protein language models with minimal wet-lab data through few-shot learning
Zhou, Z., Zhang, L., Yu, Y., Wu, B., Li, M., Hong, L., & Tan, P. (2024) · Nature Communications · doi:10.1038/s41467-024-49798-6 · Cited by 81
Song, Y., Yuan, Q., Chen, S., Zeng, Y., Zhao, H., & Yang, Y. (2024) · Nature Communications · doi:10.1038/s41467-024-52533-w · Cited by 109
Representation learning applications in biological sequence analysis
Iuchi, H., Matsutani, T., Yamada, K., Iwano, N., Sumi, S., Hosoda, S., Zhao, S., Fukunaga, T., & Hamada, M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.05.039 · Cited by 83
SEMA: Antigen B-cell conformational epitope prediction using deep transfer learning
Shashkova, T. I., Umerenkov, D., Salnikov, M., Strashnov, P. V., Konstantinova, A. V., Lebed, I., Shcherbinin, D. N., Asatryan, M. N., Kardymon, O. L., & Ivanisenko, N. V. (2022) · Frontiers in Immunology · doi:10.3389/fimmu.2022.960985 · Cited by 81
Probing the limit of hydrologic predictability with the Transformer network
Liu, J., Bian, Y., Lawson, K., & Shen, C. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131389 · Cited by 129
Self-citationIlluminating protein space with a programmable generative model
Ingraham, J., Baranov, M., Costello, Z., Frappier, V., Ismail, A., Tie, S., Wang, W., Xue, V., Obermeyer, F., Beam, A., et al. (2022) · doi:10.1101/2022.12.01.518682 · Cited by 83
Wang, G., Liu, X., Wang, K., Gao, Y., Li, G., Baptista‐Hon, D. T., Yang, X. H., Xue, K., Tai, W. H., Jiang, Z., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02483-5 · Cited by 94
Unlocking de novo antibody design with generative artificial intelligence
Shanehsazzadeh, A., McPartlon, M., Kasun, G., Steiger, A. K., Sutton, J. M., Yassine, E., McCloskey, C., Haile, R., Shuai, R., Alverio, J., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.08.523187 · Cited by 82
Self-citationRui, H., Ashton, K. S., Min, J., Wang, C. W., & Potts, P. R. (2023) · RSC Chemical Biology · doi:10.1039/d2cb00207h · Cited by 94
Boadu, F., Cao, H., & Cheng, J. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad208 · Cited by 81
Prediction of Klebsiella phage-host specificity at the strain level
Boeckaerts, D., Stock, M., Ferriol-González, C., Oteo-Iglesias, J., Sanjuán, R., Domingo-Calap, P., De Baets, B., & Briers, Y. (2024) · Nature Communications · doi:10.1038/s41467-024-48675-6 · Cited by 80
Supervised learning model predicts protein adsorption to carbon nanotubes
Pinals, R. L., Ouassil, N., Bonis-O’Donnell, J. T. D., Wang, J., & Landry, M. P. (2022) · Science Advances · doi:10.1126/sciadv.abm0898 · Cited by 89
Wilman, W., Wróbel, S., Bielska, W., Deszynski, P., Dudzic, P., Jaszczyszyn, I., Kaniewski, J., Młokosiewicz, J., Rouyan, A., Satława, T., et al. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac267 · Cited by 79
Codon language embeddings provide strong signals for use in protein engineering
Outeiral, C., & Deane, C. M. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00791-0 · Cited by 78
CAR-Toner: an AI-driven approach for CAR tonic signaling prediction and optimization
Qiu, S., Chen, J., Wu, T., Li, L., Wang, G., Wu, H., Song, X., Liu, X., & Wang, H. (2024) · Cell Research · doi:10.1038/s41422-024-00936-1 · Cited by 79
Deep graph learning of inter-protein contacts
Xie, Z., & Xu, J. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btab761 · Cited by 67
Deep language models for interpretative and predictive materials science
Hu, Y., & Buehler, M. J. (2023) · APL Machine Learning · doi:10.1063/5.0134317 · Cited by 78
Prediction of hemolytic peptides and their hemolytic concentration
Rathore, A. S., Kumar, N., Choudhury, S., Mehta, N. K., & Raghava, G. P. S. (2025) · Communications Biology · doi:10.1038/s42003-025-07615-w · Cited by 104
Contextual AI models for single-cell protein biology
Li, M. M., Huang, Y., Sumathipala, M. G., Liang, M. Q., Valdeolivas, A., Ananthakrishnan, A. N., Liao, K. P., Marbach, D., & Žitnik, M. (2024) · Nature Methods · doi:10.1038/s41592-024-02341-3 · Cited by 99
A review of transformer models in drug discovery and beyond
Jiang, J., Chen, L., Ke, L., Dou, B., Zhang, C., Feng, H., Zhu, Y., Qiu, H., Zhang, B., & Wei, G. (2024) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2024.101081 · Cited by 104
Machine learning for metabolic pathway optimization: A review
Cheng, Y., Bi, X., Xu, Y., Liu, Y., Li, J., Du, G., Lv, X., & Liu, L. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.03.045 · Cited by 88
Wang, Y., Zhao, L., Li, Z., Xi, Y., Pan, Y., Zhao, G., & Zhang, L. (2025) · Nature Microbiology · doi:10.1038/s41564-025-02114-4 · Cited by 111
Protein language models trained on multiple sequence alignments learn phylogenetic relationships
Lupo, U., Sgarbossa, D., & Bitbol, A.-F. (2022) · Nature Communications · doi:10.1038/s41467-022-34032-y · Cited by 67
Chen, K., Zhou, Y., Ding, M., Wang, Y., Ren, Z., & Yang, Y. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae163 · Cited by 99
Language models for biological research: a primer
Simon, E. P., Swanson, K., & Zou, J. (2024) · Nature Methods · doi:10.1038/s41592-024-02354-y · Cited by 94
Paheding, S., Saleem, A. I., Siddiqui, M. F. H., Rawashdeh, N. A., Essa, A. E., & Reyes-Angulo, A. A. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10165-7 · Cited by 105
Porebski, B. T., Balmforth, M., Browne, G., Riley, A., Jamali, K., Fürst, M. J. L. J., Velic, M., Buchanan, A., Minter, R., Vaughan, T., et al. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01093-3 · Cited by 72
Cheng, P., Mao, C., Tang, J., Yang, S., Cheng, Y., Wang, W., Gu, Q., Han, W., Chen, H., Li, S.-H., et al. (2024) · Cell Research · doi:10.1038/s41422-024-00989-2 · Cited by 78
Novel machine learning approaches revolutionize protein knowledge
Bordin, N., Dallago, C., Heinzinger, M., Kim, S., Littmann, M., Rauer, C., Steinegger, M., Rost, B., & Orengo, C. (2023) · Trends in Biochemical Sciences · doi:10.1016/j.tibs.2022.11.001 · Cited by 72
The genetic architecture of protein stability
Faure, A. J., Martí-Aranda, A., Hidalgo-Carcedo, C., Beltran, A., Schmiedel, J. M., & Lehner, B. (2024) · Nature · doi:10.1038/s41586-024-07966-0 · Cited by 72
Accuracy and data efficiency in deep learning models of protein expression
Nikolados, E.-M., Wongprommoon, A., Aodha, O. M., Cambray, G., & Oyarzún, D. A. (2022) · Nature Communications · doi:10.1038/s41467-022-34902-5 · Cited by 70
Structure‐Based Drug Discovery with Deep Learning**
Özçelik, R., van Tilborg, D., Jiménez-Luna, J., & Grisoni, F. (2023) · ChemBioChem · doi:10.1002/cbic.202200776 · Cited by 86
Application of Transformers in Cheminformatics
Luong, K.-D., & Singh, A. K. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c02070 · Cited by 85
GPU-accelerated homology search with MMseqs2
(2025) · doi:10.1038/s41592-025-02819-8 · Cited by 71
Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization
Lu, A. X., Zhang, H., Ghassemi, M., & Moses, A. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.09.04.283929 · Cited by 70
Continuous evolution of user-defined genes at 1 million times the genomic mutation rate
Rix, G., Williams, R. L., Hu, V. J., Spinner, A., Pisera, A. (., Marks, D. S., & Liu, C. C. (2024) · Science · doi:10.1126/science.adm9073 · Cited by 71
Deep learning for inferring transcription factor binding sites
Koo, P. K., & Ploenzke, M. (2020) · Current Opinion in Systems Biology · doi:10.1016/j.coisb.2020.04.001 · Cited by 70
Kalakoti, Y., Yadav, S., & Sundar, D. (2022) · ACS Omega · doi:10.1021/acsomega.1c05203 · Cited by 70
Drug Design and Disease Diagnosis: The Potential of Deep Learning Models in Biology
Sreeraman, S., Mayuri, K., Kushwah, R. B. S., Sundaram, V., Veluchamy, A., Anand, T., & Saravanan, K. M. (2023) · Current Bioinformatics · doi:10.2174/1574893618666230227105703 · Cited by 75
Leveraging transformers‐based language models in proteome bioinformatics
Le, N. Q. K. (2023) · PROTEOMICS · doi:10.1002/pmic.202300011 · Cited by 81
Self-play reinforcement learning guides protein engineering
Wang, Y., Tang, H., Huang, L., Pan, L., Yang, L., Yang, H., Mu, F., & Yang, M. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00691-9 · Cited by 104
Deep learning tools to accelerate antibiotic discovery
Cesaro, A., Bagheri, M., Torres, M. D. T., Wan, F., & de la Fuente‐Núñez, C. (2023) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2023.2250721 · Cited by 86
Deep learning for protein structure prediction and design—progress and applications
Jänes, J., & Beltrão, P. (2024) · Molecular Systems Biology · doi:10.1038/s44320-024-00016-x · Cited by 84
A survey of geometric graph neural networks: data structures, models and applications
Han, J., Cen, J., Wu, L., Li, Z., Kong, X., Jiao, R., Yu, Z., Xu, T., Wu, F., Wang, Z., et al. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-025-41426-w · Cited by 126
Biophysics-based protein language models for protein engineering
Gelman, S., Johnson, B., Freschlin, C. R., Sharma, A., D’Costa, S., Peters, J., Gitter, A., & Romero, P. A. (2025) · Nature Methods · doi:10.1038/s41592-025-02776-2 · Cited by 69
Protein structure prediction by AlphaFold 2: are attention and symmetries all you need?
Bouatta, N., Sorger, P., & AlQuraishi, M. (2021) · Acta Crystallographica Section D Structural Biology · doi:10.1107/s2059798321007531 · Cited by 69
Connecting chemistry and biology through molecular descriptors
Fernández-Torras, A., Comajuncosa-Creus, A., Duran-Frigola, M., & Aloy, P. (2022) · Current Opinion in Chemical Biology · doi:10.1016/j.cbpa.2021.09.001 · Cited by 68
Prediction of RNA–protein interactions using a nucleotide language model
Yamada, K., & Hamada, M. (2022) · Bioinformatics Advances · doi:10.1093/bioadv/vbac023 · Cited by 68
Harnessing generative AI to decode enzyme catalysis and evolution for enhanced engineering
Xie, W. J., & Warshel, A. (2023) · National Science Review · doi:10.1093/nsr/nwad331 · Cited by 69
Addressing the antibody germline bias and its effect on language models for improved antibody design
Olsen, T. H., Moal, I. H., & Deane, C. M. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae618 · Cited by 67
Machine Learning‐Enhanced Nanoparticle Design for Precision Cancer Drug Delivery
Wang, Q., Liu, Y., Li, C., Xu, S., Xu, S., & Liu, B. (2025) · Advanced Science · doi:10.1002/advs.202503138 · Cited by 103
Target sequence-conditioned design of peptide binders using masked language modeling
Chen, T., Quinn, Z., Dumas, M., Peng, C. Y., Hong, L., Lopez-Gonzalez, M., Mestre, A. A., Watson, R., Vincoff, S., Zhao, L., et al. (2025) · Nature Biotechnology · doi:10.1038/s41587-025-02761-2 · Cited by 131
Yuan, Q., Chen, S., Wang, Y., Zhao, H., & Yang, Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac444 · Cited by 68
Zhu, Y.-H., Zhang, C., Yu, D.-J., & Zhang, Y. (2022) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1010793 · Cited by 67
Applications of deep learning in understanding gene regulation
Li, Z., Gao, E., Zhou, J., Han, W., Xu, X., & Gao, X. (2023) · Cell Reports Methods · doi:10.1016/j.crmeth.2022.100384 · Cited by 92
Theoretical and Data-Driven Approaches for Biomolecular Condensates
Saar, K. L., Qian, D., Good, L. L., Morgunov, A. S., Collepardo‐Guevara, R., Best, R. B., & Knowles, T. P. J. (2023) · Chemical Reviews · doi:10.1021/acs.chemrev.2c00586 · Cited by 76
Cordoves‐Delgado, G., & García‐Jacas, C. R. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c02061 · Cited by 80
AI Methods for Antimicrobial Peptides: Progress and Challenges
Brizuela, C. A., Liu, G., Stokes, J. M., & de la Fuente‐Núñez, C. (2025) · Microbial Biotechnology · doi:10.1111/1751-7915.70072 · Cited by 87
Transformer-based protein generation with regularized latent space optimization
Castro, E., Godavarthi, A., Rubinfien, J., Givechian, K. B., Bhaskar, D., & Krishnaswamy, S. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00532-1 · Cited by 93
Calibrated geometric deep learning improves kinase–drug binding predictions
Luo, Y., Liu, Y., & Peng, J. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00751-0 · Cited by 84
Machine learning prediction of enzyme optimum pH
Gado, J. E., Knotts, M., Shaw, A. Y., Marks, D., Gauthier, N., Sander, C., & Beckham, G. (2024) · bioRxiv · doi:10.1038/s42256-025-01026-6 · Cited by 50
Protein design via deep learning
Ding, W., Nakai, K., & Gong, H. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac102 · Cited by 66
Hoarfrost, A., Aptekmann, A., Farfañuk, G., & Bromberg, Y. (2022) · Nature Communications · doi:10.1038/s41467-022-30070-8 · Cited by 66
Predicting the antigenic evolution of SARS-COV-2 with deep learning
Han, W., Chen, N., Xu, X., Sahil, A., Zhou, J., Li, Z., Zhong, H., Gao, E., Zhang, R., Wang, Y., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-39199-6 · Cited by 81
Structure-informed Language Models Are Protein Designers
Zheng, Z., Deng, Y., Xue, D., Zhou, Y., Ye, F., & Gu, Q. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.02.03.526917 · Cited by 62
Nicheformer: A Foundation Model for Single-Cell and Spatial OmicsPreprint
Schaar, A. C., Tejada-Lapuerta, A., Palla, G., Gutgesell, R. M., Halle, L., Minaeva, M., Vornholz, L., Dony, L., Drummer, F., Bahrami, M., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.04.15.589472 · Cited by 66
Deep-AVPpred: Artificial Intelligence Driven Discovery of Peptide Drugs for Viral Infections
Sharma, R., Shrivastava, S., Singh, S. K., Kumar, A., Singh, A. K., & Saxena, S. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3130825 · Cited by 70
Pudžiuvelytė, I., Olechnovič, K., Godliauskaite, E., Sermokas, K., Urbaitis, T., Gasiunas, G., & Kazlauskas, D. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae157 · Cited by 65
Using Computational Models to Test Syntactic Learnability
Wilcox, E. G., Futrell, R., & Levy, R. (2024) · Linguistic Inquiry · doi:10.1162/ling_a_00491 · Cited by 62
Robert, P. A., Akbar, R., Frank, R., Pavlović, M., Widrich, M., Snapkov, I., Slabodkin, A., Chernigovskaya, M., Scheffer, L., Smorodina, E., et al. (2022) · Nature Computational Science · doi:10.1038/s43588-022-00372-4 · Cited by 77
Lessons learned from the IMMREP23 TCR-epitope prediction challenge
Nielsen, M., Eugster, A., Jensen, M. F., Goel, M., Tiffeau-Mayer, A., Pélissier, A., Valkiers, S., Martínez, M. R., Meynard-Piganeau, B., Greiff, V., et al. (2024) · ImmunoInformatics · doi:10.1016/j.immuno.2024.100045 · Cited by 64
Amimeur, T., Shaver, J. M., Ketchem, R. R., Taylor, J. A., Clark, R. H., Smith, J., Van Citters, D., Siska, C. C., Smidt, P., Sprague, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.04.12.024844 · Cited by 63
Cluster learning-assisted directed evolution
Qiu, Y., Hu, J., & Wei, G.-W. (2021) · Nature Computational Science · doi:10.1038/s43588-021-00168-y · Cited by 61
A general temperature-guided language model to design proteins of enhanced stability and activity
Jiang, F., Li, M.-C., Dong, J., Yu, Y.-X., Sun, X., Wu, B.-H., Huang, J., Kang, L.-Q., Pei, Y., Zhang, L., et al. (2024) · Science Advances · doi:10.1126/sciadv.adr2641 · Cited by 66
Self-citationTCR-BERT: learning the grammar of T-cell receptors for flexible antigen-xbinding analysesPreprint
Wu, K., Yost, K. E., Dániel, B., Belk, J. A., Xia, Y., Egawa, T., Satpathy, A. T., Chang, H. Y., & Zou, J. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.11.18.469186 · Cited by 85
Bachas, S., Rakocevic, G., Spencer, D., Sastry, A. V., Haile, R., Sutton, J. M., Kasun, G., Stachyra, A., Gutierrez, J. M., Yassine, E., et al. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.08.16.504181 · Cited by 62
Self-citationLarge scale paired antibody language models
Kenlay, H., Dreyer, F. A., Kovaltsuk, A., Miketa, D., Pires, D., & Deane, C. M. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1012646 · Cited by 60
CodonBERT large language model for mRNA vaccines
Li, S., Moayedpour, S., Li, R., Bailey, M., Riahi, S., Kogler-Anele, L., Miladi, M., Miner, J., Pertuy, F., Zheng, D., et al. (2024) · Genome Research · doi:10.1101/gr.278870.123 · Cited by 60
Kalemati, M., Emani, M. Z., & Koohi, S. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011036 · Cited by 66
Sun, Y., Li, Y. Y., Leung, C. K.-S., & Hu, P. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae135 · Cited by 68
Metadata
- Authors
- Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., & Fergus, R.
- Affiliations
- Meta (United States) (US)
- New York University (US)
- Harvard University (US)
- Yale University (US)
- Corresponding author
- Rives, Alexander
- ORCID
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2021-04-05
- Volume
- 118
- Issue
- 15
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC8053943
- OpenAlex
- W3146944767
- Cited by
- 3,286
- References
- 100
- Citations by year
- 2026: 787 · 2025: 813 · 2024: 697 · 2023: 448 · 2022: 305 · 2021: 162
- FWCI
- 126.15
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Knowledge@UChicago (University of Chicago) (submittedVersion, cc-by-nc-nd)
- University of Chicago
- Funders
- National Science Foundation (1339362)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
- Keywords
- protein language models, unsupervised learning, protein evolution, protein structure prediction, secondary structure prediction, protein tertiary structure, remote homology
- MeSH terms
- Unsupervised Machine Learning, Amino Acids, Protein Conformation, Sequence Homology, Amino Acid, Sequence Analysis, Protein
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.