Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
Open accessPublished version of a preprint
The DOI matches its Crossref record; 123 of the 123 references it cites were found in a registry or an index.
Abstract
Significance: Learning biological properties from sequence data is a logical step toward generative and predictive artificial intelligence for biology. Here, we propose scaling a deep contextual language model with unsupervised learning to sequences spanning evolutionary diversity. We find that without prior knowledge, information emerges in the learned representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity. We show the learned representations are useful across benchmarks for remote homology detection, prediction of secondary structure, long-range residue–residue contacts, and mutational effect. Unsupervised representation learning enables state-of-the-art supervised prediction of mutational effect and secondary structure and improves state-of-the-art features for long-range contact prediction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The current state-of-the-art in sequence-based modeling is the use of AI/ML embeddings pretrained on multimillion-scale protein collections.118”
From Advancing Targeted Protein Degradation via Multiomics Profiling and Artificial Intelligence (2023) · cites this paper as [118]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since then, various language model-based protein embedding methods have been applied to these and other protein prediction problems through transfer learning, including protein phenotype prediction 28,32-34, residue-residue contact prediction 32,70, fold recognition 33, protein-protein 71,72 and protein-drug interaction prediction 35,73.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has been shown that the resulting representations contain rich information about the structure and the function of the proteins 17 , 32 , 33 .”
From Turnover number predictions for kinetically uncharacterized enzymes using machine and deep learning (2023) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the evolutionary scale modeling (ESM) family (Rives et al. 2021), comprising transformer protein language models such as ESM-2 and ESMFold, surpasses earlier single-sequence protein language models, demonstrating enhanced capability in generating accurate structure predictions based on protein sequences.”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Rives et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“New work has since shown that the representations learned by language models are also powerful features for learning of variant fitness as a subsequent supervised learning task 32,34, building on earlier observations that language models can improve protein property prediction through transfer learning 27.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [32]
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Basic local alignment search tool
Altschul, S. F., Gish, W., Miller, W., Myers, E. W., & Lipman, D. J. (1990) · Journal of Molecular Biology · doi:10.1006/jmbi.1990.9999 · Cited by 98,669
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
(2008) · Journal of Machine Learning Research · Cited by 33,051
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Kabsch, W., & Sander, C. (1983) · Biopolymers · doi:10.1002/bip.360221211 · Cited by 15,320
Research works printed but not linked: 4
- A. Radford . Language models are unsupervised multitask learners. OpenAI Blog [Preprint] (2019). https://openai.com/blog/better-language-models (Accessed 6 August 2020).
- 10.1016/S0022-2836(05)80360-2 · doi:10.1016/s0022-2836(05)80360-2 · Paper page
- A. Radford K. Narasimhan T. Salimans I. Sutskever Improving language understanding by generative pre-training. OpenAI Blog [Preprint] (2018). https://openai.com/blog/language-unsupervised (Accessed 6 August 2020).
- T. Mikolov . Subword language modeling with neural networks. The website of T. Mikolov [Preprint] (2012). http://www.fit.vutbr.cz/∼imikolov/rnnlm/char.pdf (Accessed 14 March 2021).
Cited by
Items 501–600 of 1,102
Automatic Gene Function Prediction in the 2020’s
Makrodimitris, S., van Ham, R. C. H. J., & Reinders, M. J. T. (2020) · Genes · doi:10.3390/genes11111264 · Cited by 44
CpG Transformer for imputation of single-cell methylomes
De Waele, G., Clauwaert, J., Menschaert, G., & Waegeman, W. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btab746 · Cited by 37
Saleh, A., Noguchi, Y., Aramayo, R. J., Ivanova, M. E., Stevens, K. M., Montoya, A., Sunidhi, S., Carranza, N. L., Skwark, M. J., & Speck, C. (2022) · Nature Communications · doi:10.1038/s41467-022-30576-1 · Cited by 44
Pakhrin, S. C., Pokharel, S., Pratyush, P., Chaudhari, M., Ismail, H. D., & KC, D. B. (2023) · Journal of Proteome Research · doi:10.1021/acs.jproteome.2c00667 · Cited by 40
Meng, Q., Guo, F., & Tang, J. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad217 · Cited by 41
Gong, J., Jiang, L., Chen, Y., Zhang, Y., Li, X., Ma, Z., Fu, Z., He, F., Sun, P., Ren, Z., et al. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad646 · Cited by 40
EMNGly: predicting N-linked glycosylation sites using the language models for feature extraction
Hou, X., Wang, Y., Bu, D., Wang, Y., & Sun, S. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad650 · Cited by 36
Kroll, A., Ranjan, S., & Lercher, M. J. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012100 · Cited by 50
Protein stability prediction by fine-tuning a protein language model on a mega-scale dataset
Chu, S. K. S., Narang, K., & Siegel, J. B. (2023) · bioRxiv · doi:10.1371/journal.pcbi.1012248 · Cited by 41
Wang, Y., Lv, H., Teo, Q. W., Lei, R., Gopal, A. B., Ouyang, W. O., Yeung, Y.-H., Tan, T. J. C., Choi, D., Shen, I. R., et al. (2024) · Immunity · doi:10.1016/j.immuni.2024.07.022 · Cited by 47
Grigorakis, K., Ferousi, C., & Topakas, E. (2025) · Catalysts · doi:10.3390/catal15020147 · Cited by 45
Machine learning in predictive biocatalysis: A comparative review of methods and applications
Tripathi, N., Hérisson, J., & Faulon, J. (2025) · Biotechnology Advances · doi:10.1016/j.biotechadv.2025.108698 · Cited by 40
Li, M., Kang, L., Xiong, Y., Wang, Y. G., Fan, G., Tan, P., & Hong, L. (2023) · Journal of Cheminformatics · doi:10.1186/s13321-023-00688-x · Cited by 35
Malbranke, C., Bikard, D., Cocco, S., Monasson, R., & Tubiana, J. (2023) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2023.102571 · Cited by 35
Ramstein, G. P., & Buckler, E. S. (2022) · Genome Biology · doi:10.1186/s13059-022-02747-2 · Cited by 34
Huang, B., Fan, T., Wang, K., Zhang, H., Yu, C., Nie, S., Qi, Y., Zheng, W.-M., Han, J., Fan, Z., et al. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad122 · Cited by 34
Yeung, W., Zhou, Z., Li, S., & Kannan, N. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbac599 · Cited by 43
scGGAN: single-cell RNA-seq imputation by graph-based generative adversarial network
Huang, Z., Wang, J., Lü, X., Zain, A. M., & Yu, G. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad040 · Cited by 42
Meta learning addresses noisy and under-labeled data in machine learning-guided antibody engineering
Minot, M., & Reddy, S. T. (2024) · Cell Systems · doi:10.1016/j.cels.2023.12.003 · Cited by 35
Zhang, X. J., Hu, X., Zhang, T., Yang, L., Liu, C., Xu, N., Wang, H., & Sun, W. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae404 · Cited by 41
Deep Learning Methods for De Novo Peptide Sequencing
Bittremieux, W., Ananth, V., Fondrie, W. E., Melendez, C., Pominova, M., Sanders, J., Wen, B., Yilmaz, M., & Noble, W. S. (2024) · Mass Spectrometry Reviews · doi:10.1002/mas.21919 · Cited by 45
SE(3)-equivariant ternary complex prediction towards target protein degradation
Xue, F., Zhang, M. L., Li, S., Gao, X., Wohlschlegel, J. A., Huang, W., Yang, Y., & Deng, W. (2025) · Nature Communications · doi:10.1038/s41467-025-61272-5 · Cited by 56
Critical assessment of missense variant effect predictors on disease-relevant variant data
Rastogi, R., Chung, R., Li, S., Li, C., Lee, K., Woo, J., Kim, D.-W., Keum, C., Babbi, G., Martelli, P. L., et al. (2025) · Human Genetics · doi:10.1007/s00439-025-02732-2 · Cited by 35
Democratizing protein language model training, sharing and collaboration
Su, J., Li, Z., Tao, T., Han, C., He, Y., Dai, F., Yuan, Q., Gao, Y., Si, T., Zhang, X., et al. (2026) · Nature Biotechnology · doi:10.1038/s41587-025-02859-7 · Cited by 35
Self-citationPretraining model for biological sequence data
Song, B., Li, Z., Lin, X., Wang, J., Wang, T., & Fu, X. (2021) · Briefings in Functional Genomics · doi:10.1093/bfgp/elab025 · Cited by 39
Alzheimer's Disease Classification Through Imaging Genetic Data With IGnet
Wang, J. X., Li, Y., Li, X., & Lu, Z. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.846638 · Cited by 41
3D-equivariant graph neural networks for protein model quality assessment
Chen, C., Chen, X., Morehead, A., Wu, T., & Cheng, J. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad030 · Cited by 33
Meta Learning Improves Robustness and Performance in Machine Learning-Guided Protein Engineering
Minot, M., & Reddy, S. T. (2023) · doi:10.1101/2023.01.30.526201 · Cited by 35
Wang, D., Pourmirzaei, M., Abbas, U. L., Zeng, S., Manshour, N., Esmaili, F., Poudel, B., Jiang, Y., Shao, Q., Chen, J., et al. (2025) · Advanced Science · doi:10.1002/advs.202404212 · Cited by 46
Flexible protein–protein docking with a multitrack iterative transformer
Chu, L., Ruffolo, J. A., Harmalkar, A., & Gray, J. J. (2024) · Protein Science · doi:10.1002/pro.4862 · Cited by 41
GlyNet: a multi-task neural network for predicting protein–glycan interactions
Carpenter, E. J., Seth, S., Yue, N., Greiner, R., & Derda, R. (2022) · Chemical Science · doi:10.1039/d1sc05681f · Cited by 38
Phosformer: an explainable transformer model for protein kinase-specific phosphorylation predictions
Zhou, Z., Yeung, W., Gravel, N., Salcedo, M., Soleymani, S., Li, S., & Kannan, N. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad046 · Cited by 44
The promises of large language models for protein design and modeling
Valentini, G., Malchiodi, D., Gliozzo, J., Mesiti, M., Soto-Gomez, M., Cabri, A., Reese, J., Casiraghi, E., & Robinson, P. N. (2023) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2023.1304099 · Cited by 48
Accurate prediction of antibody function and structure using bio-inspired antibody language model
Jing, H., Gao, Z., Xu, S. Y., Shen, T., Peng, Z., He, S., You, T., Ye, S., Lin, W. J., & Sun, S. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae245 · Cited by 41
General-purpose foundation models for increased autonomy in robot-assisted surgery
Schmidgall, S., Kim, J. W., Kuntz, A., Ghazi, A., & Krieger, A. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00917-4 · Cited by 59
Conditional generative modeling for de novo protein design with hierarchical functions
Kucera, T., Togninalli, M., & Meng-Papaxanthos, L. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac353 · Cited by 45
RAPPPID: towards generalizable protein interaction prediction with AWD-LSTM twin networks
Szymborski, J., & Emad, A. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac429 · Cited by 34
Machine Learning Approaches to TCR Repertoire Analysis
Katayama, Y., Yokota, R., Akiyama, T., & Kobayashi, T. (2022) · Frontiers in Immunology · doi:10.3389/fimmu.2022.858057 · Cited by 37
Pre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05006 · Cited by 34
CodonBERT: Large Language Models for mRNA design and optimizationPreprint
Li, S., Moayedpour, S., Li, R., Bailey, M., Riahi, S., Kogler-Anele, L., Miladi, M., Miner, J. C., Zheng, D., Wang, J., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.09.09.556981 · Cited by 65
Language Modeling Materializes a World Model of Protein BiologyPreprint
Candido, S., Hayes, T., Derry, A., Rao, R., Lin, Z., Verkuil, R., Wu, B. Z., Lee, J. S., Bruguera, E. S., Keval, J. A., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.03.729735 · Cited by 151
Self-citationLeveraging protein language models for accurate multiple sequence alignments
(2023) · doi:10.1101/gr.277675.123 · Cited by 30
Lin, K. Y., Quan, X., Jin, C., Shi, Z., & Yang, J. (2022) · Frontiers in Genetics · doi:10.3389/fgene.2022.885627 · Cited by 41
Hu, F., Hu, Y., Zhang, W., Huang, H., Pan, Y. F., & Yin, P. (2023) · Advanced Science · doi:10.1002/advs.202301223 · Cited by 40
Superior protein thermophilicity prediction with protein language model embeddings
Haselbeck, F., John, M., Zhang, Y., Pirnay, J., Werner, J., Costa, R. D., & Grimm, D. G. (2023) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqad087 · Cited by 37
Song, Y., Yuan, Q., Zhao, H., & Yang, Y. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad360 · Cited by 35
Machine learning in RNA structure prediction: Advances and challenges
Zhang, S., Li, J., & Chen, S. (2024) · Biophysical Journal · doi:10.1016/j.bpj.2024.01.026 · Cited by 36
Kim, D. N., McNaughton, A. D., & Kumar, N. (2024) · Bioengineering · doi:10.3390/bioengineering11020185 · Cited by 39
FusPB-ESM2: Fusion model of ProtBERT and ESM-2 for cell-penetrating peptide prediction
Zhang, F., Li, J., Wen, Z., & Fang, C. (2024) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2024.108098 · Cited by 37
Qian, Y., Su, L.-J., Xu, M., & Tang, L. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3523772 · Cited by 36
Topolska, M., Beltran, A., & Lehner, B. (2025) · Nature Communications · doi:10.1038/s41467-025-57510-5 · Cited by 33
Massive experimental quantification allows interpretable deep learning of protein aggregation
Thompson, M., Martín, M., Olmo, T. S., Rajesh, C., Koo, P. K., Bolognesi, B., & Lehner, B. (2025) · Science Advances · doi:10.1126/sciadv.adt5111 · Cited by 39
Machine Learning-Guided Identification of PET Hydrolases from Natural Diversity
Norton-Baker, B., Komp, E., Gado, J. E., Denton, M. C. R., Mathews, I., Murphy, N. P., Erickson, E., Storment, O. O., Sarangi, R., Gauthier, N., et al. (2025) · ACS Catalysis · doi:10.1021/acscatal.5c03460 · Cited by 35
Large-scale design and refinement of stable proteins using sequence-only models
Singer, J. M., Novotney, S., Strickland, D., Haddox, H. K., Leiby, N., Rocklin, G. J., Chow, C. M., Roy, A., Bera, A. K., Motta, F. C., et al. (2022) · PLOS ONE · doi:10.1371/journal.pone.0265020 · Cited by 32
Parameter-efficient fine-tuning on large protein language models improves signal peptide prediction
Zeng, S., Wang, D., Jiang, L., & Xu, D. H. (2024) · Genome Research · doi:10.1101/gr.279132.124 · Cited by 52
RamanFormer: A Transformer-Based Quantification Approach for Raman Mixture Components
Koyun, O. C., Keser, R. K., Şahin, S. O., Bulut, D., Yorulmaz, M., Yücesoy, V., & Töreyin, B. U. (2024) · ACS Omega · doi:10.1021/acsomega.3c09247 · Cited by 32
SVSBI: sequence-based virtual screening of biomolecular interactions
Shen, L., Feng, H., Qiu, Y., & Wei, G. (2023) · Communications Biology · doi:10.1038/s42003-023-04866-3 · Cited by 39
Zhang, Y., Guan, J., Li, C., Wang, Z., Deng, Z., Gasser, R. B., Song, J., & Ou, H. (2023) · Research · doi:10.34133/research.0258 · Cited by 41
TCR-ESM: Employing protein language embeddings to predict TCR-peptide-MHC binding
Yadav, S., Vora, D. S., Sundar, D., & Dhanjal, J. K. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.11.037 · Cited by 41
Multimodal pretraining for unsupervised protein representation learning
Nguyen, V. T. D., & Hy, T. S. (2024) · Biology Methods and Protocols · doi:10.1093/biomethods/bpae043 · Cited by 38
Natural Language Processing Methods for the Study of Protein–Ligand Interactions
Michels, J., Bandarupalli, R., Akbari, A. A., Le, T., Xiao, H. Y., Li, J., & Hom, E. F. Y. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01907 · Cited by 38
RNA language models predict mutations that improve RNA function
Shulgina, Y., Trinidad, M. I., Langeberg, C. J., Nisonoff, H., Chithrananda, S., Skopintsev, P., Nissley, A. J., Patel, J., Boger, R. S., Shi, H., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-54812-y · Cited by 32
Variant effect predictions capture some aspects of deep mutational scanning experiments
Reeb, J., Wirth, T., & Rost, B. (2020) · BMC Bioinformatics · doi:10.1186/s12859-020-3439-4 · Cited by 32
Zhang, Y., Zhang, Y., Xiong, Y., Wang, H., Deng, Z., Song, J., & Ou, H. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab420 · Cited by 34
Chen, J., Gu, Z., Xu, Y., Deng, M., Lai, L., & Pei, J. (2023) · Protein Science · doi:10.1002/pro.4555 · Cited by 34
Is Transfer Learning Necessary for Protein Landscape Prediction?Preprint
Shanehsazzadeh, A., Belanger, D., & Dohan, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.03443 · Cited by 73
Chen, K., Zhou, Y., Ding, M., Wang, Y., Ren, Z., & Yang, Y. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.31.526427 · Cited by 43
Evolutionary-scale enzymology enables exploration of a rugged catalytic landscape
(2025) · doi:10.1126/science.adu1058 · Cited by 42
Machine learning approaches for biomolecular, biophysical, and biomaterials research
Rickert, C. A., & Lieleg, O. (2022) · Biophysics Reviews · doi:10.1063/5.0082179 · Cited by 34
Pratyush, P., Pokharel, S., Saigo, H., & KC, D. B. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05164-9 · Cited by 31
Domain-PFP allows protein function prediction using function-aware domain embedding representations
Ibtehaz, N., Kagaya, Y., & Kihara, D. (2023) · Communications Biology · doi:10.1038/s42003-023-05476-9 · Cited by 37
Shen, J., Yu, Q., Chen, S., Tan, Q., Li, J., & Li, Y. (2023) · Nature Computational Science · doi:10.1038/s43588-023-00576-2 · Cited by 36
Advancing bioinformatics with language models: components, applications, and perspectives
Liu, J., Yang, M., Yu, Y., Xu, H., Wang, T., Li, K., & Zhou, X. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag367 · Cited by 72
Self-citationCarbery, A., Buttenschoen, M., Skyner, R., von Delft, F., & Deane, C. M. (2024) · Journal of Cheminformatics · doi:10.1186/s13321-024-00821-4 · Cited by 35
MDF-DTA: A Multi-Dimensional Fusion Approach for Drug-Target Binding Affinity Prediction
Ranjan, A., Bess, A., Alvin, C., & Mukhopadhyay, S. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00310 · Cited by 36
Jin, R., Ye, Q., Wang, J., Cao, Z., Jiang, D., Wang, T., Kang, Y., Xu, W., Hsieh, C., & Hou, T. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae304 · Cited by 44
Nielsen, J. C. F., Hjørringgaard, C. U., Nygaard, M. M., Wester, A., Elster, L., Porsgaard, T., Mikkelsen, R. B., Rasmussen, S. A., Madsen, A. N., Schlein, M., et al. (2024) · Journal of Medicinal Chemistry · doi:10.1021/acs.jmedchem.4c00417 · Cited by 34
Learning the language of antibody hypervariability
Singh, R., Im, C., Qiu, Y., Mackness, B. C., Gupta, A., Joren, T., Sledzieski, S., Erlach, L., Wendt, M., Fomekong‐Nanfack, Y., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2418918121 · Cited by 55
Zhou, B., Li, X., Pan, Y., He, B., & Gao, B. (2025) · Colloids and Surfaces B Biointerfaces · doi:10.1016/j.colsurfb.2025.114970 · Cited by 42
Cai, T., Lim, H., Abbu, K. A., Qiu, Y., Nussinov, R., & Xie, L. (2021) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c01285 · Cited by 30
Machine learning in energy chemistry: introduction, challenges and perspectives
Xu, Y., Ge, J., & Ju, C. (2023) · Energy Advances · doi:10.1039/d3ya00057e · Cited by 36
High-throughput deep learning variant effect prediction with Sequence UNET
Dunham, A. S., Beltrao, P., & AlQuraishi, M. (2023) · Genome Biology · doi:10.1186/s13059-023-02948-3 · Cited by 31
Li, S., Tian, T., Zhang, Z., Zou, Z., Zhao, D. Y., & Zeng, J. (2023) · Cell Systems · doi:10.1016/j.cels.2023.05.005 · Cited by 31
Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies
Ruffolo, J. A., Chu, L.-S., Mahajan, S. P., & Gray, J. J. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.20.488972 · Cited by 31
BEND: Benchmarking DNA Language Models on biologically meaningful tasks
Marin, F. I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., & Boomsma, W. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2311.12570 · Cited by 73
Beyond sequence: Structure-based machine learning
Durairaj, J., de Ridder, D., & van Dijk, A. D. J. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2022.12.039 · Cited by 33
Computationally guided AAV engineering for enhanced gene delivery
Guo, J., Lin, L. F., Oraskovich, S. V., de Jesús, J. A. R., Listgarten, J., & Schaffer, D. V. (2024) · Trends in Biochemical Sciences · doi:10.1016/j.tibs.2024.03.002 · Cited by 33
Arif, M., Musleh, S., Fida, H., & Alam, T. (2024) · Scientific Reports · doi:10.1038/s41598-024-67433-8 · Cited by 37
DeepMineLys: Deep mining of phage lysins from human microbiome
Fu, Y., Yu, S., Li, J., Lao, Z., Yang, X., & Lin, Z. (2024) · Cell Reports · doi:10.1016/j.celrep.2024.114583 · Cited by 35
Self-citationStructure-informed protein language models are robust predictors for variant effects
(2024) · doi:10.1007/s00439-024-02695-w · Cited by 30
Hao, Y., Liu, X., Fu, H., Shao, X., & Cai, W. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae497 · Cited by 35
Evaluation of residue-residue contact prediction methods: From retrospective to prospective
Zhang, H., Bei, Z., Xi, W., Hao, M., Ju, Z., Saravanan, K. M., Zhang, H., Guo, N., & Wei, Y. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009027 · Cited by 34
Self-Supervised Pre-training for Protein Embeddings Using Tertiary Structures
Guo, Y., Wu, J., Ma, H., & Huang, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i6.20636 · Cited by 44
GPN-MSA: an alignment-based DNA language model for genome-wide variant effect prediction
Benegas, G., Albors, C., Aw, A. J., Ye, C., & Song, Y. S. (2023) · doi:10.1101/2023.10.10.561776 · Cited by 118
Medium-sized protein language models perform well at transfer learning on realistic datasets
Vieira, L. C., Handojo, M. L., & Wilke, C. O. (2025) · Scientific Reports · doi:10.1038/s41598-025-05674-x · Cited by 41
HNetGO: protein function prediction via heterogeneous network transformer
Zhang, X., Guo, H., Zhang, F., Wang, X., Wu, K., Qiu, S., Liu, B., Wang, Y., Hu, Y., & Li, J. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbab556 · Cited by 33
Dou, Z., Sun, Y., Jiang, X., Wu, X., Li, Y., Gong, B., & Wang, L. (2023) · Acta Biochimica et Biophysica Sinica · doi:10.3724/abbs.2023033 · Cited by 34
DG-Affinity: predicting antigen–antibody affinity with language models from sequences
Yuan, Y., Chen, Q., Mao, J., Li, G., & Pan, X. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05562-z · Cited by 41
M-Ionic: prediction of metal-ion-binding sites from sequence using residue embeddings
Shenoy, A., Kalakoti, Y., Sundar, D., & Elofsson, A. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btad782 · Cited by 32
ALDELE: All-Purpose Deep Learning Toolkits for Predicting the Biocatalytic Activities of Enzymes
Wang, X., Quinn, D. J., Moody, T. S., & Huang, M. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00058 · Cited by 28
Metadata
- Authors
- Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., & Fergus, R.
- Affiliations
- Meta (United States) (US)
- New York University (US)
- Harvard University (US)
- Yale University (US)
- Corresponding author
- Rives, Alexander
- ORCID
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2021-04-05
- Volume
- 118
- Issue
- 15
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC8053943
- OpenAlex
- W3146944767
- Cited by
- 3,286
- References
- 100
- Citations by year
- 2026: 787 · 2025: 813 · 2024: 697 · 2023: 448 · 2022: 305 · 2021: 162
- FWCI
- 126.15
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Knowledge@UChicago (University of Chicago) (submittedVersion, cc-by-nc-nd)
- University of Chicago
- Funders
- National Science Foundation (1339362)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
- Keywords
- protein language models, unsupervised learning, protein evolution, protein structure prediction, secondary structure prediction, protein tertiary structure, remote homology
- MeSH terms
- Unsupervised Machine Learning, Amino Acids, Protein Conformation, Sequence Homology, Amino Acid, Sequence Analysis, Protein
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.