Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
Open accessPublished version of a preprint
The DOI matches its Crossref record; 123 of the 123 references it cites were found in a registry or an index.
Abstract
Significance: Learning biological properties from sequence data is a logical step toward generative and predictive artificial intelligence for biology. Here, we propose scaling a deep contextual language model with unsupervised learning to sequences spanning evolutionary diversity. We find that without prior knowledge, information emerges in the learned representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity. We show the learned representations are useful across benchmarks for remote homology detection, prediction of secondary structure, long-range residue–residue contacts, and mutational effect. Unsupervised representation learning enables state-of-the-art supervised prediction of mutational effect and secondary structure and improves state-of-the-art features for long-range contact prediction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The current state-of-the-art in sequence-based modeling is the use of AI/ML embeddings pretrained on multimillion-scale protein collections.118”
From Advancing Targeted Protein Degradation via Multiomics Profiling and Artificial Intelligence (2023) · cites this paper as [118]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since then, various language model-based protein embedding methods have been applied to these and other protein prediction problems through transfer learning, including protein phenotype prediction 28,32-34, residue-residue contact prediction 32,70, fold recognition 33, protein-protein 71,72 and protein-drug interaction prediction 35,73.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [32]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It has been shown that the resulting representations contain rich information about the structure and the function of the proteins 17 , 32 , 33 .”
From Turnover number predictions for kinetically uncharacterized enzymes using machine and deep learning (2023) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the evolutionary scale modeling (ESM) family (Rives et al. 2021), comprising transformer protein language models such as ESM-2 and ESMFold, surpasses earlier single-sequence protein language models, demonstrating enhanced capability in generating accurate structure predictions based on protein sequences.”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Rives et al. 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“New work has since shown that the representations learned by language models are also powerful features for learning of variant fitness as a subsequent supervised learning task 32,34, building on earlier observations that language models can improve protein property prediction through transfer learning 27.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [32]
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Basic local alignment search tool
Altschul, S. F., Gish, W., Miller, W., Myers, E. W., & Lipman, D. J. (1990) · Journal of Molecular Biology · doi:10.1006/jmbi.1990.9999 · Cited by 98,669
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
(2008) · Journal of Machine Learning Research · Cited by 33,051
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Kabsch, W., & Sander, C. (1983) · Biopolymers · doi:10.1002/bip.360221211 · Cited by 15,320
Research works printed but not linked: 4
- A. Radford . Language models are unsupervised multitask learners. OpenAI Blog [Preprint] (2019). https://openai.com/blog/better-language-models (Accessed 6 August 2020).
- 10.1016/S0022-2836(05)80360-2 · doi:10.1016/s0022-2836(05)80360-2 · Paper page
- A. Radford K. Narasimhan T. Salimans I. Sutskever Improving language understanding by generative pre-training. OpenAI Blog [Preprint] (2018). https://openai.com/blog/language-unsupervised (Accessed 6 August 2020).
- T. Mikolov . Subword language modeling with neural networks. The website of T. Mikolov [Preprint] (2012). http://www.fit.vutbr.cz/∼imikolov/rnnlm/char.pdf (Accessed 14 March 2021).
Cited by
Items 101–200 of 1,102
Ankh ☥: Optimized Protein Language Model Unlocks General-Purpose Modelling
Elnaggar, A., Essam, H., Salah-Eldin, W., Moustafa, W., Elkerdawy, M., Rochereau, C., & Rost, B. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.16.524265 · Cited by 184
Modeling conformational states of proteins with AlphaFold
Sala, D., Engelberger, F., Mchaourab, H. S., & Meiler, J. (2023) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2023.102645 · Cited by 182
Language models generalize beyond natural proteins
Verkuil, R., Kabeli, O., Du, Y., Wicky, B. I. M., Milles, L. F., Dauparas, J., Baker, D., Ovchinnikov, S., Sercu, T., & Rives, A. (2022) · doi:10.1101/2022.12.21.521521 · Cited by 181
Self-citationApplications of transformer-based language models in bioinformatics: a survey
Zhang, S., Fan, R., Liu, Y., Chen, S., Liu, Q., & Zeng, W. (2023) · Bioinformatics Advances · doi:10.1093/bioadv/vbad001 · Cited by 176
Computational and artificial intelligence-based methods for antibody development
Kim, J., McFee, M., Fang, Q., Abdin, O., & Kim, P. M. (2023) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2022.12.005 · Cited by 178
Embeddings from deep learning transfer GO annotations beyond homology
Littmann, M., Heinzinger, M., Dallago, C., Olenyi, T., & Rost, B. (2021) · Scientific Reports · doi:10.1038/s41598-020-80786-0 · Cited by 175
Transformers in single-cell omics: a review and new perspectives
Szałata, A., Hrovatin, K., Becker, S., Tejada-Lapuerta, A., Cui, H., Wang, B., & Theis, F. J. (2024) · Nature Methods · doi:10.1038/s41592-024-02353-z · Cited by 171
Yuan, W., Chen, G., & Chen, C. Y.-C. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbab506 · Cited by 172
Embeddings from protein language models predict conservation and variant effects
(2021) · doi:10.1007/s00439-021-02411-y · Cited by 174
Pre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2024) · ACM Computing Surveys · doi:10.1145/3611651 · Cited by 169
ECNet is an evolutionary context-integrated deep learning framework for protein engineering
Luo, Y., Jiang, G., Yu, T., Liu, Y., Vo, L., Ding, H., Su, Y., Qian, W. W., Zhao, H., & Peng, J. (2021) · Nature Communications · doi:10.1038/s41467-021-25976-8 · Cited by 173
Learning meaningful representations of protein sequences
Detlefsen, N. S., Hauberg, S., & Boomsma, W. (2022) · Nature Communications · doi:10.1038/s41467-022-29443-w · Cited by 168
IgLM: Infilling language modeling for antibody sequence design
Shuai, R. W., Ruffolo, J. A., & Gray, J. J. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.001 · Cited by 163
Blockwise Self-Attention for Long Document Understanding
Qiu, J., Ma, H., Levy, O., Yih, W.-T., Wang, S., & Tang, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.232 · Cited by 161
Advances in machine learning for directed evolution
Wittmann, B. J., Johnston, K. E., Wu, Z., & Arnold, F. H. (2021) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2021.01.008 · Cited by 164
Neural networks to learn protein sequence–function relationships from deep mutational scanning data
Gelman, S., Fahlberg, S. A., Heinzelman, P., Romero, P. A., & Gitter, A. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2104878118 · Cited by 164
Universal Cell Embeddings: A Foundation Model for Cell Biology
Rosen, Y., Roohani, Y., Agrawal, A., Samotorcan, L., Consortium, T. S., Quake, S. R., & Leskovec, J. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.11.28.568918 · Cited by 162
Goshisht, M. K. (2024) · ACS Omega · doi:10.1021/acsomega.3c05913 · Cited by 161
TALE: Transformer-based protein function Annotation with joint sequence–Label Embedding
Cao, Y., & Shen, Y. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab198 · Cited by 157
Deciphering the language of antibodies using self-supervised learning
Leem, J., Mitchell, L. S., Farmery, J. H. R., Barton, J., & Galson, J. D. (2022) · Patterns · doi:10.1016/j.patter.2022.100513 · Cited by 154
Protein generation with evolutionary diffusion: sequence is all you need
Alamdari, S., Thakkar, N., van den Berg, R., Tenenholtz, N., Strome, R., Moses, A. M., Lu, A. X., Fusi, N., Amini, A. P., & Yang, K. K. (2023) · doi:10.1101/2023.09.11.556673 · Cited by 154
Single-sequence protein structure prediction using supervised transformer protein language models
Wang, W., Peng, Z., & Yang, J. (2022) · Nature Computational Science · doi:10.1038/s43588-022-00373-3 · Cited by 150
Hie, B. L., Yang, K. K., & Kim, P. S. (2022) · Cell Systems · doi:10.1016/j.cels.2022.01.003 · Cited by 149
DeepConsensus improves the accuracy of sequences with a gap-aware sequence transformer
Baid, G., Cook, D. E., Shafin, K., Yun, T., Llinares-López, F., Berthet, Q., Belyaeva, A., Töpfer, A., Wenger, A. M., Rowell, W. J., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01435-7 · Cited by 197
Fast end-to-end learning on protein surfaces
Sverrisson, F., Feydy, J., Correia, B. E., & Bronstein, M. M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01502 · Cited by 132
Deep generative models for peptide design
Wan, F., Kontogiorgos-Heintz, D., & de la Fuente-Nunez, C. (2022) · Digital Discovery · doi:10.1039/d1dd00024a · Cited by 141
Protein remote homology detection and structural alignment using deep learning
Hamamsy, T., Morton, J. T., Blackwell, R., Berenberg, D., Carriero, N., Gligorijevic, V., Strauss, C. E. M., Leman, J. K., Cho, K., & Bonneau, R. (2024) · Nature Biotechnology · doi:10.1038/s41587-023-01917-2 · Cited by 143
(2022) · doi:10.1016/j.str.2022.05.001 · Cited by 139
A review of biomedical datasets relating to drug discovery: a knowledge graph perspective
Bonner, S., Barrett, I. P., Ye, C., Swiers, R., Engkvist, O., Bender, A., Hoyt, C. T., & Hamilton, W. L. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac404 · Cited by 197
Born, J., & Manica, M. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00639-z · Cited by 219
Active learning-assisted directed evolution
Yang, J., Lal, R., Bowden, J. C., Astudillo, R., Hameedi, M. A., Kaur, S., Hill, M., Yue, Y., & Arnold, F. H. (2025) · Nature Communications · doi:10.1038/s41467-025-55987-8 · Cited by 144
Akiyama, M., & Sakakibara, Y. (2022) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqac012 · Cited by 138
Exploring the Limits of Out-of-Distribution Detection
Fort, S., Ren, J., & Lakshminarayanan, B. (2021) · arXiv · doi:10.48550/arxiv.2106.03004 · Cited by 142
AI applications in functional genomics
Caudai, C., Galizia, A., Geraci, F., Le Pera, L., Morea, V., Salerno, E., Via, A., & Colombo, T. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.10.009 · Cited by 137
Protein language models learn evolutionary statistics of interacting sequence motifs
Zhang, Z., Wayment-Steele, H. K., Brixi, G., Wang, H., Kern, D., & Ovchinnikov, S. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2406285121 · Cited by 140
Artificial intelligence in early drug discovery enabling precision medicine
Boniolo, F., Dorigatti, E., Ohnmacht, A. J., Saur, D., Schubert, B., & Menden, M. P. (2021) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2021.1918096 · Cited by 170
Protein language model embeddings for fast, accurate, alignment-free protein structure prediction
Weissenow, K., Heinzinger, M., & Rost, B. (2021) · doi:10.1101/2021.07.31.454572 · Cited by 139
Artificial Intelligence in Pharmaceutical Sciences
Lu, M., Yin, J., Zhu, Q., Lin, G., Mou, M., Liu, F., Pan, Z., You, N., Lian, X., Li, F., et al. (2023) · Engineering · doi:10.1016/j.eng.2023.01.014 · Cited by 157
Convolutions are competitive with transformers for protein sequence pretraining
Yang, K. K., Fusi, N., & Lu, A. X. (2024) · Cell Systems · doi:10.1016/j.cels.2024.01.008 · Cited by 137
Accelerated enzyme engineering by machine-learning guided cell-free expression
Landwehr, G. M., Bogart, J. W., Magalhaes, C., Hammarlund, E., Karim, A. S., & Jewett, M. C. (2025) · Nature Communications · doi:10.1038/s41467-024-55399-0 · Cited by 145
Cell-penetrating peptides in protein mimicry and cancer therapeutics
Zorko, M., Jones, S., & Langel, Ü. (2021) · Advanced Drug Delivery Reviews · doi:10.1016/j.addr.2021.114044 · Cited by 141
Du, Z., Ding, X., Xu, Y., & Li, Y. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad135 · Cited by 155
Lai, B., & Xu, J. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbab502 · Cited by 130
Zheng, W., Wuyun, Q., Li, Y., Zhang, C., Freddolino, L., & Zhang, Y. (2024) · Nature Methods · doi:10.1038/s41592-023-02130-4 · Cited by 130
Light attention predicts protein location from the language of life
Stärk, H., Dallago, C., Heinzinger, M., & Rost, B. (2021) · Bioinformatics Advances · doi:10.1093/bioadv/vbab035 · Cited by 130
Yan, J., Cai, J., Zhang, B., Wang, Y., Wong, D. F., & Siu, S. W. I. (2022) · Antibiotics · doi:10.3390/antibiotics11101451 · Cited by 146
Rosen, Y., Brbić, M., Roohani, Y., Swanson, K., Li, Z., & Leskovec, J. (2024) · Nature Methods · doi:10.1038/s41592-024-02191-z · Cited by 127
RiNALMo: general-purpose RNA language models can generalize well on structure prediction tasks
Penić, R. J., Vlašić, T., Huber, R. G., Wan, Y., & Šikić, M. (2025) · Nature Communications · doi:10.1038/s41467-025-60872-5 · Cited by 122
FLIP: Benchmark tasks in fitness landscape inference for proteins
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K. K. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.11.09.467890 · Cited by 125
Development and use of machine learning algorithms in vaccine target selection
Bravi, B. (2024) · npj Vaccines · doi:10.1038/s41541-023-00795-8 · Cited by 161
Xu, J., Li, F., Li, C., Guo, X., Landersdorfer, C. B., Shen, H., Peleg, A. Y., Li, J., Imoto, S., Yao, J., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad240 · Cited by 146
Scientific Large Language Models: A Survey on Biological & Chemical Domains
Zhang, Q., Ding, K., Lv, T., Wang, X., Yin, Q., Zhang, Y., Yu, J., Wang, Y., Li, X., Xiang, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3715318 · Cited by 122
Protein function prediction as approximate semantic entailment
Kulmanov, M., Guzmán-Vega, F. J., Duek Roggli, P., Lane, L., Arold, S. T., & Hoehndorf, R. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00795-w · Cited by 131
From sequence to function through structure: Deep learning for protein design
(2022) · doi:10.1016/j.csbj.2022.11.014 · Cited by 124
Villegas-Morcillo, A., Makrodimitris, S., van Ham, R. C. H. J., Gomez, A. M., Sanchez, V., & Reinders, M. J. T. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btaa701 · Cited by 123
Toward the solution of the protein structure prediction problem
Pearce, R., & Zhang, Y. (2021) · Journal of Biological Chemistry · doi:10.1016/j.jbc.2021.100870 · Cited by 122
Transformer models in biomedicine
Madan, S., Lentzen, M., Brandt, J., Rueckert, D., Hofmann‐Apitius, M., & Fröhlich, H. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02600-5 · Cited by 172
Machine learning-enabled retrobiosynthesis of molecules
Yu, T., Boob, A. G., Volk, M., Liu, X., Cui, H., & Zhao, H. (2023) · Nature Catalysis · doi:10.1038/s41929-022-00909-w · Cited by 133
Functional annotation of enzyme-encoding genes using deep learning with transformer layers
Kim, G. B., Kim, J. Y., Lee, J. A., Norsigian, C. J., Palsson, B. O., & Lee, S. Y. (2023) · Nature Communications · doi:10.1038/s41467-023-43216-z · Cited by 118
Machine learning modeling of family wide enzyme-substrate specificity screens
Goldman, S., Das, R., Yang, K. K., & Coley, C. W. (2022) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1009853 · Cited by 120
GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., et al. (2023) · The International Journal of High Performance Computing Applications · doi:10.1177/10943420231201154 · Cited by 120
Genomic language models: opportunities and challenges
Benegas, G., Ye, C., Albors, C., Li, J. C., & Song, Y. S. (2025) · Trends in Genetics · doi:10.1016/j.tig.2024.11.013 · Cited by 191
A generalized platform for artificial intelligence-powered autonomous enzyme engineering
Singh, N., Lane, S., Yu, T., Lu, J., Ramos, A., Cui, H., & Zhao, H. (2025) · Nature Communications · doi:10.1038/s41467-025-61209-y · Cited by 124
GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., et al. (2022) · doi:10.1101/2022.10.10.511571 · Cited by 120
Designing antimicrobial peptides using deep learning and molecular dynamic simulations
Cao, Q., Ge, C., Wang, X., Harvey, P. J., Zhang, Z., Ma, Y., Wang, X., Jia, X., Mobli, M., Craik, D. J., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad058 · Cited by 116
Sequence-to-sequence translation from mass spectra to peptides with a transformer model
Yilmaz, M., Fondrie, W. E., Bittremieux, W., Melendez, C. F., Nelson, R., Ananth, V., Oh, S., & Noble, W. S. (2024) · Nature Communications · doi:10.1038/s41467-024-49731-x · Cited by 114
Fang, X., Wang, F., Liu, L., He, J., Lin, D., Xiang, Y., Zhu, K., Zhang, X., Wu, H., Li, H., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00721-6 · Cited by 132
A DNA language model based on multispecies alignment predicts the effects of genome-wide variants
Benegas, G., Albors, C., Aw, A. J., Ye, C., & Song, Y. S. (2025) · Nature Biotechnology · doi:10.1038/s41587-024-02511-w · Cited by 115
Deep generative modeling for protein design
Strokach, A., & Kim, P. M. (2022) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2021.11.008 · Cited by 115
Disease diagnostics using machine learning of B cell and T cell receptor sequences
Zaslavsky, M., Craig, E., Michuda, J., Sehgal, N., Ram-Mohan, N., Lee, J.-Y., Nguyen, K. D., Hoh, R. A., Pham, T. D., Röltgen, K., et al. (2025) · Science · doi:10.1126/science.adp2407 · Cited by 136
Akbar, R., Bashour, H., Rawat, P., Robert, P. A., Smorodina, E., Cotet, T.-S., Flem-Karlsen, K., Frank, R., Mehta, B. B., Vu, M. H., et al. (2022) · mAbs · doi:10.1080/19420862.2021.2008790 · Cited by 113
NetSolP: predicting protein solubility in Escherichia coli using language models
Thumuluri, V., Martiny, H.-M., Almagro Armenteros, J. J., Salomon, J., Nielsen, H., & Johansen, A. R. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btab801 · Cited by 115
Amino acid encoding for deep learning applications
ElAbd, H., Bromberg, Y., Hoarfrost, A., Lenz, T., Franke, A., & Wendorff, M. (2020) · BMC Bioinformatics · doi:10.1186/s12859-020-03546-x · Cited by 113
A foundational large language model for edible plant genomes
Mendoza‐Revilla, J., Trop, E., Gonzalez, L., Roller, M., Dalla-Torre, H., de Almeida, B. P., Richard, G., Caton, J., Carranza, N. L., Skwark, M. J., et al. (2024) · Communications Biology · doi:10.1038/s42003-024-06465-2 · Cited by 143
CatPred: a comprehensive framework for deep learning in vitro enzyme kinetic parameters
Boorla, V. S., & Maranas, C. D. (2025) · Nature Communications · doi:10.1038/s41467-025-57215-9 · Cited by 162
Markus, B., Gruber Christian, C., Andreas, K., Kummer, A., Stefan, L., Oberdorfer, G., Elina, S., & Šnajdrová, R. (2023) · ACS Catalysis · doi:10.1021/acscatal.3c03417 · Cited by 127
A new age in protein design empowered by deep learning
Khakzad, H., Igashov, I., Schneuing, A., Goverde, C., Bronstein, M., & Correia, B. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.006 · Cited by 109
Contrastive learning on protein embeddings enlightens midnight zone
(2022) · doi:10.1093/nargab/lqac043 · Cited by 105
TMbed: transmembrane proteins predicted through language model embeddings
Bernhofer, M., & Rost, B. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04873-x · Cited by 106
Ghafarollahi, A., & Buehler, M. J. (2024) · Digital Discovery · doi:10.1039/d4dd00013g · Cited by 106
Masked Inverse Folding with Sequence Transfer for Protein Representation Learning
Yang, K. K., Yeh, H., & Zanichelli, N. (2022) · doi:10.1101/2022.05.25.493516 · Cited by 110
xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein
Chen, B., Cheng, X., Li, P., Geng, Y.-A., Gong, J., Li, S., Bei, Z., Tan, X., Wang, B., Zeng, X., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.07.05.547496 · Cited by 107
Machine learning to navigate fitness landscapes for protein engineering
Freschlin, C. R., Fahlberg, S. A., & Romero, P. A. (2022) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2022.102713 · Cited by 107
Orlando, M., Molla, G., Castellani, P., Pirillo, V., Torretta, V., & Ferronato, N. (2023) · International Journal of Molecular Sciences · doi:10.3390/ijms24043877 · Cited by 123
GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT
Chen, Y., & Zou, J. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.16.562533 · Cited by 105
Multiple sequence alignment-based RNA language model and its application to structural inference
Zhang, Y., Lang, M., Jiang, J., Gao, Z., Xu, F., Litfin, T., Chen, K., Singh, J., Huang, X., Song, G., et al. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkad1031 · Cited by 112
Learned Embeddings from Deep Learning to Visualize and Predict Protein Sets
Dallago, C., Schütze, K., Heinzinger, M., Olenyi, T., Littmann, M., Lu, A. X., Yang, K. K., Min, S., Yoon, S., Morton, J. T., et al. (2021) · Current Protocols · doi:10.1002/cpz1.113 · Cited by 104
Glycoinformatics in the Artificial Intelligence Era
Bojar, D., & Lisacek, F. (2022) · Chemical Reviews · doi:10.1021/acs.chemrev.2c00110 · Cited by 101
A novel antibacterial peptide recognition algorithm based on BERT
Zhang, Y., Lin, J., Zhao, L., Zeng, X., & Liu, X. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab200 · Cited by 105
Predicting functional effect of missense variants using graph attention neural networks
Zhang, H., Xu, M. S., Fan, X., Chung, W. K., & Shen, Y. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00561-w · Cited by 98
PLASMe: a tool to identify PLASMid contigs from short-read assemblies using transformer
Tang, X., Shang, J., Ji, Y., & Sun, Y. (2023) · Nucleic Acids Research · doi:10.1093/nar/gkad578 · Cited by 107
PLMSearch: Protein language model powers accurate and fast sequence search for remote homology
Liu, W., Wang, Z., You, R., Xie, C., Wei, H., Xiong, Y., Yang, J., & Zhu, S. (2024) · Nature Communications · doi:10.1038/s41467-024-46808-5 · Cited by 96
SignalP: The Evolution of a Web Server
Nielsen, H., Teufel, F., Brunak, S., & von HEIJNE, G. (2024) · Methods in molecular biology · doi:10.1007/978-1-0716-4007-4_17 · Cited by 127
Integrating protein language models and automatic biofoundry for enhanced protein evolution
Zhang, Q., Chen, W., Qin, M., Wang, Y., Pu, Z., Ding, K., Liu, Y., Zhang, Q., Li, D., Li, X., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-56751-8 · Cited by 139
AlQuraishi, M., & Sorger, P. K. (2021) · Nature Methods · doi:10.1038/s41592-021-01283-4 · Cited by 96
Genomic language model predicts protein co-regulation and function
Hwang, Y., Cornman, A. L., Kellogg, E. H., Ovchinnikov, S., & Girguis, P. R. (2024) · Nature Communications · doi:10.1038/s41467-024-46947-9 · Cited by 99
Qiao, Y., Zhu, X., & Gong, H. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab712 · Cited by 100
Yuan, Q., Xie, J., Xie, J., Zhao, H., & Yang, Y. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad117 · Cited by 95
pLM4ACE: A protein language model based predictor for antihypertensive peptide screening
Du, Z., Ding, X., Hsu, W. W. Y., Munir, A., Xu, Y., & Li, Y. (2023) · Food Chemistry · doi:10.1016/j.foodchem.2023.137162 · Cited by 103
Large-scale chemoproteomics expedites ligand discovery and predicts ligand behavior in cells
Offensperger, F., Tin, G., Duran-Frigola, M., Hahn, E., Dobner, S., Ende, C. W. A., Strohbach, J. W., Rukavina, A., Brennsteiner, V., Ogilvie, K., et al. (2024) · Science · doi:10.1126/science.adk5864 · Cited by 94
Metadata
- Authors
- Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., & Fergus, R.
- Affiliations
- Meta (United States) (US)
- New York University (US)
- Harvard University (US)
- Yale University (US)
- Corresponding author
- Rives, Alexander
- ORCID
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2021-04-05
- Volume
- 118
- Issue
- 15
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC8053943
- OpenAlex
- W3146944767
- Cited by
- 3,286
- References
- 100
- Citations by year
- 2026: 787 · 2025: 813 · 2024: 697 · 2023: 448 · 2022: 305 · 2021: 162
- FWCI
- 126.15
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Knowledge@UChicago (University of Chicago) (submittedVersion, cc-by-nc-nd)
- University of Chicago
- Funders
- National Science Foundation (1339362)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
- Keywords
- protein language models, unsupervised learning, protein evolution, protein structure prediction, secondary structure prediction, protein tertiary structure, remote homology
- MeSH terms
- Unsupervised Machine Learning, Amino Acids, Protein Conformation, Sequence Homology, Amino Acid, Sequence Analysis, Protein
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.