Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Rives, Alexander, Meier, Joshua, Sercu, Tom, Goyal, Siddharth, Lin, Zeming, Liu, Jason, Guo, Demi, Ott, Myle +3 more·2019·Proceedings of the National Academy of Sciences
DOI 10.1101/622803

Cited by (1,005)

Items 101–200 of 1,005

  1. FusionDTA: attention-based feature polymerizer and knowledge distillation for drug-target binding affinity prediction
    Weining Yuan, Guanxing Chen, Calvin Yu‐Chian Chen · 2021 · Briefings in Bioinformatics
  2. Pre-trained Language Models in Biomedical Domain: A Systematic Survey
    Benyou Wang, Qianqian Xie, Jiahuan Pei, Zhihong Chen, Prayag Tiwari, Zhao Li, Jie Fu · 2023 · ACM Computing Surveys
  3. ECNet is an evolutionary context-integrated deep learning framework for protein engineering
    Yunan Luo, Guangde Jiang, Tianhao Yu, Yang Liu, Lam Tung Vo, Hantian Ding, Yufeng Su, Wesley Wei Qian, Huimin Zhao, Jian Peng · 2021 · Nature Communications
  4. Learning meaningful representations of protein sequences
    Nicki Skafte Detlefsen, Søren Hauberg, Wouter Boomsma · 2022 · Nature Communications
  5. IgLM: Infilling language modeling for antibody sequence design
    Richard W. Shuai, Jeffrey A. Ruffolo, Jeffrey J. Gray · 2023 · Cell Systems
  6. Protein generation with evolutionary diffusion: sequence is all you need
    Alamdari, Sarah, Thakkar, Nitya, van den Berg, Rianne, Tenenholtz, Neil, Strome, Robert, Moses, Alan M., Lu, Alex X., Fusi, Nicolò, Amini, Ava P., Yang, Kevin K. · 2023
  7. Blockwise Self-Attention for Long Document Understanding
    Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, Jie Tang · 2020 · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)
  8. Advances in machine learning for directed evolution
    Bruce J. Wittmann, Kadina E. Johnston, Zachary Wu, Frances H. Arnold · 2021 · Current Opinion in Structural Biology
  9. Neural networks to learn protein sequence–function relationships from deep mutational scanning data
    Sam Gelman, Sarah A. Fahlberg, Pete Heinzelman, Philip A. Romero, Anthony Gitter · 2021 · Proceedings of the National Academy of Sciences
  10. Universal Cell Embeddings: A Foundation Model for Cell Biology
    Yanay Rosen, Yusuf Roohani, Ayush Agrawal, Leon Samotorcan, Stephen R. Quake, Jure Leskovec · 2023 · bioRxiv (Cold Spring Harbor Laboratory)
  11. Machine Learning and Deep Learning in Synthetic Biology: Key Architectures, Applications, and Challenges
    Manoj Kumar Goshisht · 2024 · ACS Omega
  12. TALE: Transformer-based protein function Annotation with joint sequence–Label Embedding
    Yue Cao, Yang Shen · 2021 · Bioinformatics
  13. Deciphering the language of antibodies using self-supervised learning
    Jinwoo Leem, L. Mitchell, James H. R. Farmery, Justin Barton, Jacob D. Galson · 2022 · Patterns
  14. Single-sequence protein structure prediction using supervised transformer protein language models
    Wang, Wenkai, Peng, Zhenling, Yang, Jianyi · 2022 · Nature Computational Science
  15. Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteins
    Brian Hie, Kevin Yang, Peter S. Kim · 2022 · Cell Systems
  16. DeepConsensus improves the accuracy of sequences with a gap-aware sequence transformer
    Gunjan Baid, Daniel E. Cook, Kishwar Shafin, Taedong Yun, Felipe Llinares-López, Quentin Berthet, Anastasiya Belyaeva, Armin Töpfer, Aaron M. Wenger, William J. Rowell et al. · 2022 · Nature Biotechnology
  17. Fast end-to-end learning on protein surfaces
    Freyr Sverrisson, Jean Feydy, Bruno E. Correia, Michael M. Bronstein · 2021 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings
  18. Deep generative models for peptide design
    Fang Wan, Daphne Kontogiorgos-Heintz, César de la Fuente‐Núñez · 2022 · Digital Discovery
  19. Protein remote homology detection and structural alignment using deep learning
    Tymor Hamamsy, James T. Morton, Robert Blackwell, Daniel Berenberg, Nicholas John Carriero, Vladimir Gligorijević, Charlie E. M. Strauss, Julia Koehler Leman, Kyunghyun Cho, Richard Bonneau · 2023 · Nature Biotechnology
  20. Protein language-model embeddings for fast, accurate, and alignment-free protein structure prediction
    2022
  21. A review of biomedical datasets relating to drug discovery: a knowledge graph perspective
    Stephen Bonner, Ian P. Barrett, Cheng Ye, Rowan Swiers, Ola Engkvist, Andreas Bender, Charles Tapley Hoyt, William L. Hamilton · 2022 · Briefings in Bioinformatics
  22. Regression Transformer enables concurrent sequence regression and generation for molecular language modelling
    Jannis Born, Matteo Manica · 2023 · Nature Machine Intelligence
  23. Active learning-assisted directed evolution
    Jason Yang, Ravi Lal, James C. Bowden, Raul Astudillo, Mikhail A. Hameedi, Sukhvinder Kaur, Matthew Hill, Yisong Yue, Frances H. Arnold · 2025 · Nature Communications
  24. Informative RNA base embedding for RNA structural alignment and clustering by deep representation learning
    Manato Akiyama, Yasubumi Sakakibara · 2022 · NAR Genomics and Bioinformatics
  25. AI applications in functional genomics
    Claudia Caudai, Antonella Galizia, Filippo Geraci, Loredana Le Pera, Veronica Morea, Emanuele Salerno, Allegra Via, Teresa Colombo · 2021 · Computational and Structural Biotechnology Journal
  26. Artificial intelligence in early drug discovery enabling precision medicine
    Fabio Boniolo, Emilio Dorigatti, Alexander J. Ohnmacht, Dieter Saur, Benjamin Schubert, Michael P. Menden · 2021 · Expert Opinion on Drug Discovery
  27. Artificial Intelligence in Pharmaceutical Sciences
    Mingkun Lu, Jiayi Yin, Qi Zhu, Gaole Lin, Minjie Mou, Fuyao Liu, Ziqi Pan, Nanxin You, Xichen Lian, Fengcheng Li et al. · 2023 · Engineering
  28. Accelerated enzyme engineering by machine-learning guided cell-free expression
    Grant M. Landwehr, Jonathan W. Bogart, Carol Magalhaes, Eric Hammarlund, Ashty S. Karim, Michael C. Jewett · 2025 · Nature Communications
  29. Cell-penetrating peptides in protein mimicry and cancer therapeutics
    Matjaž Zorko, Sarah Jones, Ülo Langel · 2021 · Advanced Drug Delivery Reviews
  30. UniDL4BioPep: a universal deep learning architecture for binary classification in peptide bioactivity
    Zhenjiao Du, Xingjian Ding, Yixiang Xu, Yonghui Li · 2023 · Briefings in Bioinformatics
  31. Accurate protein function prediction via graph attention networks with predicted structure information
    Ben Lai, Jinbo Xu · 2021 · Briefings in Bioinformatics
  32. Light attention predicts protein location from the language of life
    Stärk, Hannes, Dallago, Christian, Heinzinger, Michael, Rost, Burkhard · 2021 · Bioinformatics Advances
  33. Improving deep learning protein monomer and complex structure prediction using DeepMSA2 with huge metagenomics data
    Wei Hui Zheng, Qiqige Wuyun, Yang Li, Chengxin Zhang, Lydia Freddolino, Yang Zhang · 2024 · Nature Methods
  34. Recent Progress in the Discovery and Design of Antimicrobial Peptides Using Traditional Machine Learning and Deep Learning
    Jielu Yan, Jianxiu Cai, Bob Zhang, Yapeng Wang, Derek F. Wong, Shirley W. I. Siu · 2022 · Antibiotics
  35. Toward universal cell embeddings: integrating single-cell RNA-seq datasets across species with SATURN
    Yanay Rosen, Maria Brbić, Yusuf Roohani, Kyle Swanson, Ziang Li, Jure Leskovec · 2024 · Nature Methods
  36. RiNALMo: general-purpose RNA language models can generalize well on structure prediction tasks
    Rafael Josip Penić, Tin Vlašić, Roland G. Huber, Yue Wan, Mile Šikić · 2025 · Nature Communications
  37. FLIP: Benchmark tasks in fitness landscape inference for proteins
    Christian Dallago, Jody Mou, Kadina E. Johnston, Bruce J. Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, Kevin Yang · 2021 · bioRxiv (Cold Spring Harbor Laboratory)
  38. Protein language models learn evolutionary statistics of interacting sequence motifs
    Zhang, Zhidian, Wayment-Steele, Hannah K., Brixi, Garyk, Wang, Haobo, Kern, Dorothee, Ovchinnikov, Sergey · 2024 · Proceedings of the National Academy of Sciences
  39. Development and use of machine learning algorithms in vaccine target selection
    Barbara Bravi · 2024 · npj Vaccines
  40. iAMPCN: a deep-learning approach for identifying antimicrobial peptides and their functional activities
    Jing Xu, Fuyi Li, Chen Li, Xudong Guo, Cornelia B. Landersdorfer, Hsin‐Hui Shen, Anton Yariv Peleg, Jian Li, Seiya Imoto, Jianhua Yao et al. · 2023 · Briefings in Bioinformatics
  41. Scientific Large Language Models: A Survey on Biological & Chemical Domains
    Qiang Zhang, Keyan Ding, Tianwen Lv, Xinda Wang, Qingyu Yin, Yiwen Zhang, Yu Jing, Yuhao Wang, Xiaotong Li, Zhuoyi Xiang et al. · 2025 · ACM Computing Surveys
  42. Protein function prediction as approximate semantic entailment
    Kulmanov, Maxat, Guzmán-Vega, Francisco J., Duek Roggli, Paula, Lane, Lydie, Arold, Stefan T., Hoehndorf, Robert · 2024 · Nature Machine Intelligence
  43. From sequence to function through structure: Deep learning for protein design
    2022
  44. Unsupervised protein embeddings outperform hand-crafted sequence and structure features at predicting molecular function
    Amelia Villegas-Morcillo, Stavros Makrodimitris, Roeland C. H. J. van Ham, Angel Manuel Gomez, Victoria Sanchez, Marcel J. T. Reinders · 2020 · Bioinformatics
  45. Toward the solution of the protein structure prediction problem
    Robin Pearce, Yang Zhang · 2021 · Journal of Biological Chemistry
  46. Transformer models in biomedicine
    Sumit Madan, Manuel Lentzen, Johannes Brandt, Daniel Rueckert, Martin Hofmann‐Apitius, Holger Fröhlich · 2024 · BMC Medical Informatics and Decision Making
  47. Machine learning-enabled retrobiosynthesis of molecules
    Tianhao Yu, Aashutosh Girish Boob, Michael Volk, Xuan Liu, Haiyang Cui, Huimin Zhao · 2023 · Nature Catalysis
  48. Functional annotation of enzyme-encoding genes using deep learning with transformer layers
    Gi Bae Kim, Ji Yeon Kim, Jong-An Lee, C. J. Norsigian, B. Palsson, Sang Yup Lee · 2023 · Nature Communications
  49. Machine learning modeling of family wide enzyme-substrate specificity screens
    Goldman, Samuel, Das, Ria, Yang, Kevin K., Coley, Connor W. · 2022 · PLOS Computational Biology
  50. Genomic language models: opportunities and challenges
    Gonzalo Benegas, Chengzhong Ye, Carlos Albors, Jianan Canal Li, Yun S. Song · 2025 · Trends in Genetics
  51. A generalized platform for artificial intelligence-powered autonomous enzyme engineering
    Nilmani Singh, Stephan Lane, Tianhao Yu, Jingxia Lu, Adrianna Ramos, Haiyang Cui, Huimin Zhao · 2025 · Nature Communications
  52. Designing antimicrobial peptides using deep learning and molecular dynamic simulations
    Qiushi Cao, Cheng Ge, Xuejie Wang, Peta J. Harvey, Zixuan Zhang, Yuan Ma, Xianghong Wang, Xinying Jia, Mehdi Mobli, David J. Craik et al. · 2023 · Briefings in Bioinformatics
  53. Sequence-to-sequence translation from mass spectra to peptides with a transformer model
    Melih Yilmaz, William E. Fondrie, Wout Bittremieux, Carlo Melendez, Rowan Nelson, Varun Ananth, Sewoong Oh, William Stafford Noble · 2024 · Nature Communications
  54. A method for multiple-sequence-alignment-free protein structure prediction using a protein language model
    Fang, Xiaomin, Wang, Fan, Liu, Lihang, He, Jingzhou, Lin, Dayong, Xiang, Yingfei, Zhu, Kunrui, Zhang, Xiaonan, Wu, Hua, Li, Hui et al. · 2023 · Nature Machine Intelligence
  55. A DNA language model based on multispecies alignment predicts the effects of genome-wide variants
    Gonzalo Benegas, Carlos Albors, Alan J. Aw, Chengzhong Ye, Yun S. Song · 2025 · Nature Biotechnology
  56. Deep generative modeling for protein design
    Alexey V. Strokach, Philip M. Kim · 2021 · Current Opinion in Structural Biology
  57. Disease diagnostics using machine learning of B cell and T cell receptor sequences
    Maxim Zaslavsky, Erin Craig, Jackson Michuda, Nidhi Sehgal, Nikhil Ram-Mohan, Ji-Yeun Lee, Khoa D. Nguyen, Ramona A. Hoh, Tho D. Pham, Katharina Röltgen et al. · 2025 · Science
  58. Progress and challenges for the machine learning-based design of fit-for-purpose monoclonal antibodies
    Rahmad Akbar, Habib Bashour, Puneet Rawat, Philippe Auguste Robert, Eva Smorodina, Tudor‐Stefan Cotet, Karine Flem Karlsen, Robert Frank, Brij Bhushan Mehta, Mai Ha Vu et al. · 2022 · mAbs
  59. NetSolP: predicting protein solubility in Escherichia coli using language models
    Vineet Thumuluri, Hannah-Marie Martiny, José Juan Almagro Armenteros, Jesper Salomon, Henrik Nielsen, Alexander Rosenberg Johansen · 2021 · Bioinformatics
  60. Amino acid encoding for deep learning applications
    Hesham ElAbd, Yana Bromberg, Adrienne Hoarfrost, Tobias L. Lenz, André Franke, Mareike Wendorff · 2020 · BMC Bioinformatics
  61. A foundational large language model for edible plant genomes
    Javier Mendoza‐Revilla, Evan Trop, Liam Gonzalez, Maša Roller, Hugo Dalla-Torre, Bernardo P. de Almeida, Guillaume Richard, Jonathan Caton, Nicolás López Carranza, Marcin J. Skwark et al. · 2024 · Communications Biology
  62. CatPred: a comprehensive framework for deep learning in vitro enzyme kinetic parameters
    Veda Sheersh Boorla, Costas D. Maranas · 2025 · Nature Communications
  63. Accelerating Biocatalysis Discovery with Machine Learning: A Paradigm Shift in Enzyme Engineering, Discovery, and Design
    Braun Markus, Gruber Christian C, Krassnigg Andreas, Arkadij Kummer, Lutz Stefan, Gustav Oberdorfer, Siirola Elina, Radka Šnajdrová · 2023 · ACS Catalysis
  64. A new age in protein design empowered by deep learning
    Hamed Khakzad, Ilia Igashov, Arne Schneuing, Casper A. Goverde, Michael M. Bronstein, Bruno E. Correia · 2023 · Cell Systems
  65. Contrastive learning on protein embeddings enlightens midnight zone
    2022
  66. TMbed: transmembrane proteins predicted through language model embeddings
    Michael Bernhofer, Burkhard Rost · 2022 · BMC Bioinformatics
  67. ProtAgents: protein discovery via large language model multi-agent collaborations combining physics and machine learning
    Alireza Ghafarollahi, Markus J. Buehler · 2024 · Digital Discovery
  68. Convolutions are competitive with transformers for protein sequence pretraining
    Yang, Kevin K., Fusi, Nicolo, Lu, Alex X. · 2024 · Cell Systems
  69. xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein
    Bo Chen, Xingyi Cheng, Pan Li, Yangli-Ao Geng, Jing Gong, Shen Li, Zhi-Lei Bei, Xu Tan, Boyan Wang, Xin Zeng et al. · 2024 · bioRxiv
  70. Machine learning to navigate fitness landscapes for protein engineering
    Chase R. Freschlin, Sarah A. Fahlberg, Philip A. Romero · 2022 · Current Opinion in Biotechnology
  71. Microbial Enzyme Biotechnology to Reach Plastic Waste Circularity: Current Status, Problems and Perspectives
    Marco Orlando, Gianluca Molla, Pietro Castellani, Valentina Pirillo, Vincenzo Torretta, Navarro Ferronato · 2023 · International Journal of Molecular Sciences
  72. GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT
    Yiqun T. Chen, James Zou · 2023 · bioRxiv (Cold Spring Harbor Laboratory)
  73. Multiple sequence alignment-based RNA language model and its application to structural inference
    Zhang, Yikun, Lang, Mei, Jiang, Jiuhong, Gao, Zhiqiang, Xu, Fan, Litfin, Thomas, Chen, Ke, Singh, Jaswinder, Huang, Xiansong, Song, Guoli et al. · 2024 · Nucleic Acids Research
  74. Learned Embeddings from Deep Learning to Visualize and Predict Protein Sets
    Christian Dallago, Konstantin Schütze, Michael Heinzinger, Tobias Olenyi, Maria Littmann, Amy X. Lu, Kevin Yang, Seonwoo Min, Sungroh Yoon, James T. Morton et al. · 2021 · Current Protocols
  75. Glycoinformatics in the Artificial Intelligence Era
    Daniel Bojar, Frédérique Lisacek · 2022 · Chemical Reviews
  76. Differentiable biology: using deep learning for biophysics-based and data-driven modeling of molecular mechanisms
    AlQuraishi, Mohammed, Sorger, Peter K. · 2021 · Nature Methods
  77. Predicting functional effect of missense variants using graph attention neural networks
    Zhang, Haicang, Xu, Michelle S., Fan, Xiao, Chung, Wendy K., Shen, Yufeng · 2022 · Nature Machine Intelligence
  78. A novel antibacterial peptide recognition algorithm based on BERT
    Yue Zhang, Jianyuan Lin, Lianmin Zhao, Xiangxiang Zeng, Xiangrong Liu · 2021 · Briefings in Bioinformatics
  79. PLASMe: a tool to identify PLASMid contigs from short-read assemblies using transformer
    Xubo Tang, Jiayu Shang, Yongxin Ji, Yanni Sun · 2023 · Nucleic Acids Research
  80. PLMSearch: Protein language model powers accurate and fast sequence search for remote homology
    Liu, Wei, Wang, Ziye, You, Ronghui, Xie, Chenghan, Wei, Hong, Xiong, Yi, Yang, Jianyi, Zhu, Shanfeng · 2024 · Nature Communications
  81. SignalP: The Evolution of a Web Server
    Henrik Nielsen, Felix Teufel, Søren Brunak, Gunnar von HEIJNE · 2024 · Methods in molecular biology
  82. Integrating protein language models and automatic biofoundry for enhanced protein evolution
    Qiang Zhang, Wanyi Chen, Ming Qin, Yuhao Wang, Zhongji Pu, Keyan Ding, Yuyue Liu, Qunfeng Zhang, Dongfang Li, Xinjia Li et al. · 2025 · Nature Communications
  83. Genomic language model predicts protein co-regulation and function
    Hwang, Yunha, Cornman, Andre L., Kellogg, Elizabeth H., Ovchinnikov, Sergey, Girguis, Peter R. · 2024 · Nature Communications
  84. BERT-Kcr: prediction of lysine crotonylation sites by a transfer learning method with pre-trained BERT models
    Yanhua Qiao, Xiaolei Zhu, Haipeng Gong · 2021 · Bioinformatics
  85. Fast and accurate protein function prediction from sequence through pretrained language model and homology-based label diffusion
    Qianmu Yuan, Junjie Xie, Jiancong Xie, Huiying Zhao, Yuedong Yang · 2023 · Briefings in Bioinformatics
  86. pLM4ACE: A protein language model based predictor for antihypertensive peptide screening
    Zhenjiao Du, Xingjian Ding, William W.Y. Hsu, Arslan Munir, Yixiang Xu, Yonghui Li · 2023 · Food Chemistry
  87. Large-scale chemoproteomics expedites ligand discovery and predicts ligand behavior in cells
    Fabian Offensperger, Gary Tin, Miquel Duran‐Frigola, Elisa Hahn, Sarah Dobner, Christopher W. am Ende, Joseph W. Strohbach, Andrea Rukavina, Vincenth Brennsteiner, Kevin Ogilvie et al. · 2024 · Science
  88. A Comprehensive Survey of Foundation Models in Medicine
    Wasif Khan, Seowung Leem, Kyle B. See, Joshua K. Wong, Shaoting Zhang, Ruogu Fang · 2025 · IEEE Reviews in Biomedical Engineering
  89. Evolutionary origin and functional diversification of aminotransferases
    Kaan Koper, Sang‐Woo Han, Delia Casas‐Pastor, Yasuo Yoshikuni, Hiroshi A. Maeda · 2022 · Journal of Biological Chemistry
  90. Deep learning for the PSIPRED Protein Analysis Workbench
    Daniel W. A. Buchan, Lewis Moffat, Andy M C Lau, Shaun M. Kandathil, David T. Jones · 2024 · Nucleic Acids Research
  91. Masked inverse folding with sequence transfer for protein representation learning
    2022
  92. Transforming the Language of Life: Transformer Neural Networks for Protein Prediction Tasks
    Ananthan Nambiar, Maeve Heflin, Simon Liu, S. Maslov, Mark Hopkins, Anna M. Ritz · 2020 · bioRxiv
  93. The genome editing revolution
    John van der OOST, Constantinos Patinios · 2023 · Trends in biotechnology
  94. Large language models improve annotation of prokaryotic viral proteins
    Flamholz, Zachary N., Biller, Steven J., Kelly, Libusha · 2024 · Nature Microbiology
  95. Large language models and their applications in bioinformatics
    Oluwafemi Abimbola Sarumi, Dominik Heider · 2024 · Computational and Structural Biotechnology Journal
  96. NetGO 3.0: Protein Language Model Improves Large-Scale Functional Annotations
    Shaojun Wang, Ronghui You, Yunjia Liu, Yi Xiong, Shanfeng Zhu · 2023 · Genomics Proteomics & Bioinformatics
  97. Signal Peptides Generated by Attention-Based Neural Networks
    Zachary Wu, Kevin Yang, Michael J. Liszka, Alycia Lee, Alina Batzilla, David G. Wernick, David P. Weiner, Frances H. Arnold · 2020 · ACS Synthetic Biology
  98. AI-driven protein design
    Huan Yee Koh, Yizhen Zheng, Madeleine Yang, Rohit Arora, Geoffrey I. Webb, Shirui Pan, Li Li, George McDonald Church · 2025 · Nature Reviews Bioengineering
  99. Democratizing protein language models with parameter-efficient fine-tuning
    Sledzieski, Samuel, Kshirsagar, Meghana, Baek, Minkyung, Dodhia, Rahul, Lavista Ferres, Juan, Berger, Bonnie · 2024 · Proceedings of the National Academy of Sciences
  100. Illuminating protein space with a programmable generative model
    Ingraham, John, Baranov, Max, Costello, Zak, Frappier, Vincent, Ismail, Ahmed, Tie, Shan, Wang, Wujie, Xue, Vincent, Obermeyer, Fritz, Beam, Andrew et al. · 2022