The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Transformer protein language models are unsupervised structure learners
Open accessAbstract
A bstract: Unsupervised contact prediction is central to uncovering physical, structural, and functional constraints for protein structure determination and design. For decades, the predominant approach has been to infer evolutionary constraints from a set of related sequences. In the past year, protein language models have emerged as a potential alternative, but performance has fallen short of state-of-the-art approaches in bioinformatics. In this paper we demonstrate that Transformer attention maps learn contacts from the unsupervised language modeling objective. We find the highest capacity models that have been trained to date already outperform a state-of-the-art unsupervised contact prediction pipeline, suggesting these pipelines can be replaced with a single forward pass of an end-to-end model. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Among many successful neural network architectures, attention-based networks are extremely powerful for modeling complicated interactions between entities, making them suitable for modeling proteins whose functions are largely determined by interactions between amino acids (34–37).”
From Deep learning guided optimization of human antibody against SARS-CoV-2 variants with broad neutralization (2022) · cites this paper as [37]
In the citing paper
“Since then, various language model-based protein embedding methods have been applied to these and other protein prediction problems through transfer learning, including protein phenotype prediction 28,32-34, residue-residue contact prediction 32,70, fold recognition 33, protein-protein 71,72 and protein-drug interaction prediction 35,73.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [70]
In the citing paper
“Recent works have shown that increasing language model scale leads to continued improvements in downstream applications, such as residue-residue contact prediction 70.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [70]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“A similar scaling law has been observed previously for supervised secondary structure prediction (Rives et al., 2019), and parallels observations in the NLP community (Kaplan et al., 2020; Brown et al., 2020).”
BackgroundmixOmics: An R package for ‘omics feature selection and multiple data integration
Rohart, F., Gautier, B., Singh, A., & Lê Cao, K.-A. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005752 · Cited by 4,056
Improved protein structure prediction using potentials from deep learning
Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1923-7 · Cited by 3,378
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Self-citationRives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- 10.1002/(SICI)1097-0134(19990301)34:4(508::AID-PROT10)3.0.CO;2-4 · DOI
- Robert D. Finn , Alex Bateman , Jody Clements , Penelope Coggill , Ruth Y. Eberhardt , Sean R. Eddy , Andreas Heger , Kirstie Hetherington , Liisa Holm , Jaina Mistry , Erik L.L. Sonnhammer , John Tate , and Marco Punta . Pfam: The protein families database, 1 2014. ISSN 03051048. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3965110/.
- 10.1101/622803 · DOI · Paper page
- 10.1101/2020.10.12.336859 · DOI
Cited by
Sorted by citations
Items 101–200 of 342
Adapting protein language models for structure-conditioned designPreprint
Ruffolo, J. A., Bhatnagar, A., Beazer, J., Nayfach, S., Russ, J., Hill, E., Hussain, R., Gallagher, J. P., & Madani, A. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.08.03.606485 · Cited by 24
From Proteins to Ligands: Decoding Deep Learning Methods for Binding Affinity Prediction
Gorantla, R., Kubincová, A., Weiße, A. Y., & Mey, A. S. J. S. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c01208 · Cited by 24
Li, Y., Zhang, C., Zheng, W. H., Zhou, X., Bell, E. W., Yu, D., & Zhang, Y. (2021) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.26211 · Cited by 24
Structure-inducing pre-training
McDermott, M. B. A., Yap, B., Szolovits, P., & Žitnik, M. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00647-z · Cited by 29
Adams, E., Bai, L., Lee, M., Yu, Y., & AlQuraishi, M. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.02.06.636901 · Cited by 23
Wang, M., Patsenker, J., Li, H., Kluger, Y., & Kleinstein, S. H. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012153 · Cited by 22
Deep embeddings to comprehend and visualize microbiome protein space
Odrzywolek, K., Karwowska, Z., Majta, J., Byrski, A., Milanowska-Zabel, K., & Kosciolek, T. (2022) · Scientific Reports · doi:10.1038/s41598-022-14055-7 · Cited by 22
Protein and genomic language models uncover the unexplored diversity of bacterial immunity
Mordret, E., Hervé, A., Tesson, F., Vaysset, H., Clabby, T., Loubat, A., Shomar, H., Planel, R., Lavenir, R., Cury, J., et al. (2026) · Science · doi:10.1126/science.adv8275 · Cited by 36
Zhao, Y., Su, X., Zhang, W., Mai, S., Xu, Z., Qin, C., Yu, R., He, B., & Yao, J. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad191 · Cited by 23
PredictProtein – Predicting Protein Structure and Function for 29 YearsPreprint
Bernhofer, M., Dallago, C., Karl, T., Satagopam, V., Heinzinger, M., Littmann, M., Olenyi, T., Qiu, J., Schütze, K., Yachdav, G., et al. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.02.23.432527 · Cited by 21
Protein Language Models: Applications and Perspectives
Leclercq, M., & Droit, A. (2025) · Journal of Proteome Research · doi:10.1021/acs.jproteome.5c00506 · Cited by 29
Protein language models trained on biophysical dynamics inform mutation effects
Hou, C., Zhao, H., & Shen, Y. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2530466123 · Cited by 22
PITHIA: Protein Interaction Site Prediction Using Multiple Sequence Alignments and Attention
Hosseini, S., & Ilie, L. (2022) · International Journal of Molecular Sciences · doi:10.3390/ijms232112814 · Cited by 20
A-Prot: protein structure modeling using MSA transformer
Hong, Y., Lee, J., & Ko, J. (2022) · BMC Bioinformatics · doi:10.1186/s12859-022-04628-8 · Cited by 19
Hu, W., & Ohue, M. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.01.022 · Cited by 25
Zhang, L., & Liu, T. (2024) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2024.136147 · Cited by 19
Design of Peptide-Based Protein Degraders via Contrastive Deep LearningPreprint
Palepu, K., Ponnapati, M. R., Bhat, S., Tysinger, E., Stan, T., Brixi, G., Koseki, S., & Chatterjee, P. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.05.23.493169 · Cited by 18
Protein Representation Learning via Knowledge Enhanced Primary Structure ModelingPreprint
Zhou, H.-Y., Fu, Y., Zhang, Z., Bian, C., & Yu, Y. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.26.525795 · Cited by 27
Li, S., & Xu, J. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103467 · Cited by 18
Advancing protein structure prediction beyond AlphaFold2
Park, S., Myung, S., & Baek, M. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.102985 · Cited by 18
Gamouh, H., Novotný, M., & Hoksza, D. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf431 · Cited by 23
Foundation models of protein sequences: A brief overview
Bjerregaard, A., Groth, P. M., Hauberg, S., Krogh, A., & Boomsma, W. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.103004 · Cited by 16
Decoding the interactions and functions of non-coding RNA with artificial intelligence
Jung, V., Vincent-Cuaz, C., Tumescheit, C., Fournier, L. M., Darsinou, M., Xu, Z. M., Saadat, A., Wang, Y., Tsantoulis, P. K., Michielin, O. A., et al. (2025) · Nature Reviews Molecular Cell Biology · doi:10.1038/s41580-025-00857-w · Cited by 18
Bioinfo-Bench: A Simple Benchmark Framework for LLM Bioinformatics Skills EvaluationPreprint
Chen, Q., & Deng, C. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.18.563023 · Cited by 16
Fast and accurate modeling and design of antibody-antigen complex using tFoldPreprint
Wu, F., Zhao, Y. S., Wu, J., Jiang, B., He, B., Huang, L.-K., Qin, C., Yang, F., Huang, N., Xiao, Y., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.02.05.578892 · Cited by 16
Talo, M., & Bozdag, S. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf183 · Cited by 16
Evotuning protocols for Transformer-based variant effect prediction on multi-domain proteins
Yamaguchi, H., & Saitō, Y. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab234 · Cited by 15
Attention-based Convolutional Neural Networks for Protein-Protein Interaction Site Prediction
Lu, S., Li, Y., Nan, X., & Zhang, S. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669435 · Cited by 15
Generating novel protein sequences using Gibbs sampling of masked language modelsPreprint
Johnson, S. R., Monaco, S., Massie, K., & Syed, Z. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.01.26.428322 · Cited by 18
Tubiana, J., Schneidman‐Duhovny, D., & Wolfson, H. J. (2022) · Journal of Molecular Biology · doi:10.1016/j.jmb.2022.167758 · Cited by 15
Language models for protein design
Lee, J. S., Abdin, O., & Kim, P. M. (2025) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2025.103027 · Cited by 15
Cuturello, F., Celoria, M., Ansuini, A., & Cazzaniga, A. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae447 · Cited by 11
Protein sequence modelling with Bayesian flow networks
Atkinson, T. J., Barrett, T. D., Cameron, S., Guloglu, B., Greenig, M., Tan, C. B., Robinson, L., Graves, A., Copoiu, L., & Laterre, A. (2025) · Nature Communications · doi:10.1038/s41467-025-58250-2 · Cited by 20
Tran, N. H., Xu, J., & Li, M. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab493 · Cited by 13
Mansoor, S., Baek, M., Madan, U., & Horvitz, E. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.09.01.458592 · Cited by 12
Language models enable zero-shot prediction of RNA secondary structures including pseudoknotsPreprint
Gong, T., & Bu, D. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.01.27.577533 · Cited by 14
Unsupervisedly Prompting AlphaFold2 for Accurate Few-Shot Protein Structure Prediction
Zhang, J., Liu, S., Chen, M., Chu, H., Wang, M., Wang, Z., Yu, J., Ni, N., Fan, Y., Chen, D., et al. (2023) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.3c00528 · Cited by 15
Cocco, S., Posani, L., & Monasson, R. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2312335121 · Cited by 14
Natural language processing approach to model the secretion signal of type III effectors
Wagner, N., Alburquerque, M., Ecker, N., Dotan, E., Zerah, B., Pena, M. M., Potnis, N., & Pupko, T. (2022) · Frontiers in Plant Science · doi:10.3389/fpls.2022.1024405 · Cited by 12
ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular ModelingPreprint
Zheng, K., Long, S., Lu, T., Yang, J., Dai, X., Zhang, M., Nie, Z., Ma, W., & Zhou, H. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.03.04.583284 · Cited by 20
TransConv: convolution-infused transformer for protein secondary structure prediction
Das, S., Ghosh, S., & Jana, N. D. (2025) · Journal of Molecular Modeling · doi:10.1007/s00894-024-06259-7 · Cited by 12
ProtRNA: A protein-derived RNA language model by cross-modality transfer learning
Zhang, R., Ma, B., Xu, G., & Ma, J. (2025) · Cell Systems · doi:10.1016/j.cels.2025.101371 · Cited by 12
Single-Sequence, Structure Free Allosteric Residue Prediction with Protein Language ModelsPreprint
Kannan, G. R., Hie, B., & Kim, P. S. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.03.616547 · Cited by 11
PRESCOTT: a population aware, epistatic, and structural model accurately predicts missense effects
Tekpinar, M., David, L., Henry, T., & Carbone, A. (2025) · Genome Biology · doi:10.1186/s13059-025-03581-y · Cited by 18
Efficient generation of epitope-targeted antibodies with Germinal
Mille, L. S., Driscoll, C. L., Wang, J. N., Dai, H., Widatalla, T., Zhang, J. L., Zhang, X., Rao, B., Feng, L., Hie, B., et al. (2026) · Nature Biotechnology · doi:10.1038/s41587-026-03187-0 · Cited by 40
Zhang, G., Wang, H., Zhang, Z., Zhang, L., Guo, G., Yang, J., Yuan, F., & Ju, F. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae319 · Cited by 12
Towards the prediction of general biomolecular interactions with AI
Baek, M. (2024) · Nature Methods · doi:10.1038/s41592-024-02350-2 · Cited by 10
Multi‐View Biomedical Foundation Models for Molecule‐Target and Property Prediction
Suryanarayanan, P., Qiu, Y., Sethi, S., Mahajan, D., Li, H., Yang, Y., Eyigöz, E., Guzmán-Sáenz, A., Platt, D. E., Rumbell, T. H., et al. (2026) · Advanced Science · doi:10.1002/advs.202517840 · Cited by 13
Asim, M. N., Asif, T., Hassan, F., & Dengel, A. (2025) · Database · doi:10.1093/database/baaf027 · Cited by 11
Dohan, D., Gane, A., Bileschi, M. L., Belanger, D., & Colwell, L. J. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467163 · Cited by 13
Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language ModelPreprint
He, L., Zhang, S., Wu, L., Xia, H., Ju, F., Zhang, H., Liu, S., Xia, Y., Zhu, J., Deng, P., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15527 · Cited by 35
Using Sequences of Life-events to Predict Human LivesPreprint
Savcisens, G., Eliassi‐Rad, T., Hansen, L. K., Mortensen, L. H., Lilleholt, L., Rogers, A., Zettler, I., & Lehmann, S. (2023) · Research Square · doi:10.21203/rs.3.rs-2975478/v1 · Cited by 10
Semantical and geometrical protein encoding toward enhanced bioactivity and thermostability
Tan, Y., Zhou, B., Zheng, L., Fan, G., & Hong, L. (2025) · eLife · doi:10.7554/elife.98033.4 · Cited by 20
Discovery of highly active kynureninases for cancer immunotherapy through protein language model
Eom, H., Park, S., Cho, K. S., Lee, J., Kim, H., Kim, S., Yang, J., Han, Y.-H., Lee, J., Seok, C., et al. (2025) · Nucleic Acids Research · doi:10.1093/nar/gkae1245 · Cited by 10
Hyena architecture enables fast and efficient protein language modeling
Zhang, Y., Bian, B., & Okumura, M. (2024) · iMetaOmics. · doi:10.1002/imo2.45 · Cited by 10
Tzavella, K., Díaz, A., Olsen, C., & Vranken, W. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae664 · Cited by 9
Predicting purification process fit of monoclonal antibodies using machine learning
Maier, A. J., Cha, M., Burgess, S., Wang, A., Cuellar, C., Kim, S. Y., Rajan, N. S., Neyyan, J., SENGUPTA, R., O’Connor, K., et al. (2025) · mAbs · doi:10.1080/19420862.2024.2439988 · Cited by 11
A comprehensive benchmarking for evaluating TCR embeddings in modeling TCR-epitope interactions
Feng, X., Huo, M., Li, H., Yang, Y., Jiang, Y., He, L., & Li, S. C. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf030 · Cited by 10
Cutting-edge deep-learning based tools for metagenomic research
Karin, E. L., & Steinegger, M. (2025) · National Science Review · doi:10.1093/nsr/nwaf056 · Cited by 10
Johnson, M. M., Sung, K., Haddox, H. K., Vora, A. A., Araki, T., Victora, G. D., Song, Y. S., Fukuyama, J., & Matsen, F. A., IV (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013758 · Cited by 11
Lombardi, G., & Carbone, A. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.08.24.609515 · Cited by 10
Wang, Y., Wang, Z., Sadeh, G., Zancato, L., Achille, A., Karypis, G., & Rangwala, H. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.29.620988 · Cited by 12
Pratyush, P., Pokharel, S., Ismail, H. D., Bahmani, S., & KC, D. B. (2024) · Methods in molecular biology · doi:10.1007/978-1-0716-4196-5_16 · Cited by 8
Alanazi, W. T., Meng, D., & Pollastri, G. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.04.005 · Cited by 9
A survey of downstream applications of evolutionary scale modeling protein language models
Yang, Q., Yu, J., & Zheng, J. (2025) · Quantitative Biology · doi:10.1002/qub2.70013 · Cited by 9
POOE: predicting oomycete effectors based on a pre-trained large protein language model
Zhao, M., Lei, C., Zhou, K., Huang, Y., Fu, C., Yang, S., & Zhang, Z. (2023) · mSystems · doi:10.1128/msystems.01004-23 · Cited by 9
Classifying alkaliphilic proteins using embeddings from protein language model
Susanty, M., Mursalim, M. K. N., Hertadi, R., Purwarianti, A., & Mengko, T. L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108385 · Cited by 10
Comprehensive Research on Druggable Proteins: From PSSM to Pre-Trained Language Models
Chu, H., & Liu, T. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms25084507 · Cited by 8
Improvements in viral gene annotation using large language models and soft alignments
Harrigan, W., Ferrell, B. D., Wommack, K. E., Polson, S. W., Schreiber, Z. D., & Belcaid, M. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05779-6 · Cited by 9
Accelerating Antibody Design with Active LearningPreprint
Seo, S.-W., Kwak, M. W., Kang, E., Kim, C., Park, E. Y., Kang, T. H., & Kim, J. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.09.12.507690 · Cited by 8
EvoNB: A protein language model-based workflow for nanobody mutation prediction and optimization
Xiong, D., Ming, Y., Li, Y., Li, S., Chen, K., Liu, J., Duan, L., Li, H., Li, M., & He, X. D. (2025) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2025.101260 · Cited by 7
ProtChatGPT: Towards Understanding Proteins with Hybrid Representation and Large Language Models
Wang, C., Fan, H., Quan, R., Yao, L., & Yang, Y. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730064 · Cited by 9
Susanty, M., Mursalim, M. K. N., Hertadi, R., Purwarianti, A., & Rajab, T. L. (2024) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2024.108163 · Cited by 10
An explainable few-shot learning model for the directed evolution of antimicrobial peptides
Gao, Q., Ge, L., Wang, Y., Zhu, Y., Liu, Y., Zhang, H., Huang, J., & Qin, Z. (2024) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2024.138272 · Cited by 8
Fishman, N., & Hancox-Li, L. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533206 · Cited by 13
DeepFusionGO: Protein function prediction by fusing heterogeneous features through deep learning
Huang, Z., Zheng, R., & Deng, L. (2022) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm55620.2022.9994899 · Cited by 7
fastMSA: Accelerating Multiple Sequence Alignment with Dense Retrieval on Protein LanguagePreprint
Hong, L., Sun, S., Zheng, L., Tan, Q., & Li, Y. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.12.20.473431 · Cited by 9
When Geometric Deep Learning Meets Pretrained Protein Language ModelsPreprint
Wu, F., Tao, Y., Radev, D., & Xu, J. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.05.522958 · Cited by 7
Unified Deep Learning of Molecular and Protein Language Representations with T5ProtChem
Kelly, T., Xia, S., Lü, J., & Zhang, Y. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5c00051 · Cited by 6
Improving prediction accuracy in chimeric proteins with windowed multiple sequence alignment
Vedula, S., Bronstein, A. M., & Marx, A. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.07.039 · Cited by 7
Predicting antibiotic resistance genes and bacterial phenotypes based on protein language models
Wang, B., Meng, R., Li, Z., Hu, M., Wang, X., Zhao, Y., Chai, Z., Jin, Y., Yue, J., Chen, W., et al. (2025) · Frontiers in Microbiology · doi:10.3389/fmicb.2025.1628952 · Cited by 9
PLMC: Language Model of Protein Sequences Enhances Protein Crystallization Prediction
Xiong, D., Kaicheng, U., Sun, J., & Cribbs, A. P. (2024) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-024-00639-6 · Cited by 6
learnMSA2: deep protein multiple alignments with large language and hidden Markov models
Becker, F., & Stanke, M. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae381 · Cited by 6
Subcellular Enrichment Patterns of New Genes in Drosophila Evolution
Dong, C., Xia, S., Zhang, L., Arsala, D., Fang, C., Tan, S., Clark, A. G., & Long, M. (2025) · Molecular Biology and Evolution · doi:10.1093/molbev/msaf038 · Cited by 5
PRO-LDM: Protein Sequence Generation with a Conditional Latent Diffusion ModelPreprint
Zhang, S., Jiang, Z., Huang, R., Mo, S., Zhu, L.-T., Li, P., Zhang, Z., Pan, E., Chen, X., Long, Y.-F., et al. (2024) · bioRxiv · doi:10.1101/2023.08.22.554145 · Cited by 11
Direct coupling analysis and the attention mechanism
(2025) · doi:10.1186/s12859-025-06062-y · Cited by 5
Towards the explainability of protein language models
Hunklinger, A., & Ferruz, N. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01232-w · Cited by 12
Clark, J. D., Mi, X., Mitchell, D. A., & Shukla, D. (2024) · Digital Discovery · doi:10.1039/d4dd00170b · Cited by 8
Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methods
Suyunu, B., Taylan, E., & Özgür, A. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822826 · Cited by 6
SST-ResNet: A Sequence and Structure Information Integration Model for Protein Property Prediction
Zhou, G., Zhao, Y., He, S., & Bo, X. (2025) · International Journal of Molecular Sciences · doi:10.3390/ijms26062783 · Cited by 5
AI-Empowered Genome Decoding: Applications of Large Language Models in Genomics
Li, S., Fan, W., & Zhou, Y. (2025) · Frontiers of digital education. · doi:10.1007/s44366-025-0051-1 · Cited by 4
The Fine-Tuning Journey of ChatGPT in Industry Applications
Mohapatra, H. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-8332-2.ch011 · Cited by 5
Li, Y., Lu, S., Ma, Q., Nan, X., & Zhang, S. (2023) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2023.3323493 · Cited by 5
Wang, X., Zhang, M., Yang, X., Yu, D., & Ge, F. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01999 · Cited by 5
Hu, Y., Tao, F., Xu, J., Lan, W.-J., Zhang, J., & Lan, W. (2025) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2025.101267 · Cited by 5
Kotowski, K., Fabian, P., Roterman, I., & Stąpor, K. (2022) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.26452 · Cited by 6
Hasan, A. K. M. M., Ahmed, A. Y., Mahbub, S., Rahman, M. S., & Bayzid, M. S. (2023) · Bioinformatics Advances · doi:10.1093/bioadv/vbad042 · Cited by 5
Distillation of MSA Embeddings to Folded Protein Structures with Graph TransformersPreprint
Costa, A. D. S., Ponnapati, M., Jacobson, J. M., & Chatterjee, P. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.06.02.446809 · Cited by 6
Metadata
- Authors
- Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A.
- Affiliations
- University of California, Berkeley (US)
- Meta (United States) (US)
- Harvard University (US)
- Corresponding author
- Rao, Roshan
- Meier, Joshua
- Sercu, Tom
- Ovchinnikov, Sergey
- Rives, Alexander
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2020-12-15
- OpenAlex
- W3111174583
- Cited by
- 341
- References
- 63
- Citations by year
- 2026: 51 · 2025: 67 · 2024: 79 · 2023: 56 · 2022: 49 · 2021: 37 · 2020: 2
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- Funders
- National Institutes of Health (DP5OD026389)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
- Keywords
- protein language models, protein structure prediction, protein structure design, evolutionary constraints
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
