The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Transformer protein language models are unsupervised structure learners
Open accessAbstract
A bstract: Unsupervised contact prediction is central to uncovering physical, structural, and functional constraints for protein structure determination and design. For decades, the predominant approach has been to infer evolutionary constraints from a set of related sequences. In the past year, protein language models have emerged as a potential alternative, but performance has fallen short of state-of-the-art approaches in bioinformatics. In this paper we demonstrate that Transformer attention maps learn contacts from the unsupervised language modeling objective. We find the highest capacity models that have been trained to date already outperform a state-of-the-art unsupervised contact prediction pipeline, suggesting these pipelines can be replaced with a single forward pass of an end-to-end model. 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Among many successful neural network architectures, attention-based networks are extremely powerful for modeling complicated interactions between entities, making them suitable for modeling proteins whose functions are largely determined by interactions between amino acids (34–37).”
From Deep learning guided optimization of human antibody against SARS-CoV-2 variants with broad neutralization (2022) · cites this paper as [37]
In the citing paper
“Since then, various language model-based protein embedding methods have been applied to these and other protein prediction problems through transfer learning, including protein phenotype prediction 28,32-34, residue-residue contact prediction 32,70, fold recognition 33, protein-protein 71,72 and protein-drug interaction prediction 35,73.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [70]
In the citing paper
“Recent works have shown that increasing language model scale leads to continued improvements in downstream applications, such as residue-residue contact prediction 70.”
From Learning the protein language: Evolution, structure, and function (2021) · cites this paper as [70]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“A similar scaling law has been observed previously for supervised secondary structure prediction (Rives et al., 2019), and parallels observations in the NLP community (Kaplan et al., 2020; Brown et al., 2020).”
BackgroundmixOmics: An R package for ‘omics feature selection and multiple data integration
Rohart, F., Gautier, B., Singh, A., & Lê Cao, K.-A. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005752 · Cited by 4,056
Improved protein structure prediction using potentials from deep learning
Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1923-7 · Cited by 3,378
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Self-citationRives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- 10.1002/(SICI)1097-0134(19990301)34:4(508::AID-PROT10)3.0.CO;2-4 · DOI
- Robert D. Finn , Alex Bateman , Jody Clements , Penelope Coggill , Ruth Y. Eberhardt , Sean R. Eddy , Andreas Heger , Kirstie Hetherington , Liisa Holm , Jaina Mistry , Erik L.L. Sonnhammer , John Tate , and Marco Punta . Pfam: The protein families database, 1 2014. ISSN 03051048. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3965110/.
- 10.1101/622803 · DOI · Paper page
- 10.1101/2020.10.12.336859 · DOI
Cited by
Sorted by citations
Items 301–342 of 342
HieVi: Protein Large Language Model for proteome-based phage clusteringPreprint
Panigrahi, S., Ansaldi, M., & Ginet, N. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.12.17.627486
An unsupervised framework for comparing SARS-CoV-2 protein sequences using LLMsPreprint
Littlefield, S. B., & Campbell, R. H. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.12.16.628708
Abeer, A. N. M. N., Boroumand, M., Sermadiras, I., Caldwell, J. G., Stanev, V., Mody, N., Kaplan, G., Savery, J., Croasdale-Wood, R., & Pouryahya, M. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.02.02.636157
Semantical and Geometrical Protein Encoding Toward Enhanced Bioactivity and ThermostabilityPreprint
Tan, Y., Zhong, B., Zheng, L., Fan, G., & Hong, L. (2025) · eLife · doi:10.7554/elife.98033.3 · Cited by 1
Drug-Target Affinity Prediction Based on Graph Representation and Attention Fusion Mechanism
Jiang, L., Ye, W., Wei, L., Guo, W., & Hu, L. (2025) · Scientific Journal of Technology · doi:10.54691/pz2qge05
Do Protein Transformers Have Biological Intelligence?
Lin, F., Du, W., Liu, J., Milon, T. I., Meche, S., Xu, W., Qin, X., & Yuan, X. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06118-8_22
Rhodes, E. R., Scarabelli, G., Jou, J., Byerly, J., Sprenger, K. G., & Oloo, E. O. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.10.28.683791
GUANinE v1.1 reveals complementarity of supervised and genomic language models
Robson, E. S., & Ioannidis, N. M. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag427 · Cited by 1
AI-Driven Insights into Protein Sequence Analysis Landscape
Asim, M. N., Ahmed, S., & Dengel, A. (2025) · Springer eBooks · doi:10.1007/978-3-031-90454-7_5
SenSeqNet : A Deep Learning Framework for Cellular Senescence Detection From Protein Sequences
Jiang, H., Deng, D., Yuan, Y. L., Ren, J., Yang, X., Liu, S., Tan, B., Lin, L., & Lubin, L. I. U. (2025) · Aging Cell · doi:10.1111/acel.70344 · Cited by 1
Murakami, T., Hashidate, Y., & Matsunaga, Y. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2025.12.22.695878
Hou, R., Jiang, X., & Chu, M. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-14929-9_9
Genetic variation shapes human mRNA translation and disease riskPreprint
Wang, S., Chen, C., Xiao, X. G., & Wang, Z. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.10.705206
Guan, J. S., Wang, Z., & Mu, Y. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.22.707318
Rapid sequence-based screening of structure-disrupting protein mutationsPreprint
Oh, J., Qian, X., & Yoon, B.-J. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.02.24.707693
Orlov, A. V., Makus, Y. V., Ashniev, G. A., Orlova, N. N., & Nikitin, P. I. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.03.04.709491 · Cited by 2
From implicit learning to explicit structural constraints for efficient protein language modelingPreprint
Shen, L., Chao, L., Liu, T., Liu, Q., Zhou, G., Wang, H., Dong, X., Li, T., Zhang, X., & Ni, J. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.03.17.712034
From sequences to knowledge, improving and learning from sequence alignments
Blassel, L. (2022) · theses.fr (ABES) · doi:10.70675/edb2db56z9b8ez4f44zb76az28681bf8bc51
Structural prediction of protein assemblies with deep learning
Bret, H. (2023) · theses.fr (ABES) · doi:10.70675/b3849091zce01z4ab0z91efz2b71d9cc754d
Hamash, M., Khan, M. R., & Tiernan, P. (2026) · AI Computer Science and Robotics Technology · doi:10.5772/acrt20250217 · Cited by 7
Lee, U., Park, M., Song, B., & Ha, N. (2026) · Computational and Structural Biotechnology Journal · doi:10.34133/csbj.0073
Advanced Topics for Adopting Cloud and AI for Life Sciences Research
Wang, Z., Sannyasi, A., & Jiang, J. (2026) · Springer eBooks · doi:10.1007/978-3-032-14222-1_5
Transformer with BPE Tokenization for Analysis of Interactions of Chemical Substances and Proteins
Zoziuk, M. O., Krysenko, P., Dovgiy, S., MAKAROV, V. L., Yakymenko, Y., & Koroliouk, D. (2026) · doi:10.1002/9781394454518.ch9
Liu, Z., & Zhang, R. (2026) · Molecular Diversity · doi:10.1007/s11030-026-11557-8
Rahimzadeh, F., Khanli, L. M., Salehpour, P., & Golabi, F. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132559 · Cited by 1
GenNA: Conditional generation of nucleotide sequences guided by natural-language annotationsPreprint
Shen, Y., Cao, G., Wu, J., Chen, D., Feng, C., & Chen, M. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.04.22.720063
Moya‐García, A. A. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.04.24.720732
RNA Large Language Models in Virology
Shen, W., Zhu, L., Li, Y., & Huang, T. (2026) · Methods in molecular biology · doi:10.1007/978-1-0716-5264-0_17
Qurat-ul-ain, Outeiral, C., Cagiada, M., Teh, Y. W., & Deane, C. M. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.28.728196 · Cited by 1
Function-guided design of active enzymesPreprint
Hu, M., Wu, L., Yang, Y., Li, F., & Zhu, L. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.27.735025
Accelerated De-Novo Generation of Protein Ligand Molecules With Generative Transformers
Nath, S., Acharya, A., & Datta, D. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-7016-3.ch004
Timofeev, A., Bratchikov, A., & Anufriev, A. (2026) · Physchem · doi:10.3390/physchem6030056
TEDlm: domain-centric protein language models with optional structural pre-trainingPreprint
Wei, T., Kandathil, S. M., Buchan, D. W. A., & Jones, D. T. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.07.09.737428
Xiao, J., Wang, W. B., Ma, Y. B., Dong, Z. Y., Li, J., Su, J. G., & Li, J. (2026) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.6c00754
AUKAT: Conditional VAE-Driven Augmentation and Neural Modeling of Enzyme Turnover Numbers
Liu, M., Ni, X., & Bryliński, M. (2026) · Biomolecules · doi:10.3390/biom16071049
Evolutionary Reasoning Does Not Arise in Standard Usage of Protein Language Models
Ektefaie, Y., Shen, A., Jain, L., Farhat, M., & Žitnik, M. (2025) · neural information processing systems · doi:10.52202/085713-4356 · Cited by 5
Which chemical features are captured by ChemBERTa’s attention?
Alavi, S. H., Zare‐Mirakabad, F., & Gharaghani, S. (2026) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-026-00889-0
Alhadethy, N. H., Hashim, A. N., & AL-Rashid, S. Z. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-25568-6_6
Scaling Biomedical Text-Mining: Transformers, GenAI, and Drug Discovery
Agostinho, S. P., Santos, C., Ramalhão, D., Moreira, I. S., & Rosário‐Ferreira, N. (2026) · Methods in molecular biology · doi:10.1007/978-1-0716-5408-8_4
Rodriguez, A. V. C. (2026) · Crop Breeding Genetics and Genomics · doi:10.20900/cbgg20260020
Language-Model-Based Detection of Genetic Editing in BacteriaPreprint
Gabay, E., & Burstein, D. E. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.09.17.751846
Introduction to the principles of protein engineering
Sharp, J. F., Nelson, M. K., Rustembek, M., Samanta, R., & Stern, L. A. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-44536-1.00010-9
Metadata
- Authors
- Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A.
- Affiliations
- University of California, Berkeley (US)
- Meta (United States) (US)
- Harvard University (US)
- Corresponding author
- Rao, Roshan
- Meier, Joshua
- Sercu, Tom
- Ovchinnikov, Sergey
- Rives, Alexander
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2020-12-15
- OpenAlex
- W3111174583
- Cited by
- 341
- References
- 63
- Citations by year
- 2026: 51 · 2025: 67 · 2024: 79 · 2023: 56 · 2022: 49 · 2021: 37 · 2020: 2
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- Funders
- National Institutes of Health (DP5OD026389)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
- Keywords
- protein language models, protein structure prediction, protein structure design, evolutionary constraints
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
