Protein language models learn evolutionary statistics of interacting sequence motifs
Open accessPublished version of a preprint
The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Protein language models (pLMs) have emerged as potent tools for predicting and designing protein structure and function, and the degree to which these models fundamentally understand the inherent biophysics of protein structure stands as an open question. Motivated by a finding that pLM-based structure predictors erroneously predict nonphysical structures for protein isoforms, we investigated the nature of sequence context needed for contact predictions in the pLM Evolutionary Scale Modeling (ESM-2). We demonstrate by use of a “categorical Jacobian” calculation that ESM-2 stores statistics of coevolving residues, analogously to simpler modeling approaches like Markov Random Fields and Multivariate Gaussian models. We further investigated how ESM-2 “stores” information needed to predict contacts by comparing sequence masking strategies, and found that providing local windows of sequence information allowed ESM-2 to best recover predicted contacts. This suggests that pLMs predict contacts by storing motifs of pairwise contacts. Our investigation highlights the limitations of current pLMs and underscores the importance of understanding the underlying mechanisms of these models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

![Fig. 2. Deep learning structure-based methods predict isoforms as fragments of full-length structures with exposed aggregation-prone residues. (A) Top: alignment between human myoglobin (UniProt: P02144) and human isoform Q8WVH6 (UniProt: Q8WVH6). Left: X-ray structure of human myoglobin [Protein Data Bank (PDB): 3RGK] shown with the heme cofactor, and the two missing helices A and B in the isoform colored in gray. Right: predicted structures of isoform Q8WVH6 of Myoglobin from AlphaFold2, OmegaFold, and ESMFold have low root-mean-squared deviation (RMSD) to structure 3RGK (0.49, 1.01, and 0.81 Å respectively). Bottom: surfaces of protein fragments corresponding to the isoform sequence. The isoform structure models all have exposed hydrophobic residues corresponding to where helices A and B reside in the full-length structure (indicated with an arrow), quantified here using the spatial aggregation propensity (SAP) score (16). Structure predictions of isoforms from Human Prostaglandin E synthase 3 (B), Human Caspase-9 (C), and Human Nfs1 cysteine desulfurase (D) all share similar trends, where the isoform structure model contains a significant patch of residues with high SAP score. In (B–D), the structure model depicted is from ESMFold. (E) For 18 isoforms previously identified in the literature as isoforms where splicing events occur in structured domains, we calculated RMSD to a reference structure of the full-length protein, and the change in average SAP for the isoform fragment in comparison to the sequence aligned in the full-length protein. We found that for AlphaFold2, OmegaFold, and ESMFold, isoform structure models generally had low RMSD to the reference structure, predicted with relatively high pLDDT, along with increased average SAP.](https://figures.cochat.ai/paper-figures/c01ccd8ae5674e38be65c4416a7c1e7f/protein-language-models-learn-evolutionary-statistics-of-interacting-sequence-figure-2.jpg)

![Fig. 4. Categorical Jacobian is an unsupervised method to extract coevolutionary signal and uniformly evaluate any pLM. (A) Scheme of the categorical Jacobian calculation. Each residue in a sequence of length L is changed to A different types of amino acids, where A is the size of the alphabet (for proteins, A=20). By computing how the output changes with respect to the input, a matrix of size [L, A, L, A] is obtained. (B) This categorial Jacobian allows for comparing a nonlinear method like ESM-2 and a simple linear method, exemplified here for large ribosomal subunit protein RL29 (UniProt: P0A7M7). We can compare the coevolutionary weights obtained from a linear model, calculated using inverse covariance, and the categorical Jacobian calculated from ESM-2. (C) Contacts calculated from the categorical Jacobian from ESM-2 outperform the inverse covariance calculation from ref. 25 (Average long-range P@L/2 of 0.67 vs. 0.80, respectively). (D) Comparing contact accuracy from the categorical Jacobian and the supervised contact prediction head (Average long-range P@L/2 of 0.80 vs. 0.87, respectively). (E) Correlation between covariation parameters from linear model and ESM-2 Jacobians increase with model size. Top: Distribution of Spearman correlation coefficients between contacts from linear model and ESM-2 Jacobians. Bottom: average Spearman R, varying σ cutoff for linear model values close to zero. For (C–E), N = 1,431 proteins.](https://figures.cochat.ai/paper-figures/ad02f4b012a24748997166172511258f/protein-language-models-learn-evolutionary-statistics-of-interacting-sequence-figure-4.jpg)


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“We observed that the model uses information from one end to specify the other end across a distance of ~1 to 2 kb, reflecting the model’s understanding of the tight evolutionary linkage of the two terminal elements”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [70]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430
Kabsch, W., & Sander, C. (1983) · Biopolymers · doi:10.1002/bip.360221211 · Cited by 15,320
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2019) · Apollo - University of Cambridge Repository · doi:10.17863/cam.64134 · Cited by 10,460
ColabFold - Making protein folding accessible to all
Mirdita, M., Schütze, K., Moriwaki, Y., Heo, L., Ovchinnikov, S., & Steinegger, M. (2021) · doi:10.1101/2021.08.15.456425 · Cited by 9,391
Self-citationEvolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
Accurate prediction of protein structures and interactions using a 3-track network
Baek, M., DiMaio, F., Anishchenko, I., Dauparas, J., Ovchinnikov, S., Lee, G. R., Wang, J., Cong, Q., Kinch, L. N., Schaeffer, R. D., et al. (2021) · doi:10.1101/2021.06.14.448402 · Cited by 5,549
Self-citationTM-align: a protein structure alignment algorithm based on the TM-score
Zhang, Y. (2005) · Nucleic Acids Research · doi:10.1093/nar/gki524 · Cited by 3,774
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
HHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignment
Remmert, M., Biegert, A., Hauser, A., & Söding, J. (2012) · Nature Methods · doi:10.1038/nmeth.1818 · Cited by 2,288
RoFormer: Enhanced transformer with rotary position embedding
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2023.127063 · Cited by 1,957
RoFormer: Enhanced Transformer with Rotary Position Embedding
Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021) · arXiv · doi:10.48550/arxiv.2104.09864 · Cited by 1,952
Improved protein structure prediction using predicted interresidue orientations
Yang, J., Anishchenko, I., Park, H., Peng, Z., Ovchinnikov, S., & Baker, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914677117 · Cited by 1,421
Self-citationKamisetty, H., Ovchinnikov, S., & Baker, D. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1314045110 · Cited by 672
Self-citationImproved contact prediction in proteins: using pseudolikelihoods to infer Potts models
Ekeberg, M., Lövkvist, C., Lan, Y., Weigt, M., & Aurell, E. (2013) · Physical Review E · doi:10.1103/physreve.87.012707 · Cited by 661
Ovchinnikov, S., Kamisetty, H., & Baker, D. (2014) · eLife · doi:10.7554/elife.02030 · Cited by 629
Self-citationDesign of therapeutic proteins with enhanced stability
Chennamsetty, N., Voynov, V., Kayser, V., Helk, B., & Trout, B. L. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0904191106 · Cited by 598
Design of protein-binding proteins from the target structure alone
Cao, L., Coventry, B., Goreshnik, I., Huang, B., Sheffler, W., Park, J. S., Jude, K. M., Marković, I., Kadam, R. U., Verschueren, K. H. G., et al. (2022) · Nature · doi:10.1038/s41586-022-04654-9 · Cited by 577
Alternative splicing and genome complexity
Brett, D., Pospisil, H., Valcárcel, J., Reich, J., & Bork, P. (2002) · Nature Genetics · doi:10.1038/ng803 · Cited by 536
Dunn, S. D., Wahl, L. M., & Gloor, G. B. (2008) · Bioinformatics · doi:10.1093/bioinformatics/btm604 · Cited by 483
Single-sequence protein structure prediction using language models from deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Rochereau, C., Church, G. M., Sorger, P. K., & AlQuraishi, M. (2021) · doi:10.1101/2021.08.02.454840 · Cited by 455
Single-sequence protein structure prediction using a language model and deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01432-w · Cited by 455
High-resolution de novo structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392
Koropatkin, N. M., Martens, E. C., Gordon, J. I., & Smith, T. J. (2008) · Structure · doi:10.1016/j.str.2008.03.017 · Cited by 383
Learning generative models for protein fold families
Balakrishnan, S., Kamisetty, H., Carbonell, J. G., Lee, S., & Langmead, C. J. (2011) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.22934 · Cited by 316
Classen, S., Olland, S., & Berger, J. M. (2003) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1832879100 · Cited by 263
A vocabulary of ancient peptides at the origin of folded proteins
Alva, V., Söding, J., & Lupas, A. N. (2015) · eLife · doi:10.7554/elife.09410 · Cited by 234
Baldassi, C., Zamparo, M., Feinauer, C., Procaccini, A., Zecchina, R., Weigt, M., & Pagnani, A. (2014) · PLoS ONE · doi:10.1371/journal.pone.0092721 · Cited by 144
Alternative splicing and protein structure evolution
Birzele, F., Csaba, G., & Zimmer, R. (2008) · Nucleic Acids Research · doi:10.1093/nar/gkm1054 · Cited by 119
Efficient generative modeling of protein sequences using simple autoregressive models
Trinquier, J., Uguzzoni, G., Pagnani, A., Zamponi, F., & Weigt, M. (2021) · Nature Communications · doi:10.1038/s41467-021-25756-4 · Cited by 92
Nepomnyachiy, S., Ben-Tal, N., & Kolodny, R. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1707642114 · Cited by 77
Tertiary alphabet for the observable protein structural universe
Mackenzie, C. O., Zhou, J., & Grigoryan, G. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1607178113 · Cited by 72
Proteomics studies confirm the presence of alternative protein isoforms on a large scale
Tress, M. L., Bodenmiller, B., Aebersold, R., & Valencia, A. (2008) · Genome Biology · doi:10.1186/gb-2008-9-11-r162 · Cited by 71
Design of proteins from smaller fragments—learning from evolution
Höcker, B. (2014) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2014.04.007 · Cited by 61
Assessing the functional relevance of splice isoforms
Pozo, F., Martinez-Gomez, L., Walsh, T. A., Rodriguez, J. M., Di Domenico, T., Abascal, F., Vazquez, J., & Tress, M. L. (2021) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqab044 · Cited by 49
The impact of splicing on protein domain architecture
Light, S., & Elofsson, A. (2013) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2013.02.013 · Cited by 48
Structure-guided isoform identification for the human transcriptome
Sommer, M. J., Cha, S., Varabyou, A., Rincon, N., Park, S., Minkin, I., Pertea, M., Steinegger, M., & Salzberg, S. L. (2022) · eLife · doi:10.7554/elife.82556 · Cited by 40
Highly accurate isoform identification for the human transcriptome
Sommer, M. J., Cha, S., Varabyou, A., Rincon, N., Park, S., Minkin, I., Pertea, M., Steinegger, M., & Salzberg, S. L. (2022) · doi:10.1101/2022.06.08.495354 · Cited by 40
Structure of dethiobiotin synthetase at 0.97 Å resolution
Sandalova, T., Schneider, G., Käck, H., & Lindqvist, Y. (1999) · Acta Crystallographica Section D Biological Crystallography · doi:10.1107/s090744499801381x · Cited by 38
Interpreting Potts and Transformer Protein Models Through the Lens of Simplified Attention
Bhattacharya, N., Thomas, N., Rao, R., Dauparas, J., Koo, P. K., Baker, D., Song, Y. S., & Ovchinnikov, S. (2021) · Biocomputing 2022 · doi:10.1142/9789811250477_0004 · Cited by 20
Self-citationUnified framework for modeling multivariate distributions in biological sequencesPreprint
Dauparas, J., Wang, H., Swartz, A., Koo, P. K., Nitzan, M., & Ovchinnikov, S. (2019) · Cold Spring Harbor Laboratory Institutional Repository (Cold Spring Harbor Laboratory) · doi:10.48550/arxiv.1906.02598 · Cited by 16
Self-citationThe structure-fitness landscape of pairwise relations in generative sequence modelsPreprint
Marshall, D., Wang, H., Stiffler, M. A., Dauparas, J., Koo, P. K., & Ovchinnikov, S. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.11.29.402875 · Cited by 8
Self-citation
Cited by
Genome modelling and design across all domains of life with Evo 2
Brixi, G., Durrant, M. G., Ku, J., Naghipourfar, M., Poli, M., Sun, G., Brockman, G., Chang, D., Fanton, A., Gonzalez, G. A., et al. (2026) · Nature · doi:10.1038/s41586-026-10176-5 · Cited by 702
Self-citationSequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580
Self-citationTransformers and genome language models
Consens, M. E., Dufault, C., Wainberg, M., Forster, D., Karimzadeh, M., Goodarzi, H., Theis, F. J., Moses, A. M., & Wang, B. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01007-9 · Cited by 195
InterPLM: discovering interpretable features in protein language models via sparse autoencoders
Simon, E. P., & Zou, J. (2025) · Nature Methods · doi:10.1038/s41592-025-02836-7 · Cited by 169
Biophysics-based protein language models for protein engineering
Gelman, S., Johnson, B., Freschlin, C. R., Sharma, A., D’Costa, S., Peters, J., Gitter, A., & Romero, P. A. (2025) · Nature Methods · doi:10.1038/s41592-025-02776-2 · Cited by 69
Metadata
- Authors
- Zhang, Z., Wayment-Steele, H. K., Brixi, G., Wang, H., Kern, D., & Ovchinnikov, S.
- Affiliations
- Harvard University (US)
- Howard Hughes Medical Institute (US)
- Corresponding author
- Ovchinnikov, Sergey
- ORCID
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2024-10-28
- Volume
- 121
- Issue
- 45
- Pages
- e2406285121-e2406285121
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC11551344
- OpenAlex
- W4403824354
- Cited by
- 140
- References
- 39
- Citations by year
- 2026: 66 · 2025: 56 · 2024: 16
- FWCI
- 18.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- Funders
- Jane Coffin Childs (N/A)
- Howard Hughes Medical Institute (N/A)
- National Institutes of Health (DP5OD026389)
- National Foundation of Science (MCB2032259)
- Swiss National Science Foundation (205321_192371)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
- Keywords
- protein language models, ESM-2, contact prediction, sequence motifs, protein isoforms, sequence masking, protein structure prediction, model interpretability
- MeSH terms
- Markov Chains, Models, Molecular, Protein Conformation, Proteins, Evolution, Molecular, Amino Acid Motifs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.