Interpreting Potts and Transformer Protein Models Through the Lens of Simplified Attention
Open accessPublished version of a preprint
Abstract
The established approach to unsupervised protein contact prediction estimates coevolving positions using undirected graphical models. This approach trains a Potts model on a Multiple Sequence Alignment. Increasingly large Transformers are being pretrained on unlabeled, unaligned protein sequence databases and showing competitive performance on protein contact prediction. We argue that attention is a principled model of protein interactions, grounded in real properties of protein family data. We introduce an energy-based attention layer, factored attention, which, in a certain limit, recovers a Potts model, and use it to contrast Potts and Transformers. We show that the Transformer leverages hierarchical signal in protein family databases not captured by single-layer models. This raises the exciting possibility for the development of powerful structured models of protein family databases.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 2× · see citing works
“10–13 applies BERT-style pretraining directly on the database of proteins with minimal preprocessing.”
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · see citing works
UniRef: comprehensive and non-redundant UniProt reference clusters
Suzek, B. E., Huang, H., McGarvey, P., Mazumder, R., & Wu, C. H. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm098 · Cited by 1,613 · see citing works
Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model
Wang, S., Sun, S., Li, Z., Zhang, R., & Xu, J. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005324 · Cited by 931 · see citing works
Learning generative models for protein fold families
Balakrishnan, S., Kamisetty, H., Carbonell, J. G., Lee, S., & Langmead, C. J. (2011) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.22934 · Cited by 316 · see citing works
Graphical Models of Residue Coupling in Protein Families
Thomas, J., Ramakrishnan, N., & Bailey-Kellogg, C. (2008) · IEEE/ACM Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2007.70225 · Cited by 77 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Protein language models learn evolutionary statistics of interacting sequence motifs
Zhang, Z., Wayment-Steele, H. K., Brixi, G., Wang, H., Kern, D., & Ovchinnikov, S. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2406285121 · Cited by 140 · see citing works
Self-citation
- Authors
- Bhattacharya, N., Thomas, N., Rao, R., Dauparas, J., Koo, P. K., Baker, D., Song, Y. S., & Ovchinnikov, S.
- Affiliations
- University of California, Berkeley (US)
- University of Washington (US)
- Cold Spring Harbor Laboratory (US)
- Harvard University (US)
- ORCID
- Venue
- Biocomputing 2022
- Publisher
- WORLD SCIENTIFIC
- Published
- 2021-11-28
- Volume
- 27
- Pages
- 34-45
- PMC
- PMC8752338
- OpenAlex
- W3216580380
- Cited by
- 20
- References
- 8
- Citations by year
- 2025: 5 · 2024: 9 · 2023: 1 · 2022: 5
- FWCI
- 13.57
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Pacific Symposium on Biocomputing (PSB) (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Cold Spring Harbor Laboratory Institutional Repository (Cold Spring Harbor Laboratory) (acceptedVersion)
- PubMed Central (acceptedVersion, cc-by-nc)
- Funders
- National Human Genome Research Institute (T32HG000047)
- National Institute of General Medical Sciences (R35GM134922)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
- Keywords
- protein contact prediction, Potts model, transformer models, attention mechanism, protein family databases, multiple sequence alignment
- MeSH terms
- Attention, Humans, Proteins, Sequence Alignment, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
