The DOI matches its Crossref record; 88 of the 91 references it cites were found in a registry or an index.
Protein language model embeddings for fast, accurate, alignment-free protein structure prediction
Open accessPreprint — published as 10.1016/j.str.2022.05.001
Abstract
All state-of-the-art (SOTA) protein structure predictions rely on evolutionary information captured in multiple sequence alignments (MSAs), primarily on evolutionary couplings (co-evolution). Such information is not available for all proteins and is computationally expensive to generate. Prediction models based on Artificial Intelligence (AI) using only single sequences as input are easier and cheaper but perform so poorly that speed becomes irrelevant. Here, we described the first competitive AI solution exclusively inputting embeddings extracted from pre-trained protein Language Models (pLMs), namely from the transformer pLM ProtT5, from single sequences into a relatively shallow (few free parameters) convolutional neural network (CNN) trained on inter-residue distances, i.e. protein structure in 2D. The major advance originated from processing the attention heads learned by ProtT5. Although these models required at no point any MSA, they matched the performance of methods relying on co-evolution. Although not reaching the very top, our lean approach came close at substantially lower costs thereby speeding up development and each future prediction. By generating protein-specific rather than family-averaged predictions, these new solutions could distinguish between structural features differentiating members of the same family of proteins with similar structure predicted alike by all other top methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. (2020) · Nature · doi:10.1038/s41586-020-2649-2 · Cited by 23,826
Harris, C., Millman, K., van der Walt, S., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N., et al. (2020) · RePEc: Research Papers in Economics · Cited by 17,844
Suspending OpenMP Tasks on Asynchronous Events: Extending the Taskwait Construct
Pereira, R., Martin, M., Roussel, A., Carribault, P., & Gautier, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-40744-4_5 · Cited by 13,143
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
ColabFold - Making protein folding accessible to all
Mirdita, M., Schütze, K., Moriwaki, Y., Heo, L., Ovchinnikov, S., & Steinegger, M. (2021) · doi:10.1101/2021.08.15.456425 · Cited by 9,391
Cited by
Sorted by citations
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
High-resolution de novo structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392
Bilingual language model for protein sequence and structure
Heinzinger, M., Weissenow, K., Sanchez, J. G., Henkel, A., Mirdita, M., Steinegger, M., & Rost, B. (2024) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqae150 · Cited by 311
Self-citationBefore and after AlphaFold2: An overview of protein structure prediction
Bertoline, L. M. F., Lima, A. N., Krieger, J. E., & Teixeira, S. K. (2023) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2023.1120370 · Cited by 330
Fine-tuning protein language models boosts predictions across diverse tasks
Schmirler, R., Heinzinger, M., & Rost, B. (2024) · Nature Communications · doi:10.1038/s41467-024-51844-2 · Cited by 229
Self-citation
Metadata
- Authors
- Weissenow, K., Heinzinger, M., & Rost, B.
- Affiliations
- Technical University of Munich (DE)
- Corresponding author
- Weissenow, Konstantin
- Venue
- Structure
- Publisher
- openRxiv
- Published
- 2021-08-02
- Volume
- 30
- Issue
- 8
- Pages
- 1169-1177.e4
- ISSN
- 0969-2126 (ISSN-L)
- OpenAlex
- W4281291878
- Cited by
- 139
- References
- 91
- Citations by year
- 2026: 21 · 2025: 30 · 2024: 34 · 2023: 30 · 2022: 19 · 2021: 5
- FWCI
- 9.22
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Structure (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- Alexander von Humboldt-Stiftung
- Nvidia
- Deutsche Forschungsgemeinschaft (DFG-GZ: RO1320/4-1, RO1320/4\u20131, RO1320/4‐1)
- Bundesministerium für Bildung und Forschung
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Structural Biology › Enzyme Structure and Function
- Keywords
- protein structure prediction, protein language models, ProtT5, attention heads, convolutional neural networks, protein engineering, deep mutational scanning, coevolution
- MeSH terms
- Artificial Intelligence, Language, Proteins, Sequence Alignment, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
