The DOI matches its Crossref record; 88 of the 91 references it cites were found in a registry or an index.
Protein language model embeddings for fast, accurate, alignment-free protein structure prediction
Open accessPreprint — published as 10.1016/j.str.2022.05.001
Abstract
All state-of-the-art (SOTA) protein structure predictions rely on evolutionary information captured in multiple sequence alignments (MSAs), primarily on evolutionary couplings (co-evolution). Such information is not available for all proteins and is computationally expensive to generate. Prediction models based on Artificial Intelligence (AI) using only single sequences as input are easier and cheaper but perform so poorly that speed becomes irrelevant. Here, we described the first competitive AI solution exclusively inputting embeddings extracted from pre-trained protein Language Models (pLMs), namely from the transformer pLM ProtT5, from single sequences into a relatively shallow (few free parameters) convolutional neural network (CNN) trained on inter-residue distances, i.e. protein structure in 2D. The major advance originated from processing the attention heads learned by ProtT5. Although these models required at no point any MSA, they matched the performance of methods relying on co-evolution. Although not reaching the very top, our lean approach came close at substantially lower costs thereby speeding up development and each future prediction. By generating protein-specific rather than family-averaged predictions, these new solutions could distinguish between structural features differentiating members of the same family of proteins with similar structure predicted alike by all other top methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. (2020) · Nature · doi:10.1038/s41586-020-2649-2 · Cited by 23,826
Harris, C., Millman, K., van der Walt, S., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N., et al. (2020) · RePEc: Research Papers in Economics · Cited by 17,844
Suspending OpenMP Tasks on Asynchronous Events: Extending the Taskwait Construct
Pereira, R., Martin, M., Roussel, A., Carribault, P., & Gautier, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-40744-4_5 · Cited by 13,143
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
ColabFold - Making protein folding accessible to all
Mirdita, M., Schütze, K., Moriwaki, Y., Heo, L., Ovchinnikov, S., & Steinegger, M. (2021) · doi:10.1101/2021.08.15.456425 · Cited by 9,391
Flagged first, then by citations
3 of the 91 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Items 101–139 of 139
Artificial intelligence in early stages of structure-based drug discovery
Ravichandran, R., Park, S., Skorupan, S., Bessette, K., & Gentile, F. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-23932-8.00001-7 · Cited by 2
Mind the gap: An embedding guide to safely travel in sequence space
Wu, A., Trolliet, Q., Rajendran, A., Lála, J., & Angioletti‐Uberti, S. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014433 · Cited by 2
Inferring context-specific site variation with evotuned protein language models
Lytras, S., Strange, A. P., Ito, J., & Sato, K. (2026) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqag018 · Cited by 3
Refining Embedding-Based Binding Predictions by Leveraging AlphaFold2 StructuresPreprint
Endres, L., Olenyi, T., Erckert, K., Weißenow, K., Rost, B., & Littmann, M. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.08.31.505997 · Cited by 2
Self-citationPokharel, S., Barasa, K., Pratyush, P., & KC, D. B. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.04.616755 · Cited by 2
Badaczewska-Dawid, A. E., & Koliński, A. (2024) · Methods in molecular biology · doi:10.1007/978-1-0716-4196-5_4 · Cited by 1
A Survey of Pretrained Protein Language Models
Pokharel, S., Pratyush, P., Chaudhari, M., Heinzinger, M., Caragea, D., Saigo, H., & KC, D. B. (2025) · Methods in molecular biology · doi:10.1007/978-1-0716-4623-6_1 · Cited by 1
Self-citationExo-Tox: Identifying Exotoxins from secreted bacterial proteins
Krueger, T., Durmaz, D. A., & Soto, L. F. J. (2025) · BioData Mining · doi:10.1186/s13040-025-00469-2 · Cited by 1
Wang, J., Geng, A.-Y., Yan, Z., Lin, X., Li, Y.-H., Xu, J.-L., Meng, Y.-J., Zhang, Z.-L., Wei, L.-Y., Zou, Q., et al. (2025) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2025.149267 · Cited by 2
Yin, Z.-N., Zeng, Y.-H., & Gao, F. (2025) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-025-00797-1 · Cited by 1
ESM-PsyPred: Leveraging Protein Language Models for Accurate Prediction of Psychrophilic Proteins
Peng, C., Bian, Y., Yuan, C., Chen, Y., Liu, D., Lu, F., Liu, F., & Liu, Y. (2026) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-025-00810-7 · Cited by 2
Li, B., Zhang, X., Huang, Z., Tiwari, P., Zou, Q., Ding, Y., & Guo, X. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014098 · Cited by 1
Wu, J., Lin, Y., Shen, P., Zhu, L., & Yang, S. (2026) · BMC Bioinformatics · doi:10.1186/s12859-026-06516-x · Cited by 4
Lombard, V., Van, J. N., Grudinin, S., & Laine, É. (2026) · Acta Crystallographica Section D Structural Biology · doi:10.1107/s2059798326006054 · Cited by 4
Le, V. T., Yuune, J. P. T., Vu, T. T. P., Malik, M. S., & Ou, Y. (2025) · Journal of Biomolecular Structure and Dynamics · doi:10.1080/07391102.2025.2512448 · Cited by 1
Domain loss enabled evolution of novel functions in a gene superfamilyPreprint
Koludarov, I., Senoner, T., Jackson, T., Dashevsky, D., Heinzinger, M., Aird, S. D., & Rost, B. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.12.15.520616 · Cited by 1
Self-citationOpinion piece/Book chapter: Artificial Intelligence learns protein predictionPreprint
Heinzinger, M., & Rost, B. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/n7wfp · Cited by 1
Self-citationHuman O -linked Glycosylation Site Prediction Using Pretrained Protein Language ModelPreprint
Pakhrin, S. C., Chauhan, N., Khan, S., Upadhyaya, J., Keller, C., Neuman, L. N., Beck, M. R., & Blanco, E. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.23.563673 · Cited by 1
ANDES: a novel best-match approach for enhancing gene set analysis in embedding spacesPreprint
Li, L., Dannenfelser, R., Cruz, C., & Yao, V. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.11.21.568145
Rendering protein mutation movies with MutAmore
Weissenow, K., & Rost, B. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05610-8
Self-citationSaha, D., Borah, N. J., & Mamud, A. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-31550-3.00014-7
Kiyanpour, F., Motahharynia, A., Ostaszewski, M., & Gheisari, Y. (2025) · Journal of Research in Medical Sciences · doi:10.4103/jrms.jrms_71_25
Li, W., Liu, Z., & Lin, G. N. (2025) · Research Square · doi:10.21203/rs.3.rs-7469880/v1
Karim, W., Alvi, R., Zaman, S. B., Abian, A. I., Raiaan, M. A. K., Mukta, S. H., Rashid, M. R. U., Islam, R., Sebastian, Y., & Azam, S. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10743-z · Cited by 7
Additional file 1 of Rendering protein mutation movies with MutAmore
Weißenow, K., & Rost, B. (2023) · Figshare · doi:10.6084/m9.figshare.24797848
Self-citationAI-Driven Insights into Protein Sequence Analysis Landscape
Asim, M. N., Ahmed, S., & Dengel, A. (2025) · Springer eBooks · doi:10.1007/978-3-031-90454-7_5
ComplexWaveGAT: Complex-Valued Graph Attention Networks for Modeling Protein-Protein Interactions
Gao, L., Zhang, X., Zhang, K., & Sheng, V. S. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356253
The landscape of machine learning approaches for modeling protein conformational ensembles
Sledzieski, S., & Hanson, S. M. (2026) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2026.103253 · Cited by 3
Simple baselines rival protein language models in dense combinatorial protein landscapesPreprint
Talpir, I., & Fleishman, S. J. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.01.722313 · Cited by 2
Qurat-ul-ain, Outeiral, C., Cagiada, M., Teh, Y. W., & Deane, C. M. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.28.728196 · Cited by 1
Zhu, L., Lin, Y., & Yang, S. (2026) · Interdisciplinary Sciences Computational Life Sciences · doi:10.1007/s12539-026-00849-0
Interpreting embeddings from genome and protein language models
May, L. E., White, J. B., & Roell, G. W. (2026) · Biotechnology Advances · doi:10.1016/j.biotechadv.2026.108945
A Fusion-based Sequence-Structure Protein Encoder of α-synuclein under Structural Uncertainty
Angel, G., & P.Sujatha (2026) · International Conference on Inventive Research in Computing Applications (ICIRCA) · doi:10.1109/icirca69024.2026.11570573
Haroon (2026) · Journal of Molecular Modeling · doi:10.1007/s00894-026-06853-x
Hierarchical Enzyme Classification with Integrated Sequence and Folding‑Derived Representations
Liang, P.-Y., Wang, W., Wang, Z., & Bai, J. (2026) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3807503.3819491
Geng, A., Wang, J., Qu, Y., Xu, J., Meng, Y., Zou, Q., Zhang, Z., & Cui, F. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.115882
Tzavella, K., Olsen, C., & Vranken, W. (2026) · Computational and Structural Biotechnology Journal · doi:10.34133/csbj.0226
Deep learning for protein folding
Lahiri, T., & Mondal, R. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-40338-5.00015-9
Metadata
- Authors
- Weissenow, K., Heinzinger, M., & Rost, B.
- Affiliations
- Technical University of Munich (DE)
- Corresponding author
- Weissenow, Konstantin
- Venue
- Structure
- Publisher
- openRxiv
- Published
- 2021-08-02
- Volume
- 30
- Issue
- 8
- Pages
- 1169-1177.e4
- ISSN
- 0969-2126 (ISSN-L)
- OpenAlex
- W4281291878
- Cited by
- 139
- References
- 91
- Citations by year
- 2026: 21 · 2025: 30 · 2024: 34 · 2023: 30 · 2022: 19 · 2021: 5
- FWCI
- 9.22
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Structure (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Funders
- Alexander von Humboldt-Stiftung
- Nvidia
- Deutsche Forschungsgemeinschaft (DFG-GZ: RO1320/4-1, RO1320/4\u20131, RO1320/4‐1)
- Bundesministerium für Bildung und Forschung
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Structural Biology › Enzyme Structure and Function
- Keywords
- protein structure prediction, protein language models, ProtT5, attention heads, convolutional neural networks, protein engineering, deep mutational scanning, coevolution
- MeSH terms
- Artificial Intelligence, Language, Proteins, Sequence Alignment, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
