CoChat

An Analysis of Protein Language Model Embeddings for Fold Prediction

Open access
Villegas-Morcillo, Amelia, Gomez, Angel M., Sanchez, Victoria · 2022 · Briefings in Bioinformatics

Preprint — published as 10.1093/bib/bbac142 · Paper page

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Abstract

The identification of the protein fold class is a challenging problem in structural biology. Recent computational methods for fold prediction leverage deep learning techniques to extract protein fold-representative embeddings mainly using evolutionary information in the form of multiple sequence alignment (MSA) as input source. In contrast, protein language models (LM) have reshaped the field thanks to their ability to learn efficient protein representations (protein-LM embeddings) from purely sequential information in a self-supervised manner. In this paper, we analyze a framework for protein fold prediction using pre-trained protein-LM embeddings as input to several fine-tuning neural network models which are supervisedly trained with fold labels. In particular, we compare the performance of six protein-LM embeddings: the LSTM-based UniRep and SeqVec, and the transformer-based ESM-1b, ESM-MSA, ProtBERT, and ProtT5; as well as three neural networks: Multi-Layer Perceptron (MLP), ResCNN-BGRU (RBG), and Light-Attention (LAT). We separately evaluated the pairwise fold recognition (PFR) and direct fold classification (DFC) tasks on well-known benchmark datasets. The results indicate that the combination of transformer-based embeddings, particularly those obtained at amino acid-level, with the RBG and LAT fine-tuning models performs remarkably well in both tasks. To further increase prediction accuracy, we propose several ensemble strategies for PFR and DFC, which provide a significant performance boost over the current state-of-the-art results. All this suggests that moving from traditional protein representations to protein-LM embeddings is a very promising approach to protein fold-related tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2022.02.07.479394

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 2 references

Research works printed but not linked: 101

  1. 2020, Improved protein structure prediction using potentials from deep learning, Nature, 577, 706, 10.1038/s41586-019-1923-7 · DOI · Paper page
  2. 10.1126/science.abj8754 · doi:10.1126/science.abj8754 · Paper page
  3. Mihaly Varadi , Stephen Anyango , Mandar Deshpande , Sreenath Nair , Cindy Natassia , Galabina Yordanova , David Yuan , Oana Stroe , Gemma Wood , Agata Laydon , et al. AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic Acids Research, 2021. · DOI · Paper page
  4. 10.1006/jmbi.1995.0159 · DOI
  5. John-Marc Chandonia , Lindsey Guan , Shiangyi Lin , Changhua Yu , Naomi K. Fox , and Steven E. Brenner . SCOPe: improvements to the structural classification of proteins–extended database to facilitate variant interpretation and machine learning. Nucleic Acids Research, 2021. · DOI · Paper page
  6. 10.1016/S0969-2126(97)00260-8 · DOI · Paper page
  7. 2021, CATH: increased structural coverage of functional space, Nucleic acids research, 49, D266, 10.1093/nar/gkaa1079 · DOI · Paper page
  8. 10.1093/nar/28.1.235 · DOI · Paper page
  9. 2021, RCSB Protein Data Bank: powerful new tools for exploring 3d structures of biological macromolecules for basic and applied research and education in fundamental biology, biomedicine, biotechnology, bioengineering and energy sciences, Nucleic Acids Research, 49, D437, 10.1093/nar/gkaa1038 · DOI · Paper page
  10. 10.1146/annurev.bi.59.070190.005043 · DOI · Paper page
  11. 10.1038/358086a0 · DOI · Paper page
  12. 10.1006/jmbi.1999.3377 · DOI · Paper page
  13. 2010, Protein folds and protein folding, Protein Engineering, Design & Selection, 24, 11
  14. 2013, On the universe of protein folds, Annual Review of Biophysics, 42, 559, 10.1146/annurev-biophys-083012-130432 · DOI
  15. 2008, A survey of protein fold recognition algorithms, Journal of Computer Science, 4, 768, 10.3844/jcssp.2008.768.776 · DOI
  16. 2018, A comprehensive review and comparison of different computational methods for protein remote homology detection, Briefings in Bioinformatics, 19, 231, 10.1093/bib/bbw108 · DOI · Paper page
  17. 2019, In Machine Learning Paradigms, 149, 101, 10.1007/978-3-319-94030-4_5 · DOI
  18. 10.1093/nar/25.17.3389 · DOI · Paper page
  19. 10.1093/bioinformatics/bti125 · DOI · Paper page
  20. 2014, MRFalign: Protein homology detection through alignment of Markov random fields, PLoS Computational Biology, 10, e1003500, 10.1371/journal.pcbi.1003500 · DOI · Paper page
  21. 10.1142/S0219720003000186 · DOI · Paper page
  22. Jian Peng and Jinbo Xu . Boosting protein threading accuracy. In Annual International Conference on Research in Computational Molecular Biology, pages 31–45, 2009. · DOI
  23. 10.1093/bioinformatics/btr350 · DOI · Paper page
  24. 10.1093/bioinformatics/bts213 · DOI
  25. 2018, Protein alignment based on higher order conditional random fields for template-based modeling, PLoS ONE, 13, e0197912, 10.1371/journal.pone.0197912 · DOI
  26. 10.1093/bioinformatics/btx217 · DOI · Paper page
  27. 2019, Detecting distant-homology protein structures by aligning deep neural-network based contact maps, PLoS Computational Biology, 15, 1
  28. 10.1093/bioinformatics/btl102 · DOI · Paper page
  29. 10.1186/1471-2105-15-S9-S14 · DOI
  30. 2015, Improving protein fold recognition by deep learning networks, Scientific Reports, 5, 17573, 10.1038/srep17573 · DOI · Paper page
  31. 2019, Protein fold recognition based on multi-view modeling, Bioinformatics, 35, 2982, 10.1093/bioinformatics/btz040 · DOI
  32. Ke Yan , Jie Wen an Yong Xu , and Bin Liu . Protein fold recognition based on auto-weighted multi-view graph embedding learning model. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2020. · DOI
  33. 2021, MLDH-Fold: Protein fold recognition based on multi-view low-rank modeling, Neurocomputing, 421, 127, 10.1016/j.neucom.2020.09.028 · DOI
  34. Bin Liu, Yulin Zhu , and Ke Yan . Fold-LTR-TCP: protein fold recognition based on triadic closure principle. Briefings in Bioinformatics, 2019. · DOI
  35. Jiangyi Shao , Ke Yan , and Bin Liu . FoldRec-C2C: protein fold recognition by combining cluster-to-cluster model and protein similarity network. Briefings in Bioinformatics, 2020. · DOI
  36. Jiangyi Shao and Bin Liu . ProtFold-DFG: protein fold recognition by combining Directed Fusion Graph and PageRank algorithm. Briefings in Bioinformatics, 2020. · DOI
  37. 2017, Improving protein fold recognition by extracting fold-specific features from predicted residue–residue contacts, Bioinformatics, 33, 3749, 10.1093/bioinformatics/btx514 · DOI
  38. Bin Liu, Chen-Chen Li , and Ke Yan . DeepSVM-fold: protein fold recognition by combining support vector machines and pairwise sequence similarity scores generated by deep learning networks. Briefings in Bioinformatics, 2019. · DOI
  39. Yihe Pang and Bin Liu . SelfAT-Fold: protein fold recognition based on residue-based and motif-based self-attention networks. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2020. · DOI
  40. Yan Liu , Yi-Heng Zhu , Xiaoning Song , Jiangning Song , and Dong-Jun Yu . Why can deep convolutional neural networks improve protein fold recognitionã a visual explanation by interpretation. Briefings in Bioinformatics, 2021. · DOI
  41. 2021, Improving protein fold recognition using triplet network and ensemble deep learning, Briefings in Bioinformatics, 22, bbab248, 10.1093/bib/bbab248 · DOI
  42. 2021, Protein fold recognition from sequences using convolutional and recurrent neural networks, IEEE/ACM Transactions on Computational Biology and Bioinformatics, 18, 2848, 10.1109/TCBB.2020.3012732 · DOI
  43. 10.1186/s12859-021-04034-6 · DOI · Paper page
  44. 2016, Recent progress in machine learning-based methods for protein fold recognition, International journal of Molecular Sciences, 17, 2118, 10.3390/ijms17122118 · DOI · Paper page
  45. 10.1093/bioinformatics/17.4.349 · DOI · Paper page
  46. 10.1093/bioinformatics/btl170 · DOI · Paper page
  47. 10.1093/bioinformatics/btp500 · DOI · Paper page
  48. 2011, Improving taxonomy-based protein fold recognition by using global and local features, Proteins: Structure, Function, and Bioinformatics, 79, 2053, 10.1002/prot.23025 · DOI
  49. 2015, Advancing the accuracy of protein fold recognition by utilizing profiles from hidden Markov models, IEEE Transactions on Nanobioscience, 14, 761, 10.1109/TNB.2015.2457906 · DOI
  50. Daozheng Chen , Xiaoyu Tian , Bo Zhou , and Jun Gao . ProFold: Protein fold classification with additional structural features and a novel ensemble classifier. BioMed Research International, 2016:1–10, 2016. · DOI
  51. 2016, An ensemble approach to protein fold classification by integration of template-based assignment and support vector machine classifier, Bioinformatics, 33, 863
  52. 2019, Protein fold recognition using deep kernelized extreme learning machine and linear discriminant analysis, Neural Computing and Applications, 31, 4201, 10.1007/s00521-018-3346-z · DOI
  53. 2020, An enhanced protein fold recognition for low similarity datasets using convolutional and skip-gram features with deep neural network, IEEE Transactions on NanoBioscience, 20, 42
  54. 10.1186/s12859-020-3475-0 · DOI · Paper page
  55. 10.1093/bioinformatics/btx780 · DOI · Paper page
  56. 2020, Progress in neural nlp: Modeling, learning, and reasoning, Engineering, 6, 275, 10.1016/j.eng.2019.12.014 · DOI · Paper page
  57. 2021, The language of proteins: Nlp, machine learning & protein sequences, Computational and Structural Biotechnology Journal, 19, 1750, 10.1016/j.csbj.2021.03.022 · DOI · Paper page
  58. 10.1093/nar/gkg095 · DOI · Paper page
  59. 10.1093/NAR/GKAA913 · DOI · Paper page
  60. 10.1093/bioinformatics/btu739 · DOI · Paper page
  61. 10.1093/nar/gkaa1100 · DOI · Paper page
  62. 2018, Clustering huge protein sequence sets in linear time, Nature Communications, 9, 1
  63. 2019, Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold, Nature Methods, 16, 603, 10.1038/s41592-019-0437-4 · DOI · Paper page
  64. 10.1371/journal.pone.0141287 · DOI · Paper page
  65. Tomas Mikolov , Kai Chen , Greg Corrado , and Jeffrey Dean . Efficient estimation of word representations in vector space. In International Conference on Learning Representations, 2013.
  66. Tomas Mikolov , Ilya Sutskever , Kai Chen , Greg Corrado , and Jeff Dean . Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119, 2013.
  67. 10.1162/neco.1997.9.8.1735 · DOI · Paper page
  68. 10.1093/bioinformatics/btaa003 · DOI · Paper page
  69. 2019, Unified rational protein engineering with sequence-based deep representation learning, Nature Methods, 16, 1315, 10.1038/s41592-019-0598-1 · DOI · Paper page
  70. 2018, Deep contextualized word representations, arXiv preprint
  71. 10.1186/s12859-019-2669-9 · DOI · Paper page
  72. Tristan Bepler and Bonnie Berger . Learning protein sequence embeddings using information from structure. In International Conference on Learning Representations, 2019.
  73. 2017, Attention is all you need, In Advances in Neural Information Processing Systems, 30, 5998
  74. Roshan Rao , Nicholas Bhattacharya , Neil Thomas , Yan Duan , Xi Chen , John Canny , Pieter Abbeel , and Yun S. Song . Evaluating protein transfer learning with TAPE. In Advances in neural information processing systems, 2019.
  75. Alexander Rives , Joshua Meier , Tom Sercu , Siddharth Goyal , Zeming Lin , Jason Liu , Demi Guo , Myle Ott , C. Lawrence Zitnick , Jerry Ma , et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), 2021. · DOI · Paper page
  76. Roshan M. Rao , Jason Liu , Robert Verkuil , Joshua Meier , John Canny , Pieter Abbeel , Tom Sercu , and Alexander Rives . Msa transformer. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8844–8856, 2021.
  77. Ahmed Elnaggar , Michael Heinzinger , Christian Dallago , Ghalia Rehawi , Wang Yu , Llion Jones , Tom Gibbs , Tamas Feher , Christoph Angerer , Martin Steinegger , Debsindhu Bhowmik , and Burkhard Rost . ProtTrans: Towards cracking the language of lifes code through self-supervised deep learning and high performance computing. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–16, 2021. · DOI · Paper page
  78. 2018, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint
  79. Zihang Dai , Zhilin Yang , Yiming Yang , Jaime Carbonell , Quoc V. Le , and Ruslan Salakhutdinov . Transformer-XL: attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, page 2978–2988, 2019.
  80. Zhilin Yang , Zihang Dai , Yiming Yang , Jaime Carbonell , Ruslan Salakhutdinov , and Quoc V. Le . XLNet: generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, 2019.
  81. 2019, Exploring the limits of transfer learning with a unified text-to-text transformer, arXiv preprint
  82. Zhenzhong Lan , Mingda Chen , Sebastian Goodman , Kevin Gimpel , Piyush Sharma , and Radu Soricut . ALBERT: a lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations, 2020.
  83. Kevin Clark , Minh-Thang Luong , Quoc V. Le , and Christopher D. Manning . ELECTRA: pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2020.
  84. 2021, Light attention predicts protein location from the language of life, Bioinformatics Advances, 11, vbab035
  85. 2021, Unsupervised protein embeddings outperform hand-crafted sequence and structure features at predicting molecular function, Bioinformatics, 37, 162, 10.1093/bioinformatics/btaa701 · DOI · Paper page
  86. 2021, Embeddings from deep learning transfer GO annotations beyond homology, Scientific reports, 11, 1
  87. 2021, The power of universal contextualized protein embeddings in cross-species protein function prediction, Evolutionary Bioinformatics, 17, 1
  88. Joshua Meier , Roshan Rao , Robert Verkuil , Jason Liu , Tom Sercu , and Alexander Rives . Language models enable zero-shot prediction of the effects of mutations on protein function. In Advances on Neural Information Processing Systems, 2021. · DOI · Paper page
  89. Celine Marquet , Michael Heinzinger , Tobias Olenyi , Christian Dallago , Michael Bernhofer , Kyra Erckert , Dmitrii Nechaev , and Burkhard Rost . Embeddings from protein language models predict conservation and variant effects. Research Square preprint, 2021. · DOI · Paper page
  90. Jesse Vig , Ali Madani , Lav R. Varshney , Caiming Xiong , Richard Socher , and Nazneen Rajani . BERTology meets biology: Interpreting attention in protein language models. In International Conference on Learning Representations, 2021. · DOI · Paper page
  91. 10.1186/s12859-019-2669-9 · DOI · Paper page
  92. 10.1093/nar/gkw1081 · DOI · Paper page
  93. 2016, Multiplicative lstm for sequence modelling, arXiv preprint
  94. 1997, Bidirectional recurrent neural networks, IEEE Transactions on Signal Processing, 45, 2673, 10.1109/78.650093 · DOI · Paper page
  95. 2019, Axial attention in multidimensional transformers, arXiv preprint
  96. Sergey Ioffe and Christian Szegedy . Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, pages 448–456, 2015.
  97. 2014, Dropout: a simple way to prevent neural networks from overfitting, The journal of Machine Learning Research, 15, 1929
  98. 2014, Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint
  99. Hao Wang , Yitong Wang , Zheng Zhou , Xing Ji , Dihong Gong , Jingchao Zhou , Zhifeng Li , and Wei Liu . CosFace: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5265–5274, 2018. · DOI · Paper page
  100. 2014, Adam: A method for stochastic optimization, arXiv preprint
  101. 10.1006/jmbi.1990.9999 · doi:10.1006/jmbi.1990.9999 · Paper page

Metadata

Authors
Villegas-Morcillo, A., Gomez, A. M., & Sanchez, V.
Affiliations
Universidad de Granada (ES)
Corresponding author
Villegas-Morcillo, Amelia
Venue
Briefings in Bioinformatics
Publisher
openRxiv
Published
2022-02-10
Volume
23
Issue
3
ISSN
1467-5463 (ISSN-L)
OpenAlex
W4225891318
Cited by
46
References
2
Citations by year
2026: 5 · 2025: 13 · 2024: 15 · 2023: 12 · 2022: 1
FWCI
3.35
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • Briefings in Bioinformatics (publishedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Institutional Repository of the University of Granada (University of Granada) (submittedVersion, cc-by-nc-nd)
Funders
  • Family Process Institute (BES-2017-079792)
  • Ministerio de Ciencia e Innovación (PID2019-104206GB-I00)
  • Agencia Estatal de Investigación (10.13039/501100011033)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Structural Biology › Enzyme Structure and Function
Keywords
protein fold prediction, protein language models, protein embeddings, Earth system models, ProtBERT, ProtT5, ensemble learning, fine-tuning, protein representation learning
MeSH terms
Language, Natural Language Processing, Proteins, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: An Analysis of Protein Language Model Embeddings for Fold Prediction. https://app.cochat.ai/papers/10.1101/2022.02.07.479394
The paper
Villegas-Morcillo, A., Gomez, A. M., & Sanchez, V. (2022). An Analysis of Protein Language Model Embeddings for Fold Prediction. Briefings in Bioinformatics. https://doi.org/10.1101/2022.02.07.479394

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.