CoChat

ProteinGLUE: A multi-task benchmark suite for self-supervised protein modeling

Open access

Preprint — published as 10.1038/s41598-022-19608-4 · Paper page

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Abstract

Self-supervised language modeling is a rapidly developing approach for the analysis of protein sequence data. However, work in this area is heterogeneous and diverse, making comparison of models and methods difficult. Moreover, models are often evaluated only on one or two downstream tasks, making it unclear whether the models capture generally useful properties. We introduce the ProteinGLUE benchmark for the evaluation of protein representations: a set of seven tasks for evaluating learned protein representations. We also offer reference code, and we provide two baseline models with hyperparameters specifically trained for these benchmarks. Pre-training was done on two tasks, masked symbol prediction and next sentence prediction. We show that pre-training yields higher performance on a variety of downstream tasks such as secondary structure and protein interaction interface prediction, compared to no pre-training. However, the larger base model does not outperform the smaller medium. We expect the ProteinGLUE benchmark dataset introduced here, together with the two baseline pre-trained models and their performance evaluations, to be of great value to the field of protein sequence-based property prediction. Availability: code and datasets from https://github.com/ibivu/protein-glue

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2021.12.13.472460

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Improved performance on downstream tasks with pre-trained models. Prediction performances of the benchmark test set, as introduced in section ProteinGLUE Benchmark tasks, for a medium model without the pre-training step (pink), medium model including a pre-trained model until step 2 000 000 (dark red), base model without the pre-training step (grey), and base model including a pre-trained model until step 1 000 000 (dark blue). The yellow lines indicate the performance of a random or majority-class baseline. All models are trained ten times on their selected set of hyperparameters after which the mean performance and standard error is determined.
Figure 1 Improved performance on downstream tasks with pre-trained models. Prediction performances of the benchmark test set, as introduced in section ProteinGLUE Benchmark tasks, for a medium model without the pre-training step (pink), medium model including a pre-trained model until step 2 000 000 (dark red), base model without the pre-training step (grey), and base model including a pre-trained model until step 1 000 000 (dark blue). The yellow lines indicate the performance of a random or majority-class baseline. All models are trained ten times on their selected set of hyperparameters after which the mean performance and standard error is determined.CC BY · PubMed Central
Figure 2 Performance of downstream tasks monitored during pre-training. Prediction performances of the benchmark test set, as introduced in section ProteinGLUE Benchmark tasks, (a) for a medium sized model, and (b) for a base sized model, trained without the pre-training step (pink/grey), and on a pre-trained model for 350 000 (light red/blue), 700 000 (red/blue) and 1 000 000 steps (dark red/blue). Further details as in Fig. 1.
Figure 2 Performance of downstream tasks monitored during pre-training. Prediction performances of the benchmark test set, as introduced in section ProteinGLUE Benchmark tasks, (a) for a medium sized model, and (b) for a base sized model, trained without the pre-training step (pink/grey), and on a pre-trained model for 350 000 (light red/blue), 700 000 (red/blue) and 1 000 000 steps (dark red/blue). Further details as in Fig. 1.CC BY · PubMed Central
Figure 3 Performance on downstream task is generally stable between test and validation sets. Prediction performances of the benchmark validation and test set for both the converged medium and base models. Further details are given in Fig. 1.
Figure 3 Performance on downstream task is generally stable between test and validation sets. Prediction performances of the benchmark validation and test set for both the converged medium and base models. Further details are given in Fig. 1.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 64

  1. 2019, Netsurfp-2.0: Improved prediction of protein structural features by integrated deep learning, Proteins: Struct. Funct. Bioinforma, 87, 520, 10.1002/prot.25674 · DOI · Paper page
  2. 10.1093/bioinformatics/btaa629 · DOI · Paper page
  3. 10.1073/pnas.1907373117 · DOI · Paper page
  4. 2019, Improving multi-task deep neural networks via knowledge distillation for natural language understanding, arXiv preprint
  5. 2018, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint
  6. 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
  7. 2019, Evaluating protein transfer learning with tape, Adv. neural information processing systems, 32, 9689
  8. Rives, A. et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl. Acad. Sci. 118 (2021). · DOI · Paper page
  9. 2020, Progen: Language modeling for protein generation, arXiv preprint
  10. 2020, Prottrans: towards cracking the language of life’s code through self-supervised deep learning and high performance computing, arXiv preprint
  11. 2018, Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint
  12. 10.1073/pnas.37.4.205 · DOI · Paper page
  13. 10.1002/bip.360221211 · DOI · Paper page
  14. 10.1093/bioinformatics/btx005 · DOI
  15. 10.1006/jmbi.1998.2144 · DOI · Paper page
  16. 10.1093/bioinformatics/btz428 · DOI
  17. Potocnakova, L. , Bhide, M. & Pulzova, L. B. An introduction to b-cell epitope mapping and in silico epitope prediction. J. immunology research 2016 (2016). · DOI · Paper page
  18. 2021, Serendip-ce: sequence-based interface prediction for conformational epitopes, Bioinformatics, 37, 3421, 10.1093/bioinformatics/btab321 · DOI
  19. 10.1021/bi00327a032 · DOI · Paper page
  20. Gils, J. et al. How sticky are our proteins?: Quantifying hydrophobicity of the human proteome. arXiv.org (2021).
  21. 10.1093/NAR/GKAA913 · DOI · Paper page
  22. 2020, On the effect of dropping layers of pre-trained transformer models, arXiv preprint
  23. 2021, Large-scale news classification using bert language model: Spark nlp approach, In 6th International Conference on Sustainable Information Engineering and Technology, 2021, 240
  24. Blum, A. & Mitchell, T. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory, 92–100 (1998). · DOI · Paper page
  25. 2009, Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews], IEEE Transactions on Neural Networks, 20, 542, 10.1109/TNN.2009.2015974 · DOI · Paper page
  26. 10.1109/TIT.1965.1053799 · DOI · Paper page
  27. 2018, Universal language model fine-tuning for text classification, arXiv preprint
  28. Peters, M. E. et al. Deep contextualized word representations. In Proceedings of NAACL-HLT, 2227–2237 (2018).
  29. 10.1101/2021.09.03.458832 · doi:10.1101/2021.09.03.458832
  30. Vaswani, A. et al. Attention is all you need. In Advances in neural information processing systems, 5998–6008 (2017).
  31. Brown, T. B. et al. Language models are few-shot learners. In Larochelle, H. , Ranzato, M. , Hadsell, R. , Balcan, M. & Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual (2020).
  32. 10.1093/bib/bbw129 · DOI · Paper page
  33. 10.1093/bioinformatics/btaa524 · DOI · Paper page
  34. 10.7717/peerj.7126 · DOI · Paper page
  35. 2019, Unified rational protein engineering with sequence-based deep representation learning, Nat. methods, 16, 1315, 10.1038/s41592-019-0598-1 · DOI · Paper page
  36. 10.1186/s12859-019-2669-9 · DOI · Paper page
  37. 10.1093/bioinformatics/btaa003 · DOI · Paper page
  38. 2019, Learning protein sequence embeddings using information from structure, arXiv preprint
  39. 2021, Pre-training of deep bidirectional protein sequence representations with structural information, IEEE Access, 9, 123912, 10.1109/ACCESS.2021.3110269 · DOI · Paper page
  40. Rives, A. et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl. Acad. Sci. 118 (2021). · DOI · Paper page
  41. 10.1093/nar/25.17.3389 · DOI · Paper page
  42. 10.1093/nar/gkaa1100 · DOI · Paper page
  43. 2018, Clustering huge protein sequence sets in linear time, Nat. communications, 9, 1
  44. 10.1038/s41592-019-0437-4 · DOI · Paper page
  45. 2020, Bertology meets biology: Interpreting attention in protein language models, arXiv preprint
  46. 10.1093/bioinformatics/bty481 · DOI · Paper page
  47. 10.1101/2021.02.12.430858 · DOI · Paper page
  48. 10.1093/nar/gkt1240 · DOI · Paper page
  49. 2018, Deep generative models of genetic variation capture the effects of mutations, Nat. methods, 15, 816, 10.1038/s41592-018-0138-4 · DOI · Paper page
  50. 10.1016/j.cels.2017.11.003 · DOI · Paper page
  51. 10.1093/bioinformatics/btx431 · DOI · Paper page
  52. 10.1093/nar/gkv485 · DOI · Paper page
  53. 1999, Evaluation and improvement of multiple sequence methods for protein secondary structure prediction, Proteins: Struct. Funct. Bioinforma, 34, 508, 10.1002/(SICI)1097-0134(19990301)34:4<508::AID-PROT10>3.0.CO;2-4 · DOI · Paper page
  54. 2019, Critical assessment of methods of protein structure prediction (casp)—round xiii, Proteins: Struct. Funct. Bioinforma, 87, 1011, 10.1002/prot.25823 · DOI · Paper page
  55. 10.1093/bioinformatics/bty1006 · DOI · Paper page
  56. 10.1093/bioinformatics/btg224 · DOI · Paper page
  57. 10.1093/protein/1.3.159 · DOI · Paper page
  58. 10.1186/s12859-014-0430-y · DOI · Paper page
  59. 10.1093/bioinformatics/btq302 · DOI · Paper page
  60. 10.1093/nar/gkt1043 · DOI · Paper page
  61. 2019, Research for practice: troubling trends in machine-learning scholarship, Commun. ACM, 62, 45, 10.1145/3316774 · DOI
  62. 2019, Large batch optimization for deep learning: Training bert in 76 minutes, arXiv preprint
  63. Hendrycks, D. & Gimpel, K. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR abs/1606.08415 (2016). 1606.08415. · arXiv
  64. 2016, A medium-scale distributed system for computer science research: Infrastructure for the long term, Computer, 49, 54, 10.1109/MC.2016.127 · DOI

Metadata

Authors
Capel, H., Weiler, R., Dijkstra, M., Vleugels, R., Bloem, P., & Feenstra, K. A.
Affiliations
Vrije Universiteit Amsterdam (NL)
Corresponding author
Feenstra, K. Anton
Venue
Scientific Reports
Publisher
openRxiv
Published
2021-12-14
Volume
12
Issue
1
Pages
16047-16047
ISSN
2045-2322 (ISSN-L)
OpenAlex
W4297243376
Cited by
11
References
1
Citations by year
2026: 1 · 2025: 3 · 2024: 3 · 2023: 2 · 2022: 2
FWCI
0.82
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • VU Research Portal (publishedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
Keywords
protein representation, benchmark suite, next sentence prediction, pre-training, secondary structure prediction, multi-task learning
MeSH terms
Amino Acid Sequence, Amino Acids, Natural Language Processing, Proteins, Benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: ProteinGLUE: A multi-task benchmark suite for self-supervised protein modeling. https://app.cochat.ai/papers/10.1101/2021.12.13.472460
The paper
Capel, H., Weiler, R., Dijkstra, M., Vleugels, R., Bloem, P., & Feenstra, K. A. (2021). ProteinGLUE: A multi-task benchmark suite for self-supervised protein modeling. Scientific Reports. https://doi.org/10.1101/2021.12.13.472460

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.