CoChat
Sign Up
PreprintCited by 723Machine Learning in BioinformaticsProtein Structure and DynamicsDomain Adaptation and Few-Shot Learning

The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.

Evaluating Protein Transfer Learning with TAPE

Open access
Rao, Roshan, Bhattacharya, Nicholas, Thomas, Neil, Duan, Yan, Chen, Xi, Canny, John, Abbeel, Pieter, Song, Yun S.·2019·bioRxiv (Cold Spring Harbor Laboratory)
DOI 10.1101/676825
Open this paper in CoChat — free accountOpen PDF

Abstract

Protein modeling is an increasingly popular area of machine learning research. Semi-supervised learning has emerged as an important paradigm in protein modeling due to the high cost of acquiring supervised protein labels, but the current literature is fragmented when it comes to datasets and standardized evaluation techniques. To facilitate progress in this field, we introduce the Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology. We curate tasks into specific training, validation, and test splits to ensure that each task tests biologically relevant generalization that transfers to real-life scenarios. We bench-mark a range of approaches to semi-supervised protein representation learning, which span recent work as well as canonical sequence learning techniques. We find that self-supervised pretraining is helpful for almost all models on all tasks, more than doubling performance in some cases. Despite this increase, in several cases features learned by self-supervised pretraining still lag behind features extracted by state-of-the-art non-neural techniques. This gap in performance suggests a huge opportunity for innovative architecture design and improved modeling paradigms that better capture the signal in biological sequences. TAPE will help the machine learning community focus effort on scientifically relevant problems. Toward this end, all data and code used to run these experiments are available at https://github.com/songlab-cal/tape .

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/676825

Cites

64 of 64 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Show all 62 references

Research works printed but not linked: 4

  1. 2018, Association for Computational Linguistics, 1, 2227
  2. 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 8
  3. 2017, Advances in Neural Information Processing Systems, 30, 5998
  4. Tianqi Chen , Bing Xu , Chiyuan Zhang , and Carlos Guestrin . Training Deep Nets with Sublinear Memory Cost. arXiv, 2016.

Cited by

Sorted by citations

Items 701–723 of 723

Related

See all

Metadata

Authors
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S.
Affiliations
  • University of California, Berkeley (US)
  • Google (United States) (US)
Corresponding author
  • Rao, Roshan
  • Bhattacharya, Nicholas
  • Thomas, Neil
  • Song, Yun S.
ORCID
Venue
bioRxiv (Cold Spring Harbor Laboratory)
Publisher
openRxiv
Published
2019-06-20
DOI
10.1101/676825
OpenAlex
W2951433247
Cited by
719
References
62
Citations by year
2026: 55 · 2025: 114 · 2024: 150 · 2023: 148 · 2022: 118 · 2021: 100 · 2020: 31 · 2019: 3
Type
posted content
Language
en
Open access
Locations
bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
Funders
  • Open Philanthropy Project
  • National Institutes of Health
  • Defense Advanced Research Projects Agency
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
protein representation learning, semi-supervised learning, protein embeddings, transfer learning, self-supervised pre-training, benchmark dataset, protein structure prediction, protein function prediction, sequence learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review