CoChat

FLIP: Benchmark tasks in fitness landscape inference for proteins

Open access
Dallago, Christian, Mou, Jody, Johnston, Kadina E., Wittmann, Bruce J., Bhattacharya, Nicholas, Goldman, Samuel, Madani, Ali, Yang, Kevin K. · 2021 · bioRxiv (Cold Spring Harbor Laboratory)
PreprintCited by 125

The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.

Abstract

Machine learning could enable an unprecedented level of control in protein engineering for therapeutic and industrial applications. Critical to its use in designing proteins with desired properties, machine learning models must capture the protein sequence-function relationship, often termed fitness landscape . Existing bench-marks like CASP or CAFA assess structure and function predictions of proteins, respectively, yet they do not target metrics relevant for protein engineering. In this work, we introduce Fitness Landscape Inference for Proteins (FLIP), a benchmark for function prediction to encourage rapid scoring of representation learning for protein engineering. Our curated tasks, baselines, and metrics probe model generalization in settings relevant for protein engineering, e.g. low-resource and extrapolative. Currently, FLIP encompasses experimental data across adeno-associated virus stability for gene therapy, protein domain B1 stability and immunoglobulin binding, and thermostability from multiple protein families. In order to enable ease of use and future expansion to new tasks, all data are presented in a standard format. FLIP scripts and data are freely accessible at https://benchmark.protein.properties .

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2021.11.09.467890

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 26 references

Research works printed but not linked: 39

  1. 10.1002/bit.22077 · doi:10.1002/bit.22077
  2. 10.1016/j.copbio.2009.07.001 · DOI · Paper page
  3. 10.1080/10408398.2014.1000481 · doi:10.1080/10408398.2014.1000481
  4. 10.1126/science.aay8484 · DOI · Paper page
  5. 10.1093/bib/bbaa150 · DOI
  6. 10.1038/nrm2805 · DOI · Paper page
  7. 10.1093/protein/gzg077 · DOI
  8. 2019, Machine-learning-guided directed evolution for protein engineering, Nature methods, 16, 687, 10.1038/s41592-019-0496-6 · DOI · Paper page
  9. 10.1073/pnas.1901979116 · DOI · Paper page
  10. 10.1101/2020.12.04.408955 · doi:10.1101/2020.12.04.408955
  11. Joshua Meier , Roshan Rao , Robert Verkuil , Jason Liu , Tom Sercu , and Alexander Rives . Language models enable zero-shot prediction of the effects of mutations on protein function. bioRxiv, 2021. 1, 2, 3, 5, 5 · DOI · Paper page
  12. Jonathan Frazer , Pascal Notin , Mafalda Dias , Aidan Gomez , Kelly Brock , Yarin Gal , and Debora Marks . Large-scale clinical interpretation of genetic variants using evolutionary data and deep learning. bioRxiv, 2020. · DOI
  13. 10.1186/s12859-019-2669-9 · DOI · Paper page
  14. 1995, Masthead, Proteins: Structure, Function, and Bioinformatics, 23, fmi, 10.1002/prot.340230301 · doi:10.1002/prot.340230301
  15. Amy X Lu , Haoran Zhang , Marzyeh Ghassemi , and Alan M Moses . Self-supervised contrastive learning of protein representations by mutual information maximization. BioRxiv, 2020. 1
  16. 2020, Is transfer learning necessary for protein landscape prediction?, arXiv preprint
  17. Chloe Hsu , Hunter Nisonoff , Clara Fannjiang , and Jennifer Listgarten . Combining evolutionary and assay-labelled data for protein fitness prediction. bioRxiv, 2021. 1 · DOI
  18. 10.1016/j.tig.2013.09.005 · DOI · Paper page
  19. 10.1016/j.cels.2017.11.003 · DOI · Paper page
  20. Céline Marquet , Michael Heinzinger , Tobias Olenyi , Christian Dallago , Michael Bernhofer , Kyra Erckert , and Burkhard Rost . Embeddings from protein language models predict conservation and variant effects. 2021. 2 · DOI · Paper page
  21. 2021, Embeddings from deep learning transfer GO annotations beyond homology, Scientific reports, 11, 1
  22. 10.7554/eLife.16965 · DOI · Paper page
  23. 10.1021/ja058292x · DOI
  24. 2021, Deep diversification of an AAV capsid protein by machine learning, Nature Biotechnology, 39, 691, 10.1038/s41587-020-00793-4 · DOI · Paper page
  25. 10.1038/s41564-018-0356-7 · DOI
  26. 2020, Meltome atlas– thermal proteome stability across the tree of life, Nature methods, 17, 495, 10.1038/s41592-020-0801-4 · DOI · Paper page
  27. 10.1038/ng0498-313 · DOI · Paper page
  28. 10.1016/S0969-2126(01)00157-5 · DOI · Paper page
  29. 1991, Streptococcal protein G. Gene structure and protein binding properties, Journal of Biological Chemistry, 266, 399, 10.1016/S0021-9258(18)52448-0 · DOI · Paper page
  30. 10.1016/j.coph.2015.08.002 · DOI
  31. 2019, Engineering the AAV capsid to evade immune responses, Current opinion in biotechnology, 60, 99, 10.1016/j.copbio.2019.01.002 · DOI · Paper page
  32. Georgios Mikos , Weitong Chen , and Junghae Suh . Machine learning identification of capsid mutations to improve AAV production fitness. bioRxiv, 2021. 3.2 · DOI
  33. Sam Sinai , Nina Jain , George M Church , and Eric D Kelsic . Generative AAV capsid diversification by latent interpolation. bioRxiv, 2021. 3.2 · DOI
  34. 2013, High-throughput screening for terpene-synthase-cyclization activity and directed evolution of a terpene synthase, Angewandte Chemie International Edition, 52, 5571, 10.1002/anie.201301362 · doi:10.1002/anie.201301362
  35. 10.1073/pnas.0510098103 · DOI · Paper page
  36. 10.1016/0167-7799(90)90125-H · DOI
  37. 10.1038/nbt1333 · DOI · Paper page
  38. 10.1073/pnas.0901417106 · DOI · Paper page
  39. Margaux M Pinney , Daniel A Mokhtari , Eyal Akiva , Filip Yabukarski , David M Sanchez , Ruibin Liang , Tzanko Doukov , Todd J Martinez , Patricia C Babbitt , and Daniel Herschlag . Parallel molecular mechanisms for enzyme temperature adaptation. Science, 371(6533), 2021. 3.3 · DOI · Paper page

Cited by

See all 125

Metadata

Authors
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K. K.
Venue
bioRxiv (Cold Spring Harbor Laboratory)
Publisher
openRxiv
Published
2021-11-11
OpenAlex
W3197005806
Cited by
125
References
26
Type
posted content
Language
en
Open access
green

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: FLIP: Benchmark tasks in fitness landscape inference for proteins. https://app.cochat.ai/papers/10.1101/2021.11.09.467890
The paper
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K. K. (2021). FLIP: Benchmark tasks in fitness landscape inference for proteins. bioRxiv (Cold Spring Harbor Laboratory). https://doi.org/10.1101/2021.11.09.467890

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.