FLIP: Benchmark tasks in fitness landscape inference for proteins
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Machine learning could enable an unprecedented level of control in protein engineering for therapeutic and industrial applications. Critical to its use in designing proteins with desired properties, machine learning models must capture the protein sequence-function relationship, often termed fitness landscape . Existing bench-marks like CASP or CAFA assess structure and function predictions of proteins, respectively, yet they do not target metrics relevant for protein engineering. In this work, we introduce Fitness Landscape Inference for Proteins (FLIP), a benchmark for function prediction to encourage rapid scoring of representation learning for protein engineering. Our curated tasks, baselines, and metrics probe model generalization in settings relevant for protein engineering, e.g. low-resource and extrapolative. Currently, FLIP encompasses experimental data across adeno-associated virus stability for gene therapy, protein domain B1 stability and immunoglobulin binding, and thermostability from multiple protein families. In order to enable ease of use and future expansion to new tasks, all data are presented in a standard format. FLIP scripts and data are freely accessible at https://benchmark.protein.properties .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
UniProt: the universal protein knowledgebase in 2021
The UniProt Consortium, Bateman, A., Martin, M.-J., Orchard, S., Magrane, M., Agivetova, R., Ahmad, S., Alpi, E., Bowler-Barnett, E. H., Britto, R., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkaa1100 · Cited by 6,372 · cited 2×
“Rapid advances in genomic sequencing technology have led to an explosion of putative protein sequences [12, 13] deposited in databases like UniProt [14].”
BackgroundMethodMMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Steinegger, M., & Söding, J. (2017) · Nature Biotechnology · doi:10.1038/nbt.3988 · Cited by 4,840
Wu, F. Y. (1982) · Reviews of Modern Physics · doi:10.1103/revmodphys.54.235 · Cited by 3,570
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 14×
“ESM-1v [16] 750M parameter transformer pretrained on UniRef90.”
BackgroundMethodInfluentialPerformance of neural network basecalling tools for Oxford Nanopore sequencing
Wick, R. R., Judd, L. M., & Holt, K. E. (2019) · Genome Biology · doi:10.1186/s13059-019-1727-y · Cited by 3,221
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · cited 2×
“Recent efforts in sequence-function prediction [15, 16] have sought to leverage the information in these unlabeled sequences through pretraining and fine-tuning, and have successfully engineered proteins with brighter fluorescence and high catalytic efficiency [17].”
BackgroundMorcos, F., Pagnani, A., Lunt, B., Bertolino, A., Marks, D. S., Sander, C., Zecchina, R., Onuchic, J. N., Hwa, T., & Weigt, M. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1111471108 · Cited by 1,434
A large-scale evaluation of computational protein function prediction
Radivojac, P., Clark, W. T., Oron, T. R., Schnoes, A. M., Wittkop, T., Sokolov, A., Graim, K., Funk, C., Verspoor, K., Ben-Hur, A., et al. (2013) · Nature Methods · doi:10.1038/nmeth.2340 · Cited by 1,028
Research works printed but not linked: 39
- 10.1002/bit.22077 · doi:10.1002/bit.22077
- 10.1016/j.copbio.2009.07.001 · DOI · Paper page
- 10.1080/10408398.2014.1000481 · doi:10.1080/10408398.2014.1000481
- 10.1126/science.aay8484 · DOI · Paper page
- 10.1093/bib/bbaa150 · DOI
- 10.1038/nrm2805 · DOI · Paper page
- 10.1093/protein/gzg077 · DOI
- 2019, Machine-learning-guided directed evolution for protein engineering, Nature methods, 16, 687, 10.1038/s41592-019-0496-6 · DOI · Paper page
- 10.1073/pnas.1901979116 · DOI · Paper page
- 10.1101/2020.12.04.408955 · doi:10.1101/2020.12.04.408955
- Joshua Meier , Roshan Rao , Robert Verkuil , Jason Liu , Tom Sercu , and Alexander Rives . Language models enable zero-shot prediction of the effects of mutations on protein function. bioRxiv, 2021. 1, 2, 3, 5, 5 · DOI · Paper page
- Jonathan Frazer , Pascal Notin , Mafalda Dias , Aidan Gomez , Kelly Brock , Yarin Gal , and Debora Marks . Large-scale clinical interpretation of genetic variants using evolutionary data and deep learning. bioRxiv, 2020. · DOI
- 10.1186/s12859-019-2669-9 · DOI · Paper page
- 1995, Masthead, Proteins: Structure, Function, and Bioinformatics, 23, fmi, 10.1002/prot.340230301 · doi:10.1002/prot.340230301
- Amy X Lu , Haoran Zhang , Marzyeh Ghassemi , and Alan M Moses . Self-supervised contrastive learning of protein representations by mutual information maximization. BioRxiv, 2020. 1
- 2020, Is transfer learning necessary for protein landscape prediction?, arXiv preprint
- Chloe Hsu , Hunter Nisonoff , Clara Fannjiang , and Jennifer Listgarten . Combining evolutionary and assay-labelled data for protein fitness prediction. bioRxiv, 2021. 1 · DOI
- 10.1016/j.tig.2013.09.005 · DOI · Paper page
- 10.1016/j.cels.2017.11.003 · DOI · Paper page
- Céline Marquet , Michael Heinzinger , Tobias Olenyi , Christian Dallago , Michael Bernhofer , Kyra Erckert , and Burkhard Rost . Embeddings from protein language models predict conservation and variant effects. 2021. 2 · DOI · Paper page
- 2021, Embeddings from deep learning transfer GO annotations beyond homology, Scientific reports, 11, 1
- 10.7554/eLife.16965 · DOI · Paper page
- 10.1021/ja058292x · DOI
- 2021, Deep diversification of an AAV capsid protein by machine learning, Nature Biotechnology, 39, 691, 10.1038/s41587-020-00793-4 · DOI · Paper page
- 10.1038/s41564-018-0356-7 · DOI
- 2020, Meltome atlas– thermal proteome stability across the tree of life, Nature methods, 17, 495, 10.1038/s41592-020-0801-4 · DOI · Paper page
- 10.1038/ng0498-313 · DOI · Paper page
- 10.1016/S0969-2126(01)00157-5 · DOI · Paper page
- 1991, Streptococcal protein G. Gene structure and protein binding properties, Journal of Biological Chemistry, 266, 399, 10.1016/S0021-9258(18)52448-0 · DOI · Paper page
- 10.1016/j.coph.2015.08.002 · DOI
- 2019, Engineering the AAV capsid to evade immune responses, Current opinion in biotechnology, 60, 99, 10.1016/j.copbio.2019.01.002 · DOI · Paper page
- Georgios Mikos , Weitong Chen , and Junghae Suh . Machine learning identification of capsid mutations to improve AAV production fitness. bioRxiv, 2021. 3.2 · DOI
- Sam Sinai , Nina Jain , George M Church , and Eric D Kelsic . Generative AAV capsid diversification by latent interpolation. bioRxiv, 2021. 3.2 · DOI
- 2013, High-throughput screening for terpene-synthase-cyclization activity and directed evolution of a terpene synthase, Angewandte Chemie International Edition, 52, 5571, 10.1002/anie.201301362 · doi:10.1002/anie.201301362
- 10.1073/pnas.0510098103 · DOI · Paper page
- 10.1016/0167-7799(90)90125-H · DOI
- 10.1038/nbt1333 · DOI · Paper page
- 10.1073/pnas.0901417106 · DOI · Paper page
- Margaux M Pinney , Daniel A Mokhtari , Eyal Akiva , Filip Yabukarski , David M Sanchez , Ruibin Liang , Tzanko Doukov , Todd J Martinez , Patricia C Babbitt , and Daniel Herschlag . Parallel molecular mechanisms for enzyme temperature adaptation. Science, 371(6533), 2021. 3.3 · DOI · Paper page
Cited by
ProGen2: Exploring the boundaries of protein language models
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.002 · Cited by 480
Self-citationEfficient evolution of human antibodies from general protein language models
Hie, B. L., Shanker, V. R., Xu, D., Bruun, T. U. J., Weidenbacher, P. A., Tang, S., Wu, W., Pak, J. E., & Kim, P. S. (2024) · Nature Biotechnology · doi:10.1038/s41587-023-01763-2 · Cited by 449
Learning inverse folding from millions of predicted structures
Hsu, C., Verkuil, R., Liu, J., Lin, Z., Hie, B., Sercu, T., Lerer, A., & Rives, A. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.10.487779 · Cited by 416
Bilingual language model for protein sequence and structure
Heinzinger, M., Weissenow, K., Sanchez, J. G., Henkel, A., Mirdita, M., Steinegger, M., & Rost, B. (2024) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqae150 · Cited by 311
Machine learning for functional protein design
Notin, P., Rollins, N., Gal, Y., Sander, C., & Marks, D. (2024) · Nature Biotechnology · doi:10.1038/s41587-024-02127-0 · Cited by 270
Metadata
- Authors
- Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K. K.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2021-11-11
- OpenAlex
- W3197005806
- Cited by
- 125
- References
- 26
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: FLIP: Benchmark tasks in fitness landscape inference for proteins. https://app.cochat.ai/papers/10.1101/2021.11.09.467890
- The paper
- Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K. K. (2021). FLIP: Benchmark tasks in fitness landscape inference for proteins. bioRxiv (Cold Spring Harbor Laboratory). https://doi.org/10.1101/2021.11.09.467890
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.