Computational scoring and experimental evaluation of enzymes generated by neural networks
Open accessPublished version of a preprint
Abstract
In recent years, generative protein sequence models have been developed to sample novel sequences. However, predicting whether generated proteins will fold and function remains challenging. We evaluate a set of 20 diverse computational metrics to assess the quality of enzyme sequences produced by three contrasting generative models: ancestral sequence reconstruction, a generative adversarial network and a protein language model. Focusing on two enzyme families, we expressed and purified over 500 natural and generated sequences with 70–90% identity to the most similar natural sequences to benchmark computational metrics for predicting in vitro enzyme activity. Over three rounds of experiments, we developed a computational filter that improved the rate of experimental success by 50–150%. The proposed metrics and models will drive protein engineering research by serving as a benchmark for generative protein sequence models and helping to select active variants for experimental testing.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764 · see citing works
ColabFold - Making protein folding accessible to all
Mirdita, M., Schütze, K., Moriwaki, Y., Heo, L., Ovchinnikov, S., & Steinegger, M. (2021) · doi:10.1101/2021.08.15.456425 · Cited by 9,391 · see citing works
Accelerated Profile HMM Searches
Eddy, S. R. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002195 · Cited by 6,975 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · see citing works
Robust deep learning based protein sequence design using ProteinMPNN
Dauparas, J., Anishchenko, I., Bennett, N., Bai, H., Ragotte, R. J., Milles, L. F., Wicky, B. I. M., Courbet, A., de Haas, R. J., Bethel, N., et al. (2022) · doi:10.1101/2022.06.03.494563 · Cited by 2,107 · see citing works
Robust deep learning–based protein sequence design using ProteinMPNN
Dauparas, J., Anishchenko, I., Bennett, N., Bai, H., Ragotte, R. J., Milles, L. F., Wicky, B. I. M., Courbet, A., de Haas, R. J., Bethel, N., et al. (2022) · Science · doi:10.1126/science.add2187 · Cited by 2,070 · see citing works
Disease variant prediction with deep generative models of evolutionary data
Frazer, J., Notin, P., Dias, M., Gomez, A., Min, J. K., Brock, K., Gal, Y., & Marks, D. S. (2021) · Nature · doi:10.1038/s41586-021-04043-8 · Cited by 791 · see citing works
Local fitness landscape of the green fluorescent protein
Sarkisyan, K. S., Bolotin, D. A., Meer, M. V., Usmanova, D. R., Mishin, A. S., Sharonov, G. V., Ivankov, D. N., Bozhanova, N. G., Baranov, M. S., Soylemez, O., et al. (2016) · Nature · doi:10.1038/nature17995 · Cited by 591 · see citing works
ProGen2: Exploring the boundaries of protein language models
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.002 · Cited by 480 · see citing works
An evolution-based model for designing chorismate mutase enzymes
Russ, W. P., Figliuzzi, M., Stocker, C., Barrat-Charlaix, P., Socolich, M., Kast, P., Hilvert, D., Monasson, R., Cocco, S., Weigt, M., et al. (2020) · Science · doi:10.1126/science.aba3304 · Cited by 313 · see citing works
Language models generalize beyond natural proteins
Verkuil, R., Kabeli, O., Du, Y., Wicky, B. I. M., Milles, L. F., Dauparas, J., Baker, D., Ovchinnikov, S., Sercu, T., & Rives, A. (2022) · doi:10.1101/2022.12.21.521521 · Cited by 181 · cited 6× · see citing works
“The few reported results from biological assays 15,20,23,29,32,33 used different experimental systems, making comparisons difficult because many factors can contribute to poor expression and activity (Supplementary Table 1), ranging from mutations disrupting protein folding and stability 34 to…”
InfluentialZrimec, J., Börlin, C. S., Buric, F., Muhammad, A. S., Chen, R., Siewers, V., Verendel, V., Nielsen, J., Töpel, M., & Zelezniak, A. (2020) · Nature Communications · doi:10.1038/s41467-020-19921-4 · Cited by 174 · see citing works
Self-citationGene expression is encoded in all parts of a co-evolving interacting gene regulatory structure
Zrimec, J., Buric, F., Muhammad, A. S., Chen, R., Verendel, V., Töpel, M., & Zelezniak, A. (2019) · doi:10.1101/792531 · Cited by 173 · see citing works
Self-citation
Flagged first, then by citations
- Authors
- Johnson, S. R., Fu, X., Viknander, S., Goldin, C., Monaco, S., Zelezniak, A., & Yang, K. K.
- Affiliations
- New England Biolabs (United States) (US)
- Chalmers University of Technology (SE)
- Invitae (United States) (US)
- Vilnius University (LT)
- Microsoft (United States) (US)
- Corresponding author
- Zelezniak, Aleksej
- Yang, Kevin K.
- ORCID
- Venue
- Nature Biotechnology
- Published
- 2024-04-23
- Volume
- 43
- Issue
- 3
- Pages
- 396-405
- ISSN
- 0733-222X (ISSN-L)
- 1087-0156
- 1546-1696
- PMC
- PMC11919684
- OpenAlex
- W4395048825
- Cited by
- 99
- References
- 14
- Citations by year
- 2026: 32 · 2025: 39 · 2024: 25 · 2023: 3
- FWCI
- 13.07
- Type
- journal article
- Language
- en
- Update notices
- correction (2024-05-20)
- Open access
- hybrid
- doi.org
- Locations
- Nature Biotechnology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Chalmers Research (Chalmers University of Technology) (submittedVersion)
- Funders
- Vetenskapsrådet (2019-05356, 2022-06725)
- Svenska Forskningsrådet Formas (2019-01403)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Enzyme Catalysis and Immobilization
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Keywords
- enzyme design, ancestral sequence reconstruction, generative adversarial networks, protein language models, in vitro enzyme activity, protein engineering, benchmarking, enzyme families
- MeSH terms
- Algorithms, Amino Acid Sequence, Enzymes, Protein Engineering, Neural Networks, Computer, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
