A critical review of five machine learning-based algorithms for predicting protein stability changes upon mutation
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
A number of machine learning (ML)-based algorithms have been proposed for predicting mutation-induced stability changes in proteins. In this critical review, we used hypothetical reverse mutations to evaluate the performance of five representative algorithms and found all of them suffer from the problem of overfitting. This approach is based on the fact that if a wild-type protein is more stable than a mutant protein, then the same mutant is less stable than the wild-type protein. We analyzed the underlying issues and suggest that the main causes of the overfitting problem include that the numbers of training cases were too small, and the features used in the models were not sufficiently informative for the task. We make recommendations on how to avoid overfitting in this important research area and improve the reliability and robustness of ML-based algorithms in general.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scalable molecular dynamics with NAMD
Phillips, J. C., Braun, R., Wang, W., Gumbart, J., Tajkhorshid, E., Villa, E., Chipot, C., Skeel, R. D., Kalé, L., & Schulten, K. (2005) · Journal of Computational Chemistry · doi:10.1002/jcc.20289 · Cited by 16,914
The FoldX web server: an online force field
Schymkowitz, J., Borg, J., Stricher, F., Nys, R., Rousseau, F., & Serrano, L. (2005) · Nucleic Acids Research · doi:10.1093/nar/gki387 · Cited by 2,759
InterPro in 2019: improving coverage, classification and access to protein sequence annotations
Mitchell, A. L., Attwood, T. K., Babbitt, P. C., Blum, M., Bork, P., Bridge, A., Brown, S. D., Chang, H.-Y., El-Gebali, S., Fraser, M. I., et al. (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1100 · Cited by 1,426
DynaMut: predicting the impact of mutations on protein conformation, flexibility and stability
Rodrigues, C. H., Pires, D. E., & Ascher, D. B. (2018) · Nucleic Acids Research · doi:10.1093/nar/gky300 · Cited by 1,086
Pires, D. E. V., Ascher, D. B., & Blundell, T. L. (2014) · Nucleic Acids Research · doi:10.1093/nar/gku411 · Cited by 932
Zhou, H., & Zhou, Y. (2002) · Protein Science · doi:10.1110/ps.0217002 · Cited by 902 · cited 1×
“Force field-based algorithms were also introduced decades ago and are still being actively researched [13–15].”
MethodLessons from the lysozyme of phage T4
Baase, W. A., Liu, L., Tronrud, D. E., & Matthews, B. W. (2010) · Protein Science · doi:10.1002/pro.344 · Cited by 199
Assessing predictors of changes in protein stability upon mutation using self-consistency
Thiltgen, G., & Goldstein, R. A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046084 · Cited by 110 · cited 1×
“However,the performance of these algorithms has been challenged as several performance reviews failed to reproduce the high accuracies found by the authors [29, 30].”
Result
Cited by
PremPS: Predicting the impact of missense mutations on protein stability
Chen, Y., Lu, H., Zhang, N., Zhu, Z., Wang, S., & Li, M. (2020) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1008543 · Cited by 260
Rapid protein stability prediction using deep learning representations
Blaabjerg, L. M., Kassem, M. M., Good, L. L., Jonsson, N., Cagiada, M., Johansson, K. E., Boomsma, W., Stein, A., & Lindorff-Larsen, K. (2023) · eLife · doi:10.7554/elife.82593 · Cited by 177
ThermoMutDB: a thermodynamic database for missense mutations
Xavier, J. S., Nguyen, T.-B., Karmarkar, M., Portelli, S., Rezende, P. M., Velloso, J. P. L., Ascher, D. B., & Pires, D. E. V. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkaa925 · Cited by 101
“However, this database has not been updated for 7 years and many errors have been identified previously (29,30), limiting both previous methods and future developments.”
MethodBenchmarking deep learning methods for predicting CRISPR/Cas9 sgRNA on- and off-target activities
Zhang, G., Luo, Y., Dai, X., & Dai, Z. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad333 · Cited by 47
Lv, X., Chen, J., Lu, Y., Chen, Z., Xiao, N., & Yang, Y. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c00064 · Cited by 26
Metadata
- Authors
- Fang, J.
- Venue
- Briefings in Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-07-05
- Volume
- 21
- Issue
- 4
- Pages
- 1285-1292
- ISSN
- 1467-5463 (ISSN-L)
- 1477-4054
- OpenAlex
- W2955704392
- Cited by
- 130
- References
- 8
- FWCI
- 4.65
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.