Quantification of biases in predictions of protein stability changes upon mutations
Open accessPublished version of a preprint
Abstract
Motivation: Bioinformatics tools that predict protein stability changes upon point mutations have made a lot of progress in the last decades and have become accurate and fast enough to make computational mutagenesis experiments feasible, even on a proteome scale. Despite these achievements, they still suffer from important issues that must be solved to allow further improving their performances and utilizing them to deepen our insights into protein folding and stability mechanisms. One of these problems is their bias toward the learning datasets which, being dominated by destabilizing mutations, causes predictions to be better for destabilizing than for stabilizing mutations. Results: We thoroughly analyzed the biases in the prediction of folding free energy changes upon point mutations (ΔΔG0) and proposed some unbiased solutions. We started by constructing a dataset Ssym of experimentally measured ΔΔG0s with an equal number of stabilizing and destabilizing mutations, by collecting mutations for which the structure of both the wild-type and mutant protein is available. On this balanced dataset, we assessed the performances of 15 widely used ΔΔG0 predictors. After the astonishing observation that almost all these methods are strongly biased toward destabilizing mutations, especially those that use black-box machine learning, we proposed an elegant way to solve the bias issue by imposing physical symmetries under inverse mutations on the model structure, which we implemented in PoPMuSiCsym. This new predictor constitutes an efficient trade-off between accuracy and absence of biases. Some final considerations and suggestions for further improvement of the predictors are discussed. Supplementary information: Supplementary data are available at Bioinformatics online. Note: The article 10.1093/bioinformatics/bty340/, published alongside this paper, also addresses the problem of biases in protein stability change predictions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many computational methods fail to correctly evaluate the effects of mutations increasing binding affinity.
“many computational methods fail to correctly evaluate their effects”
From MutaBind2: predicting the impacts of single and multiple mutations on protein-protein interactions (2020) · cites this paper as Pucci et al., 2018
Existing computational methods perform much better for mutations decreasing than for mutations increasing binding affinity.
“Critically, existing computational methods perform much better for mutations decreasing than for mutations increasing binding affinity.”
From MutaBind2: predicting the impacts of single and multiple mutations on protein-protein interactions (2020) · cites this paper as Pucci et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
I-TASSER: a unified platform for automated protein structure and function prediction
Roy, A., Küçükural, A., & Zhang, Y. (2010) · Nature Protocols · doi:10.1038/nprot.2010.5 · Cited by 6,093 · see citing works
Hawkins, D. M. (2004) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci0342472 · Cited by 2,522 · see citing works
I-Mutant2.0: predicting stability changes upon mutation from the protein sequence or structure
Capriotti, E., Fariselli, P., & Casadio, R. (2005) · Nucleic Acids Research · doi:10.1093/nar/gki375 · Cited by 2,058 · see citing works
(2010) · doi:10.5555/1756006.1859921 · Cited by 1,924 · see citing works
Guerois, R., Nielsen, J. E., & Serrano, L. (2002) · Journal of Molecular Biology · doi:10.1016/s0022-2836(02)00442-4 · Cited by 1,728 · see citing works
The Rosetta All-Atom Energy Function for Macromolecular Modeling and Design
Alford, R. F., Leaver-Fay, A., Jeliazkov, J. R., O’Meara, M. J., DiMaio, F. P., Park, H., Shapovalov, M. V., Renfrew, P. D., Mulligan, V. K., Kappel, K., et al. (2017) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.7b00125 · Cited by 1,532 · see citing works
The coming of age of de novo protein design
Huang, P.-S., Boyken, S. E., & Baker, D. (2016) · Nature · doi:10.1038/nature19946 · Cited by 1,478 · see citing works
Deep mutational scanning: a new style of protein science
Fowler, D. M., & Fields, S. (2014) · Nature Methods · doi:10.1038/nmeth.3027 · Cited by 1,287 · see citing works
Prediction of protein stability changes for single‐site mutations using support vector machines
Cheng, J. J., Randall, A. Z., & Baldi, P. (2005) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.20810 · Cited by 1,165 · see citing works
mCSM: Predicting the effects of mutations in proteins using graph-based signatures
Pires, D. E. V., Ascher, D. B., & Blundell, T. L. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btt691 · Cited by 1,066 · see citing works
Pires, D. E. V., Ascher, D. B., & Blundell, T. L. (2014) · Nucleic Acids Research · doi:10.1093/nar/gku411 · Cited by 932 · see citing works
CUPSAT: prediction of protein stability upon point mutations
Parthiban, V., Gromiha, M. M., & Schomburg, D. (2006) · Nucleic Acids Research · doi:10.1093/nar/gkl190 · Cited by 697 · see citing works
Kellogg, E. H., Leaver‐Fay, A., & Baker, D. A. (2010) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.22921 · Cited by 655 · see citing works
Dehouck, Y., Kwasigroch, J. M., Gilis, D., & Rooman, M. J. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-151 · Cited by 534 · see citing works
Self-citationSDM: a server for predicting effects of mutations on protein stability
Pandurangan, A. P., Ochoa‐Montaño, B., Ascher, D. B., & Blundell, T. L. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkx439 · Cited by 501 · see citing works
Dehouck, Y., Grosfils, A., Folch, B., Gilis, D., Bogaerts, P., & Rooman, M. J. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp445 · Cited by 409 · see citing works
Self-citationProTherm, version 4.0: thermodynamic database for proteins and mutants
Bava, K. A. (2003) · Nucleic Acids Research · doi:10.1093/nar/gkh082 · Cited by 367 · see citing works
Potapov, V., Cohen, M., & Schreiber, G. (2009) · Protein Engineering Design and Selection · doi:10.1093/protein/gzp030 · Cited by 349 · see citing works
Performance of protein stability predictors
Khan, S., & Vihinen, M. (2010) · Human Mutation · doi:10.1002/humu.21242 · Cited by 324 · see citing works
STRUM: structure-based prediction of protein stability changes upon single-point mutation
Quan, L., Lv, Q., & Zhang, Y. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw361 · Cited by 324 · see citing works
iStable: off-the-shelf predictor integration for predicting protein stability changes
Chen, C.-W., Lin, J., & Chu, Y.-W. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-s2-s5 · Cited by 236 · see citing works
MAESTROweb: a web server for structure-based protein stability prediction
Laimer, J., Hiebl-Flach, J., Lengauer, D., et al. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btv769 · Cited by 193 · see citing works
Montanucci, L., Capriotti, E., Frank, Y., Ben-Tal, N., & Fariselli, P. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-2923-1 · Cited by 167 · see citing works
Masso, M., & Vaisman, I. I. (2008) · Bioinformatics · doi:10.1093/bioinformatics/btn353 · Cited by 160 · see citing works
INPS: predicting the impact of non-synonymous variations on protein stability from sequence
Fariselli, P., Martelli, P. L., Savojardo, C., & Casadio, R. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv291 · Cited by 147 · see citing works
Giollo, M., Martín, A. J. M., Walsh, I., Ferrari, C., & Tosatto, S. C. E. (2014) · BMC Genomics · doi:10.1186/1471-2164-15-s4-s7 · Cited by 121 · see citing works
Assessing predictors of changes in protein stability upon mutation using self-consistency
Thiltgen, G., & Goldstein, R. A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046084 · Cited by 110 · see citing works
Yarmishyn, A. A., Batagov, A. O., Tan, J. Z., Sundaram, G. M., Sampath, P., Kuznetsov, V. A., & Kurochkin, I. V. (2014) · BMC Genomics · doi:10.1186/1471-2164-15-s9-s7 · Cited by 102 · see citing works
Pandurangan, A. P., & Blundell, T. L. (2019) · Protein Science · doi:10.1002/pro.3774 · Cited by 99 · see citing works
Bioinformatics analysis of the epitope regions for norovirus capsid protein
Chen, L., Wu, D. J., Ji, L., Wu, X., Xu, D., Cao, Z., & Han, J. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-s4-s5 · Cited by 93 · see citing works
de novo computational enzyme design
Zanghellini, A. (2014) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2014.03.002 · Cited by 87 · see citing works
Masso, M., & Vaisman, I. I. (2014) · Advances in Bioinformatics · doi:10.1155/2014/278385 · Cited by 64 · see citing works
Computational protein design: a review
Ivan Coluzza, B. (2017) · Journal of Physics Condensed Matter · doi:10.1088/1361-648x/aa5c76 · Cited by 53 · see citing works
High-quality Thermodynamic Data on the Stability Changes of Proteins Upon Single-site Mutations
Pucci, F., Bourgeas, R., & Rooman, M. J. (2016) · Journal of Physical and Chemical Reference Data · doi:10.1063/1.4947493 · Cited by 41 · see citing works
Self-citationSymmetry Principles in Optimization Problems: an application to Protein Stability Prediction
Pucci, F., Bernaerts, K. V., Teheux, F., Gilis, D., & Rooman, M. J. (2015) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2015.05.068 · Cited by 31 · see citing works
Self-citationWatson, M. D., Monroe, J. G., & Raleigh, D. P. (2017) · The Journal of Physical Chemistry B · doi:10.1021/acs.jpcb.7b05684 · Cited by 26 · see citing works
Dissecting the evolvability landscape of the CalB active site toward aromatic substrates
Santos, Y. L. D. L., Chew-Fajardo, Y. L., Brault, G., & Doucet, N. (2019) · Scientific Reports · doi:10.1038/s41598-019-51940-0 · Cited by 21 · see citing works
Watson, M. D., Monroe, J. G., & Raleigh, D. P. (2017) · The Journal of Physical Chemistry · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Transfer learning to leverage larger datasets for improved prediction of protein stability changes
Dieckhaus, H., Brocidiacono, M., Randolph, N. Z., & Kuhlman, B. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314853121 · Cited by 205 · see citing works
MutaBind2: predicting the impacts of single and multiple mutations on protein-protein interactions
Zhang, N., Chen, Y., Lu, H., Zhao, F., Alvarez, R. V., Goncearenco, A., Panchenko, A. R., & Li, M. (2020) · iScience · doi:10.1016/j.isci.2020.100939 · Cited by 199 · see citing works
Rapid protein stability prediction using deep learning representations
Blaabjerg, L. M., Kassem, M. M., Good, L. L., Jonsson, N., Cagiada, M., Johansson, K. E., Boomsma, W., Stein, A., & Lindorff-Larsen, K. (2023) · eLife · doi:10.7554/elife.82593 · Cited by 177 · see citing works
Improving the Accuracy of Protein Thermostability Predictions for Single Point Mutations
Duan, J., Lupyan, D., & Wang, L. (2020) · Biophysical Journal · doi:10.1016/j.bpj.2020.05.020 · Cited by 63 · see citing works
Xu, Y., Liu, D., & Gong, H. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00716-2 · Cited by 41 · see citing works
Scarabelli, G., Oloo, E. O., Maier, J. K. X., & Rodriguez-Granillo, A. (2022) · Journal of Molecular Biology · doi:10.1016/j.jmb.2021.167375 · Cited by 33 · see citing works
On the biases in predictions of protein stability changes upon variations: the INPS test case
Montanucci, L., Savojardo, C., Martelli, P. L., Casadio, R., & Fariselli, P. (2019) · Bioinformatics · doi:10.1093/bioinformatics/bty979 · Cited by 33 · see citing works
Lv, X., Chen, J., Lu, Y., Chen, Z., Xiao, N., & Yang, Y. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c00064 · Cited by 26 · see citing works
Tsishyn, M., Hermans, P., Rooman, M., & Pucci, F. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf322 · Cited by 15 · see citing works
Self-citation
- Authors
- Pucci, F., Bernaerts, K. V., Kwasigroch, J. M., & Rooman, M.
- Affiliations
- Université Libre de Bruxelles (BE)
- Corresponding author
- Pucci, Fabrizio
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-04-26
- Volume
- 34
- Issue
- 21
- Pages
- 3659-3665
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- OpenAlex
- W2794503008
- Cited by
- 169
- References
- 38
- Citations by year
- 2026: 5 · 2025: 24 · 2024: 22 · 2023: 23 · 2022: 24 · 2021: 36 · 2020: 20 · 2019: 11 · 2018: 3 · 2017: 1
- FWCI
- 6.24
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Bioinformatics (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) (submittedVersion, other-oa)
- Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) (submittedVersion)
- Funders
- Fund for Scientific Research (FNRS)
- M.R. Research Director
- FNRS
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genomics and Rare Diseases
- Keywords
- protein stability prediction, point mutations, prediction bias, physical symmetries, protein folding free energy, computational mutagenesis
- MeSH terms
- Mutation, Proteins, Bias, Protein Folding, Protein Stability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
