Self-consistency test reveals systematic bias in programs for prediction change of stability upon mutation
Open accessAbstract
Motivation: Computational prediction of the effect of mutations on protein stability is used by researchers in many fields. The utility of the prediction methods is affected by their accuracy and bias. Bias, a systematic shift of the predicted change of stability, has been noted as an issue for several methods, but has not been investigated systematically. Presence of the bias may lead to misleading results especially when exploring the effects of combination of different mutations. Results: Here we use a protocol to measure the bias as a function of the number of introduced mutations. It is based on a self-consistency test of the reciprocity the effect of a mutation. An advantage of the used approach is that it relies solely on crystal structures without experimentally measured stability values. We applied the protocol to four popular algorithms predicting change of protein stability upon mutation, FoldX, Eris, Rosetta and I-Mutant, and found an inherent bias. For one program, FoldX, we manage to substantially reduce the bias using additional relaxation by Modeller. Authors using algorithms for predicting effects of mutations should be aware of the bias described here. Availability and implementation: All calculations were implemented by in-house PERL scripts. Supplementary information: Supplementary data are available at Bioinformatics online. Note: The article 10.1093/bioinformatics/bty348, published alongside this paper, also addresses the problem of biases in protein stability change predictions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing experimental sets are skewed toward mutations reducing protein-protein binding affinity.
“existing experimental sets are skewed toward mutations reducing protein-protein binding affinity”
From MutaBind2: predicting the impacts of single and multiple mutations on protein-protein interactions (2020) · cites this paper as Usmanova et al., 2018
Many computational methods fail to correctly evaluate the effects of mutations increasing binding affinity.
“many computational methods fail to correctly evaluate their effects”
From MutaBind2: predicting the impacts of single and multiple mutations on protein-protein interactions (2020) · cites this paper as Usmanova et al., 2018
The amplitude of the bias toward higher accuracy for mutations decreasing binding affinity increases with the number of introduced mutations in a protein.
“the amplitude of this bias increased with the number of introduced mutations in a protein (Usmanova et al., 2018).”
From MutaBind2: predicting the impacts of single and multiple mutations on protein-protein interactions (2020) · cites this paper as Usmanova et al., 2018
Existing computational methods perform much better for mutations decreasing than for mutations increasing binding affinity.
“Critically, existing computational methods perform much better for mutations decreasing than for mutations increasing binding affinity.”
From MutaBind2: predicting the impacts of single and multiple mutations on protein-protein interactions (2020) · cites this paper as Usmanova et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
Altschul, S. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.17.3389 · Cited by 73,302 · see citing works
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196 · see citing works
Comparative Protein Structure Modeling Using Modeller
Eswar, N., Webb, B., Martı́-Renom, M. A., Madhusudhan, M. S., Eramian, D., Shen, M., Pieper, U., & Šali, A. (2006) · Current Protocols in Bioinformatics · doi:10.1002/0471250953.bi0506s15 · Cited by 4,564 · see citing works
Comparative Protein Structure Modeling Using MODELLER
Webb, B., & Šali, A. (2016) · Current Protocols in Bioinformatics · doi:10.1002/cpbi.3 · Cited by 4,349 · see citing works
WHAT IF: A molecular modeling and drug design program
Vriend, G. (1990) · Journal of Molecular Graphics · doi:10.1016/0263-7855(90)80070-v · Cited by 3,476 · see citing works
The FoldX web server: an online force field
Schymkowitz, J., Borg, J., Stricher, F., Nys, R., Rousseau, F., & Serrano, L. (2005) · Nucleic Acids Research · doi:10.1093/nar/gki387 · Cited by 2,759 · see citing works
Tanford, C. (1968) · Advances in protein chemistry · doi:10.1016/s0065-3233(08)60401-5 · Cited by 2,135 · see citing works
I-Mutant2.0: predicting stability changes upon mutation from the protein sequence or structure
Capriotti, E., Fariselli, P., & Casadio, R. (2005) · Nucleic Acids Research · doi:10.1093/nar/gki375 · Cited by 2,058 · see citing works
Guerois, R., Nielsen, J. E., & Serrano, L. (2002) · Journal of Molecular Biology · doi:10.1016/s0022-2836(02)00442-4 · Cited by 1,728 · see citing works
Protein Structure Prediction Using Rosetta
Rohl, C. A., Strauss, C. E. M., Misura, K. M. S., & Baker, D. A. (2004) · Methods in enzymology on CD-ROM/Methods in enzymology · doi:10.1016/s0076-6879(04)83004-0 · Cited by 1,711 · see citing works
THE KINETICS OF FORMATION OF NATIVE RIBONUCLEASE DURING OXIDATION OF THE REDUCED POLYPEPTIDE CHAIN
Anfinsen, C. B., Haber, E., Sela, M., & White, F. H. (1961) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.47.9.1309 · Cited by 1,182 · see citing works
Comparative Protein Structure Modeling Using MODELLER
Webb, B., & Šali, A. (2014) · Current Protocols in Bioinformatics · doi:10.1002/0471250953.bi0506s47 · Cited by 1,141 · see citing works
Local fitness landscape of the green fluorescent protein
Sarkisyan, K. S., Bolotin, D. A., Meer, M. V., Usmanova, D. R., Mishin, A. S., Sharonov, G. V., Ivankov, D. N., Bozhanova, N. G., Baranov, M. S., Soylemez, O., et al. (2016) · Nature · doi:10.1038/nature17995 · Cited by 591 · see citing works
Self-citationGoldenzweig, A., Goldsmith, M., Hill, S. E., Gertman, O., Laurino, P., Ashani, Y., Dym, O., Unger, T., Albeck, S., Prilusky, J., et al. (2016) · Molecular Cell · doi:10.1016/j.molcel.2016.06.012 · Cited by 579 · see citing works
The Stability Effects of Protein Mutations Appear to be Universally Distributed
Tokuriki, N., Stricher, F., Schymkowitz, J. W. H., Serrano, L. G., & Tawfik, D. S. (2007) · Journal of Molecular Biology · doi:10.1016/j.jmb.2007.03.069 · Cited by 415 · see citing works
Eris: an automated estimator of protein stability
Yin, S., Ding, F., & Dokholyan, N. V. (2007) · Nature Methods · doi:10.1038/nmeth0607-466 · Cited by 405 · see citing works
Potapov, V., Cohen, M., & Schreiber, G. (2009) · Protein Engineering Design and Selection · doi:10.1093/protein/gzp030 · Cited by 349 · see citing works
A three-state prediction of single point mutations on protein stability changes
Capriotti, E., Fariselli, P., Rossi, I., & Casadio, R. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-s2-s6 · Cited by 328 · see citing works
High-throughput protein crystallization
Stevens, R. C. (2000) · Current Opinion in Structural Biology · doi:10.1016/s0959-440x(00)00131-7 · Cited by 235 · see citing works
Predicting free energy changes using structural ensembles
Benedix, A., Becker, C. M., de Groot, B. L., Caflisch, A., & Böckmann, R. A. (2009) · Nature Methods · doi:10.1038/nmeth0109-3 · Cited by 234 · see citing works
Protein Thermostability Calculations Using Alchemical Free Energy Simulations
Seeliger, D., & de Groot, B. L. (2010) · Biophysical Journal · doi:10.1016/j.bpj.2010.01.051 · Cited by 217 · see citing works
Predicting protein stability changes from sequences using support vector machines
Capriotti, E., Fariselli, P., Calabrese, R., & Casadio, R. (2005) · Bioinformatics · doi:10.1093/bioinformatics/bti1109 · Cited by 195 · see citing works
Frappier, V., Chartier, M., & Najmanovich, R. (2015) · Nucleic Acids Research · doi:10.1093/nar/gkv343 · Cited by 181 · see citing works
Contingency and entrenchment in protein evolution under purifying selection
Shah, P., McCandlish, D. M., & Plotkin, J. B. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1412933112 · Cited by 172 · see citing works
Gilis, D., & Rooman, M. J. (2000) · Protein Engineering Design and Selection · doi:10.1093/protein/13.12.849 · Cited by 161 · see citing works
INPS: predicting the impact of non-synonymous variations on protein stability from sequence
Fariselli, P., Martelli, P. L., Savojardo, C., & Casadio, R. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv291 · Cited by 147 · see citing works
Assessing predictors of changes in protein stability upon mutation using self-consistency
Thiltgen, G., & Goldstein, R. A. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046084 · Cited by 110 · see citing works
Accurate Stabilities of Laccase Mutants Predicted with a Modified FoldX Protocol
Christensen, N. J., & Kepp, K. P. (2012) · Journal of Chemical Information and Modeling · doi:10.1021/ci300398z · Cited by 81 · see citing works
Structure‐energy‐based predictions and network modelling of RASopathy and cancer missense mutations
Kiel, C., & Serrano, L. G. (2014) · Molecular Systems Biology · doi:10.1002/msb.20145092 · Cited by 64 · see citing works
Christensen, N. J., & Kepp, K. P. (2013) · Journal of Chemical Theory and Computation · doi:10.1021/ct4002152 · Cited by 46 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
MutaBind2: predicting the impacts of single and multiple mutations on protein-protein interactions
Zhang, N., Chen, Y., Lu, H., Zhao, F., Alvarez, R. V., Goncearenco, A., Panchenko, A. R., & Li, M. (2020) · iScience · doi:10.1016/j.isci.2020.100939 · Cited by 199 · see citing works
Rapid protein stability prediction using deep learning representations
Blaabjerg, L. M., Kassem, M. M., Good, L. L., Jonsson, N., Cagiada, M., Johansson, K. E., Boomsma, W., Stein, A., & Lindorff-Larsen, K. (2023) · eLife · doi:10.7554/elife.82593 · Cited by 177 · see citing works
Xu, Y., Liu, D., & Gong, H. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00716-2 · Cited by 41 · see citing works
On the biases in predictions of protein stability changes upon variations: the INPS test case
Montanucci, L., Savojardo, C., Martelli, P. L., Casadio, R., & Fariselli, P. (2019) · Bioinformatics · doi:10.1093/bioinformatics/bty979 · Cited by 33 · see citing works
Lv, X., Chen, J., Lu, Y., Chen, Z., Xiao, N., & Yang, Y. (2020) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.0c00064 · Cited by 26 · see citing works
Tsishyn, M., Hermans, P., Rooman, M., & Pucci, F. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf322 · Cited by 15 · see citing works
- Authors
- Usmanova, D. R., Bogatyreva, N. S., Ariño Bernad, J., Eremina, A. A., Gorshkova, A. A., Kanevskiy, G. M., Lonishin, L. R., Meister, A. V., Yakupova, A. G., Kondrashov, F. A., & Ivankov, D. N.
- Affiliations
- Columbia University Irving Medical Center (US)
- Russian Academy of Sciences (RU)
- Centre de Formació Interdisciplinària Superior, Universitat Politècnica de Catalunya, Barcelona, Spain
- University of Edinburgh (GB)
- Lomonosov Moscow State University (RU)
- Peter the Great St. Petersburg Polytechnic University (RU)
- Novosibirsk State University (RU)
- Institute of Science and Technology Austria (AT)
- Corresponding author
- Ivankov, Dmitry N
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2018-05-02
- Volume
- 34
- Issue
- 21
- Pages
- 3653-3658
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- PMC
- PMC6198859
- OpenAlex
- W2799513065
- Cited by
- 93
- References
- 30
- Citations by year
- 2026: 3 · 2025: 11 · 2024: 6 · 2023: 17 · 2022: 12 · 2021: 25 · 2020: 11 · 2019: 4 · 2018: 3
- FWCI
- 3.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Bioinformatics (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Institute of Science and Technology Austria (submittedVersion, cc-by-nc)
- Repositori digital de la UPF (Universitat Pompeu Fabra) (submittedVersion, cc-by-nc)
- Funders
- HHMI International Early Career Scientist Program (55007424)
- MINECO (BFU2015-68723-P)
- Spanish Ministry of Economy and Competitiveness Centro de Excelencia Severo Ochoa (SEV-2012-0208)
- Secretaria d'Universitats i Recerca del Departament d'Economia i Coneixement de la Generalitat’s AGAUR (SGR 0974)
- European Research Council
- European Union's Seventh Framework Programme (FP7/2007-2013)
- ERC (335980_EinME)
- Russian Scientific Foundation (RSF #14-24-00157)
- School of Molecular and Theoretical Biology
- Dynasty Foundation
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genomics and Rare Diseases
- Keywords
- mutation effects, prediction bias, self-consistency test, FoldX, Eris, Rosetta, MODELLER, crystal structure
- MeSH terms
- Algorithms, Mutation, Proteins, Software, Bias, Protein Stability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
