Problems in variation interpretation guidelines and in their implementation in computational tools
Open accessAbstract
BACKGROUND: ACMG/AMP and AMP/ASCO/CAP have released guidelines for variation interpretation, and ESHG for diagnostic sequencing. These guidelines contain recommendations including the use of computational prediction methods. The guidelines per se and the way they are implemented cause some problems. METHODS: Logical reasoning based on domain knowledge. RESULTS: According to the guidelines, several methods have to be used and they have to agree. This means that the methods with the poorest performance overrule the better ones. The choice of the prediction method(s) should be made by experts based on systematic benchmarking studies reporting all the relevant performance measures. Currently variation interpretation methods have been applied mainly to amino acid substitutions and splice site variants; however, predictors for some other types of variations are available and there will be tools for new application areas in the near future. Common problems in prediction method usage are discussed. The number of features used for method training or the number of variation types predicted by a tool are not indicators of method performance. Many published gene, protein or disease-specific benchmark studies suffer from too small dataset rendering the results useless. In the case of binary predictors, equal number of positive and negative cases is beneficial for training, the imbalance has to be corrected for performance assessment. Predictors cannot be better than the data they are based on and used for training and testing. Minor allele frequency (MAF) can help to detect likely benign cases, but the recommended MAF threshold is apparently too high. The fact that many rare variants are disease-causing or -related does not mean that rare variants in general would be harmful. How large a portion of the tested variants a tool can predict (coverage) is not a quality measure. CONCLUSION: Methods used for variation interpretation have to be carefully selected. It should be possible to use only one predictor, with proven good performance or a limited number of complementary predictors with state-of-the-art performance. Bear in mind that diseases and pathogenicity have a continuum and variants are not dichotomic i.e. either pathogenic or benign, either.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Score intervals defined by tool developers, and ACMG/AMP recommendations that require the consensus of multiple predictors, lack quantitative support.
“However, score intervals defined by tool developers, and ACMG/AMP recommendations that require the consensus of multiple predictors, lack quantitative support.”
From Calibration of computational tools for missense variant pathogenicity classification and ClinGen recommendations for PP3/BP4 criteria (2022) · cites this paper as [6]
The sections “Problems with the guidelines and in their implementation” and “Suggestions” say ACMG/AMP requires multiple predictors to agree and criticize that requirement because poorly performing methods can dictate the result. Our copy does not establish that the recommendation lacks quantitative support, and the passages found do not address score intervals defined by tool developers. Page numbers were not available in our copy. (LLM, conf=0.81)
Experimental — accuracy not yet measuredIn practice, many computational methods overlap and thus do not offer independent assessment of pathogenicity.
“In practice, many methods overlap and thus do not offer independent assessment of pathogenicity.4,6”
From Calibration of computational tools for missense variant pathogenicity classification and ClinGen recommendations for PP3/BP4 criteria (2022) · cites this paper as [6]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Richards, S., Aziz, N., Bale, S., Bick, D., Das, S., Gastier-Foster, J., Grody, W. W., Hegde, M., Lyon, E., Spector, E., et al. (2015) · Genetics in Medicine · doi:10.1038/gim.2015.30 · Cited by 32,594 · see citing works
Analysis of protein-coding genetic variation in 60,706 humans
Exome Aggregation Consortium, Lek, M., Karczewski, K. J., Minikel, E. V., Samocha, K. E., Banks, E., Fennell, T., O’Donnell-Luria, A. H., Ware, J. S., Hill, A. J., et al. (2016) · Nature · doi:10.1038/nature19057 · Cited by 9,587 · see citing works
Standards and Guidelines for the Interpretation and Reporting of Sequence Variants in Cancer
Li, M. M., Datto, M., Duncavage, E. J., Kulkarni, S., Lindeman, N. I., Roy, S., Tsimberidou, A. M., Vnencak‐Jones, C. L., Wolff, D. J., Younes, A., et al. (2016) · Journal of Molecular Diagnostics · doi:10.1016/j.jmoldx.2016.10.002 · Cited by 2,054 · see citing works
Recommendations for interpreting the loss of function PVS1 ACMG/AMP variant criterion
Tayoun, A. N. A., Pesaran, T. F., DiStefano, M. T., Oza, A. M., Rehm, H. L., Biesecker, L. G., Harrison, S. M., & ClinGen Sequence Variant Interpretation Working Group (ClinGen SVI) (2018) · Human Mutation · doi:10.1002/humu.23626 · Cited by 893 · see citing works
PredictSNP: Robust and Accurate Consensus Classifier for Prediction of Disease-Related Mutations
Bendl, J. M., Štourač, J., Šalanda, O., Pavelka, A., Wieben, E. D., Zendulka, J., Brezovský, J., & Damborský, J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003440 · Cited by 850 · see citing works
dbNSFP: A lightweight database of human nonsynonymous SNPs and their functional predictions
Liu, X., Jian, X., & Boerwinkle, E. (2011) · Human Mutation · doi:10.1002/humu.21517 · Cited by 816 · see citing works
Sherloc: a comprehensive refinement of the ACMG–AMP variant classification criteria
Nykamp, K., Anderson, M., Powers, M. P., Garcia, J., Herrera, B., Ho, Y.-Y., Kobayashi, Y., Patil, N., Thusberg, J., Westbrook, M. J., et al. (2017) · Genetics in Medicine · doi:10.1038/gim.2017.37 · Cited by 769 · see citing works
Kryukov, G. V., Pennacchio, L. A., & Sunyaev, S. R. (2007) · The American Journal of Human Genetics · doi:10.1086/513473 · Cited by 598 · see citing works
Amendola, L. M., Jarvik, G. P., Leo, M. C., McLaughlin, H. M., Akkari, Y. M. N., Amaral, M. D., Berg, J. S., Biswas, S., Bowling, K. M., Conlin, L. K., et al. (2016) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2016.03.024 · Cited by 556 · see citing works
Performance of mutation pathogenicity prediction methods on missense variants
Thusberg, J., Olatubosun, A., & Vihinen, M. (2011) · Human Mutation · doi:10.1002/humu.21445 · Cited by 537 · see citing works
Self-citationGuidelines for diagnostic next-generation sequencing
MATTHIJS, G. L., Souche, E. L., Alders, M. M., et al. (2015) · European Journal of Human Genetics · doi:10.1038/ejhg.2015.226 · Cited by 516 · see citing works
In silico prediction of splice-altering single nucleotide variants in the human genome
Jian, X., Boerwinkle, E., & Liu, X.-M. (2014) · Nucleic Acids Research · doi:10.1093/nar/gku1206 · Cited by 505 · see citing works
Using high-resolution variant frequencies to empower clinical genome interpretation
Whiffin, N., Minikel, E., Walsh, R., O’Donnell-Luria, A. H., Karczewski, K., Ing, A. Y., Barton, P. J. R., Funke, B., Cook, S. A., MacArthur, D., et al. (2017) · Genetics in Medicine · doi:10.1038/gim.2017.26 · Cited by 441 · cited 2× · see citing works
“…have been suggested to the ACMG/AMP guidelines including changes to criterion PVS1 (Abou Tayoun et al., 2018), to the entire process (Nykamp et al., 2017), allele frequency thresholds (Kelly et al., 2018; Kobayashi et al., 2017; Nykamp et al., 2017; Rim et al., 2019; Whiffin et al., 2017), etc.”
BackgroundErratum: Guidelines for diagnostic next-generation sequencing
Matthijs, G., Souche, E. L., Alders, M. M., Corveleyn, A., Eck, S. H., Feenstra, I., Race, V., Sistermans, E. A., Sturm, M., Weiss, M. M., et al. (2016) · European Journal of Human Genetics · doi:10.1038/ejhg.2016.63 · Cited by 429 · see citing works
Kelly, M., Caleshu, C. A., Morales, A., Buchan, J. G., Wolf, Z. T., Harrison, S. M., Cook, S. A., Dillon, M. W., Garcia, J., Haverfield, E. V., et al. (2018) · Genetics in Medicine · doi:10.1038/gim.2017.218 · Cited by 365 · see citing works
Underdiagnosis of Hereditary Breast Cancer: Are Genetic Testing Guidelines a Tool or an Obstacle?
Beitsch, P. D., Whitworth, P. W., Hughes, K. S., Patel, R. P., Rosen, B. P., Compagnoni, G., Baron, P. L., Simmons, R. M., Smith, L. A., Grady, I., et al. (2018) · Journal of Clinical Oncology · doi:10.1200/jco.18.01631 · Cited by 362 · see citing works
Grimm, D. G., Azencott, C., Aicheler, F., Gieraths, U., MacArthur, D. G., Samocha, K. E., Cooper, D. N., Stenson, P. D., Daly, M. J., Smoller, J. W., et al. (2015) · Human Mutation · doi:10.1002/humu.22768 · Cited by 356 · see citing works
Potapov, V., Cohen, M., & Schreiber, G. (2009) · Protein Engineering Design and Selection · doi:10.1093/protein/gzp030 · Cited by 349 · see citing works
Performance of protein stability predictors
Khan, S., & Vihinen, M. (2010) · Human Mutation · doi:10.1002/humu.21242 · Cited by 324 · see citing works
Self-citationVihinen, M. (2012) · BMC Genomics · doi:10.1186/1471-2164-13-s4-s2 · Cited by 304 · see citing works
Self-citationMassively Parallel Functional Analysis of BRCA1 RING Domain Variants
Starita, L. M., Young, D. L., Islam, M. M., Kitzman, J. O., Gullingsrud, J., Hause, R. J., Fowler, D. M., Parvin, J. D., Shendure, J., & Fields, S. (2015) · Genetics · doi:10.1534/genetics.115.175802 · Cited by 300 · see citing works
DNA damage is a pervasive cause of sequencing errors, directly confounding variant identification
Chen, L., Liu, P., Evans, T. C., & Ettwiller, L. M. (2017) · Science · doi:10.1126/science.aai8690 · Cited by 254 · see citing works
Evaluation of in silico algorithms for use with ACMG/AMP clinical variant interpretation guidelines
Ghosh, R., Oak, N., & Plon, S. E. (2017) · Genome biology · doi:10.1186/s13059-017-1353-5 · Cited by 233 · see citing works
Kobayashi, Y., Yang, S., Nykamp, K., Garcia, J., Lincoln, S. E., & Topper, S. (2017) · Genome Medicine · doi:10.1186/s13073-017-0403-7 · Cited by 232 · see citing works
PON-P2: Prediction Method for Fast and Reliable Identification of Harmful Variants
Niroula, A., Urolagin, S., & Vihinen, M. (2015) · PLoS ONE · doi:10.1371/journal.pone.0117380 · Cited by 227 · see citing works
Self-citationProspective functional classification of all possible missense variants in PPARG
UK Monogenic Diabetes Consortium, Majithia, A. R., Myocardial Infarction Genetics Consortium, UK Congenital Lipodystrophy Consortium, Tsuda, B., Agostini, M., Gnanapradeepan, K., Rice, R., Peloso, G., Patel, K. A., et al. (2016) · Nature Genetics · doi:10.1038/ng.3700 · Cited by 227 · see citing works
VarCards: an integrated genetic and clinical database for coding variants in the human genome
Li, J., Shi, L., Zhang, K., Zhang, Y., Hu, S., Zhao, T., Teng, H., Li, X., Jiang, Y. W., Ji, L., et al. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkx1039 · Cited by 197 · see citing works
The functional spectrum of low-frequency coding variation
Marth, G., Yu, F., Indap, A. R., Garimella, K. V., Gravel, S., Leong, W. F., Tyler‐Smith, C., Bainbridge, M. N., Blackwell, T. W., Zheng-Bradley, X., et al. (2011) · Genome biology · doi:10.1186/gb-2011-12-9-r84 · Cited by 190 · see citing works
VariBench: A Benchmark Database for Variations
Nair, P. S., & Vihinen, M. (2012) · Human Mutation · doi:10.1002/humu.22204 · Cited by 181 · see citing works
Self-citationVariation Interpretation Predictors: Principles, Types, Performance, and Choice
Niroula, A., & Vihinen, M. (2016) · Human Mutation · doi:10.1002/humu.22987 · Cited by 121 · see citing works
Self-citationHow good are pathogenicity predictors in detecting benign variants?
Niroula, A., & Vihinen, M. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006481 · Cited by 110 · see citing works
Self-citationGuidelines for Reporting and Using Prediction Tools for Genetic Variation Analysis
Vihinen, M. (2012) · Human Mutation · doi:10.1002/humu.22253 · Cited by 81 · see citing works
Self-citationPrediction of disease-related mutations affecting protein localization
Laurila, K., & Vihinen, M. (2009) · BMC Genomics · doi:10.1186/1471-2164-10-122 · Cited by 75 · see citing works
Self-citationMaffucci, P., Bigio, B., Rapaport, F., Cobat, A., Borghesi, A., Lopez, M., Patin, É., Bolze, A., Shang, L., Bendavid, M., et al. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1808403116 · Cited by 67 · see citing works
PON-Sol: prediction of effects of amino acid substitutions on protein solubility
Yang, Y., Niroula, A., Shen, B., & Vihinen, M. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw066 · Cited by 65 · see citing works
Self-citationWei, Q., Wang, L., Wang, Q., Kruger, W. D., & Dunbrack, R. L. (2010) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.22722 · Cited by 61 · see citing works
VariSNP, A Benchmark Database for Variations From dbSNP
Schaafsma, G. C. P., & Vihinen, M. (2014) · Human Mutation · doi:10.1002/humu.22727 · Cited by 60 · see citing works
Self-citationVariation benchmark datasets: update, criteria, quality and applications
Sarkar, A., Yang, Y., & Vihinen, M. (2019) · bioRxiv · doi:10.1093/database/baz117 · Cited by 47 · see citing works
Self-citationRiera, C., Padilla, N., & de la Cruz, X. (2016) · Human Mutation · doi:10.1002/humu.23048 · Cited by 43 · see citing works
Predicting Severity of Disease-Causing Variants
Niroula, A., & Vihinen, M. (2017) · Human Mutation · doi:10.1002/humu.23173 · Cited by 42 · see citing works
Self-citationA phenotype centric benchmark of variant prioritisation tools
Anderson, D. R., & Lassmann, T. (2018) · npj Genomic Medicine · doi:10.1038/s41525-018-0044-9 · Cited by 42 · see citing works
Masica, D. L., & Karchin, R. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004725 · Cited by 36 · see citing works
Performance of Protein Disorder Prediction Programs on Amino Acid Substitutions
Ali, H., Urolagin, S., Gurarslan, Ö., & Vihinen, M. (2014) · Human Mutation · doi:10.1002/humu.22564 · Cited by 25 · see citing works
Self-citationHow to Define Pathogenicity, Health, and Disease?
Vihinen, M. (2016) · Human Mutation · doi:10.1002/humu.23144 · Cited by 24 · see citing works
Self-citationSchaafsma, G. C. P., & Vihinen, M. (2017) · Human Mutation · doi:10.1002/humu.23236 · Cited by 24 · see citing works
Self-citationNo more hidden solutions in bioinformatics
Vihinen, M. (2015) · Nature · doi:10.1038/521261a · Cited by 21 · see citing works
Self-citationMuddled genetic terms miss and mess the message
Vihinen, M. (2015) · Trends in Genetics · doi:10.1016/j.tig.2015.05.008 · Cited by 20 · see citing works
Self-citationRepresentativeness of variation benchmark datasets
Schaafsma, G. C. P., & Vihinen, M. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2478-6 · Cited by 19 · see citing works
Self-citationde la Campa, E. Á., Padilla, N., & de la Cruz, X. (2017) · BMC Genomics · doi:10.1186/s12864-017-3914-0 · Cited by 17 · see citing works
Benchmarking subcellular localization and variant tolerance predictors on membrane proteins
Orioli, T., & Vihinen, M. (2019) · BMC Genomics · doi:10.1186/s12864-019-5865-0 · Cited by 17 · see citing works
Self-citationRim, J. H., Lee, J. S., Jung, J., Lee, J. H., Lee, S., Choi, J. R., Choi, J. Y., Lee, M. G., & Gee, H. Y. (2019) · Scientific Reports · doi:10.1038/s41598-019-41068-6 · Cited by 16 · see citing works
Majority Vote and Other Problems when using Computational Tools
Vihinen, M. (2014) · Human Mutation · doi:10.1002/humu.22600 · Cited by 15 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Desmet, F., Hamroun, G., Collod‐Beroud, G., Claustres, M., & Beroud, C. (2010). Bioinformatics identification of splice site signals and prediction of mutation effects In Mohan R. M. (Ed.), Research advances in nucleic acids research (pp. 1– 16). Kerala: Global Reseach Network.
Cited by
Sorted by citations
Pejaver, V., Byrne, A. B., Feng, B.-J., Pagel, K. A., Mooney, S. D., Karchin, R., O’Donnell-Luria, A., Harrison, S. M., Tavtigian, S. V., Greenblatt, M. S., et al. (2022) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2022.10.013 · Cited by 589 · see citing works
“4,6 Accepted standards for concordance do not exist, and variability among laboratories has been observed.”
Background
- Authors
- Vihinen, M.
- Affiliations
- Lund University (SE)
- Corresponding author
- Vihinen, Mauno
- ORCID
- Vihinen, Mauno
- Venue
- Molecular Genetics & Genomic Medicine
- Publisher
- Wiley
- Published
- 2020-03-11
- Volume
- 8
- Issue
- 9
- Pages
- e1206-e1206
- ISSN
- 2324-9269 (ISSN-L)
- PMC
- PMC7507483
- OpenAlex
- W3011148401
- Cited by
- 22
- References
- 52
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 1 · 2023: 5 · 2022: 4 · 2021: 6
- FWCI
- 2.20
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Molecular Genetics & Genomic Medicine (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Lund University Publications (Lund University) (submittedVersion)
- Funders
- Vetenskapsrådet (VR 2015‐02510)
- Cancerfonden (CAN 2017/699)
- Alfred Österlunds Stiftelse
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genomics and Rare Diseases
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genetic Associations and Epidemiology
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- variant interpretation, ACMG/AMP guidelines, computational prediction tools, benchmarking, class imbalance, minor allele frequency, rare variants, diagnostic sequencing, splice site mutation, amino acid substitutions
- MeSH terms
- Diagnosis, Computer-Assisted, Genetic Testing, Genetics, Medical, Humans, Polymorphism, Genetic, Societies, Medical, Software, Practice Guidelines as Topic, Sequence Analysis, DNA, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
