CoChat
Sign Up
Cited by 234Genomics and Rare DiseasesGenetic Associations and EpidemiologyGenomics and Phylogenetic Studies

The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.

Evaluation of in silico algorithms for use with ACMG/AMP clinical variant interpretation guidelines

Open access
Ghosh, Rajarshi, Oak, Ninad, Plon, Sharon E.·2017·Genome biology
DOI 10.1186/s13059-017-1353-5PMC PMC5704597

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: The American College of Medical Genetics and American College of Pathologists (ACMG/AMP) variant classification guidelines for clinical reporting are widely used in diagnostic laboratories for variant interpretation. The ACMG/AMP guidelines recommend complete concordance of predictions among all in silico algorithms used without specifying the number or types of algorithms. The subjective nature of this recommendation contributes to discordance of variant classification among clinical laboratories and prevents definitive classification of variants. RESULTS: Using 14,819 benign or pathogenic missense variants from the ClinVar database, we compared performance of 25 algorithms across datasets differing in distinct biological and technical variables. There was wide variability in concordance among different combinations of algorithms with particularly low concordance for benign variants. We also identify a previously unreported source of error in variant interpretation (false concordance) where concordant in silico predictions are opposite to the evidence provided by other sources. We identified recently developed algorithms with high predictive power and robust to variables such as disease mechanism, gene constraint, and mode of inheritance, although poorer performing algorithms are more frequently used based on review of the clinical genetics literature (2011-2017). CONCLUSIONS: Our analyses identify algorithms with high performance characteristics independent of underlying disease mechanisms. We describe combinations of algorithms with increased concordance that should improve in silico algorithm usage during assessment of clinically relevant variants using the ACMG/AMP guidelines.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13059-017-1353-5

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Concordance among predictions of 18 algorithms for variants in ClinVar. Binary predictions made by 18 algorithms for each pathogenic or benign variants in ClinVar are shown in the upper and lower panels. Each variant is along a row and an orange, green, or white tile depicts a pathogenic, benign, or missing data call, respectively, by the corresponding algorithm. A total of 14,819 variants with ClinVar review status one star or above (a) and 2966 variants with ClinVar review status two stars or above (b) are shown
Fig. 1 Concordance among predictions of 18 algorithms for variants in ClinVar. Binary predictions made by 18 algorithms for each pathogenic or benign variants in ClinVar are shown in the upper and lower panels. Each variant is along a row and an orange, green, or white tile depicts a pathogenic, benign, or missing data call, respectively, by the corresponding algorithm. A total of 14,819 variants with ClinVar review status one star or above (a) and 2966 variants with ClinVar review status two stars or above (b) are shownCC BY · PubMed Central
Fig. 2 Concordance among algorithms. a Distribution of proportion of variants that had concordant calls by any given pair of algorithms (among 18 algorithms) for benign (green) and pathogenic (orange) variants in ClinVar. b Scatterplots of true concordance (variant assertion matches ClinVar assertion) vs false concordance (variant assertion does not match ClinVar assertion) for combinations of three, four, or five algorithms at a time. An orange and a green point depict the true and false concordance of a combination for benign and pathogenic variants, respectively, in ClinVar. The rugs on top and bottom, left and right represent the distribution of false and true concordances, respectively. c Hierarchical clustering of 25 algorithms with scores for 14,819 variants in ClinVar. Red rectangles indicate robust clusters with an AU p value of > 0.99 (see “Methods”)
Fig. 2 Concordance among algorithms. a Distribution of proportion of variants that had concordant calls by any given pair of algorithms (among 18 algorithms) for benign (green) and pathogenic (orange) variants in ClinVar. b Scatterplots of true concordance (variant assertion matches ClinVar assertion) vs false concordance (variant assertion does not match ClinVar assertion) for combinations of three, four, or five algorithms at a time. An orange and a green point depict the true and false concordance of a combination for benign and pathogenic variants, respectively, in ClinVar. The rugs on top and bottom, left and right represent the distribution of false and true concordances, respectively. c Hierarchical clustering of 25 algorithms with scores for 14,819 variants in ClinVar. Red rectangles indicate robust clusters with an AU p value of > 0.99 (see “Methods”)CC BY · PubMed Central
Fig. 3 Performance analysis of algorithms. The AUC of a ROC are plotted for 25 algorithms. Vertical dotted line indicates an AUC of 0.9 and 99% confidence intervals for each AUC are shown. Blue dots indicate AUC > 0.89. a AUCs of the algorithms across different datasets shown in the panels and described in text. b AUCs of the algorithms across different datasets (represented in panels) to address type I circularity as described in text. The same plots for ClinVar Status * and ClinVar Status ** as in Fig. 3a are used in 3b for comparison. Any instance of ** represents variants with ClinVar review status of two stars or above. Ensemble predictors are indicated by dark green labels on the y-axis
Fig. 3 Performance analysis of algorithms. The AUC of a ROC are plotted for 25 algorithms. Vertical dotted line indicates an AUC of 0.9 and 99% confidence intervals for each AUC are shown. Blue dots indicate AUC > 0.89. a AUCs of the algorithms across different datasets shown in the panels and described in text. b AUCs of the algorithms across different datasets (represented in panels) to address type I circularity as described in text. The same plots for ClinVar Status * and ClinVar Status ** as in Fig. 3a are used in 3b for comparison. Any instance of ** represents variants with ClinVar review status of two stars or above. Ensemble predictors are indicated by dark green labels on the y-axisCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

Sorted by citations

See all 234

Related

See all

Metadata

Authors
Ghosh, R., Oak, N., & Plon, S. E.
Affiliations
Baylor College of Medicine (US)
ORCID
Venue
Genome biology
Publisher
Springer Science and Business Media LLC
Published
2017-11-28
Volume
18
Issue
1
Pages
225-225
ISSN
  • 1465-6906 (ISSN-L)
  • 1474-760X
DOI
10.1186/s13059-017-1353-5
PMC
PMC5704597
OpenAlex
W2949886398
Cited by
233
References
41
Citations by year
2026: 16 · 2025: 22 · 2024: 20 · 2023: 24 · 2022: 31 · 2021: 30 · 2020: 30 · 2019: 36 · 2018: 22 · 2017: 1
FWCI
16.16
Type
journal article
Language
en
Open access
Licence
Locations
  • Genome biology (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
National Human Genome Research Institute (5 U01 HG007436)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genomics and Rare Diseases
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genetic Associations and Epidemiology
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
Keywords
clinical variant interpretation, missense mutations, ClinVar, pathogenic variants, benign variants, disease mechanisms, mode of inheritance, algorithm comparison
MeSH terms
Algorithms, Humans, Software, Genetic Variation, Reproducibility of Results, Genome, Human, Guidelines as Topic, Computational Biology, Databases, Genetic, Genetic Association Studies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review