How good are pathogenicity predictors in detecting benign variants?
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Computational tools are widely used for interpreting variants detected in sequencing projects. The choice of computational tools is critical for reliable variant impact interpretation for precision medicine and should be based on systematic performance assessment. The performance of the methods varies widely in different performance assessments, for example due to the sizes of test datasets. To address this issue, we obtained 63,160 common amino acid substitutions (allele frequency ≥1% and <25%) from the Exome Aggregation Consortium (ExAC) database, which contains variants from 60,706 genomes or exomes. We evaluated the specificity, the capability to detect benign variants, for 10 variant interpretation tools. In addition to overall specificity of the tools, we tested their performance for variants in six geographical populations. PON-P2 had the best performance (95.5%) followed by FATHMM (86.4%) and VEST (83.5%). While these tools had excellent performance, the poorest method predicted more than one third of the benign variants to be disease-causing. The results allow choosing reliable methods for benign variant interpretation, for both research and clinical purposes, as well as provide a benchmark for method developers. AUTHOR SUMMARY In precision/personalized medicine of many conditions it is essential to investigate individual’s genome. Interpretation of the observed variation (mutation) sets is feasible only with computational approaches. We assessed the performance of variant pathogenicity/tolerance prediction programs on benign variants. Variants were obtained from high-quality ExAC database and selected to have minor allele frequency between 1 and 25%. We obtained 63,160 such cases and investigated 10 widely used predictors. Specificities of the methods showed large differences, from 0.64 to 0.96%, thus users of these methods have to be careful when choosing the one(s) they will use. We investigated further the performances on different populations, allele frequencies, separately for males and females, chromosome wise and for population unique and non-unique variants. The ranking of the tools remained the same in all these scenarios, i.e. the best methods were the best irrespective on how the data is filtered and grouped. This is to our knowledge the first evaluation of method performance on benign variants.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Richards, S., Aziz, N., Bale, S., Bick, D., Das, S., Gastier-Foster, J., Grody, W. W., Hegde, M., Lyon, E., Spector, E., et al. (2015) · Genetics in Medicine · doi:10.1038/gim.2015.30 · Cited by 32,594
Analysis of protein-coding genetic variation in 60,706 humans
Exome Aggregation Consortium, Lek, M., Karczewski, K. J., Minikel, E. V., Samocha, K. E., Banks, E., Fennell, T., O’Donnell-Luria, A. H., Ware, J. S., Hill, A. J., et al. (2016) · Nature · doi:10.1038/nature19057 · Cited by 9,587
The Ensembl Variant Effect Predictor
McLaren, W., Gil, L., Hunt, S. E., Riat, H. S., Ritchie, G. R. S., Thormann, A., Flicek, P., & Cunningham, F. (2016) · Genome Biology · doi:10.1186/s13059-016-0974-4 · Cited by 8,099
dbSNP: the NCBI database of genetic variation
Sherry, S. T. (2001) · Nucleic Acids Research · doi:10.1093/nar/29.1.308 · Cited by 7,502
ClinVar: public archive of relationships among sequence variation and human phenotype
Landrum, M. J., Lee, J. M., Riley, G. R., Jang, W., Rubinstein, W. S., Church, D. M., & Maglott, D. R. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1113 · Cited by 3,516
MutationTaster evaluates disease-causing potential of sequence alterations
Schwarz, J. M., Rödelsperger, C., Schuelke, M., & Seelow, D. (2010) · Nature Methods · doi:10.1038/nmeth0810-575 · Cited by 2,975
REVEL: An Ensemble Method for Predicting the Pathogenicity of Rare Missense Variants
Ioannidis, N. M., Rothstein, J. H., Pejaver, V., Middha, S., McDonnell, S. K., Baheti, S., Musolf, A., Li, Q., Holzinger, E., Karyadi, D., et al. (2016) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2016.08.016 · Cited by 2,908
ClinVar: public archive of interpretations of clinically relevant variants
Landrum, M. J., Lee, J. M., Benson, M., Brown, G., Chao, C., Chitipiralla, S., Gu, B., Hart, J., Hoffman, D., Hoover, J., et al. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkv1222 · Cited by 2,758
Amberger, J. S., Bocchini, C. A., Schiettecatte, F., Scott, A. F., & Hamosh, A. (2015) · Nucleic Acids Research · doi:10.1093/nar/gku1205 · Cited by 2,691
Ancient DNA and the rewriting of human history: be sparing with Occam’s razor
Haber, M., Mezzavilla, M., Xue, Y., & Tyler-Smith, C. (2016) · Genome Biology · doi:10.1186/s13059-015-0866-z · Cited by 1,924
Cited by
Livesey, B. J., & Marsh, J. A. (2025) · Genome Biology · doi:10.1186/s13059-025-03575-w · Cited by 50
Bianco, S. D., Parca, L., Petrizzelli, F., Biagini, T., Giovannetti, A., Liorni, N., Napoli, A., Carella, M., Procaccio, V., Lott, M. T., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-40797-7 · Cited by 44
Problems in variation interpretation guidelines and in their implementation in computational tools
Vihinen, M. (2020) · Molecular Genetics & Genomic Medicine · doi:10.1002/mgg3.1206 · Cited by 22
Self-citationGonzález-del Pozo, M., Fernández-Suárez, E., Bravo-Gil, N., Méndez-Vidal, C., Martín-Sánchez, M., Rodríguez-de la Rúa, E., Ramos-Jiménez, M., Morillo-Sánchez, M. J., Borrego, S., & Antiñolo, G. (2022) · npj Genomic Medicine · doi:10.1038/s41525-022-00286-0 · Cited by 21
Metadata
- Authors
- Niroula, A., & Vihinen, M.
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2019-02-11
- Volume
- 15
- Issue
- 2
- Pages
- e1006481
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W2953301805
- Cited by
- 107
- References
- 13
- FWCI
- 6.91
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.