Calibrating variant-scoring methods for clinical decision making
Open accessAbstract
SUMMARY: Identifying pathogenic variants and annotating them is a major challenge in human genetics, especially for the non-coding ones. Several tools have been developed and used to predict the functional effect of genetic variants. However, the calibration assessment of the predictions has received little attention. Calibration refers to the idea that if a model predicts a group of variants to be pathogenic with a probability P, it is expected that the same fraction P of true positive is found in the observed set. For instance, a well-calibrated classifier should label the variants such that among the ones to which it gave a probability value close to 0.7, approximately 70% actually belong to the pathogenic class. Poorly calibrated algorithms can be misleading and potentially harmful for clinical decision making. AVALIABILITY AND IMPLEMENTATION: The dataset used for testing the methods is available through the DOI:10.5281/zenodo.4448197. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several studies have attempted to benchmark methods to identify tools and tool-threshold combinations that are the most accurate and appropriate for clinically relevant variants.
“Several studies have attempted to benchmark methods to identify tools and tool-threshold combinations that are the most accurate and appropriate for clinically relevant variants.3,38,39,40,41”
From Calibration of computational tools for missense variant pathogenicity classification and ClinGen recommendations for PP3/BP4 criteria (2022) · cites this paper as [39]
The Introduction describes comparing six variant-scoring tools, and the Conclusion recommends selecting a method using both classification and calibration performance. Section 2 discusses performance across threshold settings, but the paper does not identify the most accurate tool–threshold combinations for clinically relevant variants. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
A general framework for estimating the relative pathogenicity of human genetic variants
Kircher, M., Witten, D. M., Jain, P., O'Roak, B. J., Cooper, G. M., & Shendure, J. (2014) · Nature Genetics · doi:10.1038/ng.2892 · Cited by 6,183 · see citing works
Calibration: the Achilles heel of predictive analytics
On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative, Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W. (2019) · BMC Medicine · doi:10.1186/s12916-019-1466-7 · Cited by 2,650 · see citing works
Predicting effects of noncoding variants with deep learning–based sequence model
Zhou, J., & Troyanskaya, O. G. (2015) · Nature Methods · doi:10.1038/nmeth.3547 · Cited by 2,338 · see citing works
Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119 · see citing works
DANN: a deep learning approach for annotating the pathogenicity of genetic variants
Quang, D. X., Chen, Y., & Xie, X. S. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu703 · Cited by 1,073 · see citing works
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · see citing works
A Deep Learning Mammography-based Model for Improved Breast Cancer Risk Prediction
Yala, A., Lehman, C., Schuster, T., Portnoi, T., & Barzilay, R. (2019) · Radiology · doi:10.1148/radiol.2019182716 · Cited by 758 · see citing works
Shihab, H. A., Rogers, M. F., Gough, J., Mort, M. E., Cooper, D. N., Day, I. N. M., Gaunt, T. R., & Campbell, C. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv009 · Cited by 703 · see citing works
A spectral approach integrating functional genomic annotations for coding and noncoding variants
Ionita‐Laza, I., McCallum, K., Xu, B., & Buxbaum, J. D. (2016) · Nature Genetics · doi:10.1038/ng.3477 · Cited by 588 · see citing works
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · European Urology · doi:10.1016/j.eururo.2014.11.025 · Cited by 439 · see citing works
Calibration of Risk Prediction Models: Impact on Decision-Analytic Performance
Van Calster, B., & Vickers, A. J. (2015) · Medical Decision Making · doi:10.1177/0272989x14547233 · Cited by 371 · see citing works
Genomic Analysis in the Age of Human Genome Sequencing
Lappalainen, T., Scott, A. J., Brandt, M. K., & Hall, I. M. (2019) · Cell · doi:10.1016/j.cell.2019.02.032 · Cited by 298 · see citing works
PhD-SNPg: a webserver and lightweight tool for scoring single nucleotide variants
Capriotti, E., & Fariselli, P. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkx369 · Cited by 253 · see citing works
Big Data and Predictive Analytics
Shah, N. D., Steyerberg, E. W., & Kent, D. M. (2018) · JAMA · doi:10.1001/jama.2018.5602 · Cited by 244 · see citing works
Comparison and integration of computational methods for deleterious synonymous mutation prediction
Cheng, N., Li, M., Zhao, L., Zhang, B., Yang, Y., Zheng, C.-H., & Xia, J. (2019) · Briefings in Bioinformatics · doi:10.1093/bib/bbz047 · Cited by 82 · see citing works
Bioinformatics for personal genome interpretation
Capriotti, E., Nehrt, N. L., Kann, M. G., & Bromberg, Y. (2012) · Briefings in Bioinformatics · doi:10.1093/bib/bbr070 · Cited by 68 · see citing works
Integrating molecular networks with genetic variant interpretation for precision medicine
Capriotti, E., Öztürk, K., & Carter, H. K. (2018) · WIREs Systems Biology and Medicine · doi:10.1002/wsbm.1443 · Cited by 41 · see citing works
A benchmark study of scoring methods for non-coding mutations
Drubay, D., Gautheret, D., & Michiels, S. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty008 · Cited by 21 · see citing works
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078
Flagged first, then by citations
Research works printed but not linked: 1
- Niculescu-Mizil A. et al (2005) Predicting Good Probabilities with Supervised Learning. ACM, New York. pp. 625– 632.
Cited by
Sorted by citations
Pejaver, V., Byrne, A. B., Feng, B.-J., Pagel, K. A., Mooney, S. D., Karchin, R., O’Donnell-Luria, A., Harrison, S. M., Tavtigian, S. V., Greenblatt, M. S., et al. (2022) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2022.10.013 · Cited by 589 · see citing works
Manfredi, M., Savojardo, C., Martelli, P. L., & Casadio, R. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac678 · Cited by 46 · see citing works
Estiri, H., Strasser, Z. H., Rashidian, S., Klann, J. G., Wagholikar, K. B., McCoy, T. H., & Murphy, S. N. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac070 · Cited by 40 · see citing works
PhD-SNPg: updating a webserver and lightweight tool for scoring nucleotide variants
Capriotti, E., & Fariselli, P. (2023) · Nucleic Acids Research · doi:10.1093/nar/gkad455 · Cited by 32 · see citing works
TADA—a machine learning tool for functional annotation-based prioritisation of pathogenic CNVs
Hertzberg, J., Mundlos, S., Vingron, M., & Gallone, G. (2022) · Genome biology · doi:10.1186/s13059-022-02631-z · Cited by 21 · see citing works
“As described by Benevenuta et al. [40] a potential criterion for the reliability of a classifiers class probabilities is its calibration - the fraction of true positives compared to the mean predicted value [41].”
Resources and tools for rare disease variant interpretation
Licata, L., Via, A., Turina, P., Babbi, G., Benevenuta, S., Carta, C., Casadio, R., Cicconardi, A., Facchiano, A. M., Fariselli, P., et al. (2023) · Frontiers in Molecular Biosciences · doi:10.3389/fmolb.2023.1169109 · Cited by 21 · see citing works
“These results are in the same range of those reported in previous works, not limited to RDs (Capriotti and Fariselli, 2017; Benevenuta et al., 2021).”
ResultEvaluating the relevance of sequence conservation in the prediction of pathogenic missense variants
Capriotti, E., & Fariselli, P. (2022) · Human Genetics · doi:10.1007/s00439-021-02419-4 · Cited by 18 · see citing works
“CADD is one of the most accurate and popular methods for predicting Pathogenic variants in coding and non-coding regions (Benevenuta et al. 2021).”
MethodBioPRO: Context-Infused Prompt Learning for Biomedical Entity Linking
Zhu, T., Qin, Y., Feng, M., Chen, Q., Hu, B., & Xiang, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3331149 · Cited by 8 · see citing works
“Thus, it is critical to have accurate biomedical entity linking techniques, since failure in disambiguation will introduce huge risks in medical-related decision making [12].”
BackgroundCiortea, D.-A., Petrea, C. L., Verga, G. I., Berbece, S. I., Gurău, G., Fotea, S., & Matei, M. N. (2025) · Biomedicines · doi:10.3390/biomedicines13102573 · Cited by 3 · see citing works
“In future research particular attention should be focused on laboratory harmonization (minor inter-assay differences in copeptin thresholds are plausible [59]), explicit calibration (relevant statistical methods [76]), and clinical utility assessment [77].”
Somatic mutations impose an entropic upper bound on human lifespan
Efimov, E., Fedotov, V., Malaev, L. G., Khrameeva, E. E., & Kriukov, D. V. (2026) · npj Aging · doi:10.1038/s41514-026-00421-6 · Cited by 1 · see citing works
“Raw CADD scores were transformed using a sigmoid model y ¼ 1 = ð 1 þ exp ð(cid:2) A (cid:2) x þ B ÞÞ , with A and B parameters estimated by Benevenuta et al. 161 .”
Mishra, D., Tiwari, A., Srivastava, S., Tripathi, M., & Kapoor, A. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.09.23.678112
- Authors
- Benevenuta, S., Capriotti, E., & Fariselli, P.
- Affiliations
- University of Turin (IT)
- University of Bologna (IT)
- Corresponding author
- Capriotti, Emidio
- Fariselli, Piero
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-12-15
- Volume
- 36
- Issue
- 24
- Pages
- 5709-5711
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- PMC
- PMC8023678
- OpenAlex
- W3124903445
- Cited by
- 11
- References
- 20
- Citations by year
- 2026: 1 · 2025: 2 · 2023: 3 · 2022: 5
- FWCI
- 1.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Bioinformatics (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) (submittedVersion, other-oa)
- Funders
- Ministry for Education, University and Research (D15D18000410001)
- Italian Ministry of Education (MIUR-PRIN-201744NR8S)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genomics and Rare Diseases
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genetic Associations and Epidemiology
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- variant scoring, forecast calibration, pathogenic variant prediction, non-coding variants, variant annotation, clinical decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
