CoChat
Sign Up
PreprintCited by 50Genomics and Rare DiseasesGenomics and Phylogenetic StudiesCancer Genomics and Diagnostics

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Variant effect predictor correlation with functional assays is reflective of clinical classification performance

Open access
Livesey, Benjamin J., Marsh, Joseph A.·2024·Genome biology
DOI 10.1101/2024.05.12.593741PMC PMC12016141

Preprint — published as 10.1186/s13059-025-03575-w · Paper page

Open this paper in CoChat — free account

Abstract

Background: Understanding the relationship between protein sequence and function is crucial for accurate genetic variant classification. Variant effect predictors (VEPs) play a vital role in deciphering this complex relationship, yet evaluating their performance remains challenging for several reasons including data circularity, where the same or related data is used for training and assessment. High-throughput experimental strategies like deep mutational scanning (DMS) offer a promising solution. Results: In this study, we extend upon our previous benchmarking approach, assessing the performance of 97 different VEPs using DMS experiments from 36 different human proteins. In addition, a new pairwise, VEP-centric ranking method reduces the impact of missing predictions on the overall ranking. We observe a remarkably high correspondence between VEP performance in DMS-based benchmarks and clinical variant classification, especially for predictors that have not been directly trained on human clinical variants. Conclusions: Our results suggest that comparing VEP performance against diverse functional assays represents a reliable strategy for assessing their relative performance in clinical variant classification. However, major challenges in clinical interpretation of VEP scores persist, highlighting the need for further research to fully leverage computational predictors for genetic diagnosis. We also address practical considerations for end users in terms of choice of methodology.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2024.05.12.593741

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 106

  1. 10.1002/humu.22768 · DOI · Paper page
  2. 10.1038/gim.2015.30 · DOI · Paper page
  3. 10.1136/jmedgenet-2020-107003 · DOI
  4. 10.1002/mgg3.116 · DOI
  5. 10.1371/journal.pcbi.1006481 · DOI
  6. 10.1007/s00439-018-1916-x · DOI
  7. 10.1186/s13059-023-02986-x · DOI
  8. 10.1038/nmeth.3027 · DOI · Paper page
  9. 10.1186/s40246-017-0104-8 · DOI
  10. Critical Assessment of Genome Interpretation Consortium. CAGI, the Critical Assessment of Genome Interpretation, establishes progress and prospects for computational genetic variant interpretation methods. Genome Biol. 2024;25:53.
  11. 10.1002/humu.23293 · DOI
  12. 10.1002/humu.23838 · DOI
  13. Chen B , Solis-Villa C , Hakenberg J , Qiao W , Srinivasan RR , Yasuda M , et al. Acute Intermittent Porphyria: Predicted Pathogenicity of HMBS Variants Indicates Extremely Low Penetrance of the Autosomal Dominant Disease. Hum Mutat. 2016; · DOI
  14. Notin P , Kollasch AW , Ritter D , Niekerk L van, Paul S , Spinner H , et al. ProteinGym: Large-Scale Benchmarks for Protein Design and Fitness Prediction [Internet]. bioRxiv; 2023 [cited 2024 Jan 10]. p. 2023.12.07.570727. Available from: https://www.biorxiv.org/content/10.1101/2023.12.07.570727v1 · DOI
  15. 10.15252/msb.20199380 · DOI
  16. 10.15252/msb.202211474 · DOI
  17. 10.1002/pro.4688 · DOI · Paper page
  18. 10.1186/s13059-019-1845-6 · DOI
  19. 10.1093/nar/gkx1153 · DOI · Paper page
  20. 10.1038/s41586-024-07050-7 · DOI
  21. 10.1038/s41588-018-0122-z · DOI · Paper page
  22. 10.1186/s40246-024-00663-z · DOI
  23. Livesey BJ , Badonyi M , Dias M , Frazer J , Kumar S , Lindorff-Larsen K , et al. Guidelines for releasing a variant effect predictor [Internet]. arXiv; 2024 [cited 2024 Apr 29]. Available from: http://arxiv.org/abs/2404.10807 · arXiv
  24. 10.1016/j.cels.2017.11.003 · DOI
  25. Cheng J , Novati G , Pan J , Bycroft C , Žemgulytė A , Applebaum T , et al. Accurate proteome-wide missense variant effect prediction with AlphaMissense. Science. 2023;381:eadg7492. · DOI · Paper page
  26. 10.1002/humu.10212 · DOI
  27. Pathak AK , Bora N , Badonyi M , Livesey BJ , Consortium S , Ngeow J , et al. Pervasive ancestry bias in variant effect predictors [Internet]. bioRxiv; 2024 [cited 2024 Jun 12]. p. 2024.05.20.594987. Available from: https://www.biorxiv.org/content/10.1101/2024.05.20.594987v2 · DOI
  28. 10.1186/s13059-023-03024-6 · DOI
  29. 10.1093/bioinformatics/btaa1030 · DOI
  30. 10.1016/j.ajhg.2021.08.012 · DOI
  31. Meier J , Rao R , Verkuil R , Liu J , Sercu T , Rives A . Language models enable zero-shot prediction of the effects of mutations on protein function. Adv Neural Inf Process Syst [Internet]. Curran Associates, Inc.; 2021 [cited 2024 Apr 29]. p. 29287–303. Available from: https://proceedings.neurips.cc/paper/2021/hash/f51338d736f95dd42427296047067694-Abstract.html
  32. 10.1038/s41586-021-04043-8 · DOI · Paper page
  33. 10.1073/pnas.0404380101 · DOI
  34. 10.1093/bioinformatics/btz873 · DOI
  35. Adzhubei I , Jordan DM , Sunyaev SR. Predicting Functional Effect of Human Missense Mutations Using PolyPhen-2. Curr Protoc Hum Genet Editor Board Jonathan Haines Al. 2013;0 7:Unit7.20. · DOI · Paper page
  36. 10.1126/science.add2187 · DOI · Paper page
  37. Orenbuch R , Kollasch AW , Spinner HD , Shearer CA , Hopf TA , Franceschi D , et al. Deep generative modeling of the human proteome reveals over a hundred novel genes involved in rare genetic disorders [Internet]. medRxiv; 2023 [cited 2023 Dec 7]. p. 2023.11.27.23299062. Available from: https://www.medrxiv.org/content/10.1101/2023.11.27.23299062v1 · DOI
  38. Tekpinar M , David L , Henry T , Carbone A. PRESCOTT: a population aware, epistatic and structural model accurately predicts missense effect [Internet]. medRxiv; 2024 [cited 2024 Feb 7]. p. 2024.02.03.24302219. Available from: https://www.medrxiv.org/content/10.1101/2024.02.03.24302219v1 · DOI · Paper page
  39. 10.1093/molbev/msz179 · DOI
  40. Notin P , Niekerk LV , Kollasch AW , Ritter D , Gal Y , Marks DS. TranceptEVE: Combining Family-specific and Family-agnostic Models of Protein Sequences for Improved Fitness Prediction [Internet]. bioRxiv; 2022 [cited 2023 Dec 7]. p. 2022.12.07.519495. Available from: https://www.biorxiv.org/content/10.1101/2022.12.07.519495v2 · DOI
  41. Notin P , Dias M , Frazer J , Marchena-Hurtado J , Gomez A , Marks DS , et al. Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval [Internet]. arXiv; 2022 [cited 2023 Dec 7]. Available from: http://arxiv.org/abs/2205.13760 · arXiv
  42. Su J , Han C , Zhou Y , Shan J , Zhou X , Yuan F. SaProt: Protein Language Modeling with Structure-aware Vocabulary [Internet]. bioRxiv; 2024 [cited 2024 Oct 31]. p. 2023.10.01.560349. Available from: https://www.biorxiv.org/content/10.1101/2023.10.01.560349v5 · DOI
  43. 10.1038/s41587-023-01773-0 · DOI · Paper page
  44. 10.1038/s41467-021-22732-w · DOI · Paper page
  45. Rastogi R , Chung R , Li S , Li C , Lee K , Woo J , et al. Critical assessment of missense variant effect predictors on disease-relevant variant data [Internet]. bioRxiv; 2024 [cited 2024 Nov 18]. p. 2024.06.06.597828. Available from: https://www.biorxiv.org/content/10.1101/2024.06.06.597828v1 · DOI
  46. Rao RM , Liu J , Verkuil R , Meier J , Canny J , Abbeel P , et al. MSA Transformer. Proc 38th Int Conf Mach Learn [Internet]. PMLR; 2021 [cited 2023 Dec 6]. p. 8844–56. Available from: https://proceedings.mlr.press/v139/rao21a.html
  47. 10.1007/00439-021-02411-y · DOI
  48. 10.1016/j.gpb.2022.07.005 · DOI · Paper page
  49. 10.1186/s12859-020-3439-4 · DOI
  50. Li N , Mazaika E , Theotokis P , Zhang X , Jang M , Ahmad M , et al. Variant annotation across homologous proteins (“Paralogue Annotation”) identifies disease-causing missense variants with high precision, and is widely applicable across protein families [Internet]. bioRxiv; 2023 [cited 2024 May 1]. p. 2023.08.07.552236. Available from: https://www.biorxiv.org/content/10.1101/2023.08.07.552236v1 · DOI
  51. 10.1016/j.ajhg.2016.08.016 · DOI · Paper page
  52. Molotkov I , Koboldt DC , Artomov M. SNPred outperforms other ensemble-based SNV pathogenicity predictors and elucidates the challenges of using ClinVar for evaluation of variant classification quality [Internet]. medRxiv; 2023 [cited 2023 Dec 7]. p. 2023.09.07.23295192. Available from: https://www.medrxiv.org/content/10.1101/2023.09.07.23295192v2 · DOI
  53. 10.1186/s13073-022-01120-z · DOI
  54. 10.1016/j.jmb.2014.04.026 · DOI
  55. Samocha KE , Kosmicki JA , Karczewski KJ , O’Donnell-Luria AH , Pierce-Hoffman E , MacArthur DG , et al. Regional missense constraint improves variant deleteriousness prediction [Internet]. bioRxiv; 2017 [cited 2024 Feb 7]. p. 148353. Available from: https://www.biorxiv.org/content/10.1101/148353v1 · DOI · Paper page
  56. 10.1093/nar/gks539 · DOI
  57. 10.1093/nar/gkaa1100 · DOI · Paper page
  58. 10.1093/bioinformatics/btq671 · DOI
  59. 10.1186/s13059-024-03314-7 · DOI
  60. 10.1038/s41467-022-31686-6 · DOI
  61. 10.1126/science.abj8754 · DOI · Paper page
  62. 10.1016/j.ajhg.2022.10.013 · DOI · Paper page
  63. 10.1038/s41598-024-76202-6 · DOI · Paper page
  64. 10.1016/j.ajhg.2024.07.018 · DOI · Paper page
  65. 10.1016/j.ajhg.2024.10.011 · DOI
  66. 10.1186/s13073-020-00803-9 · DOI
  67. 10.1016/j.ajhg.2020.10.015 · DOI
  68. 10.1016/j.ajhg.2023.08.012 · DOI
  69. 10.1186/s13059-023-02935-8 · DOI · Paper page
  70. 10.1038/s41598-021-97871-7 · DOI
  71. McDonnell AF , Plech M , Livesey BJ , Gerasimavicius L , Owen LJ , Hall HN , et al. Deep mutational scanning quantifies DNA binding and predicts clinical outcomes of PAX6 variants [Internet]. bioRxiv; 2023 [cited 2024 Jan 10]. p. 2023.07.25.550478. Available from: https://www.biorxiv.org/content/10.1101/2023.07.25.550478v1 · DOI
  72. 10.1038/ng.3700 · DOI · Paper page
  73. 10.1038/s41467-022-30463-9 · DOI
  74. 10.1038/s41586-022-04928-2 · DOI
  75. 10.1016/j.molcel.2019.02.003 · DOI
  76. 10.1038/s41467-024-45829-4 · DOI · Paper page
  77. 10.1038/s41586-023-06954-0 · DOI
  78. 10.1038/s41467-024-48481-0 · DOI
  79. 10.1093/bioinformatics/btab529 · DOI
  80. Schmidt A , Röner S , Mai K , Klinkhammer H , Kircher M , Ludwig KU . Predicting the pathogenicity of missense variants using features derived from AlphaFold2. Bioinforma Oxf Engl. 2023;39:btad280. · DOI · Paper page
  81. 10.1093/nar/gkad989 · DOI · Paper page
  82. 10.1186/s13073-020-00775-w · DOI
  83. Yang KK , Fusi N , Lu AX. Convolutions are competitive with transformers for protein sequence pretraining [Internet]. bioRxiv; 2023 [cited 2024 Jan 10]. p. 2022.05.19.492714. Available from: https://www.biorxiv.org/content/10.1101/2022.05.19.492714v4 · DOI
  84. 10.1016/j.jmb.2021.166915 · DOI
  85. 10.1038/s41588-023-01465-0 · DOI
  86. 10.1126/science.ade2574 · DOI · Paper page
  87. 10.1038/s42256-022-00561-w · DOI · Paper page
  88. Takeda J , Nanatsue K , Yamagishi R , Ito M , Haga N , Hirata H , et al. InMeRF: prediction of pathogenicity of missense variants by individual modeling for each amino acid substitution. NAR Genomics Bioinforma [Internet]. 2020 [cited 2023 Dec 7];2. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7671370/ · DOI
  89. 10.1101/gr.245522.118 · DOI
  90. 10.1038/s41467-023-39306-7 · DOI
  91. 10.1371/journal.pone.0236962 · DOI
  92. Petrazzini BO , Balick DJ , Forrest IS , Cho J , Rocheleau G , Jordan DM , et al. Prediction of recessive inheritance for missense variants in human disease [Internet]. medRxiv; 2021 [cited 2023 Dec 7]. p. 2021.10.25.21265472. Available from: https://www.medrxiv.org/content/10.1101/2021.10.25.21265472v1 · DOI
  93. 2023, Deciphering “the language of nature”: A transformer-based language model for deleterious mutations in proteins, The Innovation, 4, 100487
  94. 10.1016/j.ajhg.2022.01.006 · DOI
  95. Kuru N , Dereli O , Akkoyun E , Bircan A , Tastan O , Adebali O . PHACT: Phylogeny-Aware Computing of Tolerance for Missense Mutations. Mol Biol Evol. 2022;39:msac114. · DOI
  96. Dereli O , Kuru N , Akkoyun E , Bircan A , Tastan O , Adebali O . PHACTboost: A Phylogeny-Aware Pathogenicity Predictor for Missense Mutations via Boosting. Mol Biol Evol. 2024;41:msae136. · DOI
  97. 10.1093/nar/gkx369 · DOI
  98. 10.1016/j.cels.2023.10.002 · DOI · Paper page
  99. 10.1038/s41467-022-32007-7 · DOI · Paper page
  100. 10.1093/bioinformatics/btaa127 · DOI
  101. Hesslow D , Zanichelli N , Notin P , Poli I , Marks D. RITA: a Study on Scaling Up Generative Protein Sequence Models [Internet]. arXiv; 2022 [cited 2023 Dec 6]. Available from: http://arxiv.org/abs/2205.05789 · arXiv
  102. 10.1186/s13059-023-02948-3 · DOI · Paper page
  103. 10.1016/j.crmeth.2023.100687 · DOI
  104. Wang B , Lei X , Tian W , Perez-Rathke A , Tseng Y-Y , Liang J . Structure-based pathogenicity relationship identifier for predicting effects of single missense variants and discovery of higher-order cancer susceptibility clusters of mutations. Brief Bioinform. 2023;24:bbad206. · DOI · Paper page
  105. 10.1371/journal.pgen.1008922 · DOI
  106. 10.1038/s41592-019-0598-1 · DOI · Paper page

Metadata

Authors
Livesey, B. J., & Marsh, J. A.
Affiliations
Institute of Genetics and Cancer (GB)
Corresponding author
Marsh, Joseph A.
ORCID
Venue
Genome biology
Publisher
openRxiv
Published
2024-05-14
Volume
26
Issue
1
Pages
104-104
ISSN
1465-6906 (ISSN-L)
DOI
10.1101/2024.05.12.593741
PMC
PMC12016141
OpenAlex
W4409669254
Cited by
50
References
1
Citations by year
2026: 24 · 2025: 20 · 2024: 6
FWCI
25.87
Type
journal article
Language
en
Open access
Licence
Locations
  • Genome biology (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
Funders
  • Lister Institute of Preventive Medicine
  • Medical Research Council (MC_UU_00035/9)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genomics and Rare Diseases
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Cancer Research › Cancer Genomics and Diagnostics
Keywords
variant effect predictors, missense mutations, deep mutational scanning, functional assays, benchmarking, genetic diagnosis
MeSH terms
Humans, Computational Biology, Mutation, Missense

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review