Why variant effect predictors and multiplexed assays agree and disagree
Open accessAbstract
Computational variant effect predictors (VEPs) and multiplexed assays of variant effect (MAVEs) are two key tools for assessing the functional consequences of genetic variants. While their outputs are often concordant, there are also many differences. Here, we analyse missense MAVE data from 37 different human proteins, comparing them to five state-of-the-art VEPs in order to quantify and explain their points of agreement and disagreement. We find that discordance is not random but reflects fundamental differences in how each method infers functional impact. VEPs, which rely heavily on sequence conservation and basic structural features, tend to overcall pathogenicity at buried and hydrophobic residues, while underestimating impact in disordered regions and on charged surface residues. MAVEs, by contrast, capture context-specific mechanisms more accurately, but can miss pathogenic variants when the assay fails to reflect disease biology, or be subject to high levels of experimental noise. By comparing both global patterns and specific clinically relevant variants, we show how protein features, assay design, and variant type shape predictive discordance. Our findings provide a framework for interpreting when and why VEPs and MAVEs diverge and point toward strategies for improving variant interpretation through integrated, mechanism-aware approaches.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MAVE-VEP correlation coefficients (Spearman ρ) were around 0.5-0.6 for a raft of top-ranking VEPs in one recent study.
“This is illustrated by the finding that results from VEPs and MAVEs, while showing broadly similar, and usually high, overall predictive performances, actually show only modest correlation: MAVE-VEP correlation coefficients (Spearman ρ) were around 0.5–0.6 for a raft of top-ranking VEPs in one recent study (43).”
From Massively parallel functional genomic assays in endocrinology: from promise to delivery (2026) · cites this paper as [43]
In all published receptor MAVEs, discriminatory performance was high, with areas under ROC curves for identifying pathogenic mutations generally in excess of 0.90.
“Although different performance metrics were reported for each receptor MAVE, illustrating the evolving approaches taken in the field, in all cases discriminatory performance was high, with areas under ROC curves for identifying pathogenic mutations generally in excess of 0.90 (43, 59).”
From Massively parallel functional genomic assays in endocrinology: from promise to delivery (2026) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 35
- Weile, J. et al. A framework for exhaustively mapping functional missense variants. Mol. Syst. Biol. 13, 957 (2017).
- Sun, S. et al. A proactive genotype-to-patient-phenotype map for cystathionine beta-synthase. Genome Med. 12, 13 (2020).
- Grønbæk-Thygesen, M. et al. Deep mutational scanning reveals a correlation between degradation and toxicity of thousands of aspartoacylase variants. Nat. Commun. 15, 4026 (2024).
- Gersing, S. et al. A comprehensive map of human glucokinase variant activity. Genome Biol. 24, 97 (2023).
- Livesey, B. J. et al. Guidelines for releasing a variant effect predictor. Genome Biol. 26, 97 (2025).
- Radford, E. J. et al. Saturation genome editing of DDX3X clarifies pathogenicity of germline and somatic variation. Nat. Commun. 14, 7702 (2023).
- Livesey, B. J. & Marsh, J. A. Variant effect predictor correlation with functional assays is reflective of clinical classification performance. Genome Biol. 26, 104 (2025).
- Rubin, A. F. et al. MaveDB 2024: a curated community database with over seven million variant effects from multiplexed functional assays. Genome Biol. 26, 13 (2025).
- Notin, P. et al. ProteinGym: large-scale benchmarks for protein fit- ness prediction and design. Adv. Neural Inf. Process. Syst. 36, 64331-64379 (2023).
- Clausen, L. et al. A mutational atlas for Parkin proteostasis. Nat. Commun. 15, 1541 (2024).
- Lin, Y.-J., Menon, A. S., Hu, Z. & Brenner, S. E. Variant Impact Pre- dictor database (VIPdb), version 2: trends from three decades of genetic variant impact predictors. Hum. Genomics 18, 90 (2024).
- Molotkov, I., Koboldt, D. C. & Artomov, M. SNPred outperforms other ensemble-based SNV pathogenicity predictors and eluci- dates the challenges of using ClinVar for evaluation of variant classification quality. 2023.09.07.23295192 Preprint at https://doi. org/10.1101/2023.09.07.23295192 (2023). · DOI
- Liu, X., Li, C., Mou, C., Dong, Y. & Tu, Y. dbNSFP v4: a comprehensive database of transcript-specific functional predictions and annota- tions for human nonsynonymous and splice-site SNVs. Genome Med. 12, 103 (2020).
- Li, C., Zhi, D., Wang, K. & Liu, X. MetaRNN: differentiating rare pathogenic and rare benign missense SNVs and InDels using deep learning. Genome Med. 14, 115 (2022).
- Jiang, T. T., Fang, L. & Wang, K. Deciphering "the language of nat- ure": A transformer-based language model for deleterious muta- tions in proteins. Innovation 4, 100487 (2023).
- Shang, Y., Badonyi, M. & Marsh, J. A. Clinical evidence yield as a framework for evaluating computational predictors and multi- plexed assays of variant effect. 2026.03.27.714777 Preprint at https://doi.org/10.64898/2026.03.27.714777 (2026). · DOI
- Jagota, M. et al. Cross-protein transfer learning substantially improves disease variant prediction. Genome Biol. 24, 182 (2023).
- Tekpinar, M., David, L., Henry, T. & Carbone, A. PRESCOTT: a population aware, epistatic, and structural model accurately pre- dicts missense effects. Genome Biol. 26, 113 (2025).
- Meier, J. et al. Language models enable zero-shot prediction of the effects of mutations on protein function. Adv. Neural Inf. Process. Syst. 34, 29287-29303 (2021).
- Su, J. et al. SaProt: protein language modeling with structure-aware vocabulary. 2023.10.01.560349 Preprint at https://doi.org/10.1101/ 2023.10.01.560349 (2023). · DOI
- Jonsson, N. F., Marsh, J. A. & Lindorff-Larsen, K. Disagreement among variant effect predictors guides experimental prioritization of target proteins. 2026.03.18.712765 Preprint at https://doi.org/10. 64898/2026.03.18.712765 (2026). · DOI
- Livesey, B. J. & Marsh, J. A. Updated benchmarking of variant effect predictors using deep mutational scanning. Mol. Syst. Biol. e11474 https://doi.org/10.15252/msb.202211474 (2023). · DOI
- Fawzy, M. & Marsh, J. A. Understanding the heterogeneous per- formance of variant effect predictors across human protein-coding genes. Sci. Rep. 14, 26114 (2024).
- Livesey, B. J. & Marsh, J. A. The properties of human disease mutations at protein interfaces. PLOS Comput. Biol. 18, e1009858 (2022).
- Fawzy, M. & Marsh, J. A. Assessing variant effect predictors and disease mechanisms in intrinsically disordered proteins. PLoS Comput. Biol. 21, e1013400 (2025).
- Luppino, F., Lenz, S., Chow, C. F. W. & Toth-Petroczy, A. Deep learning tools predict variants in disordered regions with lower sensitivity. BMC Genomics 26, 367 (2025).
- Zeng, T. & Li, Y. I. Predicting RNA splicing from DNA sequence using Pangolin. Genome Biol. 23, 103 (2022).
- Lee, M. K. et al. Human α-synuclein-harboring familial Parkinson's disease-linked Ala-53 → Thr mutation causes neurodegenerative disease with α-synuclein aggregation in transgenic mice. Proc. Natl. Acad. Sci. USA 99, 8968-8973 (2002).
- VCV002447337.2 -ClinVar -NCBI. https://www.ncbi.nlm.nih.gov/ clinvar/variation/2447337/?oq=%22MSH2%22[GENE]+%22S612P% ]&m=NM_000251.3(MSH2):c.1834T%3EC%20(p. Ser612Pro)%3Fterm=%22MSH2%22%5BGENE%5D%20S612P.
- Froese, D. S. et al. Structural basis for the regulation of human 5,10- methylenetetrahydrofolate reductase by phosphorylation and S-adenosylmethionine inhibition. Nat. Commun. 9, 2261 (2018).
- Gerasimavicius, L., Teichmann, S. A. & Marsh, J. A. Leveraging protein structural information to improve variant effect prediction. Curr. Opin. Struct. Biol. 92, 103023 (2025).
- Gerasimavicius, L., Livesey, B. J. & Marsh, J. A. Loss-of-function, gain-of-function and dominant-negative mutations have pro- foundly different effects on protein structure. Nat. Commun. 13, 3895 (2022).
- Fowler, D. M. et al. An Atlas of Variant Effects to understand the genome at nucleotide resolution. Genome Biol. 24, 147 (2023).
- Esposito, D. et al. MaveDB: an open-source platform to distribute and interpret data from multiplexed assays of variant effect. Gen- ome Biol. 20, 223 (2019).
- Scheibenreif, L., Littmann, M., Orengo, C. & Rost, B. FunFam protein families improve residue level molecular function prediction. BMC Bioinforma. 20, 400 (2019).
Cited by
Sorted by citations
Massively parallel functional genomic assays in endocrinology: from promise to delivery
Xu, J., Aslanzadeh, V., Kudla, G., & Semple, R. K. (2026) · Journal of Molecular Endocrinology · doi:10.1530/jme-25-0134
- Authors
- Livesey, B. J., & Marsh, J. A.
- Affiliations
- Institute of Genetics and Cancer (GB)
- Corresponding author
- Livesey, Benjamin J.
- Marsh, Joseph A.
- Venue
- Nature Communications
- Publisher
- openRxiv
- Published
- 2025-08-01
- Volume
- 17
- Issue
- 1
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC13620131
- OpenAlex
- W4412840383
- Cited by
- 7
- References
- 0
- Citations by year
- 2026: 6 · 2025: 1
- FWCI
- 18.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Nature Communications (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Medical Research Council (MC_UU_00035/9)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genomics and Rare Diseases
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Genetics › Genetic Associations and Epidemiology
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- variant effect prediction, MAVE, missense mutations, pathogenicity prediction, protein disorder, buried residues, hydrophobic residues, sequence conservation, variant interpretation, protein structure
- MeSH terms
- Humans, Proteins, Genetic Variation, Computational Biology, Mutation, Missense
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
