Updated benchmarking of variant effect predictors using deep mutational scanning
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
The assessment of variant effect predictor (VEP) performance is fraught with biases introduced by benchmarking against clinical observations. In this study, building on our previous work, we use independently generated measurements of protein function from deep mutational scanning (DMS) experiments for 26 human proteins to benchmark 55 different VEPs, while introducing minimal data circularity. Many top‐performing VEPs are unsupervised methods including EVE, DeepSequence and ESM‐1v, a protein language model that ranked first overall. However, the strong performance of recent supervised VEPs, in particular VARITY, shows that developers are taking data circularity and bias issues seriously. We also assess the performance of DMS and unsupervised VEPs for discriminating between known pathogenic and putatively benign missense variants. Our findings are mixed, demonstrating that some DMS datasets perform exceptionally at variant classification, while others are poor. Notably, we observe a striking correlation between VEP agreement with DMS data and performance in identifying clinically relevant variants, strongly supporting the validity of our rankings and the utility of DMS for independent benchmarking.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The mutational constraint spectrum quantified from variation in 141,456 humans
Karczewski, K. J., Francioli, L. C., Tiao, G., Cummings, B. B., Alföldi, J., Wang, Q., Collins, R. L., Laricchia, K. M., Ganna, A., Birnbaum, D. P., et al. (2020) · Nature · doi:10.1038/s41586-020-2308-7 · Cited by 9,434
Amino acid substitution matrices from protein blocks.
Henikoff, S., & Henikoff, J. G. (1992) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.89.22.10915 · Cited by 6,218
ClinVar: public archive of relationships among sequence variation and human phenotype
Landrum, M. J., Lee, J. M., Riley, G. R., Jang, W., Rubinstein, W. S., Church, D. M., & Maglott, D. R. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1113 · Cited by 3,516
REVEL: An Ensemble Method for Predicting the Pathogenicity of Rare Missense Variants
Ioannidis, N. M., Rothstein, J. H., Pejaver, V., Middha, S., McDonnell, S. K., Baheti, S., Musolf, A., Li, Q., Holzinger, E., Karyadi, D., et al. (2016) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2016.08.016 · Cited by 2,908
Predicting the Functional Effect of Amino Acid Substitutions and Indels
Choi, Y., Sims, G. E., Murphy, S., Miller, J. R., & Chan, A. P. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046688 · Cited by 2,890
UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches
Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B., Wu, C. H., & the UniProt Consortium (2015) · Bioinformatics · doi:10.1093/bioinformatics/btu739 · Cited by 1,928
Learning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., & Harmeling, S. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206594 · Cited by 1,849
Identifying a High Fraction of the Human Genome to be under Selective Constraint Using GERP++
Davydov, E. V., Goode, D. L., Sirota, M., Cooper, G. M., Sidow, A., & Batzoglou, S. (2010) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001025 · Cited by 1,757
The mutational constraint spectrum quantified from variation in 141,456 humans
Karczewski, K. J., Francioli, L. C., Tiao, G., Cummings, B. B., Alföldi, J., Wang, Q., Collins, R. L., Laricchia, K. M., Ganna, A., Birnbaum, D. P., et al. (2019) · doi:10.1101/531210 · Cited by 1,662
SIFT missense predictions for genomes
Vaser, R., Adusumalli, S., Leng, S. N., Sikic, M., & Ng, P. C. (2016) · Nature Protocols · doi:10.1038/nprot.2015.123 · Cited by 1,510
Cited by
Accurate proteome-wide missense variant effect prediction with AlphaMissense
Cheng, J., Novati, G., Pan, J., Bycroft, C., Žemgulytė, A., Applebaum, T., Pritzel, A., Wong, L. H., Zielinski, M., Sargeant, T., et al. (2023) · Science · doi:10.1126/science.adg7492 · Cited by 2,163
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580
“To compile the nucleotide information from human DMS studies, we narrowed the scope of the set of datasets used in our human benchmark to the human datasets used in reference ( 45 ) to benchmark mutational effect predictors.”
Schubach, M., Maass, T., Nazaretyan, L., Röner, S., & Kircher, M. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkad989 · Cited by 455
“It should also be noted that distinguishing methods by their means of training only (e.g. supervised vs. unsupervised learning approaches) is insufficient and that the training objective needs to be considered ( 46 ,47 ).”
BackgroundMethodGenome-wide prediction of disease variant effects with a deep protein language model
Brandes, N., Goldman, G., Wang, C. H., Ye, C. J., & Ntranos, V. (2023) · Nature Genetics · doi:10.1038/s41588-023-01465-0 · Cited by 421
(2023) · doi:10.1101/2023.12.07.570727 · Cited by 207
“Combining DMS with clinical annotations has been a fruitful direction to avoid biases [Frazer et al., 2021, Livesey and Marsh, 2023].”
BackgroundCross-protein transfer learning substantially improves disease variant prediction
(2023) · doi:10.1186/s13059-023-03024-6 · Cited by 68
Livesey, B. J., & Marsh, J. A. (2025) · Genome Biology · doi:10.1186/s13059-025-03575-w · Cited by 50
Self-citationGerasimavicius, L., Livesey, B. J., & Marsh, J. A. (2023) · Protein Science · doi:10.1002/pro.4688 · Cited by 41
Self-citationProtein representations: Encoding biological information for machine learning in biocatalysis
Harding-Larsen, D., Funk, J., Madsen, N. G., Gharabli, H., Acevedo-Rocha, C. G., Mazurenko, S., & Welner, D. H. (2024) · Biotechnology Advances · doi:10.1016/j.biotechadv.2024.108459 · Cited by 38
Expert-guided protein language models enable accurate and blazingly fast fitness prediction
Marquet, C., Schlensok, J., Abakarova, M., Rost, B., & Laine, E. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae621 · Cited by 33
Fawzy, M., & Marsh, J. A. (2024) · Scientific Reports · doi:10.1038/s41598-024-76202-6 · Cited by 22
Self-citation(2024) · doi:10.1101/2024.06.12.598724 · Cited by 22
Deep mutational scanning quantifies DNA binding and predicts clinical outcomes of PAX6 variants
McDonnell, A. F., Plech, M., Livesey, B. J., Gerasimavicius, L., Owen, L. J., Hall, H. N., FitzPatrick, D. R., Marsh, J. A., & Kudla, G. (2024) · Molecular Systems Biology · doi:10.1038/s44320-024-00043-8 · Cited by 16
Self-citationTsishyn, M., Hermans, P., Rooman, M., & Pucci, F. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf322 · Cited by 15
Proteome-scale prediction of molecular mechanisms underlying dominant genetic diseases
(2024) · doi:10.1371/journal.pone.0307312 · Cited by 15
Insights for variant clinical interpretation based on a benchmark of 65 variant effect predictors
Radjasandirane, R., Diharce, J., Gelly, J.-C., & de Brevern, A. G. (2025) · Genomics · doi:10.1016/j.ygeno.2025.111036 · Cited by 15
Multiplex and multimodal mapping of variant effects in secreted proteins via MultiSTEP
Popp, N. A., Powell, R. L., Wheelock, M. K., Holmes, K. J., Zapp, B. D., Sheldon, K. M., Fletcher, S. N., Wu, X., Fayer, S., Rubin, A. F., et al. (2025) · Nature Structural & Molecular Biology · doi:10.1038/s41594-025-01582-w · Cited by 15
Variant scoring tools for deep mutational scanning
Çubuk, H., Jin, X., Phipson, B., Marsh, J. A., & Rubin, A. F. (2025) · Molecular Systems Biology · doi:10.1038/s44320-025-00137-x · Cited by 13
Self-citationLuppino, F., Adzhubei, I. A., Cassa, C. A., & Toth-Petroczy, A. (2023) · Nature Communications · doi:10.1038/s41467-023-37661-z · Cited by 12
Why variant effect predictors and multiplexed assays agree and disagree
Livesey, B. J., & Marsh, J. A. (2025) · doi:10.1101/2025.07.31.667868 · Cited by 7
Self-citationProteome-scale prediction of molecular mechanisms underlying dominant genetic diseases
Badonyi, M., & Marsh, J. A. (2023) · doi:10.1101/2023.09.08.556798 · Cited by 6
Self-citationGerasimavicius, L., Livesey, B. J., & Marsh, J. A. (2023) · doi:10.1101/2023.02.03.527007 · Cited by 5
Self-citationProteome-wide prediction of the functional impact of missense variants with ProteoCast
Abakarova, M., Freiberger, M. I., Liehrmann, A., Rera, M., & Laine, E. (2026) · Nature Communications · doi:10.1038/s41467-026-72140-1 · Cited by 5
Finucane, H. K., Parsa, S., Guez, J., Kanai, M., Satterstrom, F. K., Nkambule, L. L., Daly, M. J., Seed, C., & Karczewski, K. J. (2024) · doi:10.1101/2024.09.17.613327 · Cited by 4
“Second, as in previous benchmarking work 3 , we aggregate pairwise comparisons to avoid restricting to the small set of variants scored by all VSMs (Supplementary Information).”
MethodSystematic characterization of indel variants using a yeast-based protein folding sensor
Larsen-Ledet, S., Lindemose, S., Panfilova, A., Gersing, S., Suhr, C. H., Genzor, A. V., Lanters, H., Nielsen, S. V., Lindorff-Larsen, K., Winther, J. R., et al. (2025) · Structure · doi:10.1016/j.str.2024.11.017 · Cited by 3
Aslanzadeh, V., Glendinning, S., Sari, Y., Clarke, M., Rodger, C. J., Lin, X., Lenka, D. R., Richardson, L., Liang, J. R., Kleinz, T., et al. (2026) · doi:10.64898/2026.02.09.704817 · Cited by 1
“…in the clinical classification task, with GEMME only outperformed by SNPred and MetaRNN, two predictors whose performance might be overestimated due to the use of clinical data in training; such predictors trained on clinical data might not perform as well for previously unreported variants [44].”
Deciphering GB1’s Single Mutational Landscape: Insights from MuMi Analysis
Guclu, T. F., Atilgan, A. R., & Atilgan, C. (2024) · The Journal of Physical Chemistry B · doi:10.1021/acs.jpcb.4c04916
Rendering protein mutation movies with MutAmore
Weissenow, K., & Rost, B. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05610-8
Metadata
- Authors
- Livesey, B. J., & Marsh, J. A.
- Venue
- Molecular Systems Biology
- Published
- 2023-06-13
- Volume
- 19
- Issue
- 8
- Pages
- e11474-e11474
- ISSN
- 1744-4292 (ISSN-L)
- OpenAlex
- W4380422659
- Cited by
- 108
- References
- 25
- FWCI
- 16.26
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.