Updated benchmarking of variant effect predictors using deep mutational scanning
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
The assessment of variant effect predictor (VEP) performance is fraught with biases introduced by benchmarking against clinical observations. In this study, building on our previous work, we use independently generated measurements of protein function from deep mutational scanning (DMS) experiments for 26 human proteins to benchmark 55 different VEPs, while introducing minimal data circularity. Many top‐performing VEPs are unsupervised methods including EVE, DeepSequence and ESM‐1v, a protein language model that ranked first overall. However, the strong performance of recent supervised VEPs, in particular VARITY, shows that developers are taking data circularity and bias issues seriously. We also assess the performance of DMS and unsupervised VEPs for discriminating between known pathogenic and putatively benign missense variants. Our findings are mixed, demonstrating that some DMS datasets perform exceptionally at variant classification, while others are poor. Notably, we observe a striking correlation between VEP agreement with DMS data and performance in identifying clinically relevant variants, strongly supporting the validity of our rankings and the utility of DMS for independent benchmarking.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The mutational constraint spectrum quantified from variation in 141,456 humans
Karczewski, K. J., Francioli, L. C., Tiao, G., Cummings, B. B., Alföldi, J., Wang, Q., Collins, R. L., Laricchia, K. M., Ganna, A., Birnbaum, D. P., et al. (2020) · Nature · doi:10.1038/s41586-020-2308-7 · Cited by 9,434
Amino acid substitution matrices from protein blocks.
Henikoff, S., & Henikoff, J. G. (1992) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.89.22.10915 · Cited by 6,218
ClinVar: public archive of relationships among sequence variation and human phenotype
Landrum, M. J., Lee, J. M., Riley, G. R., Jang, W., Rubinstein, W. S., Church, D. M., & Maglott, D. R. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1113 · Cited by 3,516
REVEL: An Ensemble Method for Predicting the Pathogenicity of Rare Missense Variants
Ioannidis, N. M., Rothstein, J. H., Pejaver, V., Middha, S., McDonnell, S. K., Baheti, S., Musolf, A., Li, Q., Holzinger, E., Karyadi, D., et al. (2016) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2016.08.016 · Cited by 2,908
Predicting the Functional Effect of Amino Acid Substitutions and Indels
Choi, Y., Sims, G. E., Murphy, S., Miller, J. R., & Chan, A. P. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046688 · Cited by 2,890
UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches
Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B., Wu, C. H., & the UniProt Consortium (2015) · Bioinformatics · doi:10.1093/bioinformatics/btu739 · Cited by 1,928
Learning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., & Harmeling, S. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206594 · Cited by 1,849
Identifying a High Fraction of the Human Genome to be under Selective Constraint Using GERP++
Davydov, E. V., Goode, D. L., Sirota, M., Cooper, G. M., Sidow, A., & Batzoglou, S. (2010) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001025 · Cited by 1,757
The mutational constraint spectrum quantified from variation in 141,456 humans
Karczewski, K. J., Francioli, L. C., Tiao, G., Cummings, B. B., Alföldi, J., Wang, Q., Collins, R. L., Laricchia, K. M., Ganna, A., Birnbaum, D. P., et al. (2019) · doi:10.1101/531210 · Cited by 1,662
SIFT missense predictions for genomes
Vaser, R., Adusumalli, S., Leng, S. N., Sikic, M., & Ng, P. C. (2016) · Nature Protocols · doi:10.1038/nprot.2015.123 · Cited by 1,510
Deep mutational scanning: a new style of protein science
Fowler, D. M., & Fields, S. (2014) · Nature Methods · doi:10.1038/nmeth.3027 · Cited by 1,287 · cited 1×
“DMS encompasses a wide variety of high-throughput experimental techniques, whereby functional scores for large numbers of amino acid variants are measured (Fowler & Fields, 2014).”
BackgroundInferring the molecular and phenotypic impact of amino acid variants with MutPred2
Pejaver, V., Urresti, J., Lugo-Martinez, J., Pagel, K. A., Lin, G. N., Nam, H.-J., Mort, M., Cooper, D. N., Sebat, J., Iakoucheva, L. M., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-19669-x · Cited by 923
Accurate classification of BRCA1 variants with saturation genome editing
Findlay, G. M., Daza, R. M., Martin, B., Zhang, M. D., Leith, A. P., Gasperini, M., Janizek, J. D., Huang, X., Starita, L. M., & Shendure, J. (2018) · Nature · doi:10.1038/s41586-018-0461-z · Cited by 843
Disease variant prediction with deep generative models of evolutionary data
Frazer, J., Notin, P., Dias, M., Gomez, A., Min, J. K., Brock, K., Gal, Y., & Marks, D. S. (2021) · Nature · doi:10.1038/s41586-021-04043-8 · Cited by 791 · cited 16×
“EVE, which has the lowest coverage of the top VEPs (84.0”
BackgroundInfluentialMutation effects predicted from sequence co-variation
Hopf, T. A., Ingraham, J. B., Poelwijk, F. J., Schärfe, C. P. I., Springer, M., Sander, C., & Marks, D. S. (2017) · Nature Biotechnology · doi:10.1038/nbt.3769 · Cited by 775
Deep generative models of genetic variation capture the effects of mutations
Riesselman, A. J., Ingraham, J. B., & Marks, D. S. (2018) · Nature Methods · doi:10.1038/s41592-018-0138-4 · Cited by 601
Mutational processes shape the landscape of TP53 mutations in human cancer
Giacomelli, A. O., Yang, X., Lintner, R. E., McFarland, J. M., Duby, M., Kim, J., Howard, T. P., Takeda, D. Y., Ly, S. H., Kim, E., et al. (2018) · Nature Genetics · doi:10.1038/s41588-018-0204-y · Cited by 556
Multiplex assessment of protein variant abundance by massively parallel sequencing
Matreyek, K. A., Starita, L. M., Stephany, J. J., Martin, B., Chiasson, M. A., Gray, V. E., Kircher, M., Khechaduri, A., Dines, J. N., Hause, R. J., et al. (2018) · Nature Genetics · doi:10.1038/s41588-018-0122-z · Cited by 436
Identifying novel constrained elements by exploiting biased substitution patterns
Garber, M., Guttman, M., Clamp, M., Zody, M. C., Friedman, N., & Xie, X. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp190 · Cited by 370
Grimm, D. G., Azencott, C., Aicheler, F., Gieraths, U., MacArthur, D. G., Samocha, K. E., Cooper, D. N., Stenson, P. D., Daly, M. J., Smoller, J. W., et al. (2015) · Human Mutation · doi:10.1002/humu.22768 · Cited by 356 · cited 1×
“Such benchmarking studies are frequently conducted both by VEP authors and independent groups, traditionally by comparing VEP classifications to sets of known pathogenic and benign variants (Gunning et al, 2021; Niroula & Vihinen, 2019).”
Regional missense constraint improves variant deleteriousness prediction
Samocha, K. E., Kosmicki, J. A., Karczewski, K. J., O’Donnell-Luria, A. H., Pierce-Hoffman, E., MacArthur, D. G., Neale, B. M., & Daly, M. J. (2017) · doi:10.1101/148353 · Cited by 351
PERCH: A Unified Framework for Disease Gene Prioritization: HUMAN MUTATION
Feng, B.-J. (2017) · Human Mutation · doi:10.1002/humu.23158 · Cited by 289
Jones, E. M., Lubock, N. B., Venkatakrishnan, A., Wang, J., Tseng, A. M., Paggi, J. M., Latorraca, N. R., Cancilla, D., Satyadi, M., Davis, J. E., et al. (2020) · eLife · doi:10.7554/elife.54895 · Cited by 168
Deconstruction of the Ras switching cycle through saturation mutagenesis
Bandaru, P., Shah, N. H., Bhattacharyya, M., Barton, J. P., Kondo, Y., Cofsky, J. C., Gee, C. L., Chakraborty, A. K., Kortemme, T., Ranganathan, R., et al. (2017) · eLife · doi:10.7554/elife.27810 · Cited by 117
Structure of PDE3A-SLFN12 complex reveals requirements for activation of SLFN12 RNase
Garvie, C. W., Wu, X., Papanastasiou, M., Lee, S., Fuller, J., Schnitzler, G. R., Horner, S. W., Baker, A., Zhang, T., Mullahoo, J. P., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-24495-w · Cited by 85
Cited by
Accurate proteome-wide missense variant effect prediction with AlphaMissense
Cheng, J., Novati, G., Pan, J., Bycroft, C., Žemgulytė, A., Applebaum, T., Pritzel, A., Wong, L. H., Zielinski, M., Sargeant, T., et al. (2023) · Science · doi:10.1126/science.adg7492 · Cited by 2,163
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580
“To compile the nucleotide information from human DMS studies, we narrowed the scope of the set of datasets used in our human benchmark to the human datasets used in reference ( 45 ) to benchmark mutational effect predictors.”
Schubach, M., Maass, T., Nazaretyan, L., Röner, S., & Kircher, M. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkad989 · Cited by 455
“It should also be noted that distinguishing methods by their means of training only (e.g. supervised vs. unsupervised learning approaches) is insufficient and that the training objective needs to be considered ( 46 ,47 ).”
BackgroundMethodGenome-wide prediction of disease variant effects with a deep protein language model
Brandes, N., Goldman, G., Wang, C. H., Ye, C. J., & Ntranos, V. (2023) · Nature Genetics · doi:10.1038/s41588-023-01465-0 · Cited by 421
(2023) · doi:10.1101/2023.12.07.570727 · Cited by 207
“Combining DMS with clinical annotations has been a fruitful direction to avoid biases [Frazer et al., 2021, Livesey and Marsh, 2023].”
Background
Metadata
- Authors
- Livesey, B. J., & Marsh, J. A.
- Venue
- Molecular Systems Biology
- Published
- 2023-06-13
- Volume
- 19
- Issue
- 8
- Pages
- e11474-e11474
- ISSN
- 1744-4292 (ISSN-L)
- OpenAlex
- W4380422659
- Cited by
- 108
- References
- 25
- FWCI
- 16.26
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.