Reports from CAGI: The Critical Assessment of Genome Interpretation
Open accessThe DOI matches its Crossref record.
Abstract
New genomic and large-scale data hold the promise of revolutionizing our understanding and treatment of human disease, and are already influencing clinical practice. Multiple barriers stand between the acquisition of the data and fully realizing these and other benefits. In particular, we need powerful and well-characterized computational methods for deducing the phenotypic impact of genomic and system-level perturbations. Many such methods have been developed, but currently, even though some are already deployed in clinical settings, we often remain ignorant of how they actually perform, as well as how and when they should be applied. Further, it is already clear that new and more sophisticated approaches must be developed to fully meet these challenges. The Critical Assessment of Genome Interpretation (CAGI, \ˈkā-jē\) conducts community experiments to objectively assess computational methods for determining the phenotypic impacts of genomic variation. The primary goals are to establish the state of the art, to show where future progress may best be made, to highlight innovations and progress, and to build a strong collaborative community. In the CAGI experiments, participants are typically provided genetic variants and make blind predictions of resulting phenotypes. These predictions are evaluated against gold-standard experimental or clinical data by independent assessors. Four CAGI experiments have been conducted to date—a pilot in 2010, and three full-scale events in 2011, 2013, and 2016. Each edition of CAGI involves about 10 challenges. The experiment is conducted over a period of about a year, starting with the identification and development of suitable challenges, followed by a period during which participants are invited to submit their predictions, then a term in which the independent assessors evaluate the results, and concluding with a meeting to discuss the outcomes. CAGI challenges span a wide range of relationships between genetic variation and disease. For single base variants, there are challenges that address the problem of interpreting the impact of missense mutations on protein activity using a variety of molecular and cellular phenotypes, challenges that test the ability to predict the effect of mutations in cancer driver genes on cell growth, and challenges on the effect of single-base variants on RNA expression levels and splicing (including Beer, 2017; Capriotti, Martelli, Fariselli, & Casadio, 2017; Carraro et al., 2017; Katsonis & Lichtarge, 2017; Kreimer et al., 2017; Niroula & Vihinen 2017; Pejaver et al., 2017; Tang et al., 2017; Tang & Fenton, 2017; Xu et al., 2017; Yin et al., 2017; Zeng, Edwards, Guo, & Gifford, 2017; Zhang et al., 2017). At the level of full exome and genome sequence, there are challenges that assess methods for assigning complex traits phenotypes and that evaluate the ability to associate genome sequence and an extensive profile of phenotypic traits (including Cai et al., 2017; Daneshjou et al., 2014; Daneshjou et al., 2017; Giollo et al., 2017; Laksshman, Bhat, Viswanath, & Li, 2017; Pal, Kundu, Yin, & Moult, 2017a; Wang et al., 2017). CAGI has also included challenges in which participants were asked to identify causative variants for rare diseases in gene panel, exome, and whole-genome sequence data (including Chandonia et al., 2017; Kundu, Pal, Yin, & Moult, 2017; Pal, Kundu, Yin, & Moult, 2017b). Many challenges have focused on cancer, given its prevalence and the impact of genetics. This special issue of Human Mutation contains a selection papers reporting the assessments of challenge results, as well as papers from some individual participating teams, describing their methods, and the results obtained. Most papers report on the recent challenges, from CAGI4, held in 2016. As CAGI best helps further development when challenges reoccur year after year, some manuscripts discuss results from the earlier editions of CAGI and their development over time. Together, these results from CAGI offer powerful insights into the appropriate level of confidence to place in variant annotations and interpretation methods, and which classes of approaches are most suitable for a particular application. They reveal limitations of current data collection and analysis approaches and point to areas for future research and new approaches. The fifth CAGI edition is presently underway. Full information about this and the previous CAGI editions is available at http://www.genomeinterpretation.org. We are most grateful to all CAGI participants. The primary contributors whose work is assessed in CAGI is that of the predictors: Allison Abad, Ogun Adebali, Ivan Adzhubey, Talal Amin, Johnathan R. Azaria, Giulia Babbi, Eraan Bachar, Benjamin Bachman, Minkyung Baek, Greet De Baets, Michael Beer, Violeta Beleva-Guthrie, Bonnie Berger, Brady Bernard, Rajendra Bhat, Rohit Bhattacharya, Samuele Bovo, Marcus Breese, Aharon S. Brodie, Yana Bromberg, Binghuang Cai, Colin Campbell, Chen Cao,
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
Wainberg, M., Merico, D., Delong, A., & Frey, B. J. (2018) · Nature Biotechnology · doi:10.1038/nbt.4233 · Cited by 585
Wells, D. K., van Buuren, M. M., Dang, K. K., Hubbard-Lucey, V. M., Sheehan, K. C. F., Campbell, K. M., Lamb, A., Ward, J. P., Sidney, J., Blazquez, A. B., et al. (2020) · Cell · doi:10.1016/j.cell.2020.09.015 · Cited by 538
“, 2017), and others (Guinney et al., 2017; Hoskins et al., 2017; Keller et al., 2017; Kreimer et al., 2017), no such effort for neoepitope prediction has been undertaken.”
BackgroundMMSplice: modular modeling improves the predictions of genetic variant effects on splicing
Cheng, J., Nguyen, T. Y. D., Cygan, K. J., Çelik, M. H., Fairbrother, W. G., Avsec, Ž., & Gagneur, J. (2019) · Genome Biology · doi:10.1186/s13059-019-1653-z · Cited by 235
Olson, N. D., Wagner, J., McDaniel, J., Stephens, S. H., Westreich, S. T., Prasanna, A. G., Johanson, E., Boja, E., Maier, E. J., Serang, O., et al. (2022) · Cell Genomics · doi:10.1016/j.xgen.2022.100129 · Cited by 202
VIPdb, a genetic variant impact predictor database
Hu, Z., Yu, C., Furutsuki, M., Andreoletti, G., Ly, M., Hoskins, R., Adhikari, A. N., & Brenner, S. E. (2019) · Human Mutation · doi:10.1002/humu.23858 · Cited by 37
Metadata
- Authors
- Hoskins, R. A., Repo, S., Barsky, D., Andreoletti, G., Moult, J., & Brenner, S. E.
- Venue
- Human Mutation
- Publisher
- Wiley
- Published
- 2017-08-17
- Volume
- 38
- Issue
- 9
- Pages
- 1039-1041
- ISSN
- 1059-7794 (ISSN-L)
- 1098-1004
- OpenAlex
- W2735037223
- Cited by
- 36
- References
- 0
- FWCI
- 3.60
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Reports from CAGI: The Critical Assessment of Genome Interpretation. https://app.cochat.ai/papers/10.1002/humu.23290
- The paper
- Hoskins, R. A., Repo, S., Barsky, D., Andreoletti, G., Moult, J., & Brenner, S. E. (2017). Reports from CAGI: The Critical Assessment of Genome Interpretation. Human Mutation. https://doi.org/10.1002/humu.23290
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.