Artificial intelligence to identify genetic alterations in conventional histopathology
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Precision oncology relies on the identification of targetable molecular alterations in tumor tissues. In many tumor types, a limited set of molecular tests is currently part of standard diagnostic workflows. However, universal testing for all targetable alterations, especially rare ones, is limited by the cost and availability of molecular assays. From 2017 to 2021, multiple studies have shown that artificial intelligence (AI) methods can predict the probability of specific genetic alterations directly from conventional hematoxylin and eosin (H&E) tissue slides. Although these methods are currently less accurate than gold standard testing (e.g. immunohistochemistry, polymerase chain reaction or next‐generation sequencing), they could be used as pre‐screening tools to reduce the workload of genetic analyses. In this systematic literature review, we summarize the state of the art in predicting molecular alterations from H&E using AI. We found that AI methods perform reasonably well across multiple tumor types, although few algorithms have been broadly validated. In addition, we found that genetic alterations in FGFR, IDH, PIK3CA, BRAF, TP53, and DNA repair pathways are predictable from H&E in multiple tumor types, while many other genetic alterations have rarely been investigated or were only poorly predictable. Finally, we discuss the next steps for the implementation of AI‐based surrogate tests in diagnostic workflows. © 2022 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Type, Density, and Location of Immune Cells Within Human Colorectal Tumors Predict Clinical Outcome
Galon, J., Costes, A., Sanchez-Cabo, F., Kirilovsky, A., Mlecnik, B., Lagorce-Pagès, C., Tosolini, M., Camus, M., Berger, A., Wind, P., et al. (2006) · Science · doi:10.1126/science.1129139 · Cited by 6,516
Tumour heterogeneity and resistance to cancer therapies
Dagogo-Jack, I., & Shaw, A. T. (2018) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.166 · Cited by 3,846 · cited 1×
“Intratumor heterogeneity is an intrinsic feature of cancers that represents the spatial and temporal variations of cell populations within the same tumor [157].”
BackgroundGrandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455 · cited 1×
“While AI has been used to automate board games [24], video games [25], and other complex tasks, in the context of medicine it is mostly used for image analysis.”
MethodCoudray, N., Ocampo, P. S., Sakellaropoulos, T., Narula, N., Snuderl, M., Fenyö, D., Moreira, A. L., Razavian, N., & Tsirigos, A. (2018) · Nature Medicine · doi:10.1038/s41591-018-0177-5 · Cited by 2,930 · cited 9×
“In 2018, a landmark study by Coudray et al surprisingly showed that multiple clinically relevant mutations in lung cancer are predictable from digitized H&E slides alone [29].”
BackgroundMethodInfluentialClinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799
Development of tumor mutation burden as an immunotherapy biomarker: utility for the oncology clinic
Chan, T. A., Yarchoan, M., Jaffee, E., Swanton, C., Quezada, S. A., Stenzinger, A., & Peters, S. (2019) · Annals of Oncology · doi:10.1093/annonc/mdy495 · Cited by 2,683
Data-efficient and weakly supervised computational pathology on whole-slide images
Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Chen, R. J., Barbieri, M., & Mahmood, F. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-020-00682-w · Cited by 2,282
Causability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885
Kather, J. N., Pearson, A. T., Halama, N., Jäger, D., Krause, J., Loosen, S. H., Marx, A., Boor, P., Tacke, F., Neumann, U. P., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0462-y · Cited by 1,458 · cited 3×
“While the prediction performance in this initial publication was too low for any immediate clinical application, it proved the concept and sparked dozens of follow-up publications [36 – 38].”
BackgroundMethodSelf-citationPredicting cancer outcomes from histology and genomics using convolutional networks
Mobadersany, P., Yousefi, S., Amgad, M., Gutman, D. A., Barnholtz-Sloan, J. S., Velázquez Vega, J. E., Brat, D. J., & Cooper, L. A. D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1717139115 · Cited by 1,084
Intratumor Heterogeneity: The Rosetta Stone of Therapy Resistance
Marusyk, A., Janiszewska, M., & Polyak, K. (2020) · Cancer Cell · doi:10.1016/j.ccell.2020.03.007 · Cited by 1,009
Deep learning in cancer pathology: a new generation of clinical biomarkers
Echle, A., Rindtorff, N. T., Brinker, T. J., Luedde, T., Pearson, A. T., & Kather, J. N. (2021) · British Journal of Cancer · doi:10.1038/s41416-020-01122-x · Cited by 667 · cited 1×
“Also, most studies use surgical resection specimens, which yield large contiguous areas of tumor tissue on a slide [132].”
BackgroundSelf-citationDeep learning for prediction of colorectal cancer outcome: a discovery and validation study
Skrede, O.-J., De Raedt, S., Kleppe, A., Hveem, T. S., Liestøl, K., Maddison, J., Askautrud, H. A., Pradhan, M., Nesheim, J. A., Albregtsen, F., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(19)32998-8 · Cited by 627
Deep learning-based classification of mesothelioma improves prediction of patient outcome
Courtiol, P., Maussion, C., Moarii, M., Pronier, E., Pilcer, S., Sefta, M., Manceron, P., Toldo, S., Zaslavskiy, M., Le Stang, N., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0583-3 · Cited by 544 · cited 1×
“In prognostication, DL has been shown to outperform some established risk factors in colorectal cancer [26–28], breast cancer [27], lung cancer [29], sarcoma [30], bladder cancer [31], glioma [32], mesothelioma [33], and hepatocellular carcinoma [34,35], among other tumor types.”
MethodIntegrating spatial gene expression and breast tumour morphology via deep learning
He, B., Bergenstråhle, L., Stenbeck, L., Abid, A., Andersson, A., Borg, Å., Maaskola, J., Lundeberg, J., & Zou, J. (2020) · Nature Biomedical Engineering · doi:10.1038/s41551-020-0578-x · Cited by 507 · cited 2×
“It has been shown that such DL-based spatial heterogeneity re fl ects an underlying genetic heterogeneity in bladder cancer [46,54].”
BackgroundMethodYamashita, R., Long, J., Longacre, T., Peng, L., Berry, G., Martin, B., Higgins, J., Rubin, D. L., & Shen, J. (2021) · The Lancet Oncology · doi:10.1016/s1470-2045(20)30535-0 · Cited by 391 · cited 3×
“Consecutively, it was demonstrated that DL can predict MSI status from histology as well [28,36,57,60,61,64,66,67,69,83,112].”
BackgroundClinical-Grade Detection of Microsatellite Instability in Colorectal Tumors by Deep Learning
Echle, A., Grabsch, H. I., Quirke, P., van den Brandt, P. A., West, N. P., Hutchins, G. G. A., Heij, L. R., Tan, X., Richman, S. D., Krause, J., et al. (2020) · Gastroenterology · doi:10.1053/j.gastro.2020.06.021 · Cited by 376
Self-citationMultiple instance learning for histopathological breast cancer image classification
Sudharshan, P. J., Petitjean, C., Spanhol, F., Oliveira, L. E., Heutte, L., & Honeine, P. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.09.049 · Cited by 369
The impact of site-specific digital histology signatures on deep learning model accuracy and bias
Howard, F. M., Dolezal, J., Kochanny, S., Schulte, J., Chen, H., Heij, L., Huo, D., Nanda, R., Olopade, O. I., Kather, J. N., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-24698-1 · Cited by 330
Self-citationCouture, H. D., Williams, L. A., Geradts, J., Nyante, S. J., Butler, E. N., Marron, J. S., Perou, C. M., Troester, M. A., & Niethammer, M. (2018) · npj Breast Cancer · doi:10.1038/s41523-018-0079-1 · Cited by 327
Obuchowski (2005) · doi:10.2214/ajr.184.2.01840364 · Cited by 321
Bilal, M., Raza, S. E. A., Azam, A., Graham, S., Ilyas, M., Cree, I. A., Snead, D., Minhas, F., & Rajpoot, N. M. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00180-1 · Cited by 299
Self supervised contrastive learning for digital histopathology
Ciga, O., Xu, T., & Martel, A. L. (2022) · Machine Learning with Applications · doi:10.1016/j.mlwa.2021.100198 · Cited by 294
Pathologic Predictors of Microsatellite Instability in Colorectal Cancer
Greenson, J. K., Huang, S.-C., Herron, C., Moreno, V., Bonner, J. D., Tomsho, L. P., Ben-Izhak, O., Cohen, H. I., Trougouboff, P., Bejhar, J., et al. (2009) · American Journal of Surgical Pathology · doi:10.1097/pas.0b013e31817ec2b1 · Cited by 286 · cited 3×
“…patterns, speci fi c morphological patterns have been linked to speci fi c genetic alterations in cancer, such as BRAF mutations [11] and microsatellite instability (MSI) in colorectal cancer [12], hormone receptor overexpression in breast cancer [13], and EGFR mutations in lung cancer [14].”
BackgroundDeep learning-based survival prediction for multiple cancer types using histopathology images
Wulczyn, E., Steiner, D. F., Xu, Z., Sadhwani, A., Wang, H., Flament-Auvigne, I., Mermel, C. H., Chen, P.-H. C., Liu, Y., & Stumpe, M. C. (2020) · PLOS ONE · doi:10.1371/journal.pone.0233678 · Cited by 266
Sirinukunwattana, K., Domingo, E., Richman, S. D., Redmond, K. L., Blake, A., Verrill, C., Leedham, S. J., Chatzipli, A., Hardy, C., Whalley, C. M., et al. (2021) · Gut · doi:10.1136/gutjnl-2019-319866 · Cited by 264 · cited 2×
“[65] 2021 Gene expression Consensus molecular subtype AUROC = 0.”
BackgroundA Graph-Transformer for Whole Slide Image Classification
Zheng, Y., Gindra, R. H., Green, E. J., Burks, E. J., Betke, M., Beane, J. E., & Kolachalama, V. B. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3176598 · Cited by 255
Halligan, S., Altman, D. G., & Mallett, S. (2015) · European Radiology · doi:10.1007/s00330-014-3487-0 · Cited by 255 · cited 1×
“For example, the AUROC is affected by class imbalance [92].”
BackgroundInterpretable survival prediction for colorectal cancer using deep learning
Wulczyn, E., Steiner, D. F., Moran, M., Plass, M., Reihs, R., Tan, F., Flament-Auvigne, I., Brown, T., Regitnig, P., Chen, P.-H. C., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00427-2 · Cited by 231 · cited 1×
“Second, new features can be discovered, potentially yielding new mechanistic insight [152 – 154].”
BackgroundNoorbakhsh, J., Farahmand, S., Foroughi pour, A., Namburi, S., Caruana, D., Rimm, D., Soltanieh-ha, M., Zarringhalam, K., & Chuang, J. H. (2020) · Nature Communications · doi:10.1038/s41467-020-20030-5 · Cited by 206
Levy-Jurgenson, A., Tekpli, X., Kristensen, V. N., & Yakhini, Z. (2020) · Scientific Reports · doi:10.1038/s41598-020-75708-z · Cited by 155
Determining breast cancer biomarker status and associated morphological features using deep learning
Gamble, P., Jaroensri, R., Wang, H., Tan, F., Moran, M., Brown, T., Flament-Auvigne, I., Rakha, E. A., Toss, M., Dabbs, D. J., et al. (2021) · Communications Medicine · doi:10.1038/s43856-021-00013-3 · Cited by 154 · cited 1×
“Commonly used interpretability methods in digital pathology research are heat maps, class activation maps, t-distributed stochastic neighbor embedding (t-SNE) plots, and generation of top predicted patients and tiles [30,37,155,156].”
MethodHong, R., Liu, W., DeLair, D., Razavian, N., & Fenyö, D. (2021) · Cell Reports Medicine · doi:10.1016/j.xcrm.2021.100400 · Cited by 154
Quality control stress test for deep learning-based diagnostic model in digital pathology
Schömig-Markiefka, B., Pryalukhin, A., Hulla, W., Bychkov, A., Fukuoka, J., Madabhushi, A., Achter, V., Nieroda, L., Büttner, R., Quaas, A., et al. (2021) · Modern Pathology · doi:10.1038/s41379-021-00859-x · Cited by 146
Sha, L., Osinski, B. L., Ho, I. Y., Tan, T. L., Willis, C., Weiss, H., Beaubier, N., Mahon, B. M., Taxter, T. J., & Yip, S. S. F. (2019) · Journal of Pathology Informatics · doi:10.4103/jpi.jpi_24_19 · Cited by 130
Muti, H. S., Heij, L. R., Keller, G., Kohlruss, M., Langer, R., Dislich, B., Cheong, J.-H., Kim, Y.-W., Kim, H., Kook, M.-C., et al. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00133-3 · Cited by 126 · cited 5×
“A few studies have compared the performance of DL systems applied to biopsy samples [61,93] or virtual biopsies [69] and found a decline of predictive performance.”
BackgroundResultInfluentialSelf-citationEchle, A., Ghaffari Laleh, N., Quirke, P., Grabsch, H. I., Muti, H. S., Saldanha, O. L., Brockmoeller, S. F., van den Brandt, P. A., Hutchins, G. G. A., Richman, S. D., et al. (2022) · ESMO Open · doi:10.1016/j.esmoop.2022.100400 · Cited by 111 · cited 3×
“Also, an AUROC does not specify a fixed threshold or operating point, but this is required for clinical application [93].”
BackgroundResultSelf-citationLiao, H., Long, Y., Han, R., Wang, W., Xu, L., Liao, M., Zhang, Z., Wu, Z., Shang, X., Li, X., et al. (2020) · Clinical and Translational Medicine · doi:10.1002/ctm2.102 · Cited by 92 · cited 9×
“Mutations in the BRAF gene were predictable when a DL model was trained on tissue microarrays (TMAs) in thyroid cancer and tested on WSIs from the independent TCGA thyroid cancer dataset with an AUROC of 0.98, proving the transferability of the method from TMAs to WSIs [89].”
BackgroundInfluentialBenchmarking artificial intelligence methods for end-to-end computational pathology
Laleh, N. G., Muti, H. S., Lavinia Loeffler, C. M., Echle, A., Saldanha, O. L., Mahmood, F., Lu, M. Y., Trautwein, C., Langer, R., Dislich, B., et al. (2021) · doi:10.1101/2021.08.09.455633 · Cited by 13 · cited 1×
“tasks, with residual networks (‘resnets’) giving a good tradeoff between performance and computational effort [81,134].”
BackgroundSelf-citation
Cited by
Multistain deep learning for prediction of prognosis and therapy response in colorectal cancer
Foersch, S., Glasner, C., Woerl, A.-C., Eckstein, M., Wagner, D.-C., Schulz, S., Kellers, F., Fernandez, A., Tserea, K., Kloth, M., et al. (2023) · Nature Medicine · doi:10.1038/s41591-022-02134-1 · Cited by 251
Self-citationAn MRI deep learning model predicts outcome in rectal cancer
Jiang, X., Zhao, H., Saldanha, O. L., Nebelung, S., Kuhl, C., Amygdalos, I., Lang, S. A., Wu, X., Meng, X., Truhn, D., et al. (2023) · Radiology · doi:10.1148/radiol.222223 · Cited by 127
Self-citationTruhn, D., Loeffler, C. M., Müller‐Franzes, G., Nebelung, S., Hewitt, K. J., Brandner, S., Bressem, K. K., Foersch, S., & Kather, J. N. (2024) · The Journal of Pathology · doi:10.1002/path.6232 · Cited by 124
“Based on our experiments, we conclude that it is likely that LLMs such as GPT-4 will play a signi fi cant role in advancing the development of deep learning models in precision oncology [2,3].”
BackgroundMethodSelf-citationRegression-based Deep-Learning predicts molecular biomarkers from pathology slides
El Nahhas, O. S. M., Loeffler, C. M. L., Carrero, Z. I., van Treeck, M., Kolbinger, F. R., Hewitt, K. J., Muti, H. S., Graziani, M., Zeng, Q., Calderaro, J., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-45589-1 · Cited by 108
Self-citationExplainability and causability in digital pathology
Plass, M., Kargl, M., Kiehl, T., Regitnig, P., Geißler, C., Evans, T., Zerbe, N., Carvalho, R., Holzinger, A., & Müller, H. (2023) · The Journal of Pathology Clinical Research · doi:10.1002/cjp2.322 · Cited by 99
Metadata
- Authors
- Cifci, D., Foersch, S., & Kather, J. N.
- Venue
- The Journal of Pathology
- Publisher
- Wiley
- Published
- 2022-04-21
- Volume
- 257
- Issue
- 4
- Pages
- 430-444
- ISSN
- 0022-3417 (ISSN-L)
- 1096-9896
- OpenAlex
- W4221030665
- Cited by
- 126
- References
- 43
- FWCI
- 8.64
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.