CoChat

Applying interpretable machine learning in computational biology—pitfalls, recommendations and opportunities for new developments

Open access
Chen, Valerie, Yang, Muyu, Cui, Wenbo, Kim, Joon Sik, Talwalkar, Ameet, Ma, Jian · 2024 · Nature Methods
Cited by 115

The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.

Abstract

Recent advances in machine learning have enabled the development of next-generation predictive models for complex computational biology problems, thereby spurring the use of interpretable machine learning (IML) to unveil biological insights. However, guidelines for using IML in computational biology are generally underdeveloped. We provide an overview of IML methods and evaluation techniques and discuss common pitfalls encountered when applying IML methods to computational biology problems. We also highlight open questions, especially in the era of large language models, and call for collaboration between IML and computational biology researchers. This Perspective discusses the methodologies, application and evaluation of interpretable machine learning (IML) approaches in computational biology, with particular focus on common pitfalls when using IML and how to avoid them.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41592-024-02359-7

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 28 references

Research works printed but not linked: 52

  1. Doshi-Velez, F. & Kim, B. Towards a rigorous science of interpretable machine learning. Preprint at https://arxiv.org/abs/1702.08608 (2017). · arXiv
  2. Azodi, C. B., Tang, J. & Shiu, S. -H. Opening the black box: interpretable machine learning for geneticists. Trends Genet. 36, 442–455 (2020). · DOI · Paper page
  3. Eraslan, G., Avsec, Ž., Gagneur, J. & Theis, F. J. Deep learning: new computational modelling techniques for genomics. Nat. Rev. Genet. 20, 389–403 (2019). This paper gives an extensive review of the application of deep learning models in genomics. · DOI · Paper page
  4. Novakovsky, G., Dexter, N., Libbrecht, M. W., Wasserman, W. W. & Mostafavi, S. Obtaining genetics insights from deep learning via explainable artificial intelligence. Nat. Rev. Genet. 24, 125–137 (2023). This paper provides a comprehensive review for the commonly applied IML methods in biology through the examples from regulatory genomics. · DOI · Paper page
  5. Klauschen, F. et al. Toward explainable artificial intelligence for precision pathology. Annu. Rev. Pathol. 19, 541–570 (2024). · DOI
  6. Schwessinger, R. et al. DeepC: predicting 3D genome folding using megabase-scale transfer learning. Nat. Methods 17, 1118–1124 (2020). · DOI · Paper page
  7. Karimi, M., Wu, D., Wang, Z. & Shen, Y. DeepAffinity: interpretable deep learning of compound–protein affinity through unified recurrent and convolutional neural networks. Bioinformatics 35, 3329–3338 (2019). · DOI · Paper page
  8. Taujale, R. et al. Mapping the glycosyltransferase fold landscape using interpretable deep learning. Nat. Commun. 12, 5656 (2021). · DOI
  9. Tasaki, S., Gaiteri, C., Mostafavi, S. & Wang, Y. Deep learning decodes the principles of differential gene expression. Nat. Mach. Intell. 2, 376–386 (2020). · DOI · Paper page
  10. Tao, Y. et al. Interpretable deep learning for chromatin-informed inference of transcriptional programs driven by somatic alterations across cancers. Nucleic Acids Res. 50, 10869–10881 (2022). · DOI
  11. Barnett, A. J. et al. A case-based interpretable deep learning model for classification of mass lesions in digital mammography. Nat. Mach. Intell. 3, 1061–1070 (2021). · DOI · Paper page
  12. Zaritsky, A. et al. Interpretable deep learning uncovers cellular properties in label-free live cell images that are predictive of highly metastatic melanoma. Cell Syst. 12, 733–747 (2021). · DOI
  13. DeGrave, A. J., Cai, Z. R., Janizek, J. D., Daneshjou, R. & Lee, S. -I. Auditing the inference processes of medical-image classifiers by leveraging generative AI and the expertise of physicians. Nat. Biomed. Eng. https://doi.org/10.1038/s41551-023-01160-9 (2023). · DOI
  14. Heil, B. J. et al. Reproducibility standards for machine learning in the life sciences. Nat. Methods 18, 1132–1135 (2021). · DOI · Paper page
  15. Whalen, S., Schreiber, J., Noble, W. S. & Pollard, K. S. Navigating the pitfalls of applying machine learning in genomics. Nat. Rev. Genet. 23, 169–181 (2022). · DOI · Paper page
  16. Sapoval, N. et al. Current progress and open challenges for applying deep learning across the biosciences. Nat. Commun. 13, 1728 (2022). · DOI · Paper page
  17. Chen, V., Li, J., Kim, J. S., Plumb, G. & Talwalkar, A. Interpretable machine learning: moving from mythos to diagnostics. Commun. ACM 65, 43–50 (2022). This paper describes the disconnect between IML techniques and downstream use cases and outlines paths forward to bridge the disconnect. · DOI · Paper page
  18. Räuker, T., Ho, A., Casper, S. & Hadfield-Menell, D. Toward transparent AI: a survey on interpreting the inner structures of deep neural networks. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 464–483 (IEEE, 2023). · DOI · Paper page
  19. Yang, M. & Ma, J. Machine learning methods for exploring sequence determinants of 3D genome organization. J. Mol. Biol. 434, 167666 (2022). · DOI
  20. Sundararajan, M., Taly, A. & Yan, Q. Axiomatic attribution for deep networks. In International Conference on Machine Learning, 3319–3328 (PMLR, 2017).
  21. Nair, S., Shrikumar, A., Schreiber, J. & Kundaje, A. fastISM: performant in silico saturation mutagenesis for convolutional neural networks. Bioinformatics 38, 2397–2403 (2022). · DOI
  22. Ribeiro, M. T., Singh, S. & Guestrin, C. “Why should I trust you?” explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD, 1135–1144 (2016). · DOI · Paper page
  23. Tseng, A., Shrikumar, A. & Kundaje, A. Fourier-transform-based attribution priors improve the interpretability and stability of deep learning models for genomics. Adv. Neural Inf. Process. Syst. 33, 1913–1923 (2020).
  24. Hastie, T. & Tibshirani, R. Generalized additive models: some applications. J. Am. Stat. Assoc. 82, 371–386 (1987). · DOI · Paper page
  25. Janizek, J. D. et al. PAUSE: principled feature attribution for unsupervised gene expression analysis. Genome Biol. 24, 81 (2023). This paper proposes an approach to combining the post hoc and by-design explanation methods. · DOI
  26. Vaswani, A. et al. Attention is all you need. In Advances in Neural Information Processing Systems, 5998–6008 (2017).
  27. Karbalayghareh, A., Sahin, M. & Leslie, C. S. Chromatin interaction–aware gene regulatory modeling with graph attention networks. Genome Res. 32, 930–944 (2022).
  28. Jain, S. & Wallace, B. C. Attention is not explanation. In Proceedings of NAACL-HLT, 3543–3556 (2019).
  29. Bai, B. et al. Why attentions may not be interpretable? In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 25–34 (2021). · DOI
  30. Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S. & Garriga-Alonso, A. Towards automated circuit discovery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Processing Systems, 16318–16352 (2023).
  31. Friedman, D., Wettig, A. & Chen, D. Learning transformer programs. In Advances in Neural Information Processing Systems, vol. 36 (2023).
  32. Alvarez Melis, D. & Jaakkola, T. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems, vol. 31 (2018).
  33. Yang, M. & Kim, B. Benchmarking attribution methods with relative feature importance. Preprint at https://arxiv.org/abs/1907.09701 (2019). · arXiv
  34. Kim, J. S., Plumb, G. & Talwalkar, A. Sanity simulations for saliency methods. In Proceedings of the 39th International Conference on Machine Learning, 11173–11200 (2022).
  35. Zhou, Y., Booth, S., Ribeiro, M. T. & Shah, J. Do feature attribution methods correctly attribute features? In Proceedings of the AAAI Conference on Artificial Intelligence, 36, 9623–9633 (2022). · DOI · Paper page
  36. Agarwal, C. et al. Openxai: towards a transparent evaluation of model explanations. In Advances in Neural Information Processing Systems, 35 (2022).
  37. Krishna, S. et al. The disagreement problem in explainable machine learning: a practitioner’s perspective. Preprint at https://arxiv.org/abs/2202.01602 (2022). · DOI · Paper page
  38. Zhao, Y., Shao, J. & Asmann, Y. W. Assessment and optimization of explainable machine learning models applied to transcriptomic data. Genomics Proteomics Bioinformatics 20, 899–911 (2022). · DOI
  39. Tang, Z. et al. Interpretable classification of Alzheimer’s disease pathologies with a convolutional neural network pipeline. Nat. Commun. 10, 2173 (2019). · DOI · Paper page
  40. Zeiler, M. D. & Fergus, R. Visualizing and understanding convolutional networks. In (eds. Fleet, D et al.) Computer Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol 8689. Springer, Cham. https://doi.org/10.1007/978-3-319-10590-1_53 (2014). · DOI · Paper page
  41. Rahman, M. A. & Rangwala, H. IDMIL: an alignment-free interpretable Deep Multiple Instance Learning (MIL) for predicting disease from whole-metagenomic data. Bioinformatics 36, i39–i47 (2020). · DOI
  42. Nagao, Y., Sakamoto, M., Chinen, T., Okada, Y. & Takao, D. Robust classification of cell cycle phase and biological feature extraction by image-based deep learning. Mol. Biol. Cell 31, 1346–1354 (2020). · DOI · Paper page
  43. Lafarge, M. W. et al. Capturing single-cell phenotypic variation via unsupervised representation learning. In International Conference on Medical Imaging with Deep Learning, 315–325 (PMLR, 2019).
  44. Tan, J. et al. Cell-type-specific prediction of 3D chromatin organization enables high-throughput in silico genetic screening. Nat. Biotechnol. 41, 1140–1150 (2023). · DOI
  45. Tang, X. et al. Explainable multi-task learning for multi-modality biological data analysis. Nat. Commun. 14, 2546 (2023). · DOI
  46. Washburn, J. D. et al. Evolutionarily informed deep learning methods for predicting relative transcript abundance from DNA sequence. Proc. Natl Acad. Sci. USA 116, 5542–5549 (2019). · DOI · Paper page
  47. Liu, K., Casper, S., Hadfield-Menell, D. & Andreas, J. Cognitive dissonance: why do language model outputs disagree with internal representations of truthfulness? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing 4791–4797 (2023). · DOI
  48. Srivastava, D., Aydin, B., Mazzoni, E. O. & Mahony, S. An interpretable bimodal neural network characterizes the sequence and preexisting chromatin predictors of induced transcription factor binding. Genome Biol. 22, 20 (2021). · DOI
  49. Liu, C., Huang, H. & Yang, P. Multi-task learning from multimodal single-cell omics with Matilda. Nucleic Acids Res. 51, e45 (2023). · DOI
  50. Liang, P. P. et al. MultiViz: towards visualizing and understanding multimodal models. In Eleventh International Conference on Learning Representations (2023).
  51. Valeri, J. A. et al. BioAutoMATED: an end-to-end automated machine learning tool for explanation and design of biological sequences. Cell Syst. 14, 525–542 (2023). · DOI
  52. Ji, Y., Zhou, Z., Liu, H. & Davuluri, R. V. DNABERT: pre-trained bidirectional encoder representations from transformers model for DNA-language in genome. Bioinformatics 37, 2112–2120 (2021). · DOI · Paper page

Cited by

See all 115

Metadata

Authors
Chen, V., Yang, M., Cui, W., Kim, J. S., Talwalkar, A., & Ma, J.
Venue
Nature Methods
Published
2024-08
Volume
21
Issue
8
Pages
1454-1461
ISSN
  • 1548-7091 (ISSN-L)
  • 1548-7105
OpenAlex
W4401455176
Cited by
115
References
28
FWCI
16.97
Type
journal article
Language
en
Open access
green
Licence
Funders
  • U.S. Department of Health & Human Services | National Institutes of Health (R01HG012303, R01HG007352, UM1HG011593)
  • National Science Foundation (IIS1705121, IIS1838017, IIS2046613, IIS2112471)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Applying interpretable machine learning in computational biology—pitfalls, recommendations and opportunities for new developments. https://app.cochat.ai/papers/10.1038/s41592-024-02359-7
The paper
Chen, V., Yang, M., Cui, W., Kim, J. S., Talwalkar, A., & Ma, J. (2024). Applying interpretable machine learning in computational biology—pitfalls, recommendations and opportunities for new developments. Nature Methods. https://doi.org/10.1038/s41592-024-02359-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.