CoChat
Sign Up
PreprintCited by 14Genomics and Chromatin DynamicsMachine Learning in BioinformaticsGenomics and Phylogenetic Studies

Uncertainty-aware genomic deep learning with knowledge distillation

Open access
Zhou, Jessica, Rizzo, Kaeli, Christensen, Trevor, Tang, Ziqi, Koo, Peter K·2024·npj Artificial Intelligence
DOI 10.1101/2024.11.13.623485PMC PMC11601481

Preprint — published as 10.1038/s44387-025-00053-3

Open this paper in CoChat — free account

Abstract

Deep neural networks (DNNs) have advanced predictive modeling for regulatory genomics, but challenges remain in ensuring the reliability of their predictions and understanding the key factors behind their decision making. Here we introduce DEGU (Distilling Ensembles for Genomic Uncertainty-aware models), a method that integrates ensemble learning and knowledge distillation to improve the robustness and explainability of DNN predictions. DEGU distills the predictions of an ensemble of DNNs into a single model, capturing both the average of the ensemble’s predictions and the variability across them, with the latter representing epistemic (or model-based) uncertainty. DEGU also includes an optional auxiliary task to estimate aleatoric, or data-based, uncertainty by modeling variability across experimental replicates. By applying DEGU across various functional genomic prediction tasks, we demonstrate that DEGU-trained models inherit the performance benefits of ensembles in a single model, with improved generalization to out-of-distribution sequences and more consistent explanations of cis-regulatory mechanisms through attribution analysis. Moreover, DEGU-trained models provide calibrated uncertainty estimates, with conformal prediction offering coverage guarantees under minimal assumptions. Overall, DEGU paves the way for robust and trustworthy applications of deep learning in genomics research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2024.11.13.623485

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The distilled student model achieved improved robustness and variant effect prediction accuracy in a single model instance.

    “This distilled student model, as shown previously14, achieved improved robustness and variant effect prediction accuracy in a single model instance, making predictions across all modelled modalities and cell types with a single device call per variant.”

    From Advancing regulatory variant effect prediction with AlphaGenome (2026) · cites this paper as [14]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 59 · Cited by 14

Research works printed but not linked: 63

  1. [1]Avsec Ž. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods 18, 1196– 1203 (2021). 34608324 PMC8490152
  2. [2]Chen K. M., Wong A. K., Troyanskaya O. G. & Zhou J. A sequence-based global map of regulatory activity for deciphering human genetics. Nature genetics 54, 940– 949 (2022). 35817977 PMC9279145
  3. [3]Dudnyk K., Cai D., Shi C., Xu J. & Zhou J. Sequence basis of transcription initiation in the human genome. Science 384, eadj0116 (2024). 38662817 PMC11223672
  4. [4]Zhou J. & Troyanskaya O. G. Predicting effects of noncoding variants with deep learning–based sequence model. Nat Methods 12, 931– 934 (Oct. 2015). 26301843 PMC4768299
  5. [5]Kelley D. R., Snoek J. & Rinn J. L. Basset: learning the regulatory code of the accessible genome with deep convolutional neural networks. Genome Res 26, 990– 999 (July 2016). 27197224 PMC4937568
  6. [6]Koo P. K., Majdandzic A., Ploenzke M., Anand P. & Paul S. B. Global importance analysis: An interpretability method to quantify importance of genomic features in deep neural networks. en. PLOS Computational Biology 17. Publisher: Public Library of Science, e1008925 (May 2021). 33983921 PMC8118286
  7. [7]Avsec Ž. Base-resolution models of transcription-factor binding reveal soft motif syntax. en. Nat Genet 53. Publisher: Nature Publishing Group, 354– 366 (Mar. 2021). 33603233 PMC8812996
  8. [8]Toneyan S., Tang Z. & Koo P. K. Evaluating deep learning for predicting epigenomic profiles. Nat Mach Intell 4, 1088– 1100 (Dec. 2022). 37324054 PMC10270674
  9. [10]Taskiran I. I. Cell-type-directed design of synthetic enhancers. en. Nature 626. Publisher: Nature Publishing Group, 212– 220 (Feb. 2024). 38086419 PMC10830415
  10. [11]De Almeida B. P. Targeted design of synthetic enhancers for selected tissues in the Drosophila embryo. en. Nature 626. Publisher: Nature Publishing Group, 207– 211 (Feb. 2024). 38086418 PMC10830412
  11. [12]Gosai S. J. Machine-guided design of cell-type-targeting cis-regulatory elements. en. Nature 634. Publisher: Nature Publishing Group, 1211– 1220 (Oct. 2024). 39443793 PMC11525185
  12. [13]Yin C. Iterative deep learning-design of human enhancers exploits condensed sequence grammar to achieve cell type-specificity en. Pages: 2024.06.14.599076 Section: New Results. June 2024. PMC12221157 40472848
  13. [14]Sarkar A., Tang Z., Zhao C. & Koo P. K. Designing DNA With Tunable Regulatory Activity Using Discrete Diffusion. bioRxiv (2024).
  14. [15]Shimodaira H. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference 90, 227– 244 (Oct. 2000).
  15. [16]Karollus A., Mauermeier T. & Gagneur J. Current sequence-based models capture gene expression determinants in promoters but mostly ignore distal enhancers. Genome Biology 24, 56 (Mar. 2023). 36973806 PMC10045630
  16. [17]Seitz E. E., McCandlish D. M., Kinney J. B. & Koo P. K. Interpreting cis-regulatory mechanisms from genomic deep neural networks using surrogate models. bioRxiv, 2023.11.14.567120 (Mar. 2024). PMC11823438 39950082
  17. [18]Sasse A. Benchmarking of deep neural networks for predicting personal gene expression from DNA sequence highlights shortcomings. en. Nat Genet 55. Publisher: Nature Publishing Group, 2060– 2064 (Dec. 2023). 38036778
  18. [19]Huang C. Personal transcriptome variation is poorly explained by current genomic deep learning models. Nat Genet 55, 2056– 2059 (2023). 38036790 PMC10703684
  19. [20]Dietterich T. G. Ensemble Methods in Machine Learning in Multiple Classifier Systems (Springer Berlin Heidelberg, Berlin, Heidelberg, 2000), 1– 15.
  20. [21]Lakshminarayanan B., Pritzel A. & Blundell C. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles arXiv:1612.01474 [cs, stat]. Nov. 2017. · arXiv
  21. [22]Malina S., Cizin D. & Knowles D. A. Deep mendelian randomization: Investigating the causal knowledge of genomic deep learning models. PLoS computational biology 18, e1009880 (2022). 36265006 PMC9624391
  22. [24]Linder J., Srivastava D., Yuan H., Agarwal V. & Kelley D. R. Predicting RNA-seq coverage from DNA sequence as a unifying model of gene regulation en. Pages: 2023.08.30.555582 Section: New Results. Sept. 2023. PMC11985352 39779956
  23. [25]He A. Y. & Danko C. G. Dissection of core promoter syntax through single nucleotide resolution modeling of transcription initiation. bioRxiv (2024).
  24. [27]Gyawali P. K., Liu X., Zou J. & He Z. Ensembling improves stability and power of feature selection for deep learning models arXiv:2210.00604 [cs]. Oct. 2022. · arXiv
  25. [28]Majdandzic A., Rajesh C. & Koo P. K. Correcting gradient-based interpretations of deep neural networks for genomics. Genome Biology 24, 109 (May 2023). 37161475 PMC10169356
  26. [29]Novakovsky G., Dexter N., Libbrecht M. W., Wasserman W. W. & Mostafavi S. Obtaining genetics insights from deep learning via explainable artificial intelligence. Nature Reviews Genetics 24, 125– 137 (2023). 36192604
  27. [32]Zhou J. Sequence-based modeling of three-dimensional genome architecture from kilobase to chromosome scale. eng. Nat Genet 54, 725– 734 (May 2022). 35551308 PMC9186125
  28. [33]Karbalayghareh A., Sahin M. & Leslie C. S. Chromatin interaction–aware gene regulatory modeling with graph attention networks. Genome Res 32, 930– 944 (May 2022). 35396274 PMC9104700
  29. [34]Rastogi R., Reddy A. J., Chung R. & Ioannidis N. M. Fine-tuning sequence-to-expression models on personal genome and transcriptome data en. Pages: 2024.09.23.614632 Section: New Results. Sept. 2024. PMC13386774 42186093
  30. [36]Lal A. Decoding sequence determinants of gene expression in diverse cellular and disease states. bioRxiv, 2024– 10 (2024). 42185539
  31. [37]Hinton G., Vinyals O. & Dean J. Distilling the Knowledge in a Neural Network arXiv:1503.02531 [cs, stat]. Mar. 2015. · arXiv
  32. [38]Malinin A., Mlodozeniec B. & Gales M. Ensemble Distribution Distillation arXiv:1905.00076 [cs, stat]. Nov. 2019. · arXiv
  33. [40]De Almeida B. P., Reiter F., Pagani M. & Stark A. DeepSTARR predicts enhancer activity from DNA sequence and enables the de novo design of synthetic enhancers. en. Nat Genet 54. Publisher: Nature Publishing Group, 613– 624 (May 2022). 35551305
  34. [41]Buenrostro J. D., Wu B., Chang H. Y. & Greenleaf W. J. ATAC-seq: A Method for Assaying Chromatin Accessibility Genome-Wide. en. Current Protocols in Molecular Biology 109 (Jan. 2015). PMC4374986 25559105
  35. [42]Tang Z., Somia N., Yu Y. & Koo P. K. Evaluating the representational power of pre-trained DNA language models for regulatory genomics. bioRxiv, 2024– 02 (2024). PMC12261763 40660356
  36. [43]Lee N. K., Tang Z., Toneyan S. & Koo P. K. EvoAug: improving generalization and interpretability of genomic deep neural networks with evolution-inspired data augmentations. Genome Biology 24, 105 (May 2023). 37143118 PMC10161416
  37. [44]Yu Y., Muthukumar S. & Koo P. K. EvoAug-TF: Extending evolution-inspired data augmentations for genomic deep learning to TensorFlow. Bioinformatics 40, btae092 (2024). 38366935 PMC10918628
  38. [45]Arpit D., Wang H., Zhou Y. & Xiong C. Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization in Advances in Neural Information Processing Systems (eds Koyejo S. et al.) 35 (Curran Associates, Inc., 2022), 8265– 8277.
  39. [46]Fort S., Ren J. & Lakshminarayanan B. Exploring the limits of out-of-distribution detection. Advances in Neural Information Processing Systems 34, 7068– 7081 (2021).
  40. [47]Wilson A. G. & Izmailov P. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems 33, 4697– 4708 (2020).
  41. [48]Hoffmann L., Fortmeier I. & Elster C. Uncertainty quantification by ensemble learning for computational optical form measurements. Machine Learning: Science and Technology 2, 035030 (2021).
  42. [49]Lundberg S. & Lee S.-I. A Unified Approach to Interpreting Model Predictions arXiv:1705.07874 [cs, stat]. Nov. 2017. · arXiv
  43. [52]Koo P. K. & Ploenzke M. Improving representations of genomic sequence motifs in convolutional networks with exponential activations. Nature machine intelligence 3, 258– 266 (2021). PMC8315445 34322657
  44. [53]Bartlett P. L., Long P. M., Lugosi G. & Tsigler A. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences 117, 30063– 30070 (2020). PMC7720150 32332161
  45. [54]Han T., Srinivas S. & Lakkaraju H. Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations. en. Advances in Neural Information Processing Systems 35, 5256– 5268 (Dec. 2022).
  46. [55]Wang Z. Smoothed Geometry for Robust Attribution in Advances in Neural Information Processing Systems 33 (Curran Associates, Inc., 2020), 13623– 13634.
  47. [57]Majdandzic A. Selecting deep neural networks that yield consistent attribution-based interpretations for genomics in Machine Learning in Computational Biology (2022), 131– 149. PMC10194041 37205975
  48. [58]Gal Y. & Ghahramani Z. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning arXiv:1506.02142 [cs, stat]. Oct. 2016. · arXiv
  49. [59]Amini A., Schwarting W., Soleimany A. & Rus D. Deep Evidential Regression arXiv:1910.02600 [cs, stat]. Nov. 2020. · arXiv
  50. [60]Nix D. A. & Weigend A. S. Estimating the mean and variance of the target probability distribution in Proceedings of 1994 ieee international conference on neural networks (ICNN’94) 1 (1994), 55– 60.
  51. [61]Venkatesh B. & Thiagarajan J. J. Heteroscedastic Calibration of Uncertainty Estimators in Deep Learning arXiv:1910.14179 [cs, stat]. Oct. 2019. · arXiv
  52. [62]Barber R. F., Candes E. J., Ramdas A. & Tibshirani R. J. Predictive inference with the jackknife+ arXiv:1905.02928 [stat]. May 2020. · arXiv
  53. [63]Papadopoulos H., Proedrou K., Vovk V. & Gammerman A. Inductive Confidence Machines for Regression en. in Machine Learning: ECML 2002 (eds Elomaa T., Mannila H. & Toivonen H.) (Springer, Berlin, Heidelberg, 2002), 345– 356.
  54. [64]Vovk V., Gammerman A. & Shafer G. in Algorithmic Learning in a Random World Journal Abbreviation: Algorithmic Learning in a Random World (Jan. 2005).
  55. [65]Critical Assessment of Genome Interpretation Consortium. CAGI, the Critical Assessment of Genome Interpretation, establishes progress and prospects for computational genetic variant interpretation methods. eng. Genome Biol 25, 53 (Feb. 2024). 38389099 PMC10882881
  56. [66]Shigaki D. Integration of multiple epigenomic marks improves prediction of variant impact in saturation mutagenesis reporter assay. eng. Hum Mutat 40, 1280– 1291 (Sept. 2019). 31106481 PMC6879779
  57. [67]Linder J., Srivastava D., Yuan H., Agarwal V. & Kelley D. R. Predicting RNA-seq coverage from DNA sequence as a unifying model of gene regulation. bioRxiv, 2023– 08 (2023). PMC11985352 39779956
  58. [68]Zhang L., Bao C. & Ma K. Self-Distillation: Towards Efficient and Compact Neural Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 4388– 4403 (2022). 33735074
  59. [69]Allen-Zhu Z. & Li Y. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 (2020). · arXiv
  60. [70]The ENCODE Project Consortium. An integrated encyclopedia of DNA elements in the human genome. en. Nature 489, 57– 74 (Sept. 2012). 22955616 PMC3439153
  61. [71]Zerbino D. R., Johnson N., Juettemann T., Wilder S. P. & Flicek P. WiggleTools: parallel processing of large collections of genome-wide datasets for visualization and statistical analysis. Bioinformatics 30. _ eprint: https://academic.oup.com/bioinformatics/article-pdf/30/7/1008/48921616/bioinformatics_30_7_1008.pdf, 1008–1009 (Dec. 2013). PMC3967112 24363377
  62. [72]Kent W. J., Zweig A. S., Barber G., Hinrichs A. S. & Karolchik D. BigWig and BigBed: enabling browsing of large distributed datasets. Bioinformatics 26. _ eprint: https://academic.oup.com/bioinformatics/article-pdf/26/17/2204/48855290/bioinformatics_26_2204-2207 (July 2010). PMC2922891 20639541
  63. [73]He K., Zhang X., Ren S. & Sun J. Deep residual learning for image recognition in Proceedings of the IEEE conference on computer vision and pattern recognition (2016), 770– 778.

Authors
Zhou, J., Rizzo, K., Christensen, T., Tang, Z., & Koo, P. K.
Affiliations
Cold Spring Harbor Laboratory (US)
Corresponding author
Koo, Peter K
ORCID
Venue
npj Artificial Intelligence
Publisher
openRxiv
Published
2024-11-15
Volume
2
Issue
1
Pages
3-3
ISSN
3005-1460 (ISSN-L)
DOI
10.1101/2024.11.13.623485
PMC
PMC11601481
OpenAlex
W7118837599
Cited by
14
References
59
Citations by year
2026: 11 · 2025: 3
FWCI
22.73
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Artificial Intelligence (publishedVersion, cc-by-nc-nd)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • Cold Spring Harbor Laboratory Institutional Repository (Cold Spring Harbor Laboratory) (acceptedVersion)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institutes of Health (1s10od028632-01)
  • National Human Genome Research Institute (R01HG012131)
  • National Institute of General Medical Sciences (R01GM149921)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Chromatin Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
Keywords
uncertainty estimation, knowledge distillation, ensemble learning, epistemic uncertainty, aleatoric uncertainty, regulatory genomics, model interpretability, attribution analysis, conformal prediction, out-of-distribution generalization, degu

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review