The DOI matches its Crossref record; the references it cites are not checked yet.
Explainability in transformer models for functional genomics
Open accessAbstract
The effectiveness of deep learning methods can be largely attributed to the automated extraction of relevant features from raw data. In the field of functional genomics, this generally concerns the automatic selection of relevant nucleotide motifs from DNA sequences. To benefit from automated learning methods, new strategies are required that unveil the decision-making process of trained models. In this paper, we present a new approach that has been successful in gathering insights on the transcription process in Escherichia coli. This work builds upon a transformer-based neural network framework designed for prokaryotic genome annotation purposes. We find that the majority of subunits (attention heads) of the model are specialized towards identifying transcription factors and are able to successfully characterize both their binding sites and consensus sequences, uncovering both well-known and potentially novel elements involved in the initiation of the transcription process. With the specialization of the attention heads occurring automatically, we believe transformer models to be of high interest towards the creation of explainable neural networks in this field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
The citation lists haven’t been fetched yet
References 51 · Cited by 60
Research works printed but not linked: 44
- Alipanahi, 2015, Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning, Nat Biotechnol, 33, 831, 10.1038/nbt.3300 · DOI · Paper page
- Ayers, 1989, Promoter recognition by Escherichia coli RNA polymerase: role of the spacer DNA in functional complex formation, J Mol Biol, 207, 749, 10.1016/0022-2836(89)90241-6 · DOI
- Barne, 1997, Region 2.5 of the Escherichia coli RNA polymerase Σ70 subunit is responsible for the recognition of the ‘extended-10’ motif at promoters, EMBO J, 16, 4034, 10.1093/emboj/16.13.4034 · DOI
- Burgess, 2001, Encyclopedia of Genetics, 1831, 10.1006/rwgn.2001.1192 · DOI
- Clauwaert, 2020, Novel transformer networks for improved sequence labeling in genomics, IEEE/ACM Trans Comput Biol Bioinform, 1, 10.1109/TCBB.2020.3035021 · DOI
- Dai, 2019, Transformer-XL: attentive language models beyond a fixed-length context, arXiv
- Eraslan, 2019, Deep learning: new computational modelling techniques for genomics, Nat Rev Genet, 20, 389, 10.1038/s41576-019-0122-6 · DOI
- Ettwiller, 2016, A novel enrichment strategy reveals unprecedented number of novel transcription start sites at single base resolution in a model prokaryote and the gut microbiome, BMC Genomics, 17, 199, 10.1186/s12864-016-2539-z · DOI
- Feklistov, 2006, A basal promoter element recognized by free RNA polymerase σ subunit determines promoter recognition by RNA polymerase holoenzyme, Mol Cell, 23, 97, 10.1016/j.molcel.2006.06.010 · DOI
- Fisher, 2018, All models are wrong, but many are useful: learning a variable’s importance by studying an entire class of prediction models simultaneously, arXiv
- Grainger, 2007, Transcription factor distribution in Escherichia coli: studies with FNR protein, Nucleic Acids Res, 35, 269, 10.1093/nar/gkl1023 · DOI
- Harr, 1983, Search algorithm for pattern match analysis of nucleic acid sequences, Nucleic Acids Res, 11, 2943, 10.1093/nar/11.9.2943 · DOI
- Haugen, 2006, rRNA promoter regulation by nonoptimal binding of σ region 1.2: an additional recognition element for RNA polymerase, Cell, 125, 1069, 10.1016/j.cell.2006.04.034 · DOI
- Hill, 2018, A deep recurrent neural network discovers complex biological rules to decipher RNA protein-coding potential, Nucleic Acids Res, 46, 8105, 10.1093/nar/gky567 · DOI
- Ishida, 2009, Participation of regulator AscG of the β-glucoside utilization operon in regulation of the propionate catabolism operon, J Bacteriol, 191, 6136, 10.1128/JB.00663-09 · DOI
- Jain, 2019, Attention is not explanation, arXiv
- Xiangwu, 2019, Full-length RNA profiling reveals pervasive bidirectional transcription terminators in bacteria, Nat Microbiol, 4, 1907, 10.1038/s41564-019-0500-z · DOI
- Kanhere, 2005, A novel method for prokaryotic promoter prediction based on DNA stability, BMC Bioinformatics, 6, 1, 10.1186/1471-2105-6-1 · DOI
- Lawrence, 1990, An expectation maximization (EM) algorithm for the identification and characterization of common sites in unaligned biopolymer sequences, Proteins, 7, 41, 10.1002/prot.340070105 · DOI
- Lin, 2018, Identifying sigma70 promoters with novel pseudo nucleotide composition, IEEE/ACM Trans Comput Biol Bioinform, 16, 1
- Lozinski, 1989, Effect of the sequence-dependent structure of the 17 bp AT spacer on the strength of consensus-like E.coli promoters in vivo, Nucleic Acids Res, 17, 3855, 10.1093/nar/17.10.3855 · DOI
- Madan Babu, 2003, Evolution of transcription factors and the gene regulatory network in Escherichia coli, Nucleic Acids Res, 31, 1234, 10.1093/nar/gkg210 · DOI
- Nikam, 2019, Seq2Feature: a comprehensive web-based feature extraction tool, Bioinformatics, 35, 4797, 10.1093/bioinformatics/btz432 · DOI
- Park, PLoS Genet, 9, 10.1371/journal.pgen.1003839 · DOI
- PLoS Genet, 5
- Rahman, 2019, iPro70-FMWin: identifying Sigma70 promoters using multiple windowing and minimal features, Mol Genet Genomics, 294, 69, 10.1007/s00438-018-1487-5 · DOI
- Ruth, 2011, Mechanism of bacterial transcription initiation: RNA polymerase - promoter binding, isomerization to initiation-competent open complexes, and initiation of RNA synthesis, J Mol Biol, 412, 754, 10.1016/j.jmb.2011.01.018 · DOI
- Santos, 2019, RegulonDB v 10.5: tackling challenges to unify classic and high throughput knowledge of gene regulation in E. coli K-12, Nucleic Acids Res, 47, D212, 10.1093/nar/gky1077 · DOI
- Schwartz, 2001, IscR, an Fe-S cluster-containing transcription factor, represses expression of Escherichia coli genes encoding Fe-S cluster assembly proteins, Proc Natl Acad Sci U S A, 98, 14895, 10.1073/pnas.251550898 · DOI
- Shearwin, 2005, Transcriptional interference – a crash course, Trends Genet, 21, 339, 10.1016/j.tig.2005.04.009 · DOI
- Shrikumar, 2019, Learning important features through propagating activation differences, arXiv
- Siebenlist, 1980, E. coli RNA polymerase interacts homologously with two different promoters, Cell, 20, 269, 10.1016/0092-8674(80)90613-3 · DOI
- Simonyan, 2014, Deep inside convolutional networks: visualising image classification models and saliency maps, arXiv
- Stormo, 1989, Identifying protein-binding sites from unaligned DNA fragments, Proc Natl Acad Sci U S A, 86, 1183, 10.1073/pnas.86.4.1183 · DOI
- Sundararajan, 2017, Axiomatic attribution for deep networks, arXiv
- Treviño-Quintanilla, 2013, Anti-sigma factors in E. coli: common regulatory mechanisms controlling sigma factors availability, Curr Genomics, 14, 378, 10.2174/1389202911314060007 · DOI
- Vaswani, 2017, Attention is all you need, arXiv
- Wang, 2018, Image-based promoter prediction: a promoter prediction method based on evolutionarily generated patterns, Sci Rep, 8, 10.1038/s41598-018-36308-0 · DOI
- Yan, 2018, SMRT-Cappable-seq reveals complex operon variants in bacteria, Nat Commun, 9, 1, 10.1038/s41467-018-05997-6 · DOI
- Zeiler, 2014, Computer Vision – ECCV 2014, 818, 10.1007/978-3-319-10590-1_53 · DOI · Paper page
- Zhu, 1999, SCPD: a promoter database of the yeast Saccharomyces cerevisiae, Bioinformatics, 15, 607, 10.1093/bioinformatics/15.7.607 · DOI
- Zintgraf, 2017, Visualizing deep neural network decisions: prediction difference analysis, arXiv
- Zou, 2019, A primer on deep learning in genomics, Nat Genet, 51, 12, 10.1038/s41588-018-0295-5 · DOI · Paper page
- Zuallaert, 2018, SpliceRover: interpretable convolutional neural networks for improved splice site prediction, Bioinformatics, 34, 4180, 10.1093/bioinformatics/bty497 · DOI
Metadata
- Authors
- Clauwaert, J., Menschaert, G., & Waegeman, W.
- Affiliations
- Ghent University (BE)
- Corresponding author
- Clauwaert, Jim
- Venue
- Briefings in Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2021-04-08
- Volume
- 22
- Issue
- 5
- ISSN
- 1467-5463 (ISSN-L)
- 1477-4054
- PMC
- PMC8425421
- OpenAlex
- W3151623506
- Cited by
- 60
- References
- 51
- Citations by year
- 2026: 9 · 2025: 21 · 2024: 12 · 2023: 6 · 2022: 10 · 2021: 2
- FWCI
- 2.65
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Briefings in Bioinformatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Ghent University Academic Bibliography (Ghent University) (submittedVersion, cc-by)
- Funders
- Special Research Fund (BOF24j2016001002)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Chromatin Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Keywords
- explainable AI, transformer models, attention heads, transcription factor binding sites, prokaryotic genome annotation, consensus sequence, transcription initiation, functional genomics
- MeSH terms
- Deep Learning, Base Sequence, Binding Sites, DNA, Bacterial, Escherichia coli, Promoter Regions, Genetic, Transcription Factors, Genome, Bacterial, Genomics, Transcription Initiation Site
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
