CoChat

A Comprehensive Evaluation of Self Attention for Detecting Regulatory Feature Interactions

Open access
Jabeen, Saira, Ben-Hur, Asa · 2024 · NAR Genomics and Bioinformatics

Preprint — published as 10.1093/nargab/lqaf209 · Paper page

The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.

Abstract

The successful use of deep learning in computational biology depends on the ability to extract meaningful biological information from the trained models. Recent work has demonstrated that the attention maps generated by self-attention layers can be interpreted to predict cooperativity between binding of transcription factors, a key feature of gene regulatory networks. We extend this earlier work and demonstrate that the addition of an entropy term yields sparser attention maps that are easier to interpret and provide higher precision interpretations. Furthermore, we performed a comprehensive evaluation of the relative performance of different flavors of attention-based transcription factor cooperativity discovery, and compared methods that use raw attention scores to the use of attribution over the attention scores. Our findings demonstrate the benefit of the entropy-enhanced attention models and provide additional insights that would enable practitioners to make effective use of this valuable tool for biological discovery.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2024.08.23.609428

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. The SATORI framework. The deep learning architecture receives as input a one-hot encoded DNA sequence, which is then processed through a 1D-CNN layer, a self-attention layer, optional supplementary layers, and a classification layer. SATORI combines the attention heads and compares the attention values between pairs of filters in positive examples to a background set. It then reports pairs of filters that demonstrate statistical significance.
Figure 1. The SATORI framework. The deep learning architecture receives as input a one-hot encoded DNA sequence, which is then processed through a 1D-CNN layer, a self-attention layer, optional supplementary layers, and a classification layer. SATORI combines the attention heads and compares the attention values between pairs of filters in positive examples to a background set. It then reports pairs of filters that demonstrate statistical significance.CC BY · PubMed Central
Figure 2. Additional layers included in the SATORI-deep architecture after the self-attention layer.
Figure 2. Additional layers included in the SATORI-deep architecture after the self-attention layer.CC BY · PubMed Central
Figure 3. Comparative performance of SATORI on simulated datasets of varying levels of complexity using the Basic and Deep architectures. We show: classification AUC (a) and interpretation accuracy: F1-score (b), precision (c), and recall (d). Results are shown for four models: Basic and Deep trained with and without the entropy loss; results are reported for datasets of increasing levels of complexity, comprising 40, 60, or 80 implanted TF motifs; performance numbers are averages over three instances of each dataset. Results are reported for the version of SATORI that uses raw attention values (Attn-Raw).
Figure 3. Comparative performance of SATORI on simulated datasets of varying levels of complexity using the Basic and Deep architectures. We show: classification AUC (a) and interpretation accuracy: F1-score (b), precision (c), and recall (d). Results are shown for four models: Basic and Deep trained with and without the entropy loss; results are reported for datasets of increasing levels of complexity, comprising 40, 60, or 80 implanted TF motifs; performance numbers are averages over three instances of each dataset. Results are reported for the version of SATORI that uses raw attention values (Attn-Raw).CC BY · PubMed Central
Figure 4. The effect of entropy on attention weights for a basic model trained without (left) and with (right) the entropy loss, using a regularization parameter of 0.01 for Data-80. (a) The distribution of attention weights across accurately predicted test sequences. (b) Heatmaps of attention weights maximized over all attention heads for a true positive sequence. The X-axis and Y-axis in the heatmaps are the positions in the simulated DNA sequence after max pooling; arrows indicate the locations of embedded motifs in the original sequence, adjusted for the pooling window.
Figure 4. The effect of entropy on attention weights for a basic model trained without (left) and with (right) the entropy loss, using a regularization parameter of 0.01 for Data-80. (a) The distribution of attention weights across accurately predicted test sequences. (b) Heatmaps of attention weights maximized over all attention heads for a true positive sequence. The X-axis and Y-axis in the heatmaps are the positions in the simulated DNA sequence after max pooling; arrows indicate the locations of embedded motifs in the original sequence, adjusted for the pooling window.CC BY · PubMed Central
Figure 5. Interpretation accuracy. We compare performance of SATORI using raw attention scores (Attn-Raw), attention attribution (Attn-Attr), and DFIM based on precision (a), recall (b), and F1-score (c). All methods use the same basic architecture; datasets comprise 40, 60, or 80 implanted TF motifs; performance numbers are averages over three instances of each dataset.
Figure 5. Interpretation accuracy. We compare performance of SATORI using raw attention scores (Attn-Raw), attention attribution (Attn-Attr), and DFIM based on precision (a), recall (b), and F1-score (c). All methods use the same basic architecture; datasets comprise 40, 60, or 80 implanted TF motifs; performance numbers are averages over three instances of each dataset.CC BY · PubMed Central
Figure 6. Number of interactions inferred by SATORI basic and deep architectures with and without entropy loss, compared against DFIM on the human promoter dataset (a), accessible chromatin in Arabidopsis (b), and Arabidopsis enhancers (c).
Figure 6. Number of interactions inferred by SATORI basic and deep architectures with and without entropy loss, compared against DFIM on the human promoter dataset (a), accessible chromatin in Arabidopsis (b), and Arabidopsis enhancers (c).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 26 references

Research works printed but not linked: 5

  1. Mukund Sundararajan , Ankur Taly , and Qiqi Yan . Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR, 2017.
  2. Satyanarayan Rao , Kami Ahmad , and Srinivas Ramachandran . Cooperative binding of transcription factors is a hallmark of active enhancers. bioRxiv, pages 2020–08, 2020.
  3. Avanti Shrikumar , Peyton Greenside , and Anshul Kundaje . Learning important features through propagating activation differences. In International conference on machine learning, pages 3145–3153. PMLR, 2017.
  4. 2017, Proceedings of the 34th International Conference on Machine Learning, 70, 3319
  5. Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , et al. PyTorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.

Metadata

Authors
Jabeen, S., & Ben-Hur, A.
Affiliations
Colorado State University (US)
Corresponding author
Jabeen, Saira
Venue
NAR Genomics and Bioinformatics
Publisher
openRxiv
Published
2024-08-25
Volume
8
Issue
1
Pages
lqaf209-lqaf209
ISSN
2631-9268 (ISSN-L)
OpenAlex
W7118968264
Cited by
1
References
26
Citations by year
2025: 1
FWCI
0.30
Type
journal article
Language
en
Open access
Locations
  • NAR Genomics and Bioinformatics (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
Funders
United States Department of Energy (DE-SC0024459)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Chromatin Dynamics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
Keywords
self-attention, attention maps, transcription factor cooperativity, gene regulatory networks, interpretability, binding site interactions
MeSH terms
Deep Learning, Algorithms, Humans, Transcription Factors, Entropy, Computational Biology, Gene Regulatory Networks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Comprehensive Evaluation of Self Attention for Detecting Regulatory Feature Interactions. https://app.cochat.ai/papers/10.1101/2024.08.23.609428
The paper
Jabeen, S., & Ben-Hur, A. (2024). A Comprehensive Evaluation of Self Attention for Detecting Regulatory Feature Interactions. NAR Genomics and Bioinformatics. https://doi.org/10.1101/2024.08.23.609428

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.