A Comprehensive Evaluation of Self Attention for Detecting Regulatory Feature Interactions
Open accessPreprint — published as 10.1093/nargab/lqaf209 · Paper page
The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
The successful use of deep learning in computational biology depends on the ability to extract meaningful biological information from the trained models. Recent work has demonstrated that the attention maps generated by self-attention layers can be interpreted to predict cooperativity between binding of transcription factors, a key feature of gene regulatory networks. We extend this earlier work and demonstrate that the addition of an entropy term yields sparser attention maps that are easier to interpret and provide higher precision interpretations. Furthermore, we performed a comprehensive evaluation of the relative performance of different flavors of attention-based transcription factor cooperativity discovery, and compared methods that use raw attention scores to the use of attribution over the attention scores. Our findings demonstrate the benefit of the entropy-enhanced attention models and provide additional insights that would enable practitioners to make effective use of this valuable tool for biological discovery.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
Mann, H. B., & Whitney, D. R. (1947) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730491 · Cited by 14,320
Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
Alipanahi, B., Delong, A., Weirauch, M. T., & Frey, B. J. (2015) · Nature Biotechnology · doi:10.1038/nbt.3300 · Cited by 2,873
Opportunities and challenges in long-read sequencing data analysis
Amarasinghe, S. L., Su, S., Dong, X., Zappia, L., Ritchie, M. E., & Gouil, Q. (2020) · Genome Biology · doi:10.1186/s13059-020-1935-5 · Cited by 2,530
Huang, D. W., Sherman, B. T., Tan, Q., Collins, J. R., Alvord, W. G., Roayaei, J., Stephens, R., Baseler, M. W., Lane, H. C., & Lempicki, R. A. (2007) · Genome Biology · doi:10.1186/gb-2007-8-9-r183 · Cited by 2,479
Predicting effects of noncoding variants with deep learning–based sequence model
Zhou, J., & Troyanskaya, O. G. (2015) · Nature Methods · doi:10.1038/nmeth.3547 · Cited by 2,338
Determination and Inference of Eukaryotic Transcription Factor Sequence Specificity
Weirauch, M. T., Yang, A., Albu, M., Cote, A. G., Montenegro-Montero, A., Drewe, P., Najafabadi, H. S., Lambert, S. A., Mann, I., Cook, K., et al. (2014) · Cell · doi:10.1016/j.cell.2014.08.009 · Cited by 1,785
Cistrome and Epicistrome Features Shape the Regulatory DNA Landscape
O’Malley, R. C., Huang, S.-S. C., Song, L., Lewsey, M. G., Bartlett, A., Nery, J. R., Galli, M., Gallavotti, A., & Ecker, J. R. (2016) · Cell · doi:10.1016/j.cell.2016.04.038 · Cited by 1,608
Effective gene expression prediction from sequence by integrating long-range interactions
Avsec, Ž., Agarwal, V., Visentin, D., Ledsam, J. R., Grabska-Barwinska, A., Taylor, K. R., Assael, Y., Jumper, J., Kohli, P., & Kelley, D. R. (2021) · Nature Methods · doi:10.1038/s41592-021-01252-x · Cited by 1,326
Kelley, D. R., Snoek, J., & Rinn, J. L. (2016) · Genome Research · doi:10.1101/gr.200535.115 · Cited by 1,048
Purhonen, J., Banerjee, R., McDonald, A. E., Fellman, V., & Kallijärvi, J. (2020) · Nucleic Acids Research · doi:10.1093/nar/gkaa516 · Cited by 657
HIPPIE v2.0: enhancing meaningfulness and reliability of protein–protein interaction networks
Alanis-Lobato, G., Andrade-Navarro, M. A., & Schaefer, M. H. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw985 · Cited by 531
Identification of regulatory regions which confer muscle-specific gene expression
Wasserman, W. W., & Fickett, J. W. (1998) · Journal of Molecular Biology · doi:10.1006/jmbi.1998.1700 · Cited by 399
Predicting protein–protein interactions through sequence-based deep learning
Hashemifar, S., Neyshabur, B., Khan, A. A., & Xu, J. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty573 · Cited by 367
DeepChrome: deep-learning for predicting gene expression from histone modifications
Singh, R., Lanchantin, J., Robins, G., & Qi, Y. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw427 · Cited by 305
A sequence-based global map of regulatory activity for deciphering human genetics
Chen, K. M., Wong, A. K., Troyanskaya, O. G., & Zhou, J. (2022) · Nature Genetics · doi:10.1038/s41588-022-01102-2 · Cited by 269
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
Hao, Y., Dong, L., Wei, F., & Xu, K. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17533 · Cited by 189
Ibarra, I. L., Hollmann, N. M., Klaus, B., Augsten, S., Velten, B., Hennig, J., & Zaugg, J. B. (2020) · Nature Communications · doi:10.1038/s41467-019-13888-7 · Cited by 97
Discovering epistatic feature interactions from neural network models of regulatory DNA sequences
Greenside, P., Shimko, T., Fordyce, P., & Kundaje, A. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty575 · Cited by 91
Sudarsanam, P., Pilpel, Y., & Church, G. M. (2002) · Genome Research · doi:10.1101/gr.301202 · Cited by 88
Chromatin interaction–aware gene regulatory modeling with graph attention networks
Karbalayghareh, A., Sahin, M., & Leslie, C. S. (2022) · Genome Research · doi:10.1101/gr.275870.121 · Cited by 71
Zhang, Z., Feng, F., Qiu, Y., & Liu, J. (2023) · Nucleic Acids Research · doi:10.1093/nar/gkad436 · Cited by 42
A self-attention model for inferring cooperativity between regulatory features
Ullah, F., & Ben‐Hur, A. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab349 · Cited by 42
Self-citationAn interpretable block-attention network for identifying regulatory feature interactions
Prakash, A., & Banerjee, M. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad250 · Cited by 4
Characterizing collaborative transcription regulation with a graph-based deep learning approach
Zhang, Z., Feng, F., Yao, Y., & Liu, J. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010162 · Cited by 3
TIANA: transcription factors cooperativity inference analysis with neural attention
Li, R. Z., Han, C. Z., & Glass, C. K. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05852-0 · Cited by 1
Research works printed but not linked: 5
- Mukund Sundararajan , Ankur Taly , and Qiqi Yan . Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR, 2017.
- Satyanarayan Rao , Kami Ahmad , and Srinivas Ramachandran . Cooperative binding of transcription factors is a hallmark of active enhancers. bioRxiv, pages 2020–08, 2020.
- Avanti Shrikumar , Peyton Greenside , and Anshul Kundaje . Learning important features through propagating activation differences. In International conference on machine learning, pages 3145–3153. PMLR, 2017.
- 2017, Proceedings of the 34th International Conference on Machine Learning, 70, 3319
- Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , et al. PyTorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Metadata
- Authors
- Jabeen, S., & Ben-Hur, A.
- Affiliations
- Colorado State University (US)
- Corresponding author
- Jabeen, Saira
- ORCID
- Ben-Hur, Asa
- Venue
- NAR Genomics and Bioinformatics
- Publisher
- openRxiv
- Published
- 2024-08-25
- Volume
- 8
- Issue
- 1
- Pages
- lqaf209-lqaf209
- ISSN
- 2631-9268 (ISSN-L)
- PMC
- PMC12770979
- OpenAlex
- W7118968264
- Cited by
- 1
- References
- 26
- Citations by year
- 2025: 1
- FWCI
- 0.30
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- NAR Genomics and Bioinformatics (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Funders
- United States Department of Energy (DE-SC0024459)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Chromatin Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene Regulatory Network Analysis
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Keywords
- self-attention, attention maps, transcription factor cooperativity, gene regulatory networks, interpretability, binding site interactions
- MeSH terms
- Deep Learning, Algorithms, Humans, Transcription Factors, Entropy, Computational Biology, Gene Regulatory Networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.