Novel Transformer Networks for Improved Sequence Labeling in genomics
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
In genomics, a wide range of machine learning methodologies have been investigated to annotate biological sequences for positions of interest such as transcription start sites, translation initiation sites, methylation sites, splice sites and promoter start sites. In recent years, this area has been dominated by convolutional neural networks, which typically outperform previously-designed methods as a result of automated scanning for influential sequence motifs. However, those architectures do not allow for the efficient processing of the full genomic sequence. As an improvement, we introduce transformer architectures for whole genome sequence labeling tasks. We show that these architectures, recently introduced for natural language processing, are better suited for processing and annotating long DNA sequences. We apply existing networks and introduce an optimized method for the calculation of attention from input nucleotides. To demonstrate this, we evaluate our architecture on several sequence labeling tasks, and find it to achieve state-of-the-art performances when comparing it to specialized models for the annotation of transcription start sites, translation initiation sites and 4mC methylation in E. coli.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 2×
“(7) defined the transformer-XL, an improvement of the transformer unit for tasks constituting long sequences by introducing a recurrent mechanism that further extends the context of the predictive network.”
BackgroundPredicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
Alipanahi, B., Delong, A., Weirauch, M. T., & Frey, B. J. (2015) · Nature Biotechnology · doi:10.1038/nbt.3300 · Cited by 2,873
Cunningham, F., Achuthan, P., Akanni, W., Allen, J., Amode, M. R., Armean, I. M., Bennett, R., Bhai, J., Billis, K., Boddu, S., et al. (2019) · Nucleic Acids Research · doi:10.1093/nar/gky1113 · Cited by 925 · cited 2×
“The genome was labeled using the RegulonDB [36] database for TSSs, Ensembl [37] for TISs and MethSMRT [38] for the 4mC-methylations.”
MethodInfluentialDeepCpG: accurate prediction of single-cell DNA methylation states using deep learning
Angermueller, C., Lee, H. J., Reik, W., & Stegle, O. (2017) · Genome Biology · doi:10.1186/s13059-017-1189-z · Cited by 536 · cited 2×
“Recurrent neural network architectures, featuring recurrently updated memory cells, have been successfully applied alongside convolutional layers to improve the detection of methylation states [2] and TISs [7] using experimental data.”
BackgroundMethod
Cited by
Ji, Y., Zhou, Z., Liu, H., & Davuluri, R. V. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab083 · Cited by 1,303
Attention is all you need: utilizing attention in AI-enabled drug discovery
Zhang, Y., Liu, C., Liu, M., Liu, T., Lin, H., Huang, C.-B., & Ning, L. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad467 · Cited by 377
Machine learning for perturbational single-cell omics
Ji, Y., Lotfollahi, M., Wolf, F. A., & Theis, F. J. (2021) · Cell Systems · doi:10.1016/j.cels.2021.05.016 · Cited by 137
Explainability in transformer models for functional genomics
Clauwaert, J., Menschaert, G., & Waegeman, W. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab060 · Cited by 60
Self-citationCpG Transformer for imputation of single-cell methylomes
De Waele, G., Clauwaert, J., Menschaert, G., & Waegeman, W. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btab746 · Cited by 37
Self-citation
Metadata
- Authors
- Clauwaert, J., & Waegeman, W.
- Venue
- IEEE/ACM Transactions on Computational Biology and Bioinformatics
- Published
- 2022-01-01
- Volume
- 19
- Issue
- 1
- Pages
- 97-106
- ISSN
- 1545-5963 (ISSN-L)
- 1557-9964
- 2374-0043
- OpenAlex
- W3096508121
- Cited by
- 47
- References
- 4
- FWCI
- 1.52
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Novel Transformer Networks for Improved Sequence Labeling in genomics. https://app.cochat.ai/papers/10.1109/tcbb.2020.3035021
- The paper
- Clauwaert, J., & Waegeman, W. (2022). Novel Transformer Networks for Improved Sequence Labeling in genomics. IEEE/ACM Transactions on Computational Biology and Bioinformatics. https://doi.org/10.1109/tcbb.2020.3035021
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.