The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Effect of Tokenization on Transformers for Biological Sequences
Open accessPreprint — published as 10.1093/bioinformatics/btae196 · Paper page
Abstract
Deep learning models are transforming biological research. Many bioinformatics and comparative genomics algorithms analyze genomic data, either DNA or protein sequences. Examples include sequence alignments, phylogenetic tree inference and automatic classification of protein functions. Among these deep learning algorithms, models for processing natural languages, developed in the natural language processing (NLP) community, were recently applied to biological sequences. However, biological sequences are different than natural languages, such as English, and French, in which segmentation of the text to separate words is relatively straightforward. Moreover, biological sequences are characterized by extremely long sentences, which hamper their processing by current machine-learning models, notably the transformer architecture. In NLP, one of the first processing steps is to transform the raw text to a list of tokens. Deep-learning applications to biological sequence data mostly segment proteins and DNA to single characters. In this work, we study the effect of alternative tokenization algorithms on eight different tasks in biology, from predicting the function of proteins and their stability, through nucleotide sequence alignment, to classifying proteins to specific families. We demonstrate that applying alternative tokenization algorithms can increase accuracy and at the same time, substantially reduce the input length compared to the trivial tokenizer in which each character is a token. Furthermore, applying these tokenization algorithms allows interpreting trained models, taking into account dependencies among positions. Finally, we trained these tokenizers on a large dataset of protein sequences containing more than 400 billion amino acids, which resulted in over a three-fold decrease in the number of tokens. We then tested these tokenizers trained on large-scale data on the above specific tasks and showed that for some tasks it is highly beneficial to train database-specific tokenizers. Our study suggests that tokenizers are likely to be a critical component in future deep-network analysis of biological sequence data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Flagged first, then by citations
Research works printed but not linked: 41
- 10.1006/jmbi.1990.9999 · DOI · Paper page
- 10.1093/nar/gkz1064 · DOI
- 2022, “ProteinBERT: A Universal Deep-Learning Model of Protein Sequence and Function, Bioinformatics, 38, 2102, 10.1093/bioinformatics/btac020 · DOI · Paper page
- 10.1016/j.ejphar.2009.10.015 · DOI · Paper page
- Devlin, Jacob , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2019. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–86. Minneapolis, Minnesota: Association for Computational Linguistics.
- Dotan, Edo , Yonatan Belinkov , Oren Avram , Elya Wygoda , Noa Ecker , Michael Alburquerque , Omri Keren , Gil Loewenthal , and Tal Pupko . 2023. “Multiple Sequence Alignment as a Sequence-to-Sequence Learning Problem.” In The Eleventh International Conference on Learning Representations (ICLR 2023).
- 10.1093/bioinformatics/btx780 · DOI
- 2021, “DNABERT: Pre-Trained Bidirectional Encoder Representations from Transformers Model for DNA-Language in Genome, Bioinformatics, 37, 2112, 10.1093/bioinformatics/btab083 · DOI · Paper page
- Kokhlikyan, Narine , Vivek Miglani , Miguel Martin , Edward Wang , Bilal Alsallakh , Jonathan Reynolds , Alexander Melnikov , et al. 2020. “Captum: A Unified and Generic Model Interpretability Library for PyTorch.” arXiv. https://doi.org/10.48550/arXiv.2009.07896. · DOI · Paper page
- 2018, “DeepGO: Predicting Protein Functions from Sequence and Interactions Using a Deep Ontology-Aware Classifier, Bioinformatics, 34, 660, 10.1093/bioinformatics/btx624 · DOI · Paper page
- 10.1038/nature14539 · DOI · Paper page
- Lin, Tianyang , Yuxin Wang , Xiangyang Liu , and Xipeng Qiu . 2021. “A Survey of Transformers.” ArXiv:2106.04554. http://arxiv.org/abs/2106.04554. · arXiv
- List, Johann-Mattis , Jananan Sylvestre Pathmanathan , Philippe Lopez , and Eric Bapteste . 2016. “Unity and Disunity in Evolutionary Sciences: Process-Based Analogies Open Common Research Avenues for Biology and Linguistics.” Biology Direct 11. · DOI
- 10.1093/molbev/msab266 · DOI
- 10.1093/nar/gkr1044 · DOI
- 1975, “Comparison of the Predicted and Observed Secondary Structure of T4 Phage Lysozyme, Biochimica et Biophysica Acta (BBA) -Protein Structure, 405, 442, 10.1016/0005-2795(75)90109-9 · DOI · Paper page
- 2022, “Deciphering Microbial Gene Function Using Natural Language Processing, Nature Communications, 13, 5731, 10.1038/s41467-022-33397-4 · DOI
- 10.1093/NAR/GKAA913 · DOI · Paper page
- 10.1128/microbiolspec.VMBF-0004–2015 · DOI
- 2022, “The Complete Sequence of a Human Genome, Science (New York, N.Y, 376, 44, 10.1126/science.abj6987 · DOI · Paper page
- Ott, Myle , Sergey Edunov , Alexei Baevski , Angela Fan , Sam Gross , Nathan Ng , David Grangier , and Michael Auli . 2019. “Fairseq: A Fast, Extensible Toolkit for Sequence Modeling.” arXiv. http://arxiv.org/abs/1904.01038. · DOI
- 10.1093/molbev/msq066 · DOI
- 2019, “Evaluating Protein Transfer Learning with TAPE, Advances in Neural Information Processing Systems, 32, 9689
- Rao, Roshan M. , Jason Liu , Robert Verkuil , Joshua Meier , John Canny , Pieter Abbeel , Tom Sercu , and Alexander Rives . 2021. “MSA Transformer.” In Proceedings of the 38th International Conference on Machine Learning, 8844–56. PMLR. · DOI · Paper page
- 2008, “Comparative Genomics and Molecular Dynamics of DNA Repeats in Eukaryotes, Microbiology and Molecular Biology Reviews: MMBR, 72, 686, 10.1128/MMBR.00011-08 · DOI
- 10.1126/science.aan0693 · DOI · Paper page
- 2023, “Automated Large-Scale Prediction of Exudative AMD Progression Using Machine-Read OCT Biomarkers, PLOS Digital Health, 2, e0000106, 10.1371/journal.pdig.0000106 · DOI
- 10.1038/nature17995 · DOI · Paper page
- Schuster, Mike , and Kaisuke Nakajima . 2012. “Japanese and Korean Voice Search.” In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5149–52. · DOI · Paper page
- Sennrich, Rico , Barry Haddow , and Alexandra Birch . 2016. “Neural Machine Translation of Rare Words with Subword Units.” arXiv. https://doi.org/10.48550/arXiv.1508.07909. · DOI · Paper page
- 10.1038/nbt.3988 · DOI · Paper page
- 2018, “Clustering Huge Protein Sequence Sets in Linear Time, Nature Communications, 9, 2542, 10.1038/s41467-018-04964-5 · DOI · Paper page
- Sundararajan, Mukund , Ankur Taly , and Qiqi Yan . 2017. “Axiomatic Attribution for Deep Networks.” arXiv. http://arxiv.org/abs/1703.01365. · arXiv
- 10.1038/514550a · DOI
- Vaswani, Ashish , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Lukasz Kaiser , and Illia Polosukhin . 2017. “Attention Is All You Need.” In 31st Conference on Neural Information Processing Systems (NIPS 2017).
- Voulodimos, Athanasios , Nikolaos Doulamis , Anastasios Doulamis , and Eftychios Protopapadakis . 2018. “Deep Learning for Computer Vision: A Brief Review.” Computational Intelligence and Neuroscience 2018: 7068349. · DOI · Paper page
- Wagner, Naama , Michael Alburquerque , Noa Ecker , Edo Dotan , Ben Zerah , Michelle Mendonca Pena , Neha Potnis , and Tal Pupko . 2022. “Natural Language Processing Approach to Model the Secretion Signal of Type III Effectors.” Frontiers in Plant Science 13. · DOI
- Wolf, Thomas , Lysandre Debut , Victor Sanh , Julien Chaumond , Clement Delangue , Anthony Moi , Pierric Cistac , et al. 2020. “Transformers: State-of-the-Art Natural Language Processing.” In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 38–45. · DOI · Paper page
- 10.1186/s13059-019-1891-0 · DOI
- 10.1073/pnas.1814684116 · DOI
- Zhou, Zhihan , Yanrong Ji , Weijian Li , Pratik Dutta , Ramana Davuluri , and Han Liu . 2023. “DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome.” arXiv. https://doi.org/10.48550/arXiv.2306.15006. · DOI · Paper page
Metadata
- Authors
- Dotan, E., Jaschek, G., Pupko, T., & Belinkov, Y.
- Affiliations
- Tel Aviv University (IL)
- Yale University (US)
- Technion – Israel Institute of Technology (IL)
- Corresponding author
- Pupko, Tal
- Belinkov, Yonatan
- Venue
- Bioinformatics
- Publisher
- openRxiv
- Published
- 2023-08-17
- Volume
- 40
- Issue
- 4
- ISSN
- 1367-4803 (ISSN-L)
- PMC
- PMC11055402
- OpenAlex
- W4394769874
- Cited by
- 50
- References
- 1
- Citations by year
- 2026: 19 · 2025: 22 · 2024: 9
- FWCI
- 6.78
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Bioinformatics (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Israel Science Foundation (2818/21)
- Tel Aviv University
- Azrieli Foundation
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Fractal and DNA sequence analysis
- Keywords
- tokenization, transformer models, biological sequences, protein function prediction, protein stability prediction, DNA sequence alignment, protein family classification, model interpretability, protein sequences
- MeSH terms
- Deep Learning, Algorithms, Natural Language Processing, Proteins, Sequence Alignment, Computational Biology, Sequence Analysis, Protein
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
