CoChat
Sign Up
Cited by 176Machine Learning in BioinformaticsGenomics and Phylogenetic StudiesComputational Drug Discovery Methods

Applications of transformer-based language models in bioinformatics: a survey

Open access
Zhang, Shuang, Fan, Rui, Liu, Yuti, Chen, Shuang, Liu, Qiao, Zeng, Wanwen·2023·Bioinformatics Advances
DOI 10.1093/bioadv/vbad001PMC PMC9950855
Open this paper in CoChat — free account

Abstract

Summary: The transformer-based language models, including vanilla transformer, BERT and GPT-3, have achieved revolutionary breakthroughs in the field of natural language processing (NLP). Since there are inherent similarities between various biological sequences and natural languages, the remarkable interpretability and adaptability of these models have prompted a new wave of their application in bioinformatics research. To provide a timely and comprehensive review, we introduce key developments of transformer-based language models by describing the detailed structure of transformers and summarize their contribution to a wide range of bioinformatics research from basic sequence analysis to drug discovery. While transformer-based applications in bioinformatics are diverse and multifaceted, we identify and discuss the common challenges, including heterogeneity of training data, computational expense and model interpretability, and opportunities in the context of bioinformatics research. We hope that the broader community of NLP researchers, bioinformaticians and biologists will be brought together to foster future research and development in transformer-based language models, and inspire novel bioinformatics applications that are unattainable by traditional methods. Supplementary information: Supplementary data are available at Bioinformatics Advances online.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/bioadv/vbad001

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Transformers are well-suited to learn sequence data of biology, including DNA, RNA, and protein sequence learning, genomic regulatory prediction, protein function annotation, protein design, and further adaptation into medical image and gene expression analysis.

    “It makes it well-suited to learn the sequence data of biology [76], including the DNA [6], RNA [77], and protein [7] sequence learning, genomic regulatory prediction [78], protein function annotation [79], protein design [80], and further adaptation into medical image [81] and gene expression analysis [13].”

    From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [76]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 142

  1. Adel H. et al (2018) Overview of character-based models for natural language processing. In: Gelbukh A. (ed.) Computational Linguistics and Intelligent Text Processing, Lecture Notes in Computer Science. Springer International Publishing, Cham, pp. 3– 16.
  2. Ashburner M. et al (2000) Gene ontology: tool for the unification of biology. Nat. Genet., 25, 25– 29. 10802651 PMC3037419
  3. Avsec Ž. et al (2021) Effective gene expression prediction from sequence by integrating long-range interactions. Nat. Methods, 18, 1196– 1203. 34608324 PMC8490152
  4. Ba J.L. et al (2016) Layer normalization. arXiv, arXiv:1607.06450v1, https://arxiv.org/abs/1607.06450v1, preprint: not peer reviewed. · arXiv
  5. Bagal V. et al (2022) MolGPT: molecular generation using a transformer-decoder model. J. Chem. Inf. Model., 62, 2064– 2076. 34694798
  6. Bahdanau D. et al (2016) Neural machine translation by jointly learning to align and translate. arXiv, arXiv:1409.0473v7, https://arxiv.org/abs/1409.0473v7, preprint: not peer reviewed. · arXiv
  7. Baid G. et al (2022) DeepConsensus improves the accuracy of sequences with a gap-aware sequence transformer. Nat. Biotechnol., 1– 7. 36050551
  8. Beltagy I. et al (2020) Longformer: the long-document transformer. arXiv, arXiv:2004.05150v2, https://arxiv.org/abs/2004.05150v2, preprint: not peer reviewed. · arXiv
  9. Bengio Y. et al (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw., 5, 157– 166. 18267787
  10. Blacoe W., Lapata M. (2012) A comparison of vector-based representations for semantic composition. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, Jeju Island, Korea, pp. 546– 556.
  11. Boguski M.S., McIntosh M.W. (2003) Biomedical informatics for proteomics. Nature, 422, 233– 237. 12634797
  12. Brandes N. et al (2022) ProteinBERT: a universal deep-learning model of protein sequence and function. Bioinformatics, 38, 2102– 2110. 35020807 PMC9386727
  13. Brazma A., Vilo J. (2000) Gene expression data analysis. FEBS Lett., 480, 17– 24. 10967323
  14. Brown T. et al (2020) Language models are few-shot learners. In: Proceedings of the 34th International Conference on Neural Information Processing Systems, Vol. 33, Curran Associates Inc., Vancouver, Canada, pp. 1877– 1901.
  15. Butte A.J. (2001) Challenges in bioinformatics: infrastructure, models and analytics. Trends Biotechnol., 19, 159– 160. 11403022
  16. Calijorne Soares M.A., Parreiras F.S. (2020) A literature review on question answering techniques, paradigms and systems. J. King Saud Univ. Comput. Inf. Sci., 32, 635– 646.
  17. Cao Y., Shen Y. (2021) TALE: transformer-based protein function annotation with joint sequence-label embedding. Bioinformatics, 37, 2825– 2833. 33755048 PMC8479653
  18. Castro-Vega L.J. et al (2015) Multi-omics analysis defines core genomic alterations in pheochromocytomas and paragangliomas. Nat. Commun., 6, 6044. 25625332 PMC4354166
  19. Charoenkwan,P. et al (2021) BERT4Bitter: a bidirectional encoder representations from transformers (BERT)-based model for improving the prediction of bitter peptides. Bioinformatics, 37, 2556– 2562. 33638635
  20. Chaudhari S. et al (2021) An attentive survey of attention models. ACM Trans. Intell. Syst. Technol, 12, 53:1– 53:32.
  21. Chen J. et al (2022a) A comprehensive comparison on cell-type composition inference for spatial transcriptomics data. Brief. Bioinform., 23, bbac245. 35753702 PMC9294426
  22. Chen J. et al (2021a) TransUNet: transformers make strong encoders for medical image segmentation. arXiv, arXiv:2102.04306v1, https://arxiv.org/abs/2102.04306v1, preprint: not peer reviewed. · arXiv
  23. Chen J. et al (2021b) ViT-V-Net: vision transformer for unsupervised volumetric medical image registration. arXiv, arXiv:2104.06468v1, https://arxiv.org/abs/2104.06468v1, preprint: not peer reviewed. · arXiv
  24. Chen K. et al (2022b) Capturing large genomic contexts for accurately predicting enhancer-promoter interactions. Brief. Bioinform., 23, bbab577. 35062021
  25. Chen Y.-C. et al (2020) UNITER: UNiversal Image-TExt Representation learning. In: Vedaldi A. et al (ed.) Computer Vision – ECCV 2020, Lecture Notes in Computer Science. Springer International Publishing, Cham, pp. 104– 120.
  26. Chithrananda S. et al (2020) ChemBERTa: large-scale self-supervised pretraining for molecular property prediction. arXiv, arXiv:2010.09885v2, https://arxiv.org/abs/2010.09885v2, preprint: not peer reviewed. · arXiv
  27. Cho K. et al (2014) Learning phrase representations using RNN encoder–decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pp. 1724– 1734.
  28. Clauwaert J. et al (2021) Explainability in transformer models for functional genomics. Brief. Bioinform., 22, bbab060. 33834200 PMC8425421
  29. Dai Z. et al (2019) Transformer-XL: attentive language models beyond a fixed-length context. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, pp. 2978– 2988.
  30. Devlin J. et al (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, pp. 4171– 4186.
  31. Dosovitskiy A. et al (2021) An image is worth 16x16 words: transformers for image recognition at scale. arXiv, arXiv:2010.11929v2, https://arxiv.org/abs/2010.11929v2, preprint: not peer reviewed. · arXiv
  32. Faridani O.R. et al (2016) Single-cell sequencing of the small-RNA transcriptome. Nat. Biotechnol., 34, 1264– 1266. 27798564
  33. Feinberg E.N. et al (2018) PotentialNet for molecular property prediction. ACS Cent. Sci., 4, 1520– 1530. 30555904 PMC6276035
  34. Ferruz N. et al (2022) ProtGPT2 is a deep unsupervised language model for protein design. Nat. Commun., 13, 4348. 35896542 PMC9329459
  35. Feuk L. et al (2006) Structural variation in the human genome. Nat. Rev. Genet., 7, 85– 97. 16418744
  36. Findlay J.B.C. et al (1995) Protein sequence analysis, storage and retrieval. In: Atassi M.Z., Appella E. (ed.) Methods in Protein Structure Analysis. Springer US, Boston, MA, pp. 465– 472.
  37. Gilmer J. et al (2020) Message passing neural networks. In: Schütt K.T. et al (ed.) Machine Learning Meets Quantum Physics, Lecture Notes in Physics. Springer International Publishing, Cham, pp. 199– 214.
  38. Goeman J.J., Bühlmann P. (2007) Analyzing gene expression data in terms of gene sets: methodological issues. Bioinformatics, 23, 980– 987. 17303618
  39. Grechishnikova D. (2021) Transformer neural network for protein-specific de novo drug generation as a machine translation problem. Sci. Rep., 11, 321. 33432013 PMC7801439
  40. Han,K. et al. (2023) A survey on vision transformer. IEEE Trans. Pattern Anal. Mach. Intell., 45, 87– 110. 35180075
  41. Han,X. and Kwoh,C.K. (2019) Natural language processing approaches in bioinformatics. In: Ranganathan S. (ed.) Encyclopedia of Bioinformatics and Computational Biology. Academic Press, Oxford, pp. 561– 574.
  42. Hastie T. et al (2009) Overview of supervised learning. In: Hastie T. et al (ed.) The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics. Springer, New York, NY, pp. 9– 41.
  43. Hochreiter S., Schmidhuber J. (1997) Long short-term memory. Neural Comput., 9, 1735– 1780. 9377276
  44. Howard J., Ruder S. (2018) Universal language model fine-tuning for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, pp. 328– 339.
  45. Huang K. et al (2021) MolTrans: molecular interaction transformer for drug-target interaction prediction. Bioinformatics, 37, 830– 836. 33070179 PMC8098026
  46. Iuchi H. et al (2021) Representation learning applications in biological sequence analysis. Comput. Struct. Biotechnol. J., 19, 3198– 3208. 34141139 PMC8190442
  47. Ji,Y. et al. (2021) DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformatics, 37, 2112– 2120. PMC11025658 33538820
  48. Jiang L. et al (2022) DeepTTA: a transformer-based model for predicting cancer drug response. Brief. Bioinform., 23, bbac100. 35348595
  49. Jurenaite N. et al (2022) SetQuence & SetOmic: deep set transformer-based representations of cancer multi-omics. In: 2022 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), Ottawa, ON, Canada, pp. 1– 9.
  50. Kaczmarek E. et al (2021) Multi-omic graph transformers for cancer classification and interpretation. In: Biocomputing 2022. World Scientific, Kohala Coast, Hawaii, USA, pp. 373– 384. 34890164
  51. Kalakoti Y. et al (2022) TransDTI: transformer-based language models for estimating DTIs and building a drug recommendation workflow. ACS Omega, 7, 2706– 2717. 35097268 PMC8792915
  52. Kalyan K.S. et al (2021a) AMMU: a survey of transformer-based biomedical pretrained language models. J. Biomed. Inform., 126, 103982. 34974190
  53. Kalyan K.S. et al (2021b) AMMUS: a survey of transformer-based pretrained models in natural language processing. arXiv, arXiv:2108.05542v2, https://arxiv.org/abs/2108.05542v2, preprint: not peer reviewed. · arXiv
  54. Kang M. et al (2022) A roadmap for multi-omics data integration using deep learning. Brief. Bioinform., 23, bbab454. 34791014 PMC8769688
  55. Karametsi K. et al (2014) Identification of bitter peptides in aged cheddar cheese. J. Agric. Food Chem., 62, 8034– 8041. 25075877
  56. Karlin S., Ghandour G. (1985) Comparative statistics for DNA and protein sequences: single sequence analysis. Proc. Natl. Acad. Sci. USA, 82, 5800– 5804. 2994049 PMC390640
  57. Katharopoulos A. et al (2020) Transformers are RNNs: fast autoregressive transformers with linear attention. In: Proceedings of the 37th International Conference on Machine Learning, PMLR, Online, pp. 5156– 5165.
  58. Ke G. et al (2021) Rethinking positional encoding in language pre-training. arXiv, arXiv:2006.15595v4, https://arxiv.org/abs/2006.15595v4, preprint: not peer reviewed. · arXiv
  59. Khan A., Lee B. (2021) Gene transformer: transformers for the gene expression-based classification of lung cancer subtypes. arXiv. arXiv:2108.11833v3, https://arxiv.org/abs/2108.11833v3, preprint: not peer reviewed. · arXiv
  60. Khan S. et al (2022) Transformers in vision: a survey. ACM Comput. Surv., 54, 200:1-200:41.
  61. Kitaev N. et al (2020) Reformer: the efficient transformer. arXiv, arXiv:2001.04451v2, https://arxiv.org/abs/2001.04451v2, preprint: not peer reviewed. · arXiv
  62. Kolodziejczyk A.A. et al (2015) The technology and biology of single-cell RNA sequencing. Mol. Cell., 58, 610– 620. 26000846
  63. Koonin E.V., Galperin M.Y. (2003a) Genome annotation and analysis. In: Koonin E.V., Galperin M.Y. (ed.) Sequence — Evolution — Function: Computational Approaches in Comparative Genomics. Springer US, Boston, MA, pp. 193– 226. 21089240
  64. Koonin E.V., Galperin M.Y. (2003b) Principles and methods of sequence analysis. In: Koonin E.V., Galperin M.Y. (ed.) Sequence — Evolution — Function: Computational Approaches in Comparative Genomics. Springer US, Boston, MA, pp. 111– 192. 21089240
  65. Kothari S.C., Oh H. (1993) Neural networks for pattern recognition. In: Yovits M.C. (ed.) Advances in Computers. Academic Press, San Diego, CA, USA, pp. 119– 166.
  66. Lan Z. et al (2020) ALBERT: a lite BERT for self-supervised learning of language representations. arXiv, arXiv:1909.11942v6, https://arxiv.org/abs/1909.11942v6, preprint: not peer reviewed. · arXiv
  67. Larance M., Lamond A.I. (2015) Multidimensional proteomics for cell biology. Nat. Rev. Mol. Cell Biol., 16, 269– 280. 25857810
  68. Le Q., Mikolov T. (2014) Distributed representations of sentences and documents. In: Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ICML’14. JMLR.org, Beijing, China, p. II-1188-II–1196.
  69. LeCun Y. et al (2015) Deep learning. Nature, 521, 436– 444. 26017442
  70. Lee J. et al (2020) BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36, 1234– 1240. 31501885 PMC7703786
  71. Lee K. et al (2021) A transformer architecture based on BERT and 2D convolutional neural network to identify DNA enhancers from sequence information. Brief. Bioinform., 22, bbab005. 33539511
  72. Li H. et al (2022) KPGT: knowledge-guided pre-training of graph transformer for molecular property prediction. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22. Association for Computing Machinery, New York, NY, USA, pp. 857– 867.
  73. Lin Z. et al (2017) A structured self-attentive sentence embedding. arXiv, arXiv:1703.03130v1, https://arxiv.org/abs/1703.03130v1, preprint: not peer reviewed. · arXiv
  74. Liu C. et al (2022) eQTLs play critical roles in regulating gene expression and identifying key regulators in rice. Plant Biotechnol. J., 20, 2357– 2371. 36087348 PMC9674320
  75. Liu K. et al (2019a) Chemi-Net: a molecular graph convolutional network for accurate drug property prediction. Int. J. Mol. Sci., 20, E3389. PMC6678642 31295892
  76. Liu Q. et al (2020) A survey on contextual embeddings. arXiv, arXiv:2003.07278v2, https://arxiv.org/abs/2003.07278v2, preprint: not peer reviewed. · arXiv
  77. Liu,X. et al. (2023) Self-Supervised Learning: Generative or Contrastive. IEEE Trans. Knowl. Data Eng., 35, 857– 876.
  78. Liu Y. et al (2019b) RoBERTa: a robustly optimized BERT pretraining approach. arXiv, arXiv:1907.11692v1, https://arxiv.org/abs/1907.11692v1, preprint: not peer reviewed. · arXiv
  79. Lo Y.-C. et al (2018) Machine learning in chemoinformatics and drug discovery. Drug Discov. Today., 23, 1538– 1546. 29750902 PMC6078794
  80. Lu J. et al (2019) ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., Vancouver, Canada.
  81. Ma A. et al (2021) Biological network inference from single-cell multi-omics data using heterogeneous graph transformer. bioRxiv, doi: 10.1101/2021.10.31.466658, preprint: not peer reviewed. · DOI
  82. Meng Y. et al (2021) Bidirectional representation learning from transformers using multimodal electronic health record data to predict depression. IEEE J. Biomed. Health Inform., 25, 3121– 3129. 33661740 PMC8606118
  83. Mikolov T. et al (2013a) Distributed representations of words and phrases and their compositionality. In: Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13. Curran Associates Inc., Red Hook, NY, USA, pp. 3111–3119.
  84. Mikolov T. et al (2013b) Efficient estimation of word representations in vector space. In: Proceedings of Workshop at ICLR, 2013, Scottsdale, Arizona.
  85. Milletari F. et al (2016) V-Net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), Stanford, California, USA, pp. 565– 571.
  86. Mo S. et al (2021) Multi-modal self-supervised pre-training for regulatory genome across cell types. arXiv, arXiv:2110.05231v2, https://arxiv.org/abs/2110.05231v2, preprint: not peer reviewed. · arXiv
  87. Nadkarni P.M. et al (2011) Natural language processing: an introduction. J. Am. Med. Inform. Assoc., 18, 544– 551. 21846786 PMC3168328
  88. Nenkova A., McKeown K. (2012) A survey of text summarization techniques. In: Aggarwal C.C., Zhai C. (ed.) Mining Text Data. Springer US, Boston, MA, pp. 43– 76.
  89. Noor Z. et al (2021) Mass spectrometry–based protein identification in proteomics—a review. Brief. Bioinform., 22, 1620– 1638. 32047889
  90. Ofer D. et al (2021) The language of proteins: NLP, machine learning & protein sequences. Comput. Struct. Biotechnol. J., 19, 1750– 1758. 33897979 PMC8050421
  91. Pang M. et al (2021) Leveraging information in spatial transcriptomics to predict super-resolution gene expression from histology images in tumors. bioRxiv, doi: 10.1101/2021.11.28.470212v1, preprint: not peer reviewed. · DOI
  92. Park M. et al (2022) EpiBERTope: a sequence-based pre-trained BERT model improves linear and structural epitope prediction by learning long-distance protein interactions effectively. bioRxiv, doi: 10.1101/2022.02.27.481241, preprint: not peer reviewed. · DOI
  93. Pascanu R. et al (2013) On the difficulty of training recurrent neural networks. In: Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28, ICML’13. JMLR.org, Atlanta, USA, pp. III-1310- III–1318.
  94. Petroni F. et al (2019) Language models as knowledge bases? In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, pp. 2463– 2473.
  95. Ponting C.P., Birney E. (2005) Protein sequence analysis and domain identification. In: Walker J.M. (ed.) The Proteomics Protocols Handbook. Springer Protocols Handbooks, Humana Press, Totowa, NJ, pp. 527– 541.
  96. Portela A., Esteller M. (2010) Epigenetic modifications and human disease. Nat. Biotechnol., 28, 1057– 1068. 20944598
  97. Pundge A.M. et al (2016) Question answering system, approaches and techniques: a review. Int. J. Comput. Appl. A, 141, 34– 39.
  98. Qiao,Y. et al. (2022) BERT-Kcr: prediction of lysine crotonylation sites by a transfer learning method with pre-trained BERT models. Bioinformatics, 38, 648–654. 34643684
  99. Raad J. et al (2022) miRe2e: a full end-to-end deep model based on transformers for prediction of pre-miRNAs. Bioinformatics, 38, 1191– 1197. 34875006
  100. Radford A. et al (2021) Learning transferable visual models from natural language supervision. In: Proceedings of the 38th International Conference on Machine Learning. PMLR, Virtual, pp. 8748– 8763.
  101. Rao R. et al (2020) Transformer protein language models are unsupervised structure learners. bioRxiv, doi: 10.1101/2020.12.15.422761, preprint: not peer reviewed. · DOI · Paper page
  102. Rasmy L. et al (2021) Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ Digit. Med., 4, 86. 34017034 PMC8137882
  103. Reis-Filho J.S. (2009) Next-generation sequencing. Breast Cancer Res., 11, S12. 20030863 PMC2797692
  104. Rix U., Superti-Furga G. (2009) Target profiling of small molecules by chemical proteomics. Nat. Chem. Biol., 5, 616– 624. 19690537
  105. Romero P.A., Arnold F.H. (2009) Exploring protein fitness landscapes by directed evolution. Nat. Rev. Mol. Cell Biol., 10, 866– 876. 19935669 PMC2997618
  106. Rong Y. et al (2020) Self-supervised graph transformer on large-scale molecular data. In: Proceedings of the 34th International Conference on Neural Information Processing Systems - Volume 33, NIPS’20. Curran Associates Inc., Vancouver, Canada, pp. 12559–12571.
  107. Ronneberger O. et al (2015) U-Net: convolutional networks for biomedical image segmentation. In: Navab N. et al (ed.), Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Lecture Notes in Computer Science. Springer International Publishing, Cham, pp. 234– 241.
  108. Rotter A. et al (2010) Gene expression data analysis using closed itemset mining for labeled data. OMICS, 14, 177– 186. 20210654 PMC3116449
  109. Rung J., Brazma A. (2013) Reuse of public genome-wide gene expression data. Nat. Rev. Genet., 14, 89– 99. 23269463
  110. Sarkar I.N. (2010) Biomedical informatics and translational medicine. J. Transl. Med., 8, 22. 20187952 PMC2837642
  111. Scarselli F. et al (2009) The graph neural network model. IEEE Trans. Neural Netw., 20, 61– 80. 19068426
  112. Schouten K., Frasincar F. (2016) Survey on aspect-level sentiment analysis. IEEE Trans. Knowl. Data Eng., 28, 813– 830.
  113. Schuster M., Paliwal K.K. (1997) Bidirectional recurrent neural networks. IEEE Trans. Signal Process., 45, 2673– 2681.
  114. Skansi S. (2018) Feedforward neural networks. In: Skansi S. (ed.) Introduction to Deep Learning: From Logical Calculus to Artificial Intelligence, Undergraduate Topics in Computer Science. Springer International Publishing, Cham, pp. 79– 105.
  115. Stevens R. et al (2000) Ontology-based knowledge representation for bioinformatics. Brief. Bioinform., 1, 398– 414. 11465057
  116. Sutskever I. et al (2014) Sequence to sequence learning with neural networks. In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14. MIT Press, Cambridge, MA, USA, pp. 3104– 3112.
  117. Tang G. et al (2018) Why self-attention? A targeted evaluation of neural machine translation architectures. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, pp. 4263– 4272.
  118. Tao Y. et al (2020) From genome to phenome: predicting multiple cancer phenotypes based on somatic genomic alterations via the genomic impact transformer. In: Pacific Symposium on Biocomputing. Vol. 25, Big Island of Hawaii, USA, pp. 79– 90. PMC6932864 31797588
  119. Turian J. et al (2010) Word representations: a simple and general method for semi-supervised learning. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL ’10. Association for Computational Linguistics, Uppsala, Sweden, pp. 384– 394.
  120. Urda D. et al (2017) Deep learning to analyze RNA-seq gene expression data. In: Rojas I. et al (ed.) Advances in Computational Intelligence, Lecture Notes in Computer Science. Springer International Publishing, Cham, pp. 50– 59.
  121. Vaswani A. et al (2017) Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17. Curran Associates Inc., Red Hook, NY, USA, pp. 6000– 6010.
  122. von Heijne G. (1992) Computer analysis of DNA and protein sequences. In: Christen P., Hofmann E. (ed.) EJB Reviews 1991. Springer, Berlin, Heidelberg, pp. 85– 88. 1712725
  123. Walczak S., Cerpa N. (2003) Artificial neural networks. In: Meyers R.A. (ed.) Encyclopedia of Physical Science and Technology. 3rd edn. Academic Press, New York, pp. 631– 645.
  124. Wang S. et al (2020) Linformer: self-attention with linear complexity. arXiv, arXiv:2006.04768v3, https://arxiv.org/abs/2006.04768v3, preprint: not peer reviewed. · arXiv
  125. Wang S. et al (2019) SMILES-BERT: large scale unsupervised pre-training for molecular property prediction. In: Proceedings of the 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics. ACM, Niagara Falls, NY, USA, pp. 429– 436.
  126. Wang X. et al (2021) InferBERT: a transformer-based causal inference framework for enhancing pharmacovigilance. Front. Artif. Intell., 4. PMC8202286 34136800
  127. Wu Z. et al (2022) Knowledge-based BERT: a method to extract molecular features like computational chemists. Brief. Bioinform., 23, bbac131. 35438145
  128. Xu P. et al (2022) Multimodal learning with transformers: a survey. arXiv, arXiv:2206.06488v1, https://arxiv.org/abs/2206.06488v1, preprint: not peer reviewed. · arXiv
  129. Yang Z. et al (2019b) XLNet: generalized autoregressive pretraining for language understanding. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA.
  130. Yun S. et al (2019) Graph transformer networks. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems - Volume 32. Curran Associates Inc., Vancouver, Canada.
  131. Zaheer M. et al (2020) Big bird: transformers for longer sequences. In: Proceedings of the 34th International Conference on Neural Information Processing Systems - Volume 33. Curran Associates Inc., Vancouver, Canada, pp. 17283–17297.
  132. Zeng Y. et al (2016) A convolution BiLSTM neural network model for Chinese event extraction. In: Lin C.-Y. et al (ed.) Natural Language Understanding and Intelligent Applications, Lecture Notes in Computer Science. Springer International Publishing, Cham, pp. 275– 287.
  133. Zeng Y. et al (2022) Spatial transcriptomics prediction from histology jointly through transformer and graph neural networks. Brief. Bioinform., 23, bbac297. 35849101
  134. Zhang H. et al (2021a) Co-evolution transformer for protein contact prediction. In: Proceedings of the 35th International Conference on Neural Information Processing Systems - Volume 34. Curran Associates Inc., Virtual, pp. 14252– 14263.
  135. Zhang K. et al (2018) Residual networks of residual networks: multilevel residual networks. IEEE Trans. Circuits Syst. Video Technol., 28, 1303– 1314.
  136. Zhang L. et al (2021b) BERT-m7G: a transformer architecture based on BERT and stacking ensemble to identify RNA N7-Methylguanosine sites from sequence information. Comput. Math. Methods Med., 2021, 7764764. 34484416 PMC8413034
  137. Zhang Q. et al (2017) A review on entity relation extraction. In: 2017 Second International Conference on Mechanical, Control and Computer Engineering (ICMCCE), Harbin, China, pp. 178– 183.
  138. Zhang S. et al (2020) TensorCoder: dimension-wise attention via tensor representation for natural language modeling. arXiv, arXiv:2008.01547v2, https://arxiv.org/abs/2008.01547v2, preprint: not peer reviewed. · arXiv
  139. Zhang Z. et al (2019) ERNIE: enhanced language representation with informative entities. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, pp. 1441– 1451.
  140. Zhao G. et al (2019) Explicit sparse transformer: concentrated attention through explicit selection. arXiv, arXiv:1912.11637v1, https://arxiv.org/abs/1912.11637v1, preprint: not peer reviewed. · arXiv
  141. Zheng R. et al (2021) Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation. In: Proceedings of the 38th International Conference on Machine Learning, PMLR, Virtual, pp. 12736– 12746.
  142. Zitnik M. et al (2018) Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics, 34, i457– i466. 29949996 PMC6022705

Cited by

Sorted by citations

Authors
Zhang, S., Fan, R., Liu, Y., Chen, S., Liu, Q., & Zeng, W.
Affiliations
  • Nankai University (CN)
  • Stanford University (US)
Corresponding author
Zeng, Wanwen
ORCID
Venue
Bioinformatics Advances
Publisher
Oxford University Press (OUP)
Published
2023-01-01
Volume
3
Issue
1
Pages
vbad001-vbad001
ISSN
2635-0041 (ISSN-L)
DOI
10.1093/bioadv/vbad001
PMC
PMC9950855
OpenAlex
W4315641887
Cited by
176
References
14
Citations by year
2026: 35 · 2025: 71 · 2024: 53 · 2023: 16 · 2022: 1
FWCI
4.62
Type
journal article
Language
en
Open access
Licence
Locations
  • Bioinformatics Advances (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • National Natural Science Foundation of China (62003178)
  • National Key Research and Development Program of China (2021YFF1001000)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
Keywords
transformer language models, BERT, GPT-3, bioinformatics applications, biological sequence analysis, drug discovery, model interpretability, computational cost, natural language processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review