Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 169

The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.

Pre-trained Language Models in Biomedical Domain: A Systematic Survey

Open access
Wang, Benyou, Xie, Qianqian, Pei, Jiahuan, Chen, Zhihong, Tiwari, Prayag, Li, Zhao, Fu, Jie·2024·ACM Computing Surveys

Abstract

Pre-trained language models (PLMs) have been the de facto paradigm for most natural language processing tasks. This also benefits the biomedical domain: researchers from informatics, medicine, and computer science communities propose various PLMs trained on biomedical datasets, e.g., biomedical text, electronic health records, protein, and DNA sequences for various biomedical tasks. However, the cross-discipline characteristics of biomedical PLMs hinder their spreading among communities; some existing works are isolated from each other without comprehensive comparison and discussions. It is nontrivial to make a survey that not only systematically reviews recent advances in biomedical PLMs and their applications but also standardizes terminology and benchmarks. This article summarizes the recent progress of pre-trained language models in the biomedical domain and their applications in downstream biomedical tasks. Particularly, we discuss the motivations of PLMs in the biomedical domain and introduce the key concepts of pre-trained language models. We then propose a taxonomy of existing biomedical PLMs that categorizes them from various perspectives systematically. Plus, their applications in biomedical downstream tasks are exhaustively discussed, respectively. Last, we illustrate various limitations and future trends, which aims to provide inspiration for the future research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3611651

Cites

79 of 79 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 79 references

Research works printed but not linked: 275

  1. Asma Ben Abacha and Dina Demner-Fushman. 2016. Recognizing question entailment for medical question answering. In AMIA Annual Symposium Proceedings, Vol. 2016. American Medical Informatics Association, 310.
  2. Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. 2019. Overview of the mediqa 2019 shared task on textual inference, question entailment and question answering. In BioNLP Workshop and Shared Task. 370–379. · DOI
  3. Arda Akdemir and Tetsuo Shibuya. 2020. Transfer learning for biomedical question answering. In CLEF (Working Notes).
  4. Liliya Akhtyamova. 2020. Named entity recognition in spanish biomedical literature: Short review and bert model. In FRUCT. IEEE, 1–7.
  5. 10.1109/ACCESS.2020.3018688 · DOI
  6. Israa Alghanmi, Luis Espinosa-Anke, and Steven Schockaert. 2021. Probing pre-trained language models for disease knowledge. arXiv:2106.07285. Retrieved from https://arxiv.org/abs/2106.07285 · arXiv
  7. 10.1016/j.tibtech.2010.04.005 · DOI · Paper page
  8. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for Arabic language understanding. arXiv:2003.00104. Retrieved from https://arxiv.org/abs/2003.00104 · arXiv
  9. Haoli Bai Wei Zhang Lu Hou Lifeng Shang Jing Jin Xin Jiang Qun Liu Michael Lyu and Irwin King. 2020. Binarybert: Pushing the limit of bert quantization. arxiv:cs.CL/2012.15701. Retrieved from https://arxiv.org/abs/2012.15701 · arXiv
  10. 10.1145/3465221 · DOI
  11. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. 65–72.
  12. Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. BEiT: BERT pre-training of image transformers. In ICLR.
  13. Marco Basaldella, Fangyu Liu, Ehsan Shareghi, and Nigel Collier. 2020. Cometa: A corpus for medical entity linking in the social media. arXiv:2010.03295. Retrieved from https://arxiv.org/abs/2010.03295 · arXiv
  14. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv: 2004.05150. Retrieved from https://arxiv.org/abs/2004.05150 · arXiv
  15. 10.5555/944919.944966 · doi:10.5555/944919.944966
  16. Tristan Bepler and Bonnie Berger. 2018. Learning protein sequence embeddings using information from structure. In ICLR.
  17. 10.18653/v1/2022.bionlp-1.22 · DOI
  18. Olivier Bodenreider. 2004. The unified medical language system (umls): integrating biomedical terminology. Nucl. Acids Res. 32, suppl_1 (2004), D267–D270.
  19. Nada Boudjellal Huaping Zhang Asif Khan Arshad Ahmad Rashid Naseem Jianyun Shang and Lin Dai. 2021. Abioner: A bert-based model for arabic biomedical named-entity recognition (unpublished). · DOI
  20. 10.1093/bioinformatics/btac020 · DOI · Paper page
  21. 10.1186/s12859-015-0472-9 · DOI · Paper page
  22. 10.1093/bioinformatics/btaa668 · DOI
  23. Eric Brochu, Vlad M. Cora, and Nando De Freitas. 2010. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv:1012.2599. Retrieved from https://arxiv.org/abs/1012.2599 · arXiv
  24. Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021. Knowledgeable or educated guess? revisiting language models as knowledge bases. In ACL. 1860–1874.
  25. 10.1016/j.asoc.2020.106779 · DOI
  26. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv:2006.14799. Retrieved from https://arxiv.org/abs/2006.14799 · arXiv
  27. Souradip Chakraborty, Ekaba Bisong, Shweta Bhatt, Thomas Wagner, Riley Elliott, and Francesco Mosconi. 2020. Biomedbert: A pre-trained biomedical language model for qa and ir. In ICCL. 669–679.
  28. Pierre Chambon, Christian Bluethgen, Jean-Benoit Delbrouck, Rogier Van der Sluijs, Małgorzata Połacin, Juan Manuel Zambrano Chaves, Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P. Langlotz, and Akshay Chaudhari. 2022. RoentGen: Vision-language foundation model for chest x-ray generation. arXiv:2211.12737. Retrieved from https://arxiv.org/abs/2211.12737 · arXiv
  29. Pierre Chambon, Christian Bluethgen, Curtis P. Langlotz, and Akshay Chaudhari. 2022. Adapting pretrained vision-language foundational models to medical imaging domains. arXiv:2210.04133. Retrieved from https://arxiv.org/abs/2210.04133 · arXiv
  30. Qingyu Chen, Jingcheng Du, Sun Kim, W. John Wilbur, and Zhiyong Lu. 2019. Evaluation of five sentence similarity models on electronic medical records. In ACM-BCB. 533–533.
  31. Qingyu Chen, Jingcheng Du, Sun Kim, W. John Wilbur, and Zhiyong Lu. 2020. Deep learning with sentence embeddings pre-trained on biomedical corpora improves the performance of finding similar sentences in electronic medical records. BMC Med. Inf. Decis. Mak. 20 (2020), 1–10.
  32. Qingyu Chen, Yifan Peng, and Zhiyong Lu. 2019. Biosentvec: Creating sentence embeddings for biomedical texts. In ICHI. IEEE, 1–5.
  33. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In ICML. PMLR, 1597–1607.
  34. Tao Chen Mingfen Wu and Hexi Li. 2019. A general approach for improving deep learning-based medical relation extraction using a pre-trained model and fine-tuning. Database (2019) baz116. · DOI
  35. 10.1007/978-3-031-16443-9_65 · DOI
  36. Zhihong Chen, Guanbin Li, and Xiang Wan. 2022. Align, reason and learn: Enhancing medical vision-and-language pre-training with knowledge. In Multimedia. 5152–5161.
  37. Kevin Clark Minh-Thang Luong Quoc V. Le and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arxiv:cs.CL/2003.10555. Retrieved from https://arxiv.org/abs/2003.10555 · arXiv
  38. 10.1093/bib/6.1.57 · DOI · Paper page
  39. 10.1145/1031120.1031122 · DOI · Paper page
  40. 10.1075/nlp.11 · DOI
  41. Pedro Colon-Hernandez, Catherine Havasi, Jason Alonso, Matthew Huggins, and Cynthia Breazeal. 2021. Combining pre-trained language models and structured knowledge. arXiv:2101.12294. Retrieved from https://arxiv.org/abs/2101.12294 · arXiv
  42. Jenny Copara, Julien Knafou, Nona Naderi, Claudia Moro, Patrick Ruch, and Douglas Teodoro. 2020. Contextualized french language models for biomedical named entity recognition. In JEP/TALN/RÉCITAL. ATALA, 36–48.
  43. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-training with whole word masking for chinese bert. arXiv:1906.08101. Retrieved from https://arxiv.org/abs/1906.08101 · arXiv
  44. 10.1093/jamia/ocv080 · DOI · Paper page
  45. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR. IEEE, 248–255.
  46. Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Wang. 2021. MSˆ2: Multi-document summarization of medical studies. In EMNLP. 7494–7513.
  47. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In HLTR. 138–145.
  48. 10.1016/j.jbi.2013.12.006 · DOI · Paper page
  49. Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. 2021. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. arXiv:2103.03404. Retrieved from https://arxiv.org/abs/a2103.03404 · arXiv
  50. Nan Du, Mingqiu Wang, Linh Tran, Gang Li, and Izhak Shafran. 2020. Learning to infer entities, properties and their relations from clinical conversations. In EMNLP-IJCNLP. Association for Computational Linguistics, 4978–4989.
  51. 10.1016/j.knosys.2020.105964 · DOI
  52. 10.1016/j.ymeth.2019.06.024 · DOI
  53. 10.1145/1118890.1118891 · DOI · Paper page
  54. Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. 2020. Prottrans: Towards cracking the language of life’s code through self-supervised deep learning and high performance computing. arXiv:2007.06225. Retrieved from https://arxiv.org/abs/2007.06225 · arXiv
  55. Andre Esteva, Anuprit Kale, Romain Paulus, Kazuma Hashimoto, Wenpeng Yin, Dragomir Radev, and Richard Socher. 2020. Co-search: COVID-19 information retrieval with semantic search, question answering, and abstractive summarization. arXiv:2006.09595. Retrieved from https://arxiv.org/abs/2006.09595 · arXiv
  56. Yixing Fan, Liang Pang, JianPeng Hou, Jiafeng Guo, Yanyan Lan, and Xueqi Cheng. 2017. Matchzoo: A toolkit for deep text matching. arXiv:1707.07270. Retrieved from https://arxiv.org/abs/1707.07270 · arXiv
  57. Sajad Sotudeh Gharebagh, Nazli Goharian, and Ross Filice. 2020. Attend to medical ontologies: Content selection for clinical abstractive summarization. In ACL. 1899–1905.
  58. John Giorgi, Xindi Wang, Nicola Sahar, Won Young Shin, Gary D. Bader, and Bo Wang. 2019. End-to-end named entity recognition and relation extraction using pre-trained language models. arXiv:1912.13415. Retrieved from https://arxiv.org/abs/1912.134.15 · arXiv
  59. Graciela Gonzalez-Hernandez, Abeed Sarker, Karen O’Connor, and Guergana Savova. 2017. Capturing the patient’s perspective: A review of advances in natural language processing of health-related text. Yrbk. Med. Inf. 26, 1 (2017), 214.
  60. Hong Guan, Jianfu Li, Hua Xu, and Murthy Devarakonda. 2020. Robustly pre-trained neural model for direct temporal relation extraction. arXiv:2004.06216. Retrieved from https://arxiv.org/abs/2004.06216 · arXiv
  61. Yuting Guo, Xiangjue Dong, Mohammed Ali Al-Garadi, Abeed Sarker, Cécile Paris, and Diego Mollá Aliod. 2020. Benchmarking of transformer-based pre-trained models on social media text classification datasets. In Workshop of the Australasian Language Technology Association. 86–91.
  62. Yue Guo, Wei Qiu, Yizhong Wang, and Trevor Cohen. 2020. Automated lay language summarization of biomedical scientific reviews. arXiv:2012.12573. Retrieved from https://arxiv.org/abs/2012.12573 · arXiv
  63. Bernal Jimenez Gutierrez, Jucheng Zeng, Dongdong Zhang, Ping Zhang, and Yu Su. 2020. Document classification for covid-19 literature. In EMNLP: Findings. 3715–3722.
  64. Xu Han Zhengyan Zhang Ning Ding Yuxian Gu Xiao Liu Yuqi Huo Jiezhong Qiu Liang Zhang Wentao Han Minlie Huang Qin Jin Yanyan Lan Yang Liu Zhiyuan Liu Zhiwu Lu Xipeng Qiu Ruihua Song Jie Tang Ji-Rong Wen Jinhui Yuan Wayne Xin Zhao and Jun Zhu. 2021. Pre-trained models: Past present and future. arxiv:cs.AI/2106.07139. Retrieved from https://arxiv.org/abs/2106.07139 · arXiv
  65. 10.1016/S0092-8674(00)81683-9 · DOI · Paper page
  66. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In CVPR. 16000–16009.
  67. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In CVPR. 9729–9738.
  68. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770–778.
  69. Yun He, Ziwei Zhu, Yin Zhang, Qin Chen, and James Caverlee. 2020. Infusing disease knowledge into bert for health question answering, medical inference and disease name recognition. In EMNLP. 4604–4614.
  70. Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv:1606.08415. Retrieved from https://arxiv.org/abs/1606.08415 · arXiv
  71. 10.1093/jamia/ocz166 · DOI · Paper page
  72. 10.1016/j.jbi.2013.07.011 · DOI · Paper page
  73. 10.1093/bioinformatics/btz694 · DOI · Paper page
  74. Lu Hou Zhiqi Huang Lifeng Shang Xin Jiang Xiao Chen and Qun Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. arxiv:cs.CL/2004.04037. Retrieved from https://arxiv.org/abs/2004.04037 · arXiv
  75. Tzu-Ming Harry Hsu, Wei-Hung Weng, Willie Boag, Matthew McDermott, and Peter Szolovits. 2018. Unsupervised multimodal representation learning across medical images and reports. arXiv:1811.08615. Retrieved from https://arxiv.org/abs/1811.08615 · arXiv
  76. Jinpeng Hu, Zhuo Li, Zhihong Chen, Zhen Li, Xiang Wan, and Tsung-Hui Chang. 2022. Graph enhanced contrastive learning for radiology findings summarization. In ACL (Volume 1: Long Papers). 4677–4688.
  77. Kexin Huang Jaan Altosaar and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arxiv:1904.05342. Retrieved from http://arxiv.org/abs/1904.05342 · arXiv
  78. Kexin Huang, Abhishek Singh, Sitong Chen, Edward Moseley, Chin ying Deng, Naomi George, and Charlotta Lindvall. 2019. Clinical xlnet: Modeling sequential clinical notes and predicting prolonged mechanical ventilation. arXiv:1912.11975. Retrieved from https://arxiv.org/abs/1912.11975 · arXiv
  79. Shih-Cheng Huang, Liyue Shen, Matthew P. Lungren, and Serena Yeung. 2021. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In IEEE/CVF ICCV. 3942–3951.
  80. Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. 2020. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv:2004.00849. Retrieved from https://arxiv.org/abs/2004.00849 · arXiv
  81. Minbyul Jeong, Mujeen Sung, Gangwoo Kim, Donghyeon Kim, Wonjin Yoon, Jaehyo Yoo, and Jaewoo Kang. 2020. Transferability of natural language inference to biomedical question answering. arXiv:2007.00217. Retrieved from https://arxiv.org/abs/2007.00217 · arXiv
  82. 10.1093/bioinformatics/btab671 · DOI
  83. Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li Fang Wang and Qun Liu. 2020. Tinybert: Distilling bert for natural language understanding. arxiv:cs.CL/1909.10351. Retrieved from https://arxiv.org/abs/1909.10351 · DOI · Paper page
  84. Qiao Jin, Bhuwan Dhingra, William Cohen, and Xinghua Lu. 2019. Probing biomedical embeddings from language models. In Workshop on Evaluating Vector Space Representations for NLP. 82–89.
  85. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In EMNLP-IJCNLP. 2567–2577.
  86. 10.1038/s41597-019-0322-0 · DOI · Paper page
  87. 10.1038/sdata.2016.35 · DOI · Paper page
  88. 10.5555/1214993 · doi:10.5555/1214993
  89. Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. 2021. Ammu—A survey of transformer-based biomedical pretrained language models. arXiv:2105.00827. Retrieved from https://arxiv.org/abs/2105.00827 · arXiv
  90. 10.1016/j.jbi.2019.103323 · DOI · Paper page
  91. Sanjay Kamath, Brigitte Grau, and Yue Ma. 2019. How to pre-train your model? comparison of different pre-training models for biomedical question answering. In ECML-PKDD. Springer, 646–660.
  92. Yoshimasa Kawazoe Daisaku Shibata Emiko Shinohara Eiji Aramaki and Kazuhiko Ohe. 2020. A clinical specific bert developed with huge size of japanese clinical narrative (unpublished). · doi:10.1101/2020.07.07.20148585
  93. Muhammad Raza Khan, Morteza Ziyadi, and Mohamed AbdelHady. 2020. Mt-bioner: Multi-task learning for biomedical named entity recognition using deep bidirectional transformers. arXiv:2001.08904. Retrieved from https://arxiv.org/abs/2001.08904 · arXiv
  94. 10.1016/j.yjbinx.2019.100057 · DOI · Paper page
  95. Virapat Kieuvongngam, Bowen Tan, and Yiming Niu. 2020. Automatic text summarization of covid-19 medical research articles using bert and gpt-2. arXiv:2006.01997. Retrieved from https://arxiv/org/abs/2006.01997 · arXiv
  96. Jin-Dong Kim, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Nigel Collier. 2004. Introduction to the bio-entity recognition task at jnlpba. In NLPBA/BioNLP. Citeseer, 70–75.
  97. Jin-Dong Kim, Yue Wang, Toshihisa Takagi, and Akinori Yonezawa. 2011. Overview of genia event task in bionlp shared task 2011. In BioNLP Shared Task 2011 Workshop. 7–15.
  98. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP. 1746–1751.
  99. Young-Min Kim and Tae-Hoon Lee. 2020. Korean clinical entity recognition from diagnosis text using bert. BMC Med. Inf. Decis. Mak. 20, 7 (2020), 1–9.
  100. 10.1145/234313.234358 · doi:10.1145/234313.234358
  101. 10.1016/j.jbi.2011.07.001 · DOI · Paper page
  102. Vaishnavi Kommaraju, Karthick Gunasekaran, Kun Li, Trapit Bansal, Andrew McCallum, Ivana Williams, and Ana-Maria Istrate. 2020. Unsupervised pre-training for biomedical question answering. arXiv:2009.12952. Retrieved from https://arxiv.org/abs/2009.12952 · arXiv
  103. Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Discourse probing of pretrained language models. arXiv:2104.05882. Retrieved from https://arxiv.org/abs/2104.05882 · arXiv
  104. Zeljko Kraljevic, Anthony Shek, Daniel Bean, Rebecca Bendayan, James Teo, and Richard Dobson. 2021. Medgpt: Medical concept prediction from clinical narratives. arXiv:2107.03134. Retrieved from https://arxiv.org/abs/2107.03134 · arXiv
  105. 10.1186/gb-2008-9-s2-s1 · DOI · Paper page
  106. Martin Krallinger, Obdulia Rabal, Saber A. Akhondi, Martın Pérez Pérez, Jesús Santamaría, Gael Pérez Rodríguez, et al. 2017. Overview of the biocreative vi chemical-protein interaction track. In BioCreative Challenge Evaluation Workshop, Vol. 1. 141–146.
  107. 10.1002/prot.25823 · DOI · Paper page
  108. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In EMNLP: System Demonstrations. 66–71.
  109. 10.1093/bioinformatics/btz682 · DOI · Paper page
  110. Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C. Wallace. 2021. Does bert pretrained on clinical notes reveal sensitive data? arXiv:2104.07762. Retrieved from https://arxiv.org/abs/2104.07762 · arXiv
  111. Patrick Lewis, Myle Ott, Jingfei Du, and Veselin Stoyanov. 2020. Pretrained language models for biomedical and clinical tasks: Understanding and extending the state-of-the-art. In Clinical Natural Language Processing Workshop. 146–157. · DOI · Paper page
  112. 10.2196/14830 · DOI · Paper page
  113. Jiao Li Yueping Sun Robin J. Johnson Daniela Sciaky Chih-Hsuan Wei Robert Leaman Allan Peter Davis Carolyn J. Mattingly Thomas C. Wiegers and Zhiyong Lu. 2016. Biocreative v cdr task corpus: A resource for chemical disease relation extraction (unpublished). · DOI · Paper page
  114. Jianquan Li, Xidong Wang, Xiangbo Wu, Zhiyi Zhang, Xiaolong Xu, Jie Fu, Prayag Tiwari, Xiang Wan, and Benyou Wang. 2023. Huatuo-26M, a large-scale chinese medical QA dataset. arXiv:2305.01526. Retrieved from https://arxiv.org/abs/2305.01526 · arXiv
  115. Junlong Li, Zhuosheng Zhang, Hai Zhao, Xi Zhou, and Xiang Zhou. 2020. Task-specific objectives of pre-trained language models for dialogue adaptation. arXiv:2009.04984. Retrieved from https://arxiv.org/abs/2009.04984 · arXiv
  116. Yikuan Li, Hanyin Wang, and Yuan Luo. 2020. A comparison of pre-trained vision-and-language models for multimodal representation learning across medical images and reports. In BIBM. IEEE, 1999–2004.
  117. Yikuan Li, Ramsey M. Wehbe, Faraz S. Ahmad, Hanyin Wang, and Yuan Luo. 2022. Clinical-longformer and clinical-BigBird: Transformers for long clinical sequences. arXiv:2201.11838. Retrieved from https://arxiv.org/abs/2201.11838 · arXiv
  118. 10.1093/jamia/ocac225 · DOI
  119. Zihan Li, Yunxiang Li, Qingde Li, You Zhang, Puyang Wang, Dazhou Guo, Le Lu, Dakai Jin, and Qingqi Hong. 2022. LViT: Language meets vision transformer in medical image segmentation. arXiv:2206.14718. Retrieved from https://arxiv.org/abs/2206.14718 · arXiv
  120. 10.1007/978-3-030-87196-3_26 · DOI
  121. Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! numersense: Probing numerical commonsense knowledge of pre-trained language models. arXiv:2005.00683. Retrieved from https://arxiv.org/abs/2005.00683 · arXiv
  122. Chen Lin, Timothy Miller, Dmitriy Dligach, Steven Bethard, and Guergana Savova. 2019. A bert-based universal model for both within-and cross-sentence clinical temporal relation extraction. In Clinical Natural Language Processing Workshop. 65–71.
  123. Chen Lin, Timothy Miller, Dmitriy Dligach, Farig Sadeque, Steven Bethard, and Guergana Savova. 2020. A bert-based one-pass multi-task model for clinical temporal relation extraction. In SIGBioMed Workshop on Biomedical Language Processing. 70–75.
  124. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out. 74–81.
  125. Xinzhu Lin, Xiahui He, Qin Chen, Huaixiao Tou, Zhongyu Wei, and Ting Chen. 2019. Enhancing dialogue symptom diagnosis with global attention and symptom graph. In EMNLP-IJCNLP. 5033–5042.
  126. Qi Liu Matt J. Kusner and Phil Blunsom. 2020. A survey on contextual embeddings. arxiv:cs.CL/2003.07278. Retrieved from https://arxiv.org/abs/2003.07278 · arXiv
  127. Wenge Liu, Jianheng Tang, Jinghui Qin, Lin Xu, Zhen Li, and Xiaodan Liang. 2020. Meddg: A large-scale medical consultation dataset for building medical dialogue system. arXiv:2010.07497. Retrieved from https://arxiv.org/abs/2010.07497 · arXiv
  128. Pilar López-Úbeda, Manuel Carlos Díaz-Galiano, L. Alfonso Ureña-López, and M Teresa Martín-Valdivia. 2021. Pre-trained language models to extract information from radiological reports. CLEF eHealth (2021).
  129. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2022. Readability controllable biomedical document summarization. arXiv:2210.04705. Retrieved from https://arxiv.org/abs/2210.04705 · arXiv
  130. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for abstractive text summarization. arXiv:2303.15621. Retrieved from https://arxiv.org/abs/2303.15621 · arXiv
  131. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. CitationSum: Citation-aware graph contrastive learning for scientific paper summarization. arXiv:2301.11223. Retrieved from https://arxiv.org/abs/2301.11223 · arXiv
  132. Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R. Eguchi, Po-Ssu Huang, and Richard Socher. 2020. Progen: Language modeling for protein generation. arXiv:2004.03497. Retrieved from https://arxiv.org/abs/2004.03497. · arXiv
  133. Tittaya Mairittha, Nattaya Mairittha, and Sozo Inoue. 2020. Improving fine-tuned question answering models for electronic health records. In UBICOMP. 688–691. · DOI
  134. 10.18653/v1/2020.bionlp-1.9 · DOI
  135. 10.1145/3457607 · DOI · Paper page
  136. Xing Meng Craig H. Ganoe Ryan T. Sieberg Yvonne Y. Cheung and Saeed Hassanpour. 2019. Self-supervised contextual language representation of radiology reports to improve the identification of communication urgency. arxiv:cs.LG/1912.02703. Retrieved from https://arxiv.org/abs/1912.02703 · arXiv
  137. 10.1109/jbhi.2021.3063721 · DOI
  138. Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020. What happens to bert embeddings during fine-tuning? arXiv:2004.14448. Retrieved from https://arxiv.org/abs/2004.14448 · arXiv
  139. George Michalopoulos, Yuanxin Wang, Hussam Kaka, Helen Chen, and Alex Wong. 2020. Umlsbert: Clinical domain knowledge augmentation of contextual embeddings using the unified medical language system metathesaurus. arXiv:2010.10391. Retrieved from https://arxiv.org/abs/2010.10391 · arXiv
  140. 10.1186/1471-2105-14-2 · DOI
  141. Joshua R. Minot, Nicholas Cheney, Marc Maier, Danne C. Elbers, Christopher M. Danforth, and Peter Sheridan Dodds. 2021. Interpretable bias mitigation for textual data: reducing gender bias in patient notes while maintaining classification performance. arXiv:2103.05841. Retrieved from https://arxiv.org/abs/2103.05841 · arXiv
  142. Giacomo Miolo Giulio Mantoan and Carlotta Orsenigo. 2021. Electramed: A new pre-trained language representation model for biomedical nlp. arxiv:cs.CL/2104.09585. Retrieved from https://arxiv.org/abs/2104.09585 · arXiv
  143. A. Miranda-Escalada, E. Farré, and M. Krallinger. 2020. Named entity recognition, concept normalization and clinical coding: Overview of the cantemist track for cancer text mining in spanish, corpus, guidelines, methods and results. In IberLEF.
  144. Florian Mock Fleming Kretschmer Anton Kriese Sebastian Böcker and Manja Marz. 2021. BERTax: Taxonomic classification of DNA sequences with Deep Neural Networks (unpublished). · DOI
  145. 10.1007/978-3-031-16443-9_69 · DOI
  146. 10.1109/JBHI.2022.3207502 · DOI · Paper page
  147. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2020. On the stability of fine-tuning bert: misconceptions, explanations, and strong baselines. arXiv:2006.04884. Retrieved from https://arxiv.org/abs/2006.04884 · arXiv
  148. Martin Müller Marcel Salathé and Per Egil Kummervold. 2020. COVID-Twitter-BERT: A natural language processing model to analyse Covid-19 content on twitter. arxiv:2005.07503. Retrieved from https://arxiv.org/abs/2005.07503 · arXiv
  149. 10.3389/fpubh.2019.00081 · DOI
  150. Philip Müller, Georgios Kaissis, and Daniel Rueckert. 2022. The role of local alignment and uniformity in image-text contrastive learning on medical images. arXiv:2211.07254. Retrieved from https://arxiv.org/abs/2211.07254 · arXiv
  151. 10.1007/978-3-031-19809-0_39 · DOI
  152. Yuta Nakamura, Shouhei Hanaoka, Yukihiro Nomura, Naoto Hayashi, Osamu Abe, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. 2020. Kart: Privacy leakage framework of language models pre-trained with clinical records. arXiv:2101.00036. Retrieved from https://arxiv.org/abs/2101.00036 · arXiv
  153. Usman Naseem, Matloob Khushi, Vinay Reddy, Sakthivel Rajendran, Imran Razzak, and Jinman Kim. 2020. Bioalbert: A simple and effective pre-trained language model for biomedical named entity recognition. arXiv:2009.09223. Retrieved from https://arxiv.org/abs/2009.09223 · arXiv
  154. Anastasios Nentidis, Konstantinos Bougiatiotis, Anastasia Krithara, and Georgios Paliouras. 2019. Results of the seventh edition of the bioasq challenge. In ECML PKDD. Springer, 553–568.
  155. Oded Nov Nina Singh and Devin Mann. 2023. Putting ChatGPT’s Medical Advice to the (Turing) Test. arXiv.2301.10035. Retrieved from https://arxiv.org/abs/2301.10035 · DOI
  156. Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain J. Marshall, Ani Nenkova, and Byron C. Wallace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In ACL, Vol. 2018. NIH Public Access, 197.
  157. Ibrahim Burak Ozyurt. 2020. On the effectiveness of small, discriminatively pre-trained language representation models for biomedical text mining. In Workshop on Scholarly Document Processing. 104–112. · DOI
  158. Obioma Pelka, Sven Koitka, Johannes Rückert, Felix Nensa, and Christoph M. Friedrich. 2018. Radiology objects in COntext (ROCO): A multimodal image dataset. In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis. Springer, 180–189. · DOI · Paper page
  159. Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In BioNLP Workshop and Shared Task. 58–65. · DOI · Paper page
  160. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. 1532–1543.
  161. 10.1146/annurev-biodatasci-030421-030931 · DOI
  162. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. 2227–2237.
  163. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In EMNLP-IJCNLP. 2463–2473.
  164. Long N. Phan James T. Anibal Hieu Tran Shaurya Chanana Erol Bahadroglu Alec Peltekian and Grégoire Altan-Bonnet. 2021. Scifive: A text-to-text transformer model for biomedical literature. arxiv:cs.CL/2106.03598. Retrieved from https://arxiv.org/abs/2106.03598 · arXiv
  165. Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020. Inexpensive domain adaptation of pretrained language models: Case studies on biomedical NER and Covid-19 QA. In EMNLP. 1482–1490.
  166. Sampo Pyysalo, Tomoko Ohta, Rafal Rak, Andrew Rowley, Hong-Woo Chun, Sung-Jae Jung, Sung-Pil Choi, Jun’ichi Tsujii, and Sophia Ananiadou. 2015. Overview of the cancer genetics and pathway curation tasks of bionlp shared task 2013. BMC Bioinf. 16, 10 (2015), 1–19.
  167. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 8748–8763.
  168. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-training. OpenAI Technical Report. (2018).
  169. Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. arxiv:cs.LG/1910.10683. Retrieved from https://arxiv.org/abs/1910.10683 · arXiv
  170. Alan Ramponi, Rob van der Goot, Rosario Lombardo, and Barbara Plank. 2020. Biomedical event extraction as sequence labeling. In EMNLP. 5357–5367.
  171. Bhanu Pratap Singh Rawat, Wei-Hung Weng, So Yeon Min, Preethi Raghavan, and Peter Szolovits. 2020. Entity-enriched neural models for clinical question answering. In SIGBioMed Workshop on Biomedical Language Processing. 112–122. · DOI
  172. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In CVPR. 10684–10695.
  173. Subendhu Rongali Abhyuday Jagannatha Bhanu Pratap Singh Rawat and Hong Yu. 2021. Continual domain-tuning for pretrained language models. arxiv:cs.CL/2004.02288. Retrieved from https://arxiv.org/abs/2004.02288 · arXiv
  174. Frank Rudzicz and Raeid Saqur. 2020. Ethics of artificial intelligence in surgery. arXiv:2007.14302. Retrieved from https://arxiv.org/abs/2007.14302 · arXiv
  175. Devendra Singh Sachan, Pengtao Xie, Mrinmaya Sachan, and Eric P. Xing. 2018. Effective use of bidirectional language modeling for transfer learning in biomedical named entity recognition. In Machine Learning for Healthcare Conference. PMLR, 383–402.
  176. Victor Sanh Lysandre Debut Julien Chaumond and Thomas Wolf. 2020. Distilbert a distilled version of bert: Smaller faster cheaper and lighter. arxiv:cs.CL/1910.01108. Retrieved from https://arxiv.org/abs/1910.01108 · arXiv
  177. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv:2211.05100. Retrieved from https://arxiv.org/abs/2211.05100 · arXiv
  178. Elisa Terumi Rubel Schneider, Joao Vitor Andrioli de Souza, Yohan Bonescki Gumiel, Claudia Moro, and Emerson Cabrera Paraiso. 2021. A gpt-2 language model for biomedical texts in portuguese. In CBMS. IEEE, 474–479.
  179. 10.18653/v1/2020.clinicalnlp-1.7 · DOI
  180. Constantin Seibold, Simon Reiß, M. Saquib Sarfraz, Rainer Stiefelhagen, and Jens Kleesiek. 2022. Breaking with fixed set pathology recognition through report-guided contrastive training. arXiv:2205.07139. Retrieved from https://arxiv.org/abs/2205.07139 · arXiv
  181. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL. Association for Computational Linguistics, 1715–1725.
  182. Junyuan Shang, Tengfei Ma, Cao Xiao, and Jimeng Sun. 2019. Pre-training of graph augmented transformers for medication recommendation. arXiv:1906.00346. Retrieved from https://arxiv.org/abs/1906.00346 · arXiv
  183. Shreyas Sharma and Ron Daniel Jr au2. 2019. Bioflair: Pretrained pooled contextualized embeddings for biomedical sequence labeling tasks. arxiv:cs.CL/1908.05760. Retrieved from https://arxiv.org/abs/1908.05760. · arXiv
  184. Soumya Sharma, Bishal Santra, Abhik Jana, Santosh Tokala, Niloy Ganguly, and Pawan Goyal. 2019. Incorporating domain knowledge into medical NLI using knowledge graphs. In EMNLP-IJCNLP. 6092–6097.
  185. Golnar Sheikhshab, Inanc Birol, and Anoop Sarkar. 2018. In-domain context-aware token embeddings improve biomedical named entity recognition. In Workshop on Health Text Mining and Information Analysis. 160–164.
  186. Xiaoming Shi, Haifeng Hu, Wanxiang Che, Zhongqian Sun, Ting Liu, and Junzhou Huang. 2020. Understanding medical conversations with scattered keyword attention and weak supervision from responses. In AAAI, Vol. 34. 8838–8845.
  187. Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, and Raghav Mani. 2020. Biomegatron: Larger biomedical domain language model. In EMNLP. 4700–4706.
  188. 10.1093/jamia/ocz096 · DOI · Paper page
  189. Larry Smith, Lorraine K. Tanabe, Rie Johnson nee Ando, Cheng-Ju Kuo, I-Fang Chung, Chun-Nan Hsu, Yu-Shi Lin, Roman Klinger, Christoph M. Friedrich, Kuzman Ganchev, et al. 2008. Overview of biocreative ii gene mention recognition. Genome Biol. 9, 2 (2008), 1–19.
  190. 10.1093/bioinformatics/btx238 · DOI · Paper page
  191. Sarvesh Soni and Kirk Roberts. 2020. Evaluation of dataset selection for pre-training and fine-tuning transformer language models for clinical question answering. In LREC. 5532–5538.
  192. Peter Spyns. 1996. Natural language processing in medicine: An overview. Methods Inf. Med. 35, 4-5 (1996), 285–301.
  193. Pascal Sturmfels, Jesse Vig, Ali Madani, and Nazneen Fatema Rajani. 2020. Profile Prediction: An alignment-based pre-training task for protein sequence models. arXiv:2012.00195. Retrieved from https://arxiv.org/abs/2012.00195 · arXiv
  194. 10.18653/v1/2020.nlpcovid19-2.14 · DOI
  195. 10.18653/v1/2021.bionlp-1.1 · DOI
  196. Peng Su and K. Vijay-Shanker. 2020. Investigation of bert model on biomedical relation extraction based on revised fine-tuning mechanism. In BIBM. IEEE, 2522–2529.
  197. Sanjay Subramanian, Lucy Lu Wang, Ben Bogin, Sachin Mehta, Madeleine van Zuylen, Sravanthi Parasa, Sameer Singh, Matt Gardner, and Hannaneh Hajishirzi. 2020. MedICaT: A dataset of medical images, captions, and textual references. In EMNLP. 2112–2120.
  198. Dianbo Sui, Yubo Chen, Jun Zhao, Yantao Jia, Yuantao Xie, and Weijian Sun. 2020. Feded: Federated learning via ensemble distillation for medical relation extraction. In EMNLP. 2118–2128.
  199. 10.1007/978-3-030-32381-3_16 · DOI · Paper page
  200. Cong Sun, Zhihao Yang, Lei Wang, Yin Zhang, Hongfei Lin, and Jian Wang. 2020. Biomedical named entity recognition using bert in the machine reading comprehension framework. arXiv:2009.01560. Retrieved from https://arxiv.org/abs/2009.01560 · arXiv

200 of 275 shown

Cited by

See all 169

Metadata

Authors
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J.
Venue
ACM Computing Surveys
Published
2023-10-05
Volume
56
Issue
3
Pages
1-52
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
OpenAlex
W4385456320
Cited by
169
References
79
FWCI
14.50
Type
journal article
Language
en
Open access
hybrid
Funders
  • Chinese Key-Area Research and Development Program of Guangdong Province (2020B0101350001)
  • Shenzhen Science and Technology Program (JCYJ20220818103001002)
  • Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen, Shenzhen Key Research Project (C10120230151)
  • Shenzhen Doctoral Startup Funding (RCBS20221008093330065)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Pre-trained Language Models in Biomedical Domain: A Systematic Survey. https://app.cochat.ai/papers/10.1145/3611651
The paper
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2024). Pre-trained Language Models in Biomedical Domain: A Systematic Survey. ACM Computing Surveys. https://doi.org/10.1145/3611651

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.