The DOI matches its Crossref record; 93 of the 93 references it cites were found in a registry or an index.
Pre-trained Language Models in Biomedical Domain: A Systematic Survey
Open accessAbstract
Pre-trained language models (PLMs) have been the de facto paradigm for most natural language processing tasks. This also benefits the biomedical domain: researchers from informatics, medicine, and computer science communities propose various PLMs trained on biomedical datasets, e.g., biomedical text, electronic health records, protein, and DNA sequences for various biomedical tasks. However, the cross-discipline characteristics of biomedical PLMs hinder their spreading among communities; some existing works are isolated from each other without comprehensive comparison and discussions. It is nontrivial to make a survey that not only systematically reviews recent advances in biomedical PLMs and their applications but also standardizes terminology and benchmarks. This article summarizes the recent progress of pre-trained language models in the biomedical domain and their applications in downstream biomedical tasks. Particularly, we discuss the motivations of PLMs in the biomedical domain and introduce the key concepts of pre-trained language models. We then propose a taxonomy of existing biomedical PLMs that categorizes them from various perspectives systematically. Plus, their applications in biomedical downstream tasks are exhaustively discussed, respectively. Last, we illustrate various limitations and future trends, which aims to provide inspiration for the future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Principles that Govern the Folding of Protein Chains
Anfinsen, C. B. (1973) · Science · doi:10.1126/science.181.4096.223 · Cited by 7,096
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Federated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3298981 · Cited by 6,322
Federated learning: Challenges, methods, and future directions
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2975749 · Cited by 6,210
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 4×
“For example, BERT [66] was widely accepted in benchmarks [242, 307] makes it spread to various tasks in NLP.”
BackgroundMethodInfluentialLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“Thus, subwords (limited set of common sub-word units, also called ‘wordpieces’ [266] 3 ) are used to reduce the size of word vocabulary and therefore save parameters.”
Self-citationLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 3×
“Compared to the previous paradigm with purely supervised learning that relies on feature engineering or neural network architecture engineering [182], the current two-stage paradigm is more friendly to the scenario when supervised data is limited while large-scaled unsupervised data is tremendous.”
BackgroundMethodSelf-citationLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 3×
“Compared to the previous paradigm with purely supervised learning that relies on feature engineering or neural network architecture engineering [182], the current two-stage paradigm is more friendly to the scenario when supervised data is limited while large-scaled unsupervised data is tremendous.”
BackgroundMethodSelf-citationALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · cited 8×
“With the development of general domain language models, Naseem [168] proposed an effective domain-specific language model bioALBERT trained on the biomedical domain corpora (PubMed abstracts and PMC full-text articles) for biomedical named entity recognition.”
BackgroundMethodInfluentialLarge Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 1×
“Different from supervised pre-training, contrastive self-supervised learning [48, 89, 99] is a method for representation learning without needing labeled data.”
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 2×
“Rives et al. [202] trains a contextual transformer-based language model 21 on 250 million protein sequences.”
BackgroundSciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition
Tjong Kim Sang, E. F., & De Meulder, F. (2003) · Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - · doi:10.3115/1119176.1119195 · Cited by 2,788
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Data Resource Profile: Clinical Practice Research Datalink (CPRD)
Herrett, E., Gallagher, A. M., Bhaskaran, K., Forbes, H., Mathur, R., van Staa, T., & Smeeth, L. (2015) · International Journal of Epidemiology · doi:10.1093/ije/dyv098 · Cited by 2,667 · cited 2×
“Clinical Practice Research Datalink (CPRD) [97] is the primary care database of anonymized medical records from 674 general physicians (GP) practices in the UK, which involves over 11.”
BackgroundPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 3×
“, DALLE-2 [243], Stable Diffusion [252], and Imagen [256]) have been proposed and achieved excellent performance on text-based image generation.”
BackgroundZero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · cited 10×
“This yield many efficient foundation models in the biomedical domain such as BioBERT [156] and PubMedBERT [90] et al, that can be directly used for downstream domain-specific tasks in the fine-tuning stage.”
BackgroundMethodInfluentialDon’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055
TinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · cited 4×
“…example, BERT, the most typical pre-trained language, has many variants in the biomedical domain, e.g. , Med-BERT [199], BioBERT [121], publicly available Clinical BERT Embeddings [10], SciBERT [16], ClinicalBERT [87], and COVID-twitter-BERT [165] et al. We draw an overview for these models in Fig.”
BackgroundMethodInfluentialToward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · cited 4×
“3) Pre-training can be regarded as a kind of regularization to avoid overfitting on small data [237].”
BackgroundInfluentialHow to Fine-Tune BERT for Text Classification?
Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32381-3_16 · Cited by 1,534
VisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · cited 1×
“, LXMERT [288], VisualBERT [165], UNITER [50], and PixelBERT [116]) to learn multi-modal representations from medical images and texts.”
BackgroundPreparing a collection of radiology examinations for distribution and retrieval
Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., & McDonald, C. J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv080 · Cited by 1,164
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation
Zhang, Y., Sun, S., Galley, M., Chen, Y.-C., Brockett, C., Gao, X., Gao, J., Liu, J., & Dolan, B. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.30 · Cited by 1,076
Language (Technology) is Power: A Critical Survey of “Bias” in NLP
Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.485 · Cited by 1,050
ProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009
Rasmy, L., Xiang, Y., Xie, Z., Tao, C., & Zhi, D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00455-y · Cited by 886 · cited 6×
“, Med-BERT [248], BioBERT [156], publicly available Clinical BERT Embeddings [13], SciBERT [23], ClinicalBERT [113], and COVID-twitter-BERT [210] et al.”
BackgroundMethodInfluentialAnil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.10403 · Cited by 834
Self-citationMen Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · cited 1×
“With the rapid development of AI systems and applications in industrial products, it should be aware that they should not introduce any bias for special groups or populations [177], and some of the eforts were taken in the NLP ield [24, 74, 260, 342].”
BackgroundEvaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685
Modeling the language of life – Deep Learning Protein Sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · doi:10.1101/614313 · Cited by 606
Modeling aspects of the language of life through transfer-learning protein sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3220-8 · Cited by 606 · cited 1×
“[102, 103] trains a deep bi-directional model ELMo for proteins 14.”
BackgroundBEHRT: Transformer for Electronic Health Records
Li, Y., Rao, S., Solares, J. R. A., Hassaine, A., Ramakrishnan, R., Canoy, D., Zhu, Y., Rahimi, K., & Salimi-Khorshidi, G. (2020) · Scientific Reports · doi:10.1038/s41598-020-62922-y · Cited by 595
A large‐scale experiment to assess protein structure prediction methods
Moult, J., Pedersen, J. T., Judson, R., & Fidelis, K. (1995) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.340230303 · Cited by 555
Critical assessment of methods of protein structure prediction (CASP)—Round XIII
Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., & Moult, J. (2019) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25823 · Cited by 506
Tiu, E., Talius, E., Patel, P., Langlotz, C. P., Ng, A. Y., & Rajpurkar, P. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00936-9 · Cited by 442 · cited 2×
“05 Dual-Encoder CLIP-Image CLIP-Text MIMIC-CXR CheXpert, MIMIC-CXR, NIH-CXR, PadChest CheXZero [293] 2022.”
MethodCan large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · cited 1×
“Fortunately, we have witnessed a preliminary sign that we started to investigate large language models in medical/clinical tasks [174, 187, 270], especially BIOGPT [187], PubMedGPT 68, and Flan-PaLM [270].”
BackgroundCan large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2022) · arXiv · doi:10.48550/arxiv.2207.08143 · Cited by 343
Towards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing
Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20059-5_1 · Cited by 317 · cited 1×
“12 Dual-Encoder ResNet ClinicalBERT MIMIC-CXR COVID-Rural, NIH-CXR, Object CXR, SIIM BioViL [26] 2022.”
Mining Electronic Health Records (EHRs): A Survey
Yadav, P., Steinbach, M., Kumar, V., & Simon, G. (2018) · ACM Computing Surveys · doi:10.1145/3127881 · Cited by 299
MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs
Johnson, A. E. W., Pollard, T. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Peng, Y., Lu, Z., Mark, R. G., Berkowitz, S. J., & Horng, S. (2019) · arXiv · doi:10.48550/arxiv.1901.07042 · Cited by 269
Ayala Solares, J. R., Diletta Raimondi, F. E., Zhu, Y., Rahimian, F., Canoy, D., Tran, J., Pinho Gomes, A. C., Payberah, A. H., Zottoli, M., Nazarzadeh, M., et al. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103337 · Cited by 247
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243
True Few-Shot Learning with Language Models
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv · doi:10.48550/arxiv.2105.11447 · Cited by 226 · cited 1×
“[229] evaluates the few-shot ability of LMs when held-out examples are unavailable for choosing hyperparameters or prompts and finds that LMs do not perform well compared to random selection and under-perform selection based on held-out examples.”
BackgroundLearning protein sequence embeddings using information from structure
Bepler, T., & Berger, B. (2019) · arXiv · doi:10.48550/arxiv.1902.08661 · Cited by 219
The SPECIES and ORGANISMS Resources for Fast and Accurate Identification of Taxonomic Names in Text
Pafilis, E., Frankild, S. P., Fanini, L., Faulwetter, S., Pavloudi, C., Vasileiadou, A., Arvanitidis, C., & Jensen, L. J. (2013) · PLoS ONE · doi:10.1371/journal.pone.0065390 · Cited by 206
emrQA: A Large Corpus for Question Answering on Electronic Medical Records
Pampari, A., Raghavan, P., Liang, J., & Peng, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1258 · Cited by 191 · cited 1×
“PubMedQA [122] PubMed abstracts 1,000 BioASQ [217] MEDLINE articles 885 MEDIQA [3] online community 383 emrQA [224] Clinical notes 400,000 cMedQA [367] online community 61,343 COVID19-QA [289] Literature review 124”
BackgroundLimitations of transformers on clinical text classification
Gao, S., Alawad, M., Young, M. T., Gounley, J., Schaefferkoetter, N., Yoon, H. J., Wu, X.-C., Durbin, E. B., Doherty, J., Stroup, A., et al. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3062322 · Cited by 157
TernaryBERT: Distillation-aware Ultra-low Bit BERT
Zhang, W., Hou, L., Yin, Y., Shang, L., Chen, X., Jiang, X., & Liu, Q. (2020) · arXiv · doi:10.48550/arxiv.2009.12812 · Cited by 150 · cited 1×
“For example, current work explores quantization [18, 369], weights pruning [110], and knowledge distillation [120, 257] for BERT.”
BackgroundProtein language model embeddings for fast, accurate, alignment-free protein structure prediction
Weissenow, K., Heinzinger, M., & Rost, B. (2021) · doi:10.1101/2021.07.31.454572 · Cited by 139
Garrido-Muñoz, I., Montejo-Ráez, A., Martínez-Santiago, F., & Ureña-López, L. A. (2021) · Applied Sciences · doi:10.3390/app11073184 · Cited by 123
Li, Y., Wang, H., & Luo, Y. (2020) · 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9313289 · Cited by 81
Summarizing Medical Conversations via Identifying Important Utterances
Song, Y., Tian, Y., Wang, N., & Xia, F. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.63 · Cited by 73
Extracting Symptoms and their Status from Clinical Conversations
Du, N., Chen, K., Kannan, A., Tran, L., Chen, Y., & Shafran, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1087 · Cited by 66 · cited 1×
“MZ [321] EN Pediatrics DPL Discharge summaries 710 DX [337] CN Pediatrics DPL Patient-doctor dialogues & patient reports 527 RD [172] CN Pediatrics DPL Patient-doctor dialogues & patient reports 1,490 SD [172] CN 9 domains DPL Patient-doctor dialogues & patient reports 30,000 CMDD [180] CN Pediatrics NLU Patient-docto…”
BackgroundChatGPT as a Factual Inconsistency Evaluator for Text Summarization
Luo, Z., Xie, Q., & Ananiadou, S. (2023) · arXiv · doi:10.48550/arxiv.2303.15621 · Cited by 53
Self-citationAn Empirical Study on Information Extraction using Large Language Models
Han, R., Yang, C., Peng, T., Tiwari, P., Wan, X., Liu, L., & Wang, B. (2023) · arXiv · doi:10.48550/arxiv.2305.14450 · Cited by 52
Self-citationA pre-training and self-training approach for biomedical named entity recognition
Gao, S., Kotevska, O., Sorokine, A., & Christian, J. B. (2021) · PLOS ONE · doi:10.1371/journal.pone.0246310 · Cited by 46 · cited 6×
“BioBERT [121], BlueBERT [179], SciBERT [16], PubMedBERT [66], BioMegatron [215], Yuan et al [286], Wei et al [258], Gao et al [58], Mascio et al [148], Guo et al [69], [73], Sont et al [221].”
MethodInfluentialChen, Q., Du, J., Kim, S., Wilbur, W. J., & Lu, Z. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-1044-0 · Cited by 19
MS2: Multi-Document Summarization of Medical Studies
DeYoung, J., Beltagy, I., van Zuylen, M., Kuehl, B., & Wang, L. L. (2021) · arXiv · doi:10.48550/arxiv.2104.06486 · Cited by 10
Research works printed but not linked: 262
- Asma Ben Abacha and Dina Demner-Fushman. 2016. Recognizing question entailment for medical question answering. In AMIA Annual Symposium Proceedings, Vol. 2016. American Medical Informatics Association, 310.
- Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. 2019. Overview of the mediqa 2019 shared task on textual inference, question entailment and question answering. In BioNLP Workshop and Shared Task. 370–379. · DOI
- Arda Akdemir and Tetsuo Shibuya. 2020. Transfer learning for biomedical question answering. In CLEF (Working Notes).
- Liliya Akhtyamova. 2020. Named entity recognition in spanish biomedical literature: Short review and bert model. In FRUCT. IEEE, 1–7.
- 10.1109/ACCESS.2020.3018688 · DOI
- Israa Alghanmi, Luis Espinosa-Anke, and Steven Schockaert. 2021. Probing pre-trained language models for disease knowledge. arXiv:2106.07285. Retrieved from https://arxiv.org/abs/2106.07285 · arXiv
- 10.1016/j.tibtech.2010.04.005 · DOI · Paper page
- Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for Arabic language understanding. arXiv:2003.00104. Retrieved from https://arxiv.org/abs/2003.00104 · arXiv
- Haoli Bai Wei Zhang Lu Hou Lifeng Shang Jing Jin Xin Jiang Qun Liu Michael Lyu and Irwin King. 2020. Binarybert: Pushing the limit of bert quantization. arxiv:cs.CL/2012.15701. Retrieved from https://arxiv.org/abs/2012.15701 · arXiv
- 10.1145/3465221 · DOI
- Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. 65–72.
- Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. BEiT: BERT pre-training of image transformers. In ICLR.
- Marco Basaldella, Fangyu Liu, Ehsan Shareghi, and Nigel Collier. 2020. Cometa: A corpus for medical entity linking in the social media. arXiv:2010.03295. Retrieved from https://arxiv.org/abs/2010.03295 · arXiv
- 10.5555/944919.944966 · doi:10.5555/944919.944966
- Tristan Bepler and Bonnie Berger. 2018. Learning protein sequence embeddings using information from structure. In ICLR.
- 10.18653/v1/2022.bionlp-1.22 · DOI
- Olivier Bodenreider. 2004. The unified medical language system (umls): integrating biomedical terminology. Nucl. Acids Res. 32, suppl_1 (2004), D267–D270.
- Nada Boudjellal Huaping Zhang Asif Khan Arshad Ahmad Rashid Naseem Jianyun Shang and Lin Dai. 2021. Abioner: A bert-based model for arabic biomedical named-entity recognition (unpublished). · DOI
- 10.1186/s12859-015-0472-9 · DOI · Paper page
- 10.1093/bioinformatics/btaa668 · DOI
- Eric Brochu, Vlad M. Cora, and Nando De Freitas. 2010. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv:1012.2599. Retrieved from https://arxiv.org/abs/1012.2599 · arXiv
- Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021. Knowledgeable or educated guess? revisiting language models as knowledge bases. In ACL. 1860–1874.
- 10.1016/j.asoc.2020.106779 · DOI
- Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv:2006.14799. Retrieved from https://arxiv.org/abs/2006.14799 · arXiv
- Souradip Chakraborty, Ekaba Bisong, Shweta Bhatt, Thomas Wagner, Riley Elliott, and Francesco Mosconi. 2020. Biomedbert: A pre-trained biomedical language model for qa and ir. In ICCL. 669–679.
- Pierre Chambon, Christian Bluethgen, Jean-Benoit Delbrouck, Rogier Van der Sluijs, Małgorzata Połacin, Juan Manuel Zambrano Chaves, Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P. Langlotz, and Akshay Chaudhari. 2022. RoentGen: Vision-language foundation model for chest x-ray generation. arXiv:2211.12737. Retrieved from https://arxiv.org/abs/2211.12737 · arXiv
- Pierre Chambon, Christian Bluethgen, Curtis P. Langlotz, and Akshay Chaudhari. 2022. Adapting pretrained vision-language foundational models to medical imaging domains. arXiv:2210.04133. Retrieved from https://arxiv.org/abs/2210.04133 · arXiv
- Qingyu Chen, Jingcheng Du, Sun Kim, W. John Wilbur, and Zhiyong Lu. 2019. Evaluation of five sentence similarity models on electronic medical records. In ACM-BCB. 533–533.
- Qingyu Chen, Yifan Peng, and Zhiyong Lu. 2019. Biosentvec: Creating sentence embeddings for biomedical texts. In ICHI. IEEE, 1–5.
- Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In ICML. PMLR, 1597–1607.
- Tao Chen Mingfen Wu and Hexi Li. 2019. A general approach for improving deep learning-based medical relation extraction using a pre-trained model and fine-tuning. Database (2019) baz116. · DOI
- 10.1007/978-3-031-16443-9_65 · DOI
- Zhihong Chen, Guanbin Li, and Xiang Wan. 2022. Align, reason and learn: Enhancing medical vision-and-language pre-training with knowledge. In Multimedia. 5152–5161.
- Kevin Clark Minh-Thang Luong Quoc V. Le and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arxiv:cs.CL/2003.10555. Retrieved from https://arxiv.org/abs/2003.10555 · arXiv
- 10.1093/bib/6.1.57 · DOI · Paper page
- 10.1145/1031120.1031122 · DOI · Paper page
- 10.1075/nlp.11 · DOI
- Pedro Colon-Hernandez, Catherine Havasi, Jason Alonso, Matthew Huggins, and Cynthia Breazeal. 2021. Combining pre-trained language models and structured knowledge. arXiv:2101.12294. Retrieved from https://arxiv.org/abs/2101.12294 · arXiv
- Jenny Copara, Julien Knafou, Nona Naderi, Claudia Moro, Patrick Ruch, and Douglas Teodoro. 2020. Contextualized french language models for biomedical named entity recognition. In JEP/TALN/RÉCITAL. ATALA, 36–48.
- Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-training with whole word masking for chinese bert. arXiv:1906.08101. Retrieved from https://arxiv.org/abs/1906.08101 · arXiv
- 10.1093/jamia/ocv080 · DOI · Paper page
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR. IEEE, 248–255.
- George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In HLTR. 138–145.
- 10.1016/j.jbi.2013.12.006 · DOI · Paper page
- Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. 2021. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. arXiv:2103.03404. Retrieved from https://arxiv.org/abs/a2103.03404 · arXiv
- Nan Du, Mingqiu Wang, Linh Tran, Gang Li, and Izhak Shafran. 2020. Learning to infer entities, properties and their relations from clinical conversations. In EMNLP-IJCNLP. Association for Computational Linguistics, 4978–4989.
- 10.1016/j.knosys.2020.105964 · DOI
- 10.1016/j.ymeth.2019.06.024 · DOI
- 10.1145/1118890.1118891 · DOI · Paper page
- Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. 2020. Prottrans: Towards cracking the language of life’s code through self-supervised deep learning and high performance computing. arXiv:2007.06225. Retrieved from https://arxiv.org/abs/2007.06225 · arXiv
- Andre Esteva, Anuprit Kale, Romain Paulus, Kazuma Hashimoto, Wenpeng Yin, Dragomir Radev, and Richard Socher. 2020. Co-search: COVID-19 information retrieval with semantic search, question answering, and abstractive summarization. arXiv:2006.09595. Retrieved from https://arxiv.org/abs/2006.09595 · arXiv
- Yixing Fan, Liang Pang, JianPeng Hou, Jiafeng Guo, Yanyan Lan, and Xueqi Cheng. 2017. Matchzoo: A toolkit for deep text matching. arXiv:1707.07270. Retrieved from https://arxiv.org/abs/1707.07270 · arXiv
- Sajad Sotudeh Gharebagh, Nazli Goharian, and Ross Filice. 2020. Attend to medical ontologies: Content selection for clinical abstractive summarization. In ACL. 1899–1905.
- John Giorgi, Xindi Wang, Nicola Sahar, Won Young Shin, Gary D. Bader, and Bo Wang. 2019. End-to-end named entity recognition and relation extraction using pre-trained language models. arXiv:1912.13415. Retrieved from https://arxiv.org/abs/1912.134.15 · arXiv
- Graciela Gonzalez-Hernandez, Abeed Sarker, Karen O’Connor, and Guergana Savova. 2017. Capturing the patient’s perspective: A review of advances in natural language processing of health-related text. Yrbk. Med. Inf. 26, 1 (2017), 214.
- Hong Guan, Jianfu Li, Hua Xu, and Murthy Devarakonda. 2020. Robustly pre-trained neural model for direct temporal relation extraction. arXiv:2004.06216. Retrieved from https://arxiv.org/abs/2004.06216 · arXiv
- Yuting Guo, Xiangjue Dong, Mohammed Ali Al-Garadi, Abeed Sarker, Cécile Paris, and Diego Mollá Aliod. 2020. Benchmarking of transformer-based pre-trained models on social media text classification datasets. In Workshop of the Australasian Language Technology Association. 86–91.
- Yue Guo, Wei Qiu, Yizhong Wang, and Trevor Cohen. 2020. Automated lay language summarization of biomedical scientific reviews. arXiv:2012.12573. Retrieved from https://arxiv.org/abs/2012.12573 · arXiv
- Bernal Jimenez Gutierrez, Jucheng Zeng, Dongdong Zhang, Ping Zhang, and Yu Su. 2020. Document classification for covid-19 literature. In EMNLP: Findings. 3715–3722.
- Xu Han Zhengyan Zhang Ning Ding Yuxian Gu Xiao Liu Yuqi Huo Jiezhong Qiu Liang Zhang Wentao Han Minlie Huang Qin Jin Yanyan Lan Yang Liu Zhiyuan Liu Zhiwu Lu Xipeng Qiu Ruihua Song Jie Tang Ji-Rong Wen Jinhui Yuan Wayne Xin Zhao and Jun Zhu. 2021. Pre-trained models: Past present and future. arxiv:cs.AI/2106.07139. Retrieved from https://arxiv.org/abs/2106.07139 · arXiv
- 10.1016/S0092-8674(00)81683-9 · DOI · Paper page
- Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In CVPR. 16000–16009.
- Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In CVPR. 9729–9738.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770–778.
- Yun He, Ziwei Zhu, Yin Zhang, Qin Chen, and James Caverlee. 2020. Infusing disease knowledge into bert for health question answering, medical inference and disease name recognition. In EMNLP. 4604–4614.
- Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv:1606.08415. Retrieved from https://arxiv.org/abs/1606.08415 · arXiv
- 10.1093/jamia/ocz166 · DOI · Paper page
- 10.1016/j.jbi.2013.07.011 · DOI · Paper page
- 10.1093/bioinformatics/btz694 · DOI · Paper page
- Lu Hou Zhiqi Huang Lifeng Shang Xin Jiang Xiao Chen and Qun Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. arxiv:cs.CL/2004.04037. Retrieved from https://arxiv.org/abs/2004.04037 · arXiv
- Tzu-Ming Harry Hsu, Wei-Hung Weng, Willie Boag, Matthew McDermott, and Peter Szolovits. 2018. Unsupervised multimodal representation learning across medical images and reports. arXiv:1811.08615. Retrieved from https://arxiv.org/abs/1811.08615 · arXiv
- Jinpeng Hu, Zhuo Li, Zhihong Chen, Zhen Li, Xiang Wan, and Tsung-Hui Chang. 2022. Graph enhanced contrastive learning for radiology findings summarization. In ACL (Volume 1: Long Papers). 4677–4688.
- Kexin Huang Jaan Altosaar and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arxiv:1904.05342. Retrieved from http://arxiv.org/abs/1904.05342 · arXiv
- Kexin Huang, Abhishek Singh, Sitong Chen, Edward Moseley, Chin ying Deng, Naomi George, and Charlotta Lindvall. 2019. Clinical xlnet: Modeling sequential clinical notes and predicting prolonged mechanical ventilation. arXiv:1912.11975. Retrieved from https://arxiv.org/abs/1912.11975 · arXiv
- Shih-Cheng Huang, Liyue Shen, Matthew P. Lungren, and Serena Yeung. 2021. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In IEEE/CVF ICCV. 3942–3951.
- Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. 2020. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv:2004.00849. Retrieved from https://arxiv.org/abs/2004.00849 · arXiv
- Minbyul Jeong, Mujeen Sung, Gangwoo Kim, Donghyeon Kim, Wonjin Yoon, Jaehyo Yoo, and Jaewoo Kang. 2020. Transferability of natural language inference to biomedical question answering. arXiv:2007.00217. Retrieved from https://arxiv.org/abs/2007.00217 · arXiv
- 10.1093/bioinformatics/btab671 · DOI
- Qiao Jin, Bhuwan Dhingra, William Cohen, and Xinghua Lu. 2019. Probing biomedical embeddings from language models. In Workshop on Evaluating Vector Space Representations for NLP. 82–89.
- 10.1038/s41597-019-0322-0 · DOI · Paper page
- 10.1038/sdata.2016.35 · DOI · Paper page
- 10.5555/1214993 · doi:10.5555/1214993
- Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. 2021. Ammu—A survey of transformer-based biomedical pretrained language models. arXiv:2105.00827. Retrieved from https://arxiv.org/abs/2105.00827 · arXiv
- 10.1016/j.jbi.2019.103323 · DOI · Paper page
- Sanjay Kamath, Brigitte Grau, and Yue Ma. 2019. How to pre-train your model? comparison of different pre-training models for biomedical question answering. In ECML-PKDD. Springer, 646–660.
- Yoshimasa Kawazoe Daisaku Shibata Emiko Shinohara Eiji Aramaki and Kazuhiko Ohe. 2020. A clinical specific bert developed with huge size of japanese clinical narrative (unpublished). · doi:10.1101/2020.07.07.20148585
- Muhammad Raza Khan, Morteza Ziyadi, and Mohamed AbdelHady. 2020. Mt-bioner: Multi-task learning for biomedical named entity recognition using deep bidirectional transformers. arXiv:2001.08904. Retrieved from https://arxiv.org/abs/2001.08904 · arXiv
- 10.1016/j.yjbinx.2019.100057 · DOI · Paper page
- Virapat Kieuvongngam, Bowen Tan, and Yiming Niu. 2020. Automatic text summarization of covid-19 medical research articles using bert and gpt-2. arXiv:2006.01997. Retrieved from https://arxiv/org/abs/2006.01997 · arXiv
- Jin-Dong Kim, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Nigel Collier. 2004. Introduction to the bio-entity recognition task at jnlpba. In NLPBA/BioNLP. Citeseer, 70–75.
- Jin-Dong Kim, Yue Wang, Toshihisa Takagi, and Akinori Yonezawa. 2011. Overview of genia event task in bionlp shared task 2011. In BioNLP Shared Task 2011 Workshop. 7–15.
- Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP. 1746–1751.
- Young-Min Kim and Tae-Hoon Lee. 2020. Korean clinical entity recognition from diagnosis text using bert. BMC Med. Inf. Decis. Mak. 20, 7 (2020), 1–9.
- 10.1145/234313.234358 · doi:10.1145/234313.234358
- 10.1016/j.jbi.2011.07.001 · DOI · Paper page
- Vaishnavi Kommaraju, Karthick Gunasekaran, Kun Li, Trapit Bansal, Andrew McCallum, Ivana Williams, and Ana-Maria Istrate. 2020. Unsupervised pre-training for biomedical question answering. arXiv:2009.12952. Retrieved from https://arxiv.org/abs/2009.12952 · arXiv
- Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Discourse probing of pretrained language models. arXiv:2104.05882. Retrieved from https://arxiv.org/abs/2104.05882 · arXiv
- Zeljko Kraljevic, Anthony Shek, Daniel Bean, Rebecca Bendayan, James Teo, and Richard Dobson. 2021. Medgpt: Medical concept prediction from clinical narratives. arXiv:2107.03134. Retrieved from https://arxiv.org/abs/2107.03134 · arXiv
- 10.1186/gb-2008-9-s2-s1 · DOI · Paper page
- Martin Krallinger, Obdulia Rabal, Saber A. Akhondi, Martın Pérez Pérez, Jesús Santamaría, Gael Pérez Rodríguez, et al. 2017. Overview of the biocreative vi chemical-protein interaction track. In BioCreative Challenge Evaluation Workshop, Vol. 1. 141–146.
- Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In EMNLP: System Demonstrations. 66–71.
- Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C. Wallace. 2021. Does bert pretrained on clinical notes reveal sensitive data? arXiv:2104.07762. Retrieved from https://arxiv.org/abs/2104.07762 · arXiv
- Patrick Lewis, Myle Ott, Jingfei Du, and Veselin Stoyanov. 2020. Pretrained language models for biomedical and clinical tasks: Understanding and extending the state-of-the-art. In Clinical Natural Language Processing Workshop. 146–157. · DOI · Paper page
- 10.2196/14830 · DOI · Paper page
- Jiao Li Yueping Sun Robin J. Johnson Daniela Sciaky Chih-Hsuan Wei Robert Leaman Allan Peter Davis Carolyn J. Mattingly Thomas C. Wiegers and Zhiyong Lu. 2016. Biocreative v cdr task corpus: A resource for chemical disease relation extraction (unpublished). · DOI · Paper page
- Jianquan Li, Xidong Wang, Xiangbo Wu, Zhiyi Zhang, Xiaolong Xu, Jie Fu, Prayag Tiwari, Xiang Wan, and Benyou Wang. 2023. Huatuo-26M, a large-scale chinese medical QA dataset. arXiv:2305.01526. Retrieved from https://arxiv.org/abs/2305.01526 · arXiv
- Junlong Li, Zhuosheng Zhang, Hai Zhao, Xi Zhou, and Xiang Zhou. 2020. Task-specific objectives of pre-trained language models for dialogue adaptation. arXiv:2009.04984. Retrieved from https://arxiv.org/abs/2009.04984 · arXiv
- Yikuan Li, Ramsey M. Wehbe, Faraz S. Ahmad, Hanyin Wang, and Yuan Luo. 2022. Clinical-longformer and clinical-BigBird: Transformers for long clinical sequences. arXiv:2201.11838. Retrieved from https://arxiv.org/abs/2201.11838 · arXiv
- 10.1093/jamia/ocac225 · DOI
- Zihan Li, Yunxiang Li, Qingde Li, You Zhang, Puyang Wang, Dazhou Guo, Le Lu, Dakai Jin, and Qingqi Hong. 2022. LViT: Language meets vision transformer in medical image segmentation. arXiv:2206.14718. Retrieved from https://arxiv.org/abs/2206.14718 · arXiv
- 10.1007/978-3-030-87196-3_26 · DOI
- Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! numersense: Probing numerical commonsense knowledge of pre-trained language models. arXiv:2005.00683. Retrieved from https://arxiv.org/abs/2005.00683 · arXiv
- Chen Lin, Timothy Miller, Dmitriy Dligach, Steven Bethard, and Guergana Savova. 2019. A bert-based universal model for both within-and cross-sentence clinical temporal relation extraction. In Clinical Natural Language Processing Workshop. 65–71.
- Chen Lin, Timothy Miller, Dmitriy Dligach, Farig Sadeque, Steven Bethard, and Guergana Savova. 2020. A bert-based one-pass multi-task model for clinical temporal relation extraction. In SIGBioMed Workshop on Biomedical Language Processing. 70–75.
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out. 74–81.
- Xinzhu Lin, Xiahui He, Qin Chen, Huaixiao Tou, Zhongyu Wei, and Ting Chen. 2019. Enhancing dialogue symptom diagnosis with global attention and symptom graph. In EMNLP-IJCNLP. 5033–5042.
- Qi Liu Matt J. Kusner and Phil Blunsom. 2020. A survey on contextual embeddings. arxiv:cs.CL/2003.07278. Retrieved from https://arxiv.org/abs/2003.07278 · arXiv
- Wenge Liu, Jianheng Tang, Jinghui Qin, Lin Xu, Zhen Li, and Xiaodan Liang. 2020. Meddg: A large-scale medical consultation dataset for building medical dialogue system. arXiv:2010.07497. Retrieved from https://arxiv.org/abs/2010.07497 · arXiv
- Pilar López-Úbeda, Manuel Carlos Díaz-Galiano, L. Alfonso Ureña-López, and M Teresa Martín-Valdivia. 2021. Pre-trained language models to extract information from radiological reports. CLEF eHealth (2021).
- Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2022. Readability controllable biomedical document summarization. arXiv:2210.04705. Retrieved from https://arxiv.org/abs/2210.04705 · arXiv
- Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. CitationSum: Citation-aware graph contrastive learning for scientific paper summarization. arXiv:2301.11223. Retrieved from https://arxiv.org/abs/2301.11223 · arXiv
- Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R. Eguchi, Po-Ssu Huang, and Richard Socher. 2020. Progen: Language modeling for protein generation. arXiv:2004.03497. Retrieved from https://arxiv.org/abs/2004.03497. · arXiv
- Tittaya Mairittha, Nattaya Mairittha, and Sozo Inoue. 2020. Improving fine-tuned question answering models for electronic health records. In UBICOMP. 688–691. · DOI
- 10.18653/v1/2020.bionlp-1.9 · DOI
- 10.1145/3457607 · DOI · Paper page
- Xing Meng Craig H. Ganoe Ryan T. Sieberg Yvonne Y. Cheung and Saeed Hassanpour. 2019. Self-supervised contextual language representation of radiology reports to improve the identification of communication urgency. arxiv:cs.LG/1912.02703. Retrieved from https://arxiv.org/abs/1912.02703 · arXiv
- 10.1109/jbhi.2021.3063721 · DOI
- Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020. What happens to bert embeddings during fine-tuning? arXiv:2004.14448. Retrieved from https://arxiv.org/abs/2004.14448 · arXiv
- George Michalopoulos, Yuanxin Wang, Hussam Kaka, Helen Chen, and Alex Wong. 2020. Umlsbert: Clinical domain knowledge augmentation of contextual embeddings using the unified medical language system metathesaurus. arXiv:2010.10391. Retrieved from https://arxiv.org/abs/2010.10391 · arXiv
- 10.1186/1471-2105-14-2 · DOI
- Joshua R. Minot, Nicholas Cheney, Marc Maier, Danne C. Elbers, Christopher M. Danforth, and Peter Sheridan Dodds. 2021. Interpretable bias mitigation for textual data: reducing gender bias in patient notes while maintaining classification performance. arXiv:2103.05841. Retrieved from https://arxiv.org/abs/2103.05841 · arXiv
- Giacomo Miolo Giulio Mantoan and Carlotta Orsenigo. 2021. Electramed: A new pre-trained language representation model for biomedical nlp. arxiv:cs.CL/2104.09585. Retrieved from https://arxiv.org/abs/2104.09585 · arXiv
- A. Miranda-Escalada, E. Farré, and M. Krallinger. 2020. Named entity recognition, concept normalization and clinical coding: Overview of the cantemist track for cancer text mining in spanish, corpus, guidelines, methods and results. In IberLEF.
- Florian Mock Fleming Kretschmer Anton Kriese Sebastian Böcker and Manja Marz. 2021. BERTax: Taxonomic classification of DNA sequences with Deep Neural Networks (unpublished). · DOI
- 10.1007/978-3-031-16443-9_69 · DOI
- 10.1109/JBHI.2022.3207502 · DOI · Paper page
- Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2020. On the stability of fine-tuning bert: misconceptions, explanations, and strong baselines. arXiv:2006.04884. Retrieved from https://arxiv.org/abs/2006.04884 · arXiv
- Martin Müller Marcel Salathé and Per Egil Kummervold. 2020. COVID-Twitter-BERT: A natural language processing model to analyse Covid-19 content on twitter. arxiv:2005.07503. Retrieved from https://arxiv.org/abs/2005.07503 · arXiv
- 10.3389/fpubh.2019.00081 · DOI
- Philip Müller, Georgios Kaissis, and Daniel Rueckert. 2022. The role of local alignment and uniformity in image-text contrastive learning on medical images. arXiv:2211.07254. Retrieved from https://arxiv.org/abs/2211.07254 · arXiv
- 10.1007/978-3-031-19809-0_39 · DOI
- Yuta Nakamura, Shouhei Hanaoka, Yukihiro Nomura, Naoto Hayashi, Osamu Abe, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. 2020. Kart: Privacy leakage framework of language models pre-trained with clinical records. arXiv:2101.00036. Retrieved from https://arxiv.org/abs/2101.00036 · arXiv
- Usman Naseem, Matloob Khushi, Vinay Reddy, Sakthivel Rajendran, Imran Razzak, and Jinman Kim. 2020. Bioalbert: A simple and effective pre-trained language model for biomedical named entity recognition. arXiv:2009.09223. Retrieved from https://arxiv.org/abs/2009.09223 · arXiv
- Anastasios Nentidis, Konstantinos Bougiatiotis, Anastasia Krithara, and Georgios Paliouras. 2019. Results of the seventh edition of the bioasq challenge. In ECML PKDD. Springer, 553–568.
- Oded Nov Nina Singh and Devin Mann. 2023. Putting ChatGPT’s Medical Advice to the (Turing) Test. arXiv.2301.10035. Retrieved from https://arxiv.org/abs/2301.10035 · DOI
- Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain J. Marshall, Ani Nenkova, and Byron C. Wallace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In ACL, Vol. 2018. NIH Public Access, 197.
- Ibrahim Burak Ozyurt. 2020. On the effectiveness of small, discriminatively pre-trained language representation models for biomedical text mining. In Workshop on Scholarly Document Processing. 104–112. · DOI
- Obioma Pelka, Sven Koitka, Johannes Rückert, Felix Nensa, and Christoph M. Friedrich. 2018. Radiology objects in COntext (ROCO): A multimodal image dataset. In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis. Springer, 180–189. · DOI · Paper page
- Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In BioNLP Workshop and Shared Task. 58–65. · DOI · Paper page
- Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. 1532–1543.
- 10.1146/annurev-biodatasci-030421-030931 · DOI
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. 2227–2237.
- Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In EMNLP-IJCNLP. 2463–2473.
- Long N. Phan James T. Anibal Hieu Tran Shaurya Chanana Erol Bahadroglu Alec Peltekian and Grégoire Altan-Bonnet. 2021. Scifive: A text-to-text transformer model for biomedical literature. arxiv:cs.CL/2106.03598. Retrieved from https://arxiv.org/abs/2106.03598 · arXiv
- Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020. Inexpensive domain adaptation of pretrained language models: Case studies on biomedical NER and Covid-19 QA. In EMNLP. 1482–1490.
- Sampo Pyysalo, Tomoko Ohta, Rafal Rak, Andrew Rowley, Hong-Woo Chun, Sung-Jae Jung, Sung-Pil Choi, Jun’ichi Tsujii, and Sophia Ananiadou. 2015. Overview of the cancer genetics and pathway curation tasks of bionlp shared task 2013. BMC Bioinf. 16, 10 (2015), 1–19.
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 8748–8763.
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-training. OpenAI Technical Report. (2018).
- Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. arxiv:cs.LG/1910.10683. Retrieved from https://arxiv.org/abs/1910.10683 · arXiv
- Alan Ramponi, Rob van der Goot, Rosario Lombardo, and Barbara Plank. 2020. Biomedical event extraction as sequence labeling. In EMNLP. 5357–5367.
- Bhanu Pratap Singh Rawat, Wei-Hung Weng, So Yeon Min, Preethi Raghavan, and Peter Szolovits. 2020. Entity-enriched neural models for clinical question answering. In SIGBioMed Workshop on Biomedical Language Processing. 112–122. · DOI
- Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In CVPR. 10684–10695.
- Subendhu Rongali Abhyuday Jagannatha Bhanu Pratap Singh Rawat and Hong Yu. 2021. Continual domain-tuning for pretrained language models. arxiv:cs.CL/2004.02288. Retrieved from https://arxiv.org/abs/2004.02288 · arXiv
- Frank Rudzicz and Raeid Saqur. 2020. Ethics of artificial intelligence in surgery. arXiv:2007.14302. Retrieved from https://arxiv.org/abs/2007.14302 · arXiv
- Devendra Singh Sachan, Pengtao Xie, Mrinmaya Sachan, and Eric P. Xing. 2018. Effective use of bidirectional language modeling for transfer learning in biomedical named entity recognition. In Machine Learning for Healthcare Conference. PMLR, 383–402.
- Victor Sanh Lysandre Debut Julien Chaumond and Thomas Wolf. 2020. Distilbert a distilled version of bert: Smaller faster cheaper and lighter. arxiv:cs.CL/1910.01108. Retrieved from https://arxiv.org/abs/1910.01108 · arXiv
- Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv:2211.05100. Retrieved from https://arxiv.org/abs/2211.05100 · arXiv
- Elisa Terumi Rubel Schneider, Joao Vitor Andrioli de Souza, Yohan Bonescki Gumiel, Claudia Moro, and Emerson Cabrera Paraiso. 2021. A gpt-2 language model for biomedical texts in portuguese. In CBMS. IEEE, 474–479.
- 10.18653/v1/2020.clinicalnlp-1.7 · DOI
- Constantin Seibold, Simon Reiß, M. Saquib Sarfraz, Rainer Stiefelhagen, and Jens Kleesiek. 2022. Breaking with fixed set pathology recognition through report-guided contrastive training. arXiv:2205.07139. Retrieved from https://arxiv.org/abs/2205.07139 · arXiv
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL. Association for Computational Linguistics, 1715–1725.
- Junyuan Shang, Tengfei Ma, Cao Xiao, and Jimeng Sun. 2019. Pre-training of graph augmented transformers for medication recommendation. arXiv:1906.00346. Retrieved from https://arxiv.org/abs/1906.00346 · arXiv
- Shreyas Sharma and Ron Daniel Jr au2. 2019. Bioflair: Pretrained pooled contextualized embeddings for biomedical sequence labeling tasks. arxiv:cs.CL/1908.05760. Retrieved from https://arxiv.org/abs/1908.05760. · arXiv
- Soumya Sharma, Bishal Santra, Abhik Jana, Santosh Tokala, Niloy Ganguly, and Pawan Goyal. 2019. Incorporating domain knowledge into medical NLI using knowledge graphs. In EMNLP-IJCNLP. 6092–6097.
- Golnar Sheikhshab, Inanc Birol, and Anoop Sarkar. 2018. In-domain context-aware token embeddings improve biomedical named entity recognition. In Workshop on Health Text Mining and Information Analysis. 160–164.
- Xiaoming Shi, Haifeng Hu, Wanxiang Che, Zhongqian Sun, Ting Liu, and Junzhou Huang. 2020. Understanding medical conversations with scattered keyword attention and weak supervision from responses. In AAAI, Vol. 34. 8838–8845.
- Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, and Raghav Mani. 2020. Biomegatron: Larger biomedical domain language model. In EMNLP. 4700–4706.
- 10.1093/jamia/ocz096 · DOI · Paper page
- Larry Smith, Lorraine K. Tanabe, Rie Johnson nee Ando, Cheng-Ju Kuo, I-Fang Chung, Chun-Nan Hsu, Yu-Shi Lin, Roman Klinger, Christoph M. Friedrich, Kuzman Ganchev, et al. 2008. Overview of biocreative ii gene mention recognition. Genome Biol. 9, 2 (2008), 1–19.
- 10.1093/bioinformatics/btx238 · DOI · Paper page
- Sarvesh Soni and Kirk Roberts. 2020. Evaluation of dataset selection for pre-training and fine-tuning transformer language models for clinical question answering. In LREC. 5532–5538.
- Peter Spyns. 1996. Natural language processing in medicine: An overview. Methods Inf. Med. 35, 4-5 (1996), 285–301.
- Pascal Sturmfels, Jesse Vig, Ali Madani, and Nazneen Fatema Rajani. 2020. Profile Prediction: An alignment-based pre-training task for protein sequence models. arXiv:2012.00195. Retrieved from https://arxiv.org/abs/2012.00195 · arXiv
- 10.18653/v1/2020.nlpcovid19-2.14 · DOI
- 10.18653/v1/2021.bionlp-1.1 · DOI
- Peng Su and K. Vijay-Shanker. 2020. Investigation of bert model on biomedical relation extraction based on revised fine-tuning mechanism. In BIBM. IEEE, 2522–2529.
- Sanjay Subramanian, Lucy Lu Wang, Ben Bogin, Sachin Mehta, Madeleine van Zuylen, Sravanthi Parasa, Sameer Singh, Matt Gardner, and Hannaneh Hajishirzi. 2020. MedICaT: A dataset of medical images, captions, and textual references. In EMNLP. 2112–2120.
- Dianbo Sui, Yubo Chen, Jun Zhao, Yantao Jia, Yuantao Xie, and Weijian Sun. 2020. Feded: Federated learning via ensemble distillation for medical relation extraction. In EMNLP. 2118–2128.
- Cong Sun, Zhihao Yang, Lei Wang, Yin Zhang, Hongfei Lin, and Jian Wang. 2020. Biomedical named entity recognition using bert in the machine reading comprehension framework. arXiv:2009.01560. Retrieved from https://arxiv.org/abs/2009.01560 · arXiv
- Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. arXiv:1906.08976. Retrieved from https://arxiv.org/abs/1906.08976 · arXiv
- Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In EMNLP-IJCNLP. 5100–5111.
- Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020. Efficient transformers: A survey. arXiv:2009.06732. Retrieved from https://arxiv.org/abs/2009.06732 · arXiv
- Ashok Thillaisundaram and Theodosia Togia. 2019. Biomedical relation extraction with pre-trained language representations and minimal task-specific architecture. In Workshop on BioNLP Open Shared Tasks. 84–89. · DOI
- Robert Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Fine-tuning large neural language models for biomedical natural language processing. arXiv:2112.07869. Retrieved from https://arxiv.org/abs/2112.07869 · arXiv
- Yiqi Tong, Yidong Chen, and Xiaodong Shi. 2021. A multi-task approach for improving biomedical named entity recognition by incorporating multi-granularity information. In ACL-IJCNLP. 4804–4813.
- 10.1093/bioinformatics/btaa540 · DOI
- Ryan Turner, David Eriksson, Michael McCourt, Juha Kiili, Eero Laaksonen, Zhen Xu, and Isabelle Guyon. 2021. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimization challenge 2020. arXiv:2104.10201. Retrieved from https://arxiv.org/abs/2104.10201 · arXiv
- 10.1093/bioinformatics/btaa675 · DOI
- 10.1136/amiajnl-2011-000203 · DOI · Paper page
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS. 5998–6008.
200 of 262 shown
Cited by
Yenduri, G., Ramalingam, M., Selvi, G. C., Supriya, Y., Srivastava, G., Maddikunta, P. K. R., Raj, G. D., Jhaveri, R. H., Prabadevi, B., Wang, W., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3389497 · Cited by 596
Yu, P., Xu, H., Hu, X., & Deng, C. (2023) · Healthcare · doi:10.3390/healthcare11202776 · Cited by 257
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149
Large language models for structured reporting in radiology: past, present, and future
Busch, F., Hoffmann, L., dos Santos, D. P., Makowski, M. R., Saba, L., Prucker, P., Hadamitzky, M., Navab, N., Kather, J. N., Truhn, D., et al. (2024) · European Radiology · doi:10.1007/s00330-024-11107-6 · Cited by 134
Remy, F., Demuynck, K., & Demeester, T. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae029 · Cited by 64
Metadata
- Authors
- Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J.
- Venue
- ACM Computing Surveys
- Published
- 2023-10-05
- Volume
- 56
- Issue
- 3
- Pages
- 1-52
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4385456320
- Cited by
- 169
- References
- 93
- FWCI
- 14.50
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Funders
- Chinese Key-Area Research and Development Program of Guangdong Province (2020B0101350001)
- Shenzhen Science and Technology Program (JCYJ20220818103001002)
- Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen, Shenzhen Key Research Project (C10120230151)
- Shenzhen Doctoral Startup Funding (RCBS20221008093330065)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.