CoChat
Sign Up
PreprintCited by 14Artificial Intelligence in Healthcare and EducationTopic ModelingMachine Learning in Healthcare

A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry

Open access
Huang, Yining, Tang, Keke, Chen, Meilian, Wang, Boyuan·2024·arXiv
DOI 10.48550/arxiv.2404.15777arXiv 2404.15777
Open this paper in CoChat — free accountOpen PDF

Abstract

Since the inception of the Transformer architecture in 2017, Large Language Models (LLMs) such as GPT and BERT have evolved significantly, impacting various industries with their advanced capabilities in language understanding and generation. These models have shown potential to transform the medical field, highlighting the necessity for specialized evaluation frameworks to ensure their effective and ethical deployment. This comprehensive survey delineates the extensive application and requisite evaluation of LLMs within healthcare, emphasizing the critical need for empirical validation to fully exploit their capabilities in enhancing healthcare outcomes. Our survey is structured to provide an in-depth analysis of LLM applications across clinical settings, medical text data processing, research, education, and public health awareness. We begin by exploring the roles of LLMs in various medical applications, detailing their evaluation based on performance in tasks such as clinical diagnosis, medical text data processing, information retrieval, data analysis, and educational content generation. The subsequent sections offer a comprehensive discussion on the evaluation methods and metrics employed, including models, evaluators, and comparative experiments. We further examine the benchmarks and datasets utilized in these evaluations, providing a categorized description of benchmarks for tasks like question answering, summarization, information extraction, bioinformatics, information retrieval and general comprehensive benchmarks. This structure ensures a thorough understanding of how LLMs are assessed for their effectiveness, accuracy, usability, and ethical alignment in the medical domain. ...

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2404.15777

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Traditional NLP taxonomies separate hallucination types into distinct categories such as intrinsic/extrinsic, factuality/faithfulness, and factual mirage/silver lining.

    “Typically, these differ from traditional natural language processing (NLP) taxonomies16, which have separated hallucination types into distinct categories, for example, into “intrinsic” and “extrinsic”24, “factuality” and “faithfulness”16, “factual mirage” and “silver lining”25 errors.”

    From A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation (2025) · cites this paper as [25]

    The Introduction & Background mentions hallucinations, and §3.2.3 discusses factual accuracy as an evaluation metric, but neither describes hallucination taxonomies. Searches for “factuality,” “faithfulness,” and hallucination types found no passage supporting the distinction attributed to this source; the other named pairs carry separate citations in the citing sentence. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 126

  1. A. Rao, M. Pang, J. Kim, M. Kamineni, W. Lie, A. K. Prasad, A. Landman, K. Dreyer, and M. D. Succi, "Assessing the utility of chatgpt throughout the entire clinical workflow: development and usability study," Journal of Medical Internet Research, vol. 25, p. e48659, 2023.
  2. T. Zack, E. Lehman, M. Suzgun, J. A. Rodriguez, L. A. Celi, J. Gichoya, D. Jurafsky, P. Szolovits, D. W. Bates, R.-E. E. Abdulnour et al., "Assessing the potential of gpt-4 to perpetuate racial and gender biases in health care: a model evaluation study," The Lancet Digital Health, vol. 6, no. 1, pp. e12-e22, 2024.
  3. K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl et al., "Large language models encode clinical knowledge," Nature, vol. 620, no. 7972, pp. 172-180, 2023.
  4. R. S. Goodman, J. R. Patrinely, C. A. Stone, E. Zimmerman, R. R. Donald, S. S. Chang, S. T. Berkowitz, A. P. Finn, E. Jahangir, E. A. Scoville et al., "Accuracy and reliability of chatbot responses to physician questions," JAMA network open, vol. 6, no. 10, pp. e2 336 483-e2 336 483, 2023.
  5. H. Sun, K. Zhang, W. Lan, Q. Gu, G. Jiang, X. Yang, W. Qin, and D. Han, "An ai dietitian for type 2 diabetes mellitus management based on large language and image recognition models: Preclinical concept validation study," Journal of Medical Internet Research, vol. 25, p. e51300, 2023.
  6. K. Pushpanathan, Z. W. Lim, S. M. E. Yew, D. Z. Chen, H. A. H. Lin, J. H. L. Goh, W. M. Wong, X. Wang, M. C. J. Tan, V. T. C. Koh et al., "Popular large language model chatbots' accuracy, comprehensiveness, and self-awareness in answering ocular symptom queries," Iscience, vol. 26, no. 11, 2023.
  7. I. A. Bernstein, Y. V. Zhang, D. Govil, I. Majid, R. T. Chang, Y. Sun, A. Shue, J. C. Chou, E. Schehlein, K. L. Christopher et al., "Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions," JAMA Network Open, vol. 6, no. 8, pp. e2 330 320-e2 330 320, 2023.
  8. T. I. Wilhelm, J. Roos, and R. Kaczmarczyk, "Large language models for therapy recommendations across 3 clinical specialties: comparative study," Journal of Medical Internet Research, vol. 25, p. e49324, 2023.
  9. T. Kuroiwa, A. Sarcon, T. Ibara, E. Yamada, A. Yamamoto, K. Tsukamoto, and K. Fujita, "The potential of chatgpt as a self-diagnostic tool in common orthopedic diseases: exploratory study," Journal of Medical Internet Research, vol. 25, p. e47621, 2023.
  10. J. Chervenak, H. Lieman, M. Blanco-Breindel, and S. Jindal, "The promise and peril of using a large language model to obtain clinical information: Chatgpt performs strongly as a fertility counseling tool with limitations," Fertility and sterility, vol. 120, no. 3, pp. 575-583, 2023.
  11. I. Levkovich and Z. Elyoseph, "Suicide risk assessments through the eyes of chatgpt-3.5 versus chatgpt-4: vignette study," JMIR mental health, vol. 10, p. e51232, 2023.
  12. F. Liu, T. Zhu, X. Wu, B. Yang, C. You, C. Wang, L. Lu, Z. Liu, Y. Zheng, X. Sun et al., "A medical multimodal large language model for future pandemics," NPJ Digital Medicine, vol. 6, no. 1, p. 226, 2023.
  13. Z. W. Lim, K. Pushpanathan, S. M. E. Yew, Y. Lai, C.-H. Sun, J. S. H. Lam, D. Z. Chen, J. H. L. Goh, M. C. J. Tan, B. Sheng et al., "Benchmarking large language models' performances for myopia care: a comparative analysis of chatgpt-3.5, chatgpt-4.0, and google bard," EBioMedicine, vol. 95, 2023.
  14. P. Goktas, G. Karakaya, A. F. Kalyoncu, and E. Damadoglu, "Artificial intelligence chatbots in allergy and immunology practice: where have we been and where are we going?" The Journal of Allergy and Clinical Immunology: In Practice, vol. 11, no. 9, pp. 2697-2700, 2023.
  15. A. Rao, J. Kim, M. Kamineni, M. Pang, W. Lie, K. J. Dreyer, and M. D. Succi, "Evaluating gpt as an adjunct for radiologic decision making: Gpt-4 versus gpt-3.5 in a breast imaging pilot," Journal of the American College of Radiology, vol. 20, no. 10, pp. 990-997, 2023.
  16. A. Nicolson, J. Dowling, and B. Koopman, "Improving chest x-ray report generation by leveraging warm starting," Artificial intelligence in medicine, vol. 144, p. 102633, 2023.
  17. H. Fraser, D. Crossland, I. Bacher, M. Ranney, T. Madsen, R. Hilliard et al., "Comparison of diagnostic and triage accuracy of ada health and webmd symptom checkers, chatgpt, and physicians for patients in an emergency department: clinical data analysis study," JMIR mHealth and uHealth, vol. 11, no. 1, p. e49995, 2023.
  18. P. P. Suthar, A. Kounsal, L. Chhetri, D. Saini, and S. G. Dua, "Artificial intelligence (ai) in radiology: a deep dive into chatgpt 4.0's accuracy with the american journal of neuroradiology's (ajnr)"" case of the month""," Cureus, vol. 15, no. 8, 2023.
  19. C. Peng, X. Yang, A. Chen, K. E. Smith, N. PourNejatian, A. B. Costa, C. Martin, M. G. Flores, Y. Zhang, T. Magoc et al., "A study of generative large language model for medical research and healthcare," NPJ Digital Medicine, vol. 6, no. 1, p. 210, 2023.
  20. Q. Chen, H. Sun, H. Liu, Y. Jiang, T. Ran, X. Jin, X. Xiao, Z. Lin, H. Chen, and Z. Niu, "An extensive benchmark study on biomedical text generation and mining with chatgpt," Bioinformatics, vol. 39, no. 9, p. btad557, 2023.
  21. K. Mermin-Bunnell, Y. Zhu, A. Hornback, G. Damhorst, T. Walker, C. Robichaux, L. Mathew, N. Jaquemet, K. Peters, T. M. Johnson et al., "Use of natural language processing of patient-initiated electronic health record messages to identify patients with covid-19 infection," JAMA network open, vol. 6, no. 7, pp. e2 322 299- e2 322 299, 2023.
  22. E. Alsentzer, M. J. Rasmussen, R. Fontoura, A. L. Cull, B. Beaulieu-Jones, K. J. Gray, D. W. Bates, and V. P. Kovacheva, "Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models," NPJ Digital Medicine, vol. 6, no. 1, p. 212, 2023.
  23. D. S. Lituiev, B. Lacar, S. Pak, P. L. Abramowitsch, E. H. De Marchis, and T. A. Peterson, "Automatic extraction of social determinants of health from medical notes of chronic lower back pain patients," Journal of the American Medical Informatics Association, vol. 30, no. 8, pp. 1438-1447, 2023.
  24. T. M. Buonocore, C. Crema, A. Redolfi, R. Bellazzi, and E. Parimbelli, "Localizing in-domain adaptation of transformer-based biomedical language models," Journal of Biomedical Informatics, vol. 144, p. 104431, 2023.
  25. S. Nowak, D. Biesner, Y. Layer, M. Theis, H. Schneider, W. Block, B. Wulff, U. Attenberger, R. Sifa, and A. Sprinkart, "Transformer-based structuring of free-text radiology report databases," European Radiology, vol. 33, no. 6, pp. 4228-4236, 2023.
  26. I. de la Iglesia, M. Vivó, P. Chocrón, G. de Maeztu, K. Gojenola, and A. Atutxa, "An open source corpus and automatic tool for section identification in spanish health records," Journal of Biomedical Informatics, vol. 145, p. 104461, 2023.
  27. W. Hendrix, M. Rutten, N. Hendrix, B. van Ginneken, C. Schaefer-Prokop, E. T. Scholten, M. Prokop, and C. Jacobs, "Trends in the incidence of pulmonary nodules in chest computed tomography: 10-year results from two dutch hospitals," European Radiology, vol. 33, no. 11, pp. 8279-8288, 2023.
  28. J. Liu, D. Capurro, A. Nguyen, and K. Verspoor, "Attention-based multimodal fusion with contrast for robust clinical prediction in the face of missing modalities," Journal of Biomedical Informatics, vol. 145, p. 104466, 2023.
  29. D. Morena, J. Fernández, C. Campos, M. Castillo, G. López, M. Benavent, and J. L. Izquierdo, "Clinical profile of patients with idiopathic pulmonary fibrosis in real life," Journal of Clinical Medicine, vol. 12, no. 4, p. 1669, 2023.
  30. E. Guo, M. Gupta, J. Deng, Y.-J. Park, M. Paget, and C. Naugler, "Automated paper screening for clinical reviews using large language models: Data analysis study," Journal of Medical Internet Research, vol. 26, p. e48996, 2024.
  31. Q. Jin, W. Kim, Q. Chen, D. C. Comeau, L. Yeganova, W. J. Wilbur, and Z. Lu, "Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval," Bioinformatics, vol. 39, no. 11, p. btad651, 2023.
  32. Y. Wang, H. Zhao, S. Sciabola, and W. Wang, "cmolgpt: A conditional generative pre-trained transformer for target-specific de novo molecular generation," Molecules, vol. 28, no. 11, p. 4430, 2023.
  33. H. Li and B. Liu, "Bioseq-diabolo: Biological sequence similarity analysis using diabolo," PLoS computational biology, vol. 19, no. 6, p. e1011214, 2023.
  34. Y. Zhang, M. Lang, J. Jiang, Z. Gao, F. Xu, T. Litfin, K. Chen, J. Singh, X. Huang, G. Song et al., "Multiple sequence alignment-based rna language model and its application to structural inference," Nucleic Acids Research, vol. 52, no. 1, pp. e3-e3, 2024.
  35. L. Y. Jiang, X. C. Liu, N. P. Nejatian, M. Nasir-Moin, D. Wang, A. Abidin, K. Eaton, H. A. Riina, I. Laufer, P. Punjabi et al., "Health system-scale language models are all-purpose prediction engines," Nature, vol. 619, no. 7969, pp. 357-362, 2023.
  36. Y. Lu, X. Liu, Z. Du, Y. Gao, and G. Wang, "Medkpl: a heterogeneous knowledge enhanced prompt learning framework for transferable diagnosis," Journal of Biomedical Informatics, vol. 143, p. 104417, 2023.
  37. Z. Hussain, Z. Sheikh, A. Tahir, K. Dashtipour, M. Gogate, A. Sheikh, A. Hussain et al., "Artificial intelligence- enabled social media analysis for pharmacovigilance of covid-19 vaccinations in the united kingdom: observa- tional study," JMIR Public Health and Surveillance, vol. 8, no. 5, p. e32543, 2022.
  38. M. Májovskỳ, M. Černỳ, M. Kasal, M. Komarc, and D. Netuka, "Artificial intelligence can generate fraudulent but authentic-looking scientific medical articles: Pandora's box has been opened," Journal of medical Internet research, vol. 25, p. e46924, 2023.
  39. A. S. Doyal, D. Sender, M. Nanda, and R. A. Serrano, "Chatgpt and artificial intelligence in medical writing: concerns and ethical considerations," Cureus, vol. 15, no. 8, 2023.
  40. S. Suppadungsuk, C. Thongprayoon, P. Krisanapan, S. Tangpanithandee, O. Garcia Valencia, J. Miao, P. Mekrak- sakit, K. Kashani, and W. Cheungpasitporn, "Examining the validity of chatgpt in identifying relevant nephrology literature: findings and implications," Journal of Clinical Medicine, vol. 12, no. 17, p. 5550, 2023.
  41. A. J. Hueber and A. Kleyer, "Quality of citation data using the natural language processing tool chatgpt in rheumatology: creation of false references," RMD open, vol. 9, no. 2, p. e003248, 2023.
  42. L. Tang, Z. Sun, B. Idnay, J. G. Nestor, A. Soroush, P. A. Elias, Z. Xu, Y. Ding, G. Durrett, J. F. Rousseau et al., "Evaluating large language models on medical evidence summarization," npj Digital Medicine, vol. 6, no. 1, p. 158, 2023.
  43. M.-H. Temsah, I. Altamimi, A. Jamal, K. Alhasan, and A. Al-Eyadhy, "Chatgpt surpasses 1000 publications on pubmed: envisioning the road ahead," Cureus, vol. 15, no. 9, 2023.
  44. Y. Kaneda, R. Takahashi, U. Kaneda, S. Akashima, H. Okita, S. Misaki, A. Yamashiro, A. Ozaki, and T. Tanimoto, "Assessing the performance of gpt-3.5 and gpt-4 on the 2023 japanese nursing examination," Cureus, vol. 15, no. 8, 2023.
  45. A. Kumari, A. Kumari, A. Singh, S. K. Singh, A. Juhi, A. K. D. Dhanvijay, M. J. Pinjar, H. Mondal, and A. K. Dhanvijay, "Large language models in hematology case solving: a comparative study of chatgpt-3.5, google bard, and microsoft bing," Cureus, vol. 15, no. 8, 2023.
  46. S. W. Li, M. W. Kemp, S. J. Logan, P. S. Dimri, N. Singh, C. N. Mattar, P. Dashraath, H. Ramlal, A. P. Mahyuddin, S. Kanayan et al., "Chatgpt outscored human candidates in a virtual objective structured clinical examination in obstetrics and gynecology," American Journal of Obstetrics and Gynecology, vol. 229, no. 2, pp. 172-e1, 2023.
  47. D. Brin, V. Sorin, A. Vaid, A. Soroush, B. S. Glicksberg, A. W. Charney, G. Nadkarni, and E. Klang, "Comparing chatgpt and gpt-4 performance in usmle soft skill assessments," Scientific Reports, vol. 13, no. 1, p. 16492, 2023.
  48. R. Golan, S. J. Ripps, R. Reddy, J. Loloi, A. P. Bernstein, Z. M. Connelly, N. S. Golan, R. Ramasamy, S. Ripps, R. V. Reddy et al., "Chatgpt's ability to assess quality and readability of online medical information: evidence from a cross-sectional study," Cureus, vol. 15, no. 7, 2023.
  49. J. Mago and M. Sharma, "The potential usefulness of chatgpt in oral and maxillofacial radiology," Cureus, vol. 15, no. 7, 2023.
  50. A. K. D. Dhanvijay, M. J. Pinjar, N. Dhokane, S. R. Sorte, A. Kumari, H. Mondal, and A. K. Dhanvijay, "Performance of large language models (chatgpt, bing search, and google bard) in solving case vignettes in physiology," Cureus, vol. 15, no. 8, 2023.
  51. J. Alghamdi, Y. Lin, and S. Luo, "Towards covid-19 fake news detection using transformer-based models," Knowledge-Based Systems, vol. 274, p. 110642, 2023.
  52. E. Sezgin, F. Chekeni, J. Lee, and S. Keim, "Clinical accuracy of large language models and google search responses to postpartum depression questions: cross-sectional study," Journal of Medical Internet Research, vol. 25, p. e49240, 2023.
  53. J. Cadamuro, F. Cabitza, Z. Debeljak, S. De Bruyne, G. Frans, S. M. Perez, H. Ozdemir, A. Tolios, A. Carobene, and A. Padoan, "Potentials and pitfalls of chatgpt and natural-language artificial intelligence models for the understanding of laboratory medicine test results. an assessment by the european federation of clinical chemistry and laboratory medicine (eflm) working group on artificial intelligence (wg-ai)," Clinical Chemistry and Laboratory Medicine (CCLM), vol. 61, no. 7, pp. 1158-1166, 2023.
  54. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
  55. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
  56. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
  57. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., "Training language models to follow instructions with human feedback," Advances in neural information processing systems, vol. 35, pp. 27 730-27 744, 2022.
  58. P. P. Ray, "Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope," Internet of Things and Cyber-Physical Systems, 2023.
  59. W. Dai, J. Lin, H. Jin, T. Li, Y.-S. Tsai, D. Gašević, and G. Chen, "Can large language models provide feedback to students? a case study on chatgpt," in 2023 IEEE International Conference on Advanced Learning Technologies (ICALT). IEEE, 2023, pp. 323-325.
  60. C. Thapa, S. I. Jang, M. E. Ahmed, S. Camtepe, J. Pieprzyk, and S. Nepal, "Transformer-based language models for software vulnerability detection," in Proceedings of the 38th Annual Computer Security Applications Conference, 2022, pp. 481-496.
  61. S. I. Ross, F. Martinez, S. Houde, M. Muller, and J. D. Weisz, "The programmer's assistant: Conversational interaction with a large language model for software development," in Proceedings of the 28th International Conference on Intelligent User Interfaces, 2023, pp. 491-514.
  62. X. Zhang and Q. Yang, "Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters," in Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, 2023, pp. 4435-4439.
  63. E. Ferrara, "Should chatgpt be biased? challenges and risks of bias in large language models," arXiv preprint arXiv:2304.03738, 2023. · arXiv
  64. Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang et al., "A survey on evaluation of large language models," ACM Transactions on Intelligent Systems and Technology, 2023.
  65. Y. Cai, L. Wang, Y. Wang, G. de Melo, Y. Zhang, Y. Wang, and L. He, "Medbench: A large-scale chinese benchmark for evaluating medical large language models," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024, pp. 17 709-17 717.
  66. Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon, "Domain-specific language model pretraining for biomedical natural language processing," ACM Transactions on Computing for Healthcare (HEALTH), vol. 3, no. 1, pp. 1-23, 2021.
  67. W. Liu, J. Tang, Y. Cheng, W. Li, Y. Zheng, and X. Liang, "Meddg: an entity-centric medical consultation dataset for entity-aware medical dialogue generation," in CCF International Conference on Natural Language Processing and Chinese Computing. Springer, 2022, pp. 447-459.
  68. D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, "What disease does this patient have? a large-scale open domain question answering dataset from medical exams," Applied Sciences, vol. 11, no. 14, p. 6421, 2021.
  69. A. Pal, L. K. Umapathi, and M. Sankarasubbu, "Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering," in Conference on health, inference, and learning. PMLR, 2022, pp. 248-260.
  70. A. B. Abacha, C. Shivade, and D. Demner-Fushman, "Overview of the mediqa 2019 shared task on textual inference, question entailment and question answering," in Proceedings of the 18th BioNLP Workshop and Shared Task, 2019, pp. 370-379.
  71. A. B. Abacha, E. Agichtein, Y. Pinter, and D. Demner-Fushman, "Overview of the medical question answering task at trec 2017 liveqa." in TREC, 2017, pp. 1-12.
  72. W. Chen, Z. Li, H. Fang, Q. Yao, C. Zhong, J. Hao, Q. Zhang, X. Huang, J. Peng, and Z. Wei, "A benchmark for automatic medical consultation system: frameworks, tasks and datasets," Bioinformatics, vol. 39, no. 1, p. btac817, 2023.
  73. A. B. Abacha and D. Demner-Fushman, "On the summarization of consumer health questions," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 2228-2234.
  74. A. B. Abacha, Y. M'rabet, Y. Zhang, C. Shivade, C. Langlotz, and D. Demner-Fushman, "Overview of the mediqa 2021 shared task on summarization in the medical domain," in Proceedings of the 20th Workshop on Biomedical Language Processing, 2021, pp. 74-85.
  75. M. Savery, A. B. Abacha, S. Gayen, and D. Demner-Fushman, "Question-driven summarization of answers to consumer health questions," Scientific Data, vol. 7, no. 1, p. 322, 2020.
  76. K. Mrini, F. Dernoncourt, S. Yoon, T. Bui, W. Chang, E. Farcas, and N. Nakashole, "A gradually soft multi-task and data-augmented approach to medical question understanding," in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 1505-1515.
  77. Z. Hongying, L. Wenxin, Z. Kunli, Y. Yajuan, C. Baobao, and S. Zhifang, "Building a pediatric medical corpus: Word segmentation and named entity annotation," in Chinese Lexical Semantics: 21st Workshop, CLSW 2020, Hong Kong, China, May 28-30, 2020, Revised Selected Papers 21. Springer, 2021, pp. 652-664.
  78. T. Guan, H. Zan, X. Zhou, H. Xu, and K. Zhang, "Cmeie: construction and evaluation of chinese medical information extraction dataset," in Natural Language Processing and Chinese Computing: 9th CCF International Conference, NLPCC 2020, Zhengzhou, China, October 14-18, 2020, Proceedings, Part I 9. Springer, 2020, pp. 270-282.
  79. W. Zhu, X. Wang, M. Chen, and B. Tang, "Overview of the promptcblue shared task in chip2023," in China Health Information Processing Conference. Springer, 2023, pp. 3-20.
  80. H. Zong, J. Yang, Z. Zhang, Z. Li, and X. Zhang, "Semantic categorization of chinese eligibility criteria in clinical trials using machine learning methods," BMC Medical Informatics and Decision Making, vol. 21, pp. 1-12, 2021.
  81. M. Herrero-Zazo, I. Segura-Bedmar, P. Martínez, and T. Declerck, "The ddi corpus: An annotated corpus with pharmacological substances and drug-drug interactions," Journal of biomedical informatics, vol. 46, no. 5, pp. 914-920, 2013.
  82. J. Li, Y. Sun, R. J. Johnson, D. Sciaky, C.-H. Wei, R. Leaman, A. P. Davis, C. J. Mattingly, T. C. Wiegers, and Z. Lu, "Biocreative v cdr task corpus: a resource for chemical disease relation extraction," Database, vol. 2016, 2016.
  83. Y. Hou, Y. Xia, L. Wu, S. Xie, Y. Fan, J. Zhu, T. Qin, and T.-Y. Liu, "Discovering drug-target interaction knowledge from biomedical literature," Bioinformatics, vol. 38, no. 22, pp. 5100-5107, 2022.
  84. B. Nye, J. J. Li, R. Patel, Y. Yang, I. J. Marshall, A. Nenkova, and B. C. Wallace, "A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature," in Proceedings of the conference. Association for Computational Linguistics. Meeting, vol. 2018. NIH Public Access, 2018, p. 197.
  85. À. Bravo, J. Piñero, N. Queralt-Rosinach, M. Rautschka, and L. I. Furlong, "Extraction of relations between genes and diseases from text and large-scale data analysis: implications for translational research," BMC bioinformatics, vol. 16, pp. 1-17, 2015.
  86. L. Smith, L. K. Tanabe, R. J. n. Ando, C.-J. Kuo, I.-F. Chung, C.-N. Hsu, Y.-S. Lin, R. Klinger, C. M. Friedrich, K. Ganchev et al., "Overview of biocreative ii gene mention recognition," Genome biology, vol. 9, pp. 1-19, 2008.
  87. N. Collier, T. Ohta, Y. Tsuruoka, Y. Tateisi, and J.-D. Kim, "Introduction to the bio-entity recognition task at jnlpba," in Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP), 2004, pp. 73-78.
  88. M. Krallinger, O. Rabal, F. Leitner, M. Vazquez, D. Salgado, Z. Lu, R. Leaman, Y. Lu, D. Ji, D. M. Lowe et al., "The chemdner corpus of chemicals and drugs and its annotation principles," Journal of cheminformatics, vol. 7, pp. 1-17, 2015.
  89. R. I. Dogan, R. Leaman, and Z. Lu, "Ncbi disease corpus: a resource for disease name recognition and concept normalization," Journal of biomedical informatics, vol. 47, pp. 1-10, 2014.
  90. M. Gerner, G. Nenadic, and C. M. Bergman, "Linnaeus: a species name identification system for biomedical literature," BMC bioinformatics, vol. 11, pp. 1-17, 2010.
  91. E. Pafilis, S. P. Frankild, L. Fanini, S. Faulwetter, C. Pavloudi, A. Vasileiadou, C. Arvanitidis, and L. J. Jensen, "The species and organisms resources for fast and accurate identification of taxonomic names in text," PloS one, vol. 8, no. 6, p. e65390, 2013.
  92. M. Basaldella, F. Liu, E. Shareghi, and N. Collier, "Cometa: A corpus for medical entity linking in the social media," arXiv preprint arXiv:2010.03295, 2020. · arXiv
  93. S. Baker, I. Silins, Y. Guo, I. Ali, J. H "ogberg, U. Stenius, and A. Korhonen, "Automatic semantic classification of scientific literature according to the hallmarks of cancer," Bioinformatics, vol. 32, no. 3, pp. 432-440, 2016.
  94. Q. Chen, A. Allot, and Z. Lu, "Litcovid: an open database of covid-19 literature," Nucleic acids research, vol. 49, no. D1, pp. D1534-D1540, 2021.
  95. W. Sun, A. Rumshisky, and O. Uzuner, "Annotating temporal information in clinical narratives," Journal of biomedical informatics, vol. 46, pp. S5-S12, 2013.
  96. O. Taboureau, S. K. Nielsen, K. Audouze, N. Weinhold, D. Edsg "ard, F. S. Roque, I. Kouskoumvekaki, A. Bora, R. Curpan, T. S. Jensen et al., "Chemprot: a disease chemical biology database," Nucleic acids research, vol. 39, no. suppl_1, pp. D367-D372, 2010.
  97. L. Luo, P.-T. Lai, C.-H. Wei, C. N. Arighi, and Z. Lu, "Biored: a rich biomedical relation extraction dataset," Briefings in Bioinformatics, vol. 23, no. 5, p. bbac282, 2022.
  98. D. Polykovskiy, A. Zhebrak, B. Sanchez-Lengeling, S. Golovanov, O. Tatanov, S. Belyaev, R. Kurbanov, A. Artamonov, V. Aladinskiy, M. Veselov et al., "Molecular sets (moses): a benchmarking platform for molecular generation models," Frontiers in pharmacology, vol. 11, p. 565644, 2020.
  99. O. Prykhodko, S. V. Johansson, P.-C. Kotsias, J. Arús-Pous, E. J. Bjerrum, O. Engkvist, and H. Chen, "A de novo molecular generation method using latent vector based generative adversarial network," Journal of Cheminformatics, vol. 11, pp. 1-13, 2019.
  100. J. Sun, N. Jeliazkova, V. Chupakhin, J.-F. Golib-Dzib, O. Engkvist, L. Carlsson, J. Wegner, H. Ceulemans, I. Georgiev, V. Jeliazkov et al., "Excape-db: an integrated large scale dataset facilitating big data analysis in chemogenomics," Journal of cheminformatics, vol. 9, pp. 1-9, 2017.
  101. A. G. Murzin, S. E. Brenner, T. Hubbard, and C. Chothia, "Scop: a structural classification of proteins database for the investigation of sequences and structures," Journal of molecular biology, vol. 247, no. 4, pp. 536-540, 1995.
  102. N. Zhou, Y. Jiang, T. R. Bergquist, A. J. Lee, B. Z. Kacsoh, A. W. Crocker, K. A. Lewis, G. Georghiou, H. N. Nguyen, M. N. Hamid et al., "The cafa challenge reports improved protein function prediction and new functional annotations for hundreds of genes through experimental screens," Genome biology, vol. 20, pp. 1-23, 2019.
  103. H. Wei, Y. Xu, and B. Liu, "icircda-ltr: identification of circrna-disease associations based on learning to rank," Bioinformatics, vol. 37, no. 19, pp. 3302-3310, 2021.
  104. I. Kalvari, E. P. Nawrocki, N. Ontiveros-Palacios, J. Argasinska, K. Lamkiewicz, M. Marz, S. Griffiths-Jones, C. Toffano-Nioche, D. Gautheret, Z. Weinberg et al., "Rfam 14: expanded coverage of metagenomic, viral and microrna families," Nucleic Acids Research, vol. 49, no. D1, pp. D192-D200, 2021.
  105. K. Chen, T. Litfin, J. Singh, J. Zhan, and Y. Zhou, "The master database of all possible rna sequences and its integration with rnacmap for rna homology search," bioRxiv, pp. 2023-02, 2023.
  106. P. Brown, A.-C. Tan, M. A. El-Esawi, T. Liehr, O. Blanck, D. P. Gladue, G. M. Almeida, T. Cernava, C. O. Sorzano, A. W. Yeung et al., "Large expert-curated database for benchmarking document similarity detection in biomedical literature search," Database, vol. 2019, 2019.
  107. A. Cohan, S. Feldman, I. Beltagy, D. Downey, and D. S. Weld, "Specter: Document-level representation learning using citation-informed transformers," arXiv preprint arXiv:2004.07180, 2020. · arXiv
  108. G. Sogancıoglu, H. "Ozt "urk, and A. "Ozg "ur, "Biosses: a semantic sentence similarity estimation system for the biomedical domain," Bioinformatics, vol. 33, no. 14, pp. i49-i58, 2017.
  109. Y. Wang, N. Afzal, S. Fu, L. Wang, F. Shen, M. Rastegar-Mojarad, and H. Liu, "Medsts: a resource for clinical semantic textual similarity," Language Resources and Evaluation, vol. 54, pp. 57-72, 2020.
  110. G. Tsatsaronis, G. Balikas, P. Malakasiotis, I. Partalas, M. Zschunke, M. R. Alvers, D. Weissenborn, A. Krithara, S. Petridis, D. Polychronopoulos et al., "An overview of the bioasq large-scale biomedical semantic indexing and question answering competition," BMC bioinformatics, vol. 16, pp. 1-28, 2015.
  111. R. E. Riveros, A. Espinosa, P. M. M. Jiménez, and L. Martinez, "Studies in health technology and informatics," 2005. [Online]. Available: https://api.semanticscholar.org/CorpusID:59537299
  112. A. E. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark, and S. Horng, "Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports," Scientific data, vol. 6, no. 1, p. 317, 2019.
  113. D. Demner-Fushman, M. D. Kohli, M. B. Rosenman, S. E. Shooshan, L. Rodriguez, S. Antani, G. R. Thoma, and C. J. McDonald, "Preparing a collection of radiology examinations for distribution and retrieval," Journal of the American Medical Informatics Association, vol. 23, no. 2, pp. 304-310, 2016.
  114. M. Pavlova, N. Terhljan, A. G. Chung, A. Zhao, S. Surana, H. Aboutalebi, H. Gunraj, A. Sabri, A. Alaref, and A. Wong, "Covid-net cxr-2: An enhanced deep convolutional neural network design for detection of covid-19 cases from chest x-ray images," Frontiers in Medicine, vol. 9, p. 861680, 2022.
  115. J. P. Cohen, P. Morrison, L. Dao, K. Roth, T. Q. Duong, and M. Ghassemi, "Covid-19 image data collection: Prospective predictions are the future," arXiv preprint arXiv:2006.11988, 2020. · arXiv
  116. G. Liu, Y. Liao, F. Wang, B. Zhang, L. Zhang, X. Liang, X. Wan, S. Li, Z. Li, S. Zhang et al., "Medical-vlbert: Medical visual language bert for covid-19 ct report generation with alternate learning," IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 9, pp. 3786-3797, 2021.
  117. J. Liu, C. Wang, and S. Liu, "Utility of chatgpt in clinical practice," Journal of Medical Internet Research, vol. 25, p. e48568, 2023.
  118. H. Grewal, G. Dhillon, V. Monga, P. Sharma, V. S. Buddhavarapu, G. Sidhu, R. Kashyap et al., "Radiology gets chatty: the chatgpt saga unfolds," Cureus, vol. 15, no. 6, 2023.
  119. J. Manyika and S. Hsiao, "An overview of bard: an early experiment with generative ai," AI. Google Static Documents, vol. 2, 2023.
  120. C. P. Carrino, J. Llop, M. Pàmies, A. Gutiérrez-Fandiño, J. Armengol-Estapé, J. Silveira-Ocampo, A. Valencia, A. Gonzalez-Agirre, and M. Villegas, "Pretrained biomedical language models for clinical nlp in spanish," in Proceedings of the 21st Workshop on Biomedical Language Processing, 2022, pp. 193-199.
  121. H.-L. Li, Y.-H. Pang, and B. Liu, "Bioseq-blm: a platform for analyzing dna, rna and protein sequences based on biological language models," Nucleic acids research, vol. 49, no. 22, pp. e129-e129, 2021.
  122. A. Elnaggar, M. Heinzinger, C. Dallago, G. Rihawi, Y. Wang, L. Jones, T. Gibbs, T. Feher, C. Angerer, M. Steinegger et al., "Prottrans: Towards cracking the language of life's code through self-supervised deep learning and high performance computing. arxiv 2020," arXiv preprint arXiv:2007.06225, 2007. · arXiv
  123. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma et al., "Scaling instruction-finetuned language models," Journal of Machine Learning Research, vol. 25, no. 70, pp. 1-53, 2024.
  124. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," Journal of machine learning research, vol. 21, no. 140, pp. 1-67, 2020.
  125. J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, "Biobert: a pre-trained biomedical language representation model for biomedical text mining," Bioinformatics, vol. 36, no. 4, pp. 1234-1240, 2020.
  126. R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T.-Y. Liu, "Biogpt: generative pre-trained transformer for biomedical text generation and mining," Briefings in bioinformatics, vol. 23, no. 6, p. bbac409, 2022.

Cited by

Sorted by citations

Related

Authors
Huang, Y., Tang, K., Chen, M., & Wang, B.
Affiliations
  • South China Normal University (CN)
  • Chinese Academy of Sciences (CN)
  • Chinese University of Hong Kong, Shenzhen (CN)
ORCID
Tang, Keke
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2404.15777
arXiv
2404.15777
OpenAlex
W4395483680
Cited by
14
References
0
Citations by year
2026: 2 · 2025: 9 · 2024: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
large language models, LLM evaluation, diagnosis, clinical text processing, information retrieval, benchmark dataset, evaluation metrics, public health awareness, bioinformatics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review