CoChat
Sign Up
Cited by 143Topic ModelingArtificial Intelligence in Healthcare and EducationMachine Learning in Healthcare

Prompt Engineering Paradigms for Medical Applications: Scoping Review

Open access
Zaghir, Jamil, Naguib, Marco, Bjelogrlic, Mina, Névéol, Aurélie, Tannier, Xavier, Lovis, Christian·2024·Journal of Medical Internet Research
DOI 10.2196/60501PMC PMC11422740

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Prompt engineering, focusing on crafting effective prompts to large language models (LLMs), has garnered attention for its capabilities at harnessing the potential of LLMs. This is even more crucial in the medical domain due to its specialized terminology and language technicity. Clinical natural language processing applications must navigate complex language and ensure privacy compliance. Prompt engineering offers a novel approach by designing tailored prompts to guide models in exploiting clinically relevant information from complex medical texts. Despite its promise, the efficacy of prompt engineering in the medical domain remains to be fully explored. Objective: The aim of the study is to review research efforts and technical approaches in prompt engineering for medical applications as well as provide an overview of opportunities and challenges for clinical practice. Methods: Databases indexing the fields of medicine, computer science, and medical informatics were queried in order to identify relevant published papers. Since prompt engineering is an emerging field, preprint databases were also considered. Multiple data were extracted, such as the prompt paradigm, the involved LLMs, the languages of the study, the domain of the topic, the baselines, and several learning, design, and architecture strategies specific to prompt engineering. We include studies that apply prompt engineering–based methods to the medical domain, published between 2022 and 2024, and covering multiple prompt paradigms such as prompt learning (PL), prompt tuning (PT), and prompt design (PD). Results: We included 114 recent prompt engineering studies. Among the 3 prompt paradigms, we have observed that PD is the most prevalent (78 papers). In 12 papers, PD, PL, and PT terms were used interchangeably. While ChatGPT is the most commonly used LLM, we have identified 7 studies using this LLM on a sensitive clinical data set. Chain-of-thought, present in 17 studies, emerges as the most frequent PD technique. While PL and PT papers typically provide a baseline for evaluating prompt-based approaches, 61% (48/78) of the PD studies do not report any nonprompt-related baseline. Finally, we individually examine each of the key prompt engineering–specific information reported across papers and find that many studies neglect to explicitly mention them, posing a challenge for advancing prompt engineering research. Conclusions: In addition to reporting on trends and the scientific landscape of prompt engineering, we provide reporting guidelines for future studies to help advance research in the medical field. We also disclose tables and figures summarizing medical prompt engineering papers available and hope that future contributions will leverage these existing works to better advance the field.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/60501

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs have demonstrated strong performance across a wide range of biomedical knowledge and reasoning tasks.

    “LLMs have demonstrated strong performance across a wide range of biomedical knowledge and reasoning tasks, accelerating their adoption in health care workflows [1-6].”

    From Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model–Generated Medical Information: Empirical Study (2026) · cites this paper as [2]

    The Results (Emerging Trends; page unavailable) describe strong results on medical question answering and several biomedical NLP tasks, supporting examples of performance across different tasks. The Discussion (Prompt Engineering Techniques Effectiveness in Medical Research; page unavailable) cautions that results are not systematically in favor of LLM-based methods, so the unqualified claim of strong performance across a wide range overstates the review. The cited passages also do not establish that performance accelerated adoption in health care workflows. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 122 · Cited by 143

Research works printed but not linked: 115

  1. [1]Brown T Mann B Ryder N Subbiah M Kaplan JD Dhariwal P Neelakantan A Shyam P Sastry G Askell A Language models are few-shot learners 2020 Advances in Neural Information Processing Systems December 6, 2020 Virtual 1877 1901
  2. [2]Kojima T Gu SS Reid M Matsuo Y Iwasawa Y Large language models are zero-shot reasoners 2022 Advances in Neural Information Processing Systems November 28, 2022 New Orleans 22199 22213
  3. [4]White J Fu Q Hays S Sandborn M Olea C Gilbert H Elnashar A Spencer-Smith J Schmidt DC A prompt pattern catalog to enhance prompt engineering with ChatGPT ArXiv Preprint posted online on February 21, 2023 10.48550/arXiv.2302.11382 · DOI · Paper page
  4. [5]Névéol A Dalianis H Velupillai S Savova G Zweigenbaum P Clinical natural language processing in languages other than English: opportunities and challenges J Biomed Semantics 2018 9 1 12 10.1186/s13326-018-0179-8 29602312 10.1186/s13326-018-0179-8 29602312 PMC5877394 · DOI
  5. [6]Luccioni AS Rogers A Mind your language (model): fact-checking LLMs and their role in NLP research and practice ArXiv Preprint posted online on June 1, 2024 10.48550/arXiv.2308.07120 · DOI
  6. [7]Thirunavukarasu AJ Ting DSJ Elangovan K Gutierrez L Tan TF Ting DSW Large language models in medicine Nat Med 2023 29 8 1930 1940 10.1038/s41591-023-02448-8 37460753 10.1038/s41591-023-02448-8 37460753 · DOI · Paper page
  7. [10]Fries J Weber L Seelam N Altay G Datta D Garda S Kang S Su R Kusa W Cahyawijaya S BigBIO: a framework for data-centric biomedical natural language processing 2022 Advances in Neural Information Processing Systems November 28, 2022 New Orleans 25792 25806
  8. [11]Weisenthal SJ ChatGPT and post-test probability ArXiv Preprint posted online on July 20, 2024 10.48550/arXiv.2311.12188 · DOI
  9. [12]Li L Ning W ProBioRE: a framework for biomedical causal relation extraction based on dual-head prompt and prototypical network 2023 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) December 5, 2023 Istanbul, Turkiye 2071 2074
  10. [13]Li Z Belkadi S Micheletti N Han L Shardlow M Nenadic G Large language models and control mechanisms improve text readability of biomedical abstracts ArXiv Preprint posted online on March 16, 2024 10.48550/arXiv.2309.13202 · DOI
  11. [14]Li Q Yang X Wang H Liu L Wang Q Wang J Zhang Y Chu M Hu S Chen Y Shen Y Fan C Zhang W Zheng J From beginner to expert: modeling medical knowledge into general LLMs ArXiv Preprint posted online on January 7, 2024 10.48550/arXiv.2312.01040 · DOI
  12. [15]Ateia S Kruschwitz U Is ChatGPT a biomedical expert? 2023 Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023) September 18-21, 2023 Thessaloniki, Greece 73 90
  13. [16]Belyaeva A Cosentino J Hormozdiari F Eswaran K Shetty S Corrado G Carroll A McLean CY Furlotte NA Multimodal LLMs for health grounded in individual-specific data 2023 Machine Learning for Multimodal Healthcare Data July 29, 2023 Honolulu, Hawaii, United States 86 102 10.1007/978-3-031-47679-2_7 · DOI
  14. [17]Chen Q Sun H Liu H Jiang Y Ran T Jin X Xiao X Lin Z Chen H Niu Z An extensive benchmark study on biomedical text generation and mining with ChatGPT Bioinformatics 2023 39 9 btad557 10.1093/bioinformatics/btad557 37682111 7264174 37682111 PMC10562950 · DOI
  15. [18]Mollá D Large language models and prompt engineering for biomedical query focused multi-document summarisation ArXiv Preprint posted online on November 9, 2023 10.48550/arXiv.2311.05169 · DOI
  16. [20]Singhal K Azizi S Tu T Mahdavi SS Wei J Chung HW Scales N Tanwani A Cole-Lewis H Pfohl S Payne P Seneviratne M Gamble P Kelly C Babiker A Schärli N Chowdhery A Mansfield P Demner-Fushman D Agüera Y Arcas B Webster D Corrado GS Matias Y Chou K Gottweis J Tomasev N Liu Y Rajkomar A Barral J Semturs C Karthikesalingam A Natarajan V Large language models encode clinical knowledge Nature 2023 620 7972 172 180 10.1038/s41586-023-06291-2 37438534 10.1038/s41586-023-06291-2 37438534 PMC10396962 · DOI · Paper page
  17. [22]Lim S Schmälzle R Artificial intelligence for health message generation: an empirical study using a large language model (LLM) and prompt engineering Front Commun 2023 8 1129082 10.3389/fcomm.2023.1129082 · DOI
  18. [23]Wu YH Lin YJ Kao HY IKM_Lab at BioLaySumm Task 1: longformer-based prompt tuning for biomedical lay summary generation 2023 The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks July 13, 2023 Toronto, Canada 10.18653/v1/2023.bionlp-1.64 · DOI
  19. [24]Zhang W Chen C Wang J Liu J Ruan T A co-adaptive duality-aware framework for biomedical relation extraction Bioinformatics 2023 39 5 btad301 10.1093/bioinformatics/btad301 37220895 7176367 37220895 PMC10209527 · DOI
  20. [25]Chen P Wang J Lin H Zhao D Yang Z Few-shot biomedical named entity recognition via knowledge-guided instance generation and prompt contrastive learning Bioinformatics 2023 39 8 btad496 10.1093/bioinformatics/btad496 37549065 7238215 37549065 PMC10444965 · DOI
  21. [26]Luo R Sun L Xia Y Qin T Zhang S Poon H Liu T BioGPT: generative pre-trained transformer for biomedical text generation and mining Brief Bioinform 2022 23 6 bbac409 10.1093/bib/bbac409 36156661 6713511 36156661 · DOI · Paper page
  22. [27]Nori H King N McKinney SM Carignan D Horvitz E Capabilities of GPT-4 on medical challenge problems ArXiv Preprint posted online on April 12, 2023 10.48550/arXiv.2303.13375 · DOI · Paper page
  23. [28]Heinz MV Bhattacharya S Trudeau B Quist R Song SH Lee CM Jacobson NC Testing domain knowledge and risk of bias of a large-scale general artificial intelligence model in mental health Digit Health 2023 9 20552076231170499 10.1177/20552076231170499 37101589 10.1177_20552076231170499 37101589 PMC10123874 · DOI
  24. [29]Ting Y Hsieh T Wang Y Kuo Y Chen Y Chan P Kao C Performance of ChatGPT incorporated chain-of-thought method in bilingual nuclear medicine physician board examinations Digit Health 2024 10 20552076231224074 10.1177/20552076231224074 38188855 10.1177_20552076231224074 38188855 PMC10771043 · DOI · Paper page
  25. [30]Casola S Labruna T Lavelli A Magnini B Testing ChatGPT for stability and reasoning: a case study using Italian medical specialty tests 2023 Proceedings of the 9th Italian Conference on Computational Linguistics November 30-Decemeber 2, 2023 Venice, Italy
  26. [31]Roemer G Li A Mahmood U Dauer L Bellamy M Artificial intelligence model GPT4 narrowly fails simulated radiological protection exam J Radiol Prot 2024 44 1 013502 10.1088/1361-6498/ad1fdf 38232401 38232401 · DOI
  27. [32]Ali S Shahab O Al Shabeeb R Ladak F Yang JO Nadkarni G Echavarria J Babar S Shaukat A Soroush A General purpose large language models match human performance on gastroenterology board exam self-assessments MedRxi Preprint posted online on September 25, 2023 10.1101/2023.09.21.23295918 · DOI
  28. [33]Patel D Raut G Zimlichman E Cheetirala S Nadkarni G Glicksberg BS Freeman R Timsina P Klang E The limits of prompt engineering in medical problem-solving: a comparative analysis with ChatGPT on calculation based USMLE medical questions MedRxiv Preprint posted online on August 9, 2023 10.1101/2023.08.06.23293710 · DOI
  29. [34]Sallam M Al-Salahat K Eid H Egger J Puladi B Human versus artificial intelligence: ChatGPT-4 outperforming Bing, Bard, ChatGPT-3.5, and humans in clinical chemistry multiple-choice questions MedRxiv Preprint posted online on January 9, 2024 10.1101/2024.01.08.24300995 PMC11421444 39319062 · DOI
  30. [36]Kung TH Cheatham M Medenilla A Sillos C De Leon L Elepaño C Madriaga M Aggabao R Diaz-Candido G Maningo J Tseng V Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models PLOS Digit Health 2023 2 2 e0000198 10.1371/journal.pdig.0000198 36812645 PDIG-D-22-00371 36812645 PMC9931230 · DOI · Paper page
  31. [37]Tanaka Y Nakata T Aiga K Etani T Muramatsu R Katagiri S Kawai H Higashino F Enomoto M Noda M Kometani M Takamura M Yoneda T Kakizaki H Nomura A Performance of generative pretrained transformer on the national medical licensing examination in Japan PLOS Digit Health 2024 3 1 e0000433 10.1371/journal.pdig.0000433 38261580 PDIG-D-23-00146 38261580 PMC10805303 · DOI
  32. [38]Rosoł M Gąsior JS Łaba J Korzeniewski K Młyńczak M Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish medical final examination Sci Rep 2023 13 1 20512 10.1038/s41598-023-46995-z 10.1038/s41598-023-46995-z 37993519 10.1038/s41598-023-46995-z 37993519 PMC10665355 · DOI
  33. [39]Sivarajkumar S Kelley M Samolyk-Mazzanti A Visweswaran S Wang Y An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing ArXiv Preprint posted online on September 14, 2023 10.2196/55318 PMC11036183 38587879 · DOI
  34. [40]Agrawal M Hegselmann S Lang H Kim Y Sontag D Large language models are few-shot clinical information extractors 2022 Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing May 25, 2022 Abu Dhabi 1998 2022 10.18653/v1/2022.emnlp-main.130 · DOI · Paper page
  35. [41]Dong B Wang Z Li Z Duan Z Xu J Pan T Zhang R Liu N Li X Wang J Liu C Dong L Mao C Gao J Wang J Toward a stable and low-resource PLM-based medical diagnostic system via prompt tuning and MoE structure Sci Rep 2023 13 1 12595 10.1038/s41598-023-39543-2 10.1038/s41598-023-39543-2 37537202 10.1038/s41598-023-39543-2 37537202 PMC10400680 · DOI
  36. [42]Gutierrez KLT Viacrusis PML Bridging the gap or widening the divide: a call for capacity-building in artificial intelligence for healthcare in the Philippines JMUST 2023 7 2 1325 1334 10.35460/2546-1621.2023-0081 · DOI
  37. [43]Islam KS Nipu AS Madiraju P Deshpande P Autocompletion of chief complaints in the electronic health records using large language models 2023 2023 IEEE International Conference on Big Data (BigData) December 15-18, 2023 Sorrento, Italy 4912 4921 10.1109/bigdata59044.2023.10386778 · DOI
  38. [44]Meoni S Ryffel T De La Clergerie É Annotate French clinical data using large language model predictions 2023 2023 IEEE 11th International Conference on Healthcare Informatics (ICHI) June 26-29, 2023 Houston, TX, United States 550 557 10.1109/ichi57859.2023.00099 · DOI
  39. [45]Meoni S De la Clergerie E Ryffel T Large language models as instructors: a study on multilingual clinical entity extraction 2023 The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks July 2023 Toronto, Canada 178 190 10.18653/v1/2023.bionlp-1.15 · DOI
  40. [46]Wang X Yang Q LingX at ROCLING 2023 MultiNER-health task: intelligent capture of Chinese medical named entities by LLMs 2023 Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023) October 20-21, 2023 Taipei City, Taiwan
  41. [47]Yang Y Li X Wang H Guan Y Jiang J Modeling clinical thinking based on knowledge hypergraph attention network and prompt learning for disease prediction SSRN Preprint posted online on June 30, 2023 10.2139/ssrn.4496800 · DOI
  42. [48]Yao Z Jaafar A Wang B Zhu Y Yang Z Yu H Do physicians know how to prompt? The need for automatic prompt optimization help in clinical note generation ArXiv Preprint posted online on July 5, 2024 10.48550/arXiv.2311.09684 · DOI
  43. [49]van Zandvoort D Wiersema L Huibers T van Dulmen S Brinkkemper S Enhancing summarization performance through transformer-based prompt engineering in automated medical reporting ArXiv Preprint posted online on January 19, 2024 10.5220/0012422600003657 · DOI
  44. [50]Zhang B Mishra R Teodoro D DS4DH at MEDIQA-Chat 2023: leveraging SVM and GPT-3 prompt engineering for medical dialogue classification and summarization 2023 Proceedings of the 5th Clinical Natural Language Processing Workshop June 12, 2023 Toronto, Canada 536 545 10.18653/v1/2023.clinicalnlp-1.57 · DOI
  45. [51]Zhu W Wang X Chen M Tang B Overview of the PromptCBLUE Shared Task in CHIP2023 ArXiv Preprint posted online on December 29, 2023 10.48550/arXiv.2312.17522 · DOI
  46. [52]Caruccio L Cirillo S Polese G Solimando G Sundaramurthy S Tortora G Can ChatGPT provide intelligent diagnoses? A comparative study between predictive models and ChatGPT to define a new medical diagnostic bot Expert Syst Appl 2024 235 121186 10.1016/j.eswa.2023.121186 · DOI
  47. [53]Lee Y Chen C Chen C Lee C Chen P Wu C Dai H Unlocking the secrets behind advanced artificial intelligence language models in deidentifying Chinese-English mixed clinical text: development and validation study J Med Internet Res 2024 26 e48443 10.2196/48443 38271060 v26i1e48443 38271060 PMC10853853 · DOI
  48. [54]Bhaumik R Srivastava V Jalali A Ghosh S Chandrasekaran R Mindwatch: a smart cloud-based AI solution for suicide ideation detection leveraging large language models MedRxiv Preprint posted online on September 26, 2023 10.1101/2023.09.25.23296062 · DOI
  49. [55]Heston TF Safety of large language models in addressing depression Cureus 2023 15 10.7759/cureus.50729 PMC10727113 38111813 · DOI · Paper page
  50. [56]Grabb D The impact of prompt engineering in large language model performance: a psychiatric example J Med Artif Intell 2023 6 20 10.21037/jmai-23-71 · DOI
  51. [57]Santos WR Paraboni I Prompt-based mental health screening from social media text ArXiv Preprint posted online on May 11, 2024 10.5753/brasnam.2024.1879 · DOI
  52. [59]Xu X Yao B Dong Y Gabriel S Yu H Hendler J Ghassemi M Dey AK Wang D Mental-LLM: leveraging large language models for mental health prediction via online text data Proc ACM Interact Mob Wearable Ubiquitous Technol 2024 8 1 1 32 10.1145/3643540 PMC11806945 39925940 · DOI · Paper page
  53. [60]Chen S Wu M Zhu KQ Lan K Zhang Z Cui L LLM-empowered chatbots for psychiatrist and patient simulation: application and evaluation ArXiv Preprint posted online on May 23, 2023 10.48550/arXiv.2305.13614 · DOI
  54. [61]Qi H Zhao Q Li J Song C Zhai W Dan L Liu S Yu YJ Wang F Zou H Supervised learning and large language model benchmarks on mental health datasets: cognitive distortions and suicidal risks in Chinese social media ResearchSquare Preprint posted online on November 02, 2023 10.21203/rs.3.rs-3523508/v1 PMC12383806 40868395 · DOI
  55. [62]Sambath V Advancements of artificial intelligence in mental health applications?: A comparative analysis of ChatGPT 3.5 and ChatGPT 4 ResearchGate Preprint posted online on December, 2023 10.13140/RG.2.2.28713.36961 · DOI
  56. [63]Choi HS Song JY Shin KH Chang JH Jang B Developing prompts from large language model for extracting clinical information from pathology and ultrasound reports in breast cancer Radiat Oncol J 2023 41 3 209 216 10.3857/roj.2023.00633 37793630 roj.2023.00633 37793630 PMC10556835 · DOI
  57. [64]Lee DT Vaid A Menon KM Freeman R Matteson DS Marin MP Nadkarni GN Development of a privacy preserving large language model for automated data extraction from thyroid cancer pathology reports MedRxiv Preprint posted online on November 8, 2023 10.1101/2023.11.08.23298252 · DOI
  58. [65]Dennstädt F Hastings J Putora PM Vu E Fischer GF Süveg K Glatzer M Riggenbach E Hà HL Cihoric N Exploring capabilities of large language models such as ChatGPT in radiation oncology Adv Radiat Oncol 2024 9 3 101400 10.1016/j.adro.2023.101400 38304112 S2452-1094(23)00228-2 38304112 PMC10831180 · DOI
  59. [66]Zhu S Gilbert M Ghanem AI Siddiqui F Thind K Feasibility of using zero-shot learning in transformer-based natural language processing algorithm for key information extraction from head and neck tumor board notes Int J Radiat Oncol Biol Phys 2023 117 2 e500 10.1016/j.ijrobp.2023.06.1743 · DOI
  60. [67]Zhao X Zhang M Ma M Su C Wang M Qiao X Guo J Ma W HW-TSC at SemEval-2023 task 7: exploring the natural language inference capabilities of ChatGPT and pre-trained language model for clinical trial 2023 Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023) July 10, 2023 Toronto, Canada 1603 1608 10.18653/v1/2023.semeval-1.221 · DOI
  61. [68]Nazary F Deldjoo Y Di Noia T ChatGPT-HealthPrompt. Harnessing the power of XAI in prompt-based healthcare decision support using ChatGPT 2023 Artificial Intelligence. ECAI 2023 International Workshops September 30-October 4, 2023 Kraków, Poland 382 397 10.1007/978-3-031-50396-2_22 · DOI
  62. [69]Wang B Lai J Cao H Jin F Tang M Yao C Zhang P Enhancing real-world data extraction in clinical research: evaluating the impact of the implementation of large language models in hospital setting ResearchSquare Preprint posted online on November 29, 2023 10.21203/rs.3.rs-3644810/v2 10.21203/rs.3.rs-3644810/v2 · DOI
  63. [70]Mishra V Sarraju A Kalwani NM Dexter JP Evaluation of prompts to simplify cardiovascular disease information using a large language model MedRxiv Preprint posted online on November 9, 2023 10.1101/2023.11.08.23298225 PMC11074888 38648104 · DOI
  64. [71]Feng R Brennan KA Azizi Z Goyal J Pedron M Chang HJ Ganesan P Ruiperez-Campillo S Deb B Clopton PL Baykaner T Rogers AJ Narayan SM Optimizing ChatGPT to detect VT recurrence from complex medical notes Circulation 2023 148 Suppl 1 A16401 10.1161/circ.148.suppl_1.16401 · DOI
  65. [72]Chowdhury M Lim E Higham A McKinnon R Ventoura N He Y De Pennington N Can large language models safely address patient questions following cataract surgery? 2023 Proceedings of the 5th Clinical Natural Language Processing Workshop June 10, 2023 Toronto, Canada 131 137 10.18653/v1/2023.clinicalnlp-1.17 · DOI · Paper page
  66. [73]Kleinig O Gao C Kovoor JG Gupta AK Bacchi S Chan WO How to use large language models in ophthalmology: from prompt engineering to protecting confidentiality Eye (Lond) 2024 38 4 649 653 10.1038/s41433-023-02772-w 37798360 10.1038/s41433-023-02772-w 37798360 PMC10920651 · DOI
  67. [74]Arsenyan V Bughdaryan S Shaya F Small K Shahnazaryan D Large language models for biomedical knowledge graph construction: information extraction from EMR notes ArXiv Preprint posted online on December 9, 2023 10.48550/arXiv.2301.12473 · DOI
  68. [75]Kwon T Ong KT Kang D Moon S Lee JR Hwang D Sohn B Sim Y Lee D Yeo J Large language models are clinical reasoners: reasoning-aware diagnosis framework with prompt-generated rationales 2024 Proceedings of the Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence February 20, 2024 Vancouver 18417 18425 10.1609/aaai.v38i16.29802 · DOI
  69. [76]Wang C Liu S Li A Liu J Text dialogue analysis for primary screening of mild cognitive impairment: development and validation study J Med Internet Res 2023 25 e51501 10.2196/51501 38157230 v25i1e51501 38157230 PMC10787336 · DOI
  70. [77]Li J Wang L Chen X Deng X Wen H You M Liu W Are you asking GPT-4 medical questions properly?—Prompt engineering in consistency and reliability with evidence-based guidelines for ChatGPT-4: a pilot study ResearchSquare. Posted online on October 3, 2023 2023 1 20 10.21203/rs.3.rs-3336823/v1 · DOI
  71. [78]Zaidat B Lahoti YS Yu A Mohamed KS Cho SK Kim JS Artificially intelligent billing in spine surgery: an analysis of a large language model Global Spine J 2023 21925682231224753 10.1177/21925682231224753 38147047 38147047 PMC11877531 · DOI
  72. [79]Datta S Lee K Paek H Manion FJ Ofoegbu N Du J Li Y Huang LC Wang J Lin B Xu H Wang X AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models J Am Med Inform Assoc 2024 31 2 375 385 10.1093/jamia/ocad218 37952206 7413158 37952206 PMC10797270 · DOI
  73. [80]White R Peng T Sripitak P Rosenberg Johansen A Snyder M CliniDigest: a case study in large language model based large-scale summarization of clinical trial descriptions 2023 Proceedings of the 2023 ACM Conference on Information Technology for Social Good September 6-8, 2023 Lisbon, Portugal 10.1145/3582515.3609559 · DOI
  74. [82]Akinci D'Antonoli T Stanzione A Bluethgen C Vernuccio F Ugga L Klontzas ME Cuocolo R Cannella R Koçak B Large language models in radiology: fundamentals, applications, ethical considerations, risks, and future directions Diagn Interv Radiol 2024 30 2 80 90 10.4274/dir.2023.232417 37789676 37789676 PMC10916534 · DOI
  75. [83]Wiest IC Ferber D Zhu J van Treeck M Meyer SK Juglan SK Carrero ZI Paech D Kleesiek J Ebert MP Truhn D Kather JN From text to tables: a local privacy preserving large language model for structured information retrieval from medical documents MedRxiv Preprint posted online on December 8, 2023 10.1101/2023.12.07.23299648 · DOI
  76. [84]Hamed E Eid A Alberry M Exploring ChatGPT's potential in facilitating adaptation of clinical guidelines: a case study of diabetic ketoacidosis guidelines Cureus 2023 15 5 e38784 10.7759/cureus.38784 37303347 37303347 PMC10249915 · DOI
  77. [85]Leypold T Schäfer B Boos A Beier JP Can AI think like a plastic surgeon? Evaluating GPT-4's clinical judgment in reconstructive procedures of the upper extremity Plast Reconstr Surg Glob Open 2023 11 12 e5471 10.1097/GOX.0000000000005471 38093728 38093728 PMC10718352 · DOI
  78. [86]Yang J Liu C Deng W Wu D Weng C Zhou Y Wang K Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT Patterns (NY) 2024 5 1 100887 10.1016/j.patter.2023.100887 38264716 S2666-3899(23)00288-X PMC10801236 38264716 · DOI
  79. [87]Xiong L Zeng Q Deng W Luo W Liu R A novel approach to nursing clinical intelligent decision-making: integration of large language models and local knowledge bases ResearchSquare Preprint posted online on December 8, 2023 10.21203/rs.3.rs-3756467/v1 · DOI
  80. [88]Zeng Q Liu Y He P A medical question classification approach based on prompt tuning and contrastive learning 2023 The Thirty Fifth International Conference on Software Engineering and Knowledge Engineering (SEKE 2023) July 1-10, 2023 San Francisco, CA, United States 632 635 10.18293/seke2023-025 · DOI
  81. [89]Zhao D Yang Y Chen P Meng J Sun S Wang J Lin H Biomedical document relation extraction with prompt learning and KNN J Biomed Inform 2023 145 104459 10.1016/j.jbi.2023.104459 37531999 S1532-0464(23)00180-6 37531999 · DOI
  82. [90]Zhu T Qin Y Feng M Chen Q Hu B Xiang Y BioPRO: context-infused prompt learning for biomedical entity linking IEEE/ACM Trans Audio Speech Lang Process 2024 32 2023 374 385 10.1109/taslp.2023.3331149 · DOI
  83. [91]Liu C Zhang S Li C Zhao H CPK-Adapter: infusing medical knowledge into K-adapter with continuous prompt 2023 2023 8th International Conference on Intelligent Computing and Signal Processing (ICSP) April 21-23, 2023 Xi'an, China 1017 1023 10.1109/icsp58490.2023.10248750 · DOI
  84. [92]Yeh HS Lavergne T Zweigenbaum P Decorate the examples: a simple method of prompt design for biomedical relation extraction 2022 Proceedings of the Thirteenth Language Resources and Evaluation Conference August 13, 2024 Marseille, France 3780 3787
  85. [93]Su Z Yu X Chen P EPTQA: a Chinese medical prompt learning method based on entity pair type question answering SSRN 2023 24 10.2139/ssrn.4563840 · DOI
  86. [94]Xu H Zhang J Wang Z Zhang S Bhalerao M Liu Y Zhu D Wang S GraphPrompt: graph-based prompt templates for biomedical synonym prediction 2023 Proceedings of the Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence February 7, 2023 Washington, DC, United States 10576 10584 10.1609/aaai.v37i9.26256 · DOI
  87. [95]Chen T Stefanidis A Jiang Z Su J Improving biomedical claim detection using prompt learning approaches 2023 2023 IEEE 4th International Conference on Pattern Recognition and Machine Learning (PRML) August 4-6, 2023 Urumqi, China 369 376 10.1109/prml59573.2023.10348317 · DOI
  88. [96]Xu Z Chen Y Hu B Improving biomedical entity linking with cross-entity interaction 2023 Proceedings of the Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence February 7, 2023 Washington 13869 13877 10.1609/aaai.v37i11.26624 · DOI
  89. [97]Wang Y Wang Y Peng Z Zhang F Zhou L Yang F Medical text classification based on the discriminative pre-training model and prompt-tuning Digit Health 2023 9 1 14 10.1177/20552076231193213 37559830 10.1177_20552076231193213 PMC10408339 37559830 · DOI
  90. [98]Tian X Wang P Mao S Open-world biomedical knowledge probing and verification 2023 Proceedings of The 12th International Joint Conference on Knowledge Graphs (IJCKG-23) December 8-9, 2023 Tokyo, Japan
  91. [99]Lu K Potash P Lin X Sun Y Qian Z Yuan Z Naumann T Cai T Lu J Prompt discriminative language models for domain adaptation 2023 Proceedings of the 5th Clinical Natural Language Processing Workshop July 14, 2023 Toronto, Canada 247 258 10.18653/v1/2023.clinicalnlp-1.30 · DOI
  92. [100]Hu Y Chen Y Xu H Towards more generalizable and accurate sentence classification in medical abstracts with less data J Healthc Inform Res 2023 7 4 542 556 10.1007/s41666-023-00141-6 37927376 141 37927376 PMC10620359 · DOI
  93. [101]Taylor N Zhang Y Joyce DW Gao Z Kormilitzin A Nevado-Holgado A Clinical prompt learning with frozen language models IEEE Trans Neural Netw Learn Syst 2023 1 11 10.1109/TNNLS.2023.3294633 37566498 37566498 · DOI
  94. [102]Landi I Alleva E Valentine AA Lepow LA Charney AW Clinical text deduplication practices for efficient pretraining and improved clinical tasks ArXiv Preprint posted online on September 29, 2023 10.48550/arXiv.2312.09469 · DOI
  95. [103]Sivarajkumar S Wang Y HealthPrompt: a zero-shot learning paradigm for clinical natural language processing AMIA Annu Symp Proc 2022 2022 972 981 37128372 123 37128372 PMC10148337
  96. [104]Sivarajkumar S Wang Y Evaluation of healthprompt for zero-shot clinical text classification 2023 2023 IEEE 11th International Conference on Healthcare Informatics (ICHI) June 26-29, 2023 Houston, TX, United States 492 494 10.1109/ichi57859.2023.00081 · DOI
  97. [105]Zhang L Liu J Intent-aware prompt learning for medical question summarization 2022 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) December 6-8, 2022 Las Vegas, NV, United States 672 679 10.1109/bibm55620.2022.9995317 · DOI
  98. [106]Alleva E Landi I Shaw LJ Böttinger E Fuchs TJ Ensari I Keyword-optimized template insertion for clinical information extraction via prompt-based learning ArXiv Preprint posted online on October 31, 2023 10.48550/arXiv.2310.20089 · DOI
  99. [107]Cui Z Yu K Yuan Z Dong X Luo W Language inference-based learning for low-resource Chinese clinical named entity recognition using language model J Biomed Inform 2024 149 104559 10.1016/j.jbi.2023.104559 38056702 S1532-0464(23)00280-0 38056702 · DOI
  100. [108]Ahmed A Zeng X Xi R Hou M Shah SA MED-Prompt: a novel prompt engineering framework for medicine prediction on free-text clinical notes J King Saud Univ Comput Inf Sci 2024 36 2 1 17 10.1016/j.jksuci.2024.101933 · DOI
  101. [109]Lu Y Liu X Du Z Gao Y Wang G MedKPL: a heterogeneous knowledge enhanced prompt learning framework for transferable diagnosis J Biomed Inform 2023 143 104417 10.1016/j.jbi.2023.104417 37315832 S1532-0464(23)00138-7 37315832 · DOI
  102. [110]Cui Y Han L Nenadic G MedTem2.0: prompt-based temporal classification of treatment events from discharge summaries 2023 Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) July 10-12, 2023 Toronto, Canada 160 183 10.18653/v1/2023.acl-srw.27 · DOI
  103. [111]Wang Z Sun J PromptEHR: conditional electronic healthcare records generation with prompt learning 2022 Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics October 11, 2022 Abu Dhabi, United Arab Emirates 2873 2855 10.18653/v1/2022.emnlp-main.185 PMC11824924 39949499 · DOI
  104. [112]Wang S Tang L Majety A Rousseau JF Shih G Ding Y Peng Y Trustworthy assertion classification through prompting J Biomed Inform 2022 132 104139 10.1016/j.jbi.2022.104139 35811026 S1532-0464(22)00153-8 35811026 PMC9378721 · DOI
  105. [114]Kwon S Wang X Liu W Druhl E Sung ML Reisman JI Li W Kerns RD Becker W Yu H ODD: a benchmark dataset for the natural language processing based opioid related aberrant behavior detection 2024 Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies June 16, 2024 Mexico City 1 22 10.18653/v1/2024.naacl-long.244 PMC11368170 39224833 · DOI
  106. [115]Su J Zhang J Peng P Wang H EGDE: a framework for bridging the gap in medical zero-shot relation triplet extraction 2023 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) December 5-8, 2023 Istanbul, Turkiye 10.1109/bibm58861.2023.10385666 · DOI
  107. [116]Li Q Wang Y You T Lu Y BioKnowPrompt: incorporating imprecise knowledge into prompt-tuning verbalizer with biomedical text for relation extraction Inf Sci 2022 617 346 358 10.1016/j.ins.2022.10.063 · DOI
  108. [117]Peng C Yang X Chen A Yu Z Smith KE Costa AB Flores MG Bian J Wu Y Generative large language models are all-purpose text analytics engines: text-to-text learning is all your need J Am Med Inform Assoc 2024 09 01 31 9 1892 1903 10.1093/jamia/ocae078 38630580 7648661 38630580 PMC11339507 · DOI
  109. [119]Duan J Lu F Liu J MVP: optimizing multi-view prompts for medical dialogue summarization 2023 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) December 5-8, 2023 Istanbul, Turkiye 10.1109/bibm58861.2023.10385916 · DOI
  110. [120]Rohanian O Jauncey H Nouriborji M Kumar V Gonalves BP Kartsonaki C Merson L Clifton D Using bottleneck adapters to identify cancer in clinical notes under low-resource constraints 2023 The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks July 23, 2023 Toronto, Canada 6239 78 10.18653/v1/2023.bionlp-1.5 · DOI
  111. [121]Elfrink A Vagliano I Abu-Hanna A Calixto I Soft-prompt tuning to predict lung cancer using primary care free-text Dutch medical notes Artificial Intelligence in Medicine 2023 Cham Springer Nature Switzerland 193 198
  112. [122]Singh Rawat BP Yu H Parameter efficient transfer learning for suicide attempt and ideation detection 2022 Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI) September 7, 2022 Abu Dhabi, United Arab Emirates 108 115 10.18653/v1/2022.louhi-1.13 PMC12947409 41767630 · DOI
  113. [123]Xu S Wan X Hu S Zhou M Xu T Wang H Mi H COSSUM: towards conversation-oriented structured summarization for automatic medical insurance assessment 2022 Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining August 14-18, 2022 Washington, DC, United States 4248 4256 10.1145/3534678.3539116 · DOI
  114. [124]Shaitarova A Zaghir J Lavelli A Krauthammer M Rinaldi F Exploring the latest highlights in medical natural language processing across multiple languages: a survey Yearb Med Inform 2023 32 1 230 243 10.1055/s-0043-1768726 38147865 38147865 PMC10751112 · DOI
  115. [126]Ducel F Fort K Lejeune G Lepage Y Do we name the languages we study? The #BenderRule in LREC and ACL articles 2022 Proceedings of the Thirteenth Language Resources and Evaluation Conference June 20-25, 2022 Marseille, France 564 573

Authors
Zaghir, J., Naguib, M., Bjelogrlic, M., Névéol, A., Tannier, X., & Lovis, C.
Affiliations
  • University of Geneva (CH)
  • Centre National de la Recherche Scientifique (FR)
  • Inserm (FR)
Corresponding author
Zaghir, Jamil
ORCID
Venue
Journal of Medical Internet Research
Publisher
JMIR Publications Inc.
Published
2024-09-10
Volume
26
Pages
e60501
ISSN
1438-8871 (ISSN-L)
DOI
10.2196/60501
PMC
PMC11422740
OpenAlex
W4402407635
Cited by
143
References
122
Citations by year
2026: 75 · 2025: 63 · 2024: 5
FWCI
4.90
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Archive ouverte UNIGE (University of Geneva) (submittedVersion, cc-by)
  • HAL (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Science Foundation
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (180888)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
prompt engineering, large language models, medical natural language processing, prompt learning, prompt tuning, chain-of-thought prompting, clinical text mining, reporting guidelines, privacy compliance, ChatGPT
MeSH terms
Humans, Medical Informatics, Natural Language Processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review