CoChat
Sign Up
Cited by 52Artificial Intelligence in Healthcare and EducationTopic ModelingMultimodal Machine Learning Applications

Towards evaluating and building versatile large language models for medicine

Open access
Wu, Chaoyi, Qiu, Pengcheng, Liu, Jinxin, Gu, Hongfei, Li, Na, Zhang, Ya, Wang, Yanfeng, Xie, Weidi·2025·npj Digital Medicine
DOI 10.1038/s41746-024-01390-4PMC PMC11770143
Open this paper in CoChat — free accountOpen PDF

Abstract

In this study, we present MedS-Bench, a comprehensive benchmark to evaluate large language models (LLMs) in clinical contexts, MedS-Bench, spanning 11 high-level clinical tasks. We evaluate nine leading LLMs, e.g., MEDITRON, Llama 3, Mistral, GPT-4, Claude-3.5, etc. and found that most models struggle with these complex tasks. To address these limitations, we developed MedS-Ins, a large-scale instruction-tuning dataset for medicine. MedS-Ins comprises 58 medically oriented language corpora, totaling 5M instances with 19K instructions, across 122 tasks. To demonstrate the dataset's utility, we conducted a proof-of-concept experiment by performing instruction tuning on a lightweight, open-source medical language model. The resulting model, MMedIns-Llama 3, significantly outperformed existing models on various clinical tasks. To promote further advancements, we have made MedS-Ins fully accessible and invite the research community to contribute to its expansion. Additionally, we have launched a dynamic leaderboard for MedS-Bench, to track the development progress of medical LLMs.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces MedS-Bench, an 11-task clinical benchmark showing that leading LLMs struggle beyond multiple-choice QA, and MedS-Ins, a 5M-instance medical instruction-tuning dataset. Instruction tuning an 8B open-source medical model on MedS-Ins yields MMedIns-Llama 3, which outperforms GPT-4 and Claude-3.5 on many evaluated clinical tasks.

Key findings

  • MedS-Bench covers 11 high-level clinical task categories, 28 datasets, and 52 distinct tasks; most evaluated LLMs struggled on complex clinical tasks despite strong MCQA performance.
  • MedS-Ins contains 58 medically oriented language corpora, 5M instances, 19K instructions, 122 tasks, 19 task categories, and five text domains.
  • MMedIns-Llama 3 achieved average scores of 63.9 on multilingual MCQA, 46.82/48.38 BLEU/ROUGE on text summarization, 83.77 accuracy on information extraction, 34.43/37.47 on concept explanation, 46.90/34.54 on rationale, 79.29 F1 on NER, 98.47 on SEER treatment planning, 97.53 on DDXPlus diagnosis, 63.35 average on MIMIC4ED clinical outcome prediction, and 86.66 macro-F1 on HoC classification.
  • On fact verification and NLI, MMedIns-Llama 3 reached 97.08 on PubMedQA Answer Verification, 79.55 on PUBLICHEALTH Health Fact Verification, 86.71 on MedNLI discriminative entailment, and 23.52/25.17 BLEU/ROUGE on MedNLI generative entailment; it slightly trailed GPT-4 and Llama 3 on EMBS Justification Verification with 12.71/14.65.

Methods

  • MedS-Bench was built from 28 existing datasets reformatted into instruction-prompted QA with hand-crafted task definitions; nine mainstream LLMs were evaluated, using 3-shot prompting except MCQA, which used zero-shot, with around 1500 test cases sampled per benchmark due to proprietary-model costs.
  • MedS-Ins was constructed by filtering medical-related tasks from Super-NaturalInstructions and prompting existing BioNLP datasets into a unified free-form QA format, yielding 5M instances, 19K instructions, and 122 tasks.
  • MMedIns-Llama 3 was trained by supervised instruction tuning from the medical-knowledge-enhanced MMed-Llama 3 base model for 5 epochs on 32 Ascend910B for 383.5 hours, with max length 2048, global batch size 128, and learning rate 1e-5.

Limitations

  • The paper states MedS-Bench covers only 11 clinical tasks, omits some mainstream LLMs, and cannot replace final evaluation in actual clinical settings; MedS-Ins remains incomplete, may miss practical medical scenarios, and has not yet shown emergent generalization to unseen clinical tasks.
  • The paper states the work is primarily English and plans multilingual expansion; the model has not undergone extensive clinical validation or human-preference safety alignment.
  • Evident from the text: many baseline results are borrowed from other works rather than uniformly re-evaluated, evaluation used sampled test cases for cost reasons, and GPT-4 was excluded from rationale evaluation because the MMedBench rationale component was primarily constructed using GPT-4 outputs.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-024-01390-4

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 21 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    MedS-Ins is the first comprehensive instruction tuning dataset for medicine.

    “construct the first, comprehensive instruction tuning dataset for medicine, MedS-Ins.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 58 source datasets are of sufficient quality and diversity to produce a comprehensive medical instruction dataset.
  • Findingown data

    MMedIns-Llama 3 achieved the highest accuracy score as InternLM 2 on PubMedQA Answer Verification.

    “MMedIns-Llama 3 continues surpassing existing models, achieving the highest accuracy score as InternLM 2, excelling in PubMedQA Answer Verification and HealthFact Verification while in EMBS, MMedIns-Llama 3 slightly falls behind the GPT-4 and Llama 3 with 12.71/14.65 in BLEU and ROUGE, which we treat as future work for further improvement.”
    Quote in the paperSubstantial claim
  • Findingown data

    Most evaluated large language models struggle with complex clinical tasks, even when using few-shot prompting.

    “most models struggle with these complex tasks.”
    • Also stated in: § Introduction · hedged
      “Our findings indicate that even the most advanced LLMs struggle with complex clinical tasks, even when utilizing few-shot prompting”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Performance on MedS-Bench tasks is indicative of practical utility in real-world clinical scenarios.
  • Findingown data

    On rationale tasks, MMedIns-Llama 3 achieves the best overall average BLEU-1/ROUGE-1 scores of 46.90/34.54 across all languages.

    “our final model, MMedIns-Llama 3, demonstrated the best overall performance with average BLEU-1/ROUGE-1 scores of 46.90/34.54 across all languages”
    Quote in the paperSubstantial claim
  • Findingown data

    Qwen 2 achieved a macro-F1 score of 40.29 on the HoC multi-label classification task.

    “Llama 3 and Qwen 2 show moderate performance, with macro-F1 scores of 38.37 and 40.29, respectively.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 22
  • Classified as probably not a substantial claim. 9

  • May combine two claims.

    Findingown data

    Med42-v2 achieved 79.54 on PUBLICHEALTH, the best among open-source models, just behind GPT-4 with 78.60.

    “On PUBLICHEALTH, Med42-v2 achieves 79.54, which is the best among all the open-source models, just behind GPT-4 with 78.60.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MMedIns-Llama 3 achieved the highest scores across all metrics on the HoC multi-label classification task, with a macro-Precision of 89.59, a macro-Recall of 85.58, and a macro-F1 score of 86.66.

    “Our MMedIns-Llama 3 clearly outperforms all other models, achieving the highest scores across all metrics, with a macro-Precision of 89.59, a macro-Recall of 85.58, and a macro-F1 score of 86.66.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MedS-Ins tokens are distributed across five text domains: exams 7%, clinical texts 35%, academic papers 13%, medical knowledge bases 43%, and daily conversations 2%.

    “7% of the tokens in our dataset are from the exams.”
    • Also stated in: § Results · plain
      “35% of the tokens in our dataset are from clinical texts.”
    • Also stated in: § Results · plain
      “43% of the tokens in our dataset are from medical knowledge.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MMedIns-Llama 3 excelled in the MedNLI generative task, with BLEU/ROUGE scores of 23.52/25.17, outperforming other models except GPT-4.

    “MMedIns-Llama 3 also excels in the generative task, with BLEU/ROUGE scores of 23.52/25.17, outperforming other models except the GPT-4.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On medical concept explanation, MMedIns-Llama 3 achieves the highest average BLEU/ROUGE scores of 34.43/37.47, significantly outperforming other models.

    “Our final model, MMedIns-Llama 3, significantly outperforms the other ones across all concept explanation tasks, particularly in Health Fact Explanation (30.50/28.53) and BioLORD Concept Explanation (38.12/43.90), and achieving the highest average scores of 34.43/37.47.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    MedS-Bench currently covers only 11 clinical tasks, which does not fully encompass the complexity of all clinical scenarios.

    “First, MedS-Bench currently covers only 11 clinical tasks, which does not fully encompass the complexity of all clinical scenarios.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MedS-Bench is a comprehensive benchmark for evaluating large language models in clinical contexts, extending beyond multiple-choice question answering to include 11 high-level clinical task categories derived from 28 existing datasets.

    “we present MedS-Bench, a comprehensive benchmark to evaluate large language models (LLMs) in clinical contexts, MedS-Bench, spanning 11 high-level clinical tasks.”
    • Also stated in: § Results · plain
      “MedS-Bench encompasses 11 high-level clinical task categories, derived from 28 existing datasets”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The selected 11 clinical task categories in MedS-Bench adequately represent real-world clinical demands.

  • May combine two claims.

    Findingown data

    InternLM 2 exhibited high recall but struggled with precision on the HoC multi-label classification task, resulting in a lower F1 score.

    “These models, especially InternLM 2, exhibit high recall but struggle with precision, resulting in lower F1 scores.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MedS-Ins has 19 task categories, including the 11 categories in MedS-Bench and additional categories such as Intent Identification, Translation, Word Relation Classification, Text Retrieval, Sentence Composition Analysis, Wrong Candidate Generation, Dialogue, and Text Completion, with MCQA extended to general Question Answering.

    “These 19 task categories include but are not limited to, the 11 categories in the MedS-Bench benchmark.”
    • Also stated in: § Results · plain
      “The additional categories encompass a range of linguistic and analytical tasks essential for comprehensive medical language processing, including Intent Identification, Translation, Word Relation Classification, Text Retrieval, Sentence Composition Analysis, Wrong Candidate Generation, Dialogue, and Text Completion and the MCQA category is extended to general Question Answering, which also include”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Eight of the task categories require the model to recall knowledge from the model, while the remaining three require fact retrieval from the given context.

    “eight of the task categories require the model to recall knowledge from the model, while the remaining three require fact retrieval from the given context.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 and Claude-3.5 achieved accuracy scores of 86.63 and 82.14 respectively on the MedNLI discriminative task.

    “For the closed-source LLMs, GPT-4 and Claude-3.5 all show relatively high scores, with 86.63 and 82.14 accuracy scores respectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On named entity recognition tasks, MMedIns-Llama 3 achieves an average F1-Score of 79.29, significantly outperforming all other models.

    “Our model, MMedIns-Llama 3, significantly outperforms all other models, achieving an average F1-Score of 79.29.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On information extraction, MMedIns-Llama 3 achieves the best overall average score of 83.77, surpassing InternLM 2 by 4.66 points.

    “Our proposed model, MMedIns-Llama 3, demonstrates the best overall performance, achieving an average score of 83.77, surpassing InternLM 2 by 4.66 points.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MMedIns-Llama 3 achieved the highest accuracy in the MedNLI discriminative task, scoring 86.71, comparable with GPT-4.

    “MMedIns-Llama 3 achieves the highest accuracy in the discriminative task, scoring 86.71, comparable with GPT-4.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On text summarization, MMedIns-Llama 3 achieves average BLEU/ROUGE scores of 46.82/48.38, significantly outperforming other models.

    “Our model (MMedIns-Llama 3), trained on the medical-specific instruction dataset (MedS-Ins), significantly outperforms the others, achieving average scores of 46.82/48.38.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On clinical decision support tasks, MMedIns-Llama 3 achieves 98.47 accuracy in treatment planning, 97.53 in diagnosis, and an average accuracy of 63.35 in clinical outcome prediction.

    “MMedIns-Llama 3 demonstrate superior accuracy in clinical decision support tasks, with a 98.47 accuracy score in treatment planning, 97.53 in diagnosis, and an average accuracy of 63.35 (mean on the scores of Hospitalization, 72h ED Revisit, and Critical Triage) in clinical outcome prediction.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingown data

    Open-source models struggle to complete zero-shot evaluation.

    “we evaluate all models in few-shot settings, as we observe that open-source models struggle to complete zero-shot evaluation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    By filtering instruction datasets in the general domain, they collected 37 tasks, for a total of 75373 samples.

    “Finally, by filtering instruction datasets in the general domain, we collect 37 tasks, for a total of 75373 samples.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Through the procedure, they prompted an extra 85 tasks into a unified free-form question-answering format, combined with filtered data, resulting in a total of 5M instances with 19K instructions, covering 122 tasks, termed MedS-Ins.

    “Through the above procedure, we prompt an extra 85 tasks into a unified free-form question-answering format, combined with the filtered data, resulting in a totaling 5M instances with 19K instructions, covering 122 tasks, termed as MedS-Ins”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Large language models have achieved high scores on multiple-choice question-answering benchmarks in healthcare and have successfully passed the UMLS examination.

    “LLMs have achieved high scores on multiple-choice question-answering (MCQA) benchmarks in healthcare, and successfully passed the UMLS examination, as noted by Singhal et al.1,2.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Singhal et al.1,2

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Around 1500 test cases per benchmark were randomly sampled to manage cost constraints.

    “we randomly sampled around 1500 test cases per benchmark to manage the cost constraints.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The randomly sampled around 1500 test cases per benchmark are representative of the full benchmark.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    The diversity of task categories in MedS-Ins guarantees a comprehensive understanding of the potential medical applications.

    “The diversity of task categories ranging from common question answering and dialogue to various downstream clinical tasks—guarantees a comprehensive understanding of the potential medical applications.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 58 source datasets are of sufficient quality and diversity to produce a comprehensive medical instruction dataset.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The significant proportion of clinical texts in MedS-Ins ensures that the instruction tuning data aligns closely with clinical demands.

    “Notably, the significant proportion of clinical texts ensures that the instruction tuning data aligns closely with clinical demands.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 58 source datasets are of sufficient quality and diversity to produce a comprehensive medical instruction dataset.

  • May combine two claims.

    Interpretationown data

    Existing LLMs perform exceptionally well on MCQA benchmarks but struggle to align with actual clinical practice, particularly in tasks such as treatment planning and explanation.

    “Our results reveal that while existing LLMs perform exceptionally well on MCQA benchmarks, they struggle to align with the actual clinical practice, particularly in tasks such as treatment planning and explanation.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    All results were obtained using a 3-shot prompting strategy except for MCQA tasks, where a zero-shot prompting setting was used to align with previous studies.

    “All results presented here were obtained using a 3-shot prompting strategy (more details in the “Evaluation Settings” section in Supplementary), except for MCQA tasks, where we used a zero-shot prompting setting to align with previous studies7,8,19.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [7] Wu, C. et al. Pmc-llama: toward building open-source language models for medicine. Journal of the American Medical Informatics Association ocae045 (2024). source

      Not checked in the free run.

    • cites [8] Chen, Z. et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079 (2023).

      Not checked in the free run.

    • cites [19] Qiu, P. et al. Towards building multilingual language model for medicine. Nature Communications 15.1 (2024): 8384. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Large language models have shown expert-level performance in clinical text summarization when appropriate prompting strategies are employed.

    “LLMs have shown expert-level performance in clinical text summarization when appropriate prompting strategies are employed3.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3] Van Veen, D. et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat. Med. 30, 1134–1142 (2024). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    MedS-Ins has been shown to significantly improve LLMs on clinical tasks.

    “which has shown to significantly improve the LLMs on clinical tasks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Performance on MedS-Bench tasks reflects clinical capabilities.

  • May combine two claims.

    Premisecitation

    MEDITRON is a large-scale medical LLM, further pre-trained on Llama 2, leveraging 21.1M medical papers, guidelines for further pre-training, and supervised finetuning on different MCQA datasets with context and chain-of-thought prompt styles.

    “MEDITRON8 is a large-scale medical LLM, further pre-trained on Llama 2. It leverages 21.1M medical papers, guidelines for further pre-training, and supervised finetuning on different MCQA datasets with context and chain-of-thought prompt styles.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8] Chen, Z. et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079 (2023).

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The benchmark cannot replace the final stage of evaluating LLMs in actual clinical settings to ensure their safety.

    “while our benchmark is more comprehensive and clinically relevant than previous MCQA benchmarks, it cannot replace the final stage of evaluating LLMs in actual clinical settings to ensure their safety.”
    Quote in the paperSubstantial claim
29 of 29

  • The paper relies on this without support.

    The selected 11 clinical task categories in MedS-Bench adequately represent real-world clinical demands.

    “Current benchmarks, which largely focus on multiple-choice problems2,7,8, fail to adequately reflect the practical utility of LLMs in real-world clinical scenarios.”
    “MedS-Bench encompasses 11 high-level clinical task categories, derived from 28 existing datasets”

  • The paper relies on this without support.

    The randomly sampled around 1500 test cases per benchmark are representative of the full benchmark.

    “we randomly sampled around 1500 test cases per benchmark to manage the cost constraints.”

  • The paper relies on this without support.

    Performance on MedS-Bench tasks is indicative of practical utility in real-world clinical scenarios.

    “This benchmark provides clinicians and researchers with a detailed understanding of where LLMs excel and where they fall short in medical tasks.”

  • The paper relies on this without support.

    The 58 source datasets are of sufficient quality and diversity to produce a comprehensive medical instruction dataset.

    “MedS-Ins comprises 58 medically oriented language corpora, totaling 5M instances with 19K instructions, across 122 tasks.”

  • The paper relies on this without support.

    Existing medical LLM benchmarks are narrow (focused on MCQA) and risk overlooking broader clinical capabilities.

    “The development of medical LLMs has largely relied on benchmarks focused on multiple-choice question answering (MCQA). However, this narrow evaluation framework risks overlooking the broader capabilities required for LLMs in various clinical scenarios.”

  • The paper relies on this without support.

    Previous instruction tuning datasets neglected texts generated from real clinical practice.

    “Previous efforts have focused primarily on constructing question-answer pairs from daily conversations, exams, or academic papers, often neglecting the texts generated from real clinical practice.”

  • The paper relies on this without support.

    Performance on MedS-Bench tasks reflects clinical capabilities.

    “our benchmark is expected to serve as an experimental arena for assessing the performance of different LLMs, offering a more accurate reflection of their clinical capabilities”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • These frameworks typically focus on narrow clinical scenarios (1-28 datasets), rely on limited data sources or synthetic data, and assess primarily accuracy without comprehensive taxonomy validation or deployment considerations.

    “However, these frameworks typically focus on narrow clinical scenarios (1–28 datasets), rely on limited data sources or synthetic data, and assess primarily accuracy without comprehensive taxonomy validation or deployment considerations.”

    From Holistic evaluation of large language models for medical tasks with MedHELM (2026) · cites this paper as [19]

  • Recent medical benchmarking efforts have begun addressing some limitations: HEALTHBENCH incorporates physician-developed rubrics, ClinicBench advances multimodal evaluation, and automated frameworks explore LLM-as-a-Judge strategies for EHR summarization.

    “Recent medical benchmarking efforts (Extended Data Table 8) have begun addressing some limitations: HEALTHBENCH10 incorporates physician-developed rubrics for health conversations, ClinicBench18,19 advances multimodal evaluation, and automated frameworks explore LLM-as-a-Judge strategies for EHR summarization19–26.”

    From Holistic evaluation of large language models for medical tasks with MedHELM (2026) · cites this paper as [19]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Mistral's MCQA average does not match its eight scores

    Minor

    The eight reported accuracies average 59.625, not 49.1.

    Reported 49.1 · recomputed 59.625

  • Med42-v2's PUBLICHEALTH score and ranking conflict with Table 8

    Minor

    The prose reports 79.54 and calls it the best open-source result. Table 8 instead reports 78.54 for Med42-v2 and 79.55 for MMedIns-Llama 3.

  • The benchmark lists 12 task categories but says it has 11

    Minor

    The enumerated list contains 12 categories, counting Explanation and Rationale separately, whereas the text says MedS-Bench encompasses 11.

  • InternLM 2 does not lead intervention extraction

    Minor

    Although InternLM 2 scores 74.42 on intervention extraction, Table 3 gives Llama 3 a higher score of 79.07.

  • MMedIns-Llama 3 does not match InternLM 2 on PubMedQA verification

    Minor

    The prose describes its accuracy as the highest, alongside InternLM 2, but Table 8 reports 97.08 for MMedIns-Llama 3 versus 99.23 for InternLM 2.

5 of 5

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 20 · Cited by 52

Research works printed but not linked: 6

  1. Wang, Y. et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022 (2022). · DOI · Paper page
  2. Longpre, S. et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning (pp. 22631-22648). PMLR.
  3. Tom, K. et al. Findings of the 2023 conference on machine translation (wmt23): Llms are here but not quite there yet. In WMT23-Eighth Conference on Machine Translation, 198–216 (2023).
  4. Zhang, W., Aljunied, M., Gao, C., Chia, Y. K. & Bing, L. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. Adv. Neural Inf. Process. Syst. 36, 5484–5505 (2023).
  5. Zhou, C. et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems 36 (2024).
  6. Lu, K. et al. # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Representations (2023).

Authors
Wu, C., Qiu, P., Liu, J., Gu, H., Li, N., Zhang, Y., Wang, Y., & Xie, W.
Affiliations
  • Shanghai Jiao Tong University (CN)
  • China Mobile (China) (CN)
Corresponding author
  • Wang, Yanfeng
  • Xie, Weidi
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2025-01-27
Volume
8
Issue
1
Pages
58-58
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-024-01390-4
PMC
PMC11770143
OpenAlex
W4406829669
Cited by
52
References
20
Citations by year
2026: 37 · 2025: 15
FWCI
12.52
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
  • Science and Technology Commission of Shanghai Municipality (18DZ2270700)
  • 111 plan (BP0719010)
  • State Key Laboratory of UHD Video and Audio Production and Presentation
  • National Key R&D Program of China (2022ZD0160702)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, clinical language models, benchmark, instruction tuning, clinical tasks, leaderboards

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review