MedExQA: Medical Question Answering Benchmark with Multiple Explanations
Open accessAbstract
This paper introduces MedExQA, a novel benchmark in medical question-answering, to evaluate large language models' (LLMs) understanding of medical knowledge through explanations. By constructing datasets across five distinct medical specialties that are underrepresented in current datasets and further incorporating multiple explanations for each question-answer pair, we address a major gap in current medical QA benchmarks which is the absence of comprehensive assessments of LLMs' ability to generate nuanced medical explanations. Our work highlights the importance of explainability in medical LLMs, proposes an effective methodology for evaluating models beyond classification accuracy, and sheds light on one specific domain, speech language pathology, where current LLMs including GPT4 lack good understanding. Our results show generation evaluation with multiple explanations aligns better with human assessment, highlighting an opportunity for a more robust automated comprehension assessment for LLMs. To diversify open-source medical LLMs (currently mostly based on Llama2), this work also proposes a new medical model, MedPhi-2, based on Phi-2 (2.7B). The model outperformed medical LLMs based on Llama2-70B in generating explanations, showing its effectiveness in the resource-constrained medical domain. We will share our benchmark datasets and the trained model.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Benchmarks frequently allow lenient scoring or use multiple expert-generated references, as in MedExQA's ensemble of clinician explanations.
“benchmarks frequently allow lenient scoring or use multiple expert-generated references, as in MedExQA's ensemble of clinician explanations (Kim et al., 2024).”
From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Kim et al., 2024
Section 3.1 (page unavailable) says MedExQA provides two explanations per question, drawn in part from online tests and exams and then human-validated. It does not identify them as clinician-generated. The Introduction (page unavailable) says existing benchmarks often lack explanations, so this paper does not support the broader claim that benchmarks frequently use multiple expert-generated references or lenient scoring. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe limitations of ROUGE/BLEU are acknowledged.
“ROUGE/BLEU are used for comparing generated text with reference comparison, but their limitations are acknowledged (Kim et al., 2024).”
From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Kim et al., 2024
Section 4.4 (page unavailable) says BLEU and ROUGE compare generated explanations with reference explanations, supporting their use for reference-based comparison. Section 5.4 reports that human evaluations contrasted with other evaluation methods, but the paper does not explicitly identify limitations of ROUGE or BLEU; its Limitation section addresses the benchmark and the small scale of human evaluation instead. (LLM, conf=0.82)
Experimental — accuracy not yet measuredBenchmarks such as MedExQA explicitly require models to provide explanations, comparing them against multiple ground-truth explanations using lexical metrics.
“Benchmarks such as MedExQA (Kim et al., 2024) explicitly require models to provide explanations, comparing them against multiple ground-truth explanations using lexical metrics (e.g., BLEU/ROUGE).”
From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Kim et al., 2024
Human evaluation remains crucial in many benchmarks.
“Human evaluation remains crucial in many benchmarks -MultiMedQA's 12-axis rubric is administered by clinicians to rate each answer qualitatively (Singhal et al., 2025), and MedExQA involves human scoring of explanation correctness (Kim et al., 2024).”
From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Kim et al., 2024
MedExQA involves human scoring of explanation correctness.
“Human evaluation remains crucial in many benchmarks -MultiMedQA's 12-axis rubric is administered by clinicians to rate each answer qualitatively (Singhal et al., 2025), and MedExQA involves human scoring of explanation correctness (Kim et al., 2024).”
From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Kim et al., 2024
ROUGE/BLEU are used for comparing generated text with reference comparison.
“ROUGE/BLEU are used for comparing generated text with reference comparison, but their limitations are acknowledged (Kim et al., 2024).”
From Trustworthy Medical Question Answering: An Evaluation-Centric Survey (2025) · cites this paper as Kim et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 21
- Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
- Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tion, pages 65-72.
- Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676. · arXiv
- Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. 2023. Meditron-70b: Scaling medical pretraining for large language mod- els. arXiv preprint arXiv:2311.16079. · arXiv
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805. · arXiv
- Emily Groves, Minhong Wang, Yusuf Abdulle, Holger Kunz, Jason Hoelscher-Obermaier, Ronin Wu, and Honghan Wu. 2023. Benchmarking and analyzing in- context learning, fine-tuning and supervised learning for biomedical knowledge curation: a focused study on chemical entities of biological interest. arXiv preprint arXiv:2312.12989. · arXiv
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language under- standing. arXiv preprint arXiv:2009.03300. · arXiv
- Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. · arXiv
- Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Ap- plied Sciences, 11(14):6421.
- Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et al. 2023. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312.15166. · arXiv
- Sunjun Kweon, Junu Kim, Jiyoun Kim, Sujeong Im, Eunbyeol Cho, Seongsu Bae, Jungwoo Oh, Gyubok Lee, Jong Hak Moon, Seng Chan You, et al. 2023. Publicly shareable clinical large language model built on synthetic clinical notes. arXiv preprint arXiv:2309.00237. · arXiv
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81. M42. 2023. Med42 -clinical large language model. Microsoft. 2023. Phi-2: The surprising power of small language models. OpenAI. 2023. Chatgpt.
- Ankit Pal, Logesh Kumar Umapathi, and Malaikan- nan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical do- main question answering. In Conference on Health, Inference, and Learning, pages 248-260.
- PMLR.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311-318.
- Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023. Towards expert-level medical question an- swering with large language models. arXiv preprint arXiv:2305.09617. · arXiv
- Augustin Toma, Patrick R Lawler, Jimmy Ba, Rahul G Krishnan, Barry B Rubin, and Bo Wang. 2023. Clini- cal camel: An open-source expert-level medical lan- guage model with dialogue-based knowledge encod- ing. arXiv preprint arXiv:2305.12031. · arXiv
- Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
- Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaek- ermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, et al. 2023. Towards generalist biomedical ai. arXiv preprint arXiv:2307.14334. · arXiv
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.09675. · arXiv
- Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, and Linda Ruth Petzold. 2023. Alpacare: Instruction-tuned large language mod- els for medical application. arXiv preprint arXiv:2310.14558. · arXiv
Cited by
Sorted by citations
NurValues: Real-World Nursing Values Evaluation for Large Language Models in Clinical Context
Yao, B., Li, Q., Zhang, Y., Yang, S., Zhang, B., Tiwari, P., & Qin, J. (2025) · arXiv · doi:10.48550/arxiv.2505.08734 · Cited by 1 · see citing works
“Insufficient value alignment may turn LLMs into new sources of conflict, whilst well-aligned LLM responses can alleviate conflicts and foster trust.”
Trustworthy Medical Question Answering: An Evaluation-Centric Survey
Wang, Y., Wang, B., Mercer, R. E., Rudzicz, F., Roy, S. S., Ren, P., Chen, Z., & Wang, X. (2025) · arXiv · doi:10.48550/arxiv.2506.03659
“Projects like MedExQA, which added speech pathology, demonstrate the value of domain expansion—but many fields remain untested.”
Influential
- Authors
- Kim, Y., Wu, J., Abdulle, Y., & Wu, H.
- Affiliations
- University College London (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 167-181
- arXiv
- 2406.06331
- OpenAlex
- W4402671133
- Cited by
- 49
- References
- 0
- Citations by year
- 2026: 36 · 2025: 12 · 2024: 1
- FWCI
- 11.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Workshop on Biomedical Language Processing (BioNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- Medical Research Council (MR/X030075/1, MR/S004149/2)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- medical question answering, benchmark, large language models, explainability, multiple explanations, speech-language pathology, LLM evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
