MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
Open accessAbstract
This paper introduces MedMCQA, a new large-scale, Multiple-Choice Question Answering (MCQA) dataset designed to address real-world medical entrance exam questions. More than 194k high-quality AIIMS \& NEET PG entrance exam MCQs covering 2.4k healthcare topics and 21 medical subjects are collected with an average token length of 12.77 and high topical diversity. Each sample contains a question, correct answer(s), and other options which requires a deeper language understanding as it tests the 10+ reasoning abilities of a model across a wide range of medical subjects \& topics. A detailed explanation of the solution, along with the above information, is provided in this study.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large-scale medical benchmarks often comprise tens of thousands of multiple-choice questions, leading to substantial token consumption and API costs, particularly for proprietary models.
“Large-scale medical benchmarks often comprise tens of thousands of multiple-choice questions (MCQs) (e.g., MedQA: 12,723 items2; MedMCQA: 6,150 items9; MedBench: 40,041 items4; Med-HALT: 59,254 items10), leading to substantial token consumption and API costs—particularly for proprietary models.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [9]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 26
- Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. Overview of the mediqa 2019 shared task on textual inference, question entailment and question answering. In MEDIQA , pages 370-379. Association for Computational Linguistics, 8 2019. doi: 10.18653/v1/W19-5039. URL https://www.aclweb.org/anthology/W19-5039 . · DOI
- Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: A human generated machine reading comprehension dataset. 2016 Conference on Neural Information Processing Systems , 2018.
- Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. In SciBERT: A Pretrained Language Model for Scientific Text , pages 3615-3620. Association for Computational Linguistics, 11 2019. doi: 10.18653/ v1/D19-1371. URL https://www.aclweb.org/ anthology/D19-1371 .
- Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer opendomain questions. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers , 2017.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , pages 4171-4186. Association for Computational Linguistics, 6 2019. doi: 10.18653/ v1/N19-1423. URL https://www.aclweb.org/ anthology/N19-1423 .
- Domain-specific language model pretraining for biomedical natural language processing. ArXiv , abs/2007.15779, 2022. · arXiv
- Karl Moritz Hermann, Tom´ aˇ s Koˇ cisk´ y, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. NIPS'15: Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1 December 2015 Pages 1693-1701 , 2015.
- Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , 2019.
- Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , 2019.
- Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics , 2019.
- Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , 2017.
- Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics , 36(4):1234-1240, February 2020. ISSN 1367-4803. doi: 10.1093/ bioinformatics/btz682.
- Timo M¨ oller, Anthony Reina, Raghavan Jayakumar, and Malte Pietsch. Covid-qa: A question answering dataset for covid-19. In COVID-QA: A Question Answering Dataset for COVID-19 . Association for Computational Linguistics, 7 2020. URL https://www.aclweb.org/ anthology/2020.nlpcovid19-acl.18 .
- Anastasios Nentidis, Konstantinos Bougiatiotis, Anastasia Krithara, and Georgios Paliouras. Results of the seventh edition of the bioasq challenge. Communications in Computer and Information Science , page 553-568, 2020. ISSN 1865-0937. doi: 10.1007/978-3-030-43887-6 51. URL http:// dx.doi.org/10.1007/978-3-030-43887-6\_51 . · DOI
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In SQuAD: 100,000+ Questions for Machine Comprehension of Text , pages 2383-2392. Association for Computational Linguistics, 11 2016. doi: 10.18653/ v1/D16-1264. URL https://www.aclweb.org/ anthology/D16-1264 .
- Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics , 7:249-266, 2019.
- Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. Newsqa: A machine comprehension dataset. Proceedings of the 2nd Workshop on Representation Learning for NLP , 2017.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, glyph[suppress] L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https:
- David Vilares and Carlos Gomez-Rodr. Head-qa: A healthcare dataset for complex reasoning. In HEAD-QA: A Healthcare Dataset for Complex Reasoning , pages 960-966. Association for Computational Linguistics, 7 2019. doi: 10.18653/ v1/P19-1092. URL https://www.aclweb.org/ anthology/P19-1092 .
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Transformers: State-ofthe-Art Natural Language Processing , pages 3845. Association for Computational Linguistics, 10 2020. doi: 10.18653/v1/2020.emnlp-demos. 6. URL https://www.aclweb.org/anthology/ 2020.emnlp-demos.6 . · DOI
- Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. Recipeqa: A challenge dataset for multimodal comprehension of cooking recipes. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , 2018.
- Yi Yang, Wen tau Yih, and Christopher Meek. Wikiqa: A challenge dataset for open-domain question answering. In WikiQA: A Challenge Dataset for Open-Domain Question Answering , pages 20132018. Association for Computational Linguistics, 9 2015. doi: 10.18653/v1/D15-1237. URL https: //www.aclweb.org/anthology/D15-1237 . · DOI · Paper page
- Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , 2018.
- Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , 2018.
- Xiao Zhang, Ji Wu, Zhiyang He, Xien Liu, and Ying Su. Medical exam question answering with largescale reading comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.
- Simon ˇ Suster and Walter Daelemans. Clicr: A dataset of clinical case reports for machine reading comprehension. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , 2018.
Cited by
Sorted by citations
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798 · see citing works
Almanac — Retrieval-Augmented Language Models for Clinical Medicine
Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al. (2024) · NEJM AI · doi:10.1056/aioa2300068 · Cited by 424 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · see citing works
“…question answering datasets: USMLE, whichincludesdifficult real-world medicalques-tions targeting medical professionals 32 ; the MedMCQA, which gathers questions from medical school entrance exams 44 ; and the PubMedQA, which includes reading comprehension questions about PubMed abstracts.”
BackgroundA review of large language models and autonomous agents in chemistry
Ramos, M. C., Collison, C. J., & White, A. D. (2025) · Chemical Science · doi:10.1039/d4sc03921a · Cited by 274 · see citing works
Retrieval augmented generation for large language models in healthcare: A systematic review
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000877 · Cited by 267 · see citing works
Medical large language models are vulnerable to data-poisoning attacks
Alber, D. A., Yang, Z., Alyakin, A., Yang, E., Rai, S., Valliani, A. A., Zhang, J., Rosenbaum, G. R., Amend-Thomas, A. K., Kurland, D. B., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03445-1 · Cited by 195 · see citing works
AI chatbots not yet ready for clinical use.
Au Yeung, J., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1161098 · Cited by 176 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147 · see citing works
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning
Tang, X., Zou, A., Zhang, Z., Li, Z., Zhao, Y., Zhang, X., Cohan, A., & Gerstein, M. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.33 · Cited by 133 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69 · see citing works
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
Cai, Y., Wang, L., Wang, Y., De Melo, G., Zhang, Y., Wang, Y., & He, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29723 · Cited by 42 · see citing works
HealthBench: Evaluating Large Language Models Towards Improved Human Health
Arora, R. K., Wei, J., Hicks, R. S., Bowman, P., Quiñonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.08775 · Cited by 28 · see citing works
Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks
Gallifant, J., Chen, S., Moreira, P. J. F., Munch, N., Gao, M., Pond, J., Celi, L. A., Aerts, H., Hartvigsen, T., & Bitterman, D. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.726 · Cited by 19 · see citing works
AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator
Fan, Z., Tang, J., Chen, W., Wang, S., Wei, Z., Xi, J., Huang, F., & Zhou, J. (2024) · arXiv · doi:10.48550/arxiv.2402.09742 · Cited by 6 · see citing works
Challenging the Evaluator: LLM Sycophancy Under User Rebuttal
Kim, S. W., & Khashabi, D. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.1222 · Cited by 6 · see citing works
Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework
Aljohani, B., & Alsanoosy, T. (2026) · Information · doi:10.3390/info17020133 · Cited by 3 · see citing works
Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator
Liao, Y., Meng, Y., Wang, Y., Liu, H., Wang, Y., & Wang, Y. (2024) · arXiv · doi:10.48550/arxiv.2403.08495 · Cited by 15 · see citing works
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence
Feng, S., Wang, Z., Wang, Y., Ebrahimi, S., Palangi, H., Miculicich, L., Kulshrestha, A., Rauschmayr, N., Choi, Y., Tsvetkov, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.11163 · Cited by 2 · see citing works
“We employ 4 such domains and 2 tasks in each domain, specifically medical (MedQA (Jin et al., 2021; Li et al., 2024b) and MedM-CQA (Pal et al., 2022)), legal (hearsay and citation prediction classification in LegalBench (Guha et al., 2024)), scientific (SciFact (Wadden et al., 2020) and the STEM subset of MMLU-pro (Wa…”
Sabotage Evaluations for Frontier Models
Benton, J., Wagner, M., Christiansen, E., Anil, C., Perez, E., Srivastav, J., Durmus, E., Ganguli, D., Kravec, S., Shlegeris, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.21514 · Cited by 2 · see citing works
Kong, S., Yang, X., Wei, Y., Wang, Z., Tang, H., Qin, J., Lan, S., Wang, Y., Bai, J., Chen, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01252 · Cited by 3 · see citing works
“Domain-specific benchmarks have emerged to assess medical LLMs. Early efforts like MedQA [7] and MedMCQA [15] focused on Western medical knowledge through multiple-choice questions, later expanded by MultiMedQA [18] to include diverse clinical formats.”
Zheng, T., Jiang, Z., Liu, J., & Feng, S. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02671-w · Cited by 1 · see citing works
Jiang, Y., Li, Q., Xu, B., Sun, H., Ding, C., Dong, J., Cai, Y., Zhang, X., & Yin, J. (2026) · arXiv · doi:10.48550/arxiv.2601.03054 · Cited by 1 · see citing works
SEMA-RAG: A Self-Evolving Multi-Agent Retrieval-Augmented Generation Framework for Medical Reasoning
Huang, Y., Chen, R., & Cheng, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.917
SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving
Zhang, Q., Chen, K., Liao, N., Lin, Z., Tang, B., Xiong, F., Li, Z., & Wang, X. (2026) · arXiv · doi:10.48550/arxiv.2606.01751
“…applicability in collaborative reasoning systems, including MATH [25], GSM-Hard [26], and AQUA-RAT [27] for mathematical reasoning; SciBench [28] and GPQA [29] for scientific reasoning; MMLU-Pro [30] for general knowledge understanding; and MedQA [31] and MedMCQA [32] for medical-domain evaluation.”
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Heineman, D., Hofmann, V., Magnusson, I., Gu, Y., Smith, N. A., Hajishirzi, H., Lo, K., & Dodge, J. (2025) · arXiv · doi:10.48550/arxiv.2508.13144
Wang, Y., Zhang, X., Yao, K., Zeng, Z., Song, K., Lin, J., & Shen, Z. (2026) · arXiv · doi:10.48550/arxiv.2606.05241
A Curated Dataset for Question Answering on Laboratory Test Reference Ranges and Interpretation
Bhasuran, B., Jin, Q., Deville, A., Wu, Y., Hanna, K., Lu, Z., & He, Z. (2026) · Scientific Data · doi:10.1038/s41597-026-07554-z
- Authors
- Pal, A., Umapathi, L. K., & Sankarasubbu, M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2203.14371
- OpenAlex
- W4221154592
- Cited by
- 157
- References
- 0
- Citations by year
- 2026: 23 · 2025: 63 · 2024: 49 · 2023: 16 · 2022: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Keywords
- medical question answering, multiple-choice questions, question answering dataset, medical knowledge, answer explanations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
