What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams
Open accessPublished version of a preprint
The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Open domain question answering (OpenQA) tasks have been recently attracting more and more attention from the natural language processing (NLP) community. In this work, we present the first free-form multiple-choice OpenQA dataset for solving medical problems, MedQA, collected from the professional medical board exams. It covers three languages: English, simplified Chinese, and traditional Chinese, and contains 12,723, 34,251, and 14,123 questions for the three languages, respectively. We implement both rule-based and popular neural methods by sequentially combining a document retriever and a machine comprehension model. Through experiments, we find that even the current best method can only achieve 36.7%, 42.0%, and 70.1% of test accuracy on the English, traditional Chinese, and simplified Chinese questions, respectively. We expect MedQA to present great challenges to existing OpenQA systems and hope that it can serve as a platform to promote much stronger OpenQA models from the NLP community in the future.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“given that medical question datasets such as PubMedQA42 and MedQA43 consist of fixed question sets susceptible to memorization”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [42]
In the citing paper
“Notably, in healthcare, medical NLP benchmarks like MedQA7, PubMedQA8 and the Massive Multitask Language Understanding (MMLU) serve as the de-facto reporting standard for state-of-the-art medical LLMs9–11 with claims of ‘superhuman’ performance in patient-facing tasks12.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [7]
In the citing paper
“Current practice relies heavily on static, fixed-length benchmarks such as, MedQA2, MMLU-Med3, and MedBench4.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [2]
In the citing paper
“Large-scale medical benchmarks often comprise tens of thousands of multiple-choice questions (MCQs) (e.g., MedQA: 12,723 items2; MedMCQA: 6,150 items9; MedBench: 40,041 items4; Med-HALT: 59,254 items10), leading to substantial token consumption and API costs—particularly for proprietary models.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [2]
In the citing paper
“(5) MedQA, introduced by Jin et al. (2021), using the USMLE subset.”
From Sabotage Evaluations for Frontier Models (2024) · cites this paper as Jin et al. (2021)
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
Research works printed but not linked: 3
- Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
- Dhingra, B., Mazaitis, K., and Cohen, W.W. (2017). Quasar: Datasets for Question Answering by Search and Reading. arXiv.
- Das, R., Dhuliawala, S., Zaheer, M., and McCallum, A. (2019). Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering. arXiv.
Cited by
Items 701–762 of 762
Uncertainty calibration for hallucination detection via token dependency and sequence divergence
Qin, S., Zhou, L.-K., Sun, L., & Wang, N.-B. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.116357
Scanlin, J., & Fulghum, J. (2026) · medRxiv · doi:10.64898/2026.09.02.26362074
Al-Farsi, M., Al-Harthy, S., & Al-Rawahi, N. (2025) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/d964147354
Perrella, A., Pecoraro, S., Maffettone, A., Salvatore, P., D'Amore, A., Morfino, V., & Bisogno, M. (2026) · Information · doi:10.3390/info17090869
GPT-5 in medical imaging: Early evidence and the case for hybrid oncology workflows
Karunanayake, N. (2026) · Intelligent Oncology · doi:10.1016/j.intonc.2026.100087
Svensson, A., & Lindberg, E. (2025) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/y427182552
Knowledge-guided multi-agent framework for intelligent acupuncture diagnosis and reasoning
Zhang, W., Cui, J., Zhao, M., Zhang, M., & Li, P. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.116139
StructTune: Structured Sparse Fine-Tuning for Efficient Adaptation of Large Language Models
Liu, Y., Po, L.-M., Cheung, K.-W., Jiang, Z., Wu, H., Li, K., & Xu, X. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.803
Neves, E. B., & Pochmann, P. G. C. (2026) · InterAção · doi:10.5902/2357797597015
Meridian topology-aware graph-language alignment for acupuncture prescription generation
Zhang, W., Cui, J., Zhang, M., Zhao, M., Mu, Y., & Li, P. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.135086
Torres, C., & González, C. I. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-37441-7_30
Şahin, M. Z., Derdiyok, F. B., Kılıç, S. A., Serbest, K., & Nas, K. (2026) · MEC International Conference on Big Data and Smart City (ICBDSC) · doi:10.1109/icadaai64945.2026.11679238
Rani, M., Mishra, B. K., & Babar, M. (2026) · MEC International Conference on Big Data and Smart City (ICBDSC) · doi:10.1109/icadaai64945.2026.11679257
Ramon‐Gonen, R., & Engelstein, H. (2026) · Computation · doi:10.3390/computation14090215
Islamaj, R., Leaman, R., Chan, J., Wan, N., Jin, Q., Xie, N., Wilbur, J., Tian, S., Yeganova, L. E., Lai, P.-T., et al. (2026) · Database · doi:10.1093/database/baag057 · Cited by 3
Praestegaard, L., & Paja, E. (2026) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew72749.2026.00084 · Cited by 1
Yin, Z., Wang, Z., Chen, M., Stockwell, I., Ning, X., Liu, J., & Zhang, W. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001634
Sorka, M., Gorenshtein, A., Abramovitch, H., Soontrapa, P., Shelly, S., & Aran, D. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag143
LLM Prompt Optimization: From Zero-Shot to Automatic Role-Playing Generation Using Ontology
Hauna, A. D. A., Khomsah, S., Yunus, A. P., & Fukui, M. (2026) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2026.p1404
Large Language Model Benchmarks in Medical Tasks
Yan, L. K. Q., Niu, Q., Li, M., Zhang, Y., Yin, C. H., Fei, C., Peng, B., Bi, Z., Feng, P., Chen, K., et al. (2026) · Medicine Advances · doi:10.1002/med4.70085 · Cited by 1
Xu, Z., Dai, X., Wang, B., & Tian, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38059-3_29
Alhwikem, F., Khan, A. R., & Jaskani, F. H. (2026) · Symmetry · doi:10.3390/sym18091572
Nguyen, B. T., Pham, H. V., Tran, T. K., & Dao, S. V. T. (2026) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2026.100467
Qiao, Z., Zhang, P., & Xue, J. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134508
Liu, F., Liu, Z., Fei, X., He, J., Xing, J., Li, J., & Chan, P. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02459-1 · Cited by 1
OphthFlowBench: A Unified Ophthalmology Multimodal Benchmark with Workflow-Aligned Evaluation
Zheng, Q., Luo, X., Ding, M., Ruli, Z., Dou, X., Wen, Y., Liu, C., & Shen, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38470-6_23
PancCADx: A Multimodal Framework for Pancreatic Cancer Diagnosis
Hu, S., Xiao, C., Qin, X., Mei, B., Cheng, B., & Wang, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38091-3_47
Zhang, R., Dong, J., Xin, J., Ballard, J., Beeche, C., Sharma, R., Zhao, X., Konz, N., Long, Q., Witschey, W. R., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38079-1_16
REM-CaR: Retrieval-Augmented Generation Enhanced Medical Case-Based Reasoner
Li, Z., Jin, B., Chen, Y., Gong, T., & Fu, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-5699-0_35
Hakeem-7B: A knowledge-first recipe for the best open Arabic medical reasoner model
Zaghloul, Y., Elasfar, Y., Helmy, A., & Fouad, M. M. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.117096
Oliveira, H. S., & Oliveira, H. P. (2026) · Machine Learning with Applications · doi:10.1016/j.mlwa.2026.101013
Bianchini, F., Bianchini, E., Boyault, C., Folco, E. G., Proponnet-Guérault, M., Vuillerme, N., & Mecella, M. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-28463-1_30
S andhi : Fine-Grained Merging for Memory Efficient Multi-Model Serving
Gupta, V., Duran, O. K., Meda, N. S., An, I., Ananthanarayanan, G., & Iyer, A. P. (2026) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3830418.3843911
TDG-LoRA: Token-Level Dynamic Gating for Mitigating Catastrophic Forgetting
Zhu, S., Huang, S., & Chen, X. (2026) · IEEE Access · doi:10.1109/access.2026.3737439
Thepade, S. D., & Renuse, M. (2026) · International Conference on Computing Communication Control and automation · doi:10.1109/iccubea69821.2026.11702120
Decision models can offload structured clinical tasks from large language modelsPreprint
Ariel, D., Laban, G., Endelman, S., Balicer, R., Dagan, N., & Cohen, S. (2026) · medRxiv · doi:10.64898/2026.10.09.26365130 · Cited by 1
A survey of LLM reasoning in healthcare and medicine: Individual modeling and collaborative agents
Lin, Q., Xiao, L., Pu, B., Shi, K., Du, Y., Xu, J., He, K., Zhao, S., Zhang, J., Cambria, E., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104827 · Cited by 7
Performance within the foundation paradox: the case for small qualitative evaluations
Simeone, M. P., Hyatt, J., Bienenstock, E. J., Babu, R., & Solís, P. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12328-0
Entity-centric evaluation of large language model responses for medical question-answering tasks
Liu, Y., & Kolachalama, V. B. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001752 · Cited by 1
Curriculum-Trained Vertical Agents for Italian Medical Question Answering
Acunzo, A., Barone, M., Dioguardi, G., Marino, R. D., Serio, F. D., Iozzino, G., Postiglione, M., Riccio, G. E., Romano, A., & Vitiello, L. (2026) · International Conference on Information Technology for Social Good (GoodIT) · doi:10.1145/3794786.3830763
Phungjununt, W., Srisomboon, K., Lee, W., Prayote, A., & Pipanmekaporn, L. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100317
Ensemble multi-retrieval methodologies with reasoning model decision in multi-step biomedical QA
Jung, J., Hwang, H., Park, Y., Minju, S., Yun, J., Hwang, H., Lee, S., Sohn, J., & Kang, J. (2026) · Database · doi:10.1093/database/baag061
Elicit and Enhance: Advancing Multimodal Reasoning in Medical Scenarios
Huang, Z., Mu, L., Gu, Y., Hu, K., Hua, S., & Zhang, X. (2026) · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · doi:10.1109/cvprf71909.2026.00539
Nguyen, K., Khan, T., & Michalas, A. (2026) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms69103.2026.00310
Precision Evidence Bench: Assessing PICO alignment and faithful reporting in clinical AIPreprint
Mukerji, A., Sanghavi, N., Lauritzen, T., Muralidharan, J., & Gombar, S. (2026) · medRxiv · doi:10.64898/2026.10.07.26364976
PharmAI: domain-specific QA generation by adaptive learning and fine-tuning of small language models
Anantharajan, S., Loganathan, S., Soundarraj, R., & Nithiyaraj, E. E. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1944175
GynoNurse: An LLM-Driven Clinical Consultation System for Gynecological Nursing
Li, Z., Deng, X., Deng, C., Feng, C., Liu, W., Chen, J., Fan, J., Yang, Y., Wang, X., & Wang, B. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-6916-7_1
Ma, Y., Di, T., Xu, J., & Wu, Y. (2026) · PLoS ONE · doi:10.1371/journal.pone.0349628
Zheng, C., Peng, C., Liang, S., Cai, Z., Peng, Z., & Chen, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-11029574/v1
Risk-Aware Decision-Theoretic Routing for Reliable Use of Large Language Models
Zhao, L., Qin, S., & Shi, Y. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728735
Wang, H., Guo, Y., Ding, S., & Fang, K. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134800
Wu, M., Wang, X., Sun, J., & Wei, B. Y. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728463
Label-Confidence-Aware Uncertainty Quantification in Natural Language Generation
Lin, Q., Feng, Y., Zhang, Y., Yang, Z., & Zhou, L. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11730026
Iyengar, S. S., Gowtham, S., Pradeep, T. G., & Jayashree, R. (2026) · IEEE International Conference on Intelligent Systems · doi:10.1109/is71416.2026.11727381
Prakash, A., Paruti, R. B., Abhimanyu, Shwetha, P. R., & Harish, S. (2026) · International Conference on Next-Gen Intelligent Computing and Applications (ICONICA) · doi:10.1109/iconica70605.2026.11736833
Auditing Cardiology Question Answering with OPM-Based Reasoning Paths
Qi, S., Kang, X., & Matsumoto, K. (2026) · International Conference on Machine Learning and Machine Intelligence (MLMI) · doi:10.1145/3841358.3841417
Translation guided Retrieval-Augmented Generation Evaluation for Turkish Medical Question Answering
Diallo, T. M., Dikenelli, O., & Bora, Ş. (2026) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap71869.2026.11727909
Li, J. (2026) · International Conference on Biomedical and Intelligent Systems (IC-BIS) · doi:10.1145/3841905.3841977
Xu, J., Han, J., Li, L., & Mu, X. (2026) · Journal of Fungi · doi:10.3390/jof12110813
Xian, R. P., Lee, A. J., Lolla, S., Wang, V., Cui, Q., Ro, R., & Abbasi-Asl, R. (2024) · arXiv · doi:10.48550/arxiv.2402.10527
Towards Transparent Reasoning: What Drives Faithfulness in Large Language Models?
McMillan, T., Dominici, G., Gjoreski, M., & Langheinrich, M. (2025) · arXiv · doi:10.48550/arxiv.2510.24236
BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing
Fries, J. A., Weber, L., Seelam, N., Altay, G., Datta, D., Garda, S., Kang, M., Su, R., Kusa, W., Cahyawijaya, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.15076 · Cited by 17
Metadata
- Authors
- Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P.
- Affiliations
- Massachusetts Institute of Technology (US)
- Huazhong University of Science and Technology (CN)
- Computer Science and Artificial Intelligence, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
- Corresponding author
- Jin, Di
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2021-07-12
- Volume
- 11
- Issue
- 14
- Pages
- 6421
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W3162922479
- Cited by
- 718
- References
- 41
- Citations by year
- 2026: 285 · 2025: 241 · 2024: 120 · 2023: 51 · 2022: 12 · 2021: 9
- FWCI
- 16.96
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- Preprints.org (acceptedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- open-domain question answering, medical question answering, multiple-choice questions, Chinese language, document retrieval, machine reading comprehension, disease diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.