What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams
Open accessPublished version of a preprint
The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Open domain question answering (OpenQA) tasks have been recently attracting more and more attention from the natural language processing (NLP) community. In this work, we present the first free-form multiple-choice OpenQA dataset for solving medical problems, MedQA, collected from the professional medical board exams. It covers three languages: English, simplified Chinese, and traditional Chinese, and contains 12,723, 34,251, and 14,123 questions for the three languages, respectively. We implement both rule-based and popular neural methods by sequentially combining a document retriever and a machine comprehension model. Through experiments, we find that even the current best method can only achieve 36.7%, 42.0%, and 70.1% of test accuracy on the English, traditional Chinese, and simplified Chinese questions, respectively. We expect MedQA to present great challenges to existing OpenQA systems and hope that it can serve as a platform to promote much stronger OpenQA models from the NLP community in the future.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“given that medical question datasets such as PubMedQA42 and MedQA43 consist of fixed question sets susceptible to memorization”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [42]
In the citing paper
“Notably, in healthcare, medical NLP benchmarks like MedQA7, PubMedQA8 and the Massive Multitask Language Understanding (MMLU) serve as the de-facto reporting standard for state-of-the-art medical LLMs9–11 with claims of ‘superhuman’ performance in patient-facing tasks12.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [7]
In the citing paper
“Current practice relies heavily on static, fixed-length benchmarks such as, MedQA2, MMLU-Med3, and MedBench4.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [2]
In the citing paper
“Large-scale medical benchmarks often comprise tens of thousands of multiple-choice questions (MCQs) (e.g., MedQA: 12,723 items2; MedMCQA: 6,150 items9; MedBench: 40,041 items4; Med-HALT: 59,254 items10), leading to substantial token consumption and API costs—particularly for proprietary models.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [2]
In the citing paper
“(5) MedQA, introduced by Jin et al. (2021), using the USMLE subset.”
From Sabotage Evaluations for Frontier Models (2024) · cites this paper as Jin et al. (2021)
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
Research works printed but not linked: 3
- Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
- Dhingra, B., Mazaitis, K., and Cohen, W.W. (2017). Quasar: Datasets for Question Answering by Search and Reading. arXiv.
- Das, R., Dhuliawala, S., Zaheer, M., and McCallum, A. (2019). Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering. arXiv.
Cited by
Items 401–500 of 762
VMHQA: A Vietnamese Multi-choice Dataset for Mental Health Domain Question Answering
Nguyen, T. A. H., Nguyen, Q.-D., Nguyen, H. M., Nguyen, A. H., & Nguyen, L. T. T. (2025) · ICST Transactions on Scalable Information Systems · doi:10.4108/eetsis.7678 · Cited by 3
Constructing a Multi-source Heterogeneous RAG Knowledge Q&A Dataset in the Telecom Domain
Wang, Y., Li, J., Xu, K., & Xu, Y. (2025) · International Conference on Computer Communication and Artificial Intelligence (CCAI) · doi:10.1109/ccai65422.2025.11189629 · Cited by 1
Castano-Villegas, N., Llano, I., Villa, M. C., Zea, J., & Velásquez, L. (2026) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2026.100447 · Cited by 3
LLMTailor: A Layer-wise Tailoring Tool for Efficient Checkpointing of Large Language Models
Sun, M., Huang, X., Guo, L., Tallent, N. R., Sato, K., & Dai, D. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3731599.3767515 · Cited by 4
Uncertainty Quantification for Multiple-Choice Questions is Just One-Token Deep
Zeng, Q., Jin, M., Yu, Q., Wang, Z. G., Hua, W., Sun, G., Meng, Y., Ma, S., Wang, Q., Juefei-Xu, F., et al. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3760887 · Cited by 1
Enabling doctor-centric medical AI with LLMs through workflow-aligned tasks and benchmarks
Xie, W., Xiao, Q., Zheng, Y., Wang, X., Chen, J., Ji, K., Gao, A., Tiwari, P., Wan, X., Jiang, F., et al. (2025) · npj Health Systems · doi:10.1038/s44401-025-00038-z · Cited by 1
CURE: Confidence-Driven Unified Reasoning Ensemble Framework for Medical Question Answering
Elshaer, Z., & Rashed, E. A. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9120299 · Cited by 2
Analytical Review of Speech and Multimodal Methods for Cognitive Impairments Recognition in People
Dolgushin, M., & Karpov, A. A. (2025) · Informatics and Automation · doi:10.15622/ia.24.6.6 · Cited by 2
DiaShift: An Explainable System for Temporal Diagnostic Shift Detection in Clinical Notes
Bakagianni, J., Dalakleidi, K. V., Stamatis, K. V., & Pavlopoulos, J. (2025) · IEEE International Conference on Bioinformatics and Bioengineering (BIBE) · doi:10.1109/bibe66822.2025.00063 · Cited by 1
Evaluating large language models for real-world perioperative clinical consultation
Li, Y., Zhan, Y., Yu, B., Xu, P., Chen, L., Zhang, C., Zhou, C., Zhang, Y., Wang, X., & Tao, D. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.132350 · Cited by 1
MediMind: A Domain-Adaptive Medical LLM Leveraging Fine-Tuning and Retrieval-Augmented Generation
Hashemi, S. M. H., Kashani, H. B., & Fatemi, S. (2025) · International Conference on the Internet of Things (IoT) · doi:10.1109/iot69654.2025.11297589 · Cited by 2
Queryome: orchestrating retrieval, reasoning, and synthesis across biomedical literature
Punuru, P., Ibtehaz, N., Giri, S. J., Srirangam, H., Tugolukova, E. A., & Kihara, D. (2026) · Journal of Biomedical Semantics · doi:10.1186/s13326-026-00366-5 · Cited by 3
Zheng, Y., Yuan, J., Liu, L., Zhao, W., & Wang, Z. (2025) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3714394.3756275 · Cited by 1
Hierarchical agent reflection for aligning LLM reasoning with clinical diagnostic processes
Wang, X., Li, X., Zhao, D., Feng, K., Liang, L., Zhang, Z., Ding, K., Chen, H., Wan, B., & Zhang, Q. (2026) · Health Information Science and Systems · doi:10.1007/s13755-025-00410-1 · Cited by 2
Hannaan, A., Khandakar, A., Hossen, M. F., Noor, M. T. A., Ayari, M. A., Al-Emadi, N. A., & Muyeen, S. M. (2026) · IEEE Access · doi:10.1109/access.2026.3652625 · Cited by 4
Ahmed, H., & Sadka, A. H. (2026) · IEEE Access · doi:10.1109/access.2026.3656396 · Cited by 2
A Survey on AI-Generated Text Detection: Methods, Challenges, and Future Directions
Ismail, A. A., Ahmed, C., & ElKorany, A. (2025) · Future International Conference on Artificial Intelligence and Cybersecurity (FICAC) · doi:10.1109/ficac65757.2025.11341850 · Cited by 2
BitLoRA: Quantization-Compatible Adapter Tuning for 1.58-bit LLM in Federated On-Device AI-Agent
Inseo, S., & Lee, K. Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131397 · Cited by 2
PC-MoE: memory-efficient and privacy-preserving collaborative training for Mixture-of-Experts LLMs
Zhang, Z. Y., Ding, B., & Low, B. K. H. (2026) · Machine Learning · doi:10.1007/s10994-025-06901-2 · Cited by 1
Hoyt, R. E., & Bajwa, M. (2026) · Applied Clinical Informatics · doi:10.1055/a-2807-4256 · Cited by 2
Augmenting Small Language Model for Better Medical Question Answering through Source Authentication
Das, S., Chatterji, S., & Mukherjee, I. (2026) · ACM Transactions on Information Systems · doi:10.1145/3797887 · Cited by 3
AI on Trial: LLM-as-a-Judge for Private and Reliable Clinical Decision-Making
Khan, T. A., Rafiq, M., & Michalas, A. (2025) · Annual Computer Security Applications Conference Workshops (ACSAC Workshops) · doi:10.1109/acsacw69556.2025.00033 · Cited by 2
Ray, S. K., Shabbir, E., Ali, R., Mohammed, A., & Wazir, S. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00060 · Cited by 2
Knauer, R., Kalmring, M., & Rodner, E. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1789412 · Cited by 1
Fine-tuning the RuBERT language model to improve the accuracy of medical query analysis
Каширина, И. Л., Старичкова, Ю. В., & Le, T. K. (2026) · Vrach i informacionnye tehnologii · doi:10.25881/18110193_2026_1_64 · Cited by 1
Lee, J., Shin, J., & Cho, B. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02567-9 · Cited by 1
Enhancing LLM-based medical decision-making by test-time knowledge acquisition
Li, S., Bao, L., Li, S., & Wan, B. (2026) · Health Information Science and Systems · doi:10.1007/s13755-026-00449-8 · Cited by 1
PKFAR: psychiatry knowledge-fused augmented reasoning with large language models
Wang, R., Yu, C., Dong, Q., Qiu, J., Wen, T., Zhang, W., & Qin, K. (2026) · Health Information Science and Systems · doi:10.1007/s13755-026-00447-w · Cited by 4
Herrera-Poyatos, D., Peláez-González, C., Zuheros, C., Tejedor, V., Montes, R., & Herrera, F. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104389 · Cited by 1
He, N., Yang, L., Hu, X., & He, Y. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1785443 · Cited by 1
Enhancing vision-language model with pretraining for reasoning medical applications
Zhang, Y., Wang, S., Meng, J., Hou, J., Overton, C. E., & Moraros, J. (2026) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2026.102765 · Cited by 1
Large Language Models for Clinical Narrative Processing: Methods, Applications, and Challenges
Livieratos, A., Lin, J., Chasani, P., Gaga, M., Fousekis, F. S., Gogos, C. A., Akinosoglou, K., Katsanos, K. H., & Gamalo, M. (2026) · Methods and Protocols · doi:10.3390/mps9030069 · Cited by 2
HAMMER: An Automatic RAG Tuning System via Hierarchical Memory-Guided Monte Carlo Tree Search
Zhou, Y. K., Wang, Z., & Fang, Y. (2026) · Proceedings of the ACM on Management of Data · doi:10.1145/3802071 · Cited by 2
Liu, J., Liu, S., & Wright, A. T. (2025) · Journal of Medical Internet Research · doi:10.2196/90061 · Cited by 2
MediHive: A Decentralized Agent Collective for Medical Reasoning
Wang, X., & Yang, C. C. (2026) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-026-00239-7 · Cited by 4
MAP: evaluation and multi-agent enhancement of large language models for inpatient pathways
Chen, Z., Peng, Z., Liang, X., Wang, C., Liang, P., Zeng, L., Ju, M., & Yuan, Y. (2026) · npj Health Systems · doi:10.1038/s44401-026-00085-0 · Cited by 17
Jiang, L., Chen, J., Lu, L., Peng, X., Liu, L., He, J., & Xu, J. (2026) · JMIR AI · doi:10.2196/86864 · Cited by 2
Yadav, I., Schindler, S., Peters, D., & Klinger, R. (2026) · Semantic Web · doi:10.1177/22104968261453132 · Cited by 5
A roadmap for medical large language models: a review of foundations, applications, and challenges
Sha, Y.-Y., Yu, L., Lin, Z.-H., Kaur, A., Lou, Y., Gornale, S. S., Zhang, T., Wong, L. S., Wang, Z., Yan, Y., et al. (2026) · Military Medical Research · doi:10.1016/j.mmr.2026.100050 · Cited by 4
The Doctor Will (Still) See You Now: On the Structural Limits of Agentic AI in Healthcare
Aránguiz-Dias, G., Meimandi, K. J., Griffith, A., Dias, C. A., Kim, G. R., Saadeddin, L., & Kochenderfer, M. J. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812224 · Cited by 4
Medical question answering: A comprehensive multimodal and LLM-driven survey
Mhedhbi, E., Zhu, X., Ayaz, M., Khan, M., & Othmani, A. (2026) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2026.109541 · Cited by 4
Sukpancharoen, S., & Srinophakun, T. R. (2026) · Chemical Engineering Journal Advances · doi:10.1016/j.ceja.2026.101375 · Cited by 1
Huang, X., Xiao, M., Qin, C., Long, Q., Chen, J., Zhou, Y., & Zhu, H. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817489 · Cited by 5
Knowledge Graph-Enhanced Long-CoT for Complex Biomolecular Reasoning
Lyu, T., Zhuang, X., Ding, K., Cao, X., Liang, L., Zhao, W., Chen, H., & Zhang, Q. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818882 · Cited by 1
Multi-view auxiliary modeling for stable LLM adaptation under view disagreement
Chen, X., Zhang, J., Long, Y., Jin, P., Su, N., Dai, X., & Chen, J. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114569 · Cited by 1
Kang, X., Ding, W., Matsumoto, K., Wang, L., Yu, H.-T., & Shi, X. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104702 · Cited by 1
Uncertainty-aware large language models: a scoping review of conformal prediction methods
Ashby, A. E., Nguyen, K. A., & Luo, Z. (2026) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2026.0061 · Cited by 1
Limited benchmarks constrain the conclusions of a general-purpose versus clinical AI comparison
Beaulieu‐Jones, B. K., & Nemati, S. (2026) · Nature Medicine · doi:10.1038/s41591-026-04638-6 · Cited by 1
Barbosa, G. P., Carvalho, E., Oliveira, D. A., Moura, A. R., Guerra, A., Pinheiro, F., Correia, R., Santos, A. M. D., Ramião, N., & Mascarenhas, M. (2026) · Diagnostics · doi:10.3390/diagnostics16193107 · Cited by 1
LLM-based reconstruction of longitudinal clinical trajectories in chronic liver disease
Paverd, H., Gao, Z., Mahani, G. K., Fabre, M. A., Burge, S., Hoare, M., & Crispin‐Ortuzar, M. (2026) · npj Precision Oncology · doi:10.1038/s41698-026-01650-4 · Cited by 1
Verejan, V. S. (2023) · Romanian Journal of Ophthalmology · doi:10.22336/rjo.2023.62 · Cited by 5
How to Use Language Expert to Assist Inference for Visual Commonsense Reasoning
Song, Z., Hu, W., Ye, H., & Hong, R. (2023) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw60847.2023.00074 · Cited by 1
Shaier, S., Bennett, K., Hunter, L., & Kann, K. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.36 · Cited by 15
Implicit Modality Mining: An End-to-End Method for Multimodal Information Extraction
Lu, J., & Guo, Q. (2024) · Journal of Electronic Research and Application · doi:10.26689/jera.v8i2.6397 · Cited by 1
Path-Aware Cross-Attention Network for Question Answering
Luo, Z., Xiong, Y., & Tang, B. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2253-2_9 · Cited by 1
del Rey, F. C., & Arias, M. C. (2024) · Revista Española de Cirugía Ortopédica y Traumatología · doi:10.1016/j.recot.2024.05.004 · Cited by 9
Luo, S., Qin, H., Li, H., & Huang, C. (2024) · Information Research · doi:10.47989/ir292837 · Cited by 2
del Rey, F. C., & Arias, M. C. (2024) · Revista Española de Cirugía Ortopédica y Traumatología · doi:10.1016/j.recot.2024.11.005 · Cited by 6
Zekaoui, N. E., Mikram, M., Rhanoui, M., & Yousfi, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-0695-5_32 · Cited by 2
Zhang, H., Dong, F., Li, W., Ren, Y., & Dong, H. (2025) · International Conference on Computer Research and Development · doi:10.1109/iccrd64588.2025.10963108 · Cited by 1
RAGNet: Enhancing Conversational AI in Medical Domain
Varshney, N., Mantoo, S., Wadhwa, M., Aatif, M., & Bhardwaj, S. (2025) · International Conference on Educational Technology Management (ICETM) · doi:10.1109/icetm63734.2025.11051911 · Cited by 1
ALSA: Context-Sensitive Prompt Privacy Preservation in Large Language Models
Ma, H., Lü, W., Liang, Y., Wang, T., Zhang, Q., Zhu, Y., & Si, J. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736840 · Cited by 12
You, Z., & Guo, Y. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105019 · Cited by 5
Improving Knowledge Graph-based BERT Pre-Training Through Entity Hot Partition and Adapter Fusion
Zhan, D., Wu, X., Chang, Y., Tian, C., Wang, T., Chen, C., Xie, L., Huang, W., Tong, E., & Niu, W. (2023) · China Automation Congress (CAC) · doi:10.1109/cac59555.2023.10451936 · Cited by 1
Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERTPreprint
Meng, Z., Liu, F., Clark, T. H., Shareghi, E., & Collier, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04810 · Cited by 9
Sekar, T., Kushal, Shankar, S., Mohammed, S., & Fiaidhi, J. (2025) · medRxiv · doi:10.1101/2025.05.03.25325604 · Cited by 5
Genome-Bench: A Scientific Reasoning Benchmark from Real-World Expert DiscussionsPreprint
Yin, M., Qu, Y., Liu, D., Yang, L., Cong, L., & Wang, M. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.06.02.657538 · Cited by 6
Jiang, L., Chen, J., Lu, L., Peng, X., Liu, L., He, J.-J., & Xu, J. (2025) · doi:10.2196/preprints.80049 · Cited by 2
Closing the Pediatric Divide: A Performance Analysis of the GPT-5 Family in Medical DiagnosticsPreprint
Mondillo, G., Abbate, F. G., Masino, M., Colosimo, S., Perrotta, A., & Frattolillo, V. (2025) · medRxiv · doi:10.1101/2025.08.28.25334657 · Cited by 2
Castano-Villegas, N., Villa, M. C., Barrientos, K. M., Llano, I., Velásquez, L., & Zea, J. (2025) · medRxiv · doi:10.1101/2025.09.23.25336206 · Cited by 1
Pandemic-Potential Viruses are a Blind Spot for Frontier Open-Source LLMsPreprint
Luebbert, L., Ektefaie, Y., Rao, A. S., Wilkason, C., Nosamiefan, D., Achonduh-Atijegbe, O. A., Soumaré, H. M., Adebayo, A. P., Olulaja, O. N., Amadi, J., et al. (2025) · medRxiv · doi:10.64898/2025.12.04.25341642 · Cited by 1
MedPI: Evaluating AI Systems in Medical Patient-facing InteractionsPreprint
Fajardo, D., Proniakin, O., Gruber, V., & Marinescu, R. V. (2026) · medRxiv · doi:10.64898/2025.12.24.25342982 · Cited by 4
Scanlin, J., Cuesta, A. G., & Varsavsky, M. (2026) · medRxiv · doi:10.64898/2026.02.13.26346256 · Cited by 4
MedMisBench: Measuring Epistemic Resilience of LLMs Under Misleading Medical ContextPreprint
Zhou, H., Zou, X., Wu, J., Wu, S., Yu, J., Segal, B., Niebuhr, T. E., Amro, S., Petrus, M., Momin, S., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.25.727671 · Cited by 4
The Dialogue That Heals: A Comprehensive Evaluation of Doctor Agents' Inquiry Capability
Gong, L., Wang, A., Lai, Y., Ma, W., & Liu, Y. (2025) · arXiv · doi:10.48550/arxiv.2509.24958 · Cited by 1
Walk the talk? Measuring the faithfulness of large language model explanations
Matton, K., Ness, R. O., Guttag, J., & Kıcıman, E. (2025) · arXiv · doi:10.48550/arxiv.2504.14150 · Cited by 1
Liao, Y., Franco, Z., Lizarraga Mazaba, J. E., & Chen, K. (2026) · Proceedings of the 17th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics · doi:10.1145/3807503.3819453 · Cited by 1
Can Large Language Models Self-Correct in Medical Question Answering? An Exploratory Study
Zhan, Z., Cui, M., & Zhang, R. (2026) · 2026 IEEE 14th International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00215 · Cited by 3
SEMA-RAG: A Self-Evolving Multi-Agent Retrieval-Augmented Generation Framework for Medical Reasoning
Huang, Y., Chen, R., & Cheng, J. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.917
Wang, Y., Zhang, X., Yao, K., Zeng, Z., Song, K., Lin, J., & Shen, Z. (2026) · arXiv · doi:10.48550/arxiv.2606.05241
SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving
Zhang, Q., Chen, K., Liao, N., Lin, Z., Tang, B., Xiong, F., Li, Z., & Wang, X. (2026) · arXiv · doi:10.48550/arxiv.2606.01751
LiveMedBench: A Contamination-Limited Medical Benchmark for LLMs with Automated Rubric Evaluation
Yan, Z., Song, D., Fang, Z., Ji, Y., Li, X., Li, Q., & Sun, L. (2026) · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3770855.3817579
Trustworthy Medical Question Answering: An Evaluation-Centric Survey
Wang, Y., Wang, B., Mercer, R. E., Rudzicz, F., Roy, S. S., Ren, P., Chen, Z., & Wang, X. (2025) · arXiv · doi:10.48550/arxiv.2506.03659
GeoCQA: A Large-Scale Geography-Domain Chinese Question Answering Dataset from Examination
Cui, Z. S., Wang, B., & Ju, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88483-3_13
Generating multiple-choice questions for medical question answering with distractors and cue-maskingPreprint
Sileo, D., Uma, K., & Moens, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.07069 · Cited by 2
Feature Enhanced Structured Reasoning for Question Answering
Li, L., Chen, H., & Qin, X. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-7224-1_15
Selvaraj, S. P. P., Yen, R. W., Forcino, R. C., & Elwyn, G. J. (2024) · doi:10.2196/preprints.57790
Zhang, H., & Shafiq, M. O. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00167
Medical Knowledge Q&A Evaluation Based on ChatGPT Ensemble Learning
Duan, P., & Su, X. B. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-1717-0_13 · Cited by 1
Wang, Z., Duan, J., Yuan, C., Chen, Q., Chen, T., Yao, H., Zhang, Y., Wang, R., Xu, K., & Shi, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4791792
Kong, L., Zhang, S., Li, J., & Liu, S. (2023) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom60117.2023.00383
Şensoy, E., & Çıtırık, M. (2024) · Ankyra Medical Journal · doi:10.51271/ankmj-0014 · Cited by 1
Learning Choice Nuance for Multiple-Choice Commonsense Question Answering
Yang, D., Deng, W., Wang, Z., Wang, K., Zhuang, Z., & Li, H. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651121
Dynamic Reasoning with Language Model and Knowledge Graph for Question Answering
Lu, Y., Wu, D., & Zhang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70546-5_26
VITRUVIUS: A conversational agent for real-time evidence based medical question answeringPreprint
Villa, M. C., Llano, I., Castano-Villegas, N., Martinez, J., Guevara, M. F., Zea, J., & Velásquez, L. (2024) · medRxiv · doi:10.1101/2024.10.03.24314861 · Cited by 2
MulMed: Addressing Multiple Medical Tasks Utilizing Large Language ModelsPreprint
Cheng, N., Li, F., & Huang, L. (2024) · Research Square · doi:10.21203/rs.3.rs-4967279/v1
Improving clinical expertise in large language models using electronic medical recordsPreprint
Zhu, L., Liu, J., Wang, J., Zhang, W., Jiang, S., Yang, H., Wang, C., Ye, Q., Ruan, T., Rui, X., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-5285540/v1
CKF: Conditional Knowledge Fusion Method for CommonSense Question Answering
Xie, M., Hao, C., Zhang, P., & Ma, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9431-7_9
Parallel Fusion of Graph and Text with Semantic Enhancement for Commonsense Question Answering
Zong, J., Li, Z., Chen, T., Zhang, L., & Zhan, Y. (2024) · Electronics · doi:10.3390/electronics13234618
Membership Feature Aggregation Attack Against Knowledge Reasoning Models in Internet of Things
Ding, Z., Tian, Y., Xiong, J., Wang, G., & Ma, J. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3516319 · Cited by 1
Metadata
- Authors
- Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P.
- Affiliations
- Massachusetts Institute of Technology (US)
- Huazhong University of Science and Technology (CN)
- Computer Science and Artificial Intelligence, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
- Corresponding author
- Jin, Di
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2021-07-12
- Volume
- 11
- Issue
- 14
- Pages
- 6421
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W3162922479
- Cited by
- 718
- References
- 41
- Citations by year
- 2026: 285 · 2025: 241 · 2024: 120 · 2023: 51 · 2022: 12 · 2021: 9
- FWCI
- 16.96
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- Preprints.org (acceptedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- open-domain question answering, medical question answering, multiple-choice questions, Chinese language, document retrieval, machine reading comprehension, disease diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.