What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams
Open accessPublished version of a preprint
The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Open domain question answering (OpenQA) tasks have been recently attracting more and more attention from the natural language processing (NLP) community. In this work, we present the first free-form multiple-choice OpenQA dataset for solving medical problems, MedQA, collected from the professional medical board exams. It covers three languages: English, simplified Chinese, and traditional Chinese, and contains 12,723, 34,251, and 14,123 questions for the three languages, respectively. We implement both rule-based and popular neural methods by sequentially combining a document retriever and a machine comprehension model. Through experiments, we find that even the current best method can only achieve 36.7%, 42.0%, and 70.1% of test accuracy on the English, traditional Chinese, and simplified Chinese questions, respectively. We expect MedQA to present great challenges to existing OpenQA systems and hope that it can serve as a platform to promote much stronger OpenQA models from the NLP community in the future.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“given that medical question datasets such as PubMedQA42 and MedQA43 consist of fixed question sets susceptible to memorization”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [42]
In the citing paper
“Notably, in healthcare, medical NLP benchmarks like MedQA7, PubMedQA8 and the Massive Multitask Language Understanding (MMLU) serve as the de-facto reporting standard for state-of-the-art medical LLMs9–11 with claims of ‘superhuman’ performance in patient-facing tasks12.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [7]
In the citing paper
“Current practice relies heavily on static, fixed-length benchmarks such as, MedQA2, MMLU-Med3, and MedBench4.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [2]
In the citing paper
“Large-scale medical benchmarks often comprise tens of thousands of multiple-choice questions (MCQs) (e.g., MedQA: 12,723 items2; MedMCQA: 6,150 items9; MedBench: 40,041 items4; Med-HALT: 59,254 items10), leading to substantial token consumption and API costs—particularly for proprietary models.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [2]
In the citing paper
“(5) MedQA, introduced by Jin et al. (2021), using the USMLE subset.”
From Sabotage Evaluations for Frontier Models (2024) · cites this paper as Jin et al. (2021)
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.740 · Cited by 2,103
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
Research works printed but not linked: 3
- Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
- Dhingra, B., Mazaitis, K., and Cohen, W.W. (2017). Quasar: Datasets for Question Answering by Search and Reading. arXiv.
- Das, R., Dhuliawala, S., Zaheer, M., and McCallum, A. (2019). Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering. arXiv.
Cited by
Items 301–400 of 762
Developing Resource-Efficient Clinical LLMs for Brazilian Portuguese
Pinto, J. G. D. S., de Freitas, A. R., Martins, A. R. D. C., Sawazaki, C. M. R., Vidal, C., & Oliveira, L. E. S. E. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-79038-6_4 · Cited by 6
Dynamic-KGQA: A Scalable Framework for Generating Adaptive Question Answering Datasets
Dammu, P. P. S., Naidu, H., & Shah, C. S. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730324 · Cited by 10
PediaBench: a comprehensive Chinese pediatric dataset for benchmarking large language models
Zhang, Q., Chen, P., Feng, L., Liu, S., Li, J., Zhao, H., Chen, M., Li, H., & Wang, Y. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-025-41345-w · Cited by 11
Selective UMLS knowledge infusion for biomedical question answering
Park, H., Son, J., Min, J., & Choi, J. (2023) · Scientific Reports · doi:10.1038/s41598-023-41423-8 · Cited by 11
Aswal, S., Ahuja, N. J., & Mehra, R. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3648363 · Cited by 6
Explanatory argumentation in natural language for correct and incorrect medical diagnoses
Molinet, B., Marro, S., Cabrio, E., & Villata, S. (2024) · Journal of Biomedical Semantics · doi:10.1186/s13326-024-00306-1 · Cited by 4
Liu, J., Ren, J., Bai, R., Zhang, Z., & Lu, Z. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125701 · Cited by 5
Generalizing Fairness to Generative Language Models via Reformulation of Non-discrimination Criteria
Sterlie, S., Weng, N., & Feragen, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92648-8_2 · Cited by 11
Schlegel, V., Nenadić, G., & Batista-Navarro, R. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000171 · Cited by 34
PubMed and Beyond: Biomedical Literature Search in the Age of Artificial IntelligencePreprint
Jin, Q., Leaman, R., & Lu, Z. (2023) · PubMed Central · doi:10.48550/arxiv.2307.09683 · Cited by 4
Yang, A. J., Woo, J. J., & Ramkumar, P. N. (2025) · Arthroscopy · doi:10.1016/j.arthro.2025.06.020 · Cited by 4
Dobreva, J., Karasmanakis, I., Ivanišević, F., Horvat, T., Kitanovski, D., Gams, M., Mishev, K., & Simjanoska, M. (2025) · Data in Brief · doi:10.1016/j.dib.2025.112146 · Cited by 6
The Compaction Cliff in Long-Running AI Agent Memory
Zerhoudi, S., Mitrović, J., & Granitzer, M. (2026) · ACM · doi:10.1145/3799682.3840567 · Cited by 4
Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
Xu, W., Chan, H. P., Li, L., Aljunied, M., Yuan, R., Wang, J., Xiao, C., Chen, G., Liu, C., Li, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.07044 · Cited by 4
Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework
Aljohani, B., & Alsanoosy, T. (2026) · Information · doi:10.3390/info17020133 · Cited by 3
IaraMed: A Women’s Healthcare Chatbot for Portuguese Speakers
Färber, F. B., Dollis, J. S., Ribeiro, P. S. F. B., Brito, I. A., Sousa, R. T., & Filho, A. R. G. (2025) · doi:10.5753/sbcas.2025.7897 · Cited by 6
Arkangel AI: A conversational agent for real-time, evidence-based medical question-answering
Villa, M. C., Castano-Villegas, N., Llano, I., Martinez, J., Guevara, M. F., Zea, J., & Velásquez, L. (2025) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2025.100274 · Cited by 6
Wang, Z., Li, K., Peng, S., Liu, L., Yang, X., Yao, K., Heinrich Herre, B., & Zhu, Y. (2025) · JMIR Medical Informatics · doi:10.2196/69286 · Cited by 3
Liu, J., Yan, L. K. Q., Wang, T., Niu, Q., Nagai‐Tanima, M., & Aoyama, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06004-4_30 · Cited by 6
Multi-choice machine reading comprehension benchmark datasets: A survey
Foolad, S., Kiani, K., & Rastgoo, R. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.114499 · Cited by 4
TRUSS-Med: Medical Question Answering via Transformers, Retrieval, and Unified State Space Models
Zhu, X., Khan, M., Mhedhbi, E., Liang, Y., & Othmani, A. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356863 · Cited by 4
Agentic memory-augmented retrieval and evidence grounding for medical question-answering tasks
Jia, S., Bit, S., Jasodanand, V. H., Liu, Y., & Kolachalama, V. B. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106339 · Cited by 13
Corpman, D. W., Auyong, D. B., Walters, A. M., Thind, B. S., Gray, A. T., Bhalla, P. I., Nathwani, R. V., & Ng, I. C. (2026) · Journal of Clinical Anesthesia · doi:10.1016/j.jclinane.2026.112164 · Cited by 3
LLM-SDaT: A knowledge-informed LLM framework for syndrome differentiation in TCM
Guan, B., Gao, S., Zheng, D., Xia, H., Wu, Z., Wu, J., & Xu, H. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108812 · Cited by 3
DxDirector: an agentic large language model driving the full-process clinical diagnosis
Xu, S., Huang, X., Wei, Z., Pang, L., Shen, H., & Cheng, X. (2026) · Nature Communications · doi:10.1038/s41467-026-71928-5 · Cited by 9
Dani, M., Prakash, M. J., Rosa, F., Akata, Z., & Liebe, S. (2024) · Communications Medicine · doi:10.1038/s43856-026-01653-z · Cited by 4
Pan, J., Jian, B., Hager, P., Zhang, Y., Liu, C., Jungmann, F., Li, H., Canisius, J., You, C., Wu, J., et al. (2026) · Nature Health · doi:10.1038/s44360-026-00152-8 · Cited by 29
Tutorial: guidance on the use of large language models for medical research
Jin, Q., Wan, N., Leaman, R., Tian, S., Wang, Z., Yang, Y., Wang, Z., Xiong, G., Lai, P.-T., Zhu, Q. L., et al. (2026) · Nature Protocols · doi:10.1038/s41596-026-01408-z · Cited by 16
Interpreting Patient Descriptions using Distantly Supervised Similar Case Retrieval
Alghanmi, I., Espinosa-Anke, L., & Schockaert, S. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532003 · Cited by 5
Walking a Tightrope – Evaluating Large Language Models in High-Risk Domains
Hung, C.-C., Rim, W. B., Frost, L., Bruckner, L., & Lawrence, C. (2023) · GenBench Workshop on Generalisation (Benchmarking) in NLP (GenBench) · doi:10.18653/v1/2023.genbench-1.8 · Cited by 26
Dolomites : Domain-Specific Long-Form Methodical Tasks
Malaviya, C., Agrawal, P., Ganchev, K., Srinivasan, P., Huot, F., Berant, J., Yatskar, M., Das, D., Lapata, M., & Alberti, C. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00727 · Cited by 12
Emerging Trends in LLM Benchmarking
Desai, A. P., Prajapati, R., Ravi, T., Luqman, M., & Yadav, P. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825845 · Cited by 6
Zong, C. F., Wan, J., Tang, S., & Zhang, L. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103246 · Cited by 6
MedMobile: a mobile-sized language model with clinical capabilities
Vishwanath, K., Stryker, J., Alyakin, A., Alber, D. A., & Oermann, E. K. (2025) · BMJ Digital Health & AI · doi:10.1136/bmjdhai-2025-000068 · Cited by 11
Saleem, S., Asim, M. N., Zulfiqar, S., & Dengel, A. (2026) · Computer Science Review · doi:10.1016/j.cosrev.2026.100938 · Cited by 9
Wang, M., He, X., Liu, L., Fang, Q.-M., Zhang, M., Chen, H., & Liu, Y. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3368288 · Cited by 5
SemPool: Simple, Robust, and Interpretable KG Pooling for Enhancing Language Models
Mavromatis, C., Karypis, P., & Karypis, G. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2238-9_12 · Cited by 8
Empowerment of Large Language Models in Psychological Counseling through Prompt Engineering
Huang, S., Fu, F., Yang, K. Q., Zhang, K., & Yang, F. (2024) · IEEE International Conference on Software Engineering and Artificial Intelligence (SEAI) · doi:10.1109/seai62072.2024.10674052 · Cited by 11
Zhou, Z., Yang, Y., & Ren, T. (2025) · International Computational Imaging Conference (CITA) · doi:10.1117/12.3056789 · Cited by 4
Evaluation of ChatGPT's Usefulness and Accuracy in Diagnostic Surgical Pathology.Preprint
Guastafierro, V., Corbitt, D. N., Bressan, A., Fernandes, B., Mintemur, O., Magnoli, F., Ronchi, S., Rosa, S. L., Uccella, S., & Renne, S. (2024) · medRxiv · doi:10.1101/2024.03.12.24304153 · Cited by 2
Toward the Clinically Effective Evaluation of Artificial Intelligence–Generated Responses
Anbumani, S., & Ahunbay, E. E. (2024) · JCO Clinical Cancer Informatics · doi:10.1200/cci-24-00258 · Cited by 4
Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator
Liao, Y., Meng, Y., Wang, Y., Liu, H., Wang, Y., & Wang, Y. (2024) · arXiv · doi:10.48550/arxiv.2403.08495 · Cited by 15
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Akhtar, M., Reuel, A., Soni, P., Ahuja, S., Ammanamanchi, P. S., Rawal, R., Zouhar, V., Yadav, S., Whitehouse, C., Ki, D., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.16763 · Cited by 3
Shao, Y., Tang, X., Sohn, J., Chen, J., Liao, Y., Zhang, J., Xiang, J., Wu, F., Zhao, Y., Wu, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.07459 · Cited by 3
Sabotage Evaluations for Frontier Models
Benton, J., Wagner, M., Christiansen, E., Anil, C., Perez, E., Srivastav, J., Durmus, E., Ganguli, D., Kravec, S., Shlegeris, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.21514 · Cited by 2
Auditing medical multi-agent AI reveals risks of false consensus
Zhu, Y., Gu, L., Wang, Z., Sang, H., Sui, D., Tang, W., Mi, L., Wang, Y., Gao, J., Yao, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.10185 · Cited by 17
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence
Feng, S., Wang, Z., Wang, Y., Ebrahimi, S., Palangi, H., Miculicich, L., Kulshrestha, A., Rauschmayr, N., Choi, Y., Tsvetkov, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.11163 · Cited by 24
Ke, Y., Lin, H., Ruan, Y., Tang, J., & Li, L. (2025) · Mathematics · doi:10.3390/math13091538 · Cited by 8
Breast-CRAG: A Breast Cancer Large Language Model Leveraging Retrieval-Augmented Generation
Chen, Z., Wang, Q., Liu, J., Sun, Y., Zheng, H., Li, H., Duan, H., & Lü, X. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-95841-0_19 · Cited by 2
Cai, Q., Yang, L., Xiao, J., Ma, J., Liu, M., & Pan, X. (2025) · Diagnostics · doi:10.3390/diagnostics15141829 · Cited by 1
Large language models driven reliable clinical decision-making: Framework and application
Du, J., Li, X., Liu, Y., Lv, T., Liu, H., & Yin, H. (2025) · Informatics and Health · doi:10.1016/j.infoh.2025.08.002 · Cited by 3
Selvaraj, S. P. P., Yen, R. W., Forcino, R. C., & Elwyn, G. J. (2025) · Patient Education and Counseling · doi:10.1016/j.pec.2025.109362 · Cited by 6
Hou, R., Chen, S., Fan, Y., Yu, G., Zhu, L., Sun, J., Liu, J., & Ruan, T. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114524 · Cited by 20
Foundation Model for Medical Imaging: A Comprehensive Review
Jiao, L., Yang, J., Li, R., Liu, F., Liu, X., Chen, P., Guo, Y., Li, L., Shang, R., & Ma, W. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3618796 · Cited by 10
Jarchow, H., Bobrowski, C., Falk, S. S. I., Hermann, A., Kulaga, A. Y., Põder, J.-C., Unfried, M., Usanov, N., Zendeh, B., Kennedy, B. K., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01996-2 · Cited by 9
Physician Perspectives on Large Language Models in Health Care: A Cross-Sectional Survey Study
Hong, H. J., Shah, N. H., Pfeffer, M. A., & Lehmann, L. S. (2025) · Applied Clinical Informatics · doi:10.1055/a-2735-0527 · Cited by 5
Contrastive text embeddings with effective sample mining for enhanced disease diagnosis
Zhang, X., Zhao, G., Ren, Y., Wang, W., Feng, Y., Zhang, X., & Liu, J. (2025) · Array · doi:10.1016/j.array.2025.100578 · Cited by 3
Srinivasan, S., Ai, X., Lo, T. W. S., Gilson, A., Zou, M., Zou, K., Kim, H., Yang, M., Pushpanathan, K., Yew, S. M. E., et al. (2025) · Ophthalmology Science · doi:10.1016/j.xops.2025.101050 · Cited by 6
Wen, A., Lu, Q., Chuang, Y.-N., Wang, G., Yuan, J., Zhang, J., Wang, L. H., Fu, S., Miller, K., Jia, H., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02253-2 · Cited by 4
A benchmark for evaluating diagnostic questioning efficiency of LLMs in patient conversations
Werthaim, M., Kimhi, M., Apartsin, A., & Aperstein, Y. (2026) · Scientific Reports · doi:10.1038/s41598-026-37022-y · Cited by 10
Evaluating reasoning large language models with human-like thinking in ophthalmic question answering
Wang, Z., Xu, C., Wang, L., Qiang, W., Li, Y., Li, D., Xu, F., Zhang, Y., Jiang, J., & Li, Z. (2026) · BMJ Open Ophthalmology · doi:10.1136/bmjophth-2025-002615 · Cited by 2
Li, W., Zhang, Y., Wang, C. R., Li, Y., He, X., Wang, A. L., Xu, M., Zhang, F., Sun, H., Wang, K., et al. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131476 · Cited by 4
MDD-thinker: A reasoning-enhanced large language model for diagnosis of major depressive disorder
Sha, Y., Pan, H., Luo, G., Shi, C., Chen, W., Wang, J., & Li, K. (2026) · Journal of Affective Disorders · doi:10.1016/j.jad.2026.121405 · Cited by 4
Cheema, M. N., Nazir, A., Harmanci, A. O., Harmanci, A. O., Cheema, Y., Masood, S., & Khokhar, F. A. (2026) · The Visual Computer · doi:10.1007/s00371-026-04385-2 · Cited by 2
CDAFlow: Enhancing LLM clinical decision-making through agentic workflow
Hou, R., Xue, D., Sun, H., He, P., Zhang, W., & Ruan, T. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131806 · Cited by 4
Medical multimodal large language models: A survey
Xiao, H., Hui, N., Xu, Y. D., Li, Z., & Peng, J. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104386 · Cited by 5
Zouaghi, M., Kacem, F., & Fathallah, H. (2026) · IEEE Middle East and North Africa Communications Conference (MENACOMM) · doi:10.1109/menacomm69507.2026.11532824 · Cited by 2
Li, J., Yuan, H., Luo, L., Lv, T., Jiang, Y., Wang, F., Zhang, P., Lv, H., Wang, J., Sun, Y., et al. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-7299-1_29 · Cited by 5
How to benchmark medical AI agents
Estévez, S. R., Ferber, D., van der Schaar, M., & Kather, J. N. (2026) · PLoS Medicine · doi:10.1371/journal.pmed.1005170 · Cited by 8
Li, L., Mo, C., Zhou, H., Yu, H., Lu, C., Wang, S., Athreya, V. M., Fitzgerald, M. B., & Wang, S. X. (2026) · Journal of Medical Internet Research · doi:10.2196/94755 · Cited by 2
An open benchmark and language models for AI in aging biology
Zhavoronkov, A. A., Naumov, V., Sidorenko, D., Aliper, A., Aladinskiy, V., Hasani, R., Amini, A., Nasto, K., Reymond, M., Shayakhmetov, R., et al. (2026) · Cell · doi:10.1016/j.cell.2026.08.026 · Cited by 3
Retrofitting language models to operate over bytes
Minixhofer, B., Murray, T., Limisiewicz, T., Korhonen, A., Zettlemoyer, L., Smith, N. A., Ponti, E. M., Soldaini, L., & Hofmann, V. (2026) · Nature · doi:10.1038/s41586-026-11111-4 · Cited by 22
Şensoy, E., & Çıtırık, M. (2024) · Oman Journal of Ophthalmology · doi:10.4103/ojo.ojo_19_24 · Cited by 2
Huang, Y., Gao, T., Zhang, J.-J., Liu, X.-H., & Wang, G.-Y. (2024) · IEEE International Conference on Bioinformatics and Biomedicine · doi:10.1109/bibm62325.2024.10822725 · Cited by 4
Probing the Dual Logic Ability of Privatized Medical-Domain LLMs
Du, Y., Zhao, S., Cai, M., Ma, M., Zhao, D., Cao, J., & Qin, B. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822782 · Cited by 4
Document Embeddings Enhance Biomedical Retrieval-Augmented Generation
Kong, Y., Yang, Z.-H., Luo, L., Ding, Z., Wang, L., Liu, W., Zhang, Y., Xu, B., Wang, J., Sun, Y.-Y., et al. (2024) · IEEE International Conference on Bioinformatics and Biomedicine · doi:10.1109/bibm62325.2024.10822781 · Cited by 2
CAST-Eval: A Domain-Specific Benchmark for Large Language Models in Civil Aviation Safety
Qiao, D., Tian, X., Xu, Y., Yang, J., Yang, L., & Wang, X. (2024) · IEEE International Conference on Electrical, Automation and Computer Engineering (ICEACE) · doi:10.1109/iceace63551.2024.10898372 · Cited by 4
Evaluating fine-tuned GPT models on different datasets in the healthcare domain
Lim, Y. H., Yeoh, P. S. Q., & Lai, K. W. (2025) · Innovation and Emerging Technologies · doi:10.1142/s2737599425500124 · Cited by 4
Physician in the Loop Design of Interactive Agents
Griot, M., Vanderdonckt, J., Yuksel, D., & Hemptinne, C. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-91760-8_7 · Cited by 2
Sakhovskiy, A., & Tutubalina, E. V. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729901 · Cited by 5
Extraction of Diagnostic Reasoning Relations for Clinical Knowledge Graphs
Socrates, V. (2022) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2022.acl-srw.33 · Cited by 2
Natural Language Explanatory Arguments for Correct and Incorrect Diagnoses of Clinical Cases
Marro, S., Molinet, B., Cabrio, E., & Villata, S. (2023) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0011927000003393 · Cited by 7
Self-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering Data
Yu, D., Sun, K., Yu, D., & Cardie, C. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.6 · Cited by 7
Sun, X., Hong, Q., Zhang, M., Li, Y., Chen, T., Huang, Z., Liang, G., Tang, W., Xu, S., Ni, X., et al. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102547 · Cited by 3
Oh, G., Kim, S., Park, S., & Kim, B.-H. (2026) · Journal of Medical Internet Research · doi:10.2196/90693 · Cited by 7
Zheng, T., Jiang, Z., Liu, J., & Feng, S. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02671-w · Cited by 3
MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systems
Chen, K., Zhen, T., Wang, H., Liu, K., Li, X., Huo, J., Yang, T., Xu, J., Dong, W., & Gao, Y. (2025) · arXiv · doi:10.48550/arxiv.2505.20824 · Cited by 32
MHB: Medical Hallucination Benchmark for Large Language Models in Complex Clinical Tasks
Lu, J., Liu, J., Zheng, X., Yang, M., Wang, J., Wang, P., & Zhang, Y. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i45.41243 · Cited by 1
KARN: Knowledge Augmented Reasoning Network for Question Answering
Li, L., Chen, H., Qin, X., Wang, Z., & Dong, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44693-1_49 · Cited by 1
The Most Disruptive Near-Term Use of AI in Cancer Care: Patient Empowerment Through Software Agents
Nothaft, F. A., & Power, B. (2024) · AI in Precision Oncology · doi:10.1089/aipo.2024.0027 · Cited by 2
Mental Chatbot Application Using Retrieval Augmented Generation
Nguyen, X. N.-T., Vo, S. N., & Pham, H.-A. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-97-7710-5_27 · Cited by 1
Applications and Case Studies in Natural Language Understanding
Han, S. C., Weld, H., Li, Y., Lee, J., & Poon, J. (2025) · Springer eBooks · doi:10.1007/978-3-031-74364-1_6 · Cited by 1
Türkçe soru cevaplama için büyük dil modelleri üzerinde geniş ölçekli etki analizi
Güven, Z. A. (2025) · Gazi Üniversitesi Mühendislik-Mimarlık Fakültesi Dergisi · doi:10.17341/gazimmfd.1538022 · Cited by 4
External Retrievals or Internal Priors? From RAG to Epitome-Augmented Generation by Fuzzy Selection
He, K., Xu, J., Lin, Q., Wang, W., Gao, Z., Wu, J., Huang, Y., & Feng, M. (2025) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2025.3581205 · Cited by 4
MedConMA: A Confidence-Driven Multi-agent Framework for Medical Q&A
Wang, R., Chen, Y., Zhang, W., Si, J., Guan, H., Peng, X., & Lü, W. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8180-8_33 · Cited by 7
MedQuery: A Graph-Driven Medical Literature-Enhanced Query Answering System
Fu, C., Xia, Y., Wang, G., Lu, R., & Tang, S. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733383 · Cited by 2
The Hitchhiker’s Guide to Autonomous Research: A Survey of Scientific Agents
Wang, X., Xu, J., Lian, S., Chen, Y., Guo, H., Zhu, F., Shao, Y., Ren, M., Yi, H., Feng, A. H., et al. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3721500 · Cited by 12
Chen, Z., Hoon, Y. H., Ong, R. Y. K., & Wong, J. J. H. (2025) · Information Systems · doi:10.1016/j.is.2025.102610 · Cited by 2
LLM-driven medical document analysis: Enhancing trustworthy pathology and differential diagnosis
Kang, L., Fu, X., Terrades, O. R., Vazquez-Corral, J., Valveny, E., & Karatzas, D. (2026) · Lecture Notes in Computer Science · doi:10.1007/978-3-032-04624-6_36 · Cited by 1
Survey on Question Answering System with Emphasis on Method and Evaluation Metrics
Jadhav, P., & Jadhav, S. M. (2025) · International Conference on Soft Computing for Security Applications (ICSCSA) · doi:10.1109/icscsa66339.2025.11170976 · Cited by 1
Metadata
- Authors
- Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P.
- Affiliations
- Massachusetts Institute of Technology (US)
- Huazhong University of Science and Technology (CN)
- Computer Science and Artificial Intelligence, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
- Corresponding author
- Jin, Di
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2021-07-12
- Volume
- 11
- Issue
- 14
- Pages
- 6421
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W3162922479
- Cited by
- 718
- References
- 41
- Citations by year
- 2026: 285 · 2025: 241 · 2024: 120 · 2023: 51 · 2022: 12 · 2021: 9
- FWCI
- 16.96
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- Preprints.org (acceptedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- open-domain question answering, medical question answering, multiple-choice questions, Chinese language, document retrieval, machine reading comprehension, disease diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.