PubMedQA: A Dataset for Biomedical Research Question Answering
Open accessThe DOI matches its Crossref record; 24 of the 26 references it cites were found in a registry or an index.
Abstract
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, Xinghua Lu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“given that medical question datasets such as PubMedQA42 and MedQA43 consist of fixed question sets susceptible to memorization”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [43]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“All analyses were performed on publicly available, de-identified benchmark datasets—MIRAGE [23], PubMedQA [24], and the medical licensing examination questions of Katz et al. [25]—used in accordance with their respective licenses and terms of use.”
From The economics of accuracy for medical reasoning with large language models (2026) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“PubMedQA (Jin et al., 2019) is a biomedical research QA dataset under the MIT license. It contains 1,000 expert-annotated questions ( pqa\_labeled\_split ) and 211K machine-labeled questions from PubMed abstracts (the most widely used biomedical literature resource).”
From MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025) · cites this paper as Jin et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Notably, in healthcare, medical NLP benchmarks like MedQA7, PubMedQA8 and the Massive Multitask Language Understanding (MMLU) serve as the de-facto reporting standard for state-of-the-art medical LLMs9–11 with claims of ‘superhuman’ performance in patient-facing tasks12.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“The ground truth question-answer pairs are sourced from the pqa\_labeled split of PubMedQA (Jin et al., 2019), which contains 1,000 expert-annotated samples, supplemented with 9,000 instances randomly selected from the pqa\_artificial split.”
From MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025) · cites this paper as Jin et al., 2019
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 2×
“We fine-tune BioBERT (Lee et al., 2019) on Pub-MedQA as a baseline.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014) · Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.3115/v1/p14-5010 · Cited by 7,037
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“Unlike other QA datasets in which questions are asked by crowd-workers for existing contexts (Rajpurkar et al., 2016; Yang et al., 2018; Kočiskỳ et al., 2018), in PubMedQA contexts are generated to answer the questions and both are written by the same authors.”
BackgroundNatural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 6×
“, 2018), Natural Questions (Kwiatkowski et al., 2019), ShARC (Saeidi et al.”
BackgroundMethodInfluentialBoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 1,855
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 6×
“Supporting fact for the answer is highlighted . et al., 2018; Yang et al., 2018; Kwiatkowski et al., 2019), the largest annotated biomedical QA dataset, BioASQ (Tsatsaronis et al., 2015) has less than 3k training instances, most of which are simple factual questions.”
BackgroundMethodInfluentialSelf-citationEnhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113
Research works printed but not linked: 1
- 2012 Conference and Labs of the Evaluation ForumQuestion Answering For Machine Reading Evaluation (QA4MRE), Rome/Forner, J.[edit.]; ea , pages 1-14.
Cited by
Items 301–400 of 1,026
OntoTune: Ontology-Driven Self-training for Aligning Large Language Models
Liu, Z., Gan, C. G., Wang, J., Zhang, Y., Bo, Z., Sun, M., Chen, H., & Zhang, W. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714816 · Cited by 33
Self-citationHe, J., Zhang, B., Rouhizadeh, H., Chen, Y., Yang, R., Lu, J. S., Chen, X., Liu, N., & Teodoro, D. (2026) · Artificial Intelligence Review · doi:10.21203/rs.3.rs-8330917/v1 · Cited by 35
A review on Natural Language Processing Models for COVID-19 research
Hall, K., Chang, V. I. C., & Jayne, C. (2022) · Healthcare Analytics · doi:10.1016/j.health.2022.100078 · Cited by 33
Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language ModelsPreprint
Kaddour, J., & Liu, Q. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.01119 · Cited by 33
OLAPH: Improving Factuality in Biomedical Long-form Question AnsweringPreprint
Jeong, M., Hwang, H., Yoon, C., Lee, T., & Kang, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.12701 · Cited by 34
Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge DistillationPreprint
Nguyen, H. T., He, Z., Gandre, S. A., Pasupulety, U., Shivakumar, S. K., & Lerman, K. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.11306 · Cited by 34
TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and LawPreprint
Hui, Z., Dong, Y. R., Shareghi, E., & Collier, N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.21134 · Cited by 33
Hou, B., Mukherjee, P., Batheja, V., Wang, K. C., Summers, R. M., & Lu, Z. (2025) · Radiology · doi:10.1148/radiol.250617 · Cited by 32
Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications
Ma, Z., Wang, W., Ding, M., Zheng, S., Liu, S. Y., Liu, J., Ji, J., Shen, L., Yuan, Y., & Chen, W. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.00669 · Cited by 32
Artificial intelligence in bioinformatics: a survey
Jiang, J., Li, Y., Cao, S., Shan, Y., Liu, Y., Fei, T., Yu, Y., Feng, Y., Li, Y., Li, Y., et al. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf576 · Cited by 35
SoK: Large Language Model Copyright Auditing via FingerprintingPreprint
Shao, S., Li, Y., He, Y., Yao, H., Yang, W., Tao, D., & Qin, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.19843 · Cited by 32
Digital forgetting in large language models: a survey of unlearning methods
Blanco-Justicia, A., Jebreel, N., Manzanares-Salor, B., Sánchez, D., Domingo-Ferrer, J., Collell, G., & Eeik Tan, K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11078-6 · Cited by 31
MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models
Pandit, S., Xu, J., Hong, J., Wang, Z., Chen, T., Xu, K., & Ding, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.14302 · Cited by 31
“For data and model access, we relied on Hugging Face resources, specifically using the qiaojin/PubMedQA dataset.”
InfluentialLiu, L., Yang, X., Lei, J., Shen, Y., Wang, J., Wei, P., Chu, Z., Qin, Z., & Ren, K. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3709941 · Cited by 31
“Here, we report the comparison results for different (Med-)LLMs on the PubMedQA benchmark [62].”
How Well Did ChatGPT Perform in Answering Questions on Different Topics in Gross Anatomy?
Bolgova, O., Shypilova, I., Sankova, L., & Mavrych, V. (2023) · European Journal of Medical and Health Sciences · doi:10.24018/ejmed.2023.5.6.1989 · Cited by 31
AlzheimerRAG: Multimodal Retrieval-Augmented Generation for Clinical Use Cases
Lahiri, A. K., & Hu, Q. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7030089 · Cited by 31
SPBERTQA: A Two-Stage Question Answering System Based on Sentence Transformers for Medical Texts
Nguyen, N. T. H., Ha, P. P.-D., Nguyen, L. T., Nguyen, K. V., & Nguyen, N. L.-T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-10986-7_30 · Cited by 31
Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey
Pei, Q., Zhou, Z., Gao, K., Jinhua, Z., Wang, Y., Wang, Z., Qin, T., Lijun, W., & Yan, R. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.48550/arxiv.2403.01528 · Cited by 31
Gao, K., He, S., He, Z., Lin, J., Pei, Q., Shao, J., & Zhang, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.14149 · Cited by 32
MEDITRON: Open Medical Foundation Models Adapted for Clinical PracticePreprint
Chen, Z., Romanou, A., Bonnet, A., Hernández-Cano, A., AlKhamissi, B., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4139743/v1 · Cited by 31
Large language model trained on clinical oncology data predicts cancer progression
Zhu, M., Lin, H., Jiang, J., Jinia, A. J., Jee, J., Pichotta, K., Waters, M., Rose, D., Schultz, N., Chalise, S., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01780-2 · Cited by 30
“For the medical domain, our dataset selection was as follows: (1) PubMedQA 43 (link), which challenges models to generate answers from NIH PubMed abstracts to be compared with expert-curated responses, covering a broad spectrum of biomedical topics.”
Automating expert-level medical reasoning evaluation of large language models
Zhou, S., Xie, W., Li, J., Zhan, Z., Song, M., Yang, H., Espinoza, C., Welton, L., Mai, X., Jin, Y., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02208-7 · Cited by 30
Self-citationMedConceptsQA: Open source medical concepts QA benchmark
Shoham, O. B., & Rappoport, N. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109089 · Cited by 30
Model Merging and Safety Alignment: One Bad Model Spoils the Bunch
Hammoud, H. A. A. K., Michieli, U., Pizzati, F., Torr, P. H. S., Bibi, A., Ghanem, B., & Özay, M. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.762 · Cited by 30
Detecting Subtle Differences between Human and Model Languages Using Spectrum of Relative Likelihood
Xu, Y., Wang, Y., An, H., Liu, Z., & Li, Y. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.564 · Cited by 31
Self-citationLarge Language Models in Psychiatry: Current Applications, Limitations, and Future Scope
Liu, Z., Bao, Y., Zeng, S., Qian, R., Deng, M., Gu, A., Li, J., Wang, W., Cai, W., Li, W., et al. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020046 · Cited by 32
Self-citationXu, T., Chen, L., Hu, Z., & Li, B. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127582 · Cited by 30
Ngo, N. T., Nguyen, C. V., Dernoncourt, F., & Nguyen, T. H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.09213 · Cited by 31
ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical ReasoningPreprint
Sun, Y., Qian, X., Xu, W., Zhang, H., Xiao, C., Li, L., Zhao, D., Huang, W., Xu, T., Bai, Q., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.09513 · Cited by 30
MedGUIDE: Benchmarking Clinical Decision-Making in Large Language ModelsPreprint
Li, X., Gao, M., Hao, Y., Li, T., Wan, G., Wang, Z., & Wang, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.11613 · Cited by 30
PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient InteractionsPreprint
Kyung, D., Chung, H., Bae, S., Kim, J., Sohn, J. H., Kim, T., Kim, S., & Choi, E. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.17818 · Cited by 30
Sviridova, E., Yeginbergen, A., Estarrona, A., Cabrio, E., Villata, S., & Agerri, R. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1026 · Cited by 29
Benchmarking Machine Reading Comprehension: A Psychological Perspective
Sugawara, S., Stenetorp, P., & Aizawa, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.137 · Cited by 31
Sequence tagging for biomedical extractive question answering
Yoon, W., Jackson, R., Lagerberg, A., & Kang, J. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac397 · Cited by 29
A Dataset of Argumentative Dialogues on Scientific Papers
Ruggeri, F., Mesgar, M., & Gurevych, I. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.425 · Cited by 29
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models
Diao, S., Pan, R. N., Dong, H., Shum, K., Zhang, J., Xiong, W., & Zhang, T. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-demo.12 · Cited by 29
AMOR: A Recipe for Building Adaptable Modular Knowledge Agents Through Process Feedback
Guan, J., Wu, W., Wen, Z., Xu, P., Wang, H., & Huang, M. (2024) · neural information processing systems · doi:10.52202/079017-4007 · Cited by 29
Model Merging and Safety Alignment: One Bad Model Spoils the BunchPreprint
Hammoud, H. A. A. K., Michieli, U., Pizzati, F., Torr, P. H. S., Bibi, A., Ghanem, B., & Özay, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.14563 · Cited by 29
PhysicianBench: Evaluating LLM Agents in Real-World EHR EnvironmentsPreprint
Liu, R., Mohiuddin, I. Q., Schoeffler, A., Renduchintala, K., Nayak, A., Vemu, P. L., Vedak, S., Black, K. C., Havlik, J. L., Ogunmola, I., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.02240 · Cited by 29
HealthBench: Evaluating Large Language Models Towards Improved Human Health
Arora, R. K., Wei, J., Hicks, R. S., Bowman, P., Quiñonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.08775 · Cited by 654
“Medical benchmarks for LLMs. Early evaluations for LLMs for health focused most on medical exam questions (Jin et al., 2021, 2019; Pal et al., 2022).”
ExpertQA: Expert-Curated Questions and Attributed Answers
Malaviya, C., Lee, S., Chen, S., Sieber, E., Yatskar, M., & Roth, D. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.167 · Cited by 28
Bao, G., Zhao, Y., Teng, Z., Yang, L., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2310.05130 · Cited by 28
A Continued Pretrained LLM Approach for Automatic Medical Note Generation
Yuan, D., Rastogi, E., Naik, G., Rajagopal, S. P., Goyal, S., Zhao, F., Chintagunta, B., & Ward, J. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) · doi:10.18653/v1/2024.naacl-short.47 · Cited by 28
UniDCP: Unifying Multiple Medical Vision-Language Tasks via Dynamic Cross-Modal Learnable Prompts
Zhan, C., Zhang, Y., Lin, Y., Wang, G., & Wang, H. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3397191 · Cited by 28
3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark
Sviridov, I., Miftakhova, A., Tereshchenko, A., Zubkova, G., Blinov, P. D., & Савченко, А. А. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1353 · Cited by 29
Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding
Zhang, Y., Cheng, H., Shen, Z., Liu, X., Wang, Y., & Gao, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.820 · Cited by 28
Wang, G., Gao, M., Yang, S., Ya, Z., He, L., Huang, L., Xiao, H., Zhang, Y., Li, W., Chen, L., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.18274 · Cited by 28
xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
Cheng, X., Wang, X., Zhang, X., Ge, T., Chen, S.-Q., Wei, F., Zhang, H., & Zhao, D. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3476 · Cited by 27
Constructing Domain-Specific Evaluation Sets for LLM-as-a-judge
Raju, R., Jain, S., Li, B., Li, J. L., & Thakker, U. (2024) · Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U) · doi:10.18653/v1/2024.customnlp4u-1.14 · Cited by 27
YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering
D’Souza, J., Giglou, H. B., & Münch, Q. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.675 · Cited by 28
ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning
Sun, Y., Qian, X., Xu, W., Zhang, H., Xiao, C., Li, L., Zhao, D., Huang, W., Xu, T., Bai, Q., et al. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1344 · Cited by 27
Generating Biomedical Question Answering Corpora From Q&A Forums
Lamúrias, A., Sousa, D., & Couto, F. M. (2020) · IEEE Access · doi:10.1109/access.2020.3020868 · Cited by 27
Bailicai: A Domain-Optimized Retrieval-Augmented Generation Framework for Medical Applications
Cui, L., Liu, Y., Ouyang, C., Yu, Y., Zhang, G., Wan, Y., & Yang, F. (2026) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020097 · Cited by 28
Are we there yet? Exploring clinical domain knowledge of BERT models
Sushil, M., Šuster, S., & Daelemans, W. M. P. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.5 · Cited by 28
He, J., Ma, M. D., Fan, J., Roth, D., Wang, W. W., & Ribeiro, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.08475 · Cited by 28
Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewards
Wu, S., Sarthi, P., Zhao, S., Lee, A., Shandilya, H., Grobelnik, A. M., Choudhary, N., Huang, E., Subbian, K., Zhang, L., et al. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2507.03041 · Cited by 27
RA-DIT: Retrieval-Augmented Dual Instruction Tuning
Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01352 · Cited by 26
Meng, W., Li, Y., Chen, L., & Dong, Z. (2025) · Electronics · doi:10.3390/electronics14020386 · Cited by 26
MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation
Riaz, H., Bhabesh, S. S., Arannil, V., Ballesteros, M., & Horwood, G. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.962 · Cited by 26
Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
Wang, W., Ma, Z., Yu, G., Cheung, Y.-F., Ding, M., Liu, J., Chen, W., & Shen, L. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.1996 · Cited by 26
Walking a Tightrope – Evaluating Large Language Models in High-Risk Domains
Hung, C.-C., Rim, W. B., Frost, L., Bruckner, L., & Lawrence, C. (2023) · GenBench Workshop on Generalisation (Benchmarking) in NLP (GenBench) · doi:10.18653/v1/2023.genbench-1.8 · Cited by 27
KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Prompts
Coscia, A. J., & Endert, A. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3346713 · Cited by 27
Zarei, M., Zarei, M., Hamzehzadeh, S., Oliyaei, S. S. B., & Hosseini, M. (2024) · Shiraz E-Medical Journal · doi:10.5812/semj-145840 · Cited by 26
A Survey of Pre-trained Language Models for Processing Scientific TextPreprint
Ho, X., Nguyen, A.-K. D., Dao, A., Jiang, J., Chida, Y., Sugimoto, K., To, H. Q., Boudin, F., & Aizawa, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.17824 · Cited by 26
RobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy ResponsePreprint
Luo, J., Luo, X., Ding, K., Yuan, J., Xiao, Z., & Zhang, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.14922 · Cited by 26
Fleming-R1: Toward Expert-Level Medical Reasoning via Reinforcement LearningPreprint
Liu, C., Li, D., Shu, Y., Chen, R., Duan, D., Fang, T., & Dai, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.15279 · Cited by 26
Aspect-Based Sentiment Analysis of Patient Feedback Using Large Language Models
Alkhnbashi, O. S., Nassr, R. M., & Hammoudeh, M. A. A. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8120167 · Cited by 25
Yang, Y., Jin, Q., Zhu, Q., Wang, Z., Álvarez, F. E., Wan, N., Hou, B., & Lu, Z. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094851 · Cited by 25
Self-citationRestrepo, D. S., Wu, C., Tang, Z., Shuai, Z., Phan, T. N. M., Ding, J.-E., Dao, C.-T., Gallifant, J., Dychiao, R. G., Artiaga, J. C. M., et al. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i27.35053 · Cited by 26
Semi-supervised Fine-tuning for Large Language Models
Luo, J., Luo, X., Chen, X., Xiao, Z., Ju, W., & Zhang, M. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.151 · Cited by 25
Zhang, X., Wang, D., Wang, B., Dou, L., Lu, X., Xu, K., Wu, D., & Zhu, Q. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.199 · Cited by 25
Self-citationShao, S., Li, Y., Yao, H., Chen, Y., Yang, Y., & Qin, Z. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792196 · Cited by 27
Advances and challenges in artificial intelligence text generation
Li, B., Yang, P., Sun, Y., Hu, Z., & Yi, M. (2024) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.2300410 · Cited by 25
AI for Biomedicine in the Era of Large Language ModelsPreprint
Bi, Z., Dip, S. A., Hajialigol, D., Kommu, S., Liu, H., Meng, L., & Wang, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.15673 · Cited by 25
Does Biomedical Training Lead to Better Medical Performance?Preprint
Dada, A., Bauer, M., Contreras, A. B., Koraş, O. A., Seibold, C., Smith, K. E., & Kleesiek, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.04067 · Cited by 25
Ma, M. D., Ye, C., Yan, Y., Wang, X., Ping, P., Chang, T. S., & Wang, W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.09923 · Cited by 25
BioDEX: Large-Scale Biomedical Adverse Drug Event Extraction for Real-World Pharmacovigilance
D’Oosterlinck, K., Remy, F., Deleu, J., Demeester, T., Develder, C., Zaporojets, K., Ghodsi, A., Ellershaw, S., Collins, J., & Potts, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.896 · Cited by 24
A Survey of Large Language Model for Drug Research and Development
Guo, H., Xing, X., Zhou, Y., Jiang, W., Chen, X., Wang, T., Jiang, Z., Wang, Y., Hou, J., Jiang, Y., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3552256 · Cited by 24
Sun, Z., Yim, W.-W., Uzuner, Ö., Xia, F., & Yetişgen, M. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104866 · Cited by 26
PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactions
Kyung, D., Chung, H., Bae, S., Kim, J., Sohn, J. H., Kim, T., Kim, S., & Choi, E. (2025) · neural information processing systems · doi:10.52202/085713-0861 · Cited by 24
Data Augmentation for Biomedical Factoid Question Answering
Pappas, D., Malakasiotis, P., & Androutsopoulos, I. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.6 · Cited by 24
Healthcare: A Growing Role for Large Language Models and Generative AI
Pahune, S. A., & Rewatkar, N. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.55573 · Cited by 24
Survey of Large Language Models for Answering Questions Across Various Fields
Shailendra, P., Ghosh, R. C., Kumar, R., & Sharma, N. (2024) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs60874.2024.10717078 · Cited by 24
Leveraging Retrieval-Augmented Generation for Swahili Language Conversation Systems
Ndimbo, E. V., Luo, Q., Fernando, G., Xu, Y., & Wang, B. (2025) · Applied Sciences · doi:10.3390/app15020524 · Cited by 24
Implementation of large language models in electronic health records
Griot, M., Vanderdonckt, J., & Yuksel, D. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0001141 · Cited by 23
“While the relevance of MCQs to evaluate LLMs is debated [10,11], they have the benefit of being reproducible, easy to administer, and most importantly available [12–14].”
DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text
Yang, X., Cheng, W., Wu, Y., Petzold, L., Wang, W. Y., & Chen, H. (2023) · arXiv · doi:10.48550/arxiv.2305.17359 · Cited by 23
“5-turbo with known prompts, PubMedQA using gpt-3.”
MethodInfluentialMedCite: Can Language Models Generate Verifiable Text for Medicine?
Wang, X., Tan, M., Jin, Q., Xiong, G., Hu, Y., Zhang, A., Lu, Z., & Zhang, M. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.967 · Cited by 23
Self-citationDivide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG
Sun, X., Xie, J., Chen, Z., Liu, Q., Wu, S., Chen, Y., Song, B., Wang, Z., Wang, W., & Wang, L. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.561 · Cited by 23
Wang, S., Zhao, F., Bu, D., Lu, Y., Gong, M., Liu, H., Yang, Z., Zeng, X., Yuan, Z., Wan, B., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64142-2 · Cited by 23
FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models
Gao, Y., Scamarcia, M. R., Fernández-Marqués, J., Naseri, M. F., Ng, C., Stripelis, D., Li, Z., Shen, T., Bai, J., Chen, D., et al. (2025) · neural information processing systems · doi:10.52202/085713-1998 · Cited by 23
Self-citationWang, B., Xia, I., Zhang, Y., Wang, J., Ouyang, F., Han, S., Cohan, A., Yu, H., & Yao, Z. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.548 · Cited by 24
The AI Productivity Index (APEX)
Vidgen, B., Fennelly, A., Pinnix, E., Mahapatra, C., Richards, Z., Bridges, A., Huang, C., Hunsberger, B., Zafar, F., Foody, B., et al. (2026) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i4.17205 · Cited by 23
Kanakarajan, K. R., & Sankarasubbu, M. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.137 · Cited by 23
Xu, S., Chen, M., & Chen, S. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5678-0_34 · Cited by 23
Nassr, R. M., Alkhnbashi, O. S., & Hammoudeh, M. A. A. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8110157 · Cited by 23
Comparing Knowledge Sources for Open-Domain Scientific Claim Verification
Vladika, J., & Matthes, F. (2024) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2024.eacl-long.128 · Cited by 23
Wang, J., Ning, H., Peng, Y., Wei, Q., Tesfai, D., Mao, W., Zhu, T., & Huang, R. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.132914 · Cited by 23
Med-R 2 : Crafting Trustworthy LLM Physicians via Retrieval and Reasoning of Evidence-Based Medicine
Lu, K., Zheng, L., Pan, D., Zhang, S., Dong, G., Leng, H., Cui, B., wu, Z., & Zhang, W. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792076 · Cited by 26
Multi-hop Inference for Question-driven Summarization
Deng, Y., Zhang, W., & Lam, W. P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.547 · Cited by 23
Counterfactual can be strong in medical question and answering
Yang, Z., Liu, Y., Ouyang, C., Ren, L., & Wen, W. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103408 · Cited by 23
Metadata
- Authors
- Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X.
- Affiliations
- University of Pittsburgh (US)
- Carnegie Mellon University (US)
- Google (United States) (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 2567-2577
- OpenAlex
- W2970482702
- Cited by
- 685
- References
- 26
- Citations by year
- 2026: 188 · 2025: 215 · 2024: 107 · 2023: 75 · 2022: 36 · 2021: 39 · 2020: 25
- FWCI
- 75.51
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- question answering, biomedical research, PubMed, scientific publications, datasets, biomedical question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.