PubMedQA: A Dataset for Biomedical Research Question Answering
Open accessThe DOI matches its Crossref record; 24 of the 26 references it cites were found in a registry or an index.
Abstract
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, Xinghua Lu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“given that medical question datasets such as PubMedQA42 and MedQA43 consist of fixed question sets susceptible to memorization”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [43]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“All analyses were performed on publicly available, de-identified benchmark datasets—MIRAGE [23], PubMedQA [24], and the medical licensing examination questions of Katz et al. [25]—used in accordance with their respective licenses and terms of use.”
From The economics of accuracy for medical reasoning with large language models (2026) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“PubMedQA (Jin et al., 2019) is a biomedical research QA dataset under the MIT license. It contains 1,000 expert-annotated questions ( pqa\_labeled\_split ) and 211K machine-labeled questions from PubMed abstracts (the most widely used biomedical literature resource).”
From MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025) · cites this paper as Jin et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Notably, in healthcare, medical NLP benchmarks like MedQA7, PubMedQA8 and the Massive Multitask Language Understanding (MMLU) serve as the de-facto reporting standard for state-of-the-art medical LLMs9–11 with claims of ‘superhuman’ performance in patient-facing tasks12.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“The ground truth question-answer pairs are sourced from the pqa\_labeled split of PubMedQA (Jin et al., 2019), which contains 1,000 expert-annotated samples, supplemented with 9,000 instances randomly selected from the pqa\_artificial split.”
From MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025) · cites this paper as Jin et al., 2019
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 2×
“We fine-tune BioBERT (Lee et al., 2019) on Pub-MedQA as a baseline.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014) · Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.3115/v1/p14-5010 · Cited by 7,037
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“Unlike other QA datasets in which questions are asked by crowd-workers for existing contexts (Rajpurkar et al., 2016; Yang et al., 2018; Kočiskỳ et al., 2018), in PubMedQA contexts are generated to answer the questions and both are written by the same authors.”
BackgroundNatural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 6×
“, 2018), Natural Questions (Kwiatkowski et al., 2019), ShARC (Saeidi et al.”
BackgroundMethodInfluentialBoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 1,855
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 6×
“Supporting fact for the answer is highlighted . et al., 2018; Yang et al., 2018; Kwiatkowski et al., 2019), the largest annotated biomedical QA dataset, BioASQ (Tsatsaronis et al., 2015) has less than 3k training instances, most of which are simple factual questions.”
BackgroundMethodInfluentialSelf-citationEnhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113
Research works printed but not linked: 1
- 2012 Conference and Labs of the Evaluation ForumQuestion Answering For Machine Reading Evaluation (QA4MRE), Rome/Forner, J.[edit.]; ea , pages 1-14.
Cited by
Items 101–200 of 1,026
Instruction Pre-Training: Language Models are Supervised Multitask Learners
Cheng, D., Gu, Y., Huang, S., Bi, J., Huang, M., & Wei, F. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.148 · Cited by 112
Wang, F., Zhang, Z., Zhang, X., Wu, Z., Mo, T., Lu, Q., Wang, W., Li, R., Xu, J., Tang, X., et al. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3768165 · Cited by 106
Systematic Analysis of Retrieval-Augmented Generation-Based LLMs for Medical Chatbot Applications
Bora, A., & Cuayáhuitl, H. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6040116 · Cited by 61
A Survey on the Optimization of Large Language Model-based Agents
Du, S., Zhao, J., Shi, J., Xie, Z., Jiang, X., Bai, Y., & He, L. (2026) · ACM Computing Surveys · doi:10.1145/3789261 · Cited by 26
UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models
Dong, Y. R., Lin, H., Belkin, M., Huerta, R., & Vulić, I. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.444 · Cited by 106
Me-LLaMA: Foundation Large Language Models for Medical Applications
Xie, Q., Chen, Q., Chen, A., Peng, C., Hu, Y., Lin, F., Peng, X., Huang, J., Zhang, J., Keloth, V., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4240043/v1 · Cited by 56
Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answering
Yang, F., Zhao, P., Wang, Z., Wang, L., Qiao, B., Zhang, J., Garg, M., Lin, Q., Rajmohan, S., & Zhang, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.29 · Cited by 104
EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries
Kweon, S., Kim, J., Kwak, H., Cha, D., Yoon, H., Kim, K., Yang, J., Won, S., & Choi, E. (2024) · neural information processing systems · doi:10.52202/079017-3958 · Cited by 103
nach0: multimodal natural and chemical languages foundation model
Livne, M., Miftahutdinov, Z., Tutubalina, E. V., Kuznetsov, M., Polykovskiy, D. A., Brundyn, A., Jhunjhunwala, A., Costa, A. B., Aliper, A., Aspuru‐Guzik, A., et al. (2024) · Chemical Science · doi:10.1039/d4sc00966e · Cited by 101
Zhou, Y., Song, L., & Shen, J. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1298 · Cited by 97
Scientific Fact-Checking: A Survey of Resources and Approaches
Vladika, J., & Matthes, F. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.387 · Cited by 96
GPT-3 Models are Poor Few-Shot Learners in the Biomedical DomainPreprint
Moradi, M., Blagec, K., Haberl, F., & Samwald, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.02555 · Cited by 95
Application of large language models in medicine
Liu, F., Zhou, H., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Hua, Y., Zhou, P., et al. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00279-5 · Cited by 94
Tian, Y., Han, Y., Chen, X., Wang, W. W., & Chawla, N. V. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703577 · Cited by 97
Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?
Jeong, D. P., Garg, S., Lipton, Z. C., & Oberst, M. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.677 · Cited by 57
Autonomous medical evaluation for guideline adherence of large language models
Fast, D., Adams, L. C., Busch, F., Fallon, C., Huppertz, M., Siepmann, R. M., Prucker, P., Bayerl, N., Truhn, D., Makowski, M. R., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01356-6 · Cited by 96
Gong, E. J., Bang, C. S., Lee, J. J., & Baik, G. H. (2025) · Journal of Medical Internet Research · doi:10.2196/84120 · Cited by 96
Yang, H., Li, M., Zhou, H., Xiao, Y., Fang, Q., Zhou, S., & Zhang, R. (2025) · Journal of Medical Internet Research · doi:10.2196/70080 · Cited by 60
CBLUE: A Chinese Biomedical Language Understanding Evaluation BenchmarkPreprint
Zhang, N., Chen, M., Bi, Z., Liang, X., Li, L., Shang, X., Yin, K., Tan, C., Xu, J., Huang, F., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08087 · Cited by 89
Artificial Intelligence in Action: Addressing the COVID-19 Pandemic with Natural Language Processing
Chen, Q., Leaman, R., Allot, A., Luo, L., Wei, C.-H., Yan, S., & Lu, Z. (2021) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-021821-061045 · Cited by 86
“According to . . . ”: Prompting Language Models Improves Quoting from Pre-Training Data
Weller, O., Marone, M., Weir, N., Lawrie, D. J., Khashabi, D., & Durme, B. V. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2024.eacl-long.140 · Cited by 67
Efficient Inference for Large Reasoning Models: A Survey
Liu, Y., Wu, J. F., He, Y., Gong, R., Xia, J., Li, L., Gao, H., Chen, H., Bi, B., Zhang, J., et al. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3701928 · Cited by 88
Chow, J. C. L., & Li, K. (2025) · Information · doi:10.3390/info16070549 · Cited by 84
AgentMD: Empowering language agents for risk prediction with large-scale clinical tool learning
Jin, Q., Wang, Z., Yang, Y., Zhu, Q., Wright, D., Huang, T. Y., Khandekar, N., Wan, N., Ai, X., Wilbur, W. J., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64430-x · Cited by 87
Self-citationPal, A., & Sankarasubbu, M. (2024) · Proceedings of the 6th Clinical Natural Language Processing Workshop · doi:10.18653/v1/2024.clinicalnlp-1.3 · Cited by 37
Abdollahi, M., Yeganli, S. F., Baharloo, M. M., & Baniasadi, A. (2024) · Electronics · doi:10.3390/electronics14010120 · Cited by 87
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
Boizard, N., Haddad, K. E., Hudelot, C., & Colombo, P. (2025) · Transactions on machine learning research · doi:10.48550/arxiv.2402.12030 · Cited by 85
Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
Ankner, Z., Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L., & Paul, M. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2405.20541 · Cited by 107
Song, Y., Miret, S., & Liu, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.201 · Cited by 83
FedSecurity: A Benchmark for Attacks and Defenses in Federated Learning and Federated LLMs
Han, S., Buyukates, B., Hu, Z., Jin, H., Jin, W., Sun, L., Wang, X., Wu, W., Xie, C., Yao, Y., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671545 · Cited by 84
Leveraging long context in retrieval augmented language models for medical question answering
Zhang, G., Xu, Z., Jin, Q., Chen, F., Fang, Y., Liu, Y., Rousseau, J. F., Xu, Z., Lu, Z., Weng, C., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01651-w · Cited by 81
“Consider the question with ID 18507507 in PubMedQA, “ The promise of specialty pharmaceuticals: are they worth the price?”
InfluentialSelf-citationMDAgents: an adaptive collaboration of LLMs for medical decision-making
Kim, Y., Park, C., Jeong, H., Chan, Y., Xu, X., McDuff, D., Lee, H., Ghassemi, M., Breazeal, C., & Park, H. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2522 · Cited by 81
“Options: A: 1st pharyngeal arch, B: 2nd pharyngeal arch, C: 3rd pharyngeal arch, D: 4th pharyngeal arch, E: 6th pharyngeal arch [36].”
BackgroundFundamental Capabilities and Applications of Large Language Models: A Survey
Li, J., Gao, Y., Yang, Y., Bai, Y., Zhou, X., Li, Y., Sun, H., Liu, Y., Si, X., Ye, Y., et al. (2025) · ACM Computing Surveys · doi:10.1145/3735632 · Cited by 82
ThoughtSource: A central hub for large language model reasoning data
Ott, S., Hebenstreit, K., Liévin, V., Hother, C. E., Moradi, M., Mayrhauser, M., Praas, R., Winther, O., & Samwald, M. (2023) · Scientific Data · doi:10.1038/s41597-023-02433-3 · Cited by 49
MAGE: Machine-generated Text Detection in the Wild
Li, Y., Li, Q., Cui, L., Bi, W., Wang, Z., Wang, L., Yang, L., Shi, S., & Zhang, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.3 · Cited by 79
Jahan, I., Laskar, M. T. R., Peng, C., & Huang, J. (2023) · The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks · doi:10.18653/v1/2023.bionlp-1.30 · Cited by 40
Large Language Models in Medical Diagnostics: Scoping Review With Bibliometric Analysis
Su, H., Sun, Y., Li, R., Zhang, A., Yang, Y., Xiao, F., Duan, Z., Chen, J., Hu, Q., Yang, T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/72062 · Cited by 81
Machine Reading Comprehension: The Role of Contextualized Language Models and BeyondPreprint
Zhang, Z., Zhao, H., & Wang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06249 · Cited by 80
The rise of small language models in healthcare: A comprehensive survey
Garg, M., Raza, S., Rayana, S., Liu, X., & Sohn, S. (2026) · Computer Science Review · doi:10.1016/j.cosrev.2026.100999 · Cited by 81
UnifiedQA-v2: Stronger Generalization via Broader Cross-Format TrainingPreprint
Khashabi, D., Kordi, Y., & Hajishirzi, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.12359 · Cited by 79
Tokenizer Choice For LLM Training: Negligible or Crucial?Preprint
Ali, M., Fromm, M., Thellmann, K., Rutmann, R., Lübbering, M., Leveling, J., Klug, K., Ebert, J., Doll, N., Buschhoff, J., et al. (2023) · NAACL-HLT · doi:10.48550/arxiv.2310.08754 · Cited by 160
Parcae: Scaling Laws For Stable Looped Language ModelsPreprint
Prairie, H., Novack, Z., Berg-Kirkpatrick, T., & Fu, D. Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.12946 · Cited by 83
Li, J., Yang, Y., Bai, Y., Zhou, X., Li, Y., Sun, H., Liu, Y., Si, X., Ye, Y., Wu, Y., et al. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.599 · Cited by 77
Promptcblue: A Chinese Prompt Tuning Benchmark for the Medical DomainPreprint
Zhu, W., Wang, X., Zheng, H., Chen, M., & Tang, B. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4685921 · Cited by 77
Language Models Hallucinate, but May Excel at Fact Verification
Guan, J., Dodge, J., Wadden, D., Huang, M., & Peng, H. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.62 · Cited by 77
Large language models in biomedicine and healthcare
Zhou, J., Li, H., Chen, S., Chen, Z., Han, Z., & Gao, X. (2025) · Yearbook of pediatric endocrinology · doi:10.1038/s44387-025-00047-1 · Cited by 77
Bias Patterns in the Application of LLMs for Clinical Decision Support
Poulain, R., Adiba, F. I., Fayyaz, H., & Beheshti, R. (2024) · Delaware journal of public health · doi:10.32481/djph.2026.03.10 · Cited by 52
Scaling Large Language Models for Next-Generation Single-Cell AnalysisPreprint
Rizvi, S. A., Levine, D., Patel, A. M., Zhang, S., Wang, E., Perry, C. J., Vrkic, I., Constante, N. M., Fu, Z., He, S., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.04.14.648850 · Cited by 76
Medical foundation large language models for comprehensive text analysis and beyond
Xie, Q., Chen, Q., Chen, A., Peng, C., Hu, Y., Lin, F., Peng, X., Huang, J., Zhang, J., Keloth, V., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01533-1 · Cited by 74
An automated framework for assessing how well LLMs cite relevant medical references
Wu, K., Wu, E., Wei, K., Zhang, A., Casasola, A., Nguyen, T., Riantawan, S., Shi, P., Ho, D., & Zou, J. (2025) · Nature Communications · doi:10.1038/s41467-025-58551-6 · Cited by 72
MLEC-QA: A Chinese Multi-Choice Biomedical Question Answering Dataset
Li, J., Zhong, S., & Chen, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.698 · Cited by 72
BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology
O’Donoghue, O., Shtedritski, A., Ginger, J., Abboud, R., Ghareeb, A. E., & Rodriques, S. G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.162 · Cited by 74
Almanac: Retrieval-Augmented Language Models for Clinical MedicinePreprint
Zakka, C., Chaurasia, A., Shad, R., Dalal, A. R., Kim, J. L., Moor, M., Alexander, K. M., Ashley, E. A., Boyd, J. H., Boyd, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.01229 · Cited by 72
LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-TuningPreprint
Pan, R., Liu, X., Diao, S., Pi, R., Zhang, J., Han, C., & Zhang, T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.17919 · Cited by 71
Moëll, B., & Aronsson, F. S. (2025) · Journal of Medical Internet Research · doi:10.2196/75849 · Cited by 71
LongHealth: A Question Answering Benchmark with Long Clinical Documents
Adams, L. C., Busch, F., Han, T., Excoffier, J.-B., Ortala, M., Löser, A., Aerts, H. J. W. L., Kather, J. N., Truhn, D., & Bressem, K. K. (2025) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-025-00204-w · Cited by 70
Shao, Y., Tang, X., Sohn, J., Chen, J., Liao, Y., Zhang, J., Xiang, J., Wu, F., Zhao, Y., Wu, C., et al. (2026) · Patterns · doi:10.1016/j.patter.2026.101658 · Cited by 69
CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning
Cui, H., Shamsi, Z., Cheon, G., Ma, X., Li, S., Tikhanovskaya, M., Norgaard, P., Mudur, N., Plomecka, M., Raccuglia, P., et al. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2503.13517 · Cited by 70
Xu, R., Cui, H., Yu, Y., Kan, X., Shi, W., Zhuang, Y., Wang, M. D., Jin, W., Ho, J. C., & Yang, C. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.916 · Cited by 71
English Machine Reading Comprehension Datasets: A Survey
Dzendzik, D., Foster, J., & Vogel, C. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.693 · Cited by 69
RadioRAG: Online Retrieval–Augmented Generation for Radiology Question Answering
Arasteh, S. T., Lotfinia, M., Bressem, K. K., Siepmann, R. M., Adams, L. C., Ferber, D., Kuhl, C. K., Kather, J. N., Nebelung, S., & Truhn, D. (2025) · Radiology Artificial Intelligence · doi:10.1148/ryai.240476 · Cited by 67
BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrievers
Xu, R., Shi, W., Yu, Y., Zhuang, Y., Zhu, Y., Wang, M. D., Ho, J. C., Zhang, C., & Yang, C. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1241 · Cited by 68
Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models
Wang, L., Zeng, X., Guo, J., Wong, K., & Gottlob, G. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i1.32068 · Cited by 69
Kanithi, P., Christophe, C., Pimentel, M. A., Raha, T., Munjal, P., Saadi, N., Javed, H. A., Maslenkova, S., Hayat, N., Rajan, R., et al. (2026) · Transactions on machine learning research · doi:10.48550/arxiv.2409.07314 · Cited by 68
AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset
Nimo, C., Olatunji, T., Owodunni, A. T., Abdullahi, T., Ayodele, E., Sanni, M., Aka, E. C., Omofoye, F., Yuehgoh, F., Faniran, T., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.96 · Cited by 72
Pre-trained language models in medicine: A survey
Luo, X., Deng, Z., Yang, B., & Luo, M. Y. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102904 · Cited by 66
ProtT3: Protein-to-Text Generation for Text-based Protein Understanding
Liu, Z., Zhang, A., Fei, H., Zhang, E., Wang, X., Kawaguchi, K., & Chua, T. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.324 · Cited by 67
Self-citationECG-LM: Understanding Electrocardiogram with a Large Language Model
Yang, K., Hong, M., Zhang, J., Luo, Y., Zhao, S., Zhang, O., Yu, X., Zhou, J., Yang, L., Zhang, P., et al. (2025) · Health Data Science · doi:10.34133/hds.0221 · Cited by 67
Xu, R., Liu, H., Nag, S., Dai, Z., Xie, Y., Tang, X., Luo, C., Li, Y., Ho, J. C., Yang, C., et al. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.575 · Cited by 66
Wan, Y., Liu, Y., Ajith, A., Grazian, C., Hoex, B., Zhang, W., Kit, C., Xie, T., & Foster, I. T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.09939 · Cited by 68
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning
Parmar, M., Mishra, S., Purohit, M., Luo, M., Mohammad, M., & Baral, C. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.10 · Cited by 20
Arsenyan, V., Bughdaryan, S., Shaya, F., Small, K. W., & Shahnazaryan, D. (2024) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2024.bionlp-1.23 · Cited by 67
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation ModelsPreprint
Diao, S., Pan, R. N., Dong, H., Shum, K., Zhang, J., Xiong, W., & Zhang, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.12420 · Cited by 67
Evaluating and Benchmarking the System One Model JevPreprint
Deußer, T., Ramamurthy, R., Sparrenberg, L., & Sifa, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.37647 · Cited by 69
ThoughtSource: A central hub for large language model reasoning data
Ott, S., Hebenstreit, K., Liévin, V., Hother, C. E., Moradi, M., Mayrhauser, M., Praas, R., Winther, O., & Samwald, M. (2023) · arXiv · doi:10.48550/arxiv.2301.11596 · Cited by 65
Generative AI in the context of assistive technologies: Trends, limitations and future directions
Fu, B., Hadid, A., & Damer, N. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105347 · Cited by 66
Benchmarking large language model-based agent systems for clinical decision tasks
Liu, Y., Carrero, Z. I., Jiang, X., Ferber, D., Wölflein, G., Zhang, L., Jayabalan, S., Lenz, T., Hui, Z., & Kather, J. N. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02443-6 · Cited by 37
HealthFC: Verifying Health Claims with Evidence-Based Medical Fact-Checking
Vladika, J., Schneider, P., & Matthes, F. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3soozndxgt86 · Cited by 67
KaLM-Embedding: Superior Training Data Brings A Stronger Embedding ModelPreprint
Hu, X., Shan, Z., Zhao, X., Sun, Z., Liu, Z., Li, D., Ye, S., Wei, X., Chen, Q., Hu, B., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.01028 · Cited by 64
Self-citationMedExpQA: Multilingual benchmarking of Large Language Models for Medical Question Answering
Alonso, I., Oronoz, M., & Agerri, R. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102938 · Cited by 62
“For our experiments we pick the 13B parameter variant of this model which outperforms LLaMA-2 (Touvron et al., 2023), Med-Alpaca (Han et al., 2023), and Chat-Doctor (Li et al., 2023) in various Medical QA tasks including MedQA (Jin et al., 2021), MedMCQA (Pal et al., 2022b), and PubMedQA (Jin et al., 2019).”
BackgroundMethodTesting Your Question Answering Software via Asking Recursively
Chen, S., Jin, S., & Xie, X. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678670 · Cited by 63
MedAdapter: Efficient Test-Time Adaptation of Large Language Models Towards Medical Reasoning
Shi, W., Xu, R., Zhuang, Y., Yu, Y., Sun, H., Wu, H., Yang, C., & Wang, M. D. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1244 · Cited by 62
Almanac: Retrieval-Augmented Language Models for Clinical Medicine
Hiesinger, W., Zakka, C., Chaurasia, A., Shad, R., Dalal, A., Kim, J., Moor, M., Alexander, K., Ashley, E., Boyd, J., et al. (2023) · Research Square · doi:10.21203/rs.3.rs-2883198/v1 · Cited by 36
Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation
Nayak, N. V., Nan, Y., Trost, A., & Bach, S. H. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.748 · Cited by 62
MultiVerS: Improving scientific claim verification with weak supervision and full-document context
Wadden, D., Lo, K., Wang, L., Cohan, A., Beltagy, I., & Hajishirzi, H. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.6 · Cited by 60
“F EVER (Thorne et al., 2018) P UBMED QA (Jin et al., 2019) was released to facilitate question-answering over biomedical research abstracts.”
BackgroundYifei, L. S., Chang, A., Malaviya, C., & Yatskar, M. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.732 · Cited by 61
Holistic evaluation of large language models for medical tasks with MedHELM
Bedi, S., Cui, H., Fuentes, M., Unell, A., Wornow, M., Banda, J. M., Kotecha, N., Keyes, T., Mai, Y., Oez, M., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04151-2 · Cited by 59
Evidence Extraction to Validate Medical Claims in Fake News Detection
Deka, P., Jurek, A., & Deepak, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20627-6_1 · Cited by 59
Adversarial prompt and fine-tuning attacks threaten medical large language models
Yang, Y., Jin, Q., Huang, F., & Lu, Z. (2025) · Nature Communications · doi:10.1038/s41467-025-64062-1 · Cited by 59
Self-citationGraph-Based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Survey
Zhu, Z., Huang, T., Wang, K., Ye, J., Chen, X., & Luo, S. (2026) · ACM Computing Surveys · doi:10.1145/3795880 · Cited by 60
Chen, K., Li, X., Yang, T., Wang, H., Dong, W., & Gao, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.13856 · Cited by 59
Making Large Language Models Better Data Creators
Lee, D., Pujara, J., Sewak, M., White, R. W., & Jauhar, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.948 · Cited by 58
Applications of Generative AI in Healthcare: algorithmic, ethical, legal and societal considerationsPreprint
Okonji, O. R., Yunusov, K., & Gordon, B. (2024) · doi:10.36227/techrxiv.171527587.75649430/v1 · Cited by 58
Jahan, I., Laskar, M. T. R., Peng, C., & Huang, J. (2023) · arXiv · doi:10.48550/arxiv.2306.04504 · Cited by 57
“(iii) Question Answering: For the question-answering task, we evaluate the performance of ChatGPT on the PubMedQA dataset (Jin et al., 2019).”
MethodSeveral Categories of Large Language Models (LLMs): A Short Survey
Pahune, S. A., & Chandrasekharan, M. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.54677 · Cited by 58
Toward universal steering and monitoring of AI models
Beaglehole, D., Radhakrishnan, A., Boix-Adserà, E., & Belkin, M. (2026) · Science · doi:10.1126/science.aea6792 · Cited by 59
Know Your Limits: A Survey of Abstention in Large Language Models
Wen, B., Yao, J., Feng, S., Xu, C., Tsvetkov, Y., Howe, B., & Wang, L. L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00754 · Cited by 56
Best Practices for Large Language Models in Radiology
Blüthgen, C., Veen, D. V., Zakka, C., Link, K. E., Fanous, A. H., Daneshjou, R., Frauenfelder, T., Langlotz, C. P., Gatidis, S., & Chaudhari, A. S. (2025) · Radiology · doi:10.1148/radiol.240528 · Cited by 56
Automatic Interactive Evaluation for Large Language Models with State Aware Patient SimulatorPreprint
Liao, Y., Meng, Y., Wang, Y., Li, L., Liu, H., Wang, Y., & Wang, Y. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4890649 · Cited by 58
PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
Lála, J., O'Donoghue, O., Shtedritski, A., Cox, S., Rodriques, S. G., & White, A. D. (2023) · arXiv · doi:10.48550/arxiv.2312.07559 · Cited by 55
“We modified PubMedQA [21] to remove the provided context (so it is closed-book) and found PaperQA beats GPT-4 by 30 points (57.9% to 86.3%).”
BackgroundMethodInfluential
Metadata
- Authors
- Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X.
- Affiliations
- University of Pittsburgh (US)
- Carnegie Mellon University (US)
- Google (United States) (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 2567-2577
- OpenAlex
- W2970482702
- Cited by
- 685
- References
- 26
- Citations by year
- 2026: 188 · 2025: 215 · 2024: 107 · 2023: 75 · 2022: 36 · 2021: 39 · 2020: 25
- FWCI
- 75.51
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- question answering, biomedical research, PubMed, scientific publications, datasets, biomedical question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.