PubMedQA: A Dataset for Biomedical Research Question Answering
Open accessThe DOI matches its Crossref record; 24 of the 26 references it cites were found in a registry or an index.
Abstract
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, Xinghua Lu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“given that medical question datasets such as PubMedQA42 and MedQA43 consist of fixed question sets susceptible to memorization”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [43]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“All analyses were performed on publicly available, de-identified benchmark datasets—MIRAGE [23], PubMedQA [24], and the medical licensing examination questions of Katz et al. [25]—used in accordance with their respective licenses and terms of use.”
From The economics of accuracy for medical reasoning with large language models (2026) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“PubMedQA (Jin et al., 2019) is a biomedical research QA dataset under the MIT license. It contains 1,000 expert-annotated questions ( pqa\_labeled\_split ) and 211K machine-labeled questions from PubMed abstracts (the most widely used biomedical literature resource).”
From MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025) · cites this paper as Jin et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Notably, in healthcare, medical NLP benchmarks like MedQA7, PubMedQA8 and the Massive Multitask Language Understanding (MMLU) serve as the de-facto reporting standard for state-of-the-art medical LLMs9–11 with claims of ‘superhuman’ performance in patient-facing tasks12.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“The ground truth question-answer pairs are sourced from the pqa\_labeled split of PubMedQA (Jin et al., 2019), which contains 1,000 expert-annotated samples, supplemented with 9,000 instances randomly selected from the pqa\_artificial split.”
From MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025) · cites this paper as Jin et al., 2019
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 2×
“We fine-tune BioBERT (Lee et al., 2019) on Pub-MedQA as a baseline.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014) · Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.3115/v1/p14-5010 · Cited by 7,037
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“Unlike other QA datasets in which questions are asked by crowd-workers for existing contexts (Rajpurkar et al., 2016; Yang et al., 2018; Kočiskỳ et al., 2018), in PubMedQA contexts are generated to answer the questions and both are written by the same authors.”
BackgroundNatural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 6×
“, 2018), Natural Questions (Kwiatkowski et al., 2019), ShARC (Saeidi et al.”
BackgroundMethodInfluentialBoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 1,855
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 6×
“Supporting fact for the answer is highlighted . et al., 2018; Yang et al., 2018; Kwiatkowski et al., 2019), the largest annotated biomedical QA dataset, BioASQ (Tsatsaronis et al., 2015) has less than 3k training instances, most of which are simple factual questions.”
BackgroundMethodInfluentialSelf-citationEnhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113
Research works printed but not linked: 1
- 2012 Conference and Labs of the Evaluation ForumQuestion Answering For Machine Reading Evaluation (QA4MRE), Rome/Forner, J.[edit.]; ea , pages 1-14.
Cited by
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170
“ation metric (average micro F1 across the ten cancer hallmarks), and create the train/dev/test split, as they are not available previously.3 4.6 Question Answering (QA) PubMedQA. The PubMedQA dataset [24] contains a set of research questions, each with a reference text from a PubMed abstract as well as an annotated la…”
MethodInfluentialA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
“The latter result was state of the art, although we caution that PubMedQA’s test set is small (500 examples), and remaining failures of Med-PaLM 2 and other strong models appear to be largely attributable to label noise intrinsic in the dataset (especially given human performance is 78.0% 17 ).”
InfluentialBioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150
“…KD-DTI [14], BC5CDR [13], DDI [15] Question Answering QA-Net [26], LUKE [27], BERT [2], PubMedBERT [9], BioELECTRA [28], LinkBERT [29] PubMedQA [16], BioASQ [30, 31] Document Classification BERT [2], BlueBERT [8], SciBERT [18], SPECTER [32], PubMedBERT [9], BioELECTRA [28], LinkBERT [29] HoC…”
BackgroundMethodInfluentialCan AI-Generated Text be Reliably Detected?
Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11156 · Cited by 810
“In §5, we show that all other detectors can also be easily spoofed to make genuine human text detected as AI-generated.”
BackgroundInfluentialChen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., & Liu, Z. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.137 · Cited by 757
“For English, we incorporate 8 datasets, including HotpotQA (Yang et al., 2018), Trivi-aQA (Joshi et al., 2017), NQ (Kwiatkowski et al., 2019), MS MARCO (Nguyen et al., 2016), COL-IEE (Kim et al., 2023), PubMedQA (Jin et al., 2019), SQuAD (Rajpurkar et al., 2016), and NLI data from SimCSE (Gao et al., 2021b).”
Self-citationAn introduction to Deep Learning in Natural Language Processing: Models, techniques, and tools
Lauriola, I., Lavelli, A., & Aiolli, F. (2022) · Neurocomputing · doi:10.1016/j.neucom.2021.05.103 · Cited by 691
DataComp-LM: In search of the next generation of training sets for language models
Li, J. J.-P., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S. Y., Bansal, H., Guha, E., Keh, S. S., Arora, K., et al. (2024) · neural information processing systems · doi:10.52202/079017-0455 · Cited by 636
Knowledge Unlearning for Mitigating Privacy Risks in Language Models
Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., & Seo, M. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.805 · Cited by 80
Evaluating Large Language Models: A Comprehensive SurveyPreprint
Guo, Z., Jin, R., Chuang, L. I. U., Huang, Y., Shi, D., Supryadi, Yu, L., Liu, Y., Li, J., Xiong, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.19736 · Cited by 479
RedPajama: an Open Dataset for Training Large Language Models
Weber, M., Fu, D. Y., Anthony, Q. G., Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., et al. (2024) · neural information processing systems · doi:10.52202/079017-3697 · Cited by 436
Almanac — Retrieval-Augmented Language Models for Clinical Medicine
Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., et al. (2024) · NEJM AI · doi:10.1056/aioa2300068 · Cited by 424
Large language models in healthcare and medical domain: A review
Nazi, Z. A., & Peng, W. (2024) · Informatics · doi:10.3390/informatics11030057 · Cited by 415
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408
“We compared GatorTronGPT with six existing transformer models using three widely used benchmark dataset including PubMedQA 47 — a biomedical question answering dataset collected from PubMed abstracts, which requires answering questions with ‘ yes/no/maybe ’ ; MedMCQA 48 — a large-scale multi-choice…”
InfluentialSearching for Best Practices in Retrieval-Augmented Generation
Wang, X., Wang, Z., Gao, X., Zhang, F., Wu, Y., Xu, Z., Shi, T., Wang, Z., Li, S., Qian, Q., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.981 · Cited by 111
Fact or Fiction: Verifying Scientific Claims
Wadden, D., Lin, S., Lo, K., Wang, L. L., van Zuylen, M., Cohan, A., & Hajishirzi, H. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.609 · Cited by 382
“PubMedQA (Jin et al., 2019) Citing”
BackgroundDeep Bidirectional Language-Knowledge Graph Pretraining
Yasunaga, M., Bosselut, A., Ren, H., Zhang, X., Manning, C. D., Liang, P., & Leskovec, J. (2022) · neural information processing systems · doi:10.52202/068431-2704 · Cited by 103
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359
Self-citationArcee’s MergeKit: A Toolkit for Merging Large Language Models
Goddard, C., Siriwardhana, S., Ehghaghi, M., Meyers, L., Karpukhin, V., Benedict, B., McQuade, M., & Solawetz, J. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-industry.36 · Cited by 353
A Survey on Retrieval-Augmented Text Generation for Large Language Models
Huang, Y., & Huang, J. X. (2026) · ACM Computing Surveys · doi:10.1145/3805774 · Cited by 352
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343
“45 The three datasets are summarized in Table 2.”
BackgroundCan large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2022) · arXiv · doi:10.48550/arxiv.2207.08143 · Cited by 343
LinkBERT: Pretraining Language Models with Document Links
Yasunaga, M., Leskovec, J., & Liang, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.551 · Cited by 327
QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2023) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 156
OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning
Ye, R., Wang, W., Chai, J., Li, D., Li, Z., Xu, Y., Du, Y., Wang, Y., & Chen, S. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671582 · Cited by 327
Benchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.372 · Cited by 310
Self-citationPMC-LLaMA: toward building open-source language models for medicine
Wu, C., Lin, W., Zhang, X., Zhang, Y., Xie, W., & Wang, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae045 · Cited by 300
Large language models surpass human experts in predicting neuroscience results
Luo, X., Rechardt, A., Sun, G., Nejad, K. K., Yáñez, F., Yilmaz, B., Lee, K., Cohen, A. O., Borghesani, V., Pashkov, A., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02046-9 · Cited by 139
Datasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 113
Scientific Large Language Models: A Survey on Biological & Chemical Domains
Zhang, Q., Ding, K., Lv, T., Wang, X., Yin, Q., Zhang, Y., Yu, J., Wang, Y., Li, X., Xiang, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3715318 · Cited by 122
AMMU: A survey of transformer-based biomedical pretrained language models
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103982 · Cited by 278
CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark
Zhang, N., Chen, M., Bi, Z., Liang, X., Li, L., Shang, X., Yin, K., Tan, C., Xu, J., Huang, F., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.544 · Cited by 152
BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains
Labrak, Y., Bazoge, A., Morin, E., Gourraud, P.-A., Rouvier, M., & Dufour, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.348 · Cited by 269
Retrieval augmented generation for large language models in healthcare: A systematic review
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000877 · Cited by 267
“Other evaluation datasets for QA tasks include MedMCQA [34], Pub-MedQA [35] and MedQA [36].”
Towards Mitigating LLM Hallucination via Self Reflection
Ji, Z., Yu, T., Xu, Y., Lee, N., Ishii, E., & Fung, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.123 · Cited by 253
Medical large language models are vulnerable to data-poisoning attacks
Alber, D. A., Yang, Z., Alyakin, A., Yang, E., Rai, S., Valliani, A. A., Zhang, J., Rosenbaum, G. R., Amend-Thomas, A. K., Kurland, D. B., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03445-1 · Cited by 242
A Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05112 · Cited by 240
“In detail, MedPaLM [14], MedPaLM 2 [15], Clinical Camel [18], evaluated their performances on the multiple QA datasets, e.g., MedQA (USMLE) [22], MedMCQA [111], PubMedQA [112], and MMLU [113].”
MethodVision-language models for medical report generation and visual question answering: a review
Hartsock, I., & Rasool, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1430984 · Cited by 229
Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
Yang, C., Zhu, Y. T., Wang, L., Wang, Y., Chen, Q., Gao, C., Yan, B., & Chen, Y. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3699518 · Cited by 226
BioMedGPT: An Open Multimodal Large Language Model for BioMedicine
Luo, Y., Zhang, J., Fan, S., Yang, K., Hong, M., Wu, Y., Qiao, M., & Nie, Z. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3505955 · Cited by 227
MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter
Liu, Z., Li, S., Luo, Y., Fei, H., Cao, Y., Kawaguchi, K., Wang, X., & Chua, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.966 · Cited by 224
Self-citationChen, Q., Hu, Y., Peng, X., Xie, Q., Jin, Q., Gilson, A., Singer, M. B., Ai, X., Lai, P.-T., Wang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-56989-2 · Cited by 219
“1.”
Self-citationKim, S., Joo, S., Kim, D., Jang, J., Ye, S., Shin, J., & Seo, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.782 · Cited by 51
Document Understanding Dataset and Evaluation (DUDE)
Landeghem, J. V., Powalski, R., Tito, R., Jurkiewicz, D., Blaschko, M. B., Borchmann, Ł., Coustaty, M., Moens, S., Pietruszka, M., Ackaert, B., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01789 · Cited by 219
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Wong, D. F., & Chao, L. S. (2023) · arXiv · doi:10.48550/arxiv.2310.14724 · Cited by 212
“• PubMedQA [78]: This is a biomedical question-and-answer (QA) dataset sourced from PubMed.”
BackgroundBiomedical Question Answering: A Survey of Approaches and Challenges
Jin, Q., Yuan, Z., Xiong, G., Yu, Q., Ying, H., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2023) · ACM Computing Surveys · doi:10.1145/3490238 · Cited by 118
Self-citationRephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
Maini, P., Seto, S., Bai, R., Grangier, D., Zhang, Y., & Jaitly, N. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.757 · Cited by 204
SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research
Sun, L., Yang, H., Zhao, Z., Ma, D., Shen, Z., Chen, B., Chen, L., & Yu, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29872 · Cited by 206
A Survey on Large Language Models with some Insights on their Capabilities and LimitationsPreprint
Matarazzo, A., & Torlone, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.04040 · Cited by 206
A comprehensive survey of large language models and multimodal large language models in medicine
Xiao, H., Zhou, F., Liu, X., Liu, T., Li, Z., Liu, X., & Huang, X. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102888 · Cited by 113
A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law
Chen, Z. Z., Ma, J., Zhang, X., Hao, N., Yan, A., Nourbakhsh, A., Yang, X., McAuley, J., Petzold, L. R., & Wang, W. Y. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2405.01769 · Cited by 205
Transformers and large language models in healthcare: A review
Nerella, S., Bandyopadhyay, S., Zhang, J., Contreras, M., Siegel, S., Bumin, A., Silva, B., Sena, J., Shickel, B., Bihorac, A., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102900 · Cited by 197
“Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets ShARe/CLEF [79], DDI [87], CHEMPROT [71], i2b2 2010 [62], HoC [88], MedNLI [75] [89] Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing NER, RE, SS, DC, QA NCBI Disease [61], BC5CDR [63], BC2GM [65], JNLPBA [90],…”
InfluentialGraph Neural Prompting with Large Language Models
Tian, Y., Song, H., Wang, Z., Wang, H., Hu, Z., Wang, F., Chawla, N. V., & Xu, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29875 · Cited by 197
A Survey of LLM-based Agents in Medicine: How far are we from Baymax?
Wang, W., Ma, Z., Wang, Z., Wu, C., Ji, J., Chen, W., Li, X., & Yuan, Y. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.539 · Cited by 49
Esteva, A., Kale, A., Paulus, R., Hashimoto, K., Yin, W., Radev, D., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00437-0 · Cited by 109
A personal health large language model for sleep and fitness coaching
Khasentino, J., Belyaeva, A., Liu, X., Yang, Z., Furlotte, N. A., Lee, C., Schenck, E., Patel, Y., Cui, J., Schneider, L. D., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03888-0 · Cited by 193
A comprehensive evaluation of large Language models on benchmark biomedical text processing tasks
Jahan, I., Laskar, M. T. R., Peng, C., & Huang, J. X. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108189 · Cited by 106
BioELECTRA:Pretrained Biomedical text Encoder using Discriminators
Kanakarajan, K. R., Kundumani, B., & Sankarasubbu, M. (2021) · Proceedings of the 20th Workshop on Biomedical Language Processing · doi:10.18653/v1/2021.bionlp-1.16 · Cited by 112
Tokenizer Choice For LLM Training: Negligible or Crucial?
Ali, M., Fromm, M., Thellmann, K., Rutmann, R., Lübbering, M., Leveling, J., Klug, K., Ebert, J., Doll, N., Buschhoff, J. S., et al. (2024) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2024.findings-naacl.247 · Cited by 181
AI chatbots not yet ready for clinical use.
Au Yeung, J., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1161098 · Cited by 176
AI chatbots not yet ready for clinical use
Yeung, J. A., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · doi:10.1101/2023.03.02.23286705 · Cited by 176
Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models
Chen, Z., Xu, L., Zheng, H., Chen, L., Tolba, A. K. R., Zhao, L., Yu, K., & Feng, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.052618 · Cited by 180
Med42-v2: A Suite of Clinical LLMsPreprint
Christophe, C., Kanithi, P., Raha, T., Khan, S., & Pimentel, M. A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.06142 · Cited by 176
Pre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2024) · ACM Computing Surveys · doi:10.1145/3611651 · Cited by 169
Fine-tuning large neural language models for biomedical natural language processing
Tinn, R., Cheng, H., Gu, Y., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2023) · Patterns · doi:10.1016/j.patter.2023.100729 · Cited by 169
Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions
Xiong, G., Jin, Q., Wang, X., Zhang, M., Lu, Z., & Zhang, A. (2024) · Biocomputing 2025 · doi:10.1142/9789819807024_0015 · Cited by 64
Self-citationPenedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., & Launay, J. (2023) · arXiv · doi:10.48550/arxiv.2306.01116 · Cited by 164
“…(Amini et al., 2019) Multiple-choice question answering 20.0 ✓ PROST (Aroca-Ouellette et al., 2021) Paraphrase identification 50.0 ✓ PubMedQA (Jin et al., 2019) Multiple-choice question answering 50.0 ✓ SciQ (Welbl et al., 2017) Multiple-choice question answering 25.0 ✓ ✓ biggest clusters in…”
BackgroundInfluentialUltraMedical: Building Specialized Generalists in Biomedicine
Zhang, K., Zeng, S., Hua, E., Ding, N., Chen, Z.-R., Ma, Z., Li, H., Cui, G., Qi, B., Zhu, X., et al. (2024) · neural information processing systems · doi:10.52202/079017-0819 · Cited by 166
Self-citationWhat Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?Preprint
Wang, T., Roberts, A., Hesslow, D., Scao, T. L., Chung, H. W., Beltagy, I., Launay, J., & Raffel, C. A. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2204.05832 · Cited by 242
MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
Pal, A., Umapathi, L. K., & Sankarasubbu, M. (2022) · arXiv · doi:10.48550/arxiv.2203.14371 · Cited by 157
MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models
Han, T., Kumar, A., Agarwal, C., & Lakkaraju, H. (2024) · neural information processing systems · doi:10.52202/079017-1054 · Cited by 158
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2301.11305 · Cited by 151
“The most common method for zero-shot machine-generated text detection is evaluating the average per-token log probability of the generated text and thresholding (Solaiman et al., 2019; Gehrmann et al., 2019; Ippolito et al., 2020).”
BackgroundMethodInfluentialAn extensive benchmark study on biomedical text generation and mining with ChatGPT
Chen, Q., Sun, H., Liu, H., Jiang, Y., Ran, T., Jin, X., Xiao, X., Lin, Z., Chen, H., & Niu, Z. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad557 · Cited by 97
Enhancing diagnostic capability with multi-agents conversational large language models
Chen, X., Yi, H., You, M., Liu, W., Wang, L., Li, H., Zhang, X., Guo, Y., Fan, L., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01550-0 · Cited by 147
Towards building multilingual language model for medicine
Qiu, P., Wu, C., Zhang, X., Lin, W., Wang, H., Zhang, Y., Wang, Y., & Xie, W. (2024) · Nature Communications · doi:10.1038/s41467-024-52417-z · Cited by 147
“Replacing the English-central LLM with a multilingual one enables to make good use of world-wide data source, thus expanding the available multimodal training data, and improving the representation quality for other modalities.”
BackgroundMethodHe, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147
SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers
Pramanick, S., Chellappa, R., & Venugopalan, S. (2024) · neural information processing systems · doi:10.52202/079017-3773 · Cited by 147
GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation
Luo, L., Zhao, Z., Haffari, G., Phung, D., Gong, C., & Pan, S. (2025) · neural information processing systems · doi:10.52202/085713-1222 · Cited by 144
Brown, A., Roman, M., & Devereux, B. J. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9120320 · Cited by 149
BioInstruct: instruction tuning of large language models for biomedical natural language processing
Tran, H., Yang, Z., Yao, Z., & Yu, H. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae122 · Cited by 144
Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired SketchingPreprint
Aytes, S. A., Baek, J., & Hwang, S. J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.05179 · Cited by 142
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., & Gardner, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.365 · Cited by 138
Christophe, C., Kanithi, P., Munjal, P., Raha, T., Hayat, N., Rajan, R., Mahrooqi, A. A., Gupta, A., Salman, M., Gosal, G., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.14779 · Cited by 139
Taiyi: a bilingual fine-tuned large language model for diverse biomedical tasks
Luo, L., Ning, J., Zhao, Y., Wang, Z., Ding, Z., Chen, P., Fu, W., Han, Q., Xu, G., Qiu, Y., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae037 · Cited by 83
A Systematic Survey of Automatic Prompt Optimization Techniques
Ramnath, K., Zhou, K., Guan, S., Mishra, S. S., Qi, X., Shen, Z., Wang, S., Woo, S., Jeoung, S., Wang, Y., et al. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1681 · Cited by 135
Jeong, M., Sohn, J., Sung, M., & Kang, J. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae238 · Cited by 132
“Jeong et al.”
Recent Advances in Large Language Models for Healthcare
Nassiri, K., & Akhloufi, M. A. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4020062 · Cited by 135
MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
Khandekar, N., Jin, Q., Xiong, G., Dunn, S., Applebaum, S. S., Anwar, Z., Sarfo-Gyamfi, M., Safranek, C. W., Anwar, A. A., Zhang, A., et al. (2024) · neural information processing systems · doi:10.52202/079017-2690 · Cited by 131
Self-citationDiagnostic performances of GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro in “Diagnosis Please” cases
Sonoda, Y., Kurokawa, R., Nakamura, Y., Kanzawa, J., Kurokawa, M., Ohizumi, Y., Gonoi, W., & Abe, O. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-024-01619-y · Cited by 100
Yang, B., Jiang, S., Xu, L., Liu, K., Li, H., Xing, G., Chen, H., Jiang, X., & Yan, Z. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3699765 · Cited by 60
MoRA: High-Rank Updating for Parameter-Efficient Fine-TuningPreprint
Jiang, T., Huang, S.-H., Luo, S., Zhang, Z.-H., Huang, H.-Z., Wei, F.-R., Deng, W.-W., Sun, F., Zhang, Q., Wang, D.-Q., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2405.12130 · Cited by 42
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.16079 · Cited by 122
“We assess the downstream performance of the trial models by evaluating the finetuned models on the training sets of PubMedQA, MedMCQA, and MedQA.”
MethodInfluentialCan Multiple-choice Questions Really Be Useful in Detecting the Abilities of LLMs?
Li, W., Li, L., Xiang, T., Liu, X., Deng, W., & García, N. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2bvpg8s5dnii · Cited by 121
Adapting Large Language Models to Domains via Reading Comprehension
Cheng, D., Huang, S.-H., & Wei, F. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.09530 · Cited by 60
Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey
Song, Z., Yan, B., Liu, Y., Fang, M., Li, M., Yan, R., & Chen, X. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.1379 · Cited by 121
Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review
Neha, F., Bhati, D., & Shukla, D. K. (2025) · AI · doi:10.3390/ai6090226 · Cited by 117
“• PubMedQA: Consists of biomedical abstracts paired with yes/no/maybe questions, requiring grounded reasoning and evidence-based answers [72].”
A survey on retrieval-augmentation generation (RAG) models for healthcare applications
El-Enen, M. A. A., Saad, S., & Nazmy, T. M. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11666-9 · Cited by 123
ClinicalAgent: Clinical Trial Multi-Agent System with Large Language Model-based Reasoning
Yue, L., Xing, S., Chen, J., & Fu, T. (2024) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3698587.3701359 · Cited by 113
Biomedical and clinical English model packages for the Stanza Python NLP library
Zhang, Y., Zhang, Y., Qi, P., Manning, C. D., & Langlotz, C. P. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab090 · Cited by 112
Metadata
- Authors
- Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X.
- Affiliations
- University of Pittsburgh (US)
- Carnegie Mellon University (US)
- Google (United States) (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 2567-2577
- OpenAlex
- W2970482702
- Cited by
- 685
- References
- 26
- Citations by year
- 2026: 188 · 2025: 215 · 2024: 107 · 2023: 75 · 2022: 36 · 2021: 39 · 2020: 25
- FWCI
- 75.51
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- question answering, biomedical research, PubMed, scientific publications, datasets, biomedical question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.