PubMedQA: A Dataset for Biomedical Research Question Answering
Open accessThe DOI matches its Crossref record; 24 of the 26 references it cites were found in a registry or an index.
Abstract
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, Xinghua Lu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“given that medical question datasets such as PubMedQA42 and MedQA43 consist of fixed question sets susceptible to memorization”
From An automated framework for assessing how well LLMs cite relevant medical references (2025) · cites this paper as [43]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“All analyses were performed on publicly available, de-identified benchmark datasets—MIRAGE [23], PubMedQA [24], and the medical licensing examination questions of Katz et al. [25]—used in accordance with their respective licenses and terms of use.”
From The economics of accuracy for medical reasoning with large language models (2026) · cites this paper as [24]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“PubMedQA (Jin et al., 2019) is a biomedical research QA dataset under the MIT license. It contains 1,000 expert-annotated questions ( pqa\_labeled\_split ) and 211K machine-labeled questions from PubMed abstracts (the most widely used biomedical literature resource).”
From MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025) · cites this paper as Jin et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Notably, in healthcare, medical NLP benchmarks like MedQA7, PubMedQA8 and the Massive Multitask Language Understanding (MMLU) serve as the de-facto reporting standard for state-of-the-art medical LLMs9–11 with claims of ‘superhuman’ performance in patient-facing tasks12.”
From Medical large language models are vulnerable to data-poisoning attacks (2025) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“The ground truth question-answer pairs are sourced from the pqa\_labeled split of PubMedQA (Jin et al., 2019), which contains 1,000 expert-annotated samples, supplemented with 9,000 instances randomly selected from the pqa\_artificial split.”
From MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025) · cites this paper as Jin et al., 2019
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 2×
“We fine-tune BioBERT (Lee et al., 2019) on Pub-MedQA as a baseline.”
MethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014) · Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.3115/v1/p14-5010 · Cited by 7,037
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“Unlike other QA datasets in which questions are asked by crowd-workers for existing contexts (Rajpurkar et al., 2016; Yang et al., 2018; Kočiskỳ et al., 2018), in PubMedQA contexts are generated to answer the questions and both are written by the same authors.”
BackgroundNatural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · cited 6×
“, 2018), Natural Questions (Kwiatkowski et al., 2019), ShARC (Saeidi et al.”
BackgroundMethodInfluentialBoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 1,855
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 6×
“Supporting fact for the answer is highlighted . et al., 2018; Yang et al., 2018; Kwiatkowski et al., 2019), the largest annotated biomedical QA dataset, BioASQ (Tsatsaronis et al., 2015) has less than 3k training instances, most of which are simple factual questions.”
BackgroundMethodInfluentialSelf-citationEnhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1152 · Cited by 1,113
Research works printed but not linked: 1
- 2012 Conference and Labs of the Evaluation ForumQuestion Answering For Machine Reading Evaluation (QA4MRE), Rome/Forner, J.[edit.]; ea , pages 1-14.
Cited by
Items 201–300 of 1,026
Efficient Compressing and Tuning Methods for Large Language Models: A Systematic Literature Review
Kim, G. I., Hwang, S., & Jang, B. (2025) · ACM Computing Surveys · doi:10.1145/3728636 · Cited by 55
New Vietnamese Corpus for Machine Reading Comprehension of Health News Articles
Nguyen, K. V., Huynh, T. V., Nguyen, D.-V., Nguyen, A. G.-T., & Nguyen, N. L.-T. (2022) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3527631 · Cited by 55
EntityBERT: Entity-centric Masking Strategy for Model Pretraining for the Clinical Domain
Lin, C., Miller, T., Dligach, D., Bethard, S. J., & Savova, G. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.21 · Cited by 55
E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commercePreprint
Zhang, D., Yuan, Z., Liu, Y., Zhuang, F., Chen, H., & Xiong, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.02835 · Cited by 55
LLM-based Agentic Reasoning Frameworks: A Survey from Methods to ScenariosPreprint
Zhao, B., Foo, L. G., Hu, P., Theobalt, C., Rahmani, H., & Liu, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.17692 · Cited by 57
KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model
Zhao, X.-P., Hu, X.-S., Shan, Z., Huang, S.-Z., Zhou, Y., Sun, Z.-T., Liu, Z.-Y., Li, D., Wei, X., Chen, Q., et al. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2506.20923 · Cited by 55
Self-citationQuantifying the reasoning abilities of LLMs on clinical cases
Qiu, P., Wu, C., Liu, S., Fan, Y., Zhao, W., Chen, Z., Gu, H., Peng, C., Zhang, Y., Wang, Y., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64769-1 · Cited by 53
Zero-shot Faithful Factual Error Correction
Huang, K.-H., Chan, H. P., & Ji, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.311 · Cited by 54
RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question
Mohammadshahi, A., Scialom, T., Yazdani, M., Yanki, P., Fan, A., Henderson, J., & Saeidi, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.428 · Cited by 53
Towards evaluating and building versatile large language models for medicine
Wu, C., Qiu, P., Liu, J., Gu, H., Li, N., Zhang, Y., Wang, Y., & Xie, W. (2025) · npj Digital Medicine · doi:10.1038/s41746-024-01390-4 · Cited by 52
ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers
Sun, H., Cohen, W. W., & Salakhutdinov, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.253 · Cited by 53
Self-citationX-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMsPreprint
Ye, R., Liu, X., Wu, Q., Pang, X., Yin, Z., Bai, L., & Chen, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.16997 · Cited by 52
Role Prompting Guided Domain Adaptation with General Capability Preserve for Large Language Models
Wang, R., Mi, F., Chen, Y., Xue, B., Wang, H., Zhu, Q., Wong, K., & Xu, R. (2024) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2024.findings-naacl.145 · Cited by 51
Unsupervised Pre-training for Biomedical Question AnsweringPreprint
Kommaraju, V., Gunasekaran, K. P., Li, K., Bansal, T., McCallum, A., Williams, I., & Istrate, A.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.12952 · Cited by 51
The AdEMAMix Optimizer: Better, Faster, Older
Pagliardini, M., Ablin, P., & Grangier, D. (2024) · arXiv · doi:10.48550/arxiv.2409.03137
Wang, Z., Duan, J., Yuan, C., Chen, Q., Chen, T., Zhang, Y., Wang, R., Shi, X., & Xu, K. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109553 · Cited by 51
Language agents achieve superhuman synthesis of scientific knowledge
Skarlinski, M. D., Cox, S., Laurent, J. M., Braza, J. D., Hinks, M., Hammerling, M. J., Ponnapati, M., Rodriques, S. G., & White, A. D. (2024) · arXiv · doi:10.48550/arxiv.2409.13740 · Cited by 49
“They do not consider the entire literature, but instead are restricted to abstracts 6 , retrieval on a fixed corpus 7 , or simply provide the relevant paper directly 8 .”
BackgroundSellergren, A., Kazemzadeh, S., Jaroensri, T., Kiraly, A., Traverse, M., Kohlberger, T., Xu, S., Jamil, F., Hughes, C., Lau, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.05201 · Cited by 49
Pashangpour, S., & Nejat, G. (2024) · Robotics · doi:10.3390/robotics13080112 · Cited by 50
The fine art of fine-tuning: A structured review of advanced LLM fine-tuning techniques
Pratap, S., Aranha, A. R., Kumar, D., Malhotra, G., Iyer, A. P. N., & Shylaja, S. S. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100144 · Cited by 51
Nentidis, A., Katsimpras, G., Vandorou, E., Krithara, A., Gascó, L., Krallinger, M., & Παλιούρας, Γ. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85251-1_18 · Cited by 47
NoiseQA: Challenge Set Evaluation for User-Centric Question Answering
Ravichander, A., Dalmia, S., Ryskina, M., Metze, F., Hovy, E. H., & Black, A. W. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.259 · Cited by 50
Data Extraction Attacks in Retrieval-Augmented Generation via BackdoorsPreprint
Peng, Y., Wang, J., Hong, Y., & Houmansadr, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.01705 · Cited by 50
Evaluating Large Language Models in Ophthalmology: Systematic Review
Zhang, Z., Zhang, H., Pan, Z., Bi, Z., Wan, Y., Song, X., & Fan, X. (2025) · Journal of Medical Internet Research · doi:10.2196/76947 · Cited by 50
BRIDGE: benchmarking large language models for understanding real-world clinical practice texts
Wu, J., Gu, B., Zhou, R., Xie, K., Snyder, D., Jiang, Y., Carducci, V., Wyss, R., Desai, R. J., Alsentzer, E., et al. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-026-01719-2 · Cited by 51
LLMs4All: A Review of Large Language Models Across Academic DisciplinesPreprint
Ye, Y., Zhang, Z., Ma, T., Wang, Z., Li, Y., Hou, S., Sun, W., Shi, K., Ma, Y., Song, W., et al. (2025) · Preprints.org · doi:10.20944/preprints202510.1383.v1 · Cited by 51
RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs
Yu, Y., Ping, W., Liu, Z., Wang, B., You, J., Zhang, C., Shoeybi, M., & Catanzaro, B. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3850 · Cited by 47
Self-citationWatermarking LLMs with Weight Quantization
Li, L., Jiang, B., Wang, P., Ren, K., Yan, H., & Qiu, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.220 · Cited by 49
Question answering systems for health professionals at the point of care—a systematic review
Kell, G., Roberts, A., Umansky, S., Qian, L., Ferrari, D., Soboczenski, F., Wallace, B., Patel, N., & Marshall, I. J. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae015 · Cited by 47
A Practical Examination of AI-Generated Text Detectors for Large Language Models
Tufts, B., Zhao, X., & Li, L. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.271 · Cited by 51
Large language models for bioinformatics
Ruan, W., Lyu, Y., Zhang, J., Cai, J., Shu, P., Yang, G., Lu, Y., Gao, S., Wang, Y., Wang, P., et al. (2025) · Quantitative Biology · doi:10.1002/qub2.70014 · Cited by 48
Self-citationALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical ReasoningPreprint
Li, S. S., Mun, J., Brahman, F., Hosseini, P., Thomas, B. G., Sin, J. M., Ren, B., Ilgen, J. S., Tsvetkov, Y., & Sap, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.14860 · Cited by 47
Task-aware Retrieval with Instructions
Asai, A., Schick, T., Lewis, P., Chen, X., Izacard, G., Riedel, S., Hajishirzi, H., & Yih, W.-T. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.225 · Cited by 46
Exploring the effectiveness of instruction tuning in biomedical language processing
Rohanian, O., Nouriborji, M., Kouchaki, S., Nooralahzadeh, F., Clifton, L., & Clifton, D. A. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103007 · Cited by 46
EYE-Llama, an in-domain large language model for ophthalmology
Haghighi, T., Gholami, S., Sokol, J. T., Kishnani, E., Ahsaniyan, A., Rahmanian, H., Hedayati, F., Leng, T., & Alam, M. N. (2025) · iScience · doi:10.1016/j.isci.2025.112984 · Cited by 46
AIBench: Towards trustworthy evaluation under the 45°law
Zhang, Z., Wang, J., Guo, Y., Wen, F., Chen, Z., Wang, H., Li, W., Sun, L., Zhou, Y., Zhang, J., et al. (2025) · Displays · doi:10.1016/j.displa.2025.103255 · Cited by 46
RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructions
Liu, W., Chen, J., Ji, K., Zhou, L., Chen, W., & Wang, B. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.192 · Cited by 49
Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text
Li, D., Hu, B., Chen, Q., Peng, W., & Wang, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.111 · Cited by 45
Characterizing LLM Abstention Behavior in Science QA with Context Perturbations
Wen, B., Howe, B., & Wang, L. L. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.197 · Cited by 47
Nentidis, A., Krithara, A., Bougiatiotis, K., Krallinger, M., Rodriguez-Penagos, C. G., Villegas, M., & Παλιούρας, Γ. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58219-7_16 · Cited by 47
Xu, R., Zhuang, Y., Zhong, Y., Yu, Y., Wang, Z., Tang, X., Wu, H., Wang, M. D., Ruan, P., Yang, D. M., et al. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2506.04405 · Cited by 45
DiffusionGemma Technical ReportPreprint
DiffusionGemma Team, Taïga, A. A., Assiene, J., Calandriello, D., Chaabouni, R., Gante, J., von Glehn, T., Keating, N., Knutsen, C., Kukla, M., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.00146 · Cited by 45
Agentic AI and the rise of in silico team science in biomedical research
Li, B., Saini, A. K., Hernandez, J. G., & Moore, J. H. (2026) · Nature Biotechnology · doi:10.1038/s41587-026-03035-1 · Cited by 45
HealthBench Professional: Evaluating Large Language Models on Real Clinician ChatsPreprint
Hicks, R. S., Trofimov, M., Lim, D., Arora, R. K., Tsimpourlas, F., Bowman, P., Sharman, M., Tong, C., Karthik, K., Dugar, A., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.27470 · Cited by 46
KEBLM: Knowledge-Enhanced Biomedical Language Models
Lai, T., Zhai, C., & Ji, H. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104392 · Cited by 44
A collaborative large language model for drug analysis
Zhou, H., Liu, F., Wu, J., Zhang, W., Huang, G., Clifton, L., Eyre, D. W., Luo, H., Liu, F., Branson, K., et al. (2025) · Nature Biomedical Engineering · doi:10.1038/s41551-025-01471-z · Cited by 44
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician ValidationPreprint
Zhang, M., Shen, Y., Li, Z., Sha, H., Hu, B., Wang, Y., Huang, C., Liu, S., Tong, J., Jiang, C., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.04078 · Cited by 43
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
Cai, Y., Wang, L., Wang, Y., De Melo, G., Zhang, Y., Wang, Y., & He, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29723 · Cited by 42
Corbeil, J.-P., Dada, A., Attendu, J.-M., Abacha, A. B., Sordoni, A., Caccia, L., Beaulieu, F., Lin, T., Kleesiek, J., & Vozila, P. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.950 · Cited by 43
MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation
He, Z., Wang, Y., Yan, A., Liu, Y., Chang, E. Y., Gentili, A., McAuley, J., & Hsu, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.540 · Cited by 42
MedS$^3$: Towards Medical Slow Thinking with Self-Evolved Soft Dual-sided Process SupervisionPreprint
Jiang, S., Liao, Y., Chen, Z., Ya, Z., Wang, Y., & Wang, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.12051 · Cited by 43
RareBench: Can LLMs Serve as Rare Diseases Specialists?
Chen, X., Mao, X., Guo, Q., Wang, L., Zhang, S., & Chen, T. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671576 · Cited by 41
“Medical Benchmarks for LLMs. Prominent medical question-and-answer data benchmarks, such as MedQA [16], PubMedQA [17], MedMCQA [38], MultiMedQA [45], and CMExam [27], primarily derive from medical examinations, typically featuring multiple-choice formats.”
Fine-tuning large language models for improved health communication in low-resource languages
Bui, N., Nguyen, G., Nguyen, N., Vo, B., Vo, L., Huynh, T., Tang, A., Tran, V. N., Huynh, T., Nguyen, H. Q., et al. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.108655 · Cited by 45
He, C., Zhou, X., Wu, Y., Yu, X., Zhang, Y., Zhang, L., Wang, D., Lyu, S., Xu, H., Wang, X., et al. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.739 · Cited by 41
From hallucinations to hazards: benchmarking LLMs for hazard analysis in safety-critical systems
Dokas, Ι. M. (2025) · Safety Science · doi:10.1016/j.ssci.2025.107056 · Cited by 42
Testing Contextualized Word Embeddings to Improve NER in Spanish Clinical Case Narratives
Akhtyamova, L., Martinez, P., Verspoor, K., & Cardiff, J. (2020) · IEEE Access · doi:10.1109/access.2020.3018688 · Cited by 30
Nonfactoid Question Answering as Query-Focused Summarization With Graph-Enhanced Multihop Inference
Deng, Y., Zhang, W., Xu, W., Shen, Y., & Lam, W. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3258413 · Cited by 34
Wang, F., Bao, R., Wang, S., Yu, W., Liu, Y., Cheng, W., & Chen, H. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.209 · Cited by 40
Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generation
Pan, R., Cao, B., Lin, H., Han, X., Zheng, J., Wang, S., Cai, X., & Sun, L. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1109 · Cited by 40
Generative Artificial Intelligence in Medicine
Shanmugam, D., Agrawal, M., Movva, R., Chen, I. Y., Ghassemi, M., Jacobs, M. L., & Pierson, E. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095332 · Cited by 42
Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs
Cha, S.-M., Cho, S., Hwang, D., & Lee, M. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2408.06621 · Cited by 47
Hallucination to truth: a review of fact-checking and factuality evaluation in large language models
Rahman, S. S., Islam, M. A., Alam, M. M., Zeba, M., Rahman, M. A., Chowa, S. S., Raiaan, M. A. K., & Azam, S. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-025-11454-w · Cited by 39
PeerQA: A Scientific Question Answering Dataset from Peer Reviews
Baumgärtner, T., Briscoe, T., & Gurevych, I. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.22 · Cited by 39
BEVERS: A General, Simple, and Performant Framework for Automatic Fact Verification
DeHaven, M., & Scott, S. (2023) · Fact Extraction and VERification Workshop (FEVER) · doi:10.18653/v1/2023.fever-1.6 · Cited by 39
A Medical Diagnostic Assistant Based on LLM
Wu, C., Lin, Z., Fang, W., & Huang, Y. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-1717-0_12 · Cited by 39
Probing Language Models for Pre-training Data Detection
Liu, Z., Zhu, T., Tan, C., Liu, B., Lu, H., & Chen, W. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.86 · Cited by 40
Self-citationSciDQA: A Deep Reading Comprehension Dataset over Scientific Papers
Singh, S., Sarkar, N., & Cohan, A. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1163 · Cited by 39
A Brief Review on Benchmarking for Large Language Models Evaluation in Healthcare
Cilar, L., Chen, H., Chen, A., Topaz, M., Tam, W., Bian, J., & Štiglic, G. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70010 · Cited by 39
Synthesizing scientific literature with retrieval-augmented language models
Asai, A., He, J., Shao, R., Shi, W., Singh, A., Chang, J. C., Lo, K., Soldaini, L., Feldman, S., D’Arcy, M., et al. (2026) · Nature · doi:10.1038/s41586-025-10072-4 · Cited by 38
“For single-paper tasks, we directly compare the model outputs to gold reference texts, following the evaluation methodologies proposed in previous work 41–43 .”
Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT
Meng, Z., Liu, F., Clark, T. H., Shareghi, E., & Collier, N. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.383 · Cited by 38
OAG-Bench: A Human-Curated Benchmark for Academic Graph Mining
Zhang, F., Shi, S., Zhu, Y., Chen, B., Cen, Y., Yu, J., Chen, Y., Wang, L., Zhao, Q., Cheng, Y., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3672354 · Cited by 38
Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm
Liu, L., Yang, X., Li, F., Chi, C., Shen, Y., Lyu, S., Zhang, M., Ma, X., Lv, X., Ma, L., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671575 · Cited by 38
Fine-grained Pluggable Gradient Ascent for Knowledge Unlearning in Language Models
Feng, X.-H., Chen, C.-J., Li, Y., & Lin, Z. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.566 · Cited by 34
Horawalavithana, S., Ayton, E., Sharma, S. K., Howland, S., Subramanian, M., Vasquez, S., Cosbey, R., Glenski, M., & Volkova, S. (2022) · BIGSCIENCE · doi:10.18653/v1/2022.bigscience-1.12 · Cited by 38
Liang, H., Ma, X., Liu, Z., Wong, Z. H., Zhao, Z., Meng, Z., He, R., Shen, C., Cai, Q., Han, Z., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.16676 · Cited by 39
Self-citationLearning to Rewrite: Generalized LLM-Generated Text Detection
Hao, W., Li, R., Zhao, W., Yang, J., & Mao, C. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.322 · Cited by 38
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation
Zhang, M., Shen, Y., Li, Z., Sha, H., Hu, B., Wang, Y., Huang, C., Liu, S., Tong, J., Jiang, C., et al. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.263 · Cited by 37
Zagirova, D., Pushkov, S., Leung, G. H. D., Liu, B. H. M., Urban, A., Sidorenko, D., Kalashnikov, A., Kozlova, E., Naumov, V., Pun, F. W., et al. (2023) · Aging · doi:10.18632/aging.205055 · Cited by 37
PaniniQA: Enhancing Patient Education Through Interactive Question Answering
Cai, P., Yao, Z., Liu, F., Wang, D., Reilly, M. H., Zhou, H., Li, L., Cao, Y., Kapoor, A., Bajracharya, A. S., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00616 · Cited by 37
Pruning as a Domain-specific LLM Extractor
Zhang, N., Liu, Y., Zhao, X., Cheng, W., Bao, R., Zhang, R., Mitra, P., & Chen, H. (2024) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2024.findings-naacl.91 · Cited by 37
Fariello, S., Fenza, G., Forte, F., Gallo, M., & Marotta, M. (2025) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2024.12.002 · Cited by 38
A survey of datasets in medicine for large language models
Zhang, D., Xue, X., Gao, P., Jin, Z., Hu, M., Wu, Y., & Ying, X. (2024) · Intelligence & Robotics · doi:10.20517/ir.2024.27 · Cited by 37
Unveiling the power of language models in chemical research question answering
Chen, X., Wang, T., Guo, T., Guo, K., Zhou, J., Li, H., Song, Z., Gao, X., & Zhang, X. (2025) · Communications Chemistry · doi:10.1038/s42004-024-01394-x · Cited by 36
Vazhentsev, A., Rvanova, L., Lazichny, I., Panchenko, A., Panov, M., Baldwin, T., & Shelmanov, A. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.113 · Cited by 38
Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching
Aytes, S. A., Baek, J., & Hwang, S. J. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1236 · Cited by 36
de Araujo, P. H. L., Röttger, P., Hovy, D., & Roth, B. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1364 · Cited by 37
A clinical environment simulator for dynamic AI evaluation
Luo, L., Kim, S. E., Zhang, X., Kernbach, J. M., Kenia, R., Acosta, J. N., Nathanson, L. A., Haimovich, A. D., Rodman, A. M., Goh, E., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04252-6 · Cited by 36
Pirá: A Bilingual Portuguese-English Dataset for Question-Answering about the Ocean
Paschoal, A. F. A., Pirozelli, P., Freire, V., Delgado, K. V., Peres, S. M., José, M. M., Cação, F. N., Oliveira, A. S., Brandão, A. A. F., Costa, A. H. R., et al. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482012 · Cited by 38
Benchmarking Large Language Models in Evidence-Based Medicine
Li, J., Deng, Y., Sun, Q., Zhu, J., Tian, Y., Li, J., & Zhu, T. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3483816 · Cited by 35
Moro, G., & Valgimigli, L. (2021) · Sensors · doi:10.3390/s21196430 · Cited by 35
Rasool, Z., Kurniawan, S., Balugo, S., Barnett, S., Vasa, R., Chesser, C., Hampstead, B. M., Belleville, S., Mouzakis, K., & Bahar-Fuchs, A. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100083 · Cited by 34
MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models
Park, D., Lee, J.-W., Park, S., Jeong, H., Koo, Y., Hwang, S., Park, S., & Lee, S. (2024) · GenBench Workshop on Generalisation (Benchmarking) in NLP (GenBench) · doi:10.18653/v1/2024.genbench-1.7 · Cited by 34
ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agents
Liao, Y., Jiang, S., Wang, Y., & Wang, Y. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.663 · Cited by 34
Critique of impure reason: Unveiling the reasoning behaviour of medical large language models
Sim, S. Z. Y., & Chen, T. (2025) · eLife · doi:10.7554/elife.106187 · Cited by 34
RGAR: Recurrence Generation-augmented Retrieval for Factual-aware Medical Question Answering
Liang, S., Zhang, L., Zhu, H., Wang, W., He, Y., & Zhou, D. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.214 · Cited by 34
Schlegel, V., Nenadić, G., & Batista-Navarro, R. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000171 · Cited by 34
Pooling And Attention: What Are Effective Designs For LLM-Based Embedding Models?Preprint
Tang, Y., & Yang, Y. (2024) · arXiv.org · doi:10.48550/arxiv.2409.02727 · Cited by 34
What Language Model to Train if You Have One Million GPU Hours?
Le Scao, T., Wang, T., Hesslow, D., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., Press, O., et al. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.54 · Cited by 33
Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset
Yue, X., Jimenez Gutierrez, B., & Sun, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.410 · Cited by 33
Metadata
- Authors
- Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X.
- Affiliations
- University of Pittsburgh (US)
- Carnegie Mellon University (US)
- Google (United States) (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 2567-2577
- OpenAlex
- W2970482702
- Cited by
- 685
- References
- 26
- Citations by year
- 2026: 188 · 2025: 215 · 2024: 107 · 2023: 75 · 2022: 36 · 2021: 39 · 2020: 25
- FWCI
- 75.51
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- question answering, biomedical research, PubMed, scientific publications, datasets, biomedical question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.