Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset
Open accessAbstract
Machine reading comprehension has made great progress in recent years owing to largescale annotated datasets.In the clinical domain, however, creating such datasets is quite difficult due to the domain expertise required for annotation.Recently, Pampari et al. (2018) tackled this issue by using expert-annotated question templates and existing i2b2 annotations to create emrQA, the first large-scale dataset for question answering (QA) based on clinical notes.In this paper, we provide an indepth analysis of this dataset and the clinical reading comprehension (CliniRC) task.From our qualitative analysis, we find that (i) emrQA answers are often incomplete, and (ii) emrQA questions are often answerable without using domain knowledge.From our quantitative experiments, surprising results include that (iii) using a small sampled subset (5%-20%), we can obtain roughly equal performance compared to the model trained on the entire dataset, (iv) this performance is close to human expert's performance, and (v) BERT models do not beat the best performing base model.Following our analysis of the emrQA, we further explore two desired aspects of CliniRC systems: the ability to utilize clinical domain knowledge and to generalize to unseen questions and contexts.We argue that both should be considered when creating future datasets.1
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Clinical NLP tasks are greatly limited by their overreliance on the same handful of data sources, small dataset sizes (typically thousands of examples), highly repetitive content, and low coverage of use cases.
“However, these NLP tasks are greatly limited by their overreliance on the same handful of data sources74, small dataset sizes (typically thousands of examples)74,76, highly repetitive content77, and low coverage of use cases20.”
From The shaky foundations of large language models and foundation models for electronic health records (2023) · cites this paper as [77]
The Related Work section says clinical QA and reading comprehension lack large-scale annotated data, supporting the concern about dataset size. But the paper does not establish that clinical NLP tasks typically have only thousands of examples or overrely on the same handful of data sources; §4.1 reports that emrQA alone has more than one million questions. The other limitations in the citing sentence are attached to different citations. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
(2013) · Cited by 5,726 · see citing works
The unified medical language system (UMLS): integrating biomedical terminology
Bodenreider, O. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh061 · Cited by 4,356 · see citing works
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · see citing works
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01603 · Cited by 2,322 · see citing works
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264 · see citing works
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · see citing works
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · see citing works
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · see citing works
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · see citing works
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · see citing works
Reading Wikipedia to answer open-domain questions
Chen, D., Fisch, A., Weston, J., & Bordes, A. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1171 · Cited by 1,546 · see citing works
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · see citing works
Long Short-Term Memory-Networks for Machine Reading
Cheng, J.-P., Dong, L., & Lapata, M. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1053 · Cited by 1,200 · see citing works
QANet: Combining Local Convolution with Global Self-Attention for Reading ComprehensionPreprint
Yu, A. W., Dohan, D., Luong, M.-T., Zhao, R., Chen, K., Norouzi, M., & Le, Q. V. (2018) · International Conference on Learning Representations · doi:10.48550/arxiv.1804.09541 · Cited by 1,169 · see citing works
Publicly Available Clinical BERT EmbeddingsPreprint
Alsentzer, E., Murphy, J. R., Boag, W., Weng, W., Jin, D., Naumann, T., & McDermott, M. B. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03323 · Cited by 1,040 · see citing works
Clinical information extraction applications: A literature review
Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., et al. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.11.011 · Cited by 910 · see citing works
Annotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907 · see citing works
ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing
Neumann, M., King, D., Beltagy, I., & Ammar, W. (2019) · BioNLP Workshop and Shared Task (BioNLP) · doi:10.18653/v1/w19-5034 · Cited by 772 · see citing works
Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base
Yih, W.-T., Chang, M.-W., He, X., & Gao, J. (2015) · doi:10.3115/v1/p15-1128 · Cited by 769 · see citing works
NewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · see citing works
What can natural language processing do for clinical decision support?
Demner‐Fushman, D., Chapman, W. W., & McDonald, C. J. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.08.007 · Cited by 690 · see citing works
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · see citing works
Gated Self-Matching Networks for Reading Comprehension and Question Answering
Wang, W., Yang, N., Wei, F., Chang, B., & Zhou, M. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1018 · Cited by 669 · see citing works
A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1223 · Cited by 494 · see citing works
Machine Comprehension Using Match-LSTM and Answer PointerPreprint
Wang, S., & Jiang, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07905 · Cited by 424 · see citing works
Data from clinical notes: a perspective on the tension between structure and flexible documentation
Rosenbloom, S. T., Denny, J. C., Xu, H., Lorenzi, N., Stead, W. W., & Johnson, K. B. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.007237 · Cited by 404 · see citing works
AskHERMES: An online question answering system for complex clinical questions
Cao, Y.-G., Liu, F., Simpson, P. M., Antieau, L. D., Bennett, A., Cimino, J. J., Ely, J., & Yu, H. Q. (2011) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2011.01.004 · Cited by 232 · see citing works
A question-entailment approach to question answering
Abacha, A. B., & Demner‐Fushman, D. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3119-4 · Cited by 221 · see citing works
Hirschman, L., Light, M., Breck, E. J., & Burger, J. D. (1999) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1034678.1034731 · Cited by 219 · see citing works
emrQA: A Large Corpus for Question Answering on Electronic Medical Records
Pampari, A., Raghavan, P., Liang, J., & Peng, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1258 · Cited by 191 · see citing works
Biomedical question answering: A survey
Athenikos, S. J., & Han, H. (2009) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2009.10.003 · Cited by 181 · see citing works
Probing Biomedical Embeddings from Language Models
Jin, Q., Dhingra, B., Cohen, W., & Lu, X. (2019) · doi:10.18653/v1/w19-2011 · Cited by 123 · see citing works
Evaluation of a Method to Identify and Categorize Section Headers in Clinical Documents
Denny, J. C., Spickard, A., Johnson, K. B., Peterson, N. B., Peterson, J. F., & Miller, R. A. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3037 · Cited by 122 · see citing works
Table Cell Search for Question Answering
Sun, H., Ma, H., He, X., Yih, W.-T., Su, Y., & Yan, X. (2016) · International World Wide Web Conference (WWW) · doi:10.1145/2872427.2883080 · Cited by 110 · see citing works
Self-citationOpen Domain Question Answering via Semantic Enrichment
Sun, H., Ma, H., Yih, W.-T., Tsai, C.-T., Liu, J., & Chang, M.-W. (2015) · International World Wide Web Conference (WWW) · doi:10.1145/2736277.2741651 · Cited by 106 · see citing works
Self-citationBeyond information retrieval--medical question answering.
Lee, M., Cimino, J. J., Zhu, H. R., Sable, C., Shanker, V. K., Ely, J., & Yu, H. Q. (2006) · PubMed Central · Cited by 106 · see citing works
The MiPACQ clinical question answering system.
Cairns, B. L., Nielsen, R. D., Masanz, J. J., Martin, J., Palmer, M. S., Ward, W., & Savova, G. (2011) · Europe PMC (PubMed Central) · Cited by 98 · see citing works
Recognizing Question Entailment for Medical Question Answering.
Abacha, A. B., & Demner‐Fushman, D. (2016) · PubMed Central · Cited by 67 · see citing works
BioASQ: A Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering
Balikas, G., Krithara, A., Partalas, I., & Paliouras, G. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24471-6_3 · Cited by 51 · see citing works
Using outcome polarity in sentence extraction for medical question-answering.
Niu, Y., Zhu, X., & Hirst, G. (2006) · Europe PMC (PubMed Central) · Cited by 37 · see citing works
Question answering systems in biology and medicine—the time is now
Wren, J. D. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr327 · Cited by 17 · see citing works
Clinical text annotation - what factors are associated with the cost of time?
(2018) · Europe PMC (PubMed Central) · Cited by 13 · see citing works
Inducing Relational Knowledge from BERT
Bouraoui, Z., Camacho-Collados, J., & Schockaert, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6242 · Cited by 12 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
2 of the 52 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Lynette Hirschman, Marc Light, Eric Breck, and John D Burger. 1999. Deep read: A reading com- prehension system. In ACL'1999, pages 325-332.
- Minsuk Lee, James Cimino, Hai Ran Zhu, Carl Sable, Vijay Shanker, John Ely, and Hong Yu. 2006. Be- yond information retrieval-medical question an- swering. In AMIA annual symposium proceedings, volume 2006, page 469.
- Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In NAACL-HLT'18, pages 2227-2237.
- Simon Šuster and Walter Daelemans. 2018. Clicr: A dataset of clinical case reports for machine reading comprehension. In NAACL-HLT'18, pages 1551- 1563.
- Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In ACL'19.
Cited by
Sorted by citations
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847 · see citing works
Predicting drug–disease associations through layer attention graph convolutional network
Yu, Z., Huang, F. F., Zhao, X., Xiao, W., & Zhang, W. (2020) · Briefings in Bioinformatics · doi:10.1093/bib/bbaa243 · Cited by 417 · see citing works
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · see citing works
A comparative study of pretrained language models for long clinical text
Li, Y., Wehbe, R. M., Ahmad, F. S., Wang, H., & Luo, Y. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac225 · Cited by 138 · see citing works
Biomedical Question Answering: A Survey of Approaches and Challenges
Jin, Q., Yuan, Z. P., Xiong, G., Yu, Q., Ying, H., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2022) · ACM Computing Surveys · doi:10.1145/3490238 · Cited by 128 · see citing works
A taxonomy and review of generalization research in NLP
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00729-y · Cited by 121 · see citing works
A Survey on Machine Reading Comprehension Systems
Baradaran, R., Ghiasi, R., & Amirkhani, H. (2022) · Natural Language Engineering · doi:10.1017/s1351324921000395 · Cited by 101 · see citing works
Kwon, T., Ong, K. T.-I., Kang, D.-J., Moon, S., Lee, J. R., Hwang, D., Sohn, B., Sim, Y., Lee, D. H., & Yeo, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29802 · Cited by 52 · see citing works
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30205 · Cited by 66 · see citing works
KALA: Knowledge-Augmented Language Model Adaptation
Kang, M., Baek, J., & Hwang, S. J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.379 · Cited by 39 · see citing works
Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text
Li, D., Hu, B., Chen, Q., Peng, W., & Wang, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.111 · Cited by 37 · see citing works
GatorTron: A Large Language Model for Clinical Natural Language Processing
Yang, X., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2022) · doi:10.1101/2022.02.27.22271257 · Cited by 33 · see citing works
TCM-SD: A Benchmark for Probing Syndrome Differentiation via Natural Language Processing
Ren, M., Huang, H., Zhou, Y., Cao, Q., Bu, Y., & Gao, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-18315-7_16 · Cited by 27 · see citing works
DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processing
Gao, Y., Dligach, D., Miller, T. A., Caskey, J., Sharma, B., Churpek, M. M., & Afshar, M. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104286 · Cited by 26 · see citing works
CliniQG4QA: Generating Diverse Questions for Domain Adaptation of Clinical Question Answering
Yue, X., Zhang, X. F., Yao, Z., Lin, S. M., & Sun, H. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669300 · Cited by 20 · see citing works
Self-citationMedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records
Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E. P., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.14089 · Cited by 65 · see citing works
Question Answering for Electronic Health Records: Scoping Review of Datasets and Models
Bardhan, J., Roberts, K. E., & Wang, D. Z. (2024) · Journal of Medical Internet Research · doi:10.2196/53636 · Cited by 13 · see citing works
Moon, S., He, H., Jia, H., Liu, H., & Fan, J. W. (2023) · JMIR AI · doi:10.2196/41818 · Cited by 12 · see citing works
PHICON: Improving Generalization of Clinical Text De-identification Models via Data Augmentation
Yue, X., & Zhou, S. (2020) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2020.clinicalnlp-1.23 · Cited by 11 · see citing works
Self-citationLearning to Ask Like a Physician
Lehman, E., Lialin, V., Legaspi, K. E., Sy, A. J., Pile, P. T., Alberto, N. R., Ragasa, R. R., Puyat, C. V., Taliño, M. K., Alberto, I. R., et al. (2022) · Proceedings of the 4th Clinical Natural Language Processing Workshop · doi:10.18653/v1/2022.clinicalnlp-1.8 · Cited by 10 · see citing works
Kim, K., Park, S., Min, J., Park, S., Kim, J. Y., Eun, J., Jung, K., Park, Y. E., Kim, E. H., Lee, E. Y., et al. (2024) · JMIR Medical Informatics · doi:10.2196/52897 · Cited by 10 · see citing works
ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination
Li, D., Yu, J., Hu, B., Xu, Z., & Zhang, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.129 · Cited by 10 · see citing works
Mahbub, M., Begoli, E., Martins, S. B., Peluso, A., Tamang, S. R., & Peterson, G. D. (2023) · IEEE Access · doi:10.1109/access.2023.3235265 · Cited by 9 · see citing works
Biomedical Question Answering: A Comprehensive Review.Preprint
Jin, Q., Yuan, Z. P., Xiong, G., Yu, Q., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2021) · arXiv (Cornell University) · Cited by 9 · see citing works
PaniniQA: Enhancing Patient Education Through Interactive Question Answering
Cai, P., Yao, Z., Liu, F., Wang, D., Reilly, M. H., Zhou, H., Li, L., Cao, Y., Kapoor, A., Bajracharya, A. S., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00616 · Cited by 7 · see citing works
Towards Generalizable Methods for Automating Risk Score Calculation
Liang, J. J., Lehman, E., Iyengar, A. S., Mahajan, D., Raghavan, P., Chang, C. Y., & Szolovits, P. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.42 · Cited by 5 · see citing works
A Unified Knowledge Graph Augmentation Service for Boosting Domain-specific NLP Tasks
Ding, R., Han, X., & Wang, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.24 · Cited by 4 · see citing works
A survey on computational methods used for drug repositioning
Abushaaban, E., & Alhajj, R. (2025) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-025-00502-8 · Cited by 3 · see citing works
Clinical Phrase Mining with Language Models
Mani, K., Yue, X., Gutiérrez, B. J., Huang, Y., Lin, S. M., & Sun, H. (2020) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9313496 · Cited by 3 · see citing works
Self-citationMdCoT: Medical Diagnosis Chain-of-Thought with Self-Diagnostic Refinement for Alzheimer’s Disease
Lu, C.-L., Zhang, Y., Wang, P., Lü, W., & Qin, L. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209984 · Cited by 2 · see citing works
Medical question answering: A comprehensive multimodal and LLM-driven survey
Mhedhbi, E., Zhu, X., Ayaz, M., Khan, M., & Othmani, A. (2026) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2026.109541 · Cited by 1 · see citing works
Automatic Question Answering and Knowledge Discovery from Electronic Health Records
Wang, P. (2021) · VTechWorks (Virginia Tech)
Towards Explainable Medical Machine Reading Comprehension With Rationale Generation
Li, D., He, S., Hu, B., & Chen, Q. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3555105
Lanz, V., Datseris, A., Boytcheva, S., Mayer, J. H., Zikánová, Š., Mikulík, R., & Pecina, P. (2026) · Journal of Medical Internet Research · doi:10.2196/96347
- Authors
- Yue, X., Jimenez Gutierrez, B., & Sun, H.
- Affiliations
- The Ohio State University (US)
- ORCID
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 4474-4486
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3035129496
- Cited by
- 33
- References
- 52
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 3 · 2023: 7 · 2022: 10 · 2021: 3 · 2020: 4
- FWCI
- 2.29
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1815674)
- Patient-Centered Outcomes Research Institute (ME-2017C1-6413)
- Army Research Office (W911NF-17-1-0412, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- medical question answering, clinical notes, i2b2, data annotation, question templates, BERT
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
