MedRedQA for Medical Consumer Question Answering: Dataset, Tasks, and Neural Baselines
Open accessAbstract
Vincent Nguyen, Sarvnaz Karimi, Maciej Rybinski, Zhenchang Xing. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multiple widely cited research papers use AskDocs's responses as gold references.
“There are multiple widely cited research papers using AskDocs's responses as gold references (Nguyen et al., 2023;Abrar et al., 2024;Ayers et al., 2023b), which proves its good quality among public forums.”
From MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification (2026) · cites this paper as Nguyen et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works
Van Riel, N., Auwerx, K., Debbaut, P., Van Hees, S., & Schoenmakers, B. (2017) · BJGP Open · doi:10.3399/bjgpopen17x100833 · Cited by 159 · see citing works
Internet-augmented language models through few-shot prompting for open-domain question answering
Lazaridou, A., Gribovskaya, E., Stokowiec, W., & Grigorev, N. (2022) · arXiv · doi:10.48550/arxiv.2203.05115 · Cited by 64 · see citing works
Memory-assisted prompt editing to improve GPT-3 after deployment
Madaan, A., Tandon, N., Clark, P., & Yang, Y. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.183 · Cited by 60 · see citing works
Flagged first, then by citations
Research works printed but not linked: 55
- Ben Abacha and Demner-Fushman. 2016. Recogniz- ing Question Entailment for Medical Question An- swering. American Medical Informatics Association Annual Symposium Proceedings, 2016:310-318.
- T. Armstrong, A. Moffat, W. Webber, and J. Zobel. 2009. Improvements that don't add up: Ad-hoc retrieval results since 1998. In CIKM, pages 601-610, Hong Kong, China. · DOI
- John W. Ayers, Adam Poliak, Mark Dredze, Eric C. Leas, Zechariah Zhu, Jessica B. Kelley, Dennis J. Faix, Aaron M. Goodman, Christopher A. Longhurst, Michael Hogarth, and Davey M. Smith. 2023. Com- paring Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Internal Medicine. · DOI · Paper page
- Marco Basaldella and Nigel Collier. 2019. BioRed- dit: Word embeddings for user-generated biomedical NLP. In LOUHI, pages 34-38, Hong Kong. · DOI
- Asma Ben Abacha and Dina Demner-Fushman. 2019. On the summarization of consumer health questions. In ACL, pages 2228-2234, Florence, Italy. · DOI
- Asma Ben Abacha and Dina Demner-Fushman. 2019. A question-entailment approach to question answering. CoRR, abs/1901.08079. · arXiv
- Asma Ben Abacha, Chaitanya Shivade, and Dina Demner-Fushman. 2019. Overview of the MEDIQA 2019 shared task on textual inference, question en- tailment and question answering. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 370-379, Florence, Italy. · DOI
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, abs/2005.14165. · arXiv
- Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. CoRR, abs/2210.11416. · arXiv
- Anthony Cocco, Rachel Zordan, David Taylor, Tracey Weiland, Stuart Dilley, Joyce Kant, Mahesha Dom- bagolla, Andreas Hendarto, Fiona Lai, and Jennie Hutton. 2018. Dr Google in the ED: searching for online health information by adult emergency depart- ment patients. The Medical Journal of Australia, 209:342-347. · DOI · Paper page
- Dina Demner-Fushman, Yassine Mrabet, and Asma Ben Abacha. 2020. Consumer health information and question answering: helping consumers find answers to their health-related information needs. JAMIA, 27(2):194-201. 31592532[pmid. · DOI
- Renee Garett and Sean D Young. 2021. Online mis- information and vaccine hesitancy. Transl. Behav. Med., 11(12):2194-2199.
- Suzanne Graham and John Brookey. 2008. Do patients understand? The Permanente journal, 12(3):67-69. · DOI
- Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific lan- guage model pretraining for biomedical natural lan- guage processing. ACM Trans. Comput. Healthcare, 3(1). · DOI · Paper page
- Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Ming-Wei Chang. 2020. Realm: Retrieval- augmented language model pre-training. In ICML, Online. · DOI
- Matthew L. Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Effi- cient natural language response suggestion for smart reply. CoRR, abs/1705.00652. · arXiv
- Azhar Hussain, Syed Ali, Madiha Ahmed, and She- haryar Hussain. 2018. The anti-vaccination move- ment: A regression in modern medicine. Cureus, 10(7):e2919-e2919. PMC6122668[pmcid]. · DOI
- Paul Jaccard. 1912. The distribution of the flora in the alpine zone.1. New Phytologist, 11:37-50. · DOI · Paper page
- Jimmy, Guido Zuccon, Bevan Koopman, and Gianluca Demartini. 2019. Health card retrieval for consumer health search: An empirical investigation of methods. In CIKM, page 2405-2408, New York, NY, USA. · DOI
- Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In EMNLP-IJCNLP, pages 2567-2577, Hong Kong, China. · DOI · Paper page
- Alistair Johnson, Tom Pollard, Lu Shen, Li-wei Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Anthony Celi, and Roger Mark. 2016. MIMIC-III, a freely accessible critical care database. Scientific Data, 3(1):160035. · DOI · Paper page
- Roozenbeek Jon, Schneider Claudia, Dryhurst Sarah, Kerr John, Freeman Alexandra, Recchia Gabriel, van der Bles Anne Marthe, and van der Lin- den Sander. 2020. Susceptibility to misinformation about covid-19 around the world. R. Soc. Open. Sci., 7. · DOI · Paper page
- Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, and Majid Yazdani. 2022. Prompt- free and efficient few-shot learning with language models. In ACL, pages 3638-3652, Dublin, Ireland. · DOI
- Anastasia Krithara, Anastasios Nentidis, Georgios Paliouras, and Ioannis Kakadiaris. 2016. Results of the 4th edition of BioASQ challenge. In Proceedings of the Fourth BioASQ workshop, pages 1-7, Berlin, Germany. · DOI
- Bridget M. Kuehn. 2013. More Than One-Third of US Individuals Use the Internet to Self-diagnose. Jour- nal of the American Medical Association, 309(8):756- 757. · DOI
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In ACL, pages 74-81, Barcelona, Spain.
- Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin A Raf- fel. 2022a. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In NIPS, volume 35, pages 1950-1965.
- Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022b. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-114, Dublin, Ireland and Online. · DOI · Paper page
- Ilya Loshchilov and Frank Hutter. 2017. Decoupled Weight Decay Regularization. In ICLR, Toulon, France.
- Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In NIPS, pages 4765-4774.
- Taylor Nelson, Nicole Kagan, Claire Critchlow, Alan Hillard, and Albert Hsu. 2020. The danger of mis- information in the COVID-19 crisis. Mo. Med., 117(6):510-512.
- Anastasios Nentidis, Konstantinos Bougiatiotis, Anasta- sia Krithara, Georgios Paliouras, and Ioannis Kakadi- aris. 2017. Results of the fifth edition of the bioasq challenge. In Biomedical Natural Language Process- ing, pages 48-57, Vancouver, Canada. · DOI
- Anastasios Nentidis, Georgios Katsimpras, Eirini Vandorou, Anastasia Krithara, Antonio Miranda- Escalada, Luis Gasco, Martin Krallinger, and Geor- gios Paliouras. 2022. Overview of BioASQ 2022: The tenth bioasq challenge on large-scale biomedi- cal semantic indexing and question answering. In Experimental IR Meets Multilinguality, Multimodal- ity, and Interaction, pages 337-361, Cham. Springer International Publishing. · DOI
- Anastasios Nentidis, Anastasia Krithara, Konstantinos Bougiatiotis, Georgios Paliouras, and Ioannis Kakadi- aris. 2018. Results of the sixth edition of the BioASQ challenge. In Proceedings of the 6th BioASQ Work- shop A challenge on large-scale biomedical semantic indexing and question answering, pages 1-10, Brus- sels, Belgium.
- Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In NIPS Cognitive Computations Workshop, volume 1773, Barcelona, Spain.
- Vincent Nguyen, Sarvnaz Karimi, and Zhenchang Xing. 2019. ANU-CSIRO at MEDIQA 2019: Question answering using deep contextual knowledge. In Pro- ceedings of the 18th BioNLP Workshop and Shared Task, pages 478-487, Florence, Italy. · DOI
- Vincent Nguyen, Maciej Rybinski, Sarvnaz Karimi, and Zhenchang Xing. 2022. Search like an expert: Reduc- ing expertise disparity using a hybrid neural index for covid-19 queries. Journal of Biomedical Informatics, 127:104005. · DOI
- Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with BERT. CoRR, abs/1910.14424. · arXiv
- Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In EMNLP, pages 3982-3992, Hong Kong, China. · DOI · Paper page
- Kirk Roberts, M. Simpson, Dina Demner-Fushman, Ellen Voorhees, and William Hersh. 2016. State-of- the-art in biomedical literature retrieval for clinical cases: A survey of the TREC 2014 CDS track. Inf Retr., 19(1-2):113-148. · DOI
- Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. CoRR, abs/1804.04235. · arXiv
- Chaitanya Shivade. 2019. Mednli -a natural language inference dataset for the clinical domain (version 1.0.0. PhysioNet. · DOI
- Yuanhe Tian, Weicheng Ma, Fei Xia, and Yan Song. 2019. ChiMed: A Chinese medical corpus for ques- tion answering. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 250-260, Florence, Italy. · DOI
- Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971. · arXiv
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. 2017. Attention is all you need. In NIPS, pages 6000-6010.
- Ellen Voorhees, Alam Tasmeer, Demner-Fushman Dina, Hersh William, and Kyle Lo. 2020. TREC-COVID: Constructing a pandemic information retrieval test collection. ACM SIGIR Forum, 54(1):1-12. · DOI
- Somin Wadhwa, Vivek Khetan, Silvio Amir, and By- ron Wallace. 2023. Redhot: A corpus of annotated medical questions, experiences, and claims on social media. In EACL: Findings.
- Shuai Wang, Shengyao Zhuang, and Guido Zuccon. 2021. BERT-based dense retrievers require interpo- lation with bm25 for effective passage retrieval. In ICTIR, page 317-324, New York, NY, USA. · DOI
- Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In ICLR, Online.
- Dirk Weißenborn, George Tsatsaronis, and Michael Schroeder. 2013. Answering factoid questions in the biomedical domain. In CEUR Workshop Pro- ceedings, volume 1094, Valencia, Spain.
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace's Transformers: State-of-the-art Natural Language Processing. CoRR, page arXiv:1910.03771. · arXiv
- Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Erik Learned-Miller, and Jaap Kamps. 2018. From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing. In CIKM, page 497-506, Torino, Italy. · DOI
- Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. 2018. Multi-scale attentive interac- tion networks for chinese medical question answer selection. IEEE Access, 6:74061-74071. · DOI
- Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M. Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized em- beddings and earth mover distance. In EMNLP, Hong Kong, China.
- Ming Zhu, Aman Ahuja, Da-Cheng Juan, Wei Wei, and Chandan K. Reddy. 2020. Question answering with long multiple-span answers. In EMNLP (Findings), pages 3840-3849, Online. · DOI
Cited by
Sorted by citations
Huang, H., DeLucia, A., Tiyyala, V. M., & Dredze, M. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.693 · Cited by 2 · see citing works
“To assess the impact of each component in the”
- Authors
- Nguyen, V., Karimi, S., Rybinski, M., & Xing, Z.
- Affiliations
- Australian National University (AU)
- Commonwealth Scientific and Industrial Research Organisation (AU)
- Venue
- Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2023
- Pages
- 629-648
- OpenAlex
- W4392637261
- Cited by
- 5
- References
- 7
- Citations by year
- 2026: 2 · 2025: 3
- FWCI
- 0.89
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- dx.doi.org
- Licence
- cc-by
- Locations
- International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- medical question answering, consumer health information, question answering dataset, medical text understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
