An Empirical Evaluation of Large Language Models on Consumer Health Questions
Open accessAbstract
This study evaluates the performance of several Large Language Models (LLMs) on MedRedQA, a dataset of consumer-based medical questions and answers by verified experts extracted from the AskDocs subreddit. While LLMs have shown proficiency in clinical question answering (QA) benchmarks, their effectiveness on real-world, consumer-based, medical questions remains less understood. MedRedQA presents unique challenges, such as informal language and the need for precise responses suited to non-specialist queries. To assess model performance, responses were generated using five LLMs: GPT-4o mini, Llama 3.1: 70B, Mistral-123B, Mistral-7B, and Gemini-Flash. A cross-evaluation method was used, where each model evaluated its responses as well as those of others to minimize bias. The results indicated that GPT-4o mini achieved the highest alignment with expert responses according to four out of the five models' judges, while Mistral-7B scored lowest according to three out of five models' judges. This study highlights the potential and limitations of current LLMs for consumer health medical question answering, indicating avenues for further development.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multiple widely cited research papers use AskDocs's responses as gold references.
“There are multiple widely cited research papers using AskDocs's responses as gold references (Nguyen et al., 2023;Abrar et al., 2024;Ayers et al., 2023b), which proves its good quality among public forums.”
From MedScore: Generalizable Factuality Evaluation of Open-ended Long-form Medical Answers by Domain-adapted Claim Decomposition and Verification (2026) · cites this paper as Abrar et al., 2024
The Introduction (p. 3) says this study uses physician answers from AskDocs as ground truths and mentions other studies of AskDocs questions. It does not establish that multiple papers use AskDocs responses as gold references or that those papers are widely cited. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 28 · Cited by 8
Research works printed but not linked: 36
- Abacha, A. B., Mrabet, Y., Sharp, M., Goodwin, T. R., Shooshan, S. E., & Demner-Fushman, D. (2019). Bridging the gap between consumers' medication questions and trusted answers. In MEDINFO 2019: Health and Wellbeing e-Networks for All (pp. 25-29). IOS Press.
- Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., ... & McGrew, B. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774 . · arXiv
- Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., ... & Smith, D. M. (2023). Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA internal medicine , 183 (6), 589-596.
- Baydaroğlu, Ö., Yeşilköy, S., Sermet, Y., & Demir, I. (2023). A comprehensive review of ontologies in the hydrology towards guiding next generation artificial intelligence applications. Journal of Environmental Informatics , 42(2), 90-107.
- Carnino, J. M., Pellegrini, W. R., Willis, M., Cohen, M. B., Paz-Lansberg, M., Davis, E. M., ... & Levi, J. R. (2024). Assessing ChatGPT's Responses to Otolaryngology Patient Questions. Annals of Otology, Rhinology & Laryngology , 00034894241249621.
- Chern, S., Chern, E., Neubig, G., & Liu, P. (2024). Can large language models be trusted for evaluation? scalable meta-evaluation of llms as evaluators via agent debate. arXiv preprint arXiv:2401.16788 . · arXiv
- Del Fiol, G., Workman, T. E., & Gorman, P. N. (2014). Clinical questions raised by clinicians at the point of care: a systematic review. JAMA internal medicine , 174 (5), 710-718.
- Demner-Fushman, D., Mrabet, Y., & Ben Abacha, A. (2020). Consumer health information and question answering: helping consumers find answers to their health-related information needs. Journal of the American Medical Informatics Association , 27 (2), 194-201.
- Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., ... & Ganapathy, R. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783 . · arXiv
- Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 . · arXiv
- Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., ... & Sayed, W. E. (2023). Mistral 7B. arXiv preprint arXiv:2310.06825 . · arXiv
- Jin, D., Pan, E., Oufattole, N., Weng, W. H., Fang, H., & Szolovits, P. (2021). What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences , 11 (14), 6421.
- Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., & Lu, X. (2019). Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146 . · arXiv
- Kanithi, P. K., Christophe, C., Pimentel, M. A., Raha, T., Saadi, N., Javed, H., ... & Khan, S. (2024). Medic: Towards a comprehensive framework for evaluating llms in clinical applications. arXiv preprint arXiv:2409.07314 . · arXiv
- Li, L., Zhou, J., Gao, Z., Hua, W., Fan, L., Yu, H., ... & Ma, S. (2024). A scoping review of using large language models (llms) to investigate electronic health records (ehrs). arXiv preprint arXiv:2405.03066 . · arXiv
- Lin, C. Y. (2004, July). Rouge: A package for automatic evaluation of summaries. In Text summarization branches out (pp. 74-81).
- Myrzakhan, A., Bsharat, S. M., & Shen, Z. (2024). Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena. arXiv preprint arXiv:2406.07545 . · arXiv
- Nachane, S. S., Gramopadhye, O., Chanda, P., Ramakrishnan, G., Jadhav, K. S., Nandwani, Y., ... &Joshi, S. (2024). Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answering. arXiv preprint arXiv:2403.04890 . · arXiv
- Nguyen, V., Karimi, S., Rybinski, M., & Xing, Z. (2023, November). MedRedQA for Medical Consumer Question Answering: Dataset, Tasks, and Neural Baselines. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 629-648).
- Nguyen, V. (2024). Consumer medical question answering: Challenges and approaches (Doctoral dissertation). The Australian National University, ANU College of Engineering, Computing and Cybernetics. ANU Open Research Repository. http://hdl.handle.net/1885/311723
- Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., ... & Horvitz, E. (2023). Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452 . · arXiv
- Reddit. (2024). AskDocs . Retrieved October 27, 2024, from https://www.reddit.com/r/AskDocs/ Saab, K., Tu, T., Weng, W. H., Tanno, R., Stutz, D., Wulczyn, E., ... & Natarajan, V. (2024). Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416 . · arXiv
- Sajja, R., Sermet, Y., & Demir, I. (2024b). End-to-End Deployment of the Educational AI Hub for Personalized Learning and Engagement: A Case Study on Environmental Science Education. EarthArxiv , 7566. https://doi.org/10.31223/X5XM7N · DOI
- Sajja, R., Sermet, Y., Cikmaz, M., Cwiertny, D., & Demir, I. (2024a). Artificial intelligence-enabled intelligent assistant for personalized and adaptive learning in higher education. Information , 15(10), 596.
- Sajja, R., Sermet, Y., Cwiertny, D., & Demir, I. (2023a). Platform-independent and curriculum-oriented intelligent assistant for higher education. International Journal of Educational Technology in Higher Education , 20(1), 42.
- Sajja, R., Sermet, Y., Cwiertny, D., & Demir, I. (2023b). Integrating AI and Learning Analytics for Data-Driven Pedagogical Decisions and Personalized Interventions in Education. arXiv preprint arXiv:2312.09548. · arXiv
- Samuel, D. J., Sermet, M. Y., Mount, J., Vald, G., Cwiertny, D., & Demir, I. (2024). Application of Large Language Models in Developing Conversational Agents for Water Quality Education, Communication and Operations. EarthArxiv , 7056. https://doi.org/10.31223/X5XT4K · DOI
- Sarrouti, M., Lachkar, A., & Ouatik, S. E. A. (2015, November). Biomedical question types classification using syntactic and rule based approach. In 2015 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K) (Vol. 1, pp. 265-272). IEEE.
- Sermet, Y., & Demir, I. (2021). A semantic web framework for automated smart assistants: A case study for public health. Big Data and Cognitive Computing , 5(4), 57.
- Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., ... & Batsaikhan, B. O. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 . · arXiv
- Welivita, A., & Pu, P. (2023). A survey of consumer health question answering systems. Ai Magazine , 44 (4), 482-507.
- Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024). Benchmarking retrieval-augmented generation for medicine. arXiv preprint arXiv:2402.13178 . · arXiv
- Yao, Z., Zhang, Z., Tang, C., Bian, X., Zhao, Y., Yang, Z., ... & Yu, H. (2024). Medqa-cs: Benchmarking large language models clinical skills using an ai-sce framework. arXiv preprint arXiv:2410.01553 . · arXiv
- Zhang, M., Zhu, L., Lin, S.Y., Herr, K., Chi, C.L., Demir, I., Dunn Lopez, K. and Chi, N.C., (2023). Using artificial intelligence to improve pain assessment and pain management: a scoping review. Journal of the American Medical Informatics Association , 30(3), pp.570-587.
- Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 . · arXiv
- Zhang, X., Li, Y., Wang, J., Sun, B., Ma, W., Sun, P., & Zhang, M. (2024, October). Large language models as evaluators for recommendation explanations. In Proceedings of the 18th ACM Conference on Recommender Systems (pp. 33-42).
- Authors
- Abrar, M., Sermet, Y., & Demir, I.
- Affiliations
- University of Iowa (US)
- Corresponding author
- Sermet, Yusuf
- ORCID
- Sermet, Yusuf
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- Volume
- 5
- Issue
- 1
- Pages
- 12-12
- ISSN
- 2673-7426 (ISSN-L)
- arXiv
- 2501.00208
- OpenAlex
- W4407997385
- Cited by
- 8
- References
- 28
- Citations by year
- 2026: 4 · 2025: 4
- FWCI
- 1.94
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BioMedInformatics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Keywords
- large language models, medical question answering, LLM-as-a-judge, GPT-4o-mini, Mistral, Llama 3.1, Gemini Flash
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
