Clinfo.ai: An Open-Source Retrieval-Augmented Large Language Model System for Answering Medical Questions using Scientific Literature
Open accessAbstract
The quickly-expanding nature of published medical literature makes it challenging for clinicians and researchers to keep up with and summarize recent, relevant findings in a timely manner. While several closed-source summarization tools based on large language models (LLMs) now exist, rigorous and systematic evaluations of their outputs are lacking. Furthermore, there is a paucity of high-quality datasets and appropriate benchmark tasks with which to evaluate these tools. We address these issues with four contributions: we release Clinfo.ai, an open-source WebApp that answers clinical questions based on dynamically retrieved scientific literature; we specify an information retrieval and abstractive summarization task to evaluate the performance of such retrieval-augmented LLM systems; we release a dataset of 200 questions and corresponding answers derived from published systematic reviews, which we name PubMed Retrieval and Synthesis (PubMedRS-200); and report benchmark results for Clinfo.ai and other publicly available OpenQA systems on PubMedRS-200.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most existing Modular RAG systems in healthcare rely solely on either sparse (BM25) or dense (MedCPT, embedding-based) retrieval methods.
“Most existing Modular RAG systems in healthcare, such as Self-BioRAG [16] and Clinfo.ai[24], rely solely on either sparse (BM25) or dense (MedCPT [25], embedding-based) retrieval methods.”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [24]
Extensive literature consistently reports that domain-specific models such as BioGPT are prone to hallucination or generating content beyond the retrieved evidence unless explicitly fine-tuned for retrieval-grounded generation.
“Moreover, extensive literature consistently reports that while domain-specific models such as BioGPT excel at producing semantically rich and lexically appropriate text, they are prone to hallucination or generating content beyond the retrieved evidence unless explicitly fine-tuned for retrieval-grounded generation [16,24,29,32]”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [24]
The citation lists haven’t been fetched yet
Cited by 55
- Authors
- Lozano, A., Fleming, S. L., Chiang, C.-C., & Shah, N.
- Affiliations
- Stanford University (US)
- Mayo Clinic (US)
- Stanford Health Care (US)
- Venue
- Biocomputing 2024
- Publisher
- WORLD SCIENTIFIC
- Published
- 2023-12
- Volume
- 29
- Pages
- 8-23
- OpenAlex
- W4389868615
- Cited by
- 55
- References
- 0
- Citations by year
- 2026: 12 · 2025: 24 · 2024: 19
- FWCI
- 14.15
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Pacific Symposium on Biocomputing (PSB) (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- retrieval-augmented generation, large language models, medical question answering, scientific information retrieval, abstractive summarization, open-source web application, systematic reviews, benchmark dataset
- MeSH terms
- Humans, Language, Natural Language Processing, Information Storage and Retrieval, Computational Biology, PubMed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
