ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge
Open accessAbstract
The primary aim of this research was to address the limitations observed in the medical knowledge of prevalent large language models (LLMs) such as ChatGPT, by creating a specialized language model with enhanced accuracy in medical advice. We achieved this by adapting and refining the large language model meta-AI (LLaMA) using a large dataset of 100,000 patient-doctor dialogues sourced from a widely used online medical consultation platform. These conversations were cleaned and anonymized to respect privacy concerns. In addition to the model refinement, we incorporated a self-directed information retrieval mechanism, allowing the model to access and utilize real-time information from online sources like Wikipedia and data from curated offline medical databases. The fine-tuning of the model with real-world patient-doctor interactions significantly improved the model's ability to understand patient needs and provide informed advice. By equipping the model with self-directed information retrieval from reliable online and offline sources, we observed substantial improvements in the accuracy of its responses. Our proposed ChatDoctor, represents a significant advancement in medical LLMs, demonstrating a significant improvement in understanding patient inquiries and providing accurate advice. Given the high stakes and low error tolerance in the medical field, such enhancements in providing accurate and reliable information are not only beneficial but essential.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing work has explored how AI may empower healthcare work for providers, focusing on Clinical Decision Support Systems (CDSS) and clinical practices, highlighting how AI may assist human experts throughout medical diagnosis and treatments.
“Existing work has explored how AI may empower healthcare work for providers, focusing on Clinical Decision Support Systems (CDSS) and clinical practices, highlighting how AI may assist human experts throughout medical diagnosis and treatments [41,73,76,115,127].”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [127]
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredChatDoctor adapts LLM-based agents to the medical domain; it fine-tunes an acquisition process to retrieve external knowledge from Wikipedia and medical databases.
“ChatDoctor [115] adapts LLM-based agents to the medical domain. It fine-tunes an acquisition process to retrieve external knowledge from Wikipedia and medical databases.”
From A Survey on the Memory Mechanism of Large Language Model-based Agents (2025) · cites this paper as [115]
The Introduction says ChatDoctor fine-tunes an LLM for medical use, and the Development of autonomous ChatDoctor with knowledge brain section describes retrieval from Wikipedia or a custom medical database. That section says the retrieval process uses prompts and term matching, not that the acquisition process itself was fine-tuned. (LLM, conf=0.88)
Experimental — accuracy not yet measuredIn medicine, most previous works empower LLM-based agents with external knowledge in their memory.
“In the field of medicine, most of the previous works empower LLM-based agents with external knowledge in their memory [107, 129, 132, 151, 115].”
From A Survey on the Memory Mechanism of Large Language Model-based Agents (2025) · cites this paper as [115]
The Creation of external knowledge database and Development of autonomous ChatDoctor with knowledge brain sections describe ChatDoctor using an external medical knowledge database. They support this paper as an example, but do not establish that most previous medical LLM works used external knowledge in memory. (LLM, conf=0.83)
Experimental — accuracy not yet measuredWhen LLMs are confronted with tasks requiring domain-specific expertise, such as medical and legal questions, they may exhibit pronounced hallucinations, often manifesting as factual fabrication.
“This limitation becomes particularly evident when LLMs are confronted with tasks that require domain-specific expertise, such as medical [179, 279] and legal [149, 353] questions, these models may exhibit pronounced hallucinations, often manifesting as factual fabrication.”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [179]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 16
- Training language models to follow instructions with human feedback . (2022). Accessed: April 3, 2023: http://arXiv:2203.02155.
- Self-instruct: aligning language model with self generated instructions . (2022). Accessed: December 20, 2022: http://arXiv:2212.10560.
- Aidan Gilson, Conrad W Safranek, Thomas Huang, et al.: How does chatgpt perform on the united states medical licensing examination? the implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023, 9:45312-2023.
- Abacha AB, Zweigenbaum P: Means: a medical question-answering system combining NLP techniques and semantic web technologies. Inf Process Manag. 2015, 51:570-94.
- Bertscore: Evaluating text generation with bert . (2020). Accessed: April 21, 2020: http://arXiv:1904.09675.
- Gessain A, Nakoune E, Yazdanpanah Y: Monkeypox. N Engl J Med. 2022, 387:1783-93. 10.1056/NEJMra2208860 · DOI
- A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity . (2023). Accessed: February 8, 2023: http://arXiv:2302.04023.
- Selfcheckgpt: zero-resource black-box hallucination detection for generative large language models . (2023). Accessed: March 15, 2023: http://arXiv:2303.08896.
- Salvagno M, Taccone FS, Gerli AG: Artificial intelligence hallucinations. Crit Care. 2023, 27:180. 10.1186/s13054-023-04473-y · DOI · Paper page
- Beutel G, Geerits E, Kielstein JT: Artificial hallucination: GPT on LSD? . Crit Care. 2023, 27:148. 10.1186/s13054-023-04425-6 · DOI · Paper page
- LLaMA: open and efficient foundation language models. (2023). Accessed: February 27, 2023: http://arXiv:2302.13971.
- Raise a child in large language model: towards effective and generalizable fine-tuning . (2021). Accessed: September 13, 2021: http://arXiv:2109.05687.
- Hammerling JA: A review of medical errors in laboratory diagnostics and where we are today . Laboratory Med. 2012, 43:41-4. 10.1309/LM6ER9WJR1IHQAUY · DOI
- Lee P, Bubeck S, Petro J: Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine . New England J Med. 2023, 388:1233-9. · DOI · Paper page
- Vaishya R, Misra A, Vaish A: ChatGPT: is this version good for healthcare and research? . Diabet Metabol Syndr. 2023, 17:102744.
- Hatherley JJ: Limits of trust in medical AI. J Med Ethics. 2020, 46:478-81. 10.1136/medethics-2019-105935 · DOI
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590 · see citing works
“…by ChatGPT [34]/GPT-4 [35] and the success of open-sourced instruction-tuned large language models (LLMs) in the general domain, several biomedical LLM chatbots have been developed, including ChatDoctor [49], Med-Alpaca [13], PMC-LLaMA [47], Clinical Camel [1], DoctorGLM [48], and Huatuo [46].”
BackgroundHuang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
“As a result, when confronted with problems necessitating domain-specific knowledge, such as medical (Li et al., 2023g; Singhal et al., 2023) and legal (Yu et al., 2022; Katz et al., 2023) questions, these models may exhibit pronounced hallucinations, often manifesting as factual fabrication.”
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213 · see citing works
A Survey on the Memory Mechanism of Large Language Model-based Agents
Zhang, Z., Dai, Q., Bo, X., Ma, C., Li, R., Chen, X., Zhu, J., Dong, Z., & Wen, J.-R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3748302 · Cited by 118 · see citing works
Yang, Z., Xu, X., Yao, B., Rogers, E., Zhang, S., Intille, S., Shara, N., Gao, G. G., & Wang, D. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3659625 · Cited by 120 · see citing works
DRG-LLaMA : tuning LLaMA model to predict diagnosis-related group for hospitalized patients
Wang, H., Gao, C., Dantona, C., Hull, B., & Sun, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-023-00989-3 · Cited by 99 · see citing works
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69 · see citing works
Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding
Toma, A., Lawler, P. R., Ba, J., Krishnan, R. G., Rubin, B. B., & Wang, B. (2023) · arXiv · doi:10.48550/arxiv.2305.12031 · Cited by 36 · see citing works
Unveiling Privacy Risks in LLM Agent Memory
Wang, B., He, W., Zeng, S., Xiang, Z., Xing, Y., Tang, J., & He, P. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1227 · Cited by 24 · see citing works
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149 · see citing works
Near to Mid-term Risks and Opportunities of Open-Source Generative AI
Eiras, F., Petrov, A., Vidgen, B., de Witt, C. S., Pizzati, F., Elkins, K., Mukhopadhyay, S., Bibi, A., Csaba, B., Steibel, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.17047 · Cited by 2 · see citing works
Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator
Liao, Y., Meng, Y., Wang, Y., Liu, H., Wang, Y., & Wang, Y. (2024) · arXiv · doi:10.48550/arxiv.2403.08495 · Cited by 15 · see citing works
- Authors
- Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., & Zhang, Y.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Corresponding author
- Zhang, You
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Volume
- 15
- Issue
- 6
- Pages
- e40895-e40895
- ISSN
- 2168-8184 (ISSN-L)
- arXiv
- 2303.14070
- PMC
- PMC10364849
- OpenAlex
- W4381930847
- Cited by
- 512
- References
- 0
- Citations by year
- 2026: 90 · 2025: 219 · 2024: 164 · 2023: 39
- FWCI
- 20.45
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Cureus (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Cancer Institute (R01CA240808, R01CA258987)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- healthcare chatbots, LLaMA fine-tuning, large language models in healthcare
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
