The DOI matches its Crossref record; 71 of the 73 references it cites were found in a registry or an index; 16 of the 41 sources its claims cite were checked for support.
A large language model digital patient system enhances ophthalmology history taking skills
Open accessAbstract
Clinical trainees face limited opportunities to practice medical history-taking skills due to scarce case diversity and access to real patients. To address this, we developed a large language model-based digital patient (LLMDP) system that transforms de‑identified electronic health records into voice‑enabled virtual patients capable of free‑text dialog and adaptive feedback, based on our previously established open-source retrieval-augmented framework. In a single‑center randomized controlled trial (ClinicalTrials.gov: NCT06229379; N = 84), students trained with LLMDP achieved a 10.50-point increase in medical history-taking assessment scores (95% CI: 4.66-16.33, p < 0.001) compared to those using traditional methods. LLMDP-trained students also demonstrated greater empathy. Participants reported high satisfaction with LLMDP, emphasizing its role in reducing training costs and boosting confidence for real patient interactions. These findings provide evidence that LLM-driven digital patients enhance medical history-taking skills and offer a scalable, low-risk pathway for integrating generative AI into ophthalmology education.
Overview
CoChat Summary
TL;DR
This paper develops an LLM-based digital patient (LLMDP) system from de-identified EHRs and evaluates it for ophthalmology history-taking training. In a single-center RCT with 84 fourth-year medical students, LLMDP training improved medical history-taking assessment scores by 10.50 points versus traditional training and was associated with greater empathy and high satisfaction.
Key findings
- In the RCT (N = 84), the LLMDP group scored 64.62 ± 9.52 on the medical history-taking assessment versus 54.12 ± 8.80 in the control group, a mean difference of 10.50 points (95% CI: 4.66–16.33, p < 0.001).
- In a prior validation experiment with 50 students, LLMDP training produced a mean score of 78.13 ± 8.35 versus 67.08 ± 7.21 for traditional training, an average increase of 11.05 points (95% CI: 6.61–15.48, p < 0.001).
- Automated LLMDP scoring correlated with real-patient manual scoring: Pearson’s r = 0.88 (95% CI: [0.79, 0.93], p < 0.001).
- Participants reported high satisfaction: 48.81% agreed versus 21.42% disagreed that LLMDP achieved better teaching/learning outcomes; 66.67% agreed versus 14.28% disagreed that it saved financial costs; 65.48% agreed versus 17.85% disagreed that it helped overcome psychological barriers. LLMDP-trained students also showed better empathy.
Methods
- The LLMDP system was built from a curated ophthalmic dataset of 59,343 instruction-output entries, selected Baichuan-13B-Chat after benchmarking against other LLMs, fine-tuned it over 20 epochs, and used knowledge bases covering 24 key ocular diseases with voice-enabled dialog and automated OSCE-based scoring.
- Two validation experiments were conducted: one with 50 fourth-year medical students comparing LLMDP-based training with traditional real-patient training using standardized patients, and another with 50 fourth-year students comparing automated LLMDP scoring with manual real-patient scoring.
- The main study was a single-center randomized controlled trial (ClinicalTrials.gov: NCT06229379) enrolling 84 fourth-year medical students without prior clinical training, randomized 1:1 to traditional real-patient training or LLMDP training; the primary outcome was MHTA score, analyzed with independent t tests and Benjamini–Hochberg correction for subitems.
Limitations
- The paper states that using local language models to preserve data privacy may reduce performance relative to commercial models, requiring fine-tuning, disease knowledge bases, and prompt engineering.
- The paper states that the system focuses on text-based history-taking only and does not simulate a physical pathophysiological body, physical examination, imaging or laboratory interpretation, or broader diagnostic and management processes.
- The paper states that the study was conducted at a single institution and focused on a specific history-taking skillset, so broader contexts, more extensive interventions, and long-term impact across institutions and complex clinical skills remain to be established.
- Evident from the text: participants were not blinded due to the nature of the interventions, although outcome data collection and statistical analysis were blinded.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




![Fig. 5 Convergent validity of LLMDP-based assessments against real patient evaluations. Evaluation of the correlation between the assessment methods of real patient encounters and LLMDP-based scenarios. a Study process: Another set of 50 fourth-year medical students from the Zhongshan School of Medicine, Sun Yat-sen University, who were undergoing traditional clinical internships were included. Participants were consecutively evaluated twice using both assessment methods (real patient encounters and LLMDP-based scenarios) for their medical history-taking assessment (MHTA). b Scatter plot comparing the evaluation scores measured using the two methods. Each data point on the graph represents a participant, with the position on the plot corresponding to the scores from the two methods. A moderate positive correlation between the two methods was shown. Pearson’s correlation coefficient r = 0.88 (95% CI: [0.79, 0.93]). LLMDP large language model-based digital patient; MHTA: medical history-taking assessment;.](https://figures.cochat.ai/paper-figures/2b9a50bc9e4342c68d47051ef9f3990b/a-large-language-model-digital-patient-system-enhances-ophthalmology-history-figure-5.jpg)

Verification
Claims the paper makes
11 Valid · 3 Not valid · 12 Not checked · 59 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, the standard case-based learning approach often faces a significant shortage of materials featuring authentic clinical cases5,6.”
§ A large language model digital patient system enhances ophthalmology history taking skills > Introduction
What the sources say
- Could not verifycites [5] McLean, S. F. Case-based learning and its application in medical and health-care fields: a review of worldwide literature. J. Med Educ. Curric. Dev. 3, JMECD.S20377 (2016). sourceExperimental — accuracy not yet measured
The Introduction describes authentic clinical cases as central to case-based learning, but does not say that materials featuring them are scarce. Searches for a shortage of cases, teaching materials, and resources found no passage establishing the claimed shortage; the Delivery of CBL section discusses a different problem, inadequate faculty preparation. (LLM, conf=0.73)
- Partly supportedcites [6] AlHaqwi, A. I. & Taha, W. S. Promoting excellence in teaching and learning in clinical education. J. Taibah Univ. Med. Sci. 10, 97–101 (2015). sourceExperimental — accuracy not yet measured
The 'Documentation of students’ experiences' section (page unavailable) says that a limited number of clinical cases has been a main difficulty for clinical teaching. It does not identify a shortage of materials featuring authentic cases or specifically discuss the standard case-based learning approach. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“By processing extensive medical literature and electronic health records (EHRs), LLMs has demonstrated the ability to extract detailed clinical information from patient discharge summaries38.”
§ A large language model digital patient system enhances ophthalmology history taking skills > Introduction
What the sources say
- Partly supportedcites [38] Stanceski, K. et al. The quality and safety of using generative AI to produce patient-centred discharge instructions. NPJ Digit Med. 7, 329 (2024). sourceExperimental — accuracy not yet measured
The paper’s results report that GPT-3.5-generated instructions captured most medications and follow-up actions from 100 discharge summaries, supporting the narrower claim that it can extract clinical details. The Methods (Data sources) say the study used discharge summaries from MIMIC-IV, not extensive medical literature or other EHR information. The results also document omissions and added actions, so the ability was not uniformly reliable. (LLM, conf=0.86)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Our system employs local language models to preserve data privacy, which comes at some cost to performance – the local model may not match that of commercial models58,59.”
§ A large language model digital patient system enhances ophthalmology history taking skills > Discussions
What the sources say
- Not checked — no full text of the sourcecites [58] Wan, P. et al. Outpatient reception via collaboration between nurses and a large language model: a randomized controlled trial. Nat. Med. 30, 2878–2885 (2024). source
- Could not verifycites [59] Van Veen, D. et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat. Med. 30, 1134–1142 (2024). sourceExperimental — accuracy not yet measured
The Discussion (page unavailable) says proprietary models raise data- and model-governance concerns and suggests that this could motivate using open-source models. It does not say that locally run language models preserve data privacy; searches for local hosting and privacy found no passage establishing that point. (LLM, conf=0.72)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“This finding parallels prior work showing that AI-driven virtual patient simulations can boost learner empathy: for example, trainees interacting with virtual patients in low-pressure, interactive settings produce more empathic responses53, and virtual patient encounters that include empathic feedback have yielded higher empathy scores in later standardized‐patient interviews54.”
§ A large language model digital patient system enhances ophthalmology history taking skills > Discussions
What the sources say
- Not checked — no full text of the sourcecites [53] Foster, A. et al. Using virtual patients to teach empathy: a randomized controlled study to enhance medical students’ empathic communication. Simul. Healthc. 11, 181–189 (2016). source
- Supportedcites [54] Kleinsmith, A., Rivera-Gutierrez, D., Finney, G., Cendan, J. & Lok, B. Understanding empathy training with virtual patients. Comput Hum. Behav. 52, 151–158 (2015). source
The Results (section 5; page unavailable) report that medical students’ responses to virtual patients were rated significantly more empathetic than their responses to standardized patients. The Abstract describes the virtual-patient interaction as a low-pressure setting for practicing empathetic communication. This supports the claim attached to citation 53; the later standardized-patient scores are attributed to citation 54, not this paper. (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, the current capabilities of LLMs, including advanced models like GPT-4V, remain limited in their understanding and processing of medical images56,57”
§ A large language model digital patient system enhances ophthalmology history taking skills > Discussions
What the sources say
- Supportedcites [56] Jin, Q. et al. Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine. NPJ Digit Med. 7, 190 (2024). source
The Abstract says most LLMs use only text despite clinical tasks requiring images, and identifies image comprehension as GPT-4V’s greatest challenge, with an error rate above 20%. This supports the broad claim that current LLMs, including GPT-4V, remain limited in handling medical images. (LLM, conf=0.89)
- Not checked — no full text of the sourcecites [57] Xu, P., Chen, X., Zhao, Z. & Shi, D. Unveiling the clinical incapabilities: a benchmarking study of GPT-4V(ision) for ophthalmic multimodal image analysis. Br. J. Ophthalmol. 108, 1384–1389 (2024). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
26 more claims · 1 with their cited sources checked
Create a free account to see all 85 claims and to verify the 2 references still open.
- Probably not a substantial claim. 46
- It may state two separate claims as one. 19
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“However, learners were required to review and correct transcribed text before eliciting a spoken response, introducing an explicit verification step that interrupts conversational flow.”
From Development and Validation of CPX-MATE: An End-to-End Medical Education Platform Integrating Voice-Based Virtual Patient Simulation and Automated Real-time Evaluation (2026) · cites this paper as [10]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Prior studies have demonstrated the technical feasibility of text-based VSPs and AI-assisted automated scoring systems.”
The cited source says what the claim attributes to it.
In the citing paper
“Likewise, Luo et al. reported that an LLM-based digital patient system enhanced ophthalmology history-taking skills and incorporated voice interaction via a chained STT-LLM-TTS architecture. 10”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Heard of but never used | 16 (38.1) | 15 (35.1) | | Used occasionally | 21 (50.0) | 15 (35.1) |”
§ A large language model digital patient system enhances ophthalmology history taking skills > Results > Clinical trial to evaluate LLMDP’s effectiveness in medical history taking
Reported 35.1 · recomputed 35.714286
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1×
“Then,thesystemmatchedtheinquiryquestionswiththe KB of the DPs using Sentence Bidirectional Encoder Representations from Transformers (SBERT) embeddings (distiluse-base-multilingual-cased-https v1) 66 as a semantic textual similarity model.”
Standards for Reporting Qualitative Research: A Synthesis of Recommendations
O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014) · Academic Medicine · doi:10.1097/acm.0000000000000388 · Cited by 12,211 · cited 2×
“…assigned to each question in the patient information sheet based on the diagnostic signi fi cance of each question after the expert panel discussion 64 (Supplementary Tables 6 and 7), with speci fi c scores calculated according to a prede fi ned rubric: 5 points for patient identi fi cation and…”
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083 · cited 1×
“Theirestablished effectiveness in medical consultations laysthe foundation for exploring their potential impact on medical education 35 – 37 .”
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“Theirestablished effectiveness in medical consultations laysthe foundation for exploring their potential impact on medical education 35 – 37 .”
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · cited 2×
“Large language models (LLMs), such as ChatGPT (Open AI, San Francisco, CA, USA), represent a recent signi fi cant advancement in healthcare 22 – 24 .”
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · cited 1×
“Through a literature search, we found that LLM applications in medicine such as answering medical examination questions 25,26 , providing diagnostic and treatment recommendations 27,28 , summarizing clinical research articles 29 , and generating patient education materials 30 are…”
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
The Lancet Global Health Commission on Global Eye Health: vision beyond 2020
Burton, M. J., Ramke, J., Marques, A. P., Bourne, R. R. A., Congdon, N., Jones, I., Ah Tong, B. A. M., Arunga, S., Bachani, D., Bascaran, C., et al. (2021) · The Lancet Global Health · doi:10.1016/s2214-109x(20)30488-5 · Cited by 1,624 · cited 1×
“This capability is crucial for creating VPs that accurately simulate real patients for educational purposes 39,40 .”
Research works printed but not linked: 4
- Kolb, D. A. Experience as the Source of Learning and Development (Prentice Hall, 1984).
- Kolb, D. A. Experiential Learning: Experience as the Source of Learning and Development (FT Press, 2014).
- Hu, E. J. et al. Lora: Low-rank adaptation of large language models. ICLR 1, 3 (2022).
- Kim, J., Kong, J. & Son, J. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. in International Conference on Machine Learning 5530-–5540 (PMLR, 2021).
Cited by
Wang, C., Sun, N., Pan, X., Liu, P., Ma, Z., Wang, W., & Jin, Y. (2026) · BMC Medical Education · doi:10.1186/s12909-026-09320-6 · Cited by 23
“Luo et al. [14] found that their LLM-based digital patient raised history-taking scores across every subdomain they measured (demo-graphics, present illness, past history, etc.), with voice-based feedback that correlated well with human raters ( r = 0.88).”
InfluentialDávidovics, A., Dávidovics, K., Hillebrand, P., Rendeki, S., & Németh, T. (2025) · BMC Medical Education · doi:10.1186/s12909-025-08507-7 · Cited by 12
“Emerging evidence on AI-enhanced virtual patients suggests similar benefits for history-taking and interviewing skills [40, 41], although concerns regarding accuracy and bias remain [42].”
GenAI-Supported Virtual Patients in Health Care Education: Systematic Review
Jiang, J., Ye, M. Z., Kwok, T. T.-O., & Wong, J. Y. H. (2026) · Journal of Medical Internet Research · doi:10.2196/82756 · Cited by 22
“The 3 RCTs provide the most robust evidence, demonstrating significant improvements in clinical decision-making [41], ophthalmology history-taking skills [45], and medical history-taking performance [47] attributable to GenAI-supported virtual patient interventions.”
InfluentialSheth, U., Lo, M., McCarthy, J., Baath, N., Last, N., Guo, E., Monteiro, S., & Sibbald, M. (2025) · International Medical Education · doi:10.3390/ime4040039 · Cited by 11
“From a technology adoption lens, LLM-based virtual patients are scalable and on-demand, allowing asynchronous, self-paced access to clinical scenarios at relatively low cost [10,12,13].”
Wang, M., Xiao, L., Jiao, J., Sun, M., Ni, Z., Xiao, J., Tan, M., Zhang, H., Liu, W., & Ding, J. (2026) · Nurse Education Today · doi:10.1016/j.nedt.2026.107008 · Cited by 6
Metadata
- Authors
- Luo, M.-J., Bi, S., Pang, J., Liu, L., Tsui, C.-K., Lai, Y., Chen, W., Yang, Y., Xu, K., Zhao, L., Jin, L., Lin, D., Wu, X., Chen, J., Chen, R., Liu, Z., Zou, Y., Yang, Y., Li, Y., & Lin, H.
- Affiliations
- Sun Yat-sen University (CN)
- Corresponding author
- Lin, Haotian
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2025-08-04
- Volume
- 8
- Issue
- 1
- Pages
- 502-502
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC12322286
- OpenAlex
- W4412929707
- Semantic Scholar
- 0688a4ed8971d339ae2f294a299ce0e9b836fb62
- Cited by
- 39
- References
- 73
- Citations by year
- 2026: 35 · 2025: 4
- FWCI
- 9.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Natural Science Foundation of Guangdong Province (2024A1515012292)
- Major Program of the National Natural Science Foundation of China (92368205)
- National Natural Science Foundation of China (82371111)
- Science and Technology Planning Projects of Guangdong Province (2018B010109008)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
- Social Sciences › Education › Simulation-Based Education in Healthcare
- Keywords
- large language models, digital patient, medical history, ophthalmology education, clinical trainees, retrieval-augmented generation, electronic health records, adaptive feedback, empathy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A large language model digital patient system enhances ophthalmology history taking skills. https://app.cochat.ai/papers/10.1038/s41746-025-01841-6
- The paper
- Luo, M.-J., Bi, S., Pang, J., Liu, L., Tsui, C.-K., Lai, Y., Chen, W., Yang, Y., Xu, K., Zhao, L., Jin, L., Lin, D., Wu, X., Chen, J., Chen, R., Liu, Z., Zou, Y., Yang, Y., Li, Y., & Lin, H. (2025). A large language model digital patient system enhances ophthalmology history taking skills. npj Digital Medicine. https://doi.org/10.1038/s41746-025-01841-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.