Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 71 of the 73 references it cites were found in a registry or an index; 16 of the 41 sources its claims cite were checked for support.

A large language model digital patient system enhances ophthalmology history taking skills

Open access

Abstract

Clinical trainees face limited opportunities to practice medical history-taking skills due to scarce case diversity and access to real patients. To address this, we developed a large language model-based digital patient (LLMDP) system that transforms de‑identified electronic health records into voice‑enabled virtual patients capable of free‑text dialog and adaptive feedback, based on our previously established open-source retrieval-augmented framework. In a single‑center randomized controlled trial (ClinicalTrials.gov: NCT06229379; N = 84), students trained with LLMDP achieved a 10.50-point increase in medical history-taking assessment scores (95% CI: 4.66-16.33, p < 0.001) compared to those using traditional methods. LLMDP-trained students also demonstrated greater empathy. Participants reported high satisfaction with LLMDP, emphasizing its role in reducing training costs and boosting confidence for real patient interactions. These findings provide evidence that LLM-driven digital patients enhance medical history-taking skills and offer a scalable, low-risk pathway for integrating generative AI into ophthalmology education.

Overview

CoChat Summary

TL;DR

This paper develops an LLM-based digital patient (LLMDP) system from de-identified EHRs and evaluates it for ophthalmology history-taking training. In a single-center RCT with 84 fourth-year medical students, LLMDP training improved medical history-taking assessment scores by 10.50 points versus traditional training and was associated with greater empathy and high satisfaction.

Key findings

  • In the RCT (N = 84), the LLMDP group scored 64.62 ± 9.52 on the medical history-taking assessment versus 54.12 ± 8.80 in the control group, a mean difference of 10.50 points (95% CI: 4.66–16.33, p < 0.001).
  • In a prior validation experiment with 50 students, LLMDP training produced a mean score of 78.13 ± 8.35 versus 67.08 ± 7.21 for traditional training, an average increase of 11.05 points (95% CI: 6.61–15.48, p < 0.001).
  • Automated LLMDP scoring correlated with real-patient manual scoring: Pearson’s r = 0.88 (95% CI: [0.79, 0.93], p < 0.001).
  • Participants reported high satisfaction: 48.81% agreed versus 21.42% disagreed that LLMDP achieved better teaching/learning outcomes; 66.67% agreed versus 14.28% disagreed that it saved financial costs; 65.48% agreed versus 17.85% disagreed that it helped overcome psychological barriers. LLMDP-trained students also showed better empathy.

Methods

  • The LLMDP system was built from a curated ophthalmic dataset of 59,343 instruction-output entries, selected Baichuan-13B-Chat after benchmarking against other LLMs, fine-tuned it over 20 epochs, and used knowledge bases covering 24 key ocular diseases with voice-enabled dialog and automated OSCE-based scoring.
  • Two validation experiments were conducted: one with 50 fourth-year medical students comparing LLMDP-based training with traditional real-patient training using standardized patients, and another with 50 fourth-year students comparing automated LLMDP scoring with manual real-patient scoring.
  • The main study was a single-center randomized controlled trial (ClinicalTrials.gov: NCT06229379) enrolling 84 fourth-year medical students without prior clinical training, randomized 1:1 to traditional real-patient training or LLMDP training; the primary outcome was MHTA score, analyzed with independent t tests and Benjamini–Hochberg correction for subitems.

Limitations

  • The paper states that using local language models to preserve data privacy may reduce performance relative to commercial models, requiring fine-tuning, disease knowledge bases, and prompt engineering.
  • The paper states that the system focuses on text-based history-taking only and does not simulate a physical pathophysiological body, physical examination, imaging or laboratory interpretation, or broader diagnostic and management processes.
  • The paper states that the study was conducted at a single institution and focused on a specific history-taking skillset, so broader contexts, more extensive interventions, and long-term impact across institutions and complex clinical skills remain to be established.
  • Evident from the text: participants were not blinded due to the nature of the interventions, although outcome data collection and statistical analysis were blinded.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-025-01841-6

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Overview of the study design. The LLM-based digital patient (LLMDP) system development and validation process comprises four stages: a Clinical instructors used a large language model (LLM) to extract information from electronic health records and generated a digital patient knowledge base (KB). The KB was used to construct an LLM-based digital patient (LLMDP) system. b A controlled experiment was conducted to preliminarily validate the effectiveness of LLMDP system in medical history-taking training compared to traditional patient-based training methods. c The correlation of an automated scoring model, incorporated in the LLMDP system, was tested in comparison to manual scoring to show that it provides valid scores for trainee proficiency. d A randomized clinical trial was conducted to assess the effect of the LLMDP system on students’ medical history taking abilities compared with that of traditional methods. EHR electronic health record, KB knowledge base, LLM large language model, LLMDP large language model-based digital patient, MHTA medical history-taking assessment.
Fig. 1 Overview of the study design. The LLM-based digital patient (LLMDP) system development and validation process comprises four stages: a Clinical instructors used a large language model (LLM) to extract information from electronic health records and generated a digital patient knowledge base (KB). The KB was used to construct an LLM-based digital patient (LLMDP) system. b A controlled experiment was conducted to preliminarily validate the effectiveness of LLMDP system in medical history-taking training compared to traditional patient-based training methods. c The correlation of an automated scoring model, incorporated in the LLMDP system, was tested in comparison to manual scoring to show that it provides valid scores for trainee proficiency. d A randomized clinical trial was conducted to assess the effect of the LLMDP system on students’ medical history taking abilities compared with that of traditional methods. EHR electronic health record, KB knowledge base, LLM large language model, LLMDP large language model-based digital patient, MHTA medical history-taking assessment.CC BY-NC-ND · PubMed Central
Fig. 2 Ophthalmic dataset construction, LLM selection, and model fine-tuning. Ophthalmic dataset construction, LLM benchmarking, and model fine-tuning workflow includes: a The preprocessing workflow for the ophthalmic instruction training data. EHR, guidelines, textbooks, multiple-choice questions and knowledge from ophthalmology textbooks, and patient-doctor consultation dialogs were used to construct ophthalmic dataset. After being processed into an instruction-output format, the accuracy of the data was sampled and checked by ophthalmologists. b The source distribution of the ophthalmic instruction dataset. c The distribution of ocular diseases in the ophthalmic instruction dataset. d A workflow for comparing LLMs using the Chatbot Arena approach. The LLMs were given 50 instruction tasks in Chinese, including ophthalmic knowledge, EHR information extraction, doctor’s question paraphrasing, patient’s answer paraphrasing and consultation-related queries. Two ophthalmologists compared the output of the two anonymized LLMs. Finally, the win rate between different models was calculated. e Human evaluation comparing the Baichuan-13B-Chat model with other models. The win, tie and loss rates of Baichuan-13B-Chat versus the other models are shown. f The BLEU and ROUGE-L performance metrics on the test set across the fine-tuning epochs. The model trained at 20 epochs was selected for the study. g Human evaluation comparing the fine-tuned Baichuan-13B-Chat model with other models. LLM large language model, BLEU Bilingual Evaluation Understudy, ROUGE-L Recall-Oriented Understudy for Gisting Evaluation - Longest Common Subsequence, EHR electronic health record.
Fig. 2 Ophthalmic dataset construction, LLM selection, and model fine-tuning. Ophthalmic dataset construction, LLM benchmarking, and model fine-tuning workflow includes: a The preprocessing workflow for the ophthalmic instruction training data. EHR, guidelines, textbooks, multiple-choice questions and knowledge from ophthalmology textbooks, and patient-doctor consultation dialogs were used to construct ophthalmic dataset. After being processed into an instruction-output format, the accuracy of the data was sampled and checked by ophthalmologists. b The source distribution of the ophthalmic instruction dataset. c The distribution of ocular diseases in the ophthalmic instruction dataset. d A workflow for comparing LLMs using the Chatbot Arena approach. The LLMs were given 50 instruction tasks in Chinese, including ophthalmic knowledge, EHR information extraction, doctor’s question paraphrasing, patient’s answer paraphrasing and consultation-related queries. Two ophthalmologists compared the output of the two anonymized LLMs. Finally, the win rate between different models was calculated. e Human evaluation comparing the Baichuan-13B-Chat model with other models. The win, tie and loss rates of Baichuan-13B-Chat versus the other models are shown. f The BLEU and ROUGE-L performance metrics on the test set across the fine-tuning epochs. The model trained at 20 epochs was selected for the study. g Human evaluation comparing the fine-tuned Baichuan-13B-Chat model with other models. LLM large language model, BLEU Bilingual Evaluation Understudy, ROUGE-L Recall-Oriented Understudy for Gisting Evaluation - Longest Common Subsequence, EHR electronic health record.CC BY-NC-ND · PubMed Central
Fig. 3 Development and demonstration of the LLMDP system. The framework integrates knowledge base construction, clinical interaction simulation, multi-platform deployment, and automated feedback generation. a Using the LLM to construct a KB for DPs. After 24 eye diseases were selected from a clinical teaching syllabus, a clinical inquiry question set was compiled, and EHRs for relevant clinical cases were collected. A LLM was used to answer questions based on patient notes and creating patient information sheets that were weighted according to clinical significance. The sheets were then combined with slit-lamp photographs to form a comprehensive KB for DPs. b An example of a history-taking process between the user and LLMDP system. Medical students’ questions were transcribed and matched to the KB for answers. If a match was found, then the LLMDP system provided as an output; otherwise, the LLMDP system created an answer from the patient notes. Responses were then converted into colloquial language by the LLM, transformed into speech, and provided to students through the HCI interface. c The LLMDP system can be accessed through a web browser on various devices (e.g., iPad, smartphone, PC). ① Initiate consultation by voice input; ② make corrections to the inquiry text if necessary; ③ the LLMDP system responds via voice output. d Concluding the consultation, reviewing the score and personalized guidance, and accessing detailed feedback. ① After completing the consultation, click “End” to view the score and receive personalized guidance. ② The exam results are displayed. ③ Personalized guidance is shown. ④ Drag the slider down to view the full text of the feedback. LLM large language model, DP digital patient, KB knowledge base, TTS text-to-speech, ASR automatic speech recognition, HCI human-computer interface, LLMDP large language model-based digital patient, EHR electronic health record. The language used in the study was originally Chinese and was translated into English for presentation purposes.
Fig. 3 Development and demonstration of the LLMDP system. The framework integrates knowledge base construction, clinical interaction simulation, multi-platform deployment, and automated feedback generation. a Using the LLM to construct a KB for DPs. After 24 eye diseases were selected from a clinical teaching syllabus, a clinical inquiry question set was compiled, and EHRs for relevant clinical cases were collected. A LLM was used to answer questions based on patient notes and creating patient information sheets that were weighted according to clinical significance. The sheets were then combined with slit-lamp photographs to form a comprehensive KB for DPs. b An example of a history-taking process between the user and LLMDP system. Medical students’ questions were transcribed and matched to the KB for answers. If a match was found, then the LLMDP system provided as an output; otherwise, the LLMDP system created an answer from the patient notes. Responses were then converted into colloquial language by the LLM, transformed into speech, and provided to students through the HCI interface. c The LLMDP system can be accessed through a web browser on various devices (e.g., iPad, smartphone, PC). ① Initiate consultation by voice input; ② make corrections to the inquiry text if necessary; ③ the LLMDP system responds via voice output. d Concluding the consultation, reviewing the score and personalized guidance, and accessing detailed feedback. ① After completing the consultation, click “End” to view the score and receive personalized guidance. ② The exam results are displayed. ③ Personalized guidance is shown. ④ Drag the slider down to view the full text of the feedback. LLM large language model, DP digital patient, KB knowledge base, TTS text-to-speech, ASR automatic speech recognition, HCI human-computer interface, LLMDP large language model-based digital patient, EHR electronic health record. The language used in the study was originally Chinese and was translated into English for presentation purposes.CC BY-NC-ND · PubMed Central
Fig. 4 Comparison of the effectiveness between traditional teaching methods and the LLMDP system using standardized patients during the development phase. A validation study demonstrates enhanced clinical competency through standardized assessments. a Study process: A group of 50 participants from the Zhongshan School of Medicine, Sun Yat-sen University, who had not received prior clinical training, were divided equally into two groups. The experimental group of 25 participants used the LLMDP for training, while the control group of 25 participants interacted with real patients. After completing of their respective training sessions, both groups underwent a standardized evaluation using standardized patients and were scored by ophthalmologists. A total of 6 SPs were used for this examination, each uniformly trained to play the same patient role to ensure fairness. b Performance comparison evaluated using standardized patients and scored by ophthalmologists. The LLMDP group (78.13 ± 8.35, mean ± SD) demonstrated a performance improvement in comparison to the control group (67.08 ± 7.21, mean ± SD), with an average score increase of 11.05 points (mean difference, 6.61–15.48, 95% CI, p < 0.001). LLMDP large language model-based digital patient, SPs standardized patients.
Fig. 4 Comparison of the effectiveness between traditional teaching methods and the LLMDP system using standardized patients during the development phase. A validation study demonstrates enhanced clinical competency through standardized assessments. a Study process: A group of 50 participants from the Zhongshan School of Medicine, Sun Yat-sen University, who had not received prior clinical training, were divided equally into two groups. The experimental group of 25 participants used the LLMDP for training, while the control group of 25 participants interacted with real patients. After completing of their respective training sessions, both groups underwent a standardized evaluation using standardized patients and were scored by ophthalmologists. A total of 6 SPs were used for this examination, each uniformly trained to play the same patient role to ensure fairness. b Performance comparison evaluated using standardized patients and scored by ophthalmologists. The LLMDP group (78.13 ± 8.35, mean ± SD) demonstrated a performance improvement in comparison to the control group (67.08 ± 7.21, mean ± SD), with an average score increase of 11.05 points (mean difference, 6.61–15.48, 95% CI, p < 0.001). LLMDP large language model-based digital patient, SPs standardized patients.CC BY-NC-ND · PubMed Central
Fig. 5 Convergent validity of LLMDP-based assessments against real patient evaluations. Evaluation of the correlation between the assessment methods of real patient encounters and LLMDP-based scenarios. a Study process: Another set of 50 fourth-year medical students from the Zhongshan School of Medicine, Sun Yat-sen University, who were undergoing traditional clinical internships were included. Participants were consecutively evaluated twice using both assessment methods (real patient encounters and LLMDP-based scenarios) for their medical history-taking assessment (MHTA). b Scatter plot comparing the evaluation scores measured using the two methods. Each data point on the graph represents a participant, with the position on the plot corresponding to the scores from the two methods. A moderate positive correlation between the two methods was shown. Pearson’s correlation coefficient r = 0.88 (95% CI: [0.79, 0.93]). LLMDP large language model-based digital patient; MHTA: medical history-taking assessment;.
Fig. 5 Convergent validity of LLMDP-based assessments against real patient evaluations. Evaluation of the correlation between the assessment methods of real patient encounters and LLMDP-based scenarios. a Study process: Another set of 50 fourth-year medical students from the Zhongshan School of Medicine, Sun Yat-sen University, who were undergoing traditional clinical internships were included. Participants were consecutively evaluated twice using both assessment methods (real patient encounters and LLMDP-based scenarios) for their medical history-taking assessment (MHTA). b Scatter plot comparing the evaluation scores measured using the two methods. Each data point on the graph represents a participant, with the position on the plot corresponding to the scores from the two methods. A moderate positive correlation between the two methods was shown. Pearson’s correlation coefficient r = 0.88 (95% CI: [0.79, 0.93]). LLMDP large language model-based digital patient; MHTA: medical history-taking assessment;.CC BY-NC-ND · PubMed Central
Fig. 6 A randomized controlled trial testing the efficacy of LLMDP training in clinical history taking education. A RCT with 84 medical trainees with no prior clinical experience compares ophthalmology history-taking performance across training modalities. a The trial profile. Out of 84 assessed participants, all subjects were included and randomized into control and LLMDP groups with no withdrawals. They underwent initial training and the medical history-taking assessment (MHTA). b Participants’ clinical consultation scores on MHTA was 54.12 ± 8.80 for the control group and 64.62 ± 9.52 for the LLMDP group, showing a mean difference of 10.50, 95% CI: 4.66–16.33, p < 0.001. Symbols: Blue circle (control), orange circle (LLMDP). The error bars represent the SDs. The data are shown as the means ± SDs and were analyzed using independent t tests with two-sided p values (p < 0.05 was considered to indicate statistical significance). MHTA medical history taking assessment, LLMDP large language model-based digital patient.
Fig. 6 A randomized controlled trial testing the efficacy of LLMDP training in clinical history taking education. A RCT with 84 medical trainees with no prior clinical experience compares ophthalmology history-taking performance across training modalities. a The trial profile. Out of 84 assessed participants, all subjects were included and randomized into control and LLMDP groups with no withdrawals. They underwent initial training and the medical history-taking assessment (MHTA). b Participants’ clinical consultation scores on MHTA was 54.12 ± 8.80 for the control group and 64.62 ± 9.52 for the LLMDP group, showing a mean difference of 10.50, 95% CI: 4.66–16.33, p < 0.001. Symbols: Blue circle (control), orange circle (LLMDP). The error bars represent the SDs. The data are shown as the means ± SDs and were analyzed using independent t tests with two-sided p values (p < 0.05 was considered to indicate statistical significance). MHTA medical history taking assessment, LLMDP large language model-based digital patient.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

11 Valid · 3 Not valid · 12 Not checked · 59 Own results and interpretations

26 more claims · 1 with their cited sources checked

Create a free account to see all 85 claims and to verify the 2 references still open.

See all claims — free account
  • Probably not a substantial claim. 46
  • It may state two separate claims as one. 19

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

1 of 1

Citations

Cites

Show all 73 references

Research works printed but not linked: 4

  1. Kolb, D. A. Experience as the Source of Learning and Development (Prentice Hall, 1984).
  2. Kolb, D. A. Experiential Learning: Experience as the Source of Learning and Development (FT Press, 2014).
  3. Hu, E. J. et al. Lora: Low-rank adaptation of large language models. ICLR 1, 3 (2022).
  4. Kim, J., Kong, J. & Son, J. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. in International Conference on Machine Learning 5530-–5540 (PMLR, 2021).

Cited by

See all 39

Metadata

Authors
Luo, M.-J., Bi, S., Pang, J., Liu, L., Tsui, C.-K., Lai, Y., Chen, W., Yang, Y., Xu, K., Zhao, L., Jin, L., Lin, D., Wu, X., Chen, J., Chen, R., Liu, Z., Zou, Y., Yang, Y., Li, Y., & Lin, H.
Affiliations
Sun Yat-sen University (CN)
Corresponding author
Lin, Haotian
Venue
npj Digital Medicine
Published
2025-08-04
Volume
8
Issue
1
Pages
502-502
ISSN
2398-6352 (ISSN-L)
OpenAlex
W4412929707
Cited by
39
References
73
Citations by year
2026: 35 · 2025: 4
FWCI
9.39
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Natural Science Foundation of Guangdong Province (2024A1515012292)
  • Major Program of the National Natural Science Foundation of China (92368205)
  • National Natural Science Foundation of China (82371111)
  • Science and Technology Planning Projects of Guangdong Province (2018B010109008)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Ophthalmology › Ophthalmology and Visual Health Research
  • Social Sciences › Education › Simulation-Based Education in Healthcare
Keywords
large language models, digital patient, medical history, ophthalmology education, clinical trainees, retrieval-augmented generation, electronic health records, adaptive feedback, empathy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A large language model digital patient system enhances ophthalmology history taking skills. https://app.cochat.ai/papers/10.1038/s41746-025-01841-6
The paper
Luo, M.-J., Bi, S., Pang, J., Liu, L., Tsui, C.-K., Lai, Y., Chen, W., Yang, Y., Xu, K., Zhao, L., Jin, L., Lin, D., Wu, X., Chen, J., Chen, R., Liu, Z., Zou, Y., Yang, Y., Li, Y., & Lin, H. (2025). A large language model digital patient system enhances ophthalmology history taking skills. npj Digital Medicine. https://doi.org/10.1038/s41746-025-01841-6

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.