Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index; 8 of the 54 sources its claims cite were checked for support.
Abstract
Background: Canadian psychiatry residents must demonstrate consultation competency, assessed using the standardized assessment of a clinical encounter report (STACER). However, opportunities to practice these skills and receive constructive assessment remain limited in clinical settings. Objective: This study aimed to evaluate the technical feasibility of an agentic AI system designed to support psychiatry residents’ consultation competence through simulated patient encounters with a patient agent and structured feedback from a rater agent. Methods: We conducted a two-phase technical feasibility prospective single-arm cohort study of the STACER Agentic System, a large language model–based platform integrating a patient agent and a rater agent. Phase 1 involved automated evaluation of the patient agent using a psychiatrist agent across 227 synthetic major depressive disorder cases. Performance was assessed using DeepEval metrics (correctness, clarity, medical faithfulness, turn relevance, and role adherence) with descriptive statistics and 95% CIs. Phase 2 involved a preliminary user study with 14 convenience-sampled participants: a total of 5 members of the clinical research team and 9 psychiatry residents from the University of Alberta. Participants completed simulated diagnostic interviews and case presentations. Performance was evaluated using STACER-based scoring by the rater agent and 2 psychiatrists. Interrater reliability was assessed using intraclass correlation coefficients (α=.05). Participants rated realism, behavioral consistency, psychiatric nuance, and feedback utility using Likert scales and free-text answers. Results: The patient agent demonstrated high behavioral (51/56, 91.07%) and symptom fidelity (105/110, 95.45%), with strong automated performance (medical faithfulness mean 0.99, 95% CI 0.99‐1.00; turn relevance 0.99, 95% CI 0.986‐0.992). Participants rated simulations as psychiatrically plausible and diagnostically useful, particularly for depressive symptom representation, although rapport building was moderate (mean 2.78, SD 1.56 to mean 3.00, SD 1.41, out of 5.00) due to limited nonverbal cues. The rater agent generated structured STACER-aligned feedback with high intrarater consistency, especially at the section subtotal level. Interrater reliability with psychiatrists was poor at the item level (intraclass correlation coefficient range=0.25‐0.49) but improved to good-to-excellent agreement at the section level for psychiatry resident sessions (intraclass correlation coefficient range=0.89‐0.93). The rater agent’s scores fell between those of the 2 psychiatrists for the clinical research team and were lower than both human raters for psychiatry residents. Conclusions: The STACER Agentic System demonstrates the technical feasibility of using agentic AI to simulate psychiatric consultations and deliver STACER-aligned formative feedback. By combining adaptive multiturn psychiatric simulation with competency-based evaluation, it shows promise in supporting cognitive aspects of consultation, though it remains limited in facilitating relational skills such as rapport building. These findings suggest agentic AI could expand scalable, low-risk opportunities for deliberate practice and formative feedback in competency-based psychiatric education. Further controlled studies are needed to evaluate educational effectiveness and integration into residency training.
Overview
CoChat Summary
TL;DR
This two-phase prospective single-arm technical feasibility study evaluated the STACER Agentic System, an LLM-based platform with a patient agent and rater agent for psychiatry consultation training. It found high patient-agent fidelity and structured STACER-aligned feedback, but limited rapport/nonverbal support and poor item-level interrater reliability with psychiatrists.
Key findings
- The patient agent showed high behavioral fidelity (51/56, 91.07%) and symptom fidelity (105/110, 95.45%), with automated medical faithfulness mean 0.99 (95% CI 0.99‐1.00) and turn relevance 0.99 (95% CI 0.986‐0.992).
- Participants rated simulations as psychiatrically plausible and diagnostically useful, especially for depressive symptoms, but rapport building was moderate (mean 2.78, SD 1.56 to mean 3.00, SD 1.41, out of 5.00) due to limited nonverbal cues.
- The rater agent generated structured STACER-aligned feedback with high intrarater consistency, especially at section subtotals. Interrater reliability with psychiatrists was poor at item level (ICC range 0.25‐0.49) but good-to-excellent at section level for psychiatry resident sessions (ICC range 0.89‐0.93).
- The rater agent’s scores fell between the 2 psychiatrists for the clinical research team and were lower than both human raters for psychiatry residents.
Methods
- Two-phase prospective single-arm cohort technical feasibility study of the STACER Agentic System, an LLM-powered platform integrating a patient agent and a rater agent.
- Phase 1: automated evaluation of the patient agent using a psychiatrist agent across 227 synthetic major depressive disorder cases, assessed with DeepEval metrics (correctness, clarity, medical faithfulness, turn relevance, role adherence) using descriptive statistics and 95% CIs.
- Phase 2: 14 convenience-sampled participants (5 clinical research team members and 9 University of Alberta psychiatry residents) completed simulated diagnostic interviews and case presentations; performance was evaluated using STACER-based scoring by the rater agent and 2 psychiatrists, with interrater reliability assessed using intraclass correlation coefficients and participant ratings of realism, behavioral consistency, psychiatric nuance, and feedback utility via Likert scales and free-text answers.
Limitations
- Stated: Phase 1 relied on automated LLM-as-a-judge testing, limiting ecological validity; the sample was small, interface use varied (text vs voice), and the design precluded conclusions about educational efficacy.
- Stated: The system’s limited capacity for rapport formation and nonverbal communication affects STACER domains including affect, engagement, and relational skills; evaluation focused mainly on major depressive disorder; and the rater agent requires calibration and psychometric validation.
- Evident in the text: No formal research ethics board review was sought because the work was framed as preliminary operational and technical feasibility testing; no formal sample size calculation was performed; missing data adversely affected reliability estimates, with resident 7 excluded and some items replaced with 0.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

![Figure 2. The structured assessment of a clinical encounter report agentic system uses the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders [Fifth Edition]), depression diagnostic criteria, and treatments to provide a simulated environment for psychiatric residents to practice their consultation skills and clinical competency. The patient agent mimics a patient from one of the case studies stored in the patient database, which contains 100 real cases and 227 synthetic cases. The rater agent evaluates the consultation session, provides feedback to the resident, and generates a structured assessment of a clinical encounter report. For technical validation, an external psychiatrist agent was developed to conduct automated multiturn interviews with the patient agent. Grounded in the Core of Discipline-structured assessment of a clinical encounter report expectations, it standardized the elicitation of clinically relevant information across synthetic profiles before human user-testing. STACER: structured assessment of a clinical encounter report.](https://figures.cochat.ai/paper-figures/a4e470c9b7854c8ca1c156079cfa79fa/enhancing-psychiatry-training-using-an-agentic-ai-simulated-consultation-tool-figure-2.jpg)



Verification
Claims the paper makes
4 Valid · 2 Not valid · 38 Not checked · 92 Own results and interpretations
None of the sources cited for this claim have been checked yet.
In the paper
“In Canada, psychiatry residents’ consultation competence is formally assessed using the structured assessment of a clinical encounter report (STACER) [29], which evaluates performance on the MSE, the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders [Fifth Edition])-aligned diagnostic formulation [30], and treatment planning.”
§ Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study > Introduction
What the sources say
- Not checked yetcites [29] Psychiatry clinical evaluation - core of psychiatry. URL: https://psychiatry.utoronto.ca/sites/default/files/assets/files/core-stacer-assessment-formupdateddecember2022-v2_0.pdf [Accessed 04-04-2025]
- Could not verifycites [30] Diagnostic and Statistical Manual of Mental Disorders: DSM-5. American Psychiatric Association; 2013. [doi: 10.1176/appi.books.9780890425596] sourceExperimental — accuracy not yet measured
Our copy under DOI 10.1176/appi.books.9780890425596 holds WHO 'Depression' fact-sheet content (sections dated 11 and 15 September 2026, plus a References list citing GBD 2023, Woody et al. 2017 and Evans-Lacko et al. 2018), not the DSM-5 manual text. Searched for DSM-5 diagnostic criteria, 'structured assessment of a clinical encounter report'/STACER, mental status examination and diagnostic formulation in residency assessment: no passage addresses Canadian psychiatry residents, STACER, MSE or treatment planning. The copy therefore neither supports nor contradicts the attribution, so the claim cannot be settled on our copy. (LLM, conf=0.70)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“LLM-generated simulations demonstrate clinically relevant psychiatric exchanges [23-25] and generate automated feedback aligned with expert judgment [26], offering a cost-efficient and scalable alternative to real patients for clinical training.”
§ Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study > Introduction
What the sources say
- Partly supportedcites [23] Cook, Virtual patients using large language models: scalable, contextualized simulation of clinician-patient dialogue with feedback, J Med Internet Res sourceExperimental — accuracy not yet measured
Our copy supports the general part of the claim: the Discussion (Principal Findings) reports that, per blinded human raters, the LLM-powered virtual patients approached a 'very good approximation of a real conversation' with 'easily overlooked flaws,' and the Abstract (Conclusions) states LLM-powered VPs can simulate patient-clinician dialogues and represent patient preferences. However, the psychiatric specificity is not supported: Methods (Conversation Planning) states the topics were '2 common problems in ambulatory medicine: chronic cough (a diagnostic task) and diabetes (a management task),' with permutations varying preferences, insurance/income and an anxious persona — no psychiatric patients or encounters were simulated. The Implications section only mentions psychiatric-adjacent tasks (cognitive behavioral therapy, motivational interviewing) as future potential applications, not as demonstrated results. (LLM, conf=0.80)
- Partly supportedcites [24] Cross, Assessing ChatGPT’s capability as a new age standardized patient: qualitative study, JMIR Med Educ sourceExperimental — accuracy not yet measured
Cross et al. (10.2196/63353) support the 'clinically relevant LLM-generated simulation' half of the claim: the Principal Findings (Discussion) state that ChatGPT 'still provides a valuable platform for honing diagnostic interviewing skills' and that results are 'broadly in line with those of other studies using LLMs in the role of SP'. However, the simulation was not psychiatric — the Methods (Study Design) say the prompt directed the LLM to present as a patient with a neurological condition, and the Rationale for Clinical Case Selection specifies Bell palsy; no psychiatric encounter appears anywhere in our copy. The source also qualifies the point rather than endorsing unreserved clinical relevance (an 'alignment tax' where the model refused a standard sexual-history question, lack of nonverbal cues, and the conclusion that ChatGPT 'cannot fully substitute for SPs'). The psychiatric domain attributed to this citation is unsupported by it. (LLM, conf=0.78)
- Partly supportedcites [25] Yamamoto, Enhancing medical interview skills through AI-simulated patient interactions: nonrandomized controlled trial, JMIR Med Educ sourceExperimental — accuracy not yet measured
Our copy supports the general point that LLM (GPT-4) simulated patients produced realistic, clinically usable medical-interview exchanges: Results report median reproducibility/realism scores of 7-9 and emotional-interaction scores of 7-8, and students 'learned essential questions relevant to clinical settings' (Results, positive aspects), with the Conclusions stating the platform demonstrated superior educational effectiveness while maintaining safety. However, the 'psychiatric' qualifier is not supported: Methods (Educational Platforms) specify cases built on 8 primary somatic symptoms — chest pain, abdominal pain, cough, heartburn, fatigue, fever, dizziness, shortness of breath — in fourth-year medical students at Okayama University assessed on the pre-CC OSCE; searches for psychiatric/psychological cases returned nothing. The paper also documents unnatural responses, repetitions and feedback fabrications (Discussion, Fabrications by LLMs; Results, negative aspects), and limits the platform to a supplementary role rather than a replacement for real patients. (LLM, conf=0.78)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
None of the sources cited for this claim have been checked yet.
In the paper
“Canadian psychiatry residents must demonstrate consultation competency, assessed using the standardized assessment of a clinical encounter report (STACER).”
- Also stated in: § Introduction · plain
“In Canada, psychiatry residents’ consultation competence is formally assessed using the structured assessment of a clinical encounter report (STACER) [29], which evaluates performance on the MSE, the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders [Fifth Edition])-aligned diagnostic formulation [30], and treatment planning.”
§ Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study > Abstract > Background
What the sources say
- Not checked yetcites [29] Psychiatry clinical evaluation - core of psychiatry. URL: https://psychiatry.utoronto.ca/sites/default/files/assets/files/core-stacer-assessment-formupdateddecember2022-v2_0.pdf [Accessed 04-04-2025]
- Could not verifycites [30] Diagnostic and Statistical Manual of Mental Disorders: DSM-5. American Psychiatric Association; 2013. [doi: 10.1176/appi.books.9780890425596] sourceExperimental — accuracy not yet measured
Our copy under this DOI contains only WHO depression-topic content (mhGAP, psychological treatments, self-care, contributing factors/prevention, references and site navigation), with no passage about psychiatric residency training, consultation competency, or the STACER instrument. Searched 'STACER standardized assessment of clinical encounter report psychiatry residents consultation liaison' and 'psychiatry residency training requirements consultation liaison competency Canada'; the top hits were all depression/treatment passages, none addressing the claim. Nothing in our copy contradicts the claim, and no alternative work is named in our copy for this point, so the attribution cannot be checked here. (LLM, conf=0.70)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Existing datasets are not suited for multiturn psychiatric consultations [41-43], so we developed a comprehensive synthetic benchmark of 227 DSM-5-Text Revision–based [44] major depressive disorder cases to systematically evaluate patient agent performance (Table 1).”
§ Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study > Methods > Synthetic Benchmark Dataset
What the sources say
- Partly supportedcites [41] Bedi, Holistic evaluation of large language models for medical tasks with MedHELM, Nat Med sourceExperimental — accuracy not yet measured
The Methods section (Complete taxonomy structure; page not specified) describes the MentalHealth benchmark as evaluating counselor responses to mental-health conversations, rather than an entire psychiatric consultation. It also describes MedDialog’s multi-turn conversations as a summarization task. These examples support a narrower point about the tasks those datasets evaluate, but the paper does not establish that existing datasets generally are unsuitable for multiturn psychiatric consultations. (LLM, conf=0.75)
- Not checked yetcites [42] Harrigian K Aguirre C Dredze M . On the state of social media data for mental health research. arXiv. Preprint posted online on Apr 25, 2021. URL: http://arxiv.org/abs/2011.05233 [Accessed 06-07-2026]
- Not checked yetcites [43] Schmidgall S Ziaei R Harris C Reis E Jopling J Moor M . AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments. arXiv. Preprint posted online on May 25, 2025. URL: http://arxiv.org/abs/2405.07960 [Accessed 06-07-2026]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants rated simulations as psychiatrically plausible and diagnostically useful, particularly for depressive symptom representation, although rapport building was moderate (mean 2.78, SD 1.56 to mean 3.00, SD 1.41, out of 5.00) due to limited nonverbal cues.”
§ Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study > Abstract > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
49 more claims
Create a free account to see all 136 claims and to verify the 46 references still open.
- Probably not a substantial claim. 51
- It may state two separate claims as one. 50
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study > Methods > Real-Patient Case Study Dataset
“Overall, 5 noncomorbid major depressive disorder cases (5/100, 5%) were selected for feasibility evaluation.”
§ Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study > Results > Linking Back the Patient Agent’s Behavior and Symptom Expressions
“Resident 3 selected a Schizophrenia patient profile where behaviors could not be mimicked in the consultation. Resident 9 selected a more complicated comorbid case study.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“Transformer-based LLMs [19], trained on terabytes of data, demonstrate improved realism and adaptability in simulating a wide range of patient scenarios relevant to consultation practices [20] and education [21,22].”
Diagnostic and Statistical Manual of Mental Disorders
American Psychiatric Association (2013) · doi:10.1176/appi.books.9780890425596 · Cited by 116,369
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
Diagnostic and Statistical Manual of Mental Disorders
American Psychiatric Association (2022) · doi:10.1176/appi.books.9780890425787 · Cited by 23,476
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
Eppich, W., & Cheng, A. (2015) · Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0000000000000072 · Cited by 1,106
Bloom’s taxonomy of cognitive learning objectives
Adams, N. E. (2015) · Journal of the Medical Library Association : JMLA · doi:10.3163/1536-5050.103.3.010 · Cited by 1,030
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge
Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z. Y. A., Bhattacharjee, A., Jiang, Y., Chen, C., Wu, T., et al. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.138 · Cited by 947
Kononowicz, A. A., Woodham, L. A., Edelbring, S., Stathakarou, N., Davies, D., Saxena, N., Tudor Car, L., Carlstedt-Duke, J., Car, J., & Zary, N. (2019) · Journal of Medical Internet Research · doi:10.2196/14676 · Cited by 600
Debriefing for technology-enhanced simulation: a systematic review and meta-analysis
Cheng, A., Eppich, W., Grant, V., Sherbino, J., Zendejas, B., & Cook, D. A. (2014) · Medical Education · doi:10.1111/medu.12432 · Cited by 481 · cited 2×
“This sequence mirrors established mechanisms of learning in simulation-based education, where structured debriefing is consistently identified as the primary driver of skill acquisition [53,54].”
Embodied conversational agents in clinical psychology: a scoping review
Provoost, S., Lau, H. M., Ruwaard, J., & Riper, H. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6553 · Cited by 433
Implementation of competency‐based medical education: are we addressing the concerns and challenges?
Hawkins, R. E., Welcher, C. M., Holmboe, E. S., Kirk, L. M., Norcini, J. J., Simons, K. B., & Skochelak, S. E. (2015) · Medical Education · doi:10.1111/medu.12831 · Cited by 409 · cited 2×
“Figure 1 illustrates how these agents collectively enact the 4 stages of the Kolb cycle to support competency-based medical education (CBME) [35,36].”
Objective structured clinical examination: the assessment of choice
Zayyan, M. (2011) · Oman Medical journal · doi:10.5001/omj.2011.55 · Cited by 295 · cited 1×
“This positions the technology as a complement, not a replacement, to supervised clinical encounters and traditional Objective Structured Clinical Examinations–style assessments [55].”
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge
Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z., Bhattacharjee, A., Jiang, Y., Chen, C., Wu, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.16594 · Cited by 271
Learning from clinical work: the roles of learning cues and credibility judgements
Watling, C., Driessen, E., van der Vleuten, C. P. M., & Lingard, L. (2012) · Medical Education · doi:10.1111/j.1365-2923.2011.04126.x · Cited by 261 · cited 1×
“Yet, training environments often limit opportunities for sustained supervision and deliberate rehearsal [5,6], leaving residents to encounter challenging cases without sufficient low-stakes practice [7].”
The promise, perils, problems and progress of competency‐based medical education
Touchie, C., & ten Cate, O. (2016) · Medical Education · doi:10.1111/medu.12839 · Cited by 223
Holderried, F., Stegemann-Philipps, C., Herrmann-Werner, A., Festl-Wietek, T., Holderried, M., Eickhoff, C., & Mahling, M. (2024) · JMIR Medical Education · doi:10.2196/59213 · Cited by 152 · cited 3×
“LLM-generated simulations demonstrate clinically relevant psychiatric exchanges [23-25] and generate automated feedback aligned with expert judgment [26], offering a cost-efficient and scalable alternative to real patients for clinical training.”
Cook, D. A., Overgaard, J., Pankratz, V. S., Del Fiol, G., & Aakre, C. A. (2025) · Journal of Medical Internet Research · doi:10.2196/68486 · Cited by 96 · cited 2×
“These findings, echoed by Cook et al [23], who observed that LLM-powered virtual patients generated personalized, high-quality performance feedback rated comparably to human evaluators, substantiate the potential of AI-assisted formative assessment in clinical education.”
Yamamoto, A., Koda, M., Ogawa, H., Miyoshi, T., Maeda, Y., Otsuka, F., & Ino, H. (2024) · JMIR Medical Education · doi:10.2196/58753 · Cited by 92 · cited 1×
“LLM-generated simulations demonstrate clinically relevant psychiatric exchanges [23-25] and generate automated feedback aligned with expert judgment [26], offering a cost-efficient and scalable alternative to real patients for clinical training.”
LLM-powered Multi-agent Framework for Goal-oriented Learning in Intelligent Tutoring System
Wang, T., Zhan, Y., Lian, J., Hu, Z., Yuan, N. J., Zhang, Q., Xie, X., & Xiong, H. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715244 · Cited by 82
Simulation Training in Psychiatry for Medical Education: A Review
Piot, M., Attoe, C., Billon, G., Cross, S., Rethans, J., & Falissard, B. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.658967 · Cited by 77
Confabulation: The Surprising Value of Large Language Model HallucinationsPreprint
Sui, P.-Q., Duede, E., Wu, S., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2406.04175 · Cited by 75
Simulation in Undergraduate Psychiatry: Exploring the Depth of Learner Engagement
Abdool, P. S., Nirula, L., Bonato, S., Rajji, T. K., & Silver, I. L. (2017) · Academic Psychiatry · doi:10.1007/s40596-016-0633-9 · Cited by 66
Holistic evaluation of large language models for medical tasks with MedHELM
Bedi, S., Cui, H., Fuentes, M., Unell, A., Wornow, M., Banda, J. M., Kotecha, N., Keyes, T., Mai, Y., Oez, M., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04151-2 · Cited by 59 · cited 1×
“Existing datasets are not suited for multiturn psychiatric consultations [41-43], so we developed a comprehensive synthetic benchmark of 227 DSM-5-Text Revision –based [44] major depressive disorder cases to systematically evaluate patient agent performance (Table 1).”
Tay, J. L., Xie, H., & Sim, K. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13010112 · Cited by 57 · cited 1×
“Complementing this, Tay et al [47] found that immersive and augmented reality environments enhance empathy and reduce stigma by enabling learners to experience affective and perceptual dimensions of mental illness.”
Applications of Artificial Intelligence in Psychiatry and Psychology Education: Scoping Review
Prégent, J., Chung, V.-H.-A., Adib, I. E., Désilets, M., & Hudon, A. (2025) · JMIR Medical Education · doi:10.2196/75238 · Cited by 44
Self-citationWeb-based simulation in psychiatry residency training: a pilot study
Gorrindo, T., Baer, L., Sanders, K. M., Birnbaum, R. J., Fromson, J. A., Sutton-Skinner, K. M., Romeo, S. A., & Beresin, E. V. (2011) · Academic Psychiatry · doi:10.1176/appi.ap.35.4.232 · Cited by 40 · cited 1×
“…that computer-mediated psychiatric dialogues were feasible and effectively supported specific tasks (eg, obtaining informed consent for antipsychotics) to improve trainee confidence and knowledge in controlled studies [14], as well as enhance diagnostic reasoning [15] and skill acquisition [16].”
The effect of rater accountability on performance ratings: a meta-analytic review
Harari, M. B., & Rudolph, C. W. (2017) · Human Resource Management Review · doi:10.1016/j.hrmr.2016.09.007 · Cited by 38 · cited 2×
“This shift, from disagreement in overall scoring for the clinical research team to consistently higher ratings by both psychiatrists for residents, may reflect a favorable bias toward psychiatry residents, potentially influenced by expectations of trainee competence [50].”
Formative assessment and feedback in medical education: A practical guide: AMEE Guide No. 189
Lipnevich, A. A., Mattern, K., & Feddock, C. (2026) · Medical Teacher · doi:10.1080/0142159x.2025.2569623 · Cited by 37 · cited 2×
“As a result, opportunities for iterative practice and formative feedback remain limited, despite their importance for competency development in medical education [10].”
Dupuy, L., Micoulaud‐Franchi, J., Cassoudesalle, H., Ballot, O., Dehail, P., Aouizerate, B., Cuny, E., de Sevin, É., & Philip, P. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2019.11.117 · Cited by 37
Assessing ChatGPT’s capability as a new age standardized patient: qualitative study
Cross, J., Kayalackakom, T., Robinson, R. E., Vaughans, A., Sebastian, R., Hood, R., Lewis, C., Devaraju, S., Honnavar, P., Naik, S., et al. (2025) · JMIR Medical Education · doi:10.2196/63353 · Cited by 33
Tripathi, S., Alkhulaifat, D., Doo, F. X., Rajpurkar, P., McBeth, R., Daye, D., & Cook, T. S. (2025) · NEJM AI · doi:10.1056/aip2401106 · Cited by 31 · cited 1×
“To ensure transparent and comprehensive reporting of this LLM-based system, we adhered to the Development, Evaluation, and Assessment of Large Language Models guidelines [40]; the completed Development, Evaluation, and Assessment of Large Language Models checklist is provided in Checklist 1.”
Dawood, E., Alshutwi, S. S., Alshareif, S., & Shereda, H. M. A. (2024) · Nursing Reports · doi:10.3390/nursrep14020107 · Cited by 27
Jaleel, A., Aziz, U., Farid, G., Zahid Bashir, M., Mirza, T. R., Khizar Abbas, S. M., Aslam, S., & Sikander, R. M. H. (2025) · JMIR Medical Education · doi:10.2196/68070 · Cited by 25
Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the AnswersPreprint
Rueda, A., Hassan, M. M., Perivolaris, A., Teferra, B. G., Samavi, R., Rambhatla, S., Wu, Y., Zhang, Y., Cao, B., Sharma, D., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.01482 · Cited by 25
Self-citationBeyond Consensus: Mitigating the Agreeableness Bias in LLM Judge EvaluationsPreprint
Jain, S., Ahmed, U. Z., Sahai, S., & Leong, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.11822 · Cited by 25
Communication skills in psychiatry for undergraduate students: A scoping review
Novais, F., Ganança, L., Barbosa, M., & Telles‐Correia, D. (2022) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2022.972703 · Cited by 24
AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments
Schmidgall, S., Ziaei, R., Harris, C., Reis, E., Jopling, J., & Moor, M. (2024) · arXiv · doi:10.48550/arxiv.2405.07960 · Cited by 24
The role of generative artificial intelligence in psychiatric education– a scoping review
Lee, Q. Y., Chen, M., Ong, C. W., & Ho, C. S. H. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07026-9 · Cited by 23
Nagendrappa, S., de Filippis, R., Ramalho, R., Ransing, R. S., Orsolini, L., Ullah, I., Karaliūnienė, R., Shoib, S., Abbass, Z., Hayatudeen, N., et al. (2021) · Academic Psychiatry · doi:10.1007/s40596-021-01482-3 · Cited by 20
Large Language Models are Inconsistent and Biased Evaluators
Stureborg, R., Alikaniotis, D., & Suhara, Y. (2024) · arXiv · doi:10.48550/arxiv.2405.01724 · Cited by 15
The Patients' Practises Disclosing Subjective Experiences in the Psychiatric Intake Interview
Savander, E. È., Hintikka, J. J., Wuolio, M. P., & Peräkylä, A. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.605760 · Cited by 14
A systematic review of hyperscanning in clinical encounters
Adel, L., Moses, L., Irvine, E., Greenway, K. T., Dumas, G., & Lifshitz, M. (2025) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2025.106248 · Cited by 13 · cited 1×
“Similarly, any multimodal extension would also need to capture and interpret the resident’s own nonverbal communication, as authentic rapport relies on bidirectional cues [49].”
Self-citationCommunication skills training in psychiatry
Michaelson, S., & Rahim, S. (2022) · BJPsych Advances · doi:10.1192/bja.2021.80 · Cited by 10
Pang, H. Y. M., Meshkat, S., Teferra, B. G., Rueda, A., Samavi, R., Krishnan, S., Doyle, T. E., Rambhatla, S., DeJong, S. M., Sockalingam, S., et al. (2024) · Academic Psychiatry · doi:10.1007/s40596-024-02087-2 · Cited by 9
Self-citationTraining framework for high‐stakes OSCE: Experience from volunteer standardized patients' bank
Feigerlová, E. (2024) · The Clinical Teacher · doi:10.1111/tct.13787 · Cited by 8
Wang, J., & Duan, Z. (2024) · arXiv · doi:10.48550/arxiv.2412.03801 · Cited by 8
Costa-Dookhan, K. A., Adirim, Z. L., Maslej, M. M., Donner, K., Rodak, T., Soklaridis, S., Sockalingam, S., & Thakur, A. (2025) · Academic Medicine · doi:10.1097/acm.0000000000005983 · Cited by 6
Self-citationSong, Y. O., Kim, H., Nam, Y., Choe, K., & Ha, J. (2022) · Journal of Korean Academy of Psychiatric and Mental Health Nursing · doi:10.12934/jkpmhn.2022.31.1.80 · Cited by 6
Online-based and technology-assisted psychiatric education for trainees: scoping review
Mohd Kassim, M. A., Azli Shah, S. M. Y., Lim, J. T. Y., & Mohd Daud, T. I. (2025) · JMIR Medical Education · doi:10.2196/64773 · Cited by 5
Evolution of AI in Education: Agentic Workflows
Kamalov, F., Calonge, D. S., Smail, L., Azizov, D., Thadani, D. R., Kwong, T., & Atif, A. (2025) · arXiv · doi:10.48550/arxiv.2504.20082 · Cited by 4
Arbuckle, M. R., Stern, D. A., Barkil-Oteo, A., & Asghar‐Ali, A. A. (2020) · Academic Psychiatry · doi:10.1007/s40596-020-01218-9 · Cited by 4
Lee, S., Kang, J., Kim, H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.17546 · Cited by 1
Research works printed but not linked: 3
- Kenny, Annu Rev Cyberther Telemed
- Hicke Y Geathers J Rajashekar N et al. MedSimAI: simulation and formative feedback generation to enhance deliberate practice in medical education. arXiv. Preprint posted online on Mar 1, 2025. URL: http://arxiv.org/abs/2503.05793 [Accessed 06-07-2026] · arXiv
- Harrigian K Aguirre C Dredze M . On the state of social media data for mental health research. arXiv. Preprint posted online on Apr 25, 2021. URL: http://arxiv.org/abs/2011.05233 [Accessed 06-07-2026] · arXiv
Cited by
Iqbal, M. Z., & Talukder, M. (2026) · Intelligent Hospital · doi:10.1016/j.inhs.2026.100111 · Cited by 1
Metadata
- Authors
- Rueda, A., Al-Shamali, H. F., Cote, Z., Roy, N., Janssen-Aguilar, R., Joseph, J., Teferra, B. G., Kamaleddin, M. A., Burback, L., Winkler, O., Kapralos, B., Torres, A., Sharma, D., Krishnan, S., Dumas, G., Greenshaw, A., Hudon, A., Sockalingam, S., Zhang, Y., Dubrowski, A., & Bhat, V.
- Affiliations
- St. Michael's Hospital (CA)
- University of Alberta (CA)
- Ontario Tech University (CA)
- York University (CA)
- Toronto Metropolitan University (CA)
- Centre Hospitalier Universitaire Sainte-Justine (CA)
- Institut national de psychiatrie légale Philippe-Pinel (CA)
- Centre for Addiction and Mental Health (CA)
- Corresponding author
- Bhat, Venkat
- ORCID
- Venue
- JMIR Medical Education
- Publisher
- JMIR Publications Inc.
- Published
- 2026-07-21
- Volume
- 12
- Pages
- e88580-e88580
- ISSN
- 2369-3762 (ISSN-L)
- PMC
- PMC13387278
- OpenAlex
- W7169849571
- Cited by
- 1
- References
- 43
- Citations by year
- 2026: 1
- FWCI
- 1.97
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Medical Education (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Education › Simulation-Based Education in Healthcare
- Social Sciences › Education › Innovations in Medical Education
- Keywords
- agentic AI, simulated patient encounters, psychiatry residency training, large language models, automated feedback, major depressive disorder, technical feasibility, inter-rater reliability, rapport building, nonverbal cues, clinical skills assessment
- MeSH terms
- Psychiatrists, Adult, Artificial Intelligence, Canada, Clinical Competence, Feasibility Studies, Female, Humans, Internship and Residency, Male, Prospective Studies, Psychiatry, Referral and Consultation, Cohort Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.