CoChat

Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study

Open access

The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index; 8 of the 54 sources its claims cite were checked for support.

Abstract

Background: Canadian psychiatry residents must demonstrate consultation competency, assessed using the standardized assessment of a clinical encounter report (STACER). However, opportunities to practice these skills and receive constructive assessment remain limited in clinical settings. Objective: This study aimed to evaluate the technical feasibility of an agentic AI system designed to support psychiatry residents’ consultation competence through simulated patient encounters with a patient agent and structured feedback from a rater agent. Methods: We conducted a two-phase technical feasibility prospective single-arm cohort study of the STACER Agentic System, a large language model–based platform integrating a patient agent and a rater agent. Phase 1 involved automated evaluation of the patient agent using a psychiatrist agent across 227 synthetic major depressive disorder cases. Performance was assessed using DeepEval metrics (correctness, clarity, medical faithfulness, turn relevance, and role adherence) with descriptive statistics and 95% CIs. Phase 2 involved a preliminary user study with 14 convenience-sampled participants: a total of 5 members of the clinical research team and 9 psychiatry residents from the University of Alberta. Participants completed simulated diagnostic interviews and case presentations. Performance was evaluated using STACER-based scoring by the rater agent and 2 psychiatrists. Interrater reliability was assessed using intraclass correlation coefficients (α=.05). Participants rated realism, behavioral consistency, psychiatric nuance, and feedback utility using Likert scales and free-text answers. Results: The patient agent demonstrated high behavioral (51/56, 91.07%) and symptom fidelity (105/110, 95.45%), with strong automated performance (medical faithfulness mean 0.99, 95% CI 0.99‐1.00; turn relevance 0.99, 95% CI 0.986‐0.992). Participants rated simulations as psychiatrically plausible and diagnostically useful, particularly for depressive symptom representation, although rapport building was moderate (mean 2.78, SD 1.56 to mean 3.00, SD 1.41, out of 5.00) due to limited nonverbal cues. The rater agent generated structured STACER-aligned feedback with high intrarater consistency, especially at the section subtotal level. Interrater reliability with psychiatrists was poor at the item level (intraclass correlation coefficient range=0.25‐0.49) but improved to good-to-excellent agreement at the section level for psychiatry resident sessions (intraclass correlation coefficient range=0.89‐0.93). The rater agent’s scores fell between those of the 2 psychiatrists for the clinical research team and were lower than both human raters for psychiatry residents. Conclusions: The STACER Agentic System demonstrates the technical feasibility of using agentic AI to simulate psychiatric consultations and deliver STACER-aligned formative feedback. By combining adaptive multiturn psychiatric simulation with competency-based evaluation, it shows promise in supporting cognitive aspects of consultation, though it remains limited in facilitating relational skills such as rapport building. These findings suggest agentic AI could expand scalable, low-risk opportunities for deliberate practice and formative feedback in competency-based psychiatric education. Further controlled studies are needed to evaluate educational effectiveness and integration into residency training.

Overview

CoChat Summary

TL;DR

This two-phase prospective single-arm technical feasibility study evaluated the STACER Agentic System, an LLM-based platform with a patient agent and rater agent for psychiatry consultation training. It found high patient-agent fidelity and structured STACER-aligned feedback, but limited rapport/nonverbal support and poor item-level interrater reliability with psychiatrists.

Key findings

  • The patient agent showed high behavioral fidelity (51/56, 91.07%) and symptom fidelity (105/110, 95.45%), with automated medical faithfulness mean 0.99 (95% CI 0.99‐1.00) and turn relevance 0.99 (95% CI 0.986‐0.992).
  • Participants rated simulations as psychiatrically plausible and diagnostically useful, especially for depressive symptoms, but rapport building was moderate (mean 2.78, SD 1.56 to mean 3.00, SD 1.41, out of 5.00) due to limited nonverbal cues.
  • The rater agent generated structured STACER-aligned feedback with high intrarater consistency, especially at section subtotals. Interrater reliability with psychiatrists was poor at item level (ICC range 0.25‐0.49) but good-to-excellent at section level for psychiatry resident sessions (ICC range 0.89‐0.93).
  • The rater agent’s scores fell between the 2 psychiatrists for the clinical research team and were lower than both human raters for psychiatry residents.

Methods

  • Two-phase prospective single-arm cohort technical feasibility study of the STACER Agentic System, an LLM-powered platform integrating a patient agent and a rater agent.
  • Phase 1: automated evaluation of the patient agent using a psychiatrist agent across 227 synthetic major depressive disorder cases, assessed with DeepEval metrics (correctness, clarity, medical faithfulness, turn relevance, role adherence) using descriptive statistics and 95% CIs.
  • Phase 2: 14 convenience-sampled participants (5 clinical research team members and 9 University of Alberta psychiatry residents) completed simulated diagnostic interviews and case presentations; performance was evaluated using STACER-based scoring by the rater agent and 2 psychiatrists, with interrater reliability assessed using intraclass correlation coefficients and participant ratings of realism, behavioral consistency, psychiatric nuance, and feedback utility via Likert scales and free-text answers.

Limitations

  • Stated: Phase 1 relied on automated LLM-as-a-judge testing, limiting ecological validity; the sample was small, interface use varied (text vs voice), and the design precluded conclusions about educational efficacy.
  • Stated: The system’s limited capacity for rapport formation and nonverbal communication affects STACER domains including affect, engagement, and relational skills; evaluation focused mainly on major depressive disorder; and the rater agent requires calibration and psychometric validation.
  • Evident in the text: No formal research ethics board review was sought because the work was framed as preliminary operational and technical feasibility testing; no formal sample size calculation was performed; missing data adversely affected reliability estimates, with resident 7 excluded and some items replaced with 0.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/88580

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Figures

Figure 1. Illustration of how the patient agent and rater agent map onto the four stages of the Kolb experiential learning cycle within a competency-based medical education psychiatric training context. The patient agent supports concrete experience by enabling simulated clinical encounters, while the rater agent contributes to reflective observation through structured assessment. Learners progress to abstract conceptualization by integrating feedback into clinical understanding and to active experimentation by applying these insights in subsequent simulations. CBME: competency-based medical education.
Figure 1. Illustration of how the patient agent and rater agent map onto the four stages of the Kolb experiential learning cycle within a competency-based medical education psychiatric training context. The patient agent supports concrete experience by enabling simulated clinical encounters, while the rater agent contributes to reflective observation through structured assessment. Learners progress to abstract conceptualization by integrating feedback into clinical understanding and to active experimentation by applying these insights in subsequent simulations. CBME: competency-based medical education.CC BY · PubMed Central
Figure 2. The structured assessment of a clinical encounter report agentic system uses the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders [Fifth Edition]), depression diagnostic criteria, and treatments to provide a simulated environment for psychiatric residents to practice their consultation skills and clinical competency. The patient agent mimics a patient from one of the case studies stored in the patient database, which contains 100 real cases and 227 synthetic cases. The rater agent evaluates the consultation session, provides feedback to the resident, and generates a structured assessment of a clinical encounter report. For technical validation, an external psychiatrist agent was developed to conduct automated multiturn interviews with the patient agent. Grounded in the Core of Discipline-structured assessment of a clinical encounter report expectations, it standardized the elicitation of clinically relevant information across synthetic profiles before human user-testing. STACER: structured assessment of a clinical encounter report.
Figure 2. The structured assessment of a clinical encounter report agentic system uses the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders [Fifth Edition]), depression diagnostic criteria, and treatments to provide a simulated environment for psychiatric residents to practice their consultation skills and clinical competency. The patient agent mimics a patient from one of the case studies stored in the patient database, which contains 100 real cases and 227 synthetic cases. The rater agent evaluates the consultation session, provides feedback to the resident, and generates a structured assessment of a clinical encounter report. For technical validation, an external psychiatrist agent was developed to conduct automated multiturn interviews with the patient agent. Grounded in the Core of Discipline-structured assessment of a clinical encounter report expectations, it standardized the elicitation of clinically relevant information across synthetic profiles before human user-testing. STACER: structured assessment of a clinical encounter report.CC BY · PubMed Central
Figure 3. Structured assessment of clinical encounter report agentic system workflow. The left panel shows the four phases: (1) patient selection, (2) simulated interview, (3) case presentation, and (4) assessment and feedback. The right panel illustrates interactions between the psychiatry resident and the structured assessment of clinical encounter report agentic system. The rater agent loads the case and provides an introduction (phase 1). The resident conducts a multi-turn psychiatric interview with the patient agent (phase 2), followed by a case presentation to the rater agent (phase 3). The rater agent then integrates all inputs to generate structured, structured assessment of clinical encounter report agentic system–aligned feedback and a Core of Discipline-Structured assessment of clinical encounter report agentic system assessment, with a chat interface supporting reflection and iterative learning (phase 4). STACER: structured assessment of clinical encounter report.
Figure 3. Structured assessment of clinical encounter report agentic system workflow. The left panel shows the four phases: (1) patient selection, (2) simulated interview, (3) case presentation, and (4) assessment and feedback. The right panel illustrates interactions between the psychiatry resident and the structured assessment of clinical encounter report agentic system. The rater agent loads the case and provides an introduction (phase 1). The resident conducts a multi-turn psychiatric interview with the patient agent (phase 2), followed by a case presentation to the rater agent (phase 3). The rater agent then integrates all inputs to generate structured, structured assessment of clinical encounter report agentic system–aligned feedback and a Core of Discipline-Structured assessment of clinical encounter report agentic system assessment, with a chat interface supporting reflection and iterative learning (phase 4). STACER: structured assessment of clinical encounter report.CC BY · PubMed Central
Figure 4. The eight DeepEval performance metrics are colored by the types of metrics, with mean values provided. The density plots indicate that the mean score of the patient agent playing the role of a patient with major depressive disorder using the synthetic dataset: medical faithfulness, turn relevance, and role adherence above 0.98; correctness and clarity above 0.87; and bias and toxicity close to 0. The distribution of scores for the eight metrics with red indicate the GEval measures, green indicates the harmfulness measures, and blue indicates the multiturn measures.
Figure 4. The eight DeepEval performance metrics are colored by the types of metrics, with mean values provided. The density plots indicate that the mean score of the patient agent playing the role of a patient with major depressive disorder using the synthetic dataset: medical faithfulness, turn relevance, and role adherence above 0.98; correctness and clarity above 0.87; and bias and toxicity close to 0. The distribution of scores for the eight metrics with red indicate the GEval measures, green indicates the harmfulness measures, and blue indicates the multiturn measures.CC BY · PubMed Central
Figure 5. Violin plot of participants’ feedback. Each participant’s response to the survey questions in Table 3 is indicated by a point. (A) Feedback from the clinical research team with instructions provided on the flow of the user interface. (B) Psychiatry residents’ feedback when no instructions were provided. Four of the 9 participants were unaware of the speech capability and used only text. The scores of 0 and 1 were from the text-only participants. Q: question.
Figure 5. Violin plot of participants’ feedback. Each participant’s response to the survey questions in Table 3 is indicated by a point. (A) Feedback from the clinical research team with instructions provided on the flow of the user interface. (B) Psychiatry residents’ feedback when no instructions were provided. Four of the 9 participants were unaware of the speech capability and used only text. The scores of 0 and 1 were from the text-only participants. Q: question.CC BY · PubMed Central

Verification

Claims the paper makes

4 Valid · 2 Not valid · 38 Not checked · 92 Own results and interpretations

49 more claims

Create a free account to see all 136 claims and to verify the 46 references still open.

See all claims — free account
  • Probably not a substantial claim. 51
  • It may state two separate claims as one. 50

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Research works printed but not linked: 3

  1. Kenny, Annu Rev Cyberther Telemed
  2. Hicke Y Geathers J Rajashekar N et al. MedSimAI: simulation and formative feedback generation to enhance deliberate practice in medical education. arXiv. Preprint posted online on Mar 1, 2025. URL: http://arxiv.org/abs/2503.05793 [Accessed 06-07-2026] · arXiv
  3. Harrigian K Aguirre C Dredze M . On the state of social media data for mental health research. arXiv. Preprint posted online on Apr 25, 2021. URL: http://arxiv.org/abs/2011.05233 [Accessed 06-07-2026] · arXiv

Metadata

Authors
Rueda, A., Al-Shamali, H. F., Cote, Z., Roy, N., Janssen-Aguilar, R., Joseph, J., Teferra, B. G., Kamaleddin, M. A., Burback, L., Winkler, O., Kapralos, B., Torres, A., Sharma, D., Krishnan, S., Dumas, G., Greenshaw, A., Hudon, A., Sockalingam, S., Zhang, Y., Dubrowski, A., & Bhat, V.
Affiliations
  • St. Michael's Hospital (CA)
  • University of Alberta (CA)
  • Ontario Tech University (CA)
  • York University (CA)
  • Toronto Metropolitan University (CA)
  • Centre Hospitalier Universitaire Sainte-Justine (CA)
  • Institut national de psychiatrie légale Philippe-Pinel (CA)
  • Centre for Addiction and Mental Health (CA)
Corresponding author
Bhat, Venkat
Venue
JMIR Medical Education
Published
2026-07-21
Volume
12
Pages
e88580-e88580
ISSN
2369-3762 (ISSN-L)
OpenAlex
W7169849571
Cited by
1
References
43
Citations by year
2026: 1
FWCI
1.97
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Medical Education (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Education › Simulation-Based Education in Healthcare
  • Social Sciences › Education › Innovations in Medical Education
Keywords
agentic AI, simulated patient encounters, psychiatry residency training, large language models, automated feedback, major depressive disorder, technical feasibility, inter-rater reliability, rapport building, nonverbal cues, clinical skills assessment
MeSH terms
Psychiatrists, Adult, Artificial Intelligence, Canada, Clinical Competence, Feasibility Studies, Female, Humans, Internship and Residency, Male, Prospective Studies, Psychiatry, Referral and Consultation, Cohort Studies

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.