Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator
Open accessAbstract
Large Language Models (LLMs) have demonstrated remarkable proficiency in human interactions, yet their application within the medical field remains insufficiently explored. Previous works mainly focus on the performance of medical knowledge with examinations, which is far from the realistic scenarios, falling short in assessing the abilities of LLMs on clinical tasks. In the quest to enhance the application of Large Language Models (LLMs) in healthcare, this paper introduces the Automated Interactive Evaluation (AIE) framework and the State-Aware Patient Simulator (SAPS), targeting the gap between traditional LLM evaluations and the nuanced demands of clinical practice. Unlike prior methods that rely on static medical knowledge assessments, AIE and SAPS provide a dynamic, realistic platform for assessing LLMs through multi-turn doctor-patient simulations. This approach offers a closer approximation to real clinical scenarios and allows for a detailed analysis of LLM behaviors in response to complex patient interactions. Our extensive experimental validation demonstrates the effectiveness of the AIE framework, with outcomes that align well with human evaluations, underscoring its potential to revolutionize medical LLM testing for improved healthcare delivery.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous benchmarks like AMIE, SAPS, and CRAFT-MD focus on dialogue-based evaluations but lack multimodal capabilities and do not simulate real-world biases, tool usage, multilingual, or specialist cases.
“Previous benchmarks like AMIE23, SAPS30, and CRAFT-MD31 focus on dialogue-based evaluations but lack multimodal capabilities and do not simulate real-world biases, tool usage, multilingual, or specialist cases.”
From AgentClinic: a multimodal benchmark for tool-using clinical AI agents (2026) · cites this paper as [30]
Section 2.4 (page not provided) confirms that SAPS evaluates multi-round diagnostic dialogues, supporting the dialogue-based description for SAPS. The cited paper does not establish the claimed limitations for AMIE, SAPS, and CRAFT-MD as a group: the passages do not compare their multimodal capabilities or their handling of biases, tools, multilingual cases, and specialist cases. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Binary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 10,387 · see citing works
(2004) · Cited by 9,718 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 1,981 · see citing works
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,028 · see citing works
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948 · see citing works
A Network-based End-to-End Trainable Task-oriented Dialogue System
Wen, T.-H., Vandyke, D., Mrkšić, N., Gašić, M., Rojas-Barahona, L. M., Su, P.-H., Ultes, S., & Young, S. J. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1042 · Cited by 807 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · see citing works
“Best Practice” for Patient-Centered Communication: A Narrative Review
King, A., & Hoppe, R. B. (2013) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-13-00072.1 · Cited by 588 · see citing works
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.14070 · Cited by 512 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · see citing works
Role play with large language models
Shanahan, M., McDonell, K., & Reynolds, L. (2023) · Nature · doi:10.1038/s41586-023-06647-8 · Cited by 491 · see citing works
(2021) · International Conference on Learning Representations · Cited by 435 · see citing works
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical UsagePreprint
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v1 · Cited by 429 · see citing works
Large language models in health care: Development, applications, and challenges
Yang, R., Tan, T. F., Lu, W., Thirunavukarasu, A. J., Ting, D. S. W., & Liu, N. (2023) · Health Care Science · doi:10.1002/hcs2.61 · Cited by 366 · see citing works
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · see citing works
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332 · see citing works
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · see citing works
A Diversity-Promoting Objective Function for Neural Conversation ModelsPreprint
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.03055 · Cited by 242 · see citing works
A Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05112 · Cited by 240
HuatuoGPT, Towards Taming Language Model to Be a Doctor
Zhang, H., Chen, J., Jiang, F., Yu, F., Chen, Z., Chen, G., Li, J., Wu, X., Zhang, Z., Xiao, Q., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.725 · Cited by 223 · see citing works
Task-oriented Dialogue System for Automatic Diagnosis
Wei, Z., Liu, Q., Peng, B., Tou, H., Chen, T., Huang, X., Wong, K., & Dai, X. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2033 · Cited by 200 · see citing works
Challenges and Applications of Large Language Models
Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., & McHardy, R. (2023) · arXiv · doi:10.48550/arxiv.2307.10169 · Cited by 196 · see citing works
The Challenges for Regulating Medical Use of ChatGPT and Other Large Language Models
Minssen, T., Vayena, E., & Cohen, I. G. (2023) · JAMA · doi:10.1001/jama.2023.9651 · Cited by 171 · see citing works
MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
Pal, A., Umapathi, L. K., & Sankarasubbu, M. (2022) · arXiv · doi:10.48550/arxiv.2203.14371 · Cited by 157 · see citing works
A Network-based End-to-End Trainable Task-oriented Dialogue SystemPreprint
Wen, T.-H., Vandyke, D., Mrkšić, N., Gašić, M., Rojas-Barahona, L. M., Su, P.-H., Ultes, S., & Young, S. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.04562 · Cited by 146 · see citing works
Evaluating large language models as agents in the clinic
Mehandru, N., Miao, B. Y., Almaraz, E. R., Sushil, M., Butte, A. J., & Alaa, A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01083-y · Cited by 143 · see citing works
Baichuan 2: Open Large-scale Language Models
Yang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., Yan, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.10305 · Cited by 125 · see citing works
Self-citationMedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training DataPreprint
Han, T., Adams, L. C., Papaioannou, J.-M., Grundmann, P., Oberhauser, T., Figueroa, A., Löser, A., Truhn, D., & Bressem, K. K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08247 · Cited by 117 · see citing works
HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge
Wang, H., Liu, C., Xi, N., Qiang, Z., Zhao, S., Qin, B., & Liu, T. (2023) · arXiv · doi:10.48550/arxiv.2304.06975 · Cited by 115 · see citing works
Towards Conversational Diagnostic AIPreprint
Tu, T., Palepu, A. K., Schaekermann, M., Saab, K. K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Tomašev, N., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.05654 · Cited by 114 · see citing works
DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task
Xiong, H., Wang, S., Zhu, Y., Zhao, Z., Liu, Y., Huang, L., Wang, Q., & Shen, D. (2023) · arXiv · doi:10.48550/arxiv.2304.01097 · Cited by 81 · see citing works
Six ways large language models are changing healthcare
Webster, P. C. (2023) · Nature Medicine · doi:10.1038/s41591-023-02700-1 · Cited by 58 · see citing works
A benchmark for automatic medical consultation system: frameworks, tasks and datasets
Chen, W., Li, Z., Fang, H., Yao, Q., Zhong, C., Hao, J., Zhang, Q., Huang, X., Peng, J., & Wei, Z. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac817 · Cited by 52 · see citing works
LLM-empowered Chatbots for Psychiatrist and Patient Simulation: Application and EvaluationPreprint
Chen, S., Wu, M., Zhu, K. Q., Lan, K., Zhang, Z., & Cui, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13614 · Cited by 41 · see citing works
MedDG: An Entity-Centric Medical Consultation Dataset for Entity-Aware Medical Dialogue Generation
Liu, W., Tang, J., Yi, C. X., Li, W., Zheng, Y., & Liang, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17120-8_35 · Cited by 34 · see citing works
DISC-MedLLM: Bridging General Large Language Models and Real-World Medical ConsultationPreprint
Bao, Z., Chen, W., Xiao, S., Ren, K., Wu, J., Zhong, C., Peng, J., Huang, X., & Wei, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.14346 · Cited by 26 · see citing works
HuatuoGPT, towards Taming Language Model to Be a DoctorPreprint
Zhang, H., Chen, J., Jiang, F., Yu, F. R., Chen, Z., Li, J., Chen, G., Wu, X., Zhang, Z., Xiao, Q., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.15075 · Cited by 20 · see citing works
Medical Dialogue Response Generation with Pivotal Information Recalling
Zhao, Y., Li, Y., Wu, Y., Hu, B., Chen, Q., Wang, X., Ding, Y., & Zhang, M. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3542674 · Cited by 17 · see citing works
Liao, Y., Meng, Y., Liu, H., Wang, Y., & Wang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.02077 · Cited by 3 · see citing works
Self-citationMedDG: An Entity-Centric Medical Consultation Dataset for Entity-Aware Medical Dialogue GenerationPreprint
Liu, W., Tang, J., Cheng, Y., Li, W., Zheng, Y., & Liang, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07497 · Cited by 3 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 4
- Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. arXiv preprint arXiv:2009.13081, 2020b. · arXiv
- Shreya Johri, Jaehwan Jeong, Benjamin A Tran, Daniel I Schlessinger, Shannon Wongvibulsin, Zhuo Ran Cai, Roxana Daneshjou, and Pranav Rajpurkar. Testing the limits of language models: A conversational framework for medical ai assessment. medRxiv, 2023.
- Shreya Johri, Jaehwan Jeong, Benjamin A. Tran, Daniel I Schlessinger, Shannon Wongvibulsin, Zhuo Ran Cai, Roxana Daneshjou, and Pranav Rajpurkar. CRAFT- MD: A conversational evaluation framework for comprehensive assessment of clini- cal LLMs. In AAAI 2024 Spring Symposium on Clinical Foundation Models, 2024. URL https://openreview.net/forum?id=Bk2nbTDtm8.
- Gerardo Flores, George H. Chen, Tom J. Pollard, Joyce C. Ho, and Tristan Naumann (eds.), Conference on Health, Inference, and Learning, CHIL 2022, 7-8 April 2022, Virtual Event, volume 174 of Proceedings of Machine Learning Research, pp. 248-260. PMLR, 2022a. URL https://proceedings.mlr.press/v174/pal22a.html.
Cited by
Sorted by citations
Quantifying the reasoning abilities of LLMs on clinical cases
Qiu, P., Wu, C., Liu, S., Fan, Y., Zhao, W., Chen, Z., Gu, H., Peng, C., Zhang, Y., Wang, Y., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64769-1 · Cited by 53 · see citing works
Self-citationZeng, J., Qi, W., Shen, S., Liu, X., Li, S., Wang, B., Dong, C., Zhu, X., Shi, Y., Lou, X., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/79091 · Cited by 49 · see citing works
Weisman, D., Sugarman, A., Huang, Y. M., Gelberg, L., Ganz, P. A., & Comulada, W. S. (2025) · JMIR Formative Research · doi:10.2196/65670 · Cited by 37 · see citing works
Steenstra, I., Nouraei, F., & Bickmore, T. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714014 · Cited by 31 · see citing works
Li, D., & Lutfi, S. L. (2025) · JMIR Medical Informatics · doi:10.2196/79039 · Cited by 31 · see citing works
Critique of impure reason: Unveiling the reasoning behaviour of medical large language models
Sim, S. Z. Y., & Chen, T. (2025) · eLife · doi:10.7554/elife.106187 · Cited by 24 · see citing works
AgentClinic: a multimodal benchmark for tool-using clinical AI agents
Schmidgall, S., Ziaei, R., Harris, C., Kim, J. W., Reis, E. P., Jopling, J., & Moor, M. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02674-7 · Cited by 11 · see citing works
Enhanced large language models for effective screening of depression and anxiety
Liu, J. M., Gao, M., Sabour, S., Chen, Z., Huang, M., & Lee, T. M. C. (2025) · Communications Medicine · doi:10.1038/s43856-025-01158-1 · Cited by 17 · see citing works
Ethics and Regulations in Generative AI
ZadZiabari, A., & Tabatabaei, A. (2025) · Springer eBooks · doi:10.1007/978-3-031-82963-5_8 · Cited by 5 · see citing works
AI-simulated clinical consultations: Assessing the potential of ChatGPT to support medical training
Saggar, A., Dimitrova, V., Sarikaya, D., Hogg, D., & Darling, J. C. (2026) · Archives of Disease in Childhood · doi:10.1136/archdischild-2025-329846 · Cited by 4 · see citing works
Yu, Y. Q., Li, L., & Li, Y. (2025) · medRxiv · doi:10.1101/2025.10.17.25338266 · Cited by 5 · see citing works
Imaezue, G. C., Maram, K. V., Ajayi, D., Alohali, I., & Butta, R. K. (2026) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2026_jslhr-26-00031 · Cited by 1 · see citing works
AI-Simulated Clinical Consultations: Assessing the Potential of ChatGPT to Support Medical TrainingPreprint
Saggar, A., Dimitrova, V. G., Sarıkaya, D., Hogg, D., & Darling, J. C. (2025) · medRxiv · doi:10.1101/2025.10.07.25337156
Li, D., Lutfi, S. L., Ahmad, A., & Mulki, H. (2026) · doi:10.2196/preprints.111051
Shawon, M. T. R., Irbaz, M. S., Elyazori, H. R. A., Resapu, K. R., Lin, Y., Cardenas, V. F., Eklou, K. P., Alemi, F., & Lybarger, K. (2026) · JMIR AI · doi:10.2196/100772
- Authors
- Liao, Y., Meng, Y., Wang, Y., Liu, H., Wang, Y., & Wang, Y.
- Affiliations
- Shanghai Jiao Tong University (CN)
- Corresponding author
- Wang, Yuhao
- Wang, Yanfeng
- Wang, Yu
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 1556-5068 (ISSN-L)
- arXiv
- 2403.08495
- OpenAlex
- W4400665557
- Cited by
- 15
- References
- 38
- Citations by year
- 2026: 5 · 2025: 10
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- large language models, LLM evaluation, multi-turn dialogue, clinical scenario simulation, medical consultation, medical training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
