Development of a Clinical Reasoning Documentation Assessment Tool for Resident and Fellow Admission Notes: a Shared Mental Model for Feedback
Open accessAbstract
Background Residents and fellows receive little feedback on their clinical reasoning documentation. Barriers include lack of a shared mental model and variability in the reliability and validity of existing assessment tools. Of the existing tools, the IDEA assessment tool includes a robust assessment of clinical reasoning documentation focusing on four elements (interpretive summary, differential diagnosis, explanation of reasoning for lead and alternative diagnoses) but lacks descriptive anchors threatening its reliability. Objective Our goal was to develop a valid and reliable assessment tool for clinical reasoning documentation building off the IDEA assessment tool. Design, participants, and main measures The Revised-IDEA assessment tool was developed by four clinician educators through iterative review of admission notes written by medicine residents and fellows and subsequently piloted with additional faculty to ensure response process validity. A random sample of 252 notes from July 2014 to June 2017 written by 30 trainees across several chief complaints was rated. Three raters rated 20% of the notes to demonstrate internal structure validity. A quality cut-off score was determined using Hofstee standard setting. Key results The Revised-IDEA assessment tool includes the same four domains as the IDEA assessment tool with more detailed descriptive prompts, new Likert scale anchors, and a score range of 0-10. Intraclass correlation was high for the notes rated by three raters, 0.84 (95% CI 0.74-0.90). Scores ≥6 were determined to demonstrate high-quality clinical reasoning documentation. Only 53% of notes (134/252) were high-quality. Conclusions The Revised-IDEA assessment tool is reliable and easy to use for feedback on clinical reasoning documentation in resident and fellow admission notes with descriptive anchors that facilitate a shared mental model for feedback.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The assessment tool demonstrated substantial agreement between graders and internal reliability similar or superior to other measures used in the assessment of reasoning.
“This assessment tool demonstrated substantial agreement between graders and internal reliability similar or superior to other measures used in the assessment of reasoning.39,40,41,42”
From Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial (2024) · cites this paper as [40]
The Internal Structure section reports high agreement between raters (ICC 0.84), and the Discussion says this exceeds that of the original IDEA tool. The paper reports moderate agreement between item scores (Cronbach’s α 0.53), but does not establish that this internal reliability is similar or superior to other reasoning measures. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Olson, A., Rencic, J., Cosby, K., Rusz, D., Papa, F., Croskerry, P., Zierler, B., Harkless, G., Giuliano, M. A., Schoenbaum, S., et al. (2019) · Diagnosis · doi:10.1515/dx-2018-0107 · Cited by 101 · see citing works
Flagged first, then by citations
Research works printed but not linked: 26
- Young M, Thomas A, Lubarsky S, et al. Drawing boundaries: the difficulty in defining clinical reasoning. Acad Med. 2018;93(7):990-995.
- Daniel M, Rencic J, Durning SJ, et al. Clinical reasoning assessment methods: a scoping review and practical guidance. Acad Med. 2019;94(6):902-912.
- Bowen JL. Educational strategies to promote clinical diagnostic reason- ing. N Engl J Med. 2006;355(21):2217-2225.
- March CA, Scholl G, Dversdal RK, et al. Use of electronic health record simulation to understand the accuracy of intern progress notes. J Grad Med Educ. 2016;8(2):237-40.
- Colicchio TK, Cimino JJ. Clinicians' reasoning as reflected in electronic clinical note-entry and reading/retrieval: a systematic review and qualitative synthesis. J Am Med Inform Assoc. 2019;26(2):172-184.
- Bierman JA, Hufmeyer KK, Liss DT, Weaver AC, Heiman HL. Promoting responsible electronic documentation: validity evidence for a checklist to assess progress notes in the electronic health record. Teach Learn Med. 2017;29(4):420-432.
- Tierney MJ, Pageler NM, Kahana M, Pantaleoni JL, Longhurst CA. Medical education in the electronic medical record (EMR) era: benefits, challenges, and future directions. Acad Med. 2013;88(6):748-52.
- Lessing JN, Rendón P, Durning SJ, Roesch JJ. Approaches to clinical reasoning assessment. Acad Med. 2020;95(8):1285.
- Middleman AB, Sunder PK, Yen AG. Reliability of the history and physical assessment (HAPA) form. Clin Teach. 2011;8(3):192-5.
- Habboush Y, Hoyt R, Beidas S. Electronic health records as an educational tool: viewpoint. JMIR Med Educ. 2018;4(2):e10306.
- Varpio L, Day K, Elliot-Miller P, et al. The impact of adopting EHRs: how losing connectivity affects clinical reasoning. Med Educ. 2015;49(5):476- 86.
- Berndt M, Fischer MR. The role of electronic health records in clinical reasoning. Ann NY Acad Sci. 2018;1434(1):109-114.
- Atwater AR, Rudd M, Brown A, et al. Developing teaching strategies in the EHR era: a survey of GME experts. J Grad Med Educ. 2016;8(4):581- 586.
- Schenarts PJ, Schenarts KD. Educational impact of the electronic medical record. J Surg Educ. 2012;69(1):105-12.
- Pageler NM, Friedman CP, Longhurst CA. Refocusing medical educa- tion in the EMR era. JAMA. 2013;310(21):2249-50.
- Burke HB, Hoang A, Becher D, et al. QNOTE: an instrument for measuring the quality of EHR clinical notes. J Am Med Inform Assoc. 2014;21(5):910-6.
- Stetson PD, Bakken S, Wrenn JO, Siegler EL. Assessing electronic note quality using the physician documentation quality instrument (PDQI-9). Appl Clin Inform. 2012;3(2):164-174.
- Baker EA, Ledford CH, Fogg L, Way DP, Park YS. The IDEA assessment tool: assessing the reporting, diagnostic reasoning, and decision-making skills demonstrated in medical students' hospital admission notes. Teach Learn Med. 2015;27(2):163-73.
- King MA, Phillipi CA, Buchanan PM, Lewin LO. Developing validity evidence for the written pediatric history and physical exam evaluation rubric. Acad Pediatr. 2017;17(1):68-73.
- Kotwal S, Klimpl D, Tackett S, Kauffman R, Wright S. Documentation of clinical reasoning in admission notes of hospitalists: validation of the CRANAPL assessment rubric. J Hosp Med. 2019;14:E1-e8.
- Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119(2):166.e7-16.
- Schaye V, Janjigian M, Hauck K, et al. A workshop to train medicine faculty to teach clinical reasoning. Diagn. 2019;6(2):109-113.
- Schaye V, Eliasz KL, Janjigian M, Stern DT. Theory-guided teaching: implementation of a clinical reasoning curriculum in residents. Med Teach. 2019;41(10):1192-1199.
- Horlick M, Miller L, Cocks P, Bui L, Schwartz M, Dembitzer A. Calling it like you see it: three-hour workshop improves hospitalists observation and feedback skills. Abstracts from the 38th annual meeting of the society of general internal medicine. J Gen Intern Med. 2015;30(Suppl 2):45-551.
- Thammasitboon S, Rencic JJ, Trowbridge RL, Olson AP, Sur M, Dhaliwal G. The assessment of reasoning tool (ART): structuring the conversation between teachers and learners. Diagn. 2018;5(4):197-203.
- Ende J. Feedback in clinical medical education. JAMA. 1983;250(6):777- 781.
Cited by
Sorted by citations
Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial
Goh, E., Gallo, R., Hom, J., Strong, E., Weng, Y., Kerman, H., Cool, J. A., Kanjee, Z., Parsons, A. S., Ahuja, N., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.40969 · Cited by 724 · see citing works
Clinical reasoning of a generative artificial intelligence model compared with physicians
Cabral, S., Restrepo, D., Kanjee, Z., Wilson, P., Crowe, B., Abdulnour, R.-E., & Rodman, A. (2024) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2024.0295 · Cited by 168 · see citing works
Developing and evaluating large language model–generated emergency medicine handoff notes
Hartman, V., Zhang, X., Poddar, R., McCarty, M., Fortenko, A., Sholle, E., Sharma, R., Campion, T., & Steel, P. A. D. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.48723 · Cited by 76 · see citing works
Demystifying AI: Current State and Future Role in Medical Education Assessment
Turner, L., Hashimoto, D. A., Vasisht, S., & Schaye, V. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005598 · Cited by 70 · see citing works
Self-citationSchaye, V., Guzman, B., Burk-Rafel, J., Marin, M., Reinstein, I., Kudlowitz, D., Miller, L., Chun, J., & Aphinyanaphongs, Y. (2022) · Journal of General Internal Medicine · doi:10.1007/s11606-022-07526-0 · Cited by 44 · see citing works
Self-citationBurk-Rafel, J., Sebok-Syer, S. S., Santen, S. A., Jiang, J., Caretta-Weyer, H. A., Iturrate, E., Kelleher, M., Warm, E. J., Schumacher, D. J., & Kinnear, B. (2023) · Perspectives on Medical Education · doi:10.5334/pme.1013 · Cited by 38 · see citing works
Self-citation
- Authors
- Schaye, V., Miller, L., Kudlowitz, D., Chun, J., Burk-Rafel, J., Cocks, P., Guzman, B., Aphinyanaphongs, Y., & Marin, M.
- Affiliations
- NYU Langone Health (US)
- Donald & Barbara Zucker School of Medicine at Hofstra/Northwell (US)
- New York University (US)
- Stanford Medicine (US)
- Corresponding author
- Schaye, Verity
- ORCID
- Venue
- Journal of General Internal Medicine
- Published
- 2021-05-04
- Volume
- 37
- Issue
- 3
- Pages
- 507-512
- ISSN
- 0884-8734 (ISSN-L)
- 1525-1497
- PMC
- PMC8858363
- OpenAlex
- W3159307263
- Cited by
- 54
- References
- 1
- Citations by year
- 2026: 16 · 2025: 20 · 2024: 13 · 2023: 4 · 2022: 1
- FWCI
- 4.91
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of General Internal Medicine (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- School of Medicine, New York University (N/A)
- Topics
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- admission notes, medical residents, medical fellows, scale development, shared mental models, feedback, differential diagnosis, diagnostic reasoning, inter-rater reliability, standard setting
- MeSH terms
- Clinical Reasoning, Clinical Competence, Documentation, Feedback, Humans, Models, Psychological, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
