A natural language intelligent tutoring system for training pathologists: implementation and evaluation
Open accessAbstract
Introduction We developed and evaluated a Natural Language Interface (NLI) for an Intelligent Tutoring System (ITS) in Diagnostic Pathology. The system teaches residents to examine pathologic slides and write accurate pathology reports while providing immediate feedback on errors they make in their slide review and diagnostic reports. Residents can ask for help at any point in the case, and will receive context-specific feedback.Research questions We evaluated (1) the performance of our natural language system, (2) the effect of the system on learning (3) the effect of feedback timing on learning gains and (4) the effect of ReportTutor on performance to self-assessment correlations.Methods The study uses a crossover 2 x 2 factorial design. We recruited 20 subjects from 4 academic programs. Subjects were randomly assigned to one of the four conditions--two conditions for the immediate interface, and two for the delayed interface. An expert dermatopathologist created a reference standard and 2 board certified AP/CP pathology fellows manually coded the residents' assessment reports. Subjects were given the opportunity to self grade their performance and we used a survey to determine student response to both interfaces.Results Our results show a highly significant improvement in report writing after one tutoring session with 4-fold increase in the learning gains with both interfaces but no effect of feedback timing on performance gains. Residents who used the immediate feedback interface first experienced a feature learning gain that is correlated with the number of cases they viewed. There was no correlation between performance and self-assessment in either condition.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The common themes of intelligent tutoring system use in medical education include teaching course content, diagnosing strengths or gaps in students' knowledge, providing automated feedback, curating learning materials based on student's needs, and facilitating collaboration between learners.
“The common themes of ITS use in medical education include teaching course content, diagnosing strengths or gaps in students' knowledge, providing automated feedback, curating learning materials based on student's needs, and facilitating collaboration between learners [12].”
From Artificial Intelligence Revolutionizing the Field of Medical Education (2023) · cites this paper as [12]
Cites
Cognitive Tutors: Lessons Learned
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0402_2 · Cited by 1,982 · see citing works
The generation effect: delineation of a phenomenon
Slamecka, N. J., & Graf, P. (1978) · Journal of Experimental Psychology: Human Learning and Memory · doi:10.1037/0278-7393.4.6.592 · Cited by 1,659 · see citing works
Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning
Koriat, A. (1997) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.126.4.349 · Cited by 1,363 · see citing works
The instructional effect of feedback in test-like events.
Bangert-Drowns, R. L., Kulik, C.-L. C., Kulik, J. A., & Morgan, M. (1991) · Review of Educational Research · doi:10.3102/00346543061002213 · Cited by 1,313 · see citing works
Educational Outcomes of Tutoring: A Meta-analysis of Findings
Cohen, P. A., Kulik, J. A., & Kulik, C.-L. C. (1982) · American Educational Research Journal · doi:10.3102/00028312019002237 · Cited by 1,138 · see citing works
Bouwman, M. J. (1983) · European Journal of Operational Research · doi:10.1016/0377-2217(83)90060-7 · Cited by 1,081 · see citing works
A comparison of current measures of the accuracy of feeling-of-knowing predictions
Nelson, T. O. (1984) · Psychological Bulletin · doi:10.1037/0033-2909.95.1.109 · Cited by 976 · see citing works
Wenger, É. (1988) · Knowledge-Based Systems · doi:10.1016/0950-7051(88)90091-3 · Cited by 878 · see citing works
Chi, M. T. H., Siler, S. A., Jeong, H., Yamauchi, T., & Hausmann, R. G. (2001) · Cognitive Science · doi:10.1207/s15516709cog2504_1 · Cited by 855 · see citing works
(2013) · Routledge eBooks · doi:10.4324/9780203052594 · Cited by 759 · see citing works
Timing of feedback and verbal learning.
Kulik, J. A., & Kulik, C.-L. C. (1988) · Review of Educational Research · doi:10.3102/00346543058001079 · Cited by 703 · see citing works
Anderson, J. R., Boyle, C., & Reiser, B. (1985) · Science · doi:10.1126/science.228.4698.456 · Cited by 699 · see citing works
AutoTutor: A tutor with dialogue in natural language
Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H. H., Ventura, M., Olney, A. M., & Louwerse, M. M. (2004) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195563 · Cited by 567 · see citing works
(2001) · doi:10.1609/aimag.v22i4.1591 · Cited by 381 · see citing works
Johnson, M. K., & Hasher, L. (1987) · Annual Review of Psychology · doi:10.1146/annurev.ps.38.020187.003215 · Cited by 319 · see citing works
The Architecture of Why2-Atlas: A Coach for Qualitative Physics Essay Writing
VanLehn, K. A., Jordan, P., Rosé, C. P., Bhembe, D., Böttner, M., Gaydos, A., Makatchev, M., Pappuswamy, U., Ringenberg, M. A., Roque, A., et al. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47987-2_20 · Cited by 230 · see citing works
Andes: A Coached Problem Solving Environment for Physics
Gertner, A. S., & VanLehn, K. A. (2000) · Lecture notes in computer science · doi:10.1007/3-540-45108-0_17 · Cited by 218 · see citing works
The Function of Conceptual Understanding in the Learning of Arithmetic Procedures
Ohlsson, S., & Rees, E. T. (1991) · Cognition and Instruction · doi:10.1207/s1532690xci0802_1 · Cited by 174 · see citing works
Modeling Student Knowledge: Cognitive Tutors in High School and College
Corbett, A. T., McLaughlin, M., & Scarpinatto, K. C. (2000) · User Modeling and User-Adapted Interaction · doi:10.1023/a:1026505626690 · Cited by 99 · see citing works
An intelligent tutoring system for visual classification problem solving
Crowley, R. S., & Medvedeva, O. (2006) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2005.01.005 · Cited by 98 · see citing works
Self-citationBeyond intelligent tutoring systems: Using computers as METAcognitive tools to enhance learning?
Azevedo, R. (2002) · Instructional Science · doi:10.1023/a:1013592216234 · Cited by 77 · see citing works
Evens, M. W., Chang, R.-C., Lee, Y. H., Shim, L. S., Woo, C. W., Zhang, Y., MICHAEL, J. A., & Rovick, A. A. (1997) · Conference on Applied Natural Language Processing · doi:10.3115/974281.974289 · Cited by 72 · see citing works
Cognitive and interactional aspects of correction in tutoring
Fox, B. A. (1991) · Cited by 57 · see citing works
Carroll, M., & Nelson, T. O. (1993) · The American Journal of Psychology · doi:10.2307/1423169 · Cited by 55 · see citing works
Crowley, R. S., Legowski, E. A., Medvedeva, O., Tseytlin, E., Roh, E., & Jukic, D. M. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2241 · Cited by 51 · see citing works
Self-citationThe Cognitive Basis for the Design of a Mammography Interpretation Tutor
Azevedo, R., & Lajoie, S. P. (2022) · Routledge eBooks · doi:10.4324/9781315782416-24 · Cited by 42 · see citing works
Towards computer-assisted maintenance of medical knowledge bases
Giuse, D. A., Giuse, N. B., & Miller, R. A. (1990) · Artificial Intelligence in Medicine · doi:10.1016/0933-3657(90)90014-i · Cited by 33 · see citing works
Tutorials on the structure of tutoring from teachers
Leinhardt, G., & Ohlsson, S. (1990) · Cited by 28 · see citing works
ReportTutor - an intelligent tutoring system that uses a natural language interface.
(2005) · PubMed Central · Cited by 26 · see citing works
Self-citationCIRCSIM-Tutor: An Intelligent Tutoring System Using Natural Language Dialogue
Evens, M. W., Brandle, S., Chang, R.-C., Freedman, R., Glass, M., Lee, Y. H., Shim, L. S., Woo, C. W., Yuemei, Z., Zhou, Y., et al. (2001) · Cited by 23 · see citing works
An ITS for medical classification problem-solving: Effects of tutoring and representations
Crowley, R. S., Legowski, E. A., Medvedeva, O., Tseytlin, E., Roh, E. K., & Jukic, D. M. (2005) · Cited by 14 · see citing works
Self-citationA comparison of tutor and student behavior in speech versus text based tutoring
Rosé, C. P., Litman, D. J., Bhembe, D., Forbes-Riley, K., Silliman, S., Srivastava, R., & VanLehn, K. A. (2003) · doi:10.3115/1118894.1118899 · Cited by 13 · see citing works
Flagged first, then by citations
2 of the 34 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Mousavinasab, E., Zarifsanaiey, N., R. Niakan Kalhori, S., Rakhshan, M., Keikha, L., & Ghazi Saeedi, M. (2021) · Interactive Learning Environments · doi:10.1080/10494820.2018.1558257 · Cited by 605 · see citing works
A scoping review of artificial intelligence in medical education: BEME Guide No. 84
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., et al. (2024) · Medical Teacher · doi:10.1080/0142159x.2024.2314198 · Cited by 543 · see citing works
Feedback for learners in medical education: What is known? A scoping review.
Bing-You, R., Hayes, V., Varaklis, K., Trowbridge, R., Kemp, H., & McKelvy, D. (2017) · Academic Medicine · doi:10.1097/acm.0000000000001578 · Cited by 256 · see citing works
Fontaine, G., Cossette, S., Maheu-Cadotte, M.-A., Mailhot, T., Deschênes, M.-F., Mathieu-Dupuis, G., Côté, J., Gagnon, M.-P., & Dubé, V. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-025252 · Cited by 193 · see citing works
Artificial Intelligence Revolutionizing the Field of Medical Education
Narayanan, S., Ramakrishnan, R., Durairaj, E., & Das, A. (2023) · Cureus · doi:10.7759/cureus.49604 · Cited by 164 · see citing works
A Systematic Literature Review of Intelligent Tutoring Systems With Dialogue in Natural Language
Paladines, J., & Ramírez, J. (2020) · IEEE Access · doi:10.1109/access.2020.3021383 · Cited by 123 · see citing works
Randhawa, G. K., & Jackson, M. (2020) · Healthcare Management Forum · doi:10.1177/0840470419869032 · Cited by 96 · see citing works
Artificial intelligence in dermatopathology: diagnosis, education, and research
Wells, A., Patel, S., Lee, J. B., & Motaparthi, K. (2021) · Journal of Cutaneous Pathology · doi:10.1111/cup.13954 · Cited by 95 · see citing works
Muis, K. R., Ranellucci, J., Trevors, G., & Duffy, M. C. (2015) · Learning and Instruction · doi:10.1016/j.learninstruc.2015.02.001 · Cited by 93 · see citing works
Escape from the Skinner Box: The case for contemporary intelligent learning environments
du Boulay, B. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12860 · Cited by 80 · see citing works
Advanced Technologies for Personalized Learning, Instruction, and Performance
Vandewaetere, M., & Clarebout, G. (2013) · Springer eBooks · doi:10.1007/978-1-4614-3185-5_34 · Cited by 76 · see citing works
Using Machine Learning to Assess Physician Competence: A Systematic Review
DIAS, R. D., Gupta, A., & Yule, S. (2018) · Academic Medicine · doi:10.1097/acm.0000000000002414 · Cited by 70 · see citing works
Furlan, R., Gatti, M., Menè, R., Shiffer, D., Marchiori, C., Giaj Levra, A., Saturnino, V., Brunetta, E., & Dipaola, F. (2021) · JMIR Medical Informatics · doi:10.2196/24073 · Cited by 67 · see citing works
Automated detection of heuristics and biases among pathologists in a computer-based system
Crowley, R. S., Legowski, E. A., Medvedeva, O., Reitmeyer, K., Tseytlin, E., Castine, M., Jukic, D. M., & Mello-Thoms, C. R. (2012) · Advances in Health Sciences Education · doi:10.1007/s10459-012-9374-z · Cited by 64 · see citing works
Self-citationMetacognitive scaffolds improve self-judgments of accuracy in a medical intelligent tutoring system
Feyzi-Behnagh, R., Azevedo, R., Legowski, E. A., Reitmeyer, K., Tseytlin, E., & Crowley, R. S. (2013) · Instructional Science · doi:10.1007/s11251-013-9275-4 · Cited by 59 · see citing works
Self-citationSaadawi, G. M. E., Azevedo, R., Castine, M., Payne, V. L., Medvedeva, O., Tseytlin, E., Legowski, E. A., Jukic, D. M., & Crowley, R. S. (2009) · Advances in Health Sciences Education · doi:10.1007/s10459-009-9162-6 · Cited by 57 · see citing works
Self-citationAparicio, F., Morales-Botello, M. L., Rubio, M., Hernando, A., Muñoz, R., López-Fernández, H., Glez-Peña, D., Fdez-Riverola, F., de la Villa, M., Maña, M., et al. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.12.016 · Cited by 39 · see citing works
The Role of Rubrics in Learning and Implementation of Authentic Assessment: A Literature Review
Nkhoma, C., Nkhoma, M. Z., Thomas, S., & Le, N. Q. (2020) · Informing Science and IT Education Conference · doi:10.28945/4606 · Cited by 37 · see citing works
Item difficulty in the evaluation of computer‐based instruction: An example from neuroanatomy
Chariker, J. H., Naaz, F., & Pani, J. R. (2012) · Anatomical Sciences Education · doi:10.1002/ase.1260 · Cited by 35 · see citing works
Verghese, B. G., Iyer, C., Borse, T., Cooper, S., White, J., & Sheehy, R. M. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07321-5 · Cited by 34 · see citing works
Payne, V. L., Medvedeva, O., Legowski, E. A., Castine, M., Tseytlin, E., Jukic, D. M., & Crowley, R. S. (2009) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2009.07.002 · Cited by 32 · see citing works
Self-citationSkin and Syntax: Large Language Models in Dermatopathology
Shah, A., Wahood, S., Guermazi, D., Brem, C. E., & Saliba, E. (2024) · Dermatopathology · doi:10.3390/dermatopathology11010009 · Cited by 25 · see citing works
Automated outcome scoring in a virtual reality simulator for endodontic surgery
Yin, M. S., Haddawy, P., Suebnukarn, S., & Rhienmora, P. (2017) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2017.10.001 · Cited by 21 · see citing works
An easy to author dialogue management system for serious games
Mori, D., Berta, R., Gloria, A. D., Fiore, V., & Magnani, L. (2013) · Journal on Computing and Cultural Heritage · doi:10.1145/2460376.2460381 · Cited by 20 · see citing works
The Role of Artificial Intelligence in the Diagnosis of Melanoma
Kalidindi, S. (2024) · Cureus · doi:10.7759/cureus.69818 · Cited by 22 · see citing works
Skinner, A. D., Diller, D., Kumar, R., Cannon‐Bowers, J. A., Smith, R. D., Tanaka, A., Julian, D., & Perez, R. S. (2018) · International Journal of STEM Education · doi:10.1186/s40594-018-0108-5 · Cited by 19 · see citing works
Learning with interactive computer graphics in the undergraduate neuroscience classroom
Pani, J. R., Chariker, J. H., Naaz, F., Mattingly, W. A., Roberts, J., & Sephton, S. E. (2014) · Advances in Health Sciences Education · doi:10.1007/s10459-013-9483-3 · Cited by 15 · see citing works
A Systemic Review of Large Language Models and Their Implications in Dermatology
Lewandowski, M., Kropidłowska, J., Kvinen, A., & Barańska‐Rybak, W. (2025) · Australasian Journal of Dermatology · doi:10.1111/ajd.14484 · Cited by 12 · see citing works
EFFECT OF A METACOGNITIVE INTERVENTION ON COGNITIVE HEURISTIC USE DURING DIAGNOSTIC REASONING
Payne, V. L. (2011) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 7 · see citing works
Pedagogy, Cognition, Human Rights, and Social Justice
du Boulay, B. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00355-0 · Cited by 7 · see citing works
Is feedback to medical learners associated with characteristics of improved patient care?
Hayes, V., Bing‐You, R. G., Varaklis, K., Trowbridge, R. L., Kemp, H., & McKelvy, D. J. (2017) · Perspectives on Medical Education · doi:10.1007/s40037-017-0375-8 · Cited by 7 · see citing works
Costa-Dookhan, K. A., Adirim, Z. L., Maslej, M. M., Donner, K., Rodak, T., Soklaridis, S., Sockalingam, S., & Thakur, A. (2025) · Academic Medicine · doi:10.1097/acm.0000000000005983 · Cited by 4 · see citing works
Blanchard, E. G., Wiseman, J., Naismith, L. M., & Lajoie, S. P. (2012) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2012.44 · Cited by 4 · see citing works
A Review of Artificial Intelligence in Dermatopathology
Prestwood, C. A., Gibbs, D. C., Wolner, Z. J., & Stoff, B. K. (2025) · Dermatologic Clinics · doi:10.1016/j.det.2025.05.006 · Cited by 3 · see citing works
Examining Challenges and Approaches to Building Intelligent Tutoring Systems for Teams
Sottilare, R. A., & Eduardo Salas, A. (2018) · Research on managing groups and teams · doi:10.1108/s1534-085620180000019001 · Cited by 3 · see citing works
Graduate Medical Education in Pathology: A Scoping Review
Schmidt, R. L., White, S., Timme, K. H., McFarland, M., & Lomo, L. C. (2023) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2022-0365-ra · Cited by 2 · see citing works
Morán, J. N. P. (2020) · doi:10.20868/upm.thesis.66047 · Cited by 2 · see citing works
Use of a Medical ITS Improves Reporting Performance among Community Pathologists
Crowley, R. S., Grzybicki, D. M., Legowski, E. A., Wagner, L. M., Castine, M., Medvedeva, O., Tseytlin, E., Jukic, D. M., & Raab, S. S. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13388-6_38 · Cited by 1 · see citing works
Self-citationFostering Diagnostic Accuracy in a Medical Intelligent Tutoring System.
Feyzi-Behnagh, R., Azevedo, R., Legowski, E. A., Reitmeyer, K., Tseytlin, E., & Crowley, R. S. (2013) · AIED Workshops · Cited by 1 · see citing works
Self-citationMalone, N. (2017) · Journal of International Crisis and Risk Communication Research · Cited by 1 · see citing works
DomainBuilder: the knowledge authoring system for SlideTutor Intelligent Tutoring system
Tseytlin, E., Linkov, F. Y., Castine, M., Legowski, E. A., & Jacobson, R. S. (2018) · F1000Research · doi:10.12688/f1000research.16060.1 · Cited by 1 · see citing works
Self-citationEffects of feedback timing and type on learning ESL grammar rules
Lavolette, E. (2014) · Michigan State University Libraries · doi:10.25335/m5df6j · Cited by 1 · see citing works
Knowledge Translation and Informatics in Healthcare
McKibbon, A. (2010) · doi:10.1201/9781439809792-12
Davids, J., Lam, K., Nimer, A., Giannarou, S., & Ashrafian, H. (2021) · Springer eBooks · doi:10.1007/978-3-030-58080-3_30-1
Davids, J., Lam, K., Nimer, A., Giannarou, S., & Ashrafian, H. (2022) · Springer eBooks · doi:10.1007/978-3-030-64573-1_30
Khan, M. J. (2020) · Advances in educational technologies and instructional design book series · doi:10.4018/978-1-7998-3062-7.ch017
Knowledge Translation and Informatics in Healthcare
McKibbon, A. (2010) · CRC Press eBooks · doi:10.1201/9781439809792-c6
Artificial Intelligence in Psychiatric Teaching and Training
Tom, A., Reddy, B. S. C., & Math, S. B. (2026) · Springer eBooks · doi:10.1007/978-981-95-1845-6_21
- Authors
- El Saadawi, G. M., Tseytlin, E., Legowski, E., Jukic, D., Castine, M., Fine, J., Gormley, R., & Crowley, R. S.
- Affiliations
- University of Pittsburgh (US)
- Corresponding author
- Crowley, Rebecca S.
- ORCID
- Fine, Jeffrey
- Venue
- Advances in Health Sciences Education
- Published
- 2007-10-13
- Volume
- 13
- Issue
- 5
- Pages
- 709-722
- ISSN
- 1382-4996 (ISSN-L)
- 1573-1677
- PMC
- PMC2753375
- OpenAlex
- W2020458978
- Cited by
- 48
- References
- 34
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 3 · 2023: 3 · 2022: 1 · 2021: 3 · 2020: 4 · 2019: 3 · 2018: 5 · 2017: 5 · 2015: 1 · 2014: 2 · 2013: 4 · 2012: 3
- FWCI
- 1.97
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Advances in Health Sciences Education (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- D-Scholarship@Pitt (University of Pittsburgh) (acceptedVersion)
- Funders
- National Institutes of Health (5r01lm007891-06)
- National Cancer Institute (R25CA101959)
- U.S. National Library of Medicine (R01-LM007891, LM007885, P41LM007885)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Keywords
- intelligent tutoring systems, natural language interfaces, pathology residents, immediate feedback, feedback timing, self-assessment, dermatopathology, medical education
- MeSH terms
- Computer-Assisted Instruction, Humans, Internship and Residency, Natural Language Processing, Pathology, Self-Assessment, User-Computer Interface, Writing, Program Evaluation, Models, Educational, Problem-Based Learning, Feedback, Psychological
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
