Factors affecting feeling-of-knowing in a medical intelligent tutoring system: the role of immediate feedback as a metacognitive scaffold
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Previous studies in our laboratory have shown the benefits of immediate feedback on cognitive performance for pathology residents using an intelligent tutoring system (ITS) in pathology. In this study, we examined the effect of immediate feedback on metacognitive performance, and investigated whether other metacognitive scaffolds will support metacognitive gains when immediate feedback is faded. Twenty-three participants were randomized into intervention and control groups. For both groups, periods working with the ITS under varying conditions were alternated with independent computer-based assessments. On day 1, a within-subjects design was used to evaluate the effect of immediate feedback on cognitive and metacognitive performance. On day 2, a between-subjects design was used to compare the use of other metacognitive scaffolds (intervention group) against no metacognitive scaffolds (control group) on cognitive and metacognitive performance, as immediate feedback was faded. Measurements included learning gains (a measure of cognitive performance), as well as several measures of metacognitive performance, including Goodman–Kruskal gamma correlation (G), bias, and discrimination. For the intervention group, we also computed metacognitive measures during tutoring sessions. Results showed that immediate feedback in an intelligent tutoring system had a statistically significant positive effect on learning gains, G and discrimination. Removal of immediate feedback was associated with decreasing metacognitive performance, and this decline was not prevented when students used a version of the tutoring system that provided other metacognitive scaffolds. Results obtained directly from the ITS suggest that other metacognitive scaffolds do have a positive effect on G and discrimination, as immediate feedback is faded. We conclude that immediate feedback had a positive effect on both metacognitive and cognitive gains in a medical tutoring system. Other metacognitive scaffolds were not sufficient to replace immediate feedback in this study. However, results obtained directly from the tutoring system are not consistent with results obtained from assessments. In order to facilitate transfer to real-world tasks, further research will be needed to determine the optimum methods for supporting metacognition as immediate feedback is faded.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Origins and functions of positive and negative affect: A control-process view.
Carver, C. S., & Scheier, M. F. (1990) · Psychological Review · doi:10.1037/0033-295x.97.1.19 · Cited by 2,726
Studying as Self-Regulated Learning
(1998) · Metacognition in Educational Theory and Practice · doi:10.4324/9781410602350-19 · Cited by 1,589 · cited 1×
“One explanation for this apparent discrepancy is that the experiment was conducted over a very short period of time relative to other demonstrations of the effects of metacognitive scaffolds (Winne and Hadwin 1998; Choi et al. 2005; Dabbagh and Kitsantas 2005; Puntambekar and Hubscher 2005).”
ResultDiagnostic error in internal medicine
Graber, M. L., Franklin, N., & Gordon, R. (2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.13.1493 · Cited by 1,549
Handbook of Metacognition in Education
Hacker, D. J., Dunlosky, J., & Graesser, A. C. (2009) · doi:10.4324/9780203876428 · Cited by 1,509
Self-Regulated Learning and Academic Achievement
(2013) · doi:10.4324/9781410601032 · Cited by 1,325
Scaffolding Complex Learning: The Mechanisms of Structuring and Problematizing Student Work
Reiser, B. J. (2004) · Journal of the Learning Sciences · doi:10.1207/s15327809jls1303_2 · Cited by 1,054
A comparison of current measures of the accuracy of feeling-of-knowing predictions
Nelson, T. O. (1984) · Psychological Bulletin · doi:10.1037/0033-2909.95.1.109 · Cited by 976
Aleven, V. A. W. M. M., & Koedinger, K. R. (2002) · Cognitive Science · doi:10.1207/s15516709cog2602_1 · Cited by 895
Puntambekar, S., & Hubscher, R. (2005) · Educational Psychologist · doi:10.1207/s15326985ep4001_1 · Cited by 787 · cited 4×
“123 Puntambekar and Hubscher 2005 ). Self-explanation prompts (Aleven and Koedinger 2002) have also been used to facilitate problem solving and help determine the adequacy of one’s understanding of the topic.”
BackgroundMethodResultInfluentialEffects of cognitive feedback on performance.
Balzer, W. K., Doherty, M. E., & O'Connor, R. (1989) · Psychological Bulletin · doi:10.1037//0033-2909.106.3.410 · Cited by 516
Azevedo, R., Moos, D. C., Greene, J. A., Winters, F. I., & Cromley, J. G. (2008) · Educational Technology Research and Development · doi:10.1007/s11423-007-9067-0 · Cited by 380 · cited 2×
“Learning to self-monitor and control these cognitive processes is termed self-regulated learning (SRL), and is an important part of expertise (Winne 2001; Azevedo et al. 2008; Zimmerman 2006).”
BackgroundSelf-citationCorbett, A. T., & Anderson, J. R. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365111 · Cited by 280 · cited 2×
“Previous research has demonstrated that immediate feedback is associated with efficient learning (Corbett and Anderson 2001), including within medical domains (Saadawi et al. 2008).”
BackgroundMinimizing the black box problem to enhance the validity of theories about instructional effects.
Winne, P. H. (1982) · Instructional Science · doi:10.1007/bf00120978 · Cited by 123 · cited 2×
“On the other hand, feedback throughout the entire session provides cues directly affecting student’s self regulation (Winne 1982; Winne and Marx 1982) by enhancing the student’s ability to associate these cues with actual performance or correctness (Balzer et al. 1989).”
BackgroundAn intelligent tutoring system for visual classification problem solving
Crowley, R. S., & Medvedeva, O. (2006) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2005.01.005 · Cited by 98 · cited 14×
“SlideTutor is a client-server application in which students examine virtual slides using various magnifications, identify visual features, specify qualities of these features, make hypotheses and diagnoses, and write pathology reports.”
BackgroundMethodInfluentialEl Saadawi, G. M., Tseytlin, E., Legowski, E., Jukic, D., Castine, M., Fine, J., Gormley, R., & Crowley, R. S. (2008) · Advances in Health Sciences Education · doi:10.1007/s10459-007-9081-3 · Cited by 48
Self-citation
Cited by
Supporting Self-Regulated Learning in Online Learning Environments and MOOCs: A Systematic Review
Wong, J., Baars, M., Davis, D., Van Der Zee, T., Houben, G.-J., & Paas, F. (2019) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2018.1543084 · Cited by 631
“El Saadawi et al.’s (2010) study examined the effect of immediate feedback and whether once the immediate feedback is withdrawn, other metacognitive scaffolds can be beneficial. The results indicated that metacognitive scaffolds in ITS need to be paired with immediate feedback to have an impact on learning. When the i…”
BackgroundInfluentialFeedback for learners in medical education: What is known? A scoping review.
Bing-You, R., Hayes, V., Varaklis, K., Trowbridge, R., Kemp, H., & McKelvy, D. (2017) · Academic Medicine · doi:10.1097/acm.0000000000001578 · Cited by 256
Fontaine, G., Cossette, S., Maheu-Cadotte, M.-A., Mailhot, T., Deschênes, M.-F., Mathieu-Dupuis, G., Côté, J., Gagnon, M.-P., & Dubé, V. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-025252 · Cited by 193
“In this review 11 AEEs employed designed adaptivity(36) 38 41–44 46–50 and 9 AEEs employed algorithmic adaptivity.(33) 37 39 51 52 54–57 The adaptivity method wasn’t specified in one study.”
BackgroundMethodRandhawa, G. K., & Jackson, M. (2020) · Healthcare Management Forum · doi:10.1177/0840470419869032 · Cited by 96
Artificial intelligence in dermatopathology: diagnosis, education, and research
Wells, A., Patel, S., Lee, J. B., & Motaparthi, K. (2021) · Journal of Cutaneous Pathology · doi:10.1111/cup.13954 · Cited by 95
Metadata
- Authors
- El Saadawi, G. M., Azevedo, R., Castine, M., Payne, V., Medvedeva, O., Tseytlin, E., Legowski, E., Jukic, D., & Crowley, R. S.
- Venue
- Advances in Health Sciences Education
- Published
- 2010-03
- Volume
- 15
- Issue
- 1
- Pages
- 9-30
- ISSN
- 1382-4996 (ISSN-L)
- 1573-1677
- OpenAlex
- W2010052797
- Cited by
- 55
- References
- 15
- FWCI
- 6.51
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.