A meta-analysis and systematic review of reactivity to judgements of learning
Open accessThe DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
Judgements of learning (JoL) are often used in memory research as a means for assessing an individual's metacognitive beliefs about their learning. JoL have been shown to reliably predict performance as well as learning behaviours and decisions . Participants may, however, modify their behaviour in response to performing JoL. There has, however, been little consensus as to the reliability and direction of the effect. We report on a meta-analyses that assesses the evidence that memory performance is reactive to JoL. The results indicate that overall providing JoL does not have a significant effect on memory performance (g = 0.054, 95% CI -0.027 to 0.135). However, sub-groups analysis showed that this effect depends on the nature of the stimuli to be recalled, with moderate positive reactivity observed for related word pairs (g = 0.323, 95% CI 0.083 to 0.563) and word lists (g = 0.384, 95% CI 0.146 to 0.622) but no reactivity when pairs were unrelated or a mixture of related and unrelated pairs. These results indicate that researchers should be aware that eliciting JoL may well influence participants' underlying encoding processes, especially when using related word pairs or word lists.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Recent studies established that making concurrent judgments of learning (JOLs) can significantly alter (typically enhance) memory itself - a reactivity effect.”
From When judging what you know changes what you really know: Soliciting metamemory judgments reactively enhances children’s learning (2022) · cites this paper as Double, Birney, & Walker, 2018
In the citing paper
“For instance, research has observed positive reactivity effects on learning of word lists (Double et al., 2018), related word pairs (Soderstrom et al., 2015), and general knowledge facts (Yang, Huang, et al., 2021), whereas no reactivity is found on learning of texts (Ariel, Karpicke, Witherby, & Tauber, 2021), unrelated word pairs (Soderstrom et al., 2015; Dougherty, Robey, & Buttaccio, 2018), or”
From When judging what you know changes what you really know: Soliciting metamemory judgments reactively enhances children’s learning (2022) · cites this paper as Double et al., 2018
Citations
Cites
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009) · doi:10.1002/9780470743386 · Cited by 14,225
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
Protocol Analysis: Verbal Reports as Data
Venkatesan, M., Ericsson, K. A., & Simon, H. A. (1986) · Journal of Marketing Research · doi:10.2307/3151491 · Cited by 4,347
Protocol Analysis: Verbal Reports as Data
Ericsson, K. A., & Simon, H. A. (1993) · doi:10.7551/mitpress/5657.001.0001 · Cited by 3,537
Morris, S. B., & DeShon, R. P. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.105 · Cited by 2,502
Flawed Self-Assessment: Implications for Health, Education, and the Workplace
Dunning, D., Heath, C., & Suls, J. M. (2004) · Psychological Science in the Public Interest · doi:10.1111/j.1529-1006.2004.00018.x · Cited by 1,865
The Hawthorne Effect: a randomised, controlled trial
McCarney, R., Warner, J., Iliffe, S., van Haselen, R., Griffin, M., & Fisher, P. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-30 · Cited by 1,715
Schwarzer, G., Carpenter, J. R., & Rücker, G. (2015) · Use R! · doi:10.1007/978-3-319-21416-0 · Cited by 1,230
Why People Fail to Recognize Their Own Incompetence
Dunning, D., Johnson, K., Ehrlinger, J., & Kruger, J. (2003) · Current Directions in Psychological Science · doi:10.1111/1467-8721.01235 · Cited by 1,224
Verbal Protocols of Reading: The Nature of Constructively Responsive Reading
Pressley, M., & Afflerbach, P. (1996) · College Composition and Communication · doi:10.2307/358808 · Cited by 1,003
Verbal Protocols of Reading: The Nature of Constructively Responsive Reading
Bernhardt, E. B., Pressley, M., & Afflerbach, P. (1997) · The Modern Language Journal · doi:10.2307/329341 · Cited by 913
Nelson, T. O., & Dunlosky, J. (1991) · Psychological Science · doi:10.1111/j.1467-9280.1991.tb00147.x · Cited by 580 · cited 1×
“For example, a large body of research has demonstrated that JoL made after a delay are more accurate than immediate JoL (e.g., Narens, Nelson, & Scheck, 2008; Nelson & Dunlosky, 1991), which may help guide educational practice (Dunning, Heath, & Suls, 2004).”
BackgroundThe mismeasure of memory: When retrieval fluency is misleading as a metamnemonic index.
Benjamin, A. S., Bjork, R. A., & Schwartz, B. L. (1998) · Journal of Experimental Psychology: General · doi:10.1037//0096-3445.127.1.55 · Cited by 529
Invited commentary: a critical look at some popular meta-analytic methods
Greenland, S. (1994) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117248 · Cited by 511
Kramarski, B., & Mevarech, Z. R. (2003) · American Educational Research Journal · doi:10.3102/00028312040001281 · Cited by 450 · cited 1×
“…of prompts that have specifically been devised for use in a particular domain, for instance reading (Pressley & Afflerbach, 1995) and mathematics (Kramarski & Mevarech, 2003), JoL can be implemented into most learning and training environments with little modification across content domains,…”
MethodThe Feeling of Knowing: Some Metatheoretical Implications for Consciousness and Control
Koriat, A. (2000) · Consciousness and Cognition · doi:10.1006/ccog.2000.0433 · Cited by 442
Fox, M. C., Ericsson, K. A., & Best, R. (2011) · Psychological Bulletin · doi:10.1037/a0021663 · Cited by 424
Random-Effects Meta-Analyses Are Not Always Conservative
Poole, C., & Greenland, S. (1999) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a010035 · Cited by 338
The Role of Individual Differences in the Accuracy of Confidence Judgments
Pallier, G., Wilkinson, R., Danthiir, V., Kleitman, S., Knezevic, G., Stankov, L., & Roberts, R. D. (2002) · The Journal of General Psychology · doi:10.1080/00221300209602099 · Cited by 336
The self-consistency model of subjective confidence.
Koriat, A. (2012) · Psychological Review · doi:10.1037/a0025648 · Cited by 320
Computer Environments as Metacognitive Tools for Enhancing Learning
Azevedo, R. (2005) · Educational Psychologist · doi:10.1207/s15326985ep4004_1 · Cited by 280
The effects of encoding fluency and retrieval fluency on judgments of learning
Koriat, A., & Ma’ayan, H. (2005) · Journal of Memory and Language · doi:10.1016/j.jml.2005.01.001 · Cited by 235
The influence of delaying judgments of learning on metacognitive accuracy: a meta-analytic review
Rhodes, M. G., & Tauber, S. K. (2011) · Psychological Bulletin · doi:10.1037/a0021705 · Cited by 230 · cited 3×
“This methodology may make JoL susceptible to what is commonly referred to as reactivity (Rhodes & Tauber, 2011).”
MethodInfluentialReactive self-monitoring: The effects of response desirability, goal setting, and feedback.
Kazdin, A. E. (1974) · Journal of Consulting and Clinical Psychology · doi:10.1037/h0037050 · Cited by 208
Complexity, Metacognition, and Fluid Intelligence
Stankov, L. (2000) · Intelligence · doi:10.1016/s0160-2896(99)00033-1 · Cited by 194
When you know that you know and when you think that you know but you don’t
Zechmeister, E. B., & Shaughnessy, J. J. (1980) · Bulletin of the Psychonomic Society · doi:10.3758/bf03329756 · Cited by 170
The diminishing criterion model for metacognitive regulation of time investment.
Ackerman, R. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0035098 · Cited by 152 · cited 2×
“Online self-report measures have been widely used in studies of memory (Ackerman, 2014; Koriat, 2000, 2012; Koriat & Ma’ayan, 2005), reasoning (Pallier et al.”
MethodTO THINK ALOUD OR NOT TO THINK ALOUD: The Issue of Reactivity in SLA Research Methodology
Leow, R. P., & Morgan-Short, K. (2004) · Studies in Second Language Acquisition · doi:10.1017/s0272263104026129 · Cited by 150
When asking the question changes the ultimate answer: Metamemory judgments change memory
Mitchum, A. L., Kelley, C. M., & Fox, M. C. (2016) · Journal of Experimental Psychology: General · doi:10.1037/a0039923 · Cited by 148 · cited 5×
“…(Dougherty, Scheck, Nelson, & Narens, 2005; Soderstrom et al., 2015; Zechmeister & Shaughnessy, 1980), whereas other have found no reactivity (Benjamin, Bjork, & Schwartz, 1998; Kelemen & Weaver, 1997; Tauber & Rhodes, 2012), and still others have found negative reactivity (Mitchum et al., 2016).”
BackgroundInfluentialConfidence and cognitive test performance.
Stankov, L., & Lee, J. (2008) · Journal of Educational Psychology · doi:10.1037/a0012546 · Cited by 133
Carr, S. C., & Punzo, R. P. (1993) · Behavioral Disorders · doi:10.1177/019874299301800401 · Cited by 129
Judgments of learning as memory modifiers.
Soderstrom, N. C., Clark, C. T., Halamish, V., & Bjork, E. L. (2015) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/a0038388 · Cited by 129
Noncognitive predictors of intelligence and academic achievement: An important role of confidence
Stankov, L. (2013) · Personality and Individual Differences · doi:10.1016/j.paid.2013.07.006 · Cited by 128
Izawa, K. P., Watanabe, S., Omiya, K., Hirano, Y., Oka, K., Osada, N., & Iijima, S. (2005) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/01.phm.0000156901.95289.09 · Cited by 120
TO THINK ALOUD OR NOT TO THINK ALOUD: The Issue of Reactivity in SLA Research Methodology
Leow, R. P., & Morgan‐Short, K. (2004) · Studies in Second Language Acquisition · doi:10.1017/s0272263104261022 · Cited by 119
Crawford, J. D., & Stankov, L. (1996) · Learning and Individual Differences · doi:10.1016/s1041-6080(96)90027-8 · Cited by 108
Enhanced memory at delays: Why do judgments of learning improve over time?
Kelemen, W. L., & Weaver, C. A., III (1997) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037//0278-7393.23.6.1394 · Cited by 100
Jackson, S., & Kleitman, S. (2013) · Metacognition and Learning · doi:10.1007/s11409-013-9110-y · Cited by 85
Using the past to predict the future
Dougherty, M. R., Scheck, P., Nelson, T. O., & Narens, L. (2005) · Memory & Cognition · doi:10.3758/bf03193216 · Cited by 78
Judgments of Learning as Memory Modifiers
Soderstrom, N. C., Clark, C., Halamish, V., & Bjork, E. L. (2014) · PsycEXTRA Dataset · doi:10.1037/e528942014-409 · Cited by 78
Subject reactivity in direct observational assessment: A review and critical analysis
Harris, F. C., & Lahey, B. B. (1982) · Clinical Psychology Review · doi:10.1016/0272-7358(82)90028-9 · Cited by 77
McCarthy, D. E., Minami, H., Yeh, V. M., & Bold, K. W. (2015) · Addiction · doi:10.1111/add.12996 · Cited by 75
Self-monitoring and the assessment of binge eating
Latner, J. D., & Wilson, G. T. (2002) · Behavior Therapy · doi:10.1016/s0005-7894(02)80039-9 · Cited by 70
Judgments of learning: Methods, data, and theory.
Rhodes, M. G. (2016) · Cited by 67
Testing models of decision making using confidence ratings in classification.
Balakrishnan, J. D., & Ratcliff, R. (1996) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037//0096-1523.22.3.615 · Cited by 63
Fox, M. C., & Charness, N. (2010) · Aging, Neuropsychology, and Cognition · doi:10.1080/13825580903042668 · Cited by 61 · cited 1×
“Reactivity can occur when participants are observed by an experimenter (e.g., McCarney et al., 2007), or when a participant self-reports on their own behaviour (e.g., Fox & Charness, 2010).”
BackgroundMeasuring memory monitoring with judgements of retention (JORs)
Tauber, S. K., & Rhodes, M. G. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.656665 · Cited by 47
Susser, J. A., Jin, A., & Mulligan, N. W. (2015) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0000189 · Cited by 41
Double, K. S., & Birney, D. P. (2017) · Thinking & Reasoning · doi:10.1080/13546783.2017.1289121 · Cited by 39
Self-citationTauber, S. K. (., Dunlosky, J., & Rawson, K. A. (2015) · Experimental Psychology · doi:10.1027/1618-3169/a000296 · Cited by 36
Differential Neural Correlates Underlie Judgment of Learning and Subsequent Memory Performance
Yang, H., Cai, Y., Liu, Q., Zhao, X., Wang, Q., Chen, C., & Xue, G. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01699 · Cited by 28
The role of awareness in reducing nail-biting behavior
Adesso, V. J., Vargas, J. M., & Siddall, J. W. (1979) · Behavior Therapy · doi:10.1016/s0005-7894(79)80018-0 · Cited by 18
Individual differences in memory-monitoring accuracy
Thompson, W. B. (1999) · Learning and Individual Differences · doi:10.1016/s1041-6080(99)80009-0 · Cited by 6
Research works printed but not linked: 6
- 10.1037/0096-1523.22.3.615 · DOI
- 10.1037/0278-7393.23.6.1394 · doi:10.1037/0278-7393.23.6.1394 · Paper page
- Koriat, A. & Helstrup, T. (2007). Metacognitive aspects of memory everyday memory (pp. 251–274). New York, NY: Psychology Press.
- Narens, L., Nelson, T. O. & Scheck, P. (2008). Memory monitoring and the delayed JOL effect handbook of metamemory and memory (pp. 137–153). New York, NY: Psychology Press.
- Rhodes M. G., 2016, Oxford handbook of metacognition, 65
- Yan Y., 2015, Chinese Journal of Clinical Psychology, 23, 867
Cited by
Chang, D. H., Lin, M. P.-C., Hajian, S., & Wang, Q. Q. (2023) · Sustainability · doi:10.3390/su151712921 · Cited by 370
“Additionally, JOL can influence an individual’s confidence, which in turn can affect their overall approach to learning [11].”
BackgroundA role for metamemory in cognitive offloading
Hu, X., Luo, L., & Fleming, S. M. (2019) · Cognition · doi:10.1016/j.cognition.2019.104012 · Cited by 97
Muncer, G., Higham, P. A., Gosling, C. J., Cortese, S., Wood‐Downie, H., & Hadwin, J. A. (2021) · Educational Psychology Review · doi:10.1007/s10648-021-09620-x · Cited by 119
Reactivity to Measures of Metacognition
Double, K. S., & Birney, D. P. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02755 · Cited by 118
Self-citationThe influence of making judgments of learning on memory performance: Positive, negative, or both?
Janes, J. L., Rivers, M. L., & Dunlosky, J. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-018-1463-4 · Cited by 88
Metadata
- Authors
- Double, K. S., Birney, D. P., & Walker, S. A.
- Affiliations
- The University of Sydney (AU)
- Corresponding author
- Double, Kit S.
- Venue
- Memory
- Publisher
- Informa UK Limited
- Published
- 2017-11-21
- Volume
- 26
- Issue
- 6
- Pages
- 741-750
- ISSN
- 0965-8211 (ISSN-L)
- 1464-0686
- OpenAlex
- W2768752593
- Cited by
- 120
- References
- 54
- Citations by year
- 2026: 16 · 2025: 18 · 2024: 19 · 2023: 23 · 2022: 9 · 2021: 16 · 2020: 9 · 2019: 6 · 2018: 3
- FWCI
- 0.55
- Type
- journal article
- Language
- en
- Open access
- Licence
- cc-by
- Locations
- Memory (publishedVersion)
- PubMed (publishedVersion)
- Durham Research Online (Durham University) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- Australian Research Council (DP140101147)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Functions and Memory
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Keywords
- judgments of learning, reactivity, metacognition, memory performance, encoding processes, word lists, metacognitive beliefs
- MeSH terms
- Metacognition, Humans, Judgment, Learning, Memory, Mental Recall
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.