When asking the question changes the ultimate answer: Metamemory judgments change memory
The DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index.
Abstract
Self-report measurements are ubiquitous in psychology, but they carry the potential of altering processes they are meant to measure. We assessed whether a common metamemory measure, judgments of learning, can change the ongoing process of memorizing and subsequent memory performance. Judgments of learning are a form of metamemory monitoring described as conscious reflection on one's own memory performance or encoding activities for the purpose of exerting strategic control over one's study and retrieval activities (T. O. Nelson & Narens, 1990). Much of the work examining the conscious monitoring of encoding relies heavily on a paradigm in which participants are asked to estimate the probability that they will recall a given item in a judgment of learning. In 5 experiments, we find effects of measuring judgments of learning on how people allocate their study time to difficult versus easy items, and on what they will recall. These results suggest that judgments of learning are partially constructed in response to the measurement question. The tendency of judgments of learning to alter performance places them in the company of other reactive verbal reporting methods, counseling researchers to consider incorporating control groups, creating alternative scales, and exploring other verbal reporting methods. Less directive methods of accessing participants' metacognition and other judgments should be considered as an alternative to response scales.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“All previous reactivity studies presented stimuli in a unimodal form (e.g., word pairs presented visually; Mitchum et al., 2016; Yang, Huang, et al., 2021), which is somewhat unrepresentative of materials in real educational settings.”
From When judging what you know changes what you really know: Soliciting metamemory judgments reactively enhances children’s learning (2022) · cites this paper as Mitchum et al., 2016
In the citing paper
“Some recent studies have documented that, in some situations, the reactivity effect can be negative (that is, making JOLs impairs retention).”
From When judging what you know changes what you really know: Soliciting metamemory judgments reactively enhances children’s learning (2022) · cites this paper as Mitchum, Kelley, and Fox (2016)
In the citing paper
“The positive reactivity effect on children's learning may be accounted for by the positivereactivity theory (Mitchum et al., 2016; Rivers, 2018; Dougherty, Scheck, Nelson, & Narens, 2005), which proposes that positive reactivity is primarily caused by the fact that making item-by-item JOLs encourages learners to adopt more effective encoding strategies and exert greater study effort.”
From When judging what you know changes what you really know: Soliciting metamemory judgments reactively enhances children’s learning (2022) · cites this paper as Mitchum et al., 2016
Citations
Cites
James, W. (1913) · Henry Holt and Company eBooks · doi:10.1037/11059-000 · Cited by 25,226
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221
Telling more than we can know: Verbal reports on mental processes.
Nisbett, R. E., & Wilson, T. D. (1977) · Psychological Review · doi:10.1037/0033-295x.84.3.231 · Cited by 11,317
An introduction to latent semantic analysis
Landauer, T. K., Foltz, P. W., & Laham, D. (1998) · Discourse Processes · doi:10.1080/01638539809545028 · Cited by 4,880
Ericsson (1993) · doi:10.1037/0033-295x.87.3.215 · Cited by 4,539
Protocol Analysis: Verbal Reports as Data
Ericsson, K. A., & Simon, H. A. (1993) · doi:10.7551/mitpress/5657.001.0001 · Cited by 3,537
Self-reports: How the questions shape the answers.
Schwarz, N. (1999) · American Psychologist · doi:10.1037/0003-066x.54.2.93 · Cited by 2,846
The MRC Psycholinguistic Database
Coltheart, M. (1981) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640748108400805 · Cited by 2,437
On the Rate of Gain of Information
Hick, W. E. (1952) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470215208416600 · Cited by 2,279
The construction of preference.
Slovic, P. (1995) · American Psychologist · doi:10.1037/0003-066x.50.5.364 · Cited by 2,009
The University of South Florida free association, rhyme, and word fragment norms
Nelson, D. L., McEvoy, C. L., & Schreiber, T. A. (2004) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195588 · Cited by 1,929
Kanfer, R., & Ackerman, P. L. (1989) · Journal of Applied Psychology · doi:10.1037/0021-9010.74.4.657 · Cited by 1,767
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980) · Journal of Experimental Psychology: Human Learning and Memory · doi:10.1037/0278-7393.6.2.107 · Cited by 1,477
Kanfer, R., & Ackerman, P. L. (1989) · Journal of Applied Psychology · doi:10.1037//0021-9010.74.4.657 · Cited by 1,393
Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning
Koriat, A. (1997) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.126.4.349 · Cited by 1,363
MRC psycholinguistic database: Machine-usable dictionary, version 2.00
Wilson, M. (1988) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03202594 · Cited by 1,174
Category judgment: A range-frequency model.
Parducci, A. (1965) · Psychological Review · doi:10.1037/h0022602 · Cited by 1,112
How do we know that we know? The accessibility model of the feeling of knowing
Koriat, A. (1993) · Psychological Review · doi:10.1037/0033-295x.100.4.609 · Cited by 895
Consciousness and metacognition.
Nelson, T. O. (1996) · American Psychologist · doi:10.1037//0003-066x.51.2.102 · Cited by 858
Accuracy of metacognitive monitoring affects learning of texts.
Thiede, K. W., Anderson, M. C. M., & Therriault, D. (2003) · Journal of Educational Psychology · doi:10.1037//0022-0663.95.1.66 · Cited by 826
Protocol analysis: Verbal reports as data, Rev. ed.
Ericsson, K. A., & Simon, H. A. (1993) · Cited by 655
Metacognitive and control strategies in study-time allocation.
Son, L. K., & Metcalfe, J. (2000) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037//0278-7393.26.1.204 · Cited by 466
Memory predictions are influenced by perceptual information: Evidence for metacognitive illusions.
Rhodes, M. G., & Castel, A. D. (2008) · Journal of Experimental Psychology: General · doi:10.1037/a0013684 · Cited by 459
Can quantum probability provide a new direction for cognitive modeling?
Pothos, E. M., & Busemeyer, J. R. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12001525 · Cited by 440
Fox, M. C., Ericsson, K. A., & Best, R. (2011) · Psychological Bulletin · doi:10.1037/a0021663 · Cited by 424
Self-citationKoriat, A., Ma'ayan, H., & Nussinson, R. (2006) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.135.1.36 · Cited by 422
Evidence that judgments of learning are causally related to study choice
METCALFE, J., & FINN, B. (2008) · Psychonomic Bulletin & Review · doi:10.3758/pbr.15.1.174 · Cited by 409
Boring, E. G. (1953) · Psychological Bulletin · doi:10.1037/h0090793 · Cited by 402
Thiede, K. W., & Dunlosky, J. (1999) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037//0278-7393.25.4.1024 · Cited by 396
Illusions of Competence in Monitoring One's Knowledge During Study.
Koriat, A., & Bjork, R. A. (2005) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.31.2.187 · Cited by 362
Predicting One's Own Forgetting: The Role of Experience-Based and Theory-Based Processes.
Koriat, A., Bjork, R. A., Sheffer, L., & Bar, S. K. (2004) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.133.4.643 · Cited by 343
A Region of Proximal Learning model of study time allocation
Metcalfe, J., & Kornell, N. (2005) · Journal of Memory and Language · doi:10.1016/j.jml.2004.12.001 · Cited by 327
Goal orientation and task demand effects on motivation, affect, and performance.
Steele‐Johnson, D., Beauregard, R. S., Hoover, P. B., & Schmidt, A. M. (2000) · Journal of Applied Psychology · doi:10.1037//0021-9010.85.5.724 · Cited by 296
On the effectiveness of self-paced learning
Tullis, J. G., & Benjamin, A. S. (2011) · Journal of Memory and Language · doi:10.1016/j.jml.2010.11.002 · Cited by 286
Unconscious Influences of Memory for a Prior Event
Jacoby, L. L., & Kelley, C. M. (1987) · Personality and Social Psychology Bulletin · doi:10.1177/0146167287133003 · Cited by 274
Self-citationIs study time allocated selectively to a region of proximal learning?
Metcalfe, J. (2002) · Journal of Experimental Psychology: General · doi:10.1037//0096-3445.131.3.349 · Cited by 267
Study efficacy and the region of proximal learning framework.
Kornell, N., & Metcalfe, J. (2006) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.32.3.609 · Cited by 256
Sources of information in metamemory: Judgments of learning and feelings of knowing
Schwartz, B. L. (1994) · Psychonomic Bulletin & Review · doi:10.3758/bf03213977 · Cited by 251
Learning Strategies and Transfer in the Domain of Programming
Pirolli, P., & Recker, M. (1994) · Cognition and Instruction · doi:10.1207/s1532690xci1203_2 · Cited by 251
Agenda-based regulation of study-time allocation: When agendas override item-based monitoring.
Ariel, R., Dunlosky, J., & Bailey, H. (2009) · Journal of Experimental Psychology: General · doi:10.1037/a0015928 · Cited by 249
The effects of encoding fluency and retrieval fluency on judgments of learning
Koriat, A., & Ma’ayan, H. (2005) · Journal of Memory and Language · doi:10.1016/j.jml.2005.01.001 · Cited by 235
The influence of delaying judgments of learning on metacognitive accuracy: a meta-analytic review
Rhodes, M. G., & Tauber, S. K. (2011) · Psychological Bulletin · doi:10.1037/a0021705 · Cited by 230
Do different metamemory judgments tap the same underlying aspects of memory?
Leonesio, R. J., & Nelson, T. O. (1990) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037//0278-7393.16.3.464 · Cited by 225
Parducci, A., & Wedell, D. H. (1986) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037//0096-1523.12.4.496 · Cited by 219
Utilization of Metacognitive Judgments in the Allocation of Study During Multitrial Learning
Nelson, T. O., Dunlosky, J., Graf, A., & Narens, L. (1994) · Psychological Science · doi:10.1111/j.1467-9280.1994.tb00502.x · Cited by 215
We talk, therefore we think? A cultural analysis of the effect of talking on thinking.
Kim, H. S. (2002) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.83.4.828 · Cited by 213
Metacognition Does Not Imply Awareness: Strategy Choice Is Governed by Implicit Learning and Memory
(2014) · Implicit Memory and Metacognition · doi:10.4324/9781315806136-8 · Cited by 212
Reactive self-monitoring: The effects of response desirability, goal setting, and feedback.
Kazdin, A. E. (1974) · Journal of Consulting and Clinical Psychology · doi:10.1037/h0037050 · Cited by 208
Griffin, T. D., Wiley, J., & Thiede, K. W. (2008) · Memory & Cognition · doi:10.3758/mc.36.1.93 · Cited by 187
The dynamics of learning and allocation of study time to a region of proximal learning
Metcalfe, J., & Kornell, N. (2003) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.132.4.530 · Cited by 186
Empirical comparison of Markov and quantum models of decision making
Busemeyer, J. R., Wang, Z., & Lambert-Mogiliansky, A. (2009) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2009.03.002 · Cited by 184
The Methodological Imperative in Psychology
Danziger, K. (1985) · Philosophy of the Social Sciences · doi:10.1177/004839318501500101 · Cited by 183
‘Introspectionism’ and the mythical origins of scientific psychology
Costall, A. P. (2006) · Consciousness and Cognition · doi:10.1016/j.concog.2006.09.008 · Cited by 173
A scale distortion theory of anchoring.
Frederick, S. W., & Mochon, D. (2012) · Journal of Experimental Psychology: General · doi:10.1037/a0024006 · Cited by 173
When Predictions Create Reality: Judgments of Learning May Alter What They Are Intended to Assess
Spellman, B. A., & Bjork, R. A. (1992) · Psychological Science · doi:10.1111/j.1467-9280.1992.tb00680.x · Cited by 166
Instability in memory phenomena: A common puzzle and a unifying explanation
McDaniel, M. A., & Bugg, J. M. (2008) · Psychonomic Bulletin & Review · doi:10.3758/pbr.15.2.237 · Cited by 165
Kinds of access: different methods for report reveal different kinds of metacognitive access
Overgaard, M. S., & Sandberg, K. (2012) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2011.0425 · Cited by 165
The role of memory for past test in the underconfidence with practice effect.
Finn, B., & Metcalfe, J. (2007) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.33.1.238 · Cited by 158
Judgments of learning are influenced by memory for past test
Finn, B., & Metcalfe, J. (2007) · Journal of Memory and Language · doi:10.1016/j.jml.2007.03.006 · Cited by 137
Aging and monitoring associative learning: Is monitoring accuracy spared or impaired?
Hertzog, C., Kidder, D. P., Powell-Moman, A., & Dunlosky, J. (2002) · Psychology and Aging · doi:10.1037//0882-7974.17.2.209 · Cited by 130
Delaying judgments of learning affects memory, not metamemory
Kimball, D. R., & Metcalfe, J. (2003) · Memory & Cognition · doi:10.3758/bf03196445 · Cited by 128
Aging and Metamemory: the Roles of Relatedness and Imagery
Rabinowitz, J. C., Ackerman, B. P., Craik, F. I. M., & Hinchley, J. L. (1982) · Journal of Gerontology · doi:10.1093/geronj/37.6.688 · Cited by 121
Mix me a list: Context moderates the truth effect and the mere-exposure effect
Dechêne, A., Stahl, C., Hansen, J., & Wänke, M. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.06.019 · Cited by 97
Betting on memory leads to metacognitive improvement by younger and older adults.
McGillivray, S., & Castel, A. D. (2011) · Psychology and Aging · doi:10.1037/a0022681 · Cited by 94
Accuracy and artifact: Reexamining the intensity bias in affective forecasting.
Levine, L. J., Lench, H. C., Kaplan, R. L., & Safer, M. A. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0029544 · Cited by 92
Framing effects on metacognitive monitoring and control
FINN, B. (2008) · Memory & Cognition · doi:10.3758/mc.36.4.813 · Cited by 90
Adult age differences in the effects of goals on self-regulated sentence processing.
Stine-Morrow, E. A. L., Shake, M. C., Miles, J. R., & Noh, S. R. (2006) · Psychology and Aging · doi:10.1037/0882-7974.21.4.790 · Cited by 88
Koriat, A., Ackerman, R., Adiv, S., Lockl, K., & Schneider, W. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0031768 · Cited by 85
Individual and group predictions of item difficulty for free learning.
Underwood, B. J. (1966) · Journal of Experimental Psychology · doi:10.1037/h0023107 · Cited by 83
Children’s metamemorial judgments in an event recall task
Roebers, C. M., von der Linden, N., Schneider, W., & Howie, P. M. (2007) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2006.12.006 · Cited by 79
Judgments of Learning as Memory Modifiers
Soderstrom, N. C., Clark, C., Halamish, V., & Bjork, E. L. (2014) · PsycEXTRA Dataset · doi:10.1037/e528942014-409 · Cited by 78
Mending metacognitive illusions: A comparison of mnemonic-based and theory-based procedures.
Koriat, A., & Bjork, R. A. (2006) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.32.5.1133 · Cited by 69
Self-evaluation as a moderating factor of strategy change in directed forgetting benefits
Sahakyan, L., Delaney, P. F., & Kelley, C. M. (2004) · Psychonomic Bulletin & Review · doi:10.3758/bf03206472 · Cited by 64
Self-citationFox, M. C., & Charness, N. (2010) · Aging, Neuropsychology, and Cognition · doi:10.1080/13825580903042668 · Cited by 61
Self-citationSplitting consciousness: Unconscious, conscious, and metaconscious processes in social cognition
Winkielman, P., & Schooler, J. W. (2011) · European Review of Social Psychology · doi:10.1080/10463283.2011.576580 · Cited by 57
Sometimes it does hurt to ask: The constructive role of articulating impressions
White, L. C., Pothos, E. M., & Busemeyer, J. R. (2014) · Cognition · doi:10.1016/j.cognition.2014.05.015 · Cited by 56
Calibration of metacognitive judgments: Insights from the underconfidence-with-practice effect
Hanczakowski, M., Zawadzka, K., Pasek, T., & Higham, P. A. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2013.05.003 · Cited by 53
It's Time We Brought Introspection Out of the Closet
Locke, E. A. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01090.x · Cited by 49
Measuring memory monitoring with judgements of retention (JORs)
Tauber, S. K., & Rhodes, M. G. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.656665 · Cited by 47
The effect of testing on the forgetting of related and unrelated associates.
Runquist, W. N. (1986) · Canadian Journal of Psychology/Revue Canadienne de Psychologie · doi:10.1037/h0080086 · Cited by 41
Magnitude and accuracy differences between judgements of remembering and forgetting
Serra, M. J., & England, B. D. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.685081 · Cited by 36
McCabe, D. P., & Soderstrom, N. C. (2011) · Journal of Experimental Psychology: General · doi:10.1037/a0024014 · Cited by 30
Confidence in Personal Preferences
Koriat, A. (2012) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1758 · Cited by 29
Koriat, A., & Sorka, H. (2014) · Cognition · doi:10.1016/j.cognition.2014.09.009 · Cited by 23
The picture complexity effect: Another list composition paradox.
Nguyen, K., & McDaniel, M. A. (2014) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0000071 · Cited by 23
Short, E. J., Schatschneider, C. W., Cuddy, C. L., Evans, S. W., Dellick, D. M., & Basili, L. A. (1991) · Contemporary Educational Psychology · doi:10.1016/0361-476x(91)90033-h · Cited by 22
Cited by
A meta-analysis and systematic review of reactivity to judgements of learning
Double, K. S., Birney, D. P., & Walker, S. A. (2018) · Memory · doi:10.1080/09658211.2017.1404111 · Cited by 120
“…(Dougherty, Scheck, Nelson, & Narens, 2005; Soderstrom et al., 2015; Zechmeister & Shaughnessy, 1980), whereas other have found no reactivity (Benjamin, Bjork, & Schwartz, 1998; Kelemen & Weaver, 1997; Tauber & Rhodes, 2012), and still others have found negative reactivity (Mitchum et al., 2016).”
BackgroundInfluentialEfklides, A., & Schwartz, B. L. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09896-9 · Cited by 130
“Indeed, in cognitive literature, this phenomenon is called negative reactivity, and it has caused some researchers to question the value of prompting metacognitive judgments (Mitchum et al., 2016).”
Reactivity to Measures of Metacognition
Double, K. S., & Birney, D. P. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.02755 · Cited by 118
Providing different types of group awareness information to guide collaborative learning
Schnaubert, L., & Bodemer, D. (2019) · International Journal of Computer-Supported Collaborative Learning · doi:10.1007/s11412-018-9293-y · Cited by 106
The influence of making judgments of learning on memory performance: Positive, negative, or both?
Janes, J. L., Rivers, M. L., & Dunlosky, J. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-018-1463-4 · Cited by 88
“…Mitchum et al. (2016) reported that the increased relatedness effect was largely driven by negative reactivity, wherein recall for unrelated pairs was lower for the JOL group; some trends toward positive reactivity for related pairs occurred in Mitchum et al. (2016), but they were not significant.”
BackgroundMethodResultInfluential
Metadata
- Authors
- Mitchum, A. L., Kelley, C. M., & Fox, M. C.
- Affiliations
- Florida State University (US)
- Venue
- Journal of Experimental Psychology: General
- Published
- 2016
- Volume
- 145
- Issue
- 2
- Pages
- 200-219
- ISSN
- 0096-3445 (ISSN-L)
- 1939-2222
- OpenAlex
- W2479323233
- Cited by
- 148
- References
- 85
- Citations by year
- 2026: 16 · 2025: 18 · 2024: 19 · 2023: 21 · 2022: 11 · 2021: 16 · 2020: 15 · 2019: 9 · 2018: 7 · 2017: 12 · 2016: 3
- FWCI
- 6.59
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Experimental Psychology General (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Functions and Memory
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Keywords
- judgments of learning, metamemory, metacognitive monitoring, self-report bias, measurement effects, study time allocation, memory performance, metacognition, response scales, metacognitive regulation
- MeSH terms
- Metacognition, Adult, Humans, Judgment, Learning, Mental Recall, Word Association Tests, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.