The replicability of the negative testing effect: differences across participant populations
Abstract
Although memory retrieval often enhances subsequent memory, Peterson and Mulligan (2013) reported conditions under which retrieval produces poorer subsequent recall-the negative testing effect. The item-specific-relational account proposes that the effect occurs when retrieval disrupts interitem organizational processing relative to the restudy condition. Rawson et al. (2015), in contrast, failed to replicate the negative testing effect despite repeated high-powered attempts. This article examines the discrepant results, ruling out differences in procedures, and concludes that differences in participant population produced the varying outcome. Specifically, participants from the University of North Carolina (UNC) and Kent State University (KSU) completed the same version of the negative-testing paradigm and were assessed on several measures of cognitive ability (working memory capacity, Raven's progressive matrices, and SAT or ACT score). For the UNC sample, free recall scores and the amount of category clustering (a measure of organizational processing) was greater in the restudy than retrieval condition (i.e., the negative testing effect was found); for the KSU sample, there was no difference on either measure. Furthermore, in the restudy condition, recall and clustering was greater for UNC than KSU students whereas in the retrieval condition, there was no effect of site on either measure. As expected, measures of cognitive ability were greater for the UNC than KSU sample. The results indicate that the negative testing effect is replicable but is subject to limitation related to the participant population. An analysis in terms of the relationship between cognitive ability and memory predicted this pattern of results. (PsycINFO Database Record
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Testing effects may fail to generalize across participant populations, even when procedures are held constant.
“Moreover, evidence from laboratory studies suggests that testing effects—including both positive and negative variants—may fail to generalize across participant populations, even when procedures are held constant (Mulligan et al., 2018).”
From Testing the testing effect on prolific: when retrieval practice fails to boost learning (2026) · cites this paper as Mulligan et al., 2018
Cites
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971 · see citing works
(1967) · Medical Entomology and Zoology · Cited by 7,532 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567 · see citing works
Test-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention
Roediger, H. L., & Karpicke, J. D. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01693.x · Cited by 2,867 · see citing works
The critical role of retrieval practice in long-term retention
Roediger, H. L., & Butler, A. C. (2011) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.09.003 · Cited by 1,796 · see citing works
Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125 · see citing works
The effect of testing versus restudy on retention: A meta-analytic review of the testing effect.
Rowland, C. A. (2014) · Psychological Bulletin · doi:10.1037/a0037559 · Cited by 1,066 · see citing works
Comparison of measures for the estimation of clustering in free recall.
Roenker, D. L., Thompson, C. P., & Brown, S. C. (1971) · Psychological Bulletin · doi:10.1037/h0031355 · Cited by 782 · see citing works
Is the Replicability Crisis Overblown? Three Arguments Examined
Pashler, H., & Harris, C. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612463401 · Cited by 697 · see citing works
The Value of Direct Replication
Simons, D. J. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691613514755 · Cited by 630 · see citing works
Category norms: An updated and expanded version of the Battig and Montague (1969) norms
Overschelde, J. P. V., Rawson, K. A., & Dunlosky, J. T. (2003) · Journal of Memory and Language · doi:10.1016/j.jml.2003.10.003 · Cited by 621 · see citing works
Self-citationFerguson, C. J., & Heene, M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459059 · Cited by 603 · see citing works
The influence of retrieval on retention
Carrier, L. M., & Pashler, H. (1992) · Memory & Cognition · doi:10.3758/bf03202713 · Cited by 580 · see citing works
The Enigma of Organization and Distinctiveness
Hunt, R. R., & McDaniel, M. A. (1993) · Journal of Memory and Language · doi:10.1006/jmla.1993.1023 · Cited by 532 · see citing works
The ironic effect of significant results on the credibility of multiple-study articles.
Schimmack, U. (2012) · Psychological Methods · doi:10.1037/a0029487 · Cited by 455 · see citing works
The Positive and Negative Consequences of Multiple-Choice Testing.
Roediger, H. L., III, & Marsh, E. J. (2005) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.31.5.1155 · Cited by 386 · see citing works
Nash Unsworth, A., & Spillers, G. J. (2010) · Journal of Memory and Language · doi:10.1016/j.jml.2010.02.001 · Cited by 373 · see citing works
Continuously Cumulating Meta-Analysis and Replicability
Braver, S. L., Thoemmes, F. J., & Rosenthal, R. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614529796 · Cited by 292 · see citing works
Why tests appear to prevent forgetting: A distribution-based bifurcation model
Kornell, N., Bjork, R. A., & Garcia, M. A. (2011) · Journal of Memory and Language · doi:10.1016/j.jml.2011.04.002 · Cited by 277 · see citing works
Different rates of forgetting following study versus test trials
Wheeler, M., Ewers, M., & Buonanno, J. F. (2003) · Memory · doi:10.1080/09658210244000414 · Cited by 249 · see citing works
The testing effect in free recall is associated with enhanced organizational processes
Zaromb, F. M., & Roediger, H. L. (2010) · Memory & Cognition · doi:10.3758/mc.38.8.995 · Cited by 206 · see citing works
Instability in memory phenomena: A common puzzle and a unifying explanation
McDaniel, M. A., & Bugg, J. M. (2008) · Psychonomic Bulletin & Review · doi:10.3758/pbr.15.2.237 · Cited by 170 · see citing works
Retrieval mode distinguishes the testing effect from the generation effect
Karpicke, J. D., & Zaromb, F. M. (2009) · Journal of Memory and Language · doi:10.1016/j.jml.2009.11.010 · Cited by 163 · see citing works
Dissociative effects of generation on item and order retention.
Nairne, J. S., Riegler, G. L., & Serra, M. (1991) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.17.4.702 · Cited by 158 · see citing works
Sex Differences in Episodic Memory: The Influence of Intelligence
Herlitz, A., & Yonker, J. E. (2002) · Journal of Clinical and Experimental Neuropsychology · doi:10.1076/jcen.24.1.107.970 · Cited by 147 · see citing works
An investigation of the role of working memory in procedural skill acquisition.
Woltz, D. J. (1988) · Journal of Experimental Psychology General · doi:10.1037/0096-3445.117.3.319 · Cited by 139 · see citing works
What's So Special About Working Memory?
Mogle, J. A., Lovett, B. J., Stawski, R. S., & Sliwinski, M. J. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02202.x · Cited by 114 · see citing works
Psychology’s Woes and a Partial Cure: The Value of Replication
Roediger, H. L., III (2012) · APS observer · Cited by 101 · see citing works
A Classroom Study on the Relationship Between Student Achievement and Retrieval-Enhanced Learning
Carpenter, S. K., Lund, T. J. S., Coffman, C. R., Armstrong, P. I., Lamm, M. H., & Reason, R. D. (2015) · Educational Psychology Review · doi:10.1007/s10648-015-9311-9 · Cited by 90 · see citing works
Sources of individual differences in working memory: Contributions of strategy to capacity
Cokely, E. T., Kelley, C. M., & Gilchrist, A. L. (2006) · Psychonomic Bulletin & Review · doi:10.3758/bf03213914 · Cited by 88 · see citing works
Mulligan, N. W., & Lozito, J. P. (2004) · The Psychology of learning and motivation/The psychology of learning and motivation · doi:10.1016/s0079-7421(03)45005-6 · Cited by 81 · see citing works
Self-citationCongleton, A. R., & Rajaram, S. (2011) · Memory & Cognition · doi:10.3758/s13421-011-0168-y · Cited by 75 · see citing works
Was, C. A., & Woltz, D. J. (2006) · Journal of Memory and Language · doi:10.1016/j.jml.2006.07.008 · Cited by 71 · see citing works
Relationship of iq to verbal learning and memory: Test and retest
Rapport, L. J., Axelrod, B. N., Theisen, M. E., Brines, D. B., Kalechstein, A. D., & Ricker, J. H. (1997) · Journal of Clinical and Experimental Neuropsychology · doi:10.1080/01688639708403751 · Cited by 63 · see citing works
The negative testing effect and multifactor account.
Peterson, D. J., & Mulligan, N. W. (2013) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/a0031337 · Cited by 56 · see citing works
Self-citationThe consequences of generation
Burns, D. J. (1992) · Journal of Memory and Language · doi:10.1016/0749-596x(92)90031-r · Cited by 54 · see citing works
Determ inants of Positive and Negative Generation Effects in Free Recall
Steffens, M. C., & Erdfelder, E. (1998) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/713755794 · Cited by 52 · see citing works
The generation effect: A test between single- and multifactor theories.
Burns, D. J. (1990) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037//0278-7393.16.6.1060 · Cited by 51 · see citing works
Testing the item-order account of design effects using the production effect.
Jonker, T. R., Levene, M., & MacLeod, C. M. (2013) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/a0034977 · Cited by 45 · see citing works
Testing enhances learning across a range of episodic memory abilities
Pan, S. C., Pashler, H., Potter, Z. E., & Rickard, T. C. (2015) · Journal of Memory and Language · doi:10.1016/j.jml.2015.04.001 · Cited by 41 · see citing works
Negative and positive testing effects in terms of item-specific and relational information.
Mulligan, N. W., & Peterson, D. J. (2014) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0000056 · Cited by 41 · see citing works
Self-citationMulligan, N. W., & Stone, M. (1999) · Journal of Memory and Language · doi:10.1006/jmla.1999.2648 · Cited by 40 · see citing works
Self-citationResponse-bound primes diminish affective priming in the naming task
Wentura, D., & Frings, C. (2008) · Cognition & Emotion · doi:10.1080/02699930701446064 · Cited by 31 · see citing works
Content-embedded tasks beat complex span for predicting comprehension
Was, C. A., Rawson, K. A., Bailey, H. R., & Dunlosky, J. T. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0112-x · Cited by 25 · see citing works
Self-citationIndividual and Combined Effects of Enactment and Testing on Memory for Action Phrases
Kubik, V., Söderlund, H., Nilsson, L., & Jönsson, F. U. (2014) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1027/1618-3169/a000254 · Cited by 25 · see citing works
Testing effects in mixed- versus pure-list designs
Rowland, C. A., Littrell, M. K., Sensenig, A. E., & DeLosh, E. L. (2014) · Memory & Cognition · doi:10.3758/s13421-014-0404-3 · Cited by 19 · see citing works
Unsworth, N. (2008) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470210802373092 · Cited by 17 · see citing works
The negative testing and negative generation effects are eliminated by delay.
Mulligan, N. W., & Peterson, D. J. (2014) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0000070 · Cited by 15 · see citing works
Self-citationThe testing effect is moderated by experimental design
Mulligan, N. W., Susser, J. A., & Smith, S. (2016) · Journal of Memory and Language · doi:10.1016/j.jml.2016.03.006 · Cited by 13 · see citing works
Self-citationDoes testing impair relational processing? Failed attempts to replicate the negative testing effect.
Rawson, K. A., Wissman, K. T., & Vaughn, K. E. (2015) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0000127 · Cited by 11 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Individual differences in long-term memory.
Unsworth, N. (2019) · Psychological Bulletin · doi:10.1037/bul0000176 · Cited by 99 · see citing works
Value-directed memory effects on item and context memory
Villaseñor, J. J., Sklenar, A. M., Frankenstein, A. N., Levy, P. U., McCurdy, M. P., & Leshikar, E. D. (2021) · Memory & Cognition · doi:10.3758/s13421-021-01153-6 · Cited by 32 · see citing works
Kubik, V., Jönsson, F. U., de Jonge, M., & Arshamian, A. (2020) · Quarterly Journal of Experimental Psychology · doi:10.1177/1747021820945560 · Cited by 22 · see citing works
The testing effect with free recall: Organization, attention, and order effects
Mulligan, N. W., Buchin, Z. L., & Zhang, A. L. (2022) · Journal of Memory and Language · doi:10.1016/j.jml.2022.104333 · Cited by 13 · see citing works
Self-citationKarpicke, J. D. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-15754-7.00074-2 · Cited by 3 · see citing works
The mediator effectiveness hypothesis revisited
Pyc, M. A., & Rawson, K. A. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0424-2 · Cited by 3 · see citing works
Self-citationInvestigating the replicability and generalizability of the negative testing effect.
Wissman, K. T., & Peterson, D. J. (2018) · Journal of Applied Research in Memory and Cognition · doi:10.1016/j.jarmac.2018.05.002 · Cited by 3 · see citing works
Self-citationTransitive inference and the testing effect: Retrieval practice impairs transitive inference
Mulligan, N. W., Buchin, Z. L., & Powers, A. (2023) · Quarterly Journal of Experimental Psychology · doi:10.1177/17470218231156732 · Cited by 2 · see citing works
Self-citationTesting the testing effect on prolific: when retrieval practice fails to boost learning
Sigayret, K., Parmentier, J.-F., & Silvestre, F. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1727423
Greene, R. L. (2024) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190917982.013.73
- Authors
- Mulligan, N. W., Rawson, K. A., Peterson, D. J., & Wissman, K. T.
- Affiliations
- Kent State University (US)
- Skidmore College (US)
- Venue
- Journal of Experimental Psychology Learning Memory and Cognition
- Published
- 2018-05
- Volume
- 44
- Issue
- 5
- Pages
- 752-763
- ISSN
- 0278-7393 (ISSN-L)
- 1939-1285
- OpenAlex
- W2766181325
- Cited by
- 10
- References
- 50
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 1 · 2022: 1 · 2021: 1 · 2020: 1 · 2018: 3
- FWCI
- 0.63
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Experimental Psychology Learning Memory and Cognition (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Functions and Memory
- Keywords
- memory retrieval, subsequent memory, replication, cognitive ability, working memory capacity, Raven's Progressive Matrices, category clustering, free recall, restudy
- MeSH terms
- Adult, Association Learning, Cues, Humans, Pattern Recognition, Visual, Practice, Psychological, Psychology, Experimental, Reading, Mental Recall, Research Design, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
