The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index; 9 of the 35 sources its claims cite were checked for support.
Testing the testing effect on prolific: when retrieval practice fails to boost learning
Open accessAbstract
The testing effect-whereby retrieving information through testing improves long-term retention more than restudying-is one of the most robust and well-documented phenomena in educational psychology. However, its replicability in online crowdsourced settings remains uncertain. In this paper, we report two preliminary experiments examining whether the testing effect can be detected in studies conducted on Prolific, a platform widely used for behavioral and educational research. Although not conceived as direct replications, both studies relied on validated learning materials and incorporated key methodological features known to enhance the testing effect, including delayed posttests, corrective feedback, and both factual and application-based outcome measures. In both experiments, participants were randomly assigned to a retrieval-practice (test) or restudy condition. No significant differences were found between groups at delayed test, despite multiple safeguards aimed at promoting participant engagement (e.g., prescreening filters, attention checks, fair compensation). These null findings suggest that investigating learning phenomena that require sustained cognitive engagement may be more challenging with crowdsource samples. We argue that the absence of the testing effect in our studies is unlikely to reflect a theoretical limitation of the effect itself, but rather highlights challenges inherent to online research environments. While platforms like Prolific offer valuable advantages in terms of speed and sample diversity, their constraints should be carefully considered when studying learning effects that depend on deep engagement with learning material.
Overview
CoChat Summary
## TL;DR
Two Prolific-based experiments found no significant testing effect, despite using delayed posttests, corrective feedback, and materials previously known to produce retrieval-practice benefits. The authors argue that crowdsource settings may make it harder to elicit learning effects that depend on sustained engagement.
## Key findings
- Experiment 1: no significant effect of learning condition on total post-test score, F(1, 66) = 0.13, p = 0.72; also null effects for Definition score [F(1, 66) = 0.94, p = 0.33] and Application score [F(1, 66) = 0.27, p = 0.61].
- Experiment 2: no significant difference between control and experimental groups on delayed recall, U = 1773, p = 0.17, r = 0.14; means were M = 6.51 (SD = 5.74) for control and M = 7.44 (SD = 4.65) for experimental.
- Attrition was high in both studies: Experiment 1 had 67/169 (39.6%) complete both phases; Experiment 2 had 129/426 (30.3%) complete the entire experiment.
- In Experiment 2, 56% of participants failed to comply with the 3-minute recall instruction in the screening phase, indicating engagement problems.
## Methods
- Two experiments were run on Prolific using the Gorilla Experiment Builder platform; both randomly assigned participants to a retrieval-practice/testing condition or a restudy/control condition.
- Experiment 1 used two ~500-word texts (Expressionism and Ancient Rome) with 10 key terms each, plus delayed multiple-choice posttests after 10 weeks; 67 participants completed both phases.
- Experiment 2 used a 2 min 26 s narrated lightning-formation video with 19 key ideas, a preliminary screening phase for prior meteorological knowledge and recall compliance, and a 2-day delayed free-recall test; 129 participants completed the full study.
- Analysis: linear mixed-effects models for Experiment 1 and Mann–Whitney U test for Experiment 2; scoring in Experiment 2 was done by a trained blind rater.
## Limitations
- The paper states that many inclusion criteria relied on self-report on Prolific, which may be inaccurate or strategically manipulated.
- The Experiment 2 sample was not representative of the undergraduate population typically used in testing-effect research, limiting generalizability.
- Both studies had high attrition, which reduced power and raised self-selection concerns; this limitation is explicitly stated.
- In Experiment 2, open-ended recall scoring involved subjective judgment, introducing potential unreliability; this is stated in the paper.
- Evident from the text: the studies are described as preliminary and not direct replications, so the null findings are evidence about these specific online conditions rather than a definitive test of the testing effect.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
7 Valid · 0 Not valid · 15 Not checked · 51 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Prolific and CloudResearch users consistently outperform Amazon Mechanical Turk (MTurk), Qualtrics, and SONA users on key data quality indicators (Douglas et al., 2023).”
§ Testing the testing effect on prolific: when retrieval practice fails to boost learning > Introduction Crowdsourcing platforms such as Amazon Mechanical Turk and prolific are increasingly used in psychological and educational research due to their accessibility, cost-efficiency, and participant diversity (Paolacci et al., 2010; Keith et al., 2017). However, concerns remain regarding data quality—particularly in educational psychology, where sustained cognitive engagement is essential (Hauser et al., 2019; Cheung et al., 2017). One of the most well-established effects in educational psychology is the testing effect, whereby retrieving information through testing improves long-term retention more than restudying (Roediger and Karpicke, 2006; Rowland, 2014). Because of its robustness, the testing effect provides a useful case for examining whether established learning effects can be detected in online, crowdsourced samples under controlled conditions. In this paper, we investigate whether the testing effect could be detected in two experiments conducted on the prolific platform using established materials and procedures. We report two preliminary studies conducted as part of a broader project, which were not intended as direct replications but rather as preparatory work for more complex investigations. Both experiments employed validated instructional materials widely used in educational psychology (e.g., Mayer and Chandler, 2001; Pan et al., 2023). These studies preserved the core materials and retrieval principles, but differed from prior laboratory implementations in participant population and procedural constraints associated with online data collection. Our results provide insights into the conditions under which the testing effect may or may not be observed in the present crowdsourced samples. Testing is not only a means of assessment but also a powerful learning event. Theoretical accounts highlight that retrieval practice activates mechanisms that strengthen memory traces and improve access to learned content over time (Roediger et al., 2009). Among these mechanisms, the retrieval effort hypothesis (Yang et al., 2021) posits that the cognitive effort required to retrieve information enhances encoding. The transfer-appropriate processing (Yang et al., 2021) framework further suggests that when the cognitive processes involved in initial encoding align with those required during retrieval, performance at retrieval improves. Finally, motivational and metacognitive perspectives emphasize that testing can foster engagement and self-monitoring, encouraging learners to identify knowledge gaps and consolidate understanding (Yang et al., 2021). Testing has also been shown to support better organization of knowledge, reduce interference from previously learned material, and provide meaningful feedback to students (Roediger et al., 2011). Although early explanations emphasized mere re-exposure to the material, research has shown that testing yields benefits beyond those provided by additional study opportunities (Agarwal et al., 2008). Corrective feedback further enhances these effects (Roediger et al., 2009) and the absence of feedback can sometimes be detrimental, increasing the risk of false recognitions in later tests—especially with multiple-choice formats (Roediger et al., 2009; Kang et al., 2007). A series of meta-analyses have established the reliability and generalizability of the testing effect. Effect sizes range from moderate to large, with estimates between d = 0.55 and d = 0.88 (Phelps, 2012), g = 0.50 (Rowland, 2014), and g = 0.61 to g = 0.70 (Adesope et al., 2017). The effect tends to be stronger in higher education contexts and when corrective feedback or performance incentives are included (Phelps, 2012; Pan and Rickard, 2018). Yang et al. (2021), in a large-scale meta-analysis of studies conducted in real classrooms, confirmed that the effect is observable across educational levels and learning outcomes—ranging from factual knowledge to conceptual understanding and even problem-solving. While the strongest effects are observed on retention, positive though smaller effects have also been reported on transfer tasks (Pan and Rickard, 2018), with elaborated retrieval practice and congruence between training and final test formats acting as key moderators. The testing effect has been demonstrated across a wide range of retrieval formats, including multiple-choice questions, cued recall, and free recall tasks (Adesope et al., 2017; Rowland, 2014). The critical mechanism is the act of retrieval itself rather than the specific test format. Across studies, the testing effect remains one of the most replicated and impactful findings in cognitive and educational psychology. Importantly, the testing effect is not invariant across learners, tasks, or contexts. Prior work indicates that its magnitude depends on factors such as learner characteristics, task demands, and the degree of effortful engagement during retrieval. For instance, retrieval practice is most effective when learners engage in effortful and successful retrieval, and when motivational and self-regulatory factors support sustained engagement (Butler et al., 2017; Schwerter et al., 2025). Moreover, evidence from laboratory studies suggests that testing effects—including both positive and negative variants—may fail to generalize across participant populations, even when procedures are held constant (Mulligan et al., 2018). These findings highlight that the replicability and generalizability of retrieval-based learning effects depend not only on materials and procedures, but also on who the learners are and how they engage with the task. Nevertheless, given the robust evidence for the testing effect in controlled settings, an important open question is whether this effect can be reliably detected when studies are conducted using crowdsourced participant platforms. Recent evidence further suggests that retrieval practice may fail to confer learning benefits when task demands and cognitive load are high, even in controlled laboratory settings. For example, Redifer et al. (2026) reported no testing advantage when undergraduate students studied a lengthy and conceptually demanding academic text, with higher cognitive load mediating the relationship between retrieval practice and final test performance, such that increased cognitive load was associated with poorer delayed recall. Despite the increasing popularity of crowdsourcing platforms in behavioral and educational research, concerns persist regarding the reliability and validity of data collected through such means. Online participants are often self-selected and may differ from target populations in key demographic or cognitive characteristics (Keith et al., 2017; Cheung et al., 2017). Specific issues include inattentive responding, repeated exposure to similar experimental paradigms leading to reduced naivety, deceptive behavior, and environmental distractions (Hauser et al., 2019). Workers may misrepresent themselves to qualify for studies or multitask during participation, which can compromise data quality (Cheung et al., 2017). The presence of “professional participants” or “Super Turkers” also raises concerns about sample independence and familiarity with standard experimental manipulations (Keith et al., 2017). To address these limitations, researchers have developed a range of methodological safeguards. These include the use of prescreening filters and custom qualifications, high approval thresholds (typically ≥95%), attention and comprehension checks, indirect eligibility screening, and metadata verification (Paolacci et al., 2010; Hauser et al., 2019). Fair compensation has been shown to reduce attrition and improve engagement (Keith et al., 2017; Albert and Smilek, 2023) and transparent reporting of recruitment procedures and exclusion criteria remains essential (Cheung et al., 2017; Hauser et al., 2019). Comparative studies have revealed meaningful differences across platforms. Prolific and CloudResearch users consistently outperform Amazon Mechanical Turk (MTurk), Qualtrics, and SONA users on key data quality indicators (Douglas et al., 2023). Prolific participants show higher attentiveness, longer and more appropriate completion times, fewer signs of disengagement, and greater response consistency (Albert and Smilek, 2023; Douglas et al., 2023). They also outperform MTurk participants in educational problem-solving tasks that require sustained effort and deliberate engagement (Wang et al., 2024). While MTurk workers are generally more experienced, they are also more prone to multitasking and careless responding, especially in cognitively demanding tasks (Cheung et al., 2017; Albert and Smilek, 2023). Combined with superior prescreening tools, transparent demographic targeting, and cost-efficiency (Douglas et al., 2023), these strengths make prolific particularly well-suited for research in educational psychology. To examine whether the testing effect can reliably be observed in crowdsourced samples, we conducted two experimental studies using the prolific platform. Although the two experiments differed in learning materials and procedures, both were designed to assess the potential of prolific for supporting robust educational effects. In the following sections, we present the methods and results of each experiment, followed by a general discussion of their implications for educational research using crowdsourced samples. Despite extensive evidence for the testing effect in laboratory and classroom settings, comparatively little is known about the conditions under which this effect can be detected in crowdsourced samples, particularly in studies requiring sustained engagement with learning materials. The present research addresses this gap by using the testing effect as a theoretically well-established benchmark to probe the limits and affordances of online participant platforms in educational psychology. Theoretically, these studies contribute to ongoing discussions about the role of engagement and task demands in enabling retrieval-based learning. Methodologically, they provide an empirical assessment of whether commonly recommended data-quality safeguards are sufficient to support robust learning effects in crowdsourced environments.
What the sources say
- Supportedcites Douglas et al., 2023 source
The Results section states that, in general, Prolific and CloudResearch participants provided higher-quality data than participants on MTurk, Qualtrics, or SONA across the study’s data-quality indicators. The paper’s abstract likewise reports better performance on attention checks, meaningful responses, instruction-following, and other measures. (LLM, conf=0.91)
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“No significant differences were found between groups at delayed test, despite multiple safeguards aimed at promoting participant engagement (e.g., prescreening filters, attention checks, fair compensation).”
§ Testing the testing effect on prolific: when retrieval practice fails to boost learning > Abstract The testing effect—whereby retrieving information through testing improves long-term retention more than restudying—is one of the most robust and well-documented phenomena in educational psychology. However, its replicability in online crowdsourced settings remains uncertain. In this paper, we report two preliminary experiments examining whether the testing effect can be detected in studies conducted on Prolific, a platform widely used for behavioral and educational research. Although not conceived as direct replications, both studies relied on validated learning materials and incorporated key methodological features known to enhance the testing effect, including delayed posttests, corrective feedback, and both factual and application-based outcome measures. In both experiments, participants were randomly assigned to a retrieval-practice (test) or restudy condition. No significant differences were found between groups at delayed test, despite multiple safeguards aimed at promoting participant engagement (e.g., prescreening filters, attention checks, fair compensation). These null findings suggest that investigating learning phenomena that require sustained cognitive engagement may be more challenging with crowdsource samples. We argue that the absence of the testing effect in our studies is unlikely to reflect a theoretical limitation of the effect itself, but rather highlights challenges inherent to online research environments. While platforms like Prolific offer valuable advantages in terms of speed and sample diversity, their constraints should be carefully considered when studying learning effects that depend on deep engagement with learning material.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Training performance did not significantly differ between the Expressionism text (M = 6.34, SD = 2.30) and the Ancient Rome text (M = 6.58, SD = 2.45), t(65) = 0.41, p = 0.68.”
§ Testing the testing effect on prolific: when retrieval practice fails to boost learning > Results > Experiment 1
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Results showed no significant difference between the groups (U = 1773; p = 0.17; r = 0.14)3.”
§ Testing the testing effect on prolific: when retrieval practice fails to boost learning > Results > Experiment 2
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the 120 remaining participants, only 67 (39.6%) completed both phases, resulting in substantial attrition.”
§ Testing the testing effect on prolific: when retrieval practice fails to boost learning > Results > Experiment 1
plainn = 120- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
26 more claims
Create a free account to see all 73 claims and every source verdict.
- Probably not a substantial claim. 36
- It may state two separate claims as one. 19
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Consistentt(65) = 0.41reported p = .68 · recomputed p = .683
“t(65) = 0.41, p = .68”
- ConsistentF(1, 66) = 0.13reported p = .72 · recomputed p = .720
“F(1, 66) = 0.13, p = .72”
- ConsistentF(1, 66) = 0.94reported p = .33 · recomputed p = .336
“F(1, 66) = 0.94, p = .33”
- ConsistentF(1, 66) = 0.27reported p = .61 · recomputed p = .605
“F(1, 66) = 0.27, p = .61”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Self-check results appear here once an independent check has confirmed them.
Citations
Cites
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 3×
“Crowdsourcing platforms such as Amazon Mechanical Turk and prolific are increasingly used in psychological and educational research due to their accessibility, cost-efficiency, and participant diversity (Paolacci et al., 2010; Keith et al., 2017).”
InfluentialTest-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention
Roediger, H. L., & Karpicke, J. D. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01693.x · Cited by 2,867
Cognitive Theory of Multimedia Learning
Mayer, R. E. (2014) · The Cambridge Handbook of Multimedia Learning · doi:10.1017/cbo9781139547369.005 · Cited by 1,976
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023) · PLOS ONE · doi:10.1371/journal.pone.0279720 · Cited by 1,532
The effect of testing versus restudy on retention: A meta-analytic review of the testing effect.
Rowland, C. A. (2014) · Psychological Bulletin · doi:10.1037/a0037559 · Cited by 1,066 · cited 5×
“Effect sizes range from moderate to large, with estimates between d = 0.55 and d = 0.88 (Phelps, 2012), g = 0.50 (Rowland, 2014), and g = 0.61 to g = 0.70 (Adesope et al., 2017).”
InfluentialAn MTurk Crisis? Shifts in Data Quality and the Impact on Study Results
Chmielewski, M., & Kucker, S. C. (2020) · Social Psychological and Personality Science · doi:10.1177/1948550619875149 · Cited by 1,027 · cited 1×
“Participants may join studies indiscriminately and complete multiple tasks in rapid succession, which can reduce task-specific investment and increase cognitive fatigue or multitasking (Chmielewski and Kucker, 2020).”
Relationships between student engagement and academic achievement: A meta-analysis
Lei, H., Cui, Y., & Zhou, W. (2018) · Social Behavior and Personality: an international journal · doi:10.2224/sbp.7054 · Cited by 987 · cited 1×
“Learning studies often require participants to make an effortful investment in selecting, organizing, and integrating information (Mayer, 2014; Lei et al., 2018), processes that are sensitive to attentional fluctuations and motivational states (Seli et al., 2016).”
Mayer, R. E., & Chandler, P. (2001) · Journal of Educational Psychology · doi:10.1037/0022-0663.93.2.390 · Cited by 762
Rethinking the Use of Tests: A Meta-Analysis of Practice Testing
Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017) · Review of Educational Research · doi:10.3102/0034654316689306 · Cited by 544
Test format and corrective feedback modify the effect of testing on long-term retention
Kang, S. H. K., McDermott, K. B., & Roediger, H. L. (2007) · European Journal of Cognitive Psychology · doi:10.1080/09541440601056620 · Cited by 513 · cited 1×
“Corrective feedback further enhances these effects (Roediger et al., 2009) and the absence of feedback can sometimes be detrimental, increasing the risk of false recognitions in later tests—especially with multiple-choice formats (Roediger et al., 2009; Kang et al., 2007).”
Metadata
- Authors
- Sigayret, K., Parmentier, J.-F., & Silvestre, F.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- Corresponding author
- Sigayret, Kevin
- Venue
- Frontiers in Psychology
- Publisher
- Frontiers Media SA
- Published
- 2026-01-29
- Volume
- 17
- Pages
- 1727423-1727423
- ISSN
- 1664-1078 (ISSN-L)
- PMC
- PMC12894256
- OpenAlex
- W7125926383
- Cited by
- 0
- References
- 28
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Psychology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- HAL (submittedVersion, cc-by)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Keywords
- testing effect, retrieval practice, restudying, long-term retention, Prolific, crowdsourced research, online research platform, negative results, corrective feedback, participant engagement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Testing the testing effect on prolific: when retrieval practice fails to boost learning. https://app.cochat.ai/papers/10.3389/fpsyg.2026.1727423
- The paper
- Sigayret, K., Parmentier, J.-F., & Silvestre, F. (2026). Testing the testing effect on prolific: when retrieval practice fails to boost learning. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2026.1727423
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

