CoChat

When Problem Solving Followed by Instruction Works: Evidence for Productive Failure

Open access
Sinha, Tanmay, Kapur, Manu · 2021 · Review of Educational Research

The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index; 8 of the 35 sources its claims cite were checked for support.

Abstract

When learning a new concept, should students engage in problem solving followed by instruction (PS-I) or instruction followed by problem solving (I-PS)? Noting that there is a passionate debate about the design of initial learning, we report evidence from a meta-analysis of 53 studies with 166 comparisons that compared PS-I with I-PS design. Our results showed a significant, moderate effect in favor of PS-I (Hedge’s g 0.36 [95% confidence interval 0.20; 0.51]). The effects were even stronger (Hedge’s g ranging between 0.37 and 0.58) when PS-I was implemented with high fidelity to the principles of Productive Failure (PF), a subset variant of PS-I design. Students’ grade level, intervention time span, and its (quasi-)experimental nature contributed to the efficacy of PS-I over I-PS designs. Contrasting trends were, however, observed for younger age learners (second to fifth graders) and for the learning of domain-general skills, for which effect sizes favored I-PS. Overall, an estimation of true effect sizes after accounting for publication bias suggested a strong effect size favoring PS-I (Hedge’s g 0.87).

Overview

CoChat Summary

TL;DR

This meta-analysis of 53 studies with 166 comparisons found that problem solving followed by instruction (PS-I) produced a significant, moderate advantage over instruction followed by problem solving (I-PS) for conceptual knowledge and transfer (Hedge’s g = 0.36). Effects were stronger when PS-I was implemented with high fidelity to Productive Failure principles, and after accounting for publication bias the estimated true effect was Hedge’s g = 0.87.

Key findings

  • Overall, PS-I outperformed I-PS for conceptual knowledge and transfer: Hedge’s g = 0.36, 95% CI [0.20, 0.51], p < .0001. For procedural knowledge, the effect was nonsignificant: Hedge’s g = −0.03, 95% CI [−0.20, 0.15].
  • Higher fidelity to Productive Failure (PF) design criteria was associated with larger effects favoring PS-I; the overall PF fidelity score significantly predicted effect size (β = 0.0065, p < .001, 95% CI [0.0044, 0.0087]). Effects ranged between Hedge’s g = 0.37 and 0.58 when PS-I was implemented with high PF fidelity.
  • Subgroup differences: younger learners (2nd–5th graders) showed an effect favoring I-PS (Hedge’s g = −0.09, 95% CI [−0.92, −0.16]), and domain-general skills also favored I-PS (Hedge’s g = −0.17, 95% CI [−1.11, −0.02]). Quasi-experimental comparisons (g = 0.46) and interventions spanning a few days (g = 0.41) descriptively favored PS-I more than experimental comparisons and shorter interventions, though these subgroup differences were not significant.
  • Publication-bias checks: funnel plot and Egger’s test were not significant (intercept = 0.63, p = .219), but p-curve analysis indicated evidential value and estimated a true effect size of Hedge’s g = 0.87 favoring PS-I; heterogeneity was moderate (I² = 42.01%).

Methods

  • Meta-analysis of 53 studies comprising 166 experimental comparisons that compared PS-I with I-PS; inclusion required experimental or quasi-experimental comparison and assessment of conceptual knowledge or transfer, with procedural knowledge optional.
  • Searches used Google Scholar citations to seminal PF articles and key follow-up PF articles; 45 articles met criteria after screening and duplicate removal.
  • Pooled effect sizes used a three-level multilevel meta-analysis model (participants nested within studies, studies nested within articles), Hedge’s g with 95% CIs, Cochran’s Q, and I². Subgroup and meta-regression analyses examined seven PF fidelity criteria, student characteristics, and intervention characteristics; publication bias was assessed with funnel plots, Egger’s test, trim-and-fill, and p-curve analysis.

Limitations

  • The paper states that nearly 75% of included comparisons targeted math and physics, with skewed subtopics; more studies are needed in other STEM and non-STEM domains and with postgraduates/professionals before generalizing.
  • The paper states that individual differences were not available, few studies reported actual failure rates, multiple RSM generation is a weak proxy for failure, and only one study explicitly assessed negative knowledge. It also notes that PF fidelity criteria and other characteristics were dichotomized, and that experimental versus quasi-experimental designs may not be directly comparable because learning objectives can differ.
  • The paper states that the identified list of influencing factors is important but not exhaustive, and that the current meta-analysis lacks studies involving students with learning disabilities or difficulties.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3102/00346543211019105

Cites

80 of 80 references verified0 citation errors · 0 not checkable

Statistics

Figures

Image 1
CC-BY · From the PDF
Figure 1 Venn diagram illustrating the hierarchy of PF, PS-I, and PFL learning designs. Here, we depict one category of preparation for future learning (PFL) designs where sensemaking experiences precede instruction. However, more broadly, PFL can be conceived as any experience that prepares students to learn in the future-that learning could occur not just through explicit instruction but also through exploration, practice, and so on. PF = productive failure; PS-I = problem solving followed by instruction.
Figure 1 Venn diagram illustrating the hierarchy of PF, PS-I, and PFL learning designs. Here, we depict one category of preparation for future learning (PFL) designs where sensemaking experiences precede instruction. However, more broadly, PFL can be conceived as any experience that prepares students to learn in the future-that learning could occur not just through explicit instruction but also through exploration, practice, and so on. PF = productive failure; PS-I = problem solving followed by instruction.CC-BY · From the PDF
Figure 2 PRISMA flowchart summarizing the overall process for selecting studies for inclusion.
Figure 2 PRISMA flowchart summarizing the overall process for selecting studies for inclusion.CC-BY · From the PDF
Figure 3 Funnel plot (left) and p-curve (right) to assess publication bias for n = 166 PS-I and I-PS comparisons are depicted. Funnel plots are colored by the significance level into which the effect size of each of the comparisons fall. Bubbles in the plot depict included comparisons. The Y-axis shows the standard error (SE) of each comparison, with larger comparisons (which thus have a smaller SE) plotted on top of the Y-axis. The X-axis shows the standardized effect size of each comparison. Striped vertical line corresponds to the pooled effect size estimate. Color coding-dark blue (dark gray): p < .05, blue (gray): p < .025, light blue (light gray): p < .01. The p-curve is shown on the right with a blue/solid line. The Y-axis shows the percentage of comparisons. The X-axis shows the p-values.
Figure 3 Funnel plot (left) and p-curve (right) to assess publication bias for n = 166 PS-I and I-PS comparisons are depicted. Funnel plots are colored by the significance level into which the effect size of each of the comparisons fall. Bubbles in the plot depict included comparisons. The Y-axis shows the standard error (SE) of each comparison, with larger comparisons (which thus have a smaller SE) plotted on top of the Y-axis. The X-axis shows the standardized effect size of each comparison. Striped vertical line corresponds to the pooled effect size estimate. Color coding-dark blue (dark gray): p < .05, blue (gray): p < .025, light blue (light gray): p < .01. The p-curve is shown on the right with a blue/solid line. The Y-axis shows the percentage of comparisons. The X-axis shows the p-values.CC-BY · From the PDF
Figure 4 Conjecture map between PF fidelity features and purported learning mechanisms.
Figure 4 Conjecture map between PF fidelity features and purported learning mechanisms.CC-BY · From the PDF

Verification

Claims the paper makes

7 Valid · 4 Not valid · 21 Not checked · 106 Own results and interpretations

75 more claims

Create a free account to see all 138 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 36
  • It may state two separate claims as one. 33
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • ConsistentF(3, 162) = 6.485reported p = .0004 · recomputed p < .001

    “F (3, 162) = 6.485, p = .0004”

  • Consistentχ²(1) = 33.15reported p < .001 · recomputed p < .001

    “χ 2 (1) = 33.15, p < .001”

2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 7 · the rest aren’t loaded

2 more findings

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Show all 80 references

Research works printed but not linked: 4

  1. Fukaya T., Uesaka Y., Ichikawa S. (2019). Investigating the effects of thinking after instruction approach. Educational Technology Research, 41(1), 1–11. https://doi.org/10.15077/etr.42105 · doi:10.15077/etr.42105
  2. Hall M. A. (1999). Correlation-based feature selection for machine learning [Doctoral dissertation, The University of Waikato Hamilton]. https://www.cs.waikato.ac.nz/~mhall/thesis.pdf
  3. Mazziotti C., 2015, Exploring the material conditions of learning: The Computer Supported Collaborative Learning (CSCL) Conference 2015, 2, 570
  4. Sears D. A. (2006). Effects of innovation versus efficiency tasks on collaboration and learning [Doctoral dissertation, Stanford University]. https://iase-web.org/documents/dissertations/06.Sears.pdf

Cited by

See all 212

Metadata

Authors
Sinha, T., & Kapur, M.
Affiliations
ETH Zurich (CH)
Venue
Review of Educational Research
Published
2021-06-18
Volume
91
Issue
5
Pages
761-798
ISSN
  • 0034-6543 (ISSN-L)
  • 1935-1046
OpenAlex
W3174759423
Cited by
212
References
77
Citations by year
2026: 49 · 2025: 57 · 2024: 30 · 2023: 44 · 2022: 29 · 2021: 3
FWCI
25.61
Type
journal article
Language
en
Open access
Locations
  • Review of Educational Research (publishedVersion, cc-by)
  • Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by)
  • ETH Zürich Research Collection (cc-by)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
  • Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
  • Social Sciences › Education › Science Education and Pedagogy
Keywords
productive failure, conceptual learning, publication bias, grade level effects, domain-general skills

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: When Problem Solving Followed by Instruction Works: Evidence for Productive Failure. https://app.cochat.ai/papers/10.3102/00346543211019105
The paper
Sinha, T., & Kapur, M. (2021). When Problem Solving Followed by Instruction Works: Evidence for Productive Failure. Review of Educational Research. https://doi.org/10.3102/00346543211019105

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.