Parsons Problems and Beyond: Systematic Literature Review and Empirical Study Designs
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Programming is a complex task that requires the development of many skills including knowledge of syntax, problem decomposition, algorithm development, and debugging. Code-writing activities are commonly used to help students develop these skills, but the difficulty of writing code from a blank page can overwhelm many novices. Parsons problems offer a simpler alternative to writing code by providing scrambled code blocks that must be placed in the correct order to solve a problem. In the 16 years since their introduction to the computing education community, an expansive body of literature has emerged that documents a range of tools, novel problem variations and makes numerous claims of benefits to learners. In this work, we track the origins of Parsons problems, outline their defining characteristics, and conduct a comprehensive review of the literature to document the evidence of benefits to learners and to identify gaps that require exploration. To facilitate future work, we design empirical studies and develop associated resources that are ready for deployment at a large scale. Collectively, this review and the provided experimental resources will serve as a focal point for researchers interested in advancing our understanding of Parsons problems and their benefits to learners.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2006.07.009 · Cited by 2,394
The Use of Worked Examples as a Substitute for Problem Solving in Learning Algebra
Sweller, J., & Cooper, G. A. (1985) · Cognition and Instruction · doi:10.1207/s1532690xci0201_3 · Cited by 1,214
Efficiency in learning: Evidence-based guidelines to manage cognitive load
Clark, R. C., Nguyen, F., Sweller, J., & Baddeley, M. (2006) · Performance Improvement · doi:10.1002/pfi.4930450920 · Cited by 820
Effects of schema acquisition and rule automation on mathematical problem-solving transfer.
Cooper, G., & Sweller, J. (1987) · Journal of Educational Psychology · doi:10.1037/0022-0663.79.4.347 · Cited by 688
Cognitive Load Theory: New Conceptualizations, Specifications, and Integrated Research Perspectives
Paas, F., van Gog, T., & Sweller, J. (2010) · Educational Psychology Review · doi:10.1007/s10648-010-9133-8 · Cited by 442
Structuring Effective Worked Examples
Ward, M., & Sweller, J. (1990) · Cognition and Instruction · doi:10.1207/s1532690xci0701_1 · Cited by 396
Relationships between reading, tracing and writing skills in introductory programming
Lopez, M., Whalley, J., Robbins, P., & Lister, R. (2008) · Proceedings of the Fourth international Workshop on Computing Education Research · doi:10.1145/1404520.1404531 · Cited by 363
Strategies for Computer-Based Programming Instruction: Program Completion vs. Program Generation
Van Merriënboer, J. J. G., & De Croock, M. B. M. (1992) · Journal of Educational Computing Research · doi:10.2190/mjdx-9pp4-kfmt-09pm · Cited by 207
Becker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199
Self-citationRecall of random and distorted chess positions: Implications for the theory of expertise
Gobet, F., & Simon, H. A. (1996) · Memory & Cognition · doi:10.3758/bf03200937 · Cited by 176 · cited 1×
“This, in turn, affects their ability to identify patterns and solve problems [89].”
BackgroundSubgoals, Context, and Worked Examples in Learning Computing Problem Solving
Morrison, B. B., Margulieux, L. E., & Guzdial, M. (2015) · Proceedings of the eleventh annual International Conference on International Computing Education Research · doi:10.1145/2787622.2787733 · Cited by 174 · cited 1×
“Research has been conducted on worked examples in math [10, 37, 210, 214, 236], physics [192, 194, 223] and computer programming [166, 179, 235].”
MethodSelf-citationExperiencing programming assignments in CS1: the emotional toll
Kinnunen, P., & Simon, B. (2010) · Proceedings of the Sixth international workshop on Computing education research · doi:10.1145/1839594.1839609 · Cited by 156 · cited 3×
“Many students struggle while writing textual code from scratch [124].”
BackgroundInfluentialMetacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use
Loksa, D., Margulieux, L., Becker, B. A., Craig, M., Denny, P., Pettit, R., & Prather, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3487050 · Cited by 140 · cited 2×
“) that metacognitive skills are often underdeveloped or not present in the domain [140, 182, 184].”
BackgroundSelf-citationSolving parsons problems versus fixing and writing code
Ericson, B. J., Margulieux, L. E., & Rick, J. (2017) · Proceedings of the 17th Koli Calling International Conference on Computing Education Research · doi:10.1145/3141880.3141895 · Cited by 121
Self-citation50 Years of CS1 at SIGCSE: A Review of the Evolution of Introductory Programming Education Research
Becker, B. A., & Quille, K. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287432 · Cited by 107 · cited 1×
“Computing educators and researchers have thus spent significant effort exploring a wide range of pedagogical approaches and activities to help students develop these necessary skills [18].”
Background(2019) · doi:10.1145/3287324.3287374 · Cited by 104 · cited 1×
“Although multiple recent attempts have been made to increase metacognition with novice programmers [45, 185], only a few papers focus on the effect of Parsons problems on novice programmer metacognition [82, 83, 183].”
BackgroundSelf-citationCognitive Load Theory in Computing Education Research: A Review
Duran, R., Zavgorodniaia, A., & Sorva, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3483843 · Cited by 92 · cited 1×
“Few papers make an explicit distinction between the types of cognitive load – intrinsic or extraneous – being measured or predicted [50].”
BackgroundSelf-citationExploring the Impact of Worked Examples in a Novice Programming Environment
Zhi, R., Price, T. W., Marwan, S., Milliken, A., Barnes, T., & Chi, M. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287385 · Cited by 75
On Novices' Interaction with Compiler Error Messages: A Human Factors Approach
Prather, J., Pettit, R., McMurry, K. H., Peters, A., Homer, J., Simone, N., & Cohen, M. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106169 · Cited by 72 · cited 2×
“Writing code from scratch often takes an unpredictable amount of time and learners with no prior experience often need help and better feedback than compiler errors or incorrect results [19, 187].”
BackgroundSelf-citationEvaluating the Efficiency and Effectiveness of Adaptive Parsons Problems
Ericson, B. J., Foley, J. D., & Rick, J. (2018) · Proceedings of the 2018 ACM Conference on International Computing Education Research · doi:10.1145/3230977.3231000 · Cited by 70
Self-citationOn Designing Programming Error Messages for Novices: Readability and its Constituent Factors
Denny, P., Prather, J., Becker, B. A., Mooney, C., Homer, J., Albrecht, Z. C., & Powell, G. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445696 · Cited by 60 · cited 1×
“Recent research has attempted to close this gap by attempting to understand how to make more usable error messages for novice learners [15, 16, 44, 46, 186, 187], but this usually centers around compiler or other syntax error messages reported at compile or runtime.”
BackgroundSelf-citationBenda, K., Bruckman, A., & Guzdial, M. (2012) · ACM Transactions on Computing Education · doi:10.1145/2382564.2382567 · Cited by 53
Error Message Readability and Novice Debugging Performance
Denny, P., Prather, J., & Becker, B. A. (2020) · Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3341525.3387384 · Cited by 50
Self-citationTowards an Analysis of Program Complexity From a Cognitive Perspective
Duran, R., Sorva, J., & Leite, S. (2018) · Proceedings of the 2018 ACM Conference on International Computing Education Research · doi:10.1145/3230977.3230986 · Cited by 44 · cited 1×
“The difficulty of the material or task depends on the learner’s prior knowledge and the complexity of the task [49].”
BackgroundSelf-citationEvaluating the Effectiveness of Parsons Problems for Block-based Programming
Zhi, R., Chi, M., Barnes, T., & Price, T. W. (2019) · Proceedings of the 2019 ACM Conference on International Computing Education Research · doi:10.1145/3291279.3339419 · Cited by 42
Prather, J., Margulieux, L., Whalley, J., Denny, P., Reeves, B. N., Becker, B. A., Singh, P., Powell, G., & Bosch, N. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543970 · Cited by 31 · cited 1×
“) that metacognitive skills are often underdeveloped or not present in the domain [140, 182, 184].”
BackgroundSelf-citationRedesigning an Object-Oriented Programming Course
Kaila, E., Kurvinen, E., Lokkila, E., & Laakso, M.-J. (2016) · ACM Transactions on Computing Education · doi:10.1145/2906362 · Cited by 22
Novice Reflections During the Transition to a New Programming Language
Denny, P., Becker, B. A., Bosch, N., Prather, J., Reeves, B., & Whalley, J. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499314 · Cited by 11 · cited 1×
“It’s also possible that information on novices moving to a new programming language could be useful [42] given the stark differences between programming from scratch and using Parsons problems.”
BackgroundSelf-citation
Cited by
Prompt Problems: A New Programming Exercise for the Generative AI Era
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630909 · Cited by 187
Self-citationReeves, B., Sarsa, S., Prather, J., Denny, P., Becker, B. A., Hellas, A., Kimmel, B., Powell, G., & Leinonen, J. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588805 · Cited by 56
“Since then, Parsons problems have received significant attention as an evidencebased pedagogical learning tool [7, 9].”
BackgroundInfluentialSelf-citationMetacodenition: Scaffolding the Problem-Solving Process for Novice Programmers
Pechorina, Y., Anderson, K., & Denny, P. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576130 · Cited by 33
Self-citationGenerative AI in Computer Science Education: Challenges and Opportunities
Franklin, D., Denny, P., Gonzalez-Maldonado, D. A., & Tran, M. (2025) · doi:10.1017/9781009581738 · Cited by 20
Self-citationNeurodiversity in Computing Education Research: A Systematic Literature Review
Zastudil, C., Smith, D. H., Tohamy, Y., Nasimova, R., Montross, G., & MacNeil, S. (2025) · Proceedings of the 30th ACM Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3724363.3729088 · Cited by 15
Metadata
- Authors
- Ericson, B. J., Denny, P., Prather, J., Duran, R., Hellas, A., Leinonen, J., Miller, C. S., Morrison, B. B., Pearce, J. L., & Rodger, S. H.
- Venue
- Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education
- Publisher
- ACM
- Published
- 2022-12-27
- Pages
- 191-234
- OpenAlex
- W4313331626
- Cited by
- 56
- References
- 28
- FWCI
- 15.44
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.