Parsons Problems and Beyond: Systematic Literature Review and Empirical Study Designs
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Programming is a complex task that requires the development of many skills including knowledge of syntax, problem decomposition, algorithm development, and debugging. Code-writing activities are commonly used to help students develop these skills, but the difficulty of writing code from a blank page can overwhelm many novices. Parsons problems offer a simpler alternative to writing code by providing scrambled code blocks that must be placed in the correct order to solve a problem. In the 16 years since their introduction to the computing education community, an expansive body of literature has emerged that documents a range of tools, novel problem variations and makes numerous claims of benefits to learners. In this work, we track the origins of Parsons problems, outline their defining characteristics, and conduct a comprehensive review of the literature to document the evidence of benefits to learners and to identify gaps that require exploration. To facilitate future work, we design empirical studies and develop associated resources that are ready for deployment at a large scale. Collectively, this review and the provided experimental resources will serve as a focal point for researchers interested in advancing our understanding of Parsons problems and their benefits to learners.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2006.07.009 · Cited by 2,394
Efficiency in learning: Evidence-based guidelines to manage cognitive load
Clark, R. C., Nguyen, F., Sweller, J., & Baddeley, M. (2006) · Performance Improvement · doi:10.1002/pfi.4930450920 · Cited by 820
Effects of schema acquisition and rule automation on mathematical problem-solving transfer.
Cooper, G., & Sweller, J. (1987) · Journal of Educational Psychology · doi:10.1037/0022-0663.79.4.347 · Cited by 688
Structuring Effective Worked Examples
Ward, M., & Sweller, J. (1990) · Cognition and Instruction · doi:10.1207/s1532690xci0701_1 · Cited by 396
Relationships between reading, tracing and writing skills in introductory programming
Lopez, M., Whalley, J., Robbins, P., & Lister, R. (2008) · Proceedings of the Fourth international Workshop on Computing Education Research · doi:10.1145/1404520.1404531 · Cited by 363
Strategies for Computer-Based Programming Instruction: Program Completion vs. Program Generation
Van Merriënboer, J. J. G., & De Croock, M. B. M. (1992) · Journal of Educational Computing Research · doi:10.2190/mjdx-9pp4-kfmt-09pm · Cited by 207
Becker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199
Self-citationRecall of random and distorted chess positions: Implications for the theory of expertise
Gobet, F., & Simon, H. A. (1996) · Memory & Cognition · doi:10.3758/bf03200937 · Cited by 176 · cited 1×
“This, in turn, affects their ability to identify patterns and solve problems [89].”
BackgroundSubgoals, Context, and Worked Examples in Learning Computing Problem Solving
Morrison, B. B., Margulieux, L. E., & Guzdial, M. (2015) · Proceedings of the eleventh annual International Conference on International Computing Education Research · doi:10.1145/2787622.2787733 · Cited by 174 · cited 1×
“Research has been conducted on worked examples in math [10, 37, 210, 214, 236], physics [192, 194, 223] and computer programming [166, 179, 235].”
MethodSelf-citationExperiencing programming assignments in CS1: the emotional toll
Kinnunen, P., & Simon, B. (2010) · Proceedings of the Sixth international workshop on Computing education research · doi:10.1145/1839594.1839609 · Cited by 156 · cited 3×
“Many students struggle while writing textual code from scratch [124].”
BackgroundInfluential
Cited by
Prompt Problems: A New Programming Exercise for the Generative AI Era
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630909 · Cited by 187
Self-citationReeves, B., Sarsa, S., Prather, J., Denny, P., Becker, B. A., Hellas, A., Kimmel, B., Powell, G., & Leinonen, J. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588805 · Cited by 56
Self-citationMetacodenition: Scaffolding the Problem-Solving Process for Novice Programmers
Pechorina, Y., Anderson, K., & Denny, P. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576130 · Cited by 33
Self-citationGenerative AI in Computer Science Education: Challenges and Opportunities
Franklin, D., Denny, P., Gonzalez-Maldonado, D. A., & Tran, M. (2025) · doi:10.1017/9781009581738 · Cited by 20
Self-citationNeurodiversity in Computing Education Research: A Systematic Literature Review
Zastudil, C., Smith, D. H., Tohamy, Y., Nasimova, R., Montross, G., & MacNeil, S. (2025) · Proceedings of the 30th ACM Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3724363.3729088 · Cited by 15
Metadata
- Authors
- Ericson, B. J., Denny, P., Prather, J., Duran, R., Hellas, A., Leinonen, J., Miller, C. S., Morrison, B. B., Pearce, J. L., & Rodger, S. H.
- Venue
- Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education
- Publisher
- ACM
- Published
- 2022-12-27
- Pages
- 191-234
- OpenAlex
- W4313331626
- Cited by
- 56
- References
- 26
- FWCI
- 15.44
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Parsons Problems and Beyond: Systematic Literature Review and Empirical Study Designs. https://app.cochat.ai/papers/10.1145/3571785.3574127
- The paper
- Ericson, B. J., Denny, P., Prather, J., Duran, R., Hellas, A., Leinonen, J., Miller, C. S., Morrison, B. B., Pearce, J. L., & Rodger, S. H. (2022). Parsons Problems and Beyond: Systematic Literature Review and Empirical Study Designs. Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education. https://doi.org/10.1145/3571785.3574127
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.