Solving parsons problems versus fixing and writing code
The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
Prior research has shown that Parsons problems are an engaging type of code completion problem that can be used to teach syntactic and semantic language constructs. They can also be used in summative assessments to reduce marking time and grading variability compared to code writing problems. In a Parsons problem the correct code is provided, but is broken into mixed-up code blocks that must be assembled in the correct order. Two-dimensional Parsons problems also require the code blocks to be indented correctly. Parsons problems can contain extra code blocks, called distractors, which are not needed in a correct solution. We present a study that compared the efficiency, effectiveness, and cognitive load of learning from solving two-dimensional Parsons problems with distractors, versus fixing code with the same errors as the distractors, versus writing the equivalent code. We found that solving two-dimensional Parsons problems with distractors took significantly less time than fixing code with errors or than writing the equivalent code. Additionally, there was no statistically significant difference in the learning performance, or in student retention of the knowledge one week later.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Recently, Ericson et al. found that interleaving WEs with Parsons problems can improve programming learning efficiency compared to WEs paired with writing code or fixing code [3].”
From Exploring the Impact of Worked Examples in a Novice Programming Environment (2019) · cites this paper as [3]
Citations
Cites
Cognitive Load During Problem Solving: Effects on Learning
Sweller, J. (1988) · Cognitive Science · doi:10.1016/0364-0213(88)90023-7 · Cited by 11,048
Cognitive Load Theory and Instructional Design: Recent Developments
Paas, F., Renkl, A., & Sweller, J. (2003) · Educational Psychologist · doi:10.1207/s15326985ep3801_1 · Cited by 3,041
(2000) · Cited by 2,779
Ericsson, K. A. (2006) · The Cambridge Handbook of Expertise and Expert Performance · doi:10.1017/cbo9780511816796.038 · Cited by 1,629
Active‐Constructive‐Interactive: A Conceptual Framework for Differentiating Learning Activities
Chi, M. T. H. (2009) · Topics in Cognitive Science · doi:10.1111/j.1756-8765.2008.01005.x · Cited by 1,597
Statistics for behavioral sciences
Gravetter, F. J., & Wallnau, L. B. (1985) · Cited by 1,104
Development of an instrument for measuring different types of cognitive load
Leppink, J., Paas, F., Van der Vleuten, C. P. M., Van Gog, T., & Van Merriënboer, J. J. G. (2013) · Behavior Research Methods · doi:10.3758/s13428-013-0334-1 · Cited by 1,066
Learning from Examples: Instructional Principles from the Worked Examples Research
Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000) · Review of Educational Research · doi:10.2307/1170661 · Cited by 1,029
Taking the Load Off a Learner's Mind: Instructional Design for Complex Learning
van Merrienboer, J. J. G., Kirschner, P. A., & Kester, L. (2003) · Educational Psychologist · doi:10.1207/s15326985ep3801_2 · Cited by 857
(2011) · Cited by 725
Effects of schema acquisition and rule automation on mathematical problem-solving transfer.
Cooper, G., & Sweller, J. (1987) · Journal of Educational Psychology · doi:10.1037/0022-0663.79.4.347 · Cited by 688
Some Difficulties of Learning to Program
Du Boulay, B. (1986) · Journal of Educational Computing Research · doi:10.2190/3lfx-9rrf-67t8-uvk9 · Cited by 648 · cited 1×
“They have to develop a mental model of the computer (the notional machine), the notation (syntax and semantics), the structures (programming schemas and plans), and skill in planning, developing, and debugging programs [5].”
BackgroundFailure rates in introductory programming revisited
Watson, C., & Li, F. W. B. (2014) · Proceedings of the 2014 conference on Innovation & technology in computer science education - ITiCSE '14 · doi:10.1145/2591708.2591749 · Cited by 639
Failure rates in introductory programming
Bennedsen, J., & Caspersen, M. E. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1272848.1272879 · Cited by 627
McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · Working group reports from ITiCSE on Innovation and technology in computer science education · doi:10.1145/572133.572137 · Cited by 626
Learning to program = learning to construct mechanisms and explanations
Soloway, E. (1986) · Communications of the ACM · doi:10.1145/6592.6594 · Cited by 567
Programming pedagogy—a psychological overview
Winslow, L. E. (1996) · ACM SIGCSE Bulletin · doi:10.1145/234867.234872 · Cited by 561
The role of practice in the development of performing musicians
Sloboda, J. A., Davidson, J. W., Howe, M. J. A., & Moore, D. G. (1996) · British Journal of Psychology · doi:10.1111/j.2044-8295.1996.tb02591.x · Cited by 514
Sweller, J. (2004) · Instructional Science · doi:10.1023/b:truc.0000021808.72598.4d · Cited by 470 · cited 1×
“Sweller proposed a “Borrowing and Reorganizing Principle” which means that the way humans build long-term knowledge is by imitating others [46].”
BackgroundMcCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · ACM SIGCSE Bulletin · doi:10.1145/572139.572181 · Cited by 424
Cognitive Load Theory: Recent Theoretical Advances
Sweller, J. (2010) · Cognitive Load Theory · doi:10.1017/cbo9780511844744.004 · Cited by 412
Proceedings of the 15th Annual Conference of the Cognitive Science Society
Goddard, N. (1993) · Cited by 405
Structuring Effective Worked Examples
Ward, M., & Sweller, J. (1990) · Cognition and Instruction · doi:10.1207/s1532690xci0701_1 · Cited by 396
Learning Mathematics From Examples and by Doing
Zhu, X., & Simon, H. A. (1987) · Cognition and Instruction · doi:10.1207/s1532690xci0403_1 · Cited by 357
“Thick” authenticity: new media and authentic learning
Shaffer, D. W., & Resnick, M. (1999) · The Journal of Interactive Learning Research · Cited by 337
The role of learning from examples in the acquisition of recursive programming skills.
Pirolli, P. L., & Anderson, J. R. (1985) · Canadian Journal of Psychology / Revue canadienne de psychologie · doi:10.1037/h0080061 · Cited by 336
The subgoal learning model: Creating better examples so that students can solve novel problems.
Catrambone, R. (1998) · Journal of Experimental Psychology: General · doi:10.1037//0096-3445.127.4.355 · Cited by 327
In the mind's eye: enhancing human performance
(1992) · Choice Reviews Online · doi:10.5860/choice.30-2385 · Cited by 271
The effects of overlearning and distributed practise on the retention of mathematics knowledge
Rohrer, D., & Taylor, K. (2006) · Applied Cognitive Psychology · doi:10.1002/acp.1266 · Cited by 250
Do Written Instructions Need Examples?
LeFevre, J.-A., & Dixon, P. (1986) · Cognition and Instruction · doi:10.1207/s1532690xci0301_1 · Cited by 227
Strategies for Programming Instruction in High School: Program Completion vs. Program Generation
Van Merriënboer, J. J. G. (1990) · Journal of Educational Computing Research · doi:10.2190/4nk5-17l7-twqv-1ehl · Cited by 222
Strategies for Computer-Based Programming Instruction: Program Completion vs. Program Generation
Van Merriënboer, J. J. G., & De Croock, M. B. M. (1992) · Journal of Educational Computing Research · doi:10.2190/mjdx-9pp4-kfmt-09pm · Cited by 207
Evaluating a new exam question: Parsons problems
Denny, P., Luxton-Reilly, A., & Simon, B. (2008) · Proceedings of the Fourth international Workshop on Computing Education Research · doi:10.1145/1404520.1404532 · Cited by 203
Parson's programming puzzles: a fun and effective learning tool for first programming courses
Parsons, D., & Haden, P. (2006) · Australasian Computing Education Conference · Cited by 195
A multi-national study of reading and tracing skills in novice programmers
Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J. E., Sanders, K., Seppälä, O., et al. (2004) · Working group reports from ITiCSE on Innovation and technology in computer science education · doi:10.1145/1044550.1041673 · Cited by 182
Eiriksdottir, E., & Catrambone, R. (2011) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720811419154 · Cited by 174
Developing a validated assessment of fundamental CS1 concepts
Tew, A. E., & Guzdial, M. (2010) · Proceedings of the 41st ACM technical symposium on Computer science education · doi:10.1145/1734263.1734297 · Cited by 170
Margulieux, L. E., Guzdial, M., & Catrambone, R. (2012) · Proceedings of the ninth annual international conference on International computing education research · doi:10.1145/2361276.2361291 · Cited by 149 · cited 1×
“3 Subgoal labeling is a method of teaching subgoal learning through worked examples [7, 29].”
MethodSelf-citationMeasuring cognitive load in introductory CS: adaptation of an instrument
Morrison, B. B., Dorn, B., & Guzdial, M. (2014) · Proceedings of the tenth annual conference on International computing education research · doi:10.1145/2632320.2632348 · Cited by 147
Subgoals Help Students Solve Parsons Problems
Morrison, B. B., Margulieux, L. E., Ericson, B., & Guzdial, M. (2016) · Proceedings of the 47th ACM Technical Symposium on Computing Science Education · doi:10.1145/2839509.2844617 · Cited by 125
Self-citation(2017) · Lexikon des gesamten Buchwesens Online · doi:10.1163/9789004337862_lgbo_com_030061 · Cited by 113
Analysis of Interactive Features Designed to Enhance Learning in an Ebook
Ericson, B. J., Guzdial, M. J., & Morrison, B. B. (2015) · Proceedings of the eleventh annual International Conference on International Computing Education Research · doi:10.1145/2787622.2787731 · Cited by 101
Self-citationA fresh look at novice programmers' performance and their teachers' expectations
Utting, I., Tew, A. E., McCracken, M., Thomas, L., Bouvier, D., Frye, R., Paterson, J., Caspersen, M., Kolikant, Y. B.-D., Sorva, J., et al. (2013) · Proceedings of the ITiCSE working group reports conference on Innovation and technology in computer science education-working group reports · doi:10.1145/2543882.2543884 · Cited by 79
Identifying Design Principles for CS Teacher Ebooks through Design-Based Research
Ericson, B. J., Rogers, K., Parker, M., Morrison, B., & Guzdial, M. (2016) · Proceedings of the 2016 ACM Conference on International Computing Education Research · doi:10.1145/2960310.2960335 · Cited by 78
Self-citationCunningham, K., Blanchard, S., Ericson, B., & Guzdial, M. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106190 · Cited by 77
Self-citationTuffiash, M., Roring, R. W., & Ericsson, K. A. (2007) · Journal of Experimental Psychology: Applied · doi:10.1037/1076-898x.13.3.124 · Cited by 66
Soloway's Rainfall Problem Has Become Harder
Simon (2013) · 2013 Learning and Teaching in Computing and Engineering · doi:10.1109/latice.2013.44 · Cited by 61
Distractors in Parsons Problems Decrease Learning Efficiency for Young Novice Programmers
Harms, K. J., Chen, J., & Kelleher, C. L. (2016) · Proceedings of the 2016 ACM Conference on International Computing Education Research · doi:10.1145/2960310.2960314 · Cited by 60
Two-Dimensional Parson’s Puzzles: The Concept, Tools, and First Observations
Ihantola, P., & Karavirta, V. (2011) · Journal of Information Technology Education: Innovations in Practice · doi:10.28945/1394 · Cited by 55
Benda, K., Bruckman, A., & Guzdial, M. (2012) · ACM Transactions on Computing Education · doi:10.1145/2382564.2382567 · Cited by 53
Enabling independent learning of programming concepts through programming completion puzzles
Harms, K. J., Rowlett, N., & Kelleher, C. L. (2015) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2015.7357226 · Cited by 46
Design and Development of a Web-based Interactive Software Tool for Teaching Operating Systems
Garmpis, A. (2011) · Journal of Information Technology Education Research · doi:10.28945/1357 · Cited by 41
Runestone interactive: tools for creating interactive course materials
Miller, B., & Ranum, D. (2014) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2556325.2567887 · Cited by 27
The Code Mangler: Evaluating Coding Ability Without Writing any Code
Cheng, N., & Harrington, B. (2017) · Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education · doi:10.1145/3017680.3017704 · Cited by 26
Garner, S. (2007) · Informing Science and IT Education Conference · doi:10.28945/3127 · Cited by 22
Open source widget for parson's puzzles
Ihantola, P., & Karavirta, V. (2010) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1822090.1822178 · Cited by 14
Research works printed but not linked: 6
- 10.3102/00346543070002181 · doi:10.3102/00346543070002181 · Paper page
- John D. Bransford , Ann L. Brown and Rodney R . Cocking How People Learn. NATIONAL ACADEMY PRESS , 2000 . John D. Bransford, Ann L. Brown and Rodney R. Cocking How People Learn. NATIONAL ACADEMY PRESS, 2000.
- 10.1037/0096-3445.127.4.355 · doi:10.1037/0096-3445.127.4.355 · Paper page
- Barbara Ericson Java Review Book for the AP CS A exam https://runestone.academy/runestone/static/JavaReview/index.html Barbara Ericson Java Review Book for the AP CS A exam https://runestone.academy/runestone/static/JavaReview/index.html
- William MK Trochim and James P Donnelly Research methods knowledge base ( 3 rd ed). Atomic Dog , Cincinnati, OH , 2006 . William MK Trochim and James P Donnelly Research methods knowledge base (3rd ed). Atomic Dog, Cincinnati, OH, 2006.
- Barry J. Wadsworth Piaget's Theory of Cognitive and Affective Development - Fourth Edition. Longman , New York 1989 . Barry J. Wadsworth Piaget's Theory of Cognitive and Affective Development - Fourth Edition. Longman, New York 1989.
Cited by
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366
“Learner’s typically complete Parsons problems significantly more quickly than fixing or writing code and have similar learning gains [24, 25, 94].”
Self-citationIzu, C., Schulte, C., Aggarwal, A., Cutts, Q., Duran, R., Gutica, M., Heinemann, B., Kraemer, E., Lonati, V., Mirolo, C., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372501 · Cited by 97
Cognitive Load Theory in Computing Education Research: A Review
Duran, R., Zavgorodniaia, A., & Sorva, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3483843 · Cited by 92
“— Ericson et al. [20] compared the effects of two forms of practice—Parsons problems and program writing—on learning outcomes, efficiency, and cognitive load.”
BackgroundKazemitabaar, M., Huang, O., Suh, S., Henley, A. Z., & Grossman, T. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712104 · Cited by 92
Metadata
- Authors
- Ericson, B. J., Margulieux, L. E., & Rick, J.
- Affiliations
- Georgia Institute of Technology (US)
- Georgia State University (US)
- Venue
- Proceedings of the 17th Koli Calling International Conference on Computing Education Research
- Publisher
- ACM
- Published
- 2017-11-16
- Pages
- 20-29
- OpenAlex
- W2766230222
- Semantic Scholar
- 2f1b7c75a3cc2f33fc34575bbe77eeb1f336debc
- Cited by
- 121
- References
- 56
- Citations by year
- 2026: 6 · 2025: 9 · 2024: 13 · 2023: 21 · 2022: 18 · 2021: 19 · 2020: 18 · 2019: 14 · 2018: 3
- FWCI
- 8.82
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Koli Calling International Conference on Computing Education Research (publishedVersion)
- Funders
- National Science Foundation (1432300 and 1138378)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Educational Robotics and Engineering
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Keywords
- Parsons problems, distractors, code completion, writing software, programming education, cognitive load, learning efficiency, knowledge retention, summative assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.