Developing Novice Programmers’ Self-Regulation Skills with Code Replays
Open accessAbstract
Learning programming benefits from self-regulation, but novices lack support for developing these skills of cognitive control. To support their development, we designed Code Replayer, an online tool that enables novice programmers to practice programming and then replay their coding process to reflect and identify process improvements. To evaluate the impact of replaying code on self-regulation, we conducted a formative qualitative evaluation with 21 novice programmers who used Code Replayer to practice writing code. We found that after watching code replays, participants more frequently interpreted problem prompts and planned their solutions, two crucial self-regulation behaviors that novices often overlook. We interpret our results by focusing on two focal points in the design of code replays as a programming self-regulation intervention: interpreting pauses in replays and ensuring replays of struggle are more informative and less detrimental.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some work has used fine-grained programming process logs from instrumented IDEs to uncover student work patterns and to visualize student programming processes.
“Some work has looked into using fine-grained programming process logs gathered from instrumented IDEs to uncover student work patterns [21-23, 26, 40] and to visualize student programming processes [10,14,35,38].”
From Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study (2024) · cites this paper as [38]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Motivational and self-regulated learning components of classroom academic performance.
Pintrich, P. R., & De Groot, E. V. (1990) · Journal of Educational Psychology · doi:10.1037/0022-0663.82.1.33 · Cited by 8,332 · cited 2× · see citing works
“The subscale consisted of nine Likert-type items where each item asked participants to rate how true statements were from “1: not at all true of me” to “7: very true of me” [46].”
BackgroundMethodBecoming a Self-Regulated Learner: An Overview
Zimmerman, B. J. (2002) · Theory Into Practice · doi:10.1207/s15430421tip4102_2 · Cited by 7,305 · see citing works
(2014) · Medical Entomology and Zoology · Cited by 6,828 · see citing works
Zimmerman, B. J. (2000) · Handbook of Self-Regulation · doi:10.1016/b978-012109890-2/50031-7 · Cited by 6,069 · see citing works
Reliability and Predictive Validity of the Motivated Strategies for Learning Questionnaire (Mslq)
Pintrich, P. R., Smith, D. A. F., Garcia, T., & Mckeachie, W. J. (1993) · Educational and Psychological Measurement · doi:10.1177/0013164493053003024 · Cited by 3,646 · see citing works
Protocol Analysis: Verbal Reports as Data
Ericsson, K. A., & Simon, H. A. (1993) · doi:10.7551/mitpress/5657.001.0001 · Cited by 3,537 · see citing works
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Nature · doi:10.1038/466029a · Cited by 3,441 · see citing works
Grounded Theory and Sensitizing Concepts
Bowen, G. A. (2006) · International Journal of Qualitative Methods · doi:10.1177/160940690600500304 · Cited by 1,937 · see citing works
Quantifying Qualitative Analyses of Verbal Data: A Practical Guide
Michelene, T. H. (1997) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0603_1 · Cited by 1,765 · see citing works
Handbook of Metacognition in Education
Hacker, D. J., Dunlosky, J., & Graesser, A. C. (2009) · doi:10.4324/9780203876428 · Cited by 1,509 · see citing works
Baker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993 · see citing works
The effects of stereotype activation on behavior: A review of possible mechanisms.
Wheeler, S. C., & Petty, R. E. (2001) · Psychological Bulletin · doi:10.1037/0033-2909.127.6.797 · Cited by 769 · see citing works
QuantCrit: rectifying quantitative methods through critical race theory
Garcia, N. M., López, N., & Vélez, V. N. (2017) · Race Ethnicity and Education · doi:10.1080/13613324.2017.1377675 · Cited by 495 · see citing works
Acquiring writing revision skill: Shifting from process to outcome self-regulatory goals.
Zimmerman, B. J., & Kitsantas, A. (1999) · Journal of Educational Psychology · doi:10.1037/0022-0663.91.2.241 · Cited by 467 · see citing works
Motivated Strategies for Learning Questionnaire
Pintrich, P. R., & De Groot, E. V. (2012) · PsycTESTS Dataset · doi:10.1037/t09161-000 · Cited by 399 · see citing works
Guo, P. J. (2013) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2445196.2445368 · Cited by 399 · see citing works
The Organization of Informal Learning
Rogoff, B., Callanan, M. A., Gutiérrez, K. D., & Erickson, F. (2016) · Review of Research in Education · doi:10.3102/0091732x16680994 · Cited by 320 · see citing works
Janelle, C. M., Barba, D. A., Frehlich, S. G., Tennant, L. K., & Cauraugh, J. H. (1997) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.1997.10608008 · Cited by 313 · see citing works
Dell, N. L., Vaidyanathan, V., Medhi, I., Cutrell, E. B., & Thies, W. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208589 · Cited by 267 · see citing works
Panadero, E., Klug, J., & Järvelä, S. (2015) · Scandinavian Journal of Educational Research · doi:10.1080/00313831.2015.1066436 · Cited by 258 · see citing works
Role of attention in implicit learning.
Stadler, M. A. (1995) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.21.3.674 · Cited by 256 · see citing works
Linxen, S., Sturm, C., Brühlmann, F., Cassau, V., Opwis, K., & Reinecke, K. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445488 · Cited by 254 · see citing works
Programming, Problem Solving, and Self-Awareness: Effects of Explicit Guidance
Loksa, D., Ko, A. J., Jernigan, W., Oleson, A., Mendez, C. J., & Burnett, M. M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858252 · Cited by 223 · see citing works
Self-citationHammer, D., & Berland, L. K. (2013) · Journal of the Learning Sciences · doi:10.1080/10508406.2013.802652 · Cited by 214 · see citing works
How to do better with gender on surveys
Spiel, K., Haimson, O. L., & Lottridge, D. M. (2019) · interactions · doi:10.1145/3338283 · Cited by 208 · see citing works
Self-monitoring of self-regulation during math homework behaviour using standardized diaries
Schmitz, B., & Perels, F. (2011) · Metacognition and Learning · doi:10.1007/s11409-011-9076-6 · Cited by 156 · see citing works
A theory of instruction for introductory programming skills
Xie, B., Loksa, D., Nelson, G. L., Davidson, M. J., Dong, D., Kwik, H., Tan, A. H., Hwa, L., Li, M., & Ko, A. J. (2019) · Computer Science Education · doi:10.1080/08993408.2019.1565235 · Cited by 148 · see citing works
Self-citationThe Role of Self-Regulation in Programming Problem Solving Process and Success
Loksa, D., & Ko, A. J. (2016) · Proceedings of the 2016 ACM Conference on International Computing Education Research · doi:10.1145/2960310.2960334 · Cited by 143 · see citing works
Self-citationMetacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use
Loksa, D., Margulieux, L., Becker, B. A., Craig, M., Denny, P., Pettit, R., & Prather, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3487050 · Cited by 140 · see citing works
Schoenfeld, A. H. (1992) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0202_3 · Cited by 121 · see citing works
Algorithmic Bias in EducationPreprint
Baker, R. S., & Hawn, A. (2021) · doi:10.35542/osf.io/pbmvz · Cited by 117 · see citing works
Gender-Inclusive HCI Research and Design: A Conceptual Review
Stumpf, S., Peters, A. N., Bardzell, S., Burnett, M., Busse, D. K., Cauchard, J. R., & Churchill, E. F. (2020) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000056 · Cited by 107 · see citing works
How Equity and Inequity Can Emerge in Pair Programming
Lewis, C. M., & Shah, N. (2015) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2787622.2787716 · Cited by 104 · see citing works
(2019) · doi:10.1145/3287324.3287374 · Cited by 104 · see citing works
The effectiveness of live-coding to teach introductory programming
Rubin, M. J. (2013) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2445196.2445388 · Cited by 103 · see citing works
Kalyuga, S. (2009) · Managing Cognitive Load in Adaptive Multimedia Learning · doi:10.4018/978-1-60566-048-6.ch003 · Cited by 90 · see citing works
Exploring the Pair Programming Process
Rodríguez, F. J., Price, K. M., & Boyer, K. E. (2017) · doi:10.1145/3017680.3017748 · Cited by 88 · see citing works
Metacognition and the Expertise of Computer Program Comprehension
Eteläpelto, A. (1993) · Scandinavian Journal of Educational Research · doi:10.1080/0031383930370305 · Cited by 83 · see citing works
Haaranen, L. (2017) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3059009.3059035 · Cited by 71 · see citing works
Role of Live-coding in Learning Introductory Programming
Raj, A. G. S., Patel, J. M., Halverson, R., et al. (2018) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3279720.3279725 · Cited by 63 · see citing works
Students' perceptions of the differences between formal and informal learning
Boustedt, J., Eckerdal, A., McCartney, R., Sanders, K., Thomas, L., & Zander, C. (2011) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2016911.2016926 · Cited by 57 · see citing works
Enhancing metacognitive awareness of undergraduates through using an e-educational video environment
Altıok, S., Başer, Z., & Yükseltürk, E. (2019) · Computers & Education · doi:10.1016/j.compedu.2019.05.010 · Cited by 57 · see citing works
A Comparative Study of Standard-Setting Methods
Livingston, S. A., & Zieky, M. J. (1989) · Applied Measurement in Education · doi:10.1207/s15324818ame0202_3 · Cited by 54 · see citing works
The importance of students' attention to program state
Lewis, C. M. (2012) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2361276.2361301 · Cited by 50 · see citing works
Live Coding: A Review of the Literature
Selvaraj, A., Zhang, E., Porter, L., et al. (2021) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3430665.3456382 · Cited by 47 · see citing works
Who's Learning? Using Demographics in EDM Research
Paquette, L., Ocumpaugh, J. L., Li, Z., Andres, A., & Baker, R. S. (2020) · Journal of Educational Data Mining · doi:10.5281/zenodo.4143611 · Cited by 46 · see citing works
Comparison of Time Metrics in Programming
Leinonen, J., Leppänen, L., Ihantola, P., & Hellas, A. (2017) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3105726.3106181 · Cited by 41 · see citing works
Kung, R. L., & Linder, C. (2007) · Metacognition and Learning · doi:10.1007/s11409-007-9006-9 · Cited by 39 · see citing works
Investigating Novices' In Situ Reflections on Their Programming Process
Loksa, D., Xie, B., Kwik, H., & Ko, A. J. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366846 · Cited by 39 · see citing works
Self-citationLive programming as a lecture technique
Paxton, J. (2002) · Journal of computing sciences in colleges · Cited by 37 · see citing works
Oleson, A., Xie, B., Salac, J., Everson, J., Kivuva, F. M., & Ko, A. J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543967 · Cited by 35 · see citing works
Self-citationMurphy, L., Fitzgerald, S., Hanks, B., & McCauley, R. A. (2010) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/1839594.1839604 · Cited by 34 · see citing works
ProgSnap2: A Flexible Format for Programming Process Data
Price, T., Hovemeyer, D., Rivers, K., Gao, G., Bart, A. C., Kazerouni, A. M., Becker, B. A., Petersen, A., Gusukuma, L., Edwards, S. H., et al. (2020) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3341525.3387373 · Cited by 34 · see citing works
Yan, L., Hu, A., & Piech, C. (2019) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3287324.3287483 · Cited by 32 · see citing works
Incorporating metacognition into learning
Mani, M., & Mazumder, Q. H. (2013) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2445196.2445218 · Cited by 31 · see citing works
Stereotype Threat and African-American Student Achievement
Steele, C. M. (2018) · Routledge eBooks · doi:10.4324/9780429494468-31 · Cited by 31 · see citing works
Prather, J., Margulieux, L., Whalley, J., Denny, P., Reeves, B. N., Becker, B. A., Singh, P., Powell, G., & Bosch, N. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543970 · Cited by 31 · see citing works
Aveline, M. O. (1992) · British Journal of Psychotherapy · doi:10.1111/j.1752-0118.1992.tb01198.x · Cited by 28 · see citing works
Pauses and spacing in learning to program
Leppänen, L., Leinonen, J., & Hellas, A. (2016) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2999541.2999549 · Cited by 18 · see citing works
Experiences of Computer Science Transfer Students
Kwik, H., Xie, B., & Ko, A. J. (2018) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3230977.3231004 · Cited by 16 · see citing works
Self-citationImproving student learning outcomes with pair programming
Radermacher, A., Walia, G., & Rummelt, R. (2012) · doi:10.1145/2361276.2361294 · Cited by 15 · see citing works
CodeProcess Charts: Visualizing the Process of Writing Code
Shrestha, R., Leinonen, J., Hellas, A., Ihantola, P., & Edwards, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511867 · Cited by 13 · see citing works
External Imagery in Computer Programming
Ditton, J., Swanson, H., & Edwards, J. (2021) · Proceedings of the 52nd ACM Technical Symposium on Computer Science Education · doi:10.1145/3408877.3432426 · Cited by 7 · see citing works
Gandolfi, E., Ferdig, R. E., & Clements, R. J. (2022) · British Journal of Educational Technology · doi:10.1111/bjet.13207 · Cited by 7 · see citing works
Zhang, Q., & Rutherford, T. A. (2022) · International Learning Analytics and Knowledge Conference (LAK) · doi:10.1145/3506860.3506873 · Cited by 5 · see citing works
McWilliams, R. L. (1996) · Medical Entomology and Zoology · Cited by 3 · see citing works
Quantifying Student Struggles using Heatmaps and Keystroke Data
Fjeldsted, G., & Edwards, J. (2022) · Intermountain Engineering, Technology and Computing (IETC) · doi:10.1109/ietc54973.2022.9796894 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- William T Hoyt. 2010. Interrater reliability and agreement. In The Reviewer's Guide to Quantitative Methods in the Social Sciences, Gregory R Hancock, Ralph O Mueller, and Laura M Stapleton (Eds.). Routledge, 141-154. https://doi.org/10. 4324/9781315755649-10/interrater-reliability-agreement-william-hoyt · DOI
- Ye Liu and Xiaolan Fu. 2007. How Does Distraction Task Influence the Interaction of Working Memory and Long-Term Memory? In Engineering Psychology and Cognitive Ergonomics, Don Harris (Ed.). Vol. 4562. Springer Berlin Heidelberg, Berlin, Heidelberg, 366-374.
Cited by
Sorted by citations
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196 · see citing works
“Another key component for moving forward in the era of GenAI is to finally invest in explicitly teaching novice programmers metacognitive behaviors and skills [38, 62, 68].”
Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630880 · Cited by 54 · see citing works
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B., Kimmel, B., Wright, J., & Briggs, B. (2024) · arXiv · Cited by 196 · see citing works
“Another key component for moving forward in the era of GenAI is to finally invest in explicitly teaching novice programmers metacognitive behaviors and skills [38, 62, 68].”
- Authors
- Xie, B., Lim, J. O., Pham, P. K. D., Li, M., & Ko, A. J.
- Affiliations
- Stanford University (US)
- The Information School, University of Washington, Seattle, USA
- University of Washington (US)
- Venue
- Proceedings of the 2023 ACM Conference on International Computing Education Research V.1
- Publisher
- ACM
- Published
- 2023-08-07
- Pages
- 298-313
- OpenAlex
- W4386587098
- Cited by
- 18
- References
- 68
- Citations by year
- 2026: 7 · 2025: 5 · 2024: 6
- FWCI
- 5.44
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion)
- Funders
- Microsoft
- Adobe Systems
- National Science Foundation (1539179, 1703304, 1836813, 2031265, 2100296, 2122950, 2137834, 2137312, 12566082)
- McCoy Family Center for Ethics in Society
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Keywords
- self-regulation, novice programmers, programming education, cognitive control, problem understanding, planning the solution, online learning tools
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
