Metacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools
Abstract
Most novice programmers are not explicitly aware of the problem-solving process used to approach programming problems and cannot articulate to an instructor where they are in that process. Many are now arguing that this skill, called metacognitive awareness, is crucial for novice learning. However, novices frequently learn in university CS1 courses that employ automated assessment tools (AATs), which are not typically designed to provide the cognitive scaffolding necessary for novices to develop metacognitive awareness. This paper reports on an experiment designed to understand what difficulties novice programmers currently face when learning to code with an AAT. We describe the experiences of CS1 students who participated in a think-aloud study where they were observed solving a programming problem with an AAT. Our observations show that some students mentally augmented the tool when it did not explicitly support their metacognitive awareness, while others stumbled due to the tool's lack of such support. We use these observations to formulate difficulties faced by novices that lack metacognitive awareness, compare these results to other related studies, and look toward future work in modifying AATs.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Others faced metacognitive difficulties like feeling a false sense of accomplishment at having a lot of code but still being far from a working solution.
“Others faced the kind of metacognitive difficulties described by Prather et al. like feeling a false sense of accomplishment at having a lot of code, but still being far from a working solution [71].”
From “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers (2024) · cites this paper as [71]
Instructors spend a considerable amount of time helping novices with cryptic error messages.
“Instructors report that they spend a considerable amount of time helping novices with these often cryptic messages [15,31,32,37].”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [32]
Cites
Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests
Rubin, J. N. (1994) · Cited by 1,750 · see citing works
(1994) · The MIT Press eBooks · doi:10.7551/mitpress/4561.001.0001 · Cited by 1,741 · see citing works
The Scratch Programming Language and Environment
Maloney, J. H., Resnick, M., Rusk, N., Silverman, B., & Eastmond, E. (2010) · ACM Transactions on Computing Education · doi:10.1145/1868358.1868363 · Cited by 1,423 · see citing works
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 561 · see citing works
Six Learning Barriers in End-User Programming Systems
Ko, A. J., Myers, B. A., & Aung, H. H. (2004) · doi:10.1109/vlhcc.2004.47 · Cited by 552 · see citing works
A Survey of Automated Assessment Approaches for Programming Assignments
Ala-Mutka, K. (2005) · Computer Science Education · doi:10.1080/08993400500150747 · Cited by 550 · see citing works
Storytelling alice motivates middle school girls to learn computer programming
Kelleher, C. L., Pausch, R. F., & Kiesler, S. B. (2007) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/1240624.1240844 · Cited by 505 · see citing works
Automatic test-based assessment of programming
Douce, C., Livingstone, D., & Orwell, J. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163409 · Cited by 416 · see citing works
Educational Data Mining and Learning Analytics in Programming
Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S. H., Isohanni, E., Korhonen, A., Petersen, A., Rivers, K., et al. (2015) · doi:10.1145/2858796.2858798 · Cited by 332 · see citing works
Web-CAT: automatically grading programming assignments
Edwards, S., & Pérez-Quiñones, M. A. (2008) · Annual Conference on Innovation and Technology in Computer Science Education · doi:10.1145/1597849.1384371 · Cited by 314 · see citing works
What would other programmers do
Hartmann, B., MacDougall, D., Brandt, J., & Klemmer, S. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753478 · Cited by 280 · see citing works
Edwards, S. H., & Pérez-Quiñones, M. A. (2008) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1384271.1384371 · Cited by 256 · see citing works
Automatic graders for programming classes
Hollingsworth, J. (1960) · Communications of the ACM · doi:10.1145/367415.367422 · Cited by 231 · see citing works
Programming, Problem Solving, and Self-Awareness: Effects of Explicit Guidance
Loksa, D., Ko, A. J., Jernigan, W., Oleson, A., Mendez, C. J., & Burnett, M. M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858252 · Cited by 223 · see citing works
Examining the role of self-regulated learning on introductory programming performance
Bergin, S., Reilly, R., & Traynor, D. (2005) · doi:10.1145/1089786.1089794 · Cited by 205 · see citing works
Identifying and correcting Java programming errors for introductory computer science students
Hristova, M., Misra, A., Rutter, M., & Mercuri, R. T. (2003) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/611892.611956 · Cited by 201 · see citing works
Rethinking computer science education from a test-first perspective
Edwards, S. H. (2003) · doi:10.1145/949344.949390 · Cited by 173 · see citing works
Analysis of research into the teaching and learning of programming
Sheard, J., Simon, S., Hamilton, M., & Lönnberg, J. (2009) · doi:10.1145/1584322.1584334 · Cited by 161 · see citing works
Enhancing syntax error messages appears ineffectual
Denny, P. C., Luxton-Reilly, A., & Carpenter, D. (2014) · doi:10.1145/2591708.2591748 · Cited by 148 · see citing works
Towards a Systematic Review of Automated Feedback Generation for Programming Exercises
Keuning, H., Jeuring, J. T., & Heeren, B. (2016) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2899415.2899422 · Cited by 142 · see citing works
Nelson, G. L., Xie, B., & Ko, A. J. (2017) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3105726.3106178 · Cited by 130 · see citing works
On Compiler Error Messages: What TheySayand What TheyMean
Traver, V. J. (2010) · Advances in Human-Computer Interaction · doi:10.1155/2010/602570 · Cited by 128 · see citing works
Evaluating metacognitive scaffolding in Guided Invention Activities
Roll, I., Holmes, N. G., Day, J., & Bonn, D. A. (2012) · Instructional Science · doi:10.1007/s11251-012-9208-7 · Cited by 126 · see citing works
Personifying programming tool feedback improves novice programmers' learning
Lee, M. J., & Ko, A. J. (2011) · Proceedings of the seventh international workshop on Computing education research · doi:10.1145/2016911.2016934 · Cited by 125 · see citing works
An Effective Approach to Enhancing Compiler Error Messages
Becker, B. A. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844584 · Cited by 125 · see citing works
Identifying computer science self-regulated learning strategies
Falkner, K., Vivian, R., & Falkner, N. J. G. (2014) · Proceedings of the 2014 conference on Innovation & technology in computer science education - ITiCSE '14 · doi:10.1145/2591708.2591715 · Cited by 121 · see citing works
Identifying and correcting Java programming errors for introductory computer science students
Hristova, M., Misra, A., Rutter, M., & Mercuri, R. T. (2003) · ACM SIGCSE Bulletin · doi:10.1145/792548.611956 · Cited by 120 · see citing works
Murphy, L., Lewandowski, G., McCauley, R. A., Simon, B., Thomas, L., & Zander, C. (2008) · ACM SIGCSE Bulletin · doi:10.1145/1352322.1352191 · Cited by 118 · see citing works
Identifying Top Java Errors for Novice Programmers
Jackson, J., Cobb, M. L., & Carver, C. A. (2006) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2005.1611967 · Cited by 100 · see citing works
Measuring the effectiveness of error messages designed for novice programmers
Marceau, G., Fisler, K., & Krishnamurthi, S. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953308 · Cited by 98 · see citing works
Murphy, L., Lewandowski, G., McCauley, R. A., Simon, B., Thomas, L., & Zander, C. (2008) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1352135.1352191 · Cited by 89 · see citing works
Do Developers Read Compiler Error Messages?
Barik, T., Smith, J., Lubick, K., Holmes, E., Feng, J., Murphy-Hill, E., & Parnin, C. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.59 · Cited by 86 · cited 1× · see citing works
“…other recent studies utilize aspects of a human factors approach to an automated assessment tool such as the theroy behind error message design [60], design and personification of feedback [38], and eye-tracking to determine if novices read error messages and what they are reading when they do [3].”
MethodMetacognition and the Expertise of Computer Program Comprehension
Eteläpelto, A. (1993) · Scandinavian Journal of Educational Research · doi:10.1080/0031383930370305 · Cited by 83 · see citing works
Automated Assessment in a Programming Tools Course
Fernández‐Alemán, J. L. (2010) · IEEE Transactions on Education · doi:10.1109/te.2010.2098442 · Cited by 82 · see citing works
A New Metric to Quantify Repeated Compiler Errors for Novice Programmers
Becker, B. A. (2016) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2899415.2899463 · Cited by 81 · see citing works
Marceau, G., Fisler, K., & Krishnamurthi, S. (2011) · doi:10.1145/2048237.2048241 · Cited by 77 · see citing works
On Novices' Interaction with Compiler Error Messages: A Human Factors Approach
Prather, J., Pettit, R., McMurry, K. H., Peters, A., Homer, J., Simone, N., & Cohen, M. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106169 · Cited by 72 · see citing works
Self-citationDo Enhanced Compiler Error Messages Help Students?: Results Inconclusive.
Pettit, R. S., Homer, J., & Gee, R. (2017) · Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education · doi:10.1145/3017680.3017768 · Cited by 66 · cited 2× · see citing works
“Pettit et al. enhanced CEMs in an automated assessment tool, Athene, used for C++ programming and did not find conclusive results that the ECEMs were more helpful than standard CEMs [50].”
MethodSelf-citationThe Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test
Becker, B. A., Goslin, K., & Glanville, G. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159461 · Cited by 62 · see citing works
An exploration of novice compilation behaviour in BlueJ
Jadud, M. C. (2021) · Kent Academic Repository (University of Kent) · doi:10.22024/unikent/01.02.86458 · Cited by 61 · see citing works
A qualitative think-aloud study of novice programmers' code writing strategies
Whalley, J. L., & Kasto, N. (2014) · Annual Conference on Innovation and Technology in Computer Science Education · doi:10.1145/2591708.2591762 · Cited by 61 · see citing works
BlueFix: Using Crowd-Sourced Feedback to Support Programming Students in Error Diagnosis and Repair
Watson, C., Li, F. W. B., & Godwin, J. L. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33642-3_25 · Cited by 59 · see citing works
Schorsch, T. (1995) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/199688.199769 · Cited by 59 · see citing works
Empowering students and building confidence in novice programmers through gauntlet
Flowers, T. R., Carver, C. A., & Jackson, J. (2005) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2004.1408551 · Cited by 56 · see citing works
An Empirical Study of Iterative Improvement in Programming Assignments
Pettit, R., Homer, J., Gee, R., et al. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677279 · Cited by 54 · see citing works
Self-citationA mobile learning application for parsons problems with automatic feedback
Karavirta, V., Helminen, J., & Ihantola, P. (2012) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2401796.2401798 · Cited by 51 · see citing works
Freund, S. N., & Roberts, E. S. (1996) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/236452.236560 · Cited by 48 · see citing works
An Adaptive, Collaborative Environment to Develop Good Habits in Programming
Vizcaíno, A., Contreras, J., Favela, J., & Prieto, M. (2000) · Lecture notes in computer science · doi:10.1007/3-540-45108-0_30 · Cited by 38 · see citing works
Incorporating metacognition into learning
Mani, M., & Mazumder, Q. H. (2013) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2445196.2445218 · Cited by 31 · see citing works
A qualitative think aloud study of the early neo-piagetian stages of reasoning in novice programmers
Teague, D., Corney, M., Ahadi, A., et al. (2013) · QUT ePrints (Queensland University of Technology) · doi:10.5555/2667199.2667209 · Cited by 31 · see citing works
Helping programmers understand computer programs
Shaft, T. M. (1995) · ACM SIGMIS Database the DATABASE for Advances in Information Systems · doi:10.1145/223278.223280 · Cited by 28 · see citing works
A detector for non-literal Java errors
Dy, T. T., & Rodrigo, M. M. T. (2010) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930485 · Cited by 28 · see citing works
A principled evaluation for a principled idea garden
Jernigan, W., Horvath, A., Lee, M., Burnett, M., Cuilty, T., Kuttal, S., Peters, A., Kwan, I., Bahmani, F., & Ko, A. (2015) · 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2015.7357222 · Cited by 28 · see citing works
Self-citationReconsidering Automated Feedback
Buffardi, K., & Edwards, S. H. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677313 · Cited by 27 · see citing works
Metacognitive calibration when learning to program
Hauswirth, M., & Adamoli, A. (2017) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3141880.3141904 · Cited by 24 · see citing works
Schorsch, T. (1995) · ACM SIGCSE Bulletin · doi:10.1145/199691.199769 · Cited by 24 · see citing works
Idea Garden: Situated Support for Problem Solving by End-User Programmers
Cao, J., Fleming, S. D., Burnett, M., & Scaffidi, C. P. (2014) · Interacting with Computers · doi:10.1093/iwc/iwu022 · Cited by 22 · see citing works
Barik, T., Witschey, J., Johnson, B., & Murphy-Hill, E. R. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2591062.2591124 · Cited by 21 · see citing works
Binary instant feedback on programming exercises can reduce student engagement and promote cheating
Kyrilov, A., & Noelle, D. C. (2015) · doi:10.1145/2828959.2828968 · Cited by 20 · see citing works
Pettit, R., & Prather, J. E. (2017) · Journal of computing sciences in colleges · doi:10.5555/3055338.3079060 · Cited by 19 · see citing works
Self-citationCarter, E. J. (2015) · Journal of computing sciences in colleges · Cited by 11 · see citing works
Towell, D., & Reeves, B. N. (2010) · Digital Commons - ACU (Abilene Christian University) · Cited by 9 · see citing works
LearnCS!: a new, browser-based C programming environment for CS1
Lipman, D. (2014) · Journal of computing sciences in colleges · Cited by 8 · see citing works
Hasker, R. W. (2002) · Journal of computing sciences in colleges · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · see citing works
Self-citationPrather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256 · see citing works
“We discuss these further in Section 5.”
Self-citationThe Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196 · see citing works
“Unlike our prior study [52], which we conducted before the advent of GenAI, the course in which this study took place utilized GenAI from the beginning.”
InfluentialSelf-citationMetacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use
Loksa, D., Margulieux, L., Becker, B. A., Craig, M., Denny, P., Pettit, R., & Prather, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3487050 · Cited by 140 · see citing works
Self-citationCognitive Load Theory in Computing Education Research: A Review
Duran, R., Zavgorodniaia, A., & Sorva, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3483843 · Cited by 92 · see citing works
Prather, J., Reeves, B. N., Denny, P., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Orvalho, J., Alipour, A., Alfageeh, A., Amarouche, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.12800 · Cited by 20 · see citing works
Self-citationThe Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B., Kimmel, B., Wright, J., & Briggs, B. (2024) · arXiv · Cited by 196 · see citing works
“Unlike our prior study [52], which we conducted before the advent of GenAI, the course in which this study took place utilized GenAI from the beginning.”
InfluentialSelf-citation"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv
“We discuss these further in Section 5.”
Self-citation
- Authors
- Prather, J., Pettit, R., McMurry, K., Peters, A., Homer, J., & Cohen, M.
- Affiliations
- Abilene Christian University (US)
- University of Virginia (US)
- United Services Automobile Association (US)
- Nova Southeastern University (US)
- Venue
- Proceedings of the 2018 ACM Conference on International Computing Education Research
- Publisher
- ACM
- Published
- 2018-08-08
- Pages
- 41-50
- OpenAlex
- W2885271276
- Cited by
- 129
- References
- 64
- Citations by year
- 2026: 9 · 2025: 19 · 2024: 20 · 2023: 31 · 2022: 15 · 2021: 15 · 2020: 7 · 2019: 13
- FWCI
- 9.91
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- metacognitive awareness, novice programmers, automated assessment systems, CS1, cognitive scaffolding, programming education, think-aloud study, problem-solving process
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
