Towards an Analysis of Program Complexity From a Cognitive Perspective
The DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index.
Abstract
Instructional designers, examiners, and researchers frequently need to assess the complexity of computer programs in their work. However, there is a dearth of established methodologies for assessing the complexity of a program from a learning point of view. In this article, we explore theories and methods for describing programs in terms of the demands they place on human cognition. More specifically, we draw on Cognitive Load Theory and the Model of Hierarchical Complexity in order to extend Soloway's plan-based analysis of programs and apply it at a fine level of granularity. The resulting framework of Cognitive Complexity of Computer Programs~(CCCP) generates metrics for two aspects of a program: plan depth and maximal plan interactivity. Plan depth reflects the overall complexity of the cognitive schemas that are required for reasoning about the program, and maximal plan interactivity reflects the complexity of interactions between schemas that arise from program composition. Using a number of short programs as case studies, we apply the CCCP to illustrate why one program or construct is more complex than another, to identify dependencies between constructs that a novice programmer needs to learn and to contrast the complexity of different strategies for program composition. Finally, we highlight some areas in computing education and computing education research in which the CCCP could be applied and discuss the upcoming work to validate and refine the CCCP and associated…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Cognitive Load During Problem Solving: Effects on Learning
Sweller, J. (1988) · Cognitive Science · doi:10.1016/0364-0213(88)90023-7 · Cited by 11,048 · cited 8×
“The first is schema theory and the related Cognitive Load Theory (CLT) [2, 10, 80], which are concerned with the limitations of working memory and the growth of expertise as schemas in long-term memory; here we extend the earlier work in CER by Soloway, Rist, and others”
BackgroundResultInfluentialMcCabe, T. J. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233837 · Cited by 6,059
The Taxonomy of Educational Objectives
(2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_6027 · Cited by 4,103
Cognitive Load Measurement as a Means to Advance Cognitive Load Theory
Paas, F., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. M. (2003) · Educational Psychologist · doi:10.1207/s15326985ep3801_8 · Cited by 2,527
Element Interactivity and Intrinsic, Extraneous, and Germane Cognitive Load
Sweller, J. (2010) · Educational Psychology Review · doi:10.1007/s10648-010-9128-5 · Cited by 2,020 · cited 2×
“depends on external factors such as engagement [40, 81].”
BackgroundRobinson, P. (2001) · Applied Linguistics · doi:10.1093/applin/22.1.27 · Cited by 1,273 · cited 1×
“Establishing a theoretically motivated, empirically sustainable, and pedagogically feasible methodology for sequencing topics by complexity is one of the major goals in curriculum development [64].”
BackgroundAccess to information in working memory: Exploring the focus of attention.
Oberauer, K. (2002) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037//0278-7393.28.3.411 · Cited by 1,161 · cited 17×
“1 Which plans fall within the RDA of a particular FoA depends on the way plans are merged, sequenced, and nested.”
BackgroundMethodInfluentialOn the capacity of attention: Its estimation and its role in working memory and cognitive aptitudes
Cowan, N., Elliott, E. M., Scott Saults, J., Morey, C. C., Mattox, S., Hismjatullina, A., & Conway, A. R. A. (2005) · Cognitive Psychology · doi:10.1016/j.cogpsych.2004.12.001 · Cited by 1,144 · cited 1×
“We build on the concept of focus of attention (FoA) [18, 60], which we have adapted to a programming context.”
MethodEvaluating the Quality of Learning: The Solo Taxonomy : Structure of the Observed Learning Outcome
Biggs, J., & Collis, K. F. (1977) · Cited by 1,079
Development of an instrument for measuring different types of cognitive load
Leppink, J., Paas, F., Van der Vleuten, C. P. M., Van Gog, T., & Van Merriënboer, J. J. G. (2013) · Behavior Research Methods · doi:10.3758/s13428-013-0334-1 · Cited by 1,066 · cited 1×
“However, there is an instrument for estimating cognitive load from learners’ ratings of perceived mental effort after a learning task [42], which has been adapted for programming tasks [57].”
MethodA Reconsideration of Cognitive Load Theory
Schnotz, W., & Kürschner, C. (2007) · Educational Psychology Review · doi:10.1007/s10648-007-9053-4 · Cited by 755 · cited 6×
“Cognitive Load Theory (CLT) [2, 10, 68, 80] is a framework for investigating the effects of the human cognitive apparatus on task performance and learning; the primary goal of CLT is to improve instructional design.”
BackgroundInfluentialCognitive Load Theory: How Many Types of Load Does It Really Need?
Kalyuga, S. (2011) · Educational Psychology Review · doi:10.1007/s10648-010-9150-7 · Cited by 725 · cited 1×
“It is determined by element interactivity , which is the degree of interconnectedness between the elements of information that one needs to hold in working memory simultaneously in order to perform successfully [37].”
BackgroundSome Difficulties of Learning to Program
Du Boulay, B. (1986) · Journal of Educational Computing Research · doi:10.2190/3lfx-9rrf-67t8-uvk9 · Cited by 648 · cited 1×
“These elements are chosen so that they represent primitive operations of a notional machine [22, 75] that the programmer instructs.”
BackgroundMcCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · Working group reports from ITiCSE on Innovation and technology in computer science education · doi:10.1145/572133.572137 · Cited by 626 · cited 1×
“monly perform at a level below their teachers’ expectations [45, 54, 83].”
BackgroundTen Steps to Complex Learning: A Systematic Approach to Four-Component Instructional Design
van Merriënboer, J. J. G., Kirschner, P. A., & Frèrejean, J. (2024) · Research Publications (Maastricht University) · doi:10.4324/9781003322481 · Cited by 579
A multi-national study of reading and tracing skills in novice programmers
Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J. E., Sanders, K., Seppälä, O., et al. (2004) · ACM SIGCSE Bulletin · doi:10.1145/1041624.1041673 · Cited by 575 · cited 2×
“writing code of “similar complexity” [45, 72] and would benefit”
BackgroundLearning to program = learning to construct mechanisms and explanations
Soloway, E. (1986) · Communications of the ACM · doi:10.1145/6592.6594 · Cited by 567 · cited 2×
“who have analyzed schemas as reflected in programs [63, 73, 77].”
BackgroundMethodTask complexity: A review and conceptualization framework
Liu, P., & Li, Z. (2012) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2012.09.001 · Cited by 438 · cited 1×
“As illustrated in a survey by Liu and Li [47], complexity means different to different people.”
BackgroundMcCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001) · ACM SIGCSE Bulletin · doi:10.1145/572139.572181 · Cited by 424
Choi, H.-H., van Merriënboer, J. J. G., & Paas, F. (2014) · Educational Psychology Review · doi:10.1007/s10648-014-9262-6 · Cited by 413 · cited 4×
“Cognitive Load Theory (CLT) [2, 10, 68, 80] is a framework for investigating the effects of the human cognitive apparatus on task performance and learning; the primary goal of CLT is to improve instructional design.”
BackgroundResultInfluentialRelationships between reading, tracing and writing skills in introductory programming
Lopez, M., Whalley, J., Robbins, P., & Lister, R. (2008) · Proceedings of the Fourth international Workshop on Computing Education Research · doi:10.1145/1404520.1404531 · Cited by 363 · cited 1×
“skills evolve with growing expertise [17, 44, 48, 82] The studies cited above emphasize the general types of activities that programmers engage in (Bloom) and/or the general degree”
BackgroundCognitive strategies and looping constructs: an empirical study
Soloway, E., Bonar, J., & Ehrlich, K. (1983) · Communications of the ACM · doi:10.1145/182.358436 · Cited by 337 · cited 1×
“likelihood they will write defective code [25, 74]; a sequencedplans approach may lead to better outcomes [25].”
BackgroundNotional machines and introductory programming education
Sorva, J. (2013) · ACM Transactions on Computing Education · doi:10.1145/2483710.2483713 · Cited by 319 · cited 1×
“These elements are chosen so that they represent primitive operations of a notional machine [22, 75] that the programmer instructs.”
BackgroundSelf-citationAn Empirical Investigation into Programming Language Syntax
Stefik, A., & Siebert, S. (2013) · ACM Transactions on Computing Education · doi:10.1145/2534973 · Cited by 295 · cited 1×
“and syntax [21, 79], which are not covered by the CCCP as presented.”
BackgroundNot seeing the forest for the trees: novice programmers and the SOLO taxonomy
Lister, R., Simon, B., Thompson, E., Whalley, J. L., & Prasad, C. (2006) · ACM SIGCSE Bulletin · doi:10.1145/1140123.1140157 · Cited by 264
Grover, S., & Basu, S. (2017) · Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education · doi:10.1145/3017680.3017723 · Cited by 249 · cited 1×
“There is also evidence that students mistakenly concatenate plans instead of merging and generally struggle with plans which interact in complex ways [28], that students sometimes read merged plans as if they were composed sequentially [30], and that a student’s plan composition strategy interacts with the”
BackgroundDelocalized Plans and Program Comprehension
Letovsky, S., & Soloway, E. (1986) · IEEE Software · doi:10.1109/ms.1986.233414 · Cited by 245 · cited 7×
“What Soloway’s group termed “plans” are essentially schemas in the programming domain: a plan represents a stereo-typical solution to a programming problem.”
BackgroundInfluentialHierarchical Complexity of Tasks Shows the Existence of Developmental Stages
Commons, M. L., Trudeau, E. J., Stein, S. A., Richards, F. A., & Krause, S. R. (1998) · Developmental Review · doi:10.1006/drev.1998.0467 · Cited by 227 · cited 2×
“Our other main influence is the Model of Hierarchical Complexity (MHC) [16], a neo-Piagetian theory concerned with the relative complexities of tasks; to the best of our knowledge, the MHC has not been previously applied in computing education.”
BackgroundSchema creation in programming
Rist, R. (1989) · Cognitive Science · doi:10.1016/0364-0213(89)90018-9 · Cited by 217 · cited 6×
“who have analyzed schemas as reflected in programs [63, 73, 77].”
BackgroundMethodInfluentialNovice programmer errors: language constructs and plan composition
Ebrahimi, A. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1069 · Cited by 174 · cited 1×
“A number of studies over the past few decades have investigated bugs in student code and mapped the bugs to the plans in which they appear [23, 25, 36, 59, 69].”
BackgroundA Goal/Plan Analysis of Buggy Pascal Programs
Spohrer, J. C., Soloway, E., & Pope, E. (1985) · Human–Computer Interaction · doi:10.1207/s15327051hci0102_4 · Cited by 172 · cited 1×
“who have analyzed schemas as reflected in programs [63, 73, 77].”
BackgroundConcrete and other neo-Piagetian forms of reasoning in the novice programmer
Lister, R. (2011) · UTS ePRESS (University of Technology Sydney) · Cited by 159
Differentiating Different Types of Cognitive Load: a Comparison of Different Measures
Korbach, A., Brünken, R., & Park, B. (2018) · Educational Psychology Review · doi:10.1007/s10648-017-9404-8 · Cited by 153 · cited 1×
“depends on external factors such as engagement [40, 81].”
BackgroundMeasuring cognitive load in introductory CS: adaptation of an instrument
Morrison, B. B., Dorn, B., & Guzdial, M. (2014) · Proceedings of the tenth annual conference on International computing education research · doi:10.1145/2632320.2632348 · Cited by 147 · cited 3×
“[57], who estimated cognitive load by asking students for their subjective perceptions of mental effort after the students had engaged in various types of CS study.”
BackgroundMethodUnderstanding the syntax barrier for novices
Denny, P., Luxton-Reilly, A., Tempero, E., & Hendrickx, J. (2011) · Proceedings of the 16th annual joint conference on Innovation and technology in computer science education · doi:10.1145/1999747.1999807 · Cited by 133 · cited 1×
“and syntax [21, 79], which are not covered by the CCCP as presented.”
BackgroundDesign and First Results of a Psychometric Test for Measuring Basic Programming Abilities
Mühling, A., Ruf, A., & Hubwieser, P. (2015) · Workshop in Primary and Secondary Computing Education · doi:10.1145/2818314.2818320 · Cited by 120 · cited 1×
“For instance, Mühling et al. [58] gave students a psychometric test whose items featured different programming constructs.”
BackgroundAn Experiment on Using Roles of Variables in Teaching Introductory Programming
Sajaniemi, J., & Kuittinen, M. (2005) · Computer Science Education · doi:10.1080/08993400500056563 · Cited by 119 · cited 1×
“Within CER, these cognitive theories have inspired pedagogies that explicitly teach plan-like patterns to students [20, 34, 66].”
BackgroundA cognitive approach to identifying measurable milestones for programming skill acquisition
Mead, J., Gray, S., Hamer, J., James, R., Sorva, J., Clair, C. S., & Thomas, L. (2006) · ACM SIGCSE Bulletin · doi:10.1145/1189136.1189185 · Cited by 117
Self-citationIntroduction to the Model of Hierarchical Complexity and Its Relationship to Postformal Action
Commons, M. L. (2008) · World Futures · doi:10.1080/02604020802301105 · Cited by 117 · cited 1×
“For instance, Commons [12] applied Rasch analysis to show a positive correlation between the predicted complexity of equations in physics (the pendulum test) and measured student performance.”
BackgroundDo code smells hamper novice programming? A controlled experiment on Scratch programs
Hermans, F., & Aivaloglou, E. (2016) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2016.7503706 · Cited by 117
K-8 Learning Trajectories Derived from Research Literature: Sequence, Repetition, Conditionals
Rich, K. M., Strickland, C., Binkowski, T. A., Moran, C., & Franklin, D. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106166 · Cited by 117 · cited 1×
“[62] created a set of K-8 learning trajectories for three concepts: sequencing, repetition, and conditionals; they identified challenges in basing the trajectories on a heterogeneous and sparse set of prior reports of concept difficulty.”
BackgroundNot seeing the forest for the trees: novice programmers and the SOLO taxonomy
Lister, R., Simon, B., Thompson, E., Whalley, J. L., & Prasad, C. (2006) · Proceedings of the 11th annual SIGCSE conference on Innovation and technology in computer science education · doi:10.1145/1140124.1140157 · Cited by 115 · cited 1×
“Recognizing (like [46]) that programming requires the programmer to think about low-level elements of code (the trees)”
BackgroundThe recurring rainfall problem
Fisler, K. (2014) · Proceedings of the tenth annual conference on International computing education research · doi:10.1145/2632320.2632346 · Cited by 112 · cited 2×
“A number of studies over the past few decades have investigated bugs in student code and mapped the bugs to the plans in which they appear [23, 25, 36, 59, 69].”
BackgroundA conceptual model of cognitive complexity of elements of the programming process
Cant, S. N., Jeffery, D. R., & Henderson‐Sellers, B. (1995) · Information and Software Technology · doi:10.1016/0950-5849(95)91491-h · Cited by 105 · cited 1×
“[8, 32] conceptualized CCM, a tentative framework for measuring complexity, which resembles ours in its goals and in that it, too, draws on schema theory and related findings from cognitive psychology.”
BackgroundGoing SOLO to assess novice programmers
Sheard, J., Carbone, A., Lister, R., Simon, B., Thompson, E., & Whalley, J. L. (2008) · Proceedings of the 13th annual conference on Innovation and technology in computer science education · doi:10.1145/1384271.1384328 · Cited by 102 · cited 1×
“used for analyzing students’ responses to code-reading tasks [71], the structure of students’ code [11], and aspects of program design such as testing and abstraction [9].”
MethodDo we know how difficult the rainfall problem is?
Seppälä, O., Ihantola, P., Isohanni, E., Sorva, J., & Vihavainen, A. (2015) · Proceedings of the 15th Koli Calling Conference on Computing Education Research · doi:10.1145/2828959.2828963 · Cited by 93 · cited 3×
“A number of studies over the past few decades have investigated bugs in student code and mapped the bugs to the plans in which they appear [23, 25, 36, 59, 69].”
BackgroundMethodSelf-citationThe Canterbury QuestionBank: building a repository of multiple-choice CS1 and CS2 questions
Sanders, K., Ahmadzadeh, M., Clear, T., Edwards, S. H., Goldweber, M., Johnson, C., Lister, R., McCartney, R., Patitsas, E., & Spacco, J. (2013) · Proceedings of the ITiCSE working group reports conference on Innovation and technology in computer science education-working group reports · doi:10.1145/2543882.2543885 · Cited by 91 · cited 1×
“In a related line of CER, surface-level metrics from Software Engineering such as Cyclomatic Complexity [53], lines of code, and block depth have been applied to codecomprehension tasks [38], code-writing tasks [24], and multiplechoice questions [67].”
MethodBeckmann, J. F. (2010) · Learning and Instruction · doi:10.1016/j.learninstruc.2009.02.024 · Cited by 89 · cited 9×
“The first is schema theory and the related Cognitive Load Theory (CLT) [2, 10, 80], which are concerned with the limitations of working memory and the growth of expertise as schemas in long-term memory; here we extend the earlier work in CER by Soloway, Rist, and others”
BackgroundMethodResultInfluentialBuck, D., & Stucki, D. J. (2000) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/330908.331817 · Cited by 88 · cited 1×
“Of the theoretical tools that have been used in CER for categorizing activities by complexity, Bloom’s Taxonomy [5] is probably themost common [7, 29, 52].”
MethodA Systematic Review of the Use of Bloom's Taxonomy in Computer Science Education
Masapanta-Carrión, S., & Velázquez-Iturbide, J. Á. (2018) · Proceedings of the 49th ACM Technical Symposium on Computer Science Education · doi:10.1145/3159450.3159491 · Cited by 83 · cited 1×
“Of the theoretical tools that have been used in CER for categorizing activities by complexity, Bloom’s Taxonomy [5] is probably themost common [7, 29, 52].”
MethodA fresh look at novice programmers' performance and their teachers' expectations
Utting, I., Tew, A. E., McCracken, M., Thomas, L., Bouvier, D., Frye, R., Paterson, J., Caspersen, M., Kolikant, Y. B.-D., Sorva, J., et al. (2013) · Proceedings of the ITiCSE working group reports conference on Innovation and technology in computer science education-working group reports · doi:10.1145/2543882.2543884 · Cited by 79 · cited 1×
“monly perform at a level below their teachers’ expectations [45, 54, 83].”
BackgroundSelf-citationA Study of Code Design Skills in Novice Programmers using the SOLO taxonomy
Izu, C., Weerasinghe, A., & Pope, C. (2016) · Proceedings of the 2016 ACM Conference on International Computing Education Research · doi:10.1145/2960310.2960324 · Cited by 79 · cited 1×
“[35] suggested an intuitive, SOLO-inspired learning trajectory in which the learner abstracts increasingly complex “building blocks” (language constructs and plan-like “templates”) in order to tackle the next concepts; the CCCP differs from this work, and the other research just cited, in its use of MHC to structure t…”
ResultDeveloping Assessments to Determine Mastery of Programming Fundamentals
Luxton-Reilly, A., Becker, B. A., Cao, Y., McDermott, R., Mirolo, C., Mühling, A., Petersen, A., Sanders, K., Simon, & Whalley, J. L. (2018) · doi:10.1145/3174781.3174784 · Cited by 78 · cited 1×
“, like ourselves, analyzed individual programs in order to characterize their structure [50, 51]; their analysis and metrics are fundamentally different from ours in that they focused exclusively on syntactic features whereas we have focused on cognitive plans.”
Assessing the complexity of students' knowledge in chemistry
Bernholt, S., & Parchmann, I. (2011) · Chemistry Education Research and Practice · doi:10.1039/c1rp90021h · Cited by 71 · cited 1×
“The MHC has been used for complexity analysis in diverse domains such as physics [78], bias in forensics [14], chemistry [3], and student competence in graduate courses [56].”
MethodGoing SOLO to assess novice programmers
Sheard, J., Carbone, A., Lister, R., Simon, B., Thompson, E., & Whalley, J. L. (2008) · ACM SIGCSE Bulletin · doi:10.1145/1597849.1384328 · Cited by 67
The Compound Nature of Novice Programming Assessments
Luxton-Reilly, A., & Petersen, A. (2017) · Proceedings of the Nineteenth Australasian Computing Education Conference · doi:10.1145/3013499.3013500 · Cited by 66 · cited 1×
“, like ourselves, analyzed individual programs in order to characterize their structure [50, 51]; their analysis and metrics are fundamentally different from ours in that they focused exclusively on syntactic features whereas we have focused on cognitive plans.”
BackgroundA comparison of three developmental stage scoring systems.
Dawson, T. L. (2002) · PubMed · Cited by 60
Measuring the difficulty of code comprehension tasks using software metrics
Kasto, N., & Whalley, J. L. (2013) · Australasian Computing Education Conference · doi:10.5555/2667199.2667206 · Cited by 60
Buck, D., & Stucki, D. J. (2000) · ACM SIGCSE Bulletin · doi:10.1145/331795.331817 · Cited by 59
Commons, M. L., Goodheart, E. A., Pekker, A., Dawson, T. L., Draney, K., & Adams, K. M. (2008) · PubMed · doi:10.5281/zenodo.22700913 · Cited by 55
Designing a multi-faceted SOLO taxonomy to track program design skills through an entire course
Castro, F. E. V., & Fisler, K. (2017) · Proceedings of the 17th Koli Calling International Conference on Computing Education Research · doi:10.1145/3141880.3141891 · Cited by 54 · cited 1×
“Meanwhile, the SOLO taxonomy [4] has been used for analyzing students’ responses to code-reading tasks [71], the structure of students’ code [11], and aspects of program design such as testing and abstraction [9].”
MethodProgramming: reading, writing and reversing
Teague, D., & Lister, R. (2014) · Proceedings of the 2014 conference on Innovation & technology in computer science education - ITiCSE '14 · doi:10.1145/2591708.2591712 · Cited by 51 · cited 1×
“skills evolve with growing expertise [17, 44, 48, 82] The studies cited above emphasize the general types of activities that programmers engage in (Bloom) and/or the general degree”
BackgroundHow difficult are exams?: a framework for assessing the complexity of introductory programming exams
Sheard, J., Simon, N. P., Carbone, A., Chinn, D. D., Clear, T., Corney, M., D’Souza, D., Fenwick, J., Harland, J., Laakso, M., et al. (2013) · QUT ePrints (Queensland University of Technology) · doi:10.5555/2667199.2667215 · Cited by 51 · cited 1×
“The complexity of a programming task is additionally influenced by factors such as task presentation [70], contextualization [6, 49]”
BackgroundPresenting the Formal Theory of Hierarchical Complexity
Commons, M. L., & Pekker, A. (2008) · World Futures · doi:10.1080/02604020802301204 · Cited by 51
Teaching programming strategies explicitly to novice programmers
de RAADT, M. (2026) · doi:10.26192/9yy76 · Cited by 43
Novice Programmers and the Problem Description Effect
Bouvier, D., Lovellette, E., Matta, J., Alshaigy, B., Becker, B. A., Craig, M., Jackova, J., McCartney, R., Sanders, K., & Zarb, M. (2016) · Proceedings of the 2016 ITiCSE Working Group Reports · doi:10.1145/3024906.3024912 · Cited by 43 · cited 1×
“The complexity of a programming task is additionally influenced by factors such as task presentation [70], contextualization [6, 49]”
BackgroundCorney, M., Teague, D., Ahadi, A., & Lister, R. (2012) · QUT ePrints (Queensland University of Technology) · Cited by 42
Syntax, Predicates, Idioms - What Really Affects Code Complexity?
Ajami, S., Woodbridge, Y., & Feitelson, D. G. (2017) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2017.39 · Cited by 39
Modernizing Plan-Composition Studies
Fisler, K., Krishnamurthi, S., & Siegmund, J. (2016) · Proceedings of the 47th ACM Technical Symposium on Computing Science Education · doi:10.1145/2839509.2844556 · Cited by 38 · cited 2×
“For instance, one might sequence the subplans (perhaps using separate functions for each) or interleave them (perhaps using a single loop associated with multiple subplans); such decisions may impact on readability and error rates [25, 27, 33, 74].”
BackgroundProgram complexity metrics and programmer opinions
Katzmarski, B., & Koschke, R. (2012) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2012.6240486 · Cited by 33 · cited 1×
“Such metrics provide one perspective on software complexity but do not indicate how different constructs and plans affect complexity [1], have a low correlation with perceived complexity [39], and disregard the role of prior knowledge.”
BackgroundWhat Makes Code Hard to Understand?Preprint
Hansen, M., Goldstone, R. L., & Lumsdaine, A. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1304.5257 · Cited by 32 · cited 1×
“Violating programmers’ expectations of code structure leads to poorer comprehension, as existing schemas fail to apply [e.g., 31].”
BackgroundSurely we must learn to read before we learn to write
Simon, N. P., Lopez, M., Sutton, K., & Clear, T. (2009) · Australasian Computing Education Conference · doi:10.5555/1862712.1862736 · Cited by 31
Teaching novice programming using goals and plans in a visual notation
Hu, M., Winikoff, M., & Cranefield, S. (2012) · Australasian Computing Education Conference · Cited by 29
Mastering cognitive development theory in computer science education
Gluga, R., Kay, J., Lister, R., Simon, & Kleitman, S. (2013) · Computer Science Education · doi:10.1080/08993408.2013.768830 · Cited by 25 · cited 1×
“Of the theoretical tools that have been used in CER for categorizing activities by complexity, Bloom’s Taxonomy [5] is probably themost common [7, 29, 52].”
MethodCognitive architectures: a way forward for the psychology of programming
Hansen, M. E., Lumsdaine, A., & Goldstone, R. L. (2012) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/2384592.2384596 · Cited by 22 · cited 1×
“[8, 32] conceptualized CCM, a tentative framework for measuring complexity, which resembles ours in its goals and in that it, too, draws on schema theory and related findings from cognitive psychology.”
BackgroundClassifying the tools of contextualized programming education and forms of media computation
Lukkarinen, A., & Sorva, J. (2016) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2999541.2999551 · Cited by 19 · cited 1×
“The complexity of a programming task is additionally influenced by factors such as task presentation [70], contextualization [6, 49]”
BackgroundSelf-citationNovice Difficulties with Interleaved Pattern Composition
Ginat, D., Menashe, E., & Taya, A. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-36617-8_5 · Cited by 16 · cited 1×
“There is also evidence that students mistakenly concatenate plans instead of merging and generally struggle with plans which interact in complex ways [28], that students sometimes read merged plans as if they were composed sequentially [30], and that a student’s plan composition strategy interacts with the”
BackgroundSometimes, Rainfall Accumulates: Talk-Alouds with Novice Functional Programmers
Fisler, K., & Castro, F. E. V. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106183 · Cited by 15 · cited 1×
“Each color corresponds to one of the main high-level plans identified in other work on the same problem [26, 69]: iteration (light blue), a sum (red), read (light green), average (purple), count (brown), guard against negative (dark blue), guard against division by zero (orange) and sentinel (light green).”
BackgroundForensic experts' perceptions of expert bias
Commons, M. L., Miller, P. M., Li, Y., & Gutheil, T. G. (2012) · International Journal of Law and Psychiatry · doi:10.1016/j.ijlp.2012.09.016 · Cited by 14
Bug Infestation! A Goal-Plan Analysis of CS2 Students' Recursive Binary Tree Solutions
Murphy, L., Fitzgerald, S., Grissom, S., & McCauley, R. A. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677232 · Cited by 12 · cited 1×
“A number of studies over the past few decades have investigated bugs in student code and mapped the bugs to the plans in which they appear [23, 25, 36, 59, 69].”
BackgroundSorva, J., & Vihavainen, A. (2016) · ACM Inroads · doi:10.1145/2950065 · Cited by 12 · cited 2×
“We compare two solutions that differ in plan composition: a merged-plans solution (Figure 4) and a sequenced-plans solution (Figure 5), both adapted from [76].”
BackgroundMethodSelf-citationUsing software metrics to predict the difficulty of code writing questions
Elnaffar, S. (2016) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon.2016.7474601 · Cited by 12 · cited 1×
“In a related line of CER, surface-level metrics from Software Engineering such as Cyclomatic Complexity [53], lines of code, and block depth have been applied to codecomprehension tasks [38], code-writing tasks [24], and multiple-”
MethodHierarchical complexity in physics.
Stålne, K., Commons, M. L., & Li, Y. (2014) · Behavioral Development · doi:10.1037/h0100591 · Cited by 9 · cited 1×
“The MHC has been used for complexity analysis in diverse domains such as physics [78], bias in forensics [14], chemistry [3], and student competence in graduate courses [56].”
MethodReport on the Eighth BRACElet workshop: BRACElet technical report 01/08 AUT university, Auckland
Clear, T., Philpott, A., Simon, N. P., & Robbins, P. (2009) · Journal of applied computing and information technology · Cited by 8
Miller, P. M., & Crone‐Todd, D. E. (2016) · Behavioral Development · doi:10.1037/bdb0000039 · Cited by 3 · cited 1×
“The MHC has been used for complexity analysis in diverse domains such as physics [78], bias in forensics [14], chemistry [3], and student competence in graduate courses [56].”
Method
Research works printed but not linked: 9
- Michael Lamport Commons, JA Rodriguez, PM Miller, SN Ross, A LoCicero, EA Goodheart, and D Danaher-Gilpin . 2007. Applying the model of hierarchical complexity. Unpublished manuscript (2007).
- 10.5555/2483716.2483726 · doi:10.5555/2483716.2483726
- 10.5555/2483716.2483722 · doi:10.5555/2483716.2483722
- W. Lewis Johnson, Elliot Soloway, Benjamin Cutler, and Steven Draper . 1983. Bug Catalogue: I. Technical Report. Yale University, YaleU/CSD/RR \#286.
- Sofia Leite Jose Carlos Principe Antonio Carlos Silva Joao Marques-Teixeira Michael L Commons and Pedro Rodrigues . In Review. Connectionist models capture hierarchical complexity transitions in development: discrimination between memory-based and operationally-based transitions. (. In Review).
- 10.5555/2459936.2459938 · doi:10.5555/2459936.2459938
- 10.1037/0278-7393.28.3.411 · DOI · Paper page
- 10.1207/s15516709cog1303_3 · doi:10.1207/s15516709cog1303_3 · Paper page
- Yvonne Rogers . 2004. New theoretical approaches for HCI. Annual review of information science and technology Vol. 38, 1 (2004), 87--143.
Cited by
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366
“Cognitive load theory defines the demand that a situation or task places on a learner’s working memory [84] and is based in part on the learner’s prior knowledge and the complexity of the task or material [22].”
Izu, C., Schulte, C., Aggarwal, A., Cutts, Q., Duran, R., Gutica, M., Heinemann, B., Kraemer, E., Lonati, V., Mirolo, C., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372501 · Cited by 97
“Du-ran et al. [34], in particular, investigated how different program structures and plan-composition strategies could lead to students’ distinct perceptions of difficulty.”
BackgroundSelf-citationCognitive Load Theory in Computing Education Research: A Review
Duran, R., Zavgorodniaia, A., & Sorva, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3483843 · Cited by 92
“Another is the work of Duran et al. [18], who proposed (but did not empirically evaluate) an approach for analyzing the complexity of computer programs in terms of the hierarchical schemas required to understand the programs.”
BackgroundSelf-citationParsons Problems and Beyond: Systematic Literature Review and Empirical Study Designs
Ericson, B. J., Denny, P., Prather, J., Duran, R., Hellas, A., Leinonen, J., Miller, C. S., Morrison, B. B., Pearce, J. L., & Rodger, S. H. (2022) · Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3571785.3574127 · Cited by 56
“The difficulty of the material or task depends on the learner’s prior knowledge and the complexity of the task [49].”
BackgroundSelf-citationThrough (Tracking) Their Eyes: Abstraction and Complexity in Program Comprehension
Kather, P., Duran, R., & Vahrenhold, J. (2021) · ACM Transactions on Computing Education · doi:10.1145/3480171 · Cited by 34
“[15] to differentiate and evaluate the complexity and difficulty of comprehending programs (see the left part of Figure 1).”
BackgroundInfluentialSelf-citation
Metadata
- Authors
- Duran, R., Sorva, J., & Leite, S.
- Affiliations
- Aalto University (FI)
- Universidade do Porto (PT)
- Venue
- Proceedings of the 2018 ACM Conference on International Computing Education Research
- Publisher
- ACM
- Published
- 2018-08-08
- Pages
- 21-30
- OpenAlex
- W2885150285
- Cited by
- 44
- References
- 84
- Citations by year
- 2026: 4 · 2025: 3 · 2024: 4 · 2023: 9 · 2022: 10 · 2021: 6 · 2020: 4 · 2019: 4
- FWCI
- 3.96
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion)
- Funders
- Conselho Nacional de Desenvolvimento Científico e Tecnológico (201365/2015-4)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- software complexity, cognitive load theory, cognitive schemas, program composition, novice programmers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.