An Empirical Investigation into Programming Language Syntax
Abstract
Recent studies in the literature have shown that syntax remains a significant barrier to novice computer science students in the field. While this syntax barrier is known to exist, whether and how it varies across programming languages has not been carefully investigated. For this article, we conducted four empirical studies on programming language syntax as part of a larger analysis into the, so called, programming language wars. We first present two surveys conducted with students on the intuitiveness of syntax, which we used to garner formative clues on what words and symbols might be easy for novices to understand. We followed up with two studies on the accuracy rates of novices using a total of six programming languages: Ruby, Java, Perl, Python, Randomo, and Quorum. Randomo was designed by randomly choosing some keywords from the ASCII table (a metaphorical placebo). To our surprise, we found that languages using a more traditional C-style syntax (both Perl and Java) did not afford accuracy rates significantly higher than a language with randomly generated keywords, but that languages which deviate (Quorum, Python, and Ruby) did. These results, including the specifics of syntax that are particularly problematic for novices, may help teachers of introductory programming courses in choosing appropriate first languages and in helping students to overcome the challenges they face with syntax.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Instructors spend a considerable amount of time helping novices with cryptic error messages.
“Instructors report that they spend a considerable amount of time helping novices with these often cryptic messages [15,31,32,37].”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [37]
Cites
Kiczales, G. (1996) · ACM Computing Surveys · doi:10.1145/242224.242420 · Cited by 5,146 · see citing works
Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333 · see citing works
An Easy Guide to Factor Analysis
Kline, P. (2014) · Routledge eBooks · doi:10.4324/9781315788135 · Cited by 3,328 · see citing works
The Scratch Programming Language and Environment
Maloney, J. H., Resnick, M., Rusk, N., Silverman, B., & Eastmond, E. (2010) · ACM Transactions on Computing Education · doi:10.1145/1868358.1868363 · Cited by 1,423 · see citing works
Usability Analysis of Visual Programming Environments: A ‘Cognitive Dimensions’ Framework
Green, T. R. G., & Petre, M. (1996) · Journal of Visual Languages & Computing · doi:10.1006/jvlc.1996.0009 · Cited by 1,252 · see citing works
Pinker, S. A. (1991) · Science · doi:10.1126/science.1857983 · Cited by 1,061 · see citing works
Lowering the barriers to programming
Kelleher, C. L., & Pausch, R. F. (2005) · ACM Computing Surveys · doi:10.1145/1089733.1089734 · Cited by 887 · see citing works
The Cornell program synthesizer
Teitelbaum, T., & Reps, T. (1981) · Communications of the ACM · doi:10.1145/358746.358755 · Cited by 712 · see citing works
The state of the art in end-user software engineering
Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., et al. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922658 · Cited by 633 · see citing works
Stimulus structures and mental representations in expert comprehension of computer programs
Pennington, N. (1987) · Cognitive Psychology · doi:10.1016/0010-0285(87)90007-7 · Cited by 526 · see citing works
Why the high attrition rate for computer science students
Beaubouef, T., & Mason, J. L. (2005) · ACM SIGCSE Bulletin · doi:10.1145/1083431.1083474 · Cited by 362 · see citing works
DrScheme: a programming environment for Scheme
Findler, R. B., Clements, J., Cormac Flanagan, A., Flatt, M., Krishnamurthi, S., Steckler, P. A., & Felleisen, M. (2002) · Journal of Functional Programming · doi:10.1017/s0956796801004208 · Cited by 342 · see citing works
Cognitive strategies and looping constructs
Soloway, E. M., Bonar, J. G., & Ehrlich, K. (1983) · Communications of the ACM · doi:10.1145/182.358436 · Cited by 338 · see citing works
Deissenboeck, F., & Pizka, M. (2006) · Software Quality Journal · doi:10.1007/s11219-006-9219-1 · Cited by 317 · see citing works
Intentional Ignorance: A History of Blind Assessment and Placebo Controls in Medicine
Kaptchuk, T. J. (1998) · Bulletin of the history of medicine · doi:10.1353/bhm.1998.0159 · Cited by 315 · see citing works
Studying the language and structure in non-programmers' solutions to programming problems
Pane, J. F., Ratanamahatana, C. A., & Myers, B. A. (2001) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2000.0410 · Cited by 287 · see citing works
A multi-national study of reading and tracing skills in novice programmers
Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J. E., Sanders, K., Seppälä, O., et al. (2004) · ACM SIGCSE Bulletin · doi:10.1145/1041624.1041673 · Cited by 284 · see citing works
Comprehension strategies in programming
Pennington, N. (1987) · Ablex Publishing Corp. eBooks · Cited by 280 · see citing works
Natural programming languages and environments
Myers, B. A., Pane, J. F., & Ko, A. J. (2004) · Communications of the ACM · doi:10.1145/1015864.1015888 · Cited by 247 · see citing works
A multi-national study of reading and tracing skills in novice programmers
Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J. E., Sanders, K., Seppälä, O., et al. (2004) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1044550.1041673 · Cited by 186 · see citing works
Tew, A. E., & Guzdial, M. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953200 · Cited by 172 · see citing works
Hubert, L. J. (1977) · Psychological Bulletin · doi:10.1037/0033-2909.84.2.289 · Cited by 153 · see citing works
All syntax errors are not equal
Denny, P., Luxton-Reilly, A., & Tempero, E. (2012) · Proceedings of the 17th ACM annual conference on Innovation and technology in computer science education · doi:10.1145/2325296.2325318 · Cited by 144 · see citing works
The paradoxical success of aspect-oriented programming
Steimann, F. (2006) · ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA) · doi:10.1145/1167473.1167514 · Cited by 139 · see citing works
Pothier, G., Tanter, É., & Piquer, J. (2007) · ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA) · doi:10.1145/1297027.1297067 · Cited by 135 · see citing works
Understanding the syntax barrier for novices
Denny, P., Luxton-Reilly, A., Tempero, E., & Hendrickx, J. (2011) · Proceedings of the 16th annual joint conference on Innovation and technology in computer science education · doi:10.1145/1999747.1999807 · Cited by 133 · see citing works
Is transactional programming actually easier?
Rossbach, C. J., Hofmann, O. S., & Witchel, E. (2010) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/1693453.1693462 · Cited by 129 · see citing works
Color and sound in algorithm animation
Brown, M. H., & Hershberger, J. E. (1992) · Computer · doi:10.1109/2.179117 · Cited by 128 · see citing works
Finding causes of program output with the Java Whyline
Ko, A. J., & Myers, B. A. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518942 · Cited by 123 · see citing works
Dann, W. P., Cosgrove, D., Slater, D., Culyba, D., & Cooper, S. (2012) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2157136.2157180 · Cited by 111 · see citing works
Usability Implications of Requiring Parameters in Objects' Constructors
Stylos, J., & Clarke, S. (2007) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2007.92 · Cited by 110 · see citing works
Measuring programming experience
Feigenspan, J., Kästner, C., Liebig, J., Apel, S., & Hanenberg, S. (2012) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2012.6240511 · Cited by 106 · see citing works
The Turing programming language
Holt, R. C., & Cordy, J. R. (1988) · Communications of the ACM · doi:10.1145/53580.53581 · Cited by 103 · see citing works
The implications of method placement on API learnability
Stylos, J., & Myers, B. A. (2008) · ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE) · doi:10.1145/1453101.1453117 · Cited by 102 · see citing works
Deisenbock, F., & Pizka, M. (2005) · IEEE International Workshop on Program Comprehension (IWPC) · doi:10.1109/wpc.2005.14 · Cited by 99 · see citing works
Keynote Address: .QL for Source Code Analysis
de Moor, O., Verbaere, M., Hajiyev, E., Avgustinov, P., Ekman, T., Ongkingco, N., Sereni, D., & Tibble, J. (2007) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2007.31 · Cited by 99 · see citing works
Stefik, A., Hundhausen, C., & Smith, D. W. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953323 · Cited by 96 · see citing works
Self-citationCleary, B., Exton, C., Buckley, J., & English, M. (2008) · Empirical Software Engineering · doi:10.1007/s10664-008-9095-3 · Cited by 86 · see citing works
Program understanding behavior during debugging of large scale software
von Mayrhauser, A., & Vans, A. M. (1997) · doi:10.1145/266399.266414 · Cited by 86 · see citing works
Uncovering principles of novice programming
Bonar, J. G., & Soloway, E. M. (1983) · ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages (POPL) · doi:10.1145/567067.567069 · Cited by 84 · see citing works
An experiment about static and dynamic type systems
Hanenberg, S. (2010) · ACM International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA) · doi:10.1145/1869459.1869462 · Cited by 82 · see citing works
Stefik, A., Hundhausen, C. D., & Patterson, R. (2011) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2011.07.002 · Cited by 78 · see citing works
Self-citationDo static type systems improve the maintainability of software systems? An empirical study
Kleinschmager, S., Robbes, R., Stefik, A., Hanenberg, S., & Tanter, E. (2012) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2012.6240483 · Cited by 73 · see citing works
Self-citationVickers, P., & Alty, J. (2002) · Interacting with Computers · doi:10.1016/s0953-5438(02)00026-7 · Cited by 71 · see citing works
An empirical study of novice program comprehension in the imperative and object-oriented styles
Ramalingam, V., & Wiedenbeck, S. (1997) · doi:10.1145/266399.266411 · Cited by 70 · see citing works
Blind learners programming through audio
Sánchez, J., & Aguayo, F. (2005) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1056808.1057018 · Cited by 70 · see citing works
Cooper, S. (2010) · ACM Transactions on Computing Education · doi:10.1145/1868358.1868362 · Cited by 63 · see citing works
An empirical study of the influence of static type systems on the usability of undocumented software
Mayer, C., Hanenberg, S., Robbes, R., Tanter, É., & Stefik, A. (2012) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2384616.2384666 · Cited by 62 · see citing works
Self-citationEmpirical studies on programming language stimuli
Stefik, A., & Gellenbeck, E. (2010) · Software Quality Journal · doi:10.1007/s11219-010-9106-7 · Cited by 59 · see citing works
Self-citationAn empirical investigation of the influence of a type of side effects on program comprehension
Dolado, J. J., Harman, M., Otero, M. C., & Hu, L. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1214329 · Cited by 57 · see citing works
Can direct manipulation lower the barriers to computer programming and promote transfer of training?
Hundhausen, C. D., Farley, S. F., & Brown, J. L. (2009) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1592440.1592442 · Cited by 55 · see citing works
Inspiring blind high school students to pursue computer science with instant messaging chatbots
Bigham, J. P., Aller, M. B., Brudvik, J. T., Leung, J. O., Yazzolino, L. A., & Ladner, R. E. (2008) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1352135.1352287 · Cited by 48 · see citing works
Hanenberg, S., Kleinschmager, S., & Josupeit-Walter, M. (2009) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2009.5316028 · Cited by 46 · see citing works
Hanenberg, S. (2010) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/1869459.1869536 · Cited by 46 · see citing works
Holt, R. C., Wortman, D. B., Barnard, D. T., & Cordy, J. R. (1977) · Communications of the ACM · doi:10.1145/359581.359586 · Cited by 42 · see citing works
Nonvisual tool for navigating hierarchical structures
Smith, A. C., Cook, J. S., Francioni, J. M., Hossain, A., Anwar, M. F. M., & Rahman, M. F. (2003) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1028630.1028654 · Cited by 42 · see citing works
Garlick, R., & Cankaya, E. C. (2010) · doi:10.1145/1822090.1822138 · Cited by 41 · see citing works
Python CS1 as preparation for C++ CS2
Enbody, R. J., Punch, W. F., & McCullen, M. (2009) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1508865.1508907 · Cited by 38 · see citing works
Lopes, C. V., Dourish, P., Lorenz, D. H., & Lieberherr, K. J. (2003) · Companion to the ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA Companion) · doi:10.1145/949344.949400 · Cited by 37 · see citing works
Spoken Language Support for Software Development
Begel, A. (2005) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2004.49 · Cited by 32 · see citing works
Colour and sound in algorithm animation
Brown, M. H., & Hershberger, J. E. (2002) · IEEE Workshop on Visual Languages (VL) · doi:10.1109/wvl.1991.238856 · Cited by 32 · see citing works
Do Programming Languages Affect Productivity? A Case Study Using Data from Open Source Projects
Delorey, D. P., Knutson, C. D., & Chun, S. B.-S. (2007) · doi:10.1109/floss.2007.5 · Cited by 32 · see citing works
Is transactional programming actually easier?
Rossbach, C. J., Hofmann, O. S., & Witchel, E. (2010) · ACM SIGPLAN Notices · doi:10.1145/1837853.1693462 · Cited by 32 · see citing works
WAD: A Feasibility study using the Wicked Audio Debugger
Stefik, A., Alexander, R., Patterson, R., & Brown, J. (2007) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2007.42 · Cited by 31 · see citing works
Self-citationMeyerovich, L. A., & Rabkin, A. S. (2012) · doi:10.1145/2384592.2384597 · Cited by 30 · see citing works
Binkley, D., Davis, M. H., Lawrie, D., & Morrell, C. H. (2019) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2019.00035 · Cited by 30 · see citing works
Deltatalk: An Empirically and Aesthetically Motivated Simplification of the Smalltalk-80 Language
Borning, A. H., & O’Shea, T. (1987) · Lecture notes in computer science · doi:10.1007/3-540-47891-4_1 · Cited by 29 · see citing works
Analysing novices analysing Prolog: what stories do novices tell themselves about Prolog?
Taylor, J. (1990) · Instructional Science · doi:10.1007/bf00116442 · Cited by 29 · see citing works
Whitney, P. J. (1997) · Medical Entomology and Zoology · Cited by 28 · see citing works
Staking claims: a history of programming language design claims and evidence
Markstrum, S. A. (2010) · Evaluation and Usability of Programming Languages and Tools · doi:10.1145/1937117.1937124 · Cited by 28 · see citing works
Stefik, A., Siebert, S., Stefik, M., & Slattery, K. (2011) · doi:10.1145/2089155.2089159 · Cited by 24 · see citing works
Self-citationNovices' expectations and prior knowledge of software development
Schulte, C., & Magenheim, J. (2005) · doi:10.1145/1089786.1089800 · Cited by 23 · see citing works
Performance of python CS1 students in mid-level non-python CS courses
Enbody, R. J., & Punch, W. F. (2010) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1734263.1734437 · Cited by 23 · see citing works
Syntactic and semantic issues in introductory programming education
McIver, L. (2017) · Figshare · doi:10.4225/03/59cb52174f8e2 · Cited by 22 · see citing works
Endrikat, S., & Hanenberg, S. (2011) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2011.46 · Cited by 22 · see citing works
Using events to debug Java programs backwards in time
Lewis, B., & Ducassé, M. (2003) · doi:10.1145/949344.949367 · Cited by 18 · see citing works
More natural end-user software engineering
Myers, B. A., Ko, A. J., Park, S. Y., Stylos, J., LaToza, T. D., & Beaton, J. (2008) · International Workshop on End-User Software Engineering (WEUSE) · doi:10.1145/1370847.1370854 · Cited by 16 · see citing works
Strategic Software Development: Productivity Comparisons Of General Development Programs
Comstock, C., Jiang, Z., & Naudé, P. (2007) · doi:10.5281/zenodo.1062877 · Cited by 11 · see citing works
Using spoken text to aid debugging: An empirical study
Stefik, A., & Gellenbeck, E. (2009) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2009.5090034 · Cited by 11 · see citing works
Self-citationUses of the LOGO programming language in undergraduate instruction
Lukas, G. (1972) · ACM Annual Conference · doi:10.1145/800194.805908 · Cited by 10 · see citing works
Mining Programming Language Vocabularies from Source Code.
Delorey, D. P., Knutson, C. D., & Davies, M. (2009) · Cited by 9 · see citing works
Pausch, R. F. (2008) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1352135.1352137 · Cited by 8 · see citing works
The linguistics of object-oriented design
Holmboe, C. (2005) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1067445.1067498 · Cited by 4 · see citing works
Understanding programmer language
Høst, E. W. (2007) · doi:10.1145/1297846.1297957 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Introductory programming: a systematic literature review
Luxton-Reilly, A., Simon, Al-Bluwi, I., Becker, B. A., Giannakos, M. N., Kumar, A. N., Ott, L. M., Paterson, J. H., Scott, M., Sheard, J., et al. (2018) · doi:10.1145/3293881.3295779 · Cited by 511 · see citing works
Bau, D., Gray, J., Kelleher, C. L., Sheldon, J., & Turbak, F. A. (2017) · Communications of the ACM · doi:10.1145/3015455 · Cited by 326 · see citing works
Altadmri, A., & Brown, N. C. C. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677258 · Cited by 261 · see citing works
Learner-Centered Design of Computing Education: Research on Computing for Everyone
Guzdial, M. (2015) · Synthesis lectures on human-centered informatics · doi:10.2200/s00684ed1v01y201511hci033 · Cited by 252 · see citing works
Using Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235 · see citing works
Becker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199 · see citing works
Weintrop, D., & Wilensky, U. (2015) · Proceedings of the eleventh annual International Conference on International Computing Education Research · doi:10.1145/2787622.2787721 · Cited by 198 · see citing works
Replication, Validation, and Use of a Language Independent CS1 Knowledge Assessment
Parker, M. C., Guzdial, M., & Engelman, S. (2016) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2960310.2960316 · Cited by 185 · see citing works
Teaching Introductory Programming
Koulouri, T., Lauria, S., & Macredie, R. D. (2014) · ACM Transactions on Computing Education · doi:10.1145/2662412 · Cited by 182 · see citing works
Novice Programmers and Introductory Programming
Robins, A. V. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.013 · Cited by 183 · see citing works
Learnable Programming: Blocks and BeyondPreprint
Bau, D., Gray, J., Kelleher, C. L., Sheldon, J., & Turbak, F. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09413 · Cited by 168 · see citing works
Non-Native English Speakers Learning Computer Programming
Guo, P. J. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173970 · Cited by 145 · see citing works
The Cambridge Handbook of Computing Education Research
(2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555 · Cited by 130 · see citing works
An Effective Approach to Enhancing Compiler Error Messages
Becker, B. A. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844584 · Cited by 125 · see citing works
A Comparative Study of Programming Languages in Rosetta Code
Nanz, S., & Furia, C. A. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.90 · Cited by 111 · see citing works
Koushik, V., Guinness, D., & Kane, S. K. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300722 · Cited by 91 · see citing works
Investigating novice programming mistakes: educator beliefs vs. student data
Brown, N. C. C., & Altadmri, A. (2014) · Proceedings of the tenth annual conference on International computing education research · doi:10.1145/2632320.2632343 · Cited by 89 · see citing works
Investigating Static Analysis Errors in Student Java Programs
Edwards, S. H., Kandru, N., & Rajagopal, M. B. M. M. (2017) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3105726.3106182 · Cited by 87 · see citing works
How do API documentation and static typing affect API usability?
Endrikat, S., Hanenberg, S., Robbes, R., & Stefik, A. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568299 · Cited by 86 · see citing works
Self-citationCurzon, P., Bell, T., Waite, J., & Dorling, M. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.018 · Cited by 80 · see citing works
The Language of Programming: A Cognitive Perspective
Fedorenko, E. G., Ivanova, A. A., Dhamala, R., & Bers, M. U. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.04.010 · Cited by 75 · see citing works
A Comparative Study of Programming Languages in Rosetta Code
Nanz, S., & Furia, C. A. (2015) · Cited by 74 · see citing works
Errors and Complications in SQL Query Formulation
Taipalus, T., Siponen, M. T., & Vartiainen, T. (2018) · ACM Transactions on Computing Education · doi:10.1145/3231712 · Cited by 74 · see citing works
Understanding Programming Expertise
Crk, I., Kluthe, T., & Stefik, A. (2015) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2829945 · Cited by 71 · see citing works
Self-citationAn empirical study on the impact of C++ lambdas and programmer experience
Uesbeck, P. M., Stefik, A., Hanenberg, S., Pedersen, J. B., & Daleiden, P. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884849 · Cited by 67 · see citing works
Self-citationThe Compound Nature of Novice Programming Assessments
Luxton-Reilly, A., & Petersen, A. (2017) · doi:10.1145/3013499.3013500 · Cited by 66 · see citing works
Programming Paradigms and Beyond
Krishnamurthi, S., & Fisler, K. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.014 · Cited by 65 · see citing works
Hermans, F. (2020) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3372782.3406262 · Cited by 63 · see citing works
Developing Assessments to Determine Mastery of Programming Fundamentals
Luxton-Reilly, A., Becker, B. A., Cao, Y., McDermott, R., Mirolo, C., Mühling, A., Petersen, A., Sanders, K., Simon, & Whalley, J. L. (2018) · doi:10.1145/3174781.3174784 · Cited by 62 · see citing works
Curriculum, Teaching and Learning, and Assessments for Introductory Programming Course
Mehmood, E., Abid, A., Farooq, M. S., & Nawaz, N. A. H. (2020) · IEEE Access · doi:10.1109/access.2020.3008321 · Cited by 66 · see citing works
Horn, M., & Bers, M. U. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.023 · Cited by 62 · see citing works
On Designing Programming Error Messages for Novices: Readability and its Constituent Factors
Denny, P., Prather, J., Becker, B. A., Mooney, C., Homer, J., Albrecht, Z. C., & Powell, G. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445696 · Cited by 60 · see citing works
The History of Computing Education Research
Guzdial, M., & du Boulay, B. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.002 · Cited by 57 · see citing works
Understanding beginners' mistakes with Haskell
Tirronen, V., UUSI-MÄKELÄ, S., & Isomöttönen, V. (2015) · Journal of Functional Programming · doi:10.1017/s0956796815000179 · Cited by 56 · see citing works
Stefik, A., & Hanenberg, S. (2014) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/2661136.2661156 · Cited by 56 · see citing works
Self-citationOut of our minds: a review of sociocultural cognition theory
Tenenberg, J., & Knobelsdorf, M. (2014) · Computer Science Education · doi:10.1080/08993408.2013.869396 · Cited by 55 · see citing works
Evaluating Code Readability and Legibility: An Examination of Human-centric Studies
Oliveira, D., Bruno, R., Madeiral, F., & Castor, F. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00041 · Cited by 54 · see citing works
A Systematic Investigation of Replications in Computing Education Research
Hao, Q., Smith IV, D. H., Iriumi, N., Tsikerdekis, M., & Ko, A. J. (2019) · ACM Transactions on Computing Education · doi:10.1145/3345328 · Cited by 51 · see citing works
Making Programming Accessible to Learners with Visual Impairments: A Literature Review
Hadwen-Bennett, A., Sentance, S., & Morrison, C. P. (2018) · International Journal of Computer Science Education in Schools · doi:10.21585/ijcses.v2i2.25 · Cited by 51 · see citing works
Prasad, P., & Sane, A. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630828 · Cited by 60 · see citing works
Teachers’ Perceptions of Student Misconceptions in Introductory Programming
Qian, Y., Hambrusch, S. E., Yadav, A. K., Gretter, S., & Li, Y. (2019) · Journal of Educational Computing Research · doi:10.1177/0735633119845413 · Cited by 46 · see citing works
Vogel, S., Hoadley, C. P., Castillo, A. R. M., & Ascenzi‐Moreno, L. (2020) · Computer Science Education · doi:10.1080/08993408.2020.1751525 · Cited by 48 · see citing works
Towards an Analysis of Program Complexity From a Cognitive Perspective
Duran, R., Sorva, J., & Leite, S. (2018) · Proceedings of the 2018 ACM Conference on International Computing Education Research · doi:10.1145/3230977.3230986 · Cited by 44 · see citing works
Hazelnut: a bidirectionally typed structure editor calculus
Omar, C., Voysey, I., Hilton, M. L., Aldrich, J., & Hammer, M. A. (2016) · ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) · doi:10.1145/3009837.3009900 · Cited by 42 · see citing works
Aggregate Compilation Behavior
Jadud, M. C., & Dorn, B. (2015) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2787622.2787718 · Cited by 41 · see citing works
Computer Science Principles for Teachers of Blind and Visually Impaired Students
Stefik, A., Ladner, R. E., Allee, W., & Mealin, S. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287453 · Cited by 39 · see citing works
Self-citationCognitive Sciences for Computing Education
Robins, A. V., Margulieux, L. E., & Morrison, B. B. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.010 · Cited by 39 · see citing works
An Analysis of HTML and CSS Syntax Errors in a Web Development Course
Park, T. H., Dorn, B., & Forte, A. (2015) · ACM Transactions on Computing Education · doi:10.1145/2700514 · Cited by 40 · see citing works
Evaluation of a Frame-based Programming Editor
Price, T., Brown, N. C. C., Lipovac, D., Barnes, T. M., & Kölling, M. (2016) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2960310.2960319 · Cited by 36 · see citing works
Computing EducationLiterature Review and Voices from the Field
Blikstein, P., & Moghadam, S. H. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.004 · Cited by 36 · see citing works
Edwards, J., Ditton, J., Trninić, D., Swanson, H., Sullivan, S., & Mano, C. D. (2020) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3372782.3406259 · Cited by 34 · see citing works
The Effects of Compilation Mechanisms and Error Message Presentation on Novice Programmer Behavior
Karvelas, I., Li, A. D. R., & Becker, B. A. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366882 · Cited by 33 · see citing works
Learning Sciences for Computing Education
Margulieux, L. E., Dorn, B., & Searle, K. A. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.009 · Cited by 32 · see citing works
Lewis, C. M., Shah, N., & Falkner, K. E. K. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.017 · Cited by 33 · see citing works
Espinal, A., Vieira, C., & Guerrero-Bequis, V. (2022) · Computer Science Education · doi:10.1080/08993408.2022.2079867 · Cited by 31 · see citing works
AudioHighlight: Code Skimming for Blind Programmers
Armaly, A., Rodeghero, P., & McMillan, C. (2018) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2018.00030 · Cited by 30 · see citing works
Ladner, R. E., & Stefik, A. (2017) · ACM SIGACCESS Accessibility and Computing · doi:10.1145/3124144.3124145 · Cited by 29 · see citing works
Self-citationAPI Blindspots: Why Experienced Developers Write Vulnerable Code
Oliveira, D. S., Tian, L., Rahman, M. S., Akefirad, R., Ellis, D. M., Perez, E., Bobhate, R., DeLong, L. A., Cappos, J., Brun, Y., et al. (2018) · Cited by 29 · see citing works
Understanding Conceptual Transfer for Students Learning New Programming Languages
Tshukudu, E., & Cutts, Q. I. (2020) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3372782.3406270 · Cited by 29 · see citing works
Learning and programming challenges of rust
Zhu, S., Zhang, Z., Qin, B., Xiong, A., & Song, L. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510164 · Cited by 29 · see citing works
Ehtesham-Ul-Haque, M., Monsur, S. M., & Billah, S. M. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545620 · Cited by 29 · see citing works
A Competency Structure Model of Object-Oriented Programming
Kramer, M., Hubwieser, P., & Brinda, T. (2016) · International Conference on Learning and Teaching in Computing and Engineering (LaTICE) · doi:10.1109/latice.2016.24 · Cited by 28 · see citing works
Teacher Learning and Professional Development
Fincher, S. A., Kolikant, Y. B., & Falkner, K. E. K. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.026 · Cited by 28 · see citing works
Motivation, Attitudes, and Dispositions
Lishinski, A., & Yadav, A. K. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.029 · Cited by 27 · see citing works
Coblenz, M. J., Oei, R., Etzel, T., Koronkevich, P., Baker, M., Bloem, Y., Myers, B. A., Sunshine, J., & Aldrich, J. (2020) · ACM Transactions on Programming Languages and Systems · doi:10.1145/3417516 · Cited by 27 · see citing works
Falkner, K. E. K., & Sheard, J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.016 · Cited by 27 · see citing works
Experiences With Efficient Methodologies for Teaching Computer Programming to Geoscientists
Jacobs, C. T., Gorman, G., Rees, H. E., & Craig, L. E. (2016) · Journal of Geoscience Education · doi:10.5408/15-101.1 · Cited by 25 · see citing works
Spiza, S., & Hanenberg, S. (2014) · International Conference on Modularity (MODULARITY) · doi:10.1145/2577080.2577098 · Cited by 26 · see citing works
Kölling, M., Brown, N. C. C., & Altadmri, A. (2017) · Journal of Visual Languages and Sentient Systems · doi:10.18293/vlss2017-009 · Cited by 26 · see citing works
Kane, S. K., Koushik, V., & Muehlbradt, A. (2018) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3202185.3202754 · Cited by 26 · see citing works
Programs in the Palm of your Hand
Cabrera, L., Maloney, J. H., & Weintrop, D. (2019) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3311927.3323138 · Cited by 26 · see citing works
Native Language's Effect on Java Compiler Errors
Reestman, K., & Dorn, B. (2019) · Proceedings of the 2019 ACM Conference on International Computing Education Research · doi:10.1145/3291279.3339423 · Cited by 25 · see citing works
Koushik, V., & Kane, S. K. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300744 · Cited by 25 · see citing works
A Controlled Experiment on Python vs C for an Introductory Programming Course
Wainer, J., & Xavier, E. C. (2018) · ACM Transactions on Computing Education · doi:10.1145/3152894 · Cited by 24 · see citing works
Qualitative Methods for Computing Education
Tenenberg, J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.008 · Cited by 24 · see citing works
Fowler, M., & Zilles, C. B. (2021) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3408877.3432413 · Cited by 24 · see citing works
Learning From Errors: Exploring the Effectiveness of Enhanced Error Messages in Learning to Program
Zhou, Z., Wang, S.-J., & Qian, Y. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.768962 · Cited by 24 · see citing works
A Comparison of Program Comprehension Strategies by Blind and Sighted Programmers
Armaly, A., Rodeghero, P., & McMillan, C. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2729548 · Cited by 24 · see citing works
Enhancing Python Compiler Error Messages via Stack
Thiselton, E., & Treude, C. (2019) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2019.8870155 · Cited by 24 · see citing works
Evidence-based programming language design : a philosophical and methodological exploration
Kaijanaho, A.-J. (2015) · Jyväskylä University Digital Archive (University of Jyväskylä) · Cited by 24 · see citing works
Haden, P. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.006 · Cited by 26 · see citing works
Do developers benefit from generic types?
Hoppe, M., & Hanenberg, S. (2013) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2509136.2509528 · Cited by 23 · see citing works
An Exploration of the Effects of Enhanced Compiler Error Messages for Computer Programming Novices
Becker, B. A. (2015) · Arrow - TU Dublin (Technological University Dublin) · doi:10.13140/rg.2.2.26637.13288 · Cited by 22 · see citing works
(2020) · Durham Research Online (Durham University) · Cited by 21 · see citing works
Semantic Transfer in Programming Languages: Exploratory Study of Relative Novices
Tshukudu, E., & Cutts, Q. I. (2020) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3341525.3387406 · Cited by 21 · see citing works
Penney, J., Pimentel, J. F., Steinmacher, I., & Gerosa, M. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-54975-5_12 · Cited by 19 · see citing works
Weintrop, D., Killen, H., Munzar, T., & Franke, B. (2019) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3287324.3287348 · Cited by 21 · see citing works
Stefik, A., Hanenberg, S., McKenney, M., Andrews, A. K. A., Yellanki, S. K., & Siebert, S. (2014) · International Conference on Program Comprehension · doi:10.1145/2597008.2597154 · Cited by 20 · see citing works
Self-citationLearning programming, syntax errors and institution-specific factors
Ahadi, A., Lister, R., Lal, S., & Hellas, A. (2018) · Australasian Computing Education Conference · doi:10.1145/3160489.3160490 · Cited by 20 · see citing works
Methodological Irregularities in Programming-Language Research
Stefik, A., & Hanenberg, S. (2017) · Computer · doi:10.1109/mc.2017.3001257 · Cited by 19 · see citing works
Self-citationKohn, T., & Manaris, B. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366920 · Cited by 18 · see citing works
Can we crowdsource language design?
Wilson, P. T., Pombrio, J. L., & Krishnamurthi, S. (2017) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3133850.3133863 · Cited by 18 · see citing works
Block-Based Development of Mobile Learning Experiences for the Internet of Things
Ruiz‐Rube, I., Mota, J. M., Person, T., Corral, J. M. R., & Dodero, J. M. (2019) · Sensors · doi:10.3390/s19245467 · Cited by 18 · see citing works
Malmi, L., Utting, I., & Ko, A. J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.022 · Cited by 17 · see citing works
The why, when, and how of computing in biology classrooms
Wright, A. M., Schwartz, R. S., Oaks, J. R., Newman, C. E., & Flanagan, S. P. (2020) · F1000Research · doi:10.12688/f1000research.20873.2 · Cited by 17 · see citing works
Software Analytics to Support Students in Object-Oriented Programming Tasks: An Empirical Study
Ardimento, P., Bernardi, M. L., & Cimitile, M. (2020) · IEEE Access · doi:10.1109/access.2020.3010172 · Cited by 17 · see citing works
Learning Outside the Classroom
Begel, A., & Ko, A. J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.027 · Cited by 16 · see citing works
The Cost of Syntax and How to Avoid It: Text versus Frame-Based Editing
Altadmri, A., Kölling, M., & Brown, N. C. C. (2016) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2016.204 · Cited by 16 · see citing works
Coblenz, M. J., Aldrich, J., Myers, B. A., & Sunshine, J. (2020) · Proceedings of the ACM on Programming Languages · doi:10.1145/3428200 · Cited by 16 · see citing works
Separation of syntax and problem solving in Introductory Computer Programming
Edwards, J., Fulton, E. K., Holmes, J. D., Valentin, J. L., Beard, D. V., & Parker, K. R. (2018) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2018.8658852 · Cited by 17 · see citing works
- Authors
- Stefik, A., & Siebert, S.
- Affiliations
- University of Nevada, Las Vegas (US)
- Washington University in St. Louis (US)
- ORCID
- Stefik, Andreas
- Venue
- ACM Transactions on Computing Education
- Published
- 2013-11
- Volume
- 13
- Issue
- 4
- Pages
- 1-40
- ISSN
- 1946-6226 (ISSN-L)
- OpenAlex
- W2157148654
- Cited by
- 295
- References
- 84
- Citations by year
- 2026: 14 · 2025: 13 · 2024: 21 · 2023: 19 · 2022: 19 · 2021: 19 · 2020: 34 · 2019: 60 · 2018: 23 · 2017: 22 · 2016: 17 · 2015: 16 · 2014: 15 · 2013: 2
- FWCI
- 25.18
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Computing Education (publishedVersion)
- Funders
- Division of Computer and Network Systems (CNS-0940521)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, programming, and type systems
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- programming language syntax, novice programmers, quorum, Python, ruby, Java, Perl, introductory programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
