The DOI matches its Crossref record; 84 of the 84 references it cites were found in a registry or an index.
An Empirical Investigation into Programming Language Syntax
Abstract
Recent studies in the literature have shown that syntax remains a significant barrier to novice computer science students in the field. While this syntax barrier is known to exist, whether and how it varies across programming languages has not been carefully investigated. For this article, we conducted four empirical studies on programming language syntax as part of a larger analysis into the, so called, programming language wars. We first present two surveys conducted with students on the intuitiveness of syntax, which we used to garner formative clues on what words and symbols might be easy for novices to understand. We followed up with two studies on the accuracy rates of novices using a total of six programming languages: Ruby, Java, Perl, Python, Randomo, and Quorum. Randomo was designed by randomly choosing some keywords from the ASCII table (a metaphorical placebo). To our surprise, we found that languages using a more traditional C-style syntax (both Perl and Java) did not afford accuracy rates significantly higher than a language with randomly generated keywords, but that languages which deviate (Quorum, Python, and Ruby) did. These results, including the specifics of syntax that are particularly problematic for novices, may help teachers of introductory programming courses in choosing appropriate first languages and in helping students to overcome the challenges they face with syntax.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Instructors report that they spend a considerable amount of time helping novices with these often cryptic messages [15,31,32,37].”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [37]
Citations
Cites
Kiczales, G. (1996) · ACM Computing Surveys · doi:10.1145/242224.242420 · Cited by 5,146
An Easy Guide to Factor Analysis
Kline, P. (2014) · Routledge eBooks · doi:10.4324/9781315788135 · Cited by 3,380
Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
The Scratch Programming Language and Environment
Maloney, J. H., Resnick, M., Rusk, N., Silverman, B., & Eastmond, E. (2010) · ACM Transactions on Computing Education · doi:10.1145/1868358.1868363 · Cited by 1,423
Usability Analysis of Visual Programming Environments: A ‘Cognitive Dimensions’ Framework
Green, T. R. G., & Petre, M. (1996) · Journal of Visual Languages & Computing · doi:10.1006/jvlc.1996.0009 · Cited by 1,252
Pinker, S. A. (1991) · Science · doi:10.1126/science.1857983 · Cited by 1,061
Lowering the barriers to programming
Kelleher, C. L., & Pausch, R. F. (2005) · ACM Computing Surveys · doi:10.1145/1089733.1089734 · Cited by 887
The Cornell program synthesizer
Teitelbaum, T., & Reps, T. (1981) · Communications of the ACM · doi:10.1145/358746.358755 · Cited by 712
The state of the art in end-user software engineering
Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., et al. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922658 · Cited by 633
A multi-national study of reading and tracing skills in novice programmers
Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J. E., Sanders, K., Seppälä, O., et al. (2004) · ACM SIGCSE Bulletin · doi:10.1145/1041624.1041673 · Cited by 575
Stimulus structures and mental representations in expert comprehension of computer programs
Pennington, N. (1987) · Cognitive Psychology · doi:10.1016/0010-0285(87)90007-7 · Cited by 526
Why the high attrition rate for computer science students
Beaubouef, T., & Mason, J. L. (2005) · ACM SIGCSE Bulletin · doi:10.1145/1083431.1083474 · Cited by 362
DrScheme: a programming environment for Scheme
Findler, R. B., Clements, J., Cormac Flanagan, A., Flatt, M., Krishnamurthi, S., Steckler, P. A., & Felleisen, M. (2002) · Journal of Functional Programming · doi:10.1017/s0956796801004208 · Cited by 342
Cognitive strategies and looping constructs
Soloway, E. M., Bonar, J. G., & Ehrlich, K. (1983) · Communications of the ACM · doi:10.1145/182.358436 · Cited by 338
Deissenboeck, F., & Pizka, M. (2006) · Software Quality Journal · doi:10.1007/s11219-006-9219-1 · Cited by 317
Intentional Ignorance: A History of Blind Assessment and Placebo Controls in Medicine
Kaptchuk, T. J. (1998) · Bulletin of the history of medicine · doi:10.1353/bhm.1998.0159 · Cited by 315
Studying the language and structure in non-programmers' solutions to programming problems
Pane, J. F., Ratanamahatana, C. A., & Myers, B. A. (2001) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2000.0410 · Cited by 296
Comprehension strategies in programming
Pennington, N. (1987) · Ablex Publishing Corp. eBooks · Cited by 280
Natural programming languages and environments
Myers, B. A., Pane, J. F., & Ko, A. J. (2004) · Communications of the ACM · doi:10.1145/1015864.1015888 · Cited by 247
A multi-national study of reading and tracing skills in novice programmers
Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J. E., Sanders, K., Seppälä, O., et al. (2004) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1044550.1041673 · Cited by 204
Tew, A. E., & Guzdial, M. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953200 · Cited by 172
Hubert, L. J. (1977) · Psychological Bulletin · doi:10.1037/0033-2909.84.2.289 · Cited by 153
All syntax errors are not equal
Denny, P., Luxton-Reilly, A., & Tempero, E. (2012) · Proceedings of the 17th ACM annual conference on Innovation and technology in computer science education · doi:10.1145/2325296.2325318 · Cited by 144
The paradoxical success of aspect-oriented programming
Steimann, F. (2006) · ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA) · doi:10.1145/1167473.1167514 · Cited by 139
Pothier, G., Tanter, É., & Piquer, J. (2007) · ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA) · doi:10.1145/1297027.1297067 · Cited by 135
Understanding the syntax barrier for novices
Denny, P., Luxton-Reilly, A., Tempero, E., & Hendrickx, J. (2011) · Proceedings of the 16th annual joint conference on Innovation and technology in computer science education · doi:10.1145/1999747.1999807 · Cited by 133
Is transactional programming actually easier?
Rossbach, C. J., Hofmann, O. S., & Witchel, E. (2010) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/1693453.1693462 · Cited by 129
Color and sound in algorithm animation
Brown, M. H., & Hershberger, J. E. (1992) · Computer · doi:10.1109/2.179117 · Cited by 128
Finding causes of program output with the Java Whyline
Ko, A. J., & Myers, B. A. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518942 · Cited by 123
Dann, W. P., Cosgrove, D., Slater, D., Culyba, D., & Cooper, S. (2012) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2157136.2157180 · Cited by 111
Usability Implications of Requiring Parameters in Objects' Constructors
Stylos, J., & Clarke, S. (2007) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2007.92 · Cited by 110
Measuring programming experience
Feigenspan, J., Kästner, C., Liebig, J., Apel, S., & Hanenberg, S. (2012) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2012.6240511 · Cited by 106
The Turing programming language
Holt, R. C., & Cordy, J. R. (1988) · Communications of the ACM · doi:10.1145/53580.53581 · Cited by 103
The implications of method placement on API learnability
Stylos, J., & Myers, B. A. (2008) · ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE) · doi:10.1145/1453101.1453117 · Cited by 102
Deisenbock, F., & Pizka, M. (2005) · IEEE International Workshop on Program Comprehension (IWPC) · doi:10.1109/wpc.2005.14 · Cited by 99
Keynote Address: .QL for Source Code Analysis
de Moor, O., Verbaere, M., Hajiyev, E., Avgustinov, P., Ekman, T., Ongkingco, N., Sereni, D., & Tibble, J. (2007) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2007.31 · Cited by 99
Stefik, A., Hundhausen, C., & Smith, D. W. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953323 · Cited by 96
Self-citationUncovering principles of novice programming
Bonar, J. G., & Soloway, E. M. (1983) · ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) · doi:10.1145/567067.567069 · Cited by 89
Cleary, B., Exton, C., Buckley, J., & English, M. (2008) · Empirical Software Engineering · doi:10.1007/s10664-008-9095-3 · Cited by 86
Program understanding behavior during debugging of large scale software
von Mayrhauser, A., & Vans, A. M. (1997) · doi:10.1145/266399.266414 · Cited by 86
An experiment about static and dynamic type systems
Hanenberg, S. (2010) · ACM International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA) · doi:10.1145/1869459.1869462 · Cited by 82
Stefik, A., Hundhausen, C. D., & Patterson, R. (2011) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2011.07.002 · Cited by 78
Self-citationDo static type systems improve the maintainability of software systems? An empirical study
Kleinschmager, S., Robbes, R., Stefik, A., Hanenberg, S., & Tanter, E. (2012) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2012.6240483 · Cited by 73
Self-citationVickers, P., & Alty, J. (2002) · Interacting with Computers · doi:10.1016/s0953-5438(02)00026-7 · Cited by 71
An empirical study of novice program comprehension in the imperative and object-oriented styles
Ramalingam, V., & Wiedenbeck, S. (1997) · doi:10.1145/266399.266411 · Cited by 70
Blind learners programming through audio
Sánchez, J., & Aguayo, F. (2005) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1056808.1057018 · Cited by 70
Cooper, S. (2010) · ACM Transactions on Computing Education · doi:10.1145/1868358.1868362 · Cited by 66
An empirical study of the influence of static type systems on the usability of undocumented software
Mayer, C., Hanenberg, S., Robbes, R., Tanter, É., & Stefik, A. (2012) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2384616.2384666 · Cited by 62
Self-citationEmpirical studies on programming language stimuli
Stefik, A., & Gellenbeck, E. (2010) · Software Quality Journal · doi:10.1007/s11219-010-9106-7 · Cited by 59
Self-citationCan direct manipulation lower the barriers to computer programming and promote transfer of training?
Hundhausen, C. D., Farley, S., & Brown, J. L. (2009) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1592440.1592442 · Cited by 59
An empirical investigation of the influence of a type of side effects on program comprehension
Dolado, J. J., Harman, M., Otero, M. C., & Hu, L. (2003) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2003.1214329 · Cited by 57
Inspiring blind high school students to pursue computer science with instant messaging chatbots
Bigham, J. P., Aller, M. B., Brudvik, J. T., Leung, J. O., Yazzolino, L. A., & Ladner, R. E. (2008) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1352135.1352287 · Cited by 48
Hanenberg, S., Kleinschmager, S., & Josupeit-Walter, M. (2009) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2009.5316028 · Cited by 46
Hanenberg, S. (2010) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/1869459.1869536 · Cited by 46
Holt, R. C., Wortman, D. B., Barnard, D. T., & Cordy, J. R. (1977) · Communications of the ACM · doi:10.1145/359581.359586 · Cited by 42
Nonvisual tool for navigating hierarchical structures
Smith, A. C., Cook, J. S., Francioni, J. M., Hossain, A., Anwar, M. F. M., & Rahman, M. F. (2003) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/1028630.1028654 · Cited by 42
Garlick, R., & Cankaya, E. C. (2010) · doi:10.1145/1822090.1822138 · Cited by 41
Python CS1 as preparation for C++ CS2
Enbody, R. J., Punch, W. F., & McCullen, M. (2009) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1508865.1508907 · Cited by 38
Lopes, C. V., Dourish, P., Lorenz, D. H., & Lieberherr, K. J. (2003) · Companion to the ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA Companion) · doi:10.1145/949344.949400 · Cited by 37
Spoken Language Support for Software Development
Begel, A. (2005) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2004.49 · Cited by 32
Colour and sound in algorithm animation
Brown, M. H., & Hershberger, J. E. (2002) · IEEE Workshop on Visual Languages (VL) · doi:10.1109/wvl.1991.238856 · Cited by 32
Do Programming Languages Affect Productivity? A Case Study Using Data from Open Source Projects
Delorey, D. P., Knutson, C. D., & Chun, S. B.-S. (2007) · doi:10.1109/floss.2007.5 · Cited by 32
Is transactional programming actually easier?
Rossbach, C. J., Hofmann, O. S., & Witchel, E. (2010) · ACM SIGPLAN Notices · doi:10.1145/1837853.1693462 · Cited by 32
WAD: A Feasibility study using the Wicked Audio Debugger
Stefik, A., Alexander, R., Patterson, R., & Brown, J. (2007) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2007.42 · Cited by 31
Self-citationMeyerovich, L. A., & Rabkin, A. S. (2012) · doi:10.1145/2384592.2384597 · Cited by 30
Binkley, D., Davis, M. H., Lawrie, D., & Morrell, C. H. (2019) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2019.00035 · Cited by 30
Deltatalk: An Empirically and Aesthetically Motivated Simplification of the Smalltalk-80 Language
Borning, A. H., & O’Shea, T. (1987) · Lecture notes in computer science · doi:10.1007/3-540-47891-4_1 · Cited by 29
Analysing novices analysing Prolog: what stories do novices tell themselves about Prolog?
Taylor, J. (1990) · Instructional Science · doi:10.1007/bf00116442 · Cited by 29
Whitney, P. J. (1997) · Medical Entomology and Zoology · Cited by 28
Staking claims: a history of programming language design claims and evidence
Markstrum, S. A. (2010) · Evaluation and Usability of Programming Languages and Tools · doi:10.1145/1937117.1937124 · Cited by 28
Stefik, A., Siebert, S., Stefik, M., & Slattery, K. (2011) · doi:10.1145/2089155.2089159 · Cited by 24
Self-citationNovices' expectations and prior knowledge of software development
Schulte, C., & Magenheim, J. (2005) · doi:10.1145/1089786.1089800 · Cited by 23
Performance of python CS1 students in mid-level non-python CS courses
Enbody, R. J., & Punch, W. F. (2010) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1734263.1734437 · Cited by 23
Syntactic and semantic issues in introductory programming education
McIver, L. (2017) · Figshare · doi:10.4225/03/59cb52174f8e2 · Cited by 22
Endrikat, S., & Hanenberg, S. (2011) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2011.46 · Cited by 22
Using events to debug Java programs backwards in time
Lewis, B., & Ducassé, M. (2003) · doi:10.1145/949344.949367 · Cited by 18
More natural end-user software engineering
Myers, B. A., Ko, A. J., Park, S. Y., Stylos, J., LaToza, T. D., & Beaton, J. (2008) · International Workshop on End-User Software Engineering (WEUSE) · doi:10.1145/1370847.1370854 · Cited by 16
Strategic Software Development: Productivity Comparisons Of General Development Programs
Comstock, C., Jiang, Z., & Naudé, P. (2007) · doi:10.5281/zenodo.1062877 · Cited by 11
Using spoken text to aid debugging: An empirical study
Stefik, A., & Gellenbeck, E. (2009) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2009.5090034 · Cited by 11
Self-citationUses of the LOGO programming language in undergraduate instruction
Lukas, G. (1972) · ACM Annual Conference · doi:10.1145/800194.805908 · Cited by 10
Mining Programming Language Vocabularies from Source Code.
Delorey, D. P., Knutson, C. D., & Davies, M. (2009) · Cited by 9
Pausch, R. F. (2008) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1352135.1352137 · Cited by 8
The linguistics of object-oriented design
Holmboe, C. (2005) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1067445.1067498 · Cited by 4
Understanding programmer language
Høst, E. W. (2007) · doi:10.1145/1297846.1297957 · Cited by 3
Research works printed but not linked: 9
- Binkley D., Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC’09)
- 10.5555/646147.679038 · DOI
- Brown M. H., Proceedings of the IEEE Workshop on Visual Languages (VL’91)
- Delorey D. P., 2009, Proceedings of the Psychology of Programming Interest Group Conference (PPIG
- Feigenspan J., Proceedings of the IEEE 20th International Conference on Program Comprehension (ICPC’12)
- Kleinschmager S., Proceedings of the IEEE 20th International Conference on Program Comprehension (ICPC’12)
- Pennington N., 1987, Empirical Studies of Programmers: Second Workshop, G. M. Olson, S. Sheppard, E. Soloway, and B. Shneiderman Eds., Greenwood Publishing Group Inc., 100--113
- Stefik A., Proceedings of the 17th IEEE International Conference on Program Comprehension (ICPC’09)
- Vogt W. P., Quantitative Research Methods for Professionals in Education and Other Fields
Cited by
Sorted by citations
Introductory programming: a systematic literature review
Luxton-Reilly, A., Simon, Al-Bluwi, I., Becker, B. A., Giannakos, M. N., Kumar, A. N., Ott, L. M., Paterson, J. H., Scott, M., Sheard, J., et al. (2018) · doi:10.1145/3293881.3295779 · Cited by 511
Bau, D., Gray, J., Kelleher, C. L., Sheldon, J., & Turbak, F. A. (2017) · Communications of the ACM · doi:10.1145/3015455 · Cited by 326
Altadmri, A., & Brown, N. C. C. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677258 · Cited by 261
Learner-Centered Design of Computing Education: Research on Computing for Everyone
Guzdial, M. (2015) · Synthesis lectures on human-centered informatics · doi:10.2200/s00684ed1v01y201511hci033 · Cited by 260
Using Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235
Metadata
- Authors
- Stefik, A., & Siebert, S.
- Affiliations
- University of Nevada, Las Vegas (US)
- Washington University in St. Louis (US)
- ORCID
- Stefik, Andreas
- Venue
- ACM Transactions on Computing Education
- Published
- 2013-11
- Volume
- 13
- Issue
- 4
- Pages
- 1-40
- ISSN
- 1946-6226 (ISSN-L)
- OpenAlex
- W2157148654
- Cited by
- 295
- References
- 84
- Citations by year
- 2026: 14 · 2025: 13 · 2024: 21 · 2023: 19 · 2022: 19 · 2021: 19 · 2020: 34 · 2019: 60 · 2018: 23 · 2017: 22 · 2016: 17 · 2015: 16 · 2014: 15 · 2013: 2
- FWCI
- 25.18
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Computing Education (publishedVersion)
- Funders
- Division of Computer and Network Systems (CNS-0940521)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Software › Logic, programming, and type systems
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- programming language syntax, novice programmers, quorum, Python, ruby, Java, Perl, introductory programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
