The DOI matches its Crossref record; 72 of the 73 references it cites were found in a registry or an index.
Using Commutative Assessments to Compare Conceptual Understanding in Blocks-based and Text-based Programs
Abstract
Blocks-based programming environments are becoming increasingly common in introductory programming courses, but to date, little comparative work has been done to understand if and how this approach affects students' emerging understanding of fundamental programming concepts. In an effort to understand how tools like Scratch and Blockly differ from more conventional text-based introductory programming languages with respect to conceptual understanding, we developed a set of "commutative" assessments. Each multiple-choice question on the assessment includes a short program that can be displayed in either a blocks- based or text-based form. The set of potential answers for each question includes the correct answer along with choices informed by prior research on novice programming misconceptions. In this paper we introduce the Commutative Assessment, discuss the theoretical and practical motivations for the assessment, and present findings from a study that used the assessment. The study had 90 high school students take the assessment at three points over the course of the first ten weeks of an introduction to programming course, alternating the modality (blocks vs. text) for each question over the course of the three administrations of the assessment. Our analysis reveals differences on performance between blocks-based and text-based questions as well as differences in the frequency of misconceptions based on the modality. Future work, potential implications, and limitations of…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The pre-test, and isomorphic post-test1 and post-test2, were adapted from the block-based version of the Commutative Assessments [25], with 16 multiple-choice questions on variables, loops, procedures, and conditionals.”
From Exploring the Impact of Worked Examples in a Novice Programming Environment (2019) · cites this paper as [25]
Citations
Cites
(1978) · Medical Entomology and Zoology · Cited by 53,488
Vygotsky, L. S. (1964) · Bulletin of the Orton Society · doi:10.1007/bf02928399 · Cited by 20,704
The origins of intelligence in children.
Piaget, J. (1952) · doi:10.1037/11494-000 · Cited by 10,687
Ong, W. J. (1982) · Taylor & Francis eBooks · doi:10.4324/9780203328064 · Cited by 7,079
Cognition in Practice: Mind, Mathematics and Culture in Everyday Life
Lave, J. (1988) · doi:10.1017/cbo9780511609268 · Cited by 5,148
Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Mind in Society: Development of Higher Psychological Processes
VYGOTSKY, L. S. (1980) · doi:10.2307/j.ctvjf9vz4 · Cited by 3,123
Language, Thought, and Reality: Selected Writings of Benjamin Lee Whorf
Miller, G. A., & Carroll, J. B. (1957) · The American Journal of Psychology · doi:10.2307/1419256 · Cited by 2,748
Voices of the Mind: A Sociocultural Approach to Mediated Action
Baker‐Sennett, J., Rogoff, B., Bell, N. J., & Wertsch, J. V. (1992) · The American Journal of Psychology · doi:10.2307/1423207 · Cited by 1,505
van Deursen, A., Klint, P., & Visser, J. (2000) · ACM SIGPLAN Notices · doi:10.1145/352029.352035 · Cited by 1,303
Changing minds: computers, learning, and literacy
(2001) · Choice Reviews Online · doi:10.5860/choice.38-2855 · Cited by 813
Some Difficulties of Learning to Program
Du Boulay, B. (1986) · Journal of Educational Computing Research · doi:10.2190/3lfx-9rrf-67t8-uvk9 · Cited by 648
Wilensky, U. J., & Reisman, K. (2006) · Cognition and Instruction · doi:10.1207/s1532690xci2402_1 · Cited by 593
Self-citationWindows on Mathematical Meanings
Noss, R., & Hoyles, C. (1996) · Springer eBooks · doi:10.1007/978-94-009-1696-8 · Cited by 586
Windows on mathematical meanings: learning cultures and computers
(1997) · Choice Reviews Online · doi:10.5860/choice.35-0959 · Cited by 543
Alice: a 3-D tool for introductory programming concepts
Cooper, S., Dann, W. P., & Pausch, R. F. (2000) · Journal of computing sciences in colleges · Cited by 489
Abelson, H., & diSessa, A. A. (1981) · The MIT Press eBooks · doi:10.7551/mitpress/6933.001.0001 · Cited by 434
To block or not to block, that is the question
Weintrop, D., & Wilensky, U. J. (2015) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/2771839.2771860 · Cited by 426
Self-citationInventing Graphing: Meta Representational Expertise in Children
DiSessa, A. A. (1991) · The Journal of Mathematical Behavior · Cited by 390
Karel the Robot: A Gentle Introduction to the Art of Programming
Pattis, R. E., Roberts, J., & Stehlik, M. (1994) · Cumulative Index of Computer Aided Architectural Design · Cited by 352
The fairy performance assessment
Werner, L., Denner, J., Campe, S., & Kawamoto, D. C. (2012) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2157136.2157200 · Cited by 330
Language-Independent Conceptual “Bugs” in Novice Programming
Pea, R. D. (1986) · Journal of Educational Computing Research · doi:10.2190/689t-1r2a-x4w4-29j2 · Cited by 320
An Empirical Investigation into Programming Language Syntax
Stefik, A., & Siebert, S. (2013) · ACM Transactions on Computing Education · doi:10.1145/2534973 · Cited by 295
From Scratch to “Real” Programming
Armoni, M., Meerbaum–Salant, O., & Ben‐Ari, M. (2015) · ACM Transactions on Computing Education · doi:10.1145/2677087 · Cited by 267
Learning computer science concepts with scratch
Meerbaum–Salant, O., Armoni, M., & Ben‐Ari, M. (2010) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/1839594.1839607 · Cited by 228
Modeling how students learn to program
Piech, C., Sahami, M., Koller, D., Cooper, S., & Blikstein, P. (2012) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2157136.2157182 · Cited by 200
Habits of programming in scratch
Meerbaum-Salant, O., Armoni, M., & Ben-Ari, M. (2011) · Proceedings of the 16th annual joint conference on Innovation and technology in computer science education · doi:10.1145/1999747.1999796 · Cited by 199
Programming-languages as a conceptual framework for teaching mathematics
Feurzeig, W., Papert, S. A., Bloom, M., Grant, R., & Solomon, C. J. (1970) · ACM SIGCUE Outlook · doi:10.1145/965754.965757 · Cited by 193
How programming environment shapes perception, learning and goals
Lewis, C. M. (2010) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1734263.1734383 · Cited by 192
Mini-languages: a way to learn programming principles
Brusilovsky, P. L., Calabrese, E., Hvorecký, J., KOUCHNIRENKO, A., & Miller, P. L. (1997) · Education and Information Technologies · doi:10.1023/a:1018636507883 · Cited by 181
Learning Electricity with NIELS: Thinking with Electrons and Thinking in Levels
Sengupta, P., & Wilensky, U. J. (2009) · International Journal of Computers for Mathematical Learning · doi:10.1007/s10758-009-9144-z · Cited by 175
Self-citationProgramming Environments Based on Structured Editors: The MENTOR Experience,
Donzeau-Gouge, V. V., Huet, G., Kahn, G., & Lang, B. (1980) · HAL · Cited by 174
Tew, A. E., & Guzdial, M. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953200 · Cited by 172
Developing a validated assessment of fundamental CS1 concepts
Tew, A. E., & Guzdial, M. (2010) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1734263.1734297 · Cited by 170
Visual program simulation in introductory programming education
Sorva, J. (2012) · Aaltodoc (Aalto University) · Cited by 155
Computer Science curricula 2013
Sahami, M., & Roach, S. (2013) · ACM SIGCSE Bulletin · doi:10.1145/2490868.2490870 · Cited by 154
Assessing computational learning in K-12
Grover, S., Cooper, S., & Pea, R. D. (2014) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2591708.2591713 · Cited by 153
A diagnosis of beginning programmers' misconceptions of BASIC programming statements
Bayman, P., & Mayer, R. E. (1983) · Communications of the ACM · doi:10.1145/358172.358408 · Cited by 152
A Comparison of Programming Languages and Algebraic Notation as Expressive Languages for Physics
Sherin, B. L. (2001) · International Journal of Computers for Mathematical Learning · doi:10.1023/a:1011434026437 · Cited by 152
Comprehension and recall of miniature programs
Gilmore, D., & Green, T. R. G. (1984) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(84)80037-1 · Cited by 148
Blikstein, P., & Wilensky, U. J. (2009) · International Journal of Computers for Mathematical Learning · doi:10.1007/s10758-009-9148-8 · Cited by 146
Self-citationUnderstanding the syntax barrier for novices
Denny, P., Luxton-Reilly, A., Tempero, E., & Hendrickx, J. (2011) · Proceedings of the 16th annual joint conference on Innovation and technology in computer science education · doi:10.1145/1999747.1999807 · Cited by 133
Powers, K. D., Ecott, S., & Hirshfield, L. (2007) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1227310.1227386 · Cited by 116
Pascal and High School Students: A Study of Errors
Sleeman, D. H., Putnam, R. T., Baxter, J. A., & Kuspa, L. K. (1986) · Journal of Educational Computing Research · doi:10.2190/2xpp-ltyh-98nq-bu77 · Cited by 110
Bringing "No Ceiling" to Scratch: Can One Language Serve Kids and Computer Scientists?
Harvey, B. W. (2010) · Cited by 101
Collaboration, Collusion and Plagiarism in Computer Science Coursework
FRASER, R. (2014) · Informatics in Education · doi:10.15388/infedu.2014.10 · Cited by 98
How students invent representations of motion
Sherin, B. L. (2000) · The Journal of Mathematical Behavior · doi:10.1016/s0732-3123(01)00052-9 · Cited by 93
Powers, K. D., Ecott, S., & Hirshfield, L. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1227504.1227386 · Cited by 91
LogoBlocks: A Graphical Programming Language for Interacting with the World
Begel, A., & Resnick, M. (2000) · Cited by 89
On Ways of Symbolizing: The Case of Laura and the Velocity Sign
Nemirovsky, R. (1996) · Springer eBooks · doi:10.1007/978-3-642-61189-6_10 · Cited by 80
Evaluating assessments of novice programming environments
Gross, P., & Powers, K. D. (2005) · doi:10.1145/1089786.1089796 · Cited by 80
Blue—a language for teaching object-oriented programming
Kölling, M., & Rosenberg, J. (1996) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/236452.236537 · Cited by 70
Buffum, P. S., Lobene, E. V., Frankosky, M. H., Boyer, K. E., Wiebe, E., & Lester, J. C. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677295 · Cited by 68
On ways of symbolizing: The case of Laura and the velocity sign
Nemirovsky, R. (1994) · The Journal of Mathematical Behavior · doi:10.1016/0732-3123(94)90002-7 · Cited by 61
Wertsch, J. V. (1993) · Harvard University Press eBooks · doi:10.4159/9780674045101 · Cited by 60
Capitalism and Arithmetic: The New Math of the 15th Century
Swetz, F. (1989) · Science & Society · Cited by 60
Empirical studies on programming language stimuli
Stefik, A., & Gellenbeck, E. (2010) · Software Quality Journal · doi:10.1007/s11219-010-9106-7 · Cited by 59
Stefik, A., & Hanenberg, S. (2014) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/2661136.2661156 · Cited by 56
Real Time Assessment of Computational Thinking
Koh, K. H., Basawapatna, A. R., Nickerson, H., & Repenning, A. (2014) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2014.6883021 · Cited by 55
The same but different students' understandings of primitive and object variables
Sorva, J. (2008) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1595356.1595360 · Cited by 54
The Case for Validated Tools in Computer Science Education Research
Tew, A. E., & Dorn, B. (2013) · Computer · doi:10.1109/mc.2013.259 · Cited by 54
Esper, S., Foster, S. R., & Griswold, W. G. (2013) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2462476.2465593 · Cited by 43
Garlick, R., & Cankaya, E. C. (2010) · doi:10.1145/1822090.1822138 · Cited by 41
Programming Languages in Education: The Search for an Easy Start
Mendelsohn, P., Green, T. R. G., & Brna, P. (1990) · Elsevier eBooks · doi:10.1016/b978-0-12-350772-3.50016-1 · Cited by 37
RoboBuilder: A Program-to-Play Constructionist Video Game
Weintrop, D. (2012) · Cited by 36
Self-citationProgramming osmosis: Knowledge transfer from imperative to visual programming environments
Parsons, D., & Haden, P. (2007) · Cited by 36
Restructuring Change, Interpreting Changes: The DeltaTick Modeling and Analysis Toolkit
Wilkerson, M. H. (2010) · Cited by 26
A Visual Programming Language for Novices
Bonar, J. G., & Liffick, B. W. (1987) · doi:10.21236/ada218940 · Cited by 24
Sudol, L. A., & Studer, C. (2010) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1734263.1734411 · Cited by 16
Olson, I. C., & Horn, M. (2011) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/1999030.1999058 · Cited by 14
Blue—a language for teaching object-oriented programming
Kölling, M., & Rosenberg, J. (1996) · ACM SIGCSE Bulletin · doi:10.1145/236462.236537 · Cited by 11
The Journal of Mathematical Behavior
Moschkovich, J. N. (2000) · The Journal of Mathematical Behavior · doi:10.1016/s0732-3123(01)00065-7
Research works printed but not linked: 14
- Begel A. and Klopfer E. 2007. Starlogo TNG: An introduction to game development. Journal of E-Learning. (2007). Begel A. and Klopfer E. 2007. Starlogo TNG: An introduction to game development. Journal of E-Learning. (2007).
- 10.5555/364133.364161 · DOI
- Dijkstra E. W., How do we tell truths that might hurt? Selected Writings on Computing: A Personal Perspective
- 10.5555/335023 · DOI
- Doukakis D., 2007, Proc. of the 8th HERCMA Conference
- Harvey B., Proc. of Constructionism 2010
- Luria A. R., Language and cognition. Winston
- Palmer S. E., Fundamental aspects of cognitive representation. Cognition and categorization
- Parsons D., 2007, Proc.of The 12th Annual NACCQ Conference
- Slany W., 2014, Proc. of Constructionism 2014
- Wagh A., 2012, Proc. of Constructionism 2012 (Athens
- Weintrop D., 2012, Proc. of Constructionism 2012 (Athens
- Weintrop D., 2014, Proc. of Constructionism 2014
- Wilensky U., 2010, Proc. of Constructionism 2010
Cited by
Sorted by citations
Román-González, ., Pérez-González, J. C., & Fernández, C. J. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.08.047 · Cited by 867
Students’ Misconceptions and Other Difficulties in Introductory Programming: A Literature Review
Qian, Y., & Lehman, J. (2018) · ACM Transactions on Computing Education · doi:10.1145/3077618 · Cited by 529
Comparing Block-Based and Text-Based Programming in High School Computer Science Classrooms
Weintrop, D., & Wilensky, U. J. (2017) · ACM Transactions on Computing Education · doi:10.1145/3089799 · Cited by 338
Self-citationReplication, Validation, and Use of a Language Independent CS1 Knowledge Assessment
Parker, M. C., Guzdial, M., & Engelman, S. (2016) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2960310.2960316 · Cited by 185
Combining Assessment Tools for a Comprehensive Evaluation of Computational Thinking Interventions
Román-González, ., Moreno-León, J., & Robles, G. (2019) · Springer eBooks · doi:10.1007/978-981-13-6528-7_6 · Cited by 184
Metadata
- Authors
- Weintrop, D., & Wilensky, U.
- Affiliations
- Northwestern University (US)
- Venue
- Proceedings of the eleventh annual International Conference on International Computing Education Research
- Publisher
- ACM
- Published
- 2015-08-09
- Pages
- 101-110
- OpenAlex
- W2016385238
- Cited by
- 198
- References
- 73
- Citations by year
- 2026: 6 · 2025: 4 · 2024: 6 · 2023: 13 · 2022: 19 · 2021: 20 · 2020: 24 · 2019: 59 · 2018: 12 · 2017: 20 · 2016: 10 · 2015: 5
- FWCI
- 11.21
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Science Education and Pedagogy
- Social Sciences › Education › Educational Robotics and Engineering
- Keywords
- text-based programming, introductory programming, conceptual understanding, Scratch, Blockly, high school students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review