A Framework for Using Hypothesis-Driven Approaches to Support Data-Driven Learning Analytics in Measuring Computational Thinking in Block-Based Programming Environments

Grover, Shuchi; Basu, Satabdi; Bienkowski, Marie; Eagle, Michael; Diana, Nicholas; Stamper, John

doi:10.1145/3105910

Cited by 93Teaching and Learning ProgrammingOnline Learning and AnalyticsStatistics Education and Methodologies

A Framework for Using Hypothesis-Driven Approaches to Support Data-Driven Learning Analytics in Measuring Computational Thinking in Block-Based Programming Environments

Grover, Shuchi, Basu, Satabdi, Bienkowski, Marie, Eagle, Michael, Diana, Nicholas, Stamper, John·2017·ACM Transactions on Computing Education

DOI 10.1145/3105910

Open this paper in CoChat — free account

Abstract

Systematic endeavors to take computer science (CS) and computational thinking (CT) to scale in middle and high school classrooms are underway with curricula that emphasize the enactment of authentic CT skills, especially in the context of programming in block-based programming environments. There is, therefore, a growing need to measure students’ learning of CT in the context of programming and also support all learners through this process of learning computational problem solving. The goal of this research is to explore hypothesis-driven approaches that can be combined with data-driven ones to better interpret student actions and processes in log data captured from block-based programming environments with the goal of measuring and assessing students’ CT skills. Informed by past literature and based on our empirical work examining a dataset from the use of the Fairy Assessment in the Alice programming environment in middle schools, we present a framework that formalizes a process where a hypothesis-driven approach informed by Evidence-Centered Design effectively complements data-driven learning analytics in interpreting students’ programming process and assessing CT in block-based programming environments. We apply the framework to the design of Alice tasks for high school CS to be used for measuring CT during programming.

Integrity signals

Registry

Crossref · 10.1145/3105910

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Programming is an optimal medium for CT education.
“Programming, in particular, has been highlighted as an optimal medium for CT education.”
From ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12 (2024) · cites this paper as [19]
The nuances of CT can be effectively evaluated through programming exercises.
“Secondly, the nuances of CT can be effectively evaluated through programming exercises [19].”
From ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12 (2024) · cites this paper as [19]

Computational thinking
Wing, J. M. (2006) · Communications of the ACM · doi:10.1145/1118178.1118215 · Cited by 8,102 · see citing works
Untitled work
(1980) · Medical Entomology and Zoology · Cited by 7,534 · see citing works
Computational Thinking in K–12
Grover, S., & Pea, R. D. (2013) · Educational Researcher · doi:10.3102/0013189x12463051 · Cited by 2,547 · see citing works
Self-citation
Educational Data Mining: A review of the state of the art
Romero, C., & Ventura, S. (2010) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2010.2053532 · Cited by 2,251 · see citing works
Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century
Pellegrino, J. W., & Hilton, M. L. (2012) · Cited by 2,038 · see citing works
The State of Educational Data Mining in 2009: A Review and Future Visions
Baker, R. S. J. D., & Yacef, K. (2009) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554657 · Cited by 1,365 · see citing works
Analyzing and Visualizing State Sequences in R with TraMineR
Gabadinho, A., Ritschard, G., Müller, N. S., & Studer, M. (2011) · Journal of Statistical Software · doi:10.18637/jss.v040.i04 · Cited by 1,364 · see citing works
Educational Data Mining and Learning Analytics
Baker, R. S., & Inventado, P. S. (2014) · Springer eBooks · doi:10.1007/978-1-4614-3305-7_4 · Cited by 1,333 · see citing works
The State of Educational Data Mining in 2009: A Review and Future Visions
Baker, R. S., & Yacef, K. (2009) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554658 · Cited by 1,296 · see citing works
Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters
Gwet, K. L. (2014) · Medical Entomology and Zoology · Cited by 1,095 · see citing works
The Social and Technological Dimensions of Scaffolding and Related Theoretical Concepts for Learning, Education, and Human Activity
Pea, R. D. (2004) · Journal of the Learning Sciences · doi:10.1207/s15327809jls1303_6 · Cited by 1,047 · see citing works
Proceedings of the 22nd international conference on Machine learning - ICML '05
(2005) · doi:10.1145/1102351 · Cited by 884 · see citing works
The Knowledge‐Learning‐Instruction Framework: Bridging the Science‐Practice Chasm to Enhance Robust Student Learning
Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012) · Cognitive Science · doi:10.1111/j.1551-6709.2012.01245.x · Cited by 772 · cited 1× · see citing works
“Also, prior literature in both constrained Intelligent Tutoring Systems and OELEs underscores the need to see evidence of appropriate as well as repeated use of constructs—either demonstrating growth over time or mastery—to warrant a claim of evidence of skill [14, 35, 67].”
Background
Learning to program = learning to construct mechanisms and explanations
Soloway, E. (1986) · CACM · doi:10.1145/6592.6594 · Cited by 615 · see citing works
Implications of Evidence‐Centered Design for Educational Testing
Mislevy, R. J., & Haertel, G. D. (2006) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.2006.00075.x · Cited by 550 · see citing works
On the cognitive effects of learning computer programming
Pea, R. D., & Kurland, D. M. (1984) · New Ideas in Psychology · doi:10.1016/0732-118x(84)90018-7 · Cited by 546 · see citing works
Educational Data Mining and Learning Analytics
Baker, R. S., & Siemens, G. (2014) · Cambridge University Press eBooks · doi:10.1017/cbo9781139519526.016 · Cited by 491 · see citing works
WISE design for knowledge integration
Linn, M. C., Clark, D. B., & Slotta, J. D. (2003) · Science Education · doi:10.1002/sce.10086 · Cited by 419 · see citing works
Designing for deeper learning in a blended computer science course for middle school students
Grover, S., Pea, R. D., & Cooper, S. (2015) · Computer Science Education · doi:10.1080/08993408.2015.1033142 · Cited by 410 · see citing works
Self-citation
Stealth Assessment: Measuring and Supporting Learning in Video Games
Shute, V., & Ventura, M. (2013) · doi:10.7551/mitpress/9589.001.0001 · Cited by 406 · see citing works
Learning computer science concepts with Scratch
Meerbaum–Salant, O., Armoni, M., & Ben‐Ari, M. (2013) · Computer Science Education · doi:10.1080/08993408.2013.832022 · Cited by 360 · see citing works
Computer games created by middle school girls: Can they be used to measure understanding of computer science concepts?
Denner, J., Werner, L., & Ortiz, E. (2011) · Computers & Education · doi:10.1016/j.compedu.2011.08.006 · Cited by 343 · see citing works
Avatars Versus Agents: A Meta-Analysis Quantifying the Effect of Agency on Social Influence
Fox, J., Ahn, S. J. (., Janssen, J. H., Yeykelis, L., Segovia, K. Y., & Bailenson, J. N. (2015) · Human–Computer Interaction · doi:10.1080/07370024.2014.921494 · Cited by 320 · see citing works
The fairy performance assessment
Werner, L., Denner, J., Campe, S., & Kawamoto, D. C. (2012) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2157136.2157200 · Cited by 305 · see citing works
Programming Pluralism: Using Learning Analytics to Detect Patterns in the Learning of Computer Programming
Blikstein, P., Worsley, M. B., Piech, C., Sahami, M., Cooper, S., & Koller, D. (2014) · Journal of the Learning Sciences · doi:10.1080/10508406.2014.954750 · Cited by 266 · see citing works
Learning to program with Alice
Dann, W. P., Cooper, S., & Pausch, R. F. (2004) · Cited by 238 · see citing works
Using Learning Analytics to Understand the Learning Pathways of Novice Programmers
Berland, M., Martin, T., Benton, T., Smith, C. P., & Davis, D. (2013) · Journal of the Learning Sciences · doi:10.1080/10508406.2013.836655 · Cited by 217 · see citing works
Modeling how students learn to program
Piech, C., Sahami, M., Koller, D., Cooper, S., & Blikstein, P. (2012) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2157136.2157182 · Cited by 200 · see citing works
Children's Mental Models of Recursive Logo Programs
Kurland, D. M., & Pea, R. D. (1985) · Journal of Educational Computing Research · doi:10.2190/jv9y-5pd0-mx22-9j4y · Cited by 177 · see citing works
Alice, Greenfoot, and Scratch -- A Discussion
Utting, I., Cooper, S., Kölling, M., Maloney, J. H., & Resnick, M. (2010) · ACM Transactions on Computing Education · doi:10.1145/1868358.1868364 · Cited by 173 · see citing works
The Potentials of Educational Data Mining for Researching Metacognition, Motivation and Self-Regulated Learning
Winne, P. H., & Baker, R. S. (2013) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554619 · Cited by 162 · see citing works
Assessing computational learning in K-12
Grover, S., Cooper, S., & Pea, R. D. (2014) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2591708.2591713 · Cited by 153 · see citing works
Self-citation
Learner modeling for adaptive scaffolding in a Computational Thinking-based science learning environment
Basu, S., Biswas, G., & Kinnebrew, J. S. (2017) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-017-9187-0 · Cited by 152 · see citing works
Self-citation
Design and Discovery in Educational Assessment: Evidence-Centered Design, Psychometrics, and Educational Data Mining
Mislevy, R. J., Behrens, J. T., Dicerbo, K. E., & Levy, R. (2012) · Zenodo · doi:10.5281/zenodo.3554641 · Cited by 139 · see citing works
Learning Program Embeddings to Propagate Feedback on Student CodePreprint
Piech, C., Huang, J., Nguyen, A., Phulsuksombati, M., Sahami, M., & Guibas, L. J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05969 · Cited by 128 · see citing works
Leveraging educational data mining for real-time performance assessment of scientific inquiry skills within microworlds.
Gobert, J. D., Sao Pedro, M. A., Baker, R. S. J. D., Toto, E., & Montalvo, O. (2012) · Zenodo · doi:10.5281/zenodo.3554645 · Cited by 122 · see citing works
Combining Unsupervised and Supervised Classification to Build User Models for Exploratory Learning Environments
Amershi, S., & Conati, C. (2009) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554659 · Cited by 110 · see citing works
But My Program Runs! Discourse Rules for Novice Programmers
Joni, S.-N. A., & Soloway, E. M. (1986) · Journal of Educational Computing Research · doi:10.2190/6e5w-ar7c-nx76-hut2 · Cited by 105 · see citing works
Towards the Automatic Recognition of Computational Thinking for Adaptive Visual Language Learning
Koh, K. H., Basawapatna, A. R., Bennett, V., & Repenning, A. (2010) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2010.17 · Cited by 103 · see citing works
Programming in the wild
Fields, D. A., Giang, M. T., & Kafai, Y. B. (2014) · Workshop in Primary and Secondary Computing Education · doi:10.1145/2670757.2670768 · Cited by 102 · see citing works
Remedying misperceptions of computer science among middle school students
Grover, S., Pea, R. D., & Cooper, S. (2014) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2538862.2538934 · Cited by 97 · see citing works
Self-citation
Dann, W., Cooper, S. & Pausch, R. (2006). Learning to program with Alice. Upper Saddle River, NJ: Prentice-Hall
Abramson, G. W. (2008) · NSUWorks (Nova Southeastern University) · Cited by 96 · see citing works
Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill
Pedro, M. S., Baker, R. S., Gobert, J. D., Montalvo, O., & Nakama, A. (2011) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-011-9101-0 · Cited by 93 · see citing works
Assessing Algorithmic and Computational Thinking in K-12: Lessons from a Middle School Classroom
Grover, S. (2017) · Springer eBooks · doi:10.1007/978-3-319-52691-1_17 · Cited by 89 · see citing works
Self-citation
Identifying Key Features of Student Performance in Educational Video Games and Simulations through Cluster Analysis
Kerr, D., & Chung, G. K. W. K. (2012) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554647 · Cited by 84 · see citing works
Increasing diversity in k-12 computer science
Goode, J. (2008) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1352135.1352259 · Cited by 79 · see citing works
Identifying Productive Inquiry in Virtual Labs Using Sequence Mining
Perez, S., Massey-Allard, J., Butler, D., Ives, J., Bonn, D., Yee, N., & Roll, I. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-61425-0_24 · Cited by 79 · see citing works
Putting ECD into Practice: The Interplay of Theory and Data in Evidence Models within a Digital Learning Environment
Rupp, A. A., Levy, R., DiCerbo, K. E., Sweet, S. J., Crawford, A., Caliço, T., Benson, M., Fay, D. M., Kunze, K. L., Mislevy, R. J., et al. (2012) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554643 · Cited by 62 · see citing works
Motivating programming: using storytelling to make computer programming attractive to middle school girls
Pausch, R. F., & Kelleher, C. L. (2006) · Cited by 56 · see citing works
Skill progression in MIT app inventor
Xie, B., & Abelson, H. (2016) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2016.7739687 · Cited by 54 · see citing works
A nationwide exam as a tool for improving a new curriculum
Bargury, I. Z., Pârv, B., & Lanzberg, D. (2013) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2462476.2462479 · Cited by 48 · see citing works
AgentSheets: an Interactive Simulation Environment with End-User Programmable Agents
Repenning, A. (2000) · Cited by 47 · see citing works
A future for computing education research
Cooper, S., Grover, S., Guzdial, M., & Simon, B. (2014) · Communications of the ACM · doi:10.1145/2668899 · Cited by 47 · see citing works
Self-citation
Serious Games Analytics to Measure Implicit Science Learning
Rowe, E. A., Asbell‐Clarke, J., & Baker, R. S. (2015) · Springer eBooks · doi:10.1007/978-3-319-05834-4_15 · Cited by 44 · see citing works
A Conceptual Framework for a Psychometric Theory for Standard Setting with Examples of Its Use for Evaluating the Functioning of Two Standard Setting Methods
Reckase, M. D. (2006) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.2006.00052.x · Cited by 43 · see citing works
Measuring the usability and capability of App inventor to create mobile Applications
Xie, B., Shabir, I., & Abelson, H. (2015) · doi:10.1145/2824823.2824824 · Cited by 38 · see citing works
Combining Big Data and Thick Data Analyses for Understanding Youth Learning Trajectories in a Summer Coding Camp
Fields, D. A., Quirke, L. C., Amely, J., & Maughan, J. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844631 · Cited by 33 · see citing works
Mere belief in social action improves complex learning
Okita, S. Y., Bailenson, J. N., & Schwartz, D. L. (2008) · International Conference of Learning Sciences · Cited by 31 · see citing works
A principled evaluation for a principled idea garden
Jernigan, W., Horvath, A., Lee, M., Burnett, M., Cuilty, T., Kuttal, S., Peters, A., Kwan, I., Bahmani, F., & Ko, A. (2015) · 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2015.7357222 · Cited by 28 · see citing works
Evidence-centered Design for Diagnostic Assessment within Digital Learning Environments: Integrating Modern Psychometrics and Educational Data Mining
Rupp, A. A., Nugent, R., & Nelson, B. C. (2012) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554639 · Cited by 22 · see citing works
A First Step in Learning Analytics: Pre-processing Low-Level Alice Logging Data of Middle School Students
Werner, L., McDowell, C., & Denner, J. (2013) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554631 · Cited by 18 · see citing works
Using the personal software process to motivate good programming practices
Grove, R. F. (1998) · doi:10.1145/282991.283046 · Cited by 16 · see citing works
Leveraging Educational Data Mining for Real-time Performance Assessment of Scientific Inquiry Skills within Microworlds
Gobert, J. D., Pedro, M. S., Baker, R. S., et al. (2012) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3554646 · Cited by 13 · see citing works
Assessing Problem-Solving Process At Scale
Grover, S., Bienkowski, M. A., Niekrasz, J., & Hauswirth, M. (2016) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2876034.2893425 · Cited by 12 · see citing works
Self-citation
Using the ECD Framework to Support Evidentiary Reasoning in the Context of a Simulation Study for Detecting Learner Differences in Epistemic Games
Sweet, S. J., & Rupp, A. A. (2012) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554649 · Cited by 8 · see citing works
A First Step in Learning Analytics: Pre-processing Low-Level Alice Logging Data of Middle School Students
Werner, L., McDowell, C., & Denner, J. (2013) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3554632 · Cited by 2 · see citing works

Flagged first, then by citations

Sorted by citations

Mapping computational thinking through programming in K-12 education: A conceptual model based on a systematic literature Review
Tikva, C., & Tambouris, E. (2021) · Computers & Education · doi:10.1016/j.compedu.2020.104083 · Cited by 296 · see citing works
Combining Assessment Tools for a Comprehensive Evaluation of Computational Thinking Interventions
Román-González‬, ‪., Moreno-León, J., & Robles, G. (2019) · Springer eBooks · doi:10.1007/978-981-13-6528-7_6 · Cited by 184 · see citing works
Can computational talent be detected? Predictive validity of the Computational Thinking Test
Román-González‬, ‪., Pérez-González, J. C., Moreno-León, J., & Robles, G. (2018) · International Journal of Child-Computer Interaction · doi:10.1016/j.ijcci.2018.06.004 · Cited by 146 · see citing works
Block-based versus text-based programming environments on novice student learning outcomes: a meta-analysis study
Xu, Z., Ritzhaupt, A. D., Tian, F., & Umapathy, K. (2019) · Computer Science Education · doi:10.1080/08993408.2019.1565233 · Cited by 115 · see citing works
Exploring the Effectiveness and Moderators of Block-Based Visual Programming on Student Learning: A Meta-Analysis
Hu, Y., Chen, C.-H., & Su, C. (2020) · Journal of Educational Computing Research · doi:10.1177/0735633120945935 · Cited by 98 · see citing works
Assessing implicit computational thinking in Zoombinis puzzle gameplay
Rowe, E. A., Almeda, M. V., Asbell‐Clarke, J., Scruggs, R., Baker, R. S., Bardar, E., & Gasca, S. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106707 · Cited by 88 · see citing works
Trends and development in research on computational thinking
Tekdal, M. (2021) · Education and Information Technologies · doi:10.1007/s10639-021-10617-w · Cited by 80 · see citing works
What We Can Learn About Student Learning From Open-Ended Programming Projects in Middle School Computer Science
Grover, S., Basu, S., & Schank, P. K. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159522 · Cited by 76 · see citing works
Self-citation
Clickstream for learning analytics to assess students’ behavior with Scratch
Amo, D., Alier, M., García‐Peñalvo, F. J., Escudero, D. F., & Casañ, M. J. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2018.10.057 · Cited by 70 · see citing works
Applications of Learning Analytics in High Schools: A Systematic Literature Review
de Sousa, E. B. G., Rosa, B. A. B., Mello, R. F., Falcão, T. P., Vesin, B., & Gašević, D. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.737891 · Cited by 65 · see citing works
Persistence in a Game-Based Learning Environment: The Case of Elementary School Students Learning Computational Thinking
Israel‐Fishelson, R., & Hershkovitz, A. (2019) · Journal of Educational Computing Research · doi:10.1177/0735633119887187 · Cited by 63 · see citing works
The Associations Between Computational Thinking and Creativity: The Role of Personal Characteristics
Israel‐Fishelson, R., Hershkovitz, A., Eguíluz, A., Garaizar, P., & Guenaga, M. (2020) · Journal of Educational Computing Research · doi:10.1177/0735633120940954 · Cited by 71 · see citing works
The Competent Computational Thinking Test: Development and Validation of an Unplugged Computational Thinking Test for Upper Primary School
El‐Hamamsy, L., Zapata-Cáceres, M., Martín, E., Francesco Mondada, C., Zufferey, J. D., & Bruno, B. (2022) · Journal of Educational Computing Research · doi:10.1177/07356331221081753 · Cited by 60 · see citing works
ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12
Chen, L., Xiao, S., Chen, Y., Song, Y., Wu, R., & Sun, L. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642229 · Cited by 60 · see citing works
Examining Student Regulation of Collaborative, Computational, Problem-Solving Processes in Open-Ended Learning Environments
Emara, M., Hutchins, N. M., Grover, S., Snyder, C., & Biswas, G. (2021) · Journal of Learning Analytics · doi:10.18608/jla.2021.7230 · Cited by 53 · see citing works
Self-citation
Concepts before coding: non-programming interactives to advance learning of introductory programming concepts in middle school
Grover, S., Jackiw, N., & Lundh, P. (2019) · Computer Science Education · doi:10.1080/08993408.2019.1568955 · Cited by 52 · see citing works
Self-citation
The effect of cues for calibration on learners' self-regulated learning through changes in learners’ learning behaviour and outcomes
Laer, S. V., & Elen, J. (2019) · Computers & Education · doi:10.1016/j.compedu.2019.02.016 · Cited by 42 · see citing works
A principled approach to designing computational thinking concepts and practices assessments for upper elementary grades
Basu, S., Rutstein, D. W., Xu, Y., Wang, H., & Shear, L. (2021) · Computer Science Education · doi:10.1080/08993408.2020.1866939 · Cited by 40 · see citing works
Self-citation
How do students develop computational thinking? Assessing early programmers in a maze-based online game
Guenaga, M., Eguíluz, A., Garaizar, P., & Gibaja, J. (2021) · Computer Science Education · doi:10.1080/08993408.2021.1903248 · Cited by 34 · see citing works
Designing an Assessment for Introductory Programming Concepts in Middle School Computer Science
Grover, S. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366896 · Cited by 34 · see citing works
Self-citation
A Log-Based Analysis of the Associations Between Creativity and Computational Thinking
Israel‐Fishelson, R., Hershkovitz, A., Eguíluz, A., Garaizar, P., & Guenaga, M. (2020) · Journal of Educational Computing Research · doi:10.1177/0735633120973429 · Cited by 34 · see citing works
Using Rubrics Integrating Design and Coding to Assess Middle School Students' Open-ended Block-based Programming Projects
Basu, S. (2019) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3287324.3287412 · Cited by 29 · see citing works
Self-citation
From Critique to Computational Thinking: A Peer-Assessment-Supported Problem Identification, Flow Definition, Coding, and Testing Approach for Computer Programming Instruction
Fang, J.-W., Shao, D., Hwang, G., & Chang, S. (2022) · Journal of Educational Computing Research · doi:10.1177/07356331211060470 · Cited by 28 · see citing works
Data-driven and knowledge-guided denoising diffusion model for flood forecasting
Shao, P., Feng, J., Lu, J., Zhang, P., & Zou, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122908 · Cited by 28 · see citing works
Evidence of the ISTE Standards for Educators leading to learning gains
Crompton, H. (2023) · Journal of Digital Learning in Teacher Education · doi:10.1080/21532974.2023.2244089 · Cited by 27 · see citing works
Early Performance Prediction using Interpretable Patterns in Programming Process Data
Gao, G., Marwan, S., & Price, T. (2021) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3408877.3432439 · Cited by 27 · see citing works
A Generative Artificial Intelligence (AI)-Based Human-Computer Collaborative Programming Learning Method to Improve Computational Thinking, Learning Attitudes, and Learning Achievement
Zhao, G., Yang, L., Hu, B., & Wang, J. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251336154 · Cited by 27 · see citing works
A Principled Approach to Designing a Computational Thinking Practices Assessment for Early Grades
Basu, S., Rutstein, D. W., Xu, Y., & Shear, L. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366849 · Cited by 25 · see citing works
Self-citation
A systematic review of the use of log-based process data in computer-based assessments
He, S., & Cui, Y. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105245 · Cited by 24 · see citing works
Data Mining Framework to Analyze the Evolution of Computational Thinking Skills in Game Building Workshops
de Souza, A. A., Barcelos, T., Muñoz, R., Villarroel, R. H., & da Silva, L. A. (2019) · IEEE Access · doi:10.1109/access.2019.2924343 · Cited by 24 · see citing works
The influence of block-based programming to computational thinking skills: A systematic review
Fadhillah, M. R., Budiyanto, C. W., & Hatta, P. (2023) · AIP conference proceedings · doi:10.1063/5.0105716 · Cited by 20 · see citing works
Debugging in Computational Thinking: A Meta-analysis on the Effects of Interventions on Debugging Skills
Sun, C., Yang, S. F., & Becker, B. J. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241227793 · Cited by 20 · see citing works
Data-driven and knowledge-guided denoising diffusion probabilistic model for runoff uncertainty prediction
Shao, P., Feng, J., Lu, J., & Tang, Z. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131556 · Cited by 20 · see citing works
Micro‐persistence and difficulty in a game‐based learning environment for computational thinking acquisition
Israel‐Fishelson, R., & Hershkovitz, A. (2021) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12527 · Cited by 20 · see citing works
Predictive Student Modeling in Block-Based Programming Environments with Bayesian Hierarchical Models
Emerson, A., Geden, M., Smith, A., Wiebe, E., Mott, B., Boyer, K. E., & Lester, J. C. (2020) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3340631.3394853 · Cited by 20 · see citing works
Design and analysis of microworlds and puzzles for block-based programming
Pelánek, R., & Effenberger, T. (2020) · Computer Science Education · doi:10.1080/08993408.2020.1832813 · Cited by 22 · see citing works
Assessment Methods and Interventions to Develop Computational Thinking — A Literature Review
Varghese, V. V. V., & Renumol, V. G. (2021) · International Conference on Innovative Trends in Information Technology (ICITIIT) · doi:10.1109/icitiit51526.2021.9399606 · Cited by 19 · see citing works
Investigating Challenges Faced by Learners with Visual Impairments using Block-Based Programming/Hybrid Environments
Mountapmbeme, A., & Ludi, S. A. (2020) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3373625.3417998 · Cited by 17 · see citing works
Enhancing middle school students’ computational thinking competency through game-based learning
Pan, Y., Adams, E. L., Ketterlin–Geller, L. R., Larson, E. C., & Clark, C. (2024) · Educational Technology Research and Development · doi:10.1007/s11423-024-10400-x · Cited by 32 · see citing works
Impact of Educational Coaching Programs and Mentoring Services on Users’ Perception and Preferences: A Qualitative and Quantitative Approach
OKOYE, K. O., Hosseini, S., Arrona‐Palacios, A., & Escamilla, J. (2021) · IEEE Access · doi:10.1109/access.2021.3069130 · Cited by 15 · see citing works
A Phenomenographic Analysis of College Students’ Conceptions of and Approaches to Programming Learning: Insights From a Comparison of Computer Science and Non-Computer Science Contexts
Chou, T.-L., Tang, K., & Tsai, C. (2021) · Journal of Educational Computing Research · doi:10.1177/0735633121995950 · Cited by 14 · see citing works
How is computational thinking assessed in European K-12 education? A systematic review
Babazadeh, M., & Negrini, L. (2022) · International Journal of Computer Science Education in Schools · doi:10.21585/ijcses.v5i4.138 · Cited by 14 · see citing works
Large-scale path modeling of remixing to computational thinking
Xing, W. (2019) · Interactive Learning Environments · doi:10.1080/10494820.2019.1573199 · Cited by 14 · see citing works
Comparing the effects of robots and IoT objects on STEM learning outcomes and computational thinking skills between programming-experienced learners and programming-novice learners
Hu, C., Tseng, H., Chen, M., Goh, A. P. I., & Chen, N. (2020) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt49669.2020.00033 · Cited by 14 · see citing works
Research trends in programming education: A systematic review of the articles published between 2012-2020
Rovshenov, A., & Sarsar, F. (2023) · Journal of Educational Technology and Online Learning · doi:10.31681/jetol.1201010 · Cited by 12 · see citing works
A Systematic Umbrella Review on Computational Thinking Assessment in Higher Education
Zhang, X., Aivaloglou, E., & Specht, M. M. (2024) · European Journal of STEM Education · doi:10.20897/ejsteme/14175 · Cited by 12 · see citing works
Towards high-quality informatics K-12 education in Europe: key insights from the literature
Sampson, D. G., Kampylis, P., Moreno-León, J., & Bocconi, S. (2025) · Smart Learning Environments · doi:10.1186/s40561-025-00366-5 · Cited by 15 · see citing works
Novel Scratch Programming Blocks for Web Scraping
Park, Y., & Shin, Y. (2022) · Electronics · doi:10.3390/electronics11162584 · Cited by 10 · see citing works
BlockLens: Visual Analytics of Student Coding Behaviors in Block-Based Programming Environments
Tsung, S., Wei, H., Li, H., Wang, Y., Xia, M., & Qu, H. (2022) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3491140.3528298 · Cited by 10 · see citing works
Shooting for the Stars
Israel‐Fishelson, R., & Hershkovitz, A. (2020) · Advances in educational technologies and instructional design book series · doi:10.4018/978-1-7998-5074-8.ch012 · Cited by 9 · see citing works
Children's computational thinking as the development of a possibility space
Kopcha, T. J., & Ocak, C. (2023) · Computers and Education Open · doi:10.1016/j.caeo.2023.100156 · Cited by 8 · see citing works
Reflecting on Computational Thinking Studies for High School Education
Sondakh, D. E. (2019) · CogITo Smart Journal · doi:10.31154/cogito.v4i2.136.243-256 · Cited by 7 · see citing works
Impact of Hint Content on Performance and Learning: A Study with Primary School Children in a Scratch Course
Greifenstein, L., Brune, M., Fuchs, T., Heuer, U., & Fraser, G. (2023) · Workshop in Primary and Secondary Computing Education · doi:10.1145/3605468.3605498 · Cited by 7 · see citing works
Enhancing computational thinking in students with autism spectrum disorder and intellectual disabilities: a robot programming approach
Kim, M., Kim, J., & Lee, W. (2024) · International Journal of Developmental Disabilities · doi:10.1080/20473869.2024.2394735 · Cited by 6 · see citing works
Combining Data-Driven Models and Expert Knowledge for Personalized Support to Foster Computational Thinking Skills
Lallé, S., Yalçın, Ö. N., & Conati, C. (2021) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3448139.3448175 · Cited by 6 · see citing works
Predicting Long-Term Student Outcomes from Short-Term EdTech Log Data
Gao, G., Leon, A. A., Jetten, A., Turner, J., Almoubayyed, H., Fancsali, S. E., & Brunskill, E. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706552 · Cited by 5 · see citing works
Using Data to Inform Computing Education Research and Practice
Price, T., Franke, B., Grover, S., & McGill, M. M. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366967 · Cited by 5 · see citing works
Self-citation
Using video analysis and learning analytics to understand programming trajectories in data science activities with Scratch
Fernandez, C. D. O., Freitas, J. A., Lopes, R. D. D., & Blikstein, P. (2022) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3501712.3529742 · Cited by 5 · see citing works
Improving Computational Thinking for Middle School Students Through Python Programming: Interaction Effect Analysis of Grade Level and Gender
Sun, L., & Liu, J. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251332751 · Cited by 4 · see citing works
Integrating Computational Thinking Diagnostic Mechanism and Reflective Learning: An Innovative Approach to Enhance Learning Outcomes in Introductory Programming
Wu, T., Lee, H., Chen, P., Wang, W., & Huang, Y. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70121 · Cited by 5 · see citing works
ESL Teachers’ Experiences While Designing Projects with Block-Based Coding: Language Instruction and Computational Thinking
Kim, E., & Dinç, E. (2025) · Journal of Educational Technology Systems · doi:10.1177/00472395241312652 · Cited by 4 · see citing works
“Doing Physics” And “Doing Code”: Students’ Framing During Computational Modeling in Physics
Conlin, L. D., Hutchins, N. M., Grover, S., & Biswas, G. (2020) · Proceedings. · doi:10.22318/icls2020.733 · Cited by 4 · see citing works
Self-citation
A Data-Driven Approach to Automatically Assessing Concept-Level CS Competencies Based on Student Programs.
Akram, B., Azizolsoltani, H., Min, W., Wiebe, E., Navied, A., Mott, B., Boyer, K. E., & Lester, J. C. (2020) · Cited by 4 · see citing works
A Conceptual Assessment Framework for K-12 Computer Science Rubric Design
Akram, B., Min, W., Wiebe, E., Navied, A., Mott, B., Boyer, K. E., & Lester, J. C. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3372643 · Cited by 3 · see citing works
Design of Mini-Games for Assessing Computational Thinking
Varghese, V. V. V., & Renumol, V. G. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-19-0095-2_70 · Cited by 3 · see citing works
OnlineBioinfo: Leveraging the Teaching of Programming Skills to Life Science Students Through Learning Analytics
de Melo-Minardi, R. C., de Melo, E. C., & Bastos, L. L. (2022) · Frontiers in Education · doi:10.3389/feduc.2022.727019 · Cited by 3 · see citing works
A Novel Three-Way Decision Model for Improving Computational Thinking Based on Grey Correlation Analysis
Xu, R., Jiang, C., & Sun, L. (2022) · Scientific Programming · doi:10.1155/2022/3575457 · Cited by 3 · see citing works
Automated Assessment of Computer Science Competencies from Student Programs with Gaussian Process Regression.
Akram, B., Azizsoltani, H., Min, W., Wiebe, E., Mott, B., Navied, A., Boyer, K. E., & Lester, J. C. (2020) · Educational Data Mining · Cited by 3 · see citing works
Implementation of Learning Analytics in Primary and Secondary School: A Systematic Literature Review
Abdullah, S. H., Puad, M. H. M., Murad, M. A. A., & Marlisah, E. (2022) · International Journal of Academic Research in Progressive Education and Development · doi:10.6007/ijarped/v11-i2/13136 · Cited by 2 · see citing works
Modeling hidden states in learning: a hidden Markov model approach to stealth assessment for problem solving in computational thinking
Liu, Z., & Fulwider, G. C. (2025) · Journal of Research on Technology in Education · doi:10.1080/15391523.2025.2586536 · Cited by 2 · see citing works
Effects of a Self‐Regulated Cooperative Programming Approach on Elementary Students' Programming Knowledge, Metacognitive Awareness, and Self‐Efficacy
Liu, C., Cui, Z., & Weng, X. (2025) · Journal of Computer Assisted Learning · doi:10.1002/jcal.70167 · Cited by 2 · see citing works
Systematic Review of Educational Level and Evaluation Tools for Computational Thinking Skill
Rahim, F. R., Widodo, A., Suhandi, A., & Ha, M. (2024) · Jurnal Penelitian Pendidikan IPA · doi:10.29303/jppipa.v10i2.5209 · Cited by 2 · see citing works
El uso del software Alice como herramienta para el aprendizaje de programación: una revisión de literatura
Morán, J. A. C., Rodas‐Silva, J., & Vinueza, M. (2019) · LACCEI International Multi-Conference for Engineering, Education and Technology (LACCEI) · doi:10.18687/laccei2019.1.1.161 · Cited by 2 · see citing works
Insights Into Young Children's Coding With Data Analytics
Unahalekhaka, A., Blake-West, J. C., & Nguyen, X. (2021) · Advances in early childhood and K-12 education · doi:10.4018/978-1-7998-7308-2.ch015 · Cited by 2 · see citing works
Examining Young Children's Computational Artifacts
Unahalekhaka, A., & Govind, M. (2021) · Advances in early childhood and K-12 education · doi:10.4018/978-1-7998-7308-2.ch014 · Cited by 2 · see citing works
Applying Learning Analytics to Support Instruction
Feng, M., Krumm, A. E., & Grover, S. (2018) · Routledge eBooks · doi:10.4324/9781351136501-12 · Cited by 2 · see citing works
Self-citation
Understanding the Characteristics of Students’ Behavioral Processes in Solving Computational Thinking Problems Based on the Behavioral Sequences
Guo, Q., Li, H., & Zhu, S. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241251397 · Cited by 1 · see citing works
Learning analytics dashboard para apoyar la enseñanza de la programación en educación a distancia
Martínez, V. A. M., Ramírez, L. A., & Montané-Jiménez, L. G. (2024) · Revista de Investigación en Tecnologías de la Información · doi:10.36825/riti.12.26.013 · Cited by 1 · see citing works
Soft Grades: A Calibrated and Accurate Method for Course-Grade Estimation that Expresses Uncertainty
Woodrow, J., & Piech, C. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706568 · Cited by 1 · see citing works
A Learner‐Centred Exploration of Teachers' Solution Pathways in K‐12 Programming‐Based Mathematical Problem‐Solving
Ye, H., Liang, B., & Ng, O. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70102 · Cited by 1 · see citing works
Assessing Computational Thinking Using Log Data: A Scoping Review
Mergianian, C., Hienonen, N., Dindar, M., & Vainikainen, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-03870-8_21 · Cited by 1 · see citing works
Impact of Generative AI-assisted programming on the computational thinking of high school students
Guo, R., Li, G., Miao, H., Pi, Z., & Xie, L. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1703177 · Cited by 1 · see citing works
Exploring K-8 Teachers’ Preferences in a Teaching Augmentation System for Block-Based Programming Environments
Kong, M., Mauriello, M. L., & Pollock, L. (2022) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3564721.3564725 · Cited by 1 · see citing works
Analysis of Learning Data by Problem Type in Educational Programming Language(EPL) Based Learning: Focusing on Entry Which was Developed by Naver
Lee, J., Son, Y., Lee, M., & Lee, J. (2025) · The Journal of Korean Association of Computer Education · doi:10.32431/kace.2025.28.3.002 · Cited by 1 · see citing works
Assessment of Students' Computer Science Focal Knowledge, Skills, and Abilities in Game-Based Learning Environments.
Akram, B. (2019) · NCSU Libraries Repository (North Carolina State University Libraries) · Cited by 1 · see citing works
Método de evaluación del pensamiento computacional y alfabetización en código
Cordenonzi, W. H., & Pino, J. C. D. (2021) · Praxis & Saber · doi:10.19053/22160159.v12.n31.2021.11750
Improvement of Fill-in-the-Blank Questions for Object-Oriented Programming Education
Murata, M., Kato, N., & Kakeshita, T. (2023) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-43393-1_48
Novice Programmers' Unproductive Persistence: Using Learning Analytics to Interrogate Learning Theories
Smith, J. M. (2023) · doi:10.12794/metadc2179234
Optimizing Assessment of Knowledge Transfer Through Computational Thinking: An L&D Project
Chichekian, T., Jawhar, T., Corliss, D., Maheux, C., & Zhang, Y. (2023) · Proceedings. · doi:10.22318/icls2023.232733
Understanding and Addressing Accessibility Barriers Faced by People with Visual Impairments on Block-Based Programming Environments
Mountapmbeme, A. (2022) · doi:10.12794/metadc2048605
Improving Embedded Software Development Courses: A Student Attentive Data-Driven Approach
Kissich, M., Weinbauer, K., & Baunach, M. C. (2025) · European Conference on Software Engineering Education (ECSEE) · doi:10.1145/3723010.3723013
Capturing and Analyzing User Interactions in Block-Based Programming With Starlogo Nova
Gomez, M. J., Wendel, D., & Klopfer, E. D. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016581
From flip to click: a global bibliometric analysis of blended learning trends in programming education (2000–2023)
Samala, A. D., Howard, N.-J., & Rawas, S. (2026) · SN Social Sciences · doi:10.1007/s43545-026-01760-7

A framework for hypothesis-driven approaches to support data-driven learning analytics in measuring computational thinking in block-based programming
Grover, S., Bienkowski, M. A., Basu, S., Eagle, M. J., Diana, N., & Stamper, J. C. (2017) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3027385.3029440 · Cited by 15 · see citing works
Self-citation
Assessing Middle School Students' Computational Thinking Through Programming Trajectory Analysis
Akram, B., Min, W., Wiebe, E., Mott, B., Boyer, K. E., & Lester, J. C. (2019) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3287324.3293798 · Cited by 9 · see citing works
CT-Blocks Analyser: Analysing CT-Blocks Projects
Vinayakumar, R., Soman, K., & Menon, P. (2018) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt.2018.8493970 · Cited by 1 · see citing works
Programming trajectories analytics in block-based programming language learning
Jiang, B., Zhao, W., Zhang, N., & Qiu, F. (2019) · Interactive Learning Environments · doi:10.1080/10494820.2019.1643741 · Cited by 26 · see citing works
Assessing Chinese middle school students’ computational thinking: an evidence-centered design approach
Zhu, S., Guo, Q., Yang, S., Yang, H. H., Li, H., & Sun, Z. (2025) · Educational Technology Research and Development · doi:10.1007/s11423-025-10475-0 · Cited by 3 · see citing works

Authors: Grover, S., Basu, S., Bienkowski, M., Eagle, M., Diana, N., & Stamper, J.
Affiliations: SRI International (US)
Carnegie Mellon University (US)
ORCID: Grover, Shuchi
Basu, Satabdi
Eagle, Michael
Diana, Nicholas
Stamper, John
Venue: ACM Transactions on Computing Education
Publisher: Association for Computing Machinery (ACM)
Published: 2017-08-28
Volume: 17
Issue: 3
Pages: 1-25
ISSN: 1946-6226 (ISSN-L)
DOI: 10.1145/3105910
OpenAlex: W2753422637
Cited by: 93
References: 63
Citations by year: 2026: 2 · 2025: 15 · 2024: 9 · 2023: 9 · 2022: 12 · 2021: 15 · 2020: 16 · 2019: 11 · 2018: 4
FWCI: 10.16
Type: journal article
Language: en
Open access: closed
Licence: www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations: ACM Transactions on Computing Education (publishedVersion)
Funders: National Science Foundation (IIS-1522990)
Topics: Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
Keywords: computational thinking, block-based programming, learning analytics, hypothesis-driven analysis, data-driven analysis, evidence-centered design, Alice programming environment, assessment of computational thinking, high school computer science

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

A Framework for Using Hypothesis-Driven Approaches to Support Data-Driven Learning Analytics in Measuring Computational Thinking in Block-Based Programming Environments

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks