Visual StoryCoder: A Multimodal Programming Environment for Children’s Creation of Stories
Open accessAbstract
Computational thinking (CT) education reaches only a fraction of young children, in part because CT learning tools often require expensive hardware or fluent literacy. Block-based programming environments address these challenges through symbolic graphical interfaces, but users often need instructor support to advance. Alternatively, voice-based tools provide direct instruction on CT concepts but can present memory and navigation challenges to users. In this work, we present Visual StoryCoder, a multimodal tablet application that combines the strengths of each of these approaches to overcome their respective weaknesses. Visual StoryCoder introduces children ages 5–8 to CT through creative storytelling, offers direct instruction via a pedagogical voice agent, and eases use through a block-like graphical interface. In a between-subjects evaluation comparing Visual StoryCoder to a leading block-based programming app for this age group (N = 24), we show that Visual StoryCoder is more understandable to independent learners, leads to higher-quality code after app familiarization, and encourages personally meaningful projects.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For children without access to educational resources or guidance from educators, learning within paradigms without classroom support is often ineffective.
“Moreover, for children without access to educational resources or guidance from educators, learning within paradigms without classroom support is often ineffective [14, 39, 51].”
From ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12 (2024) · cites this paper as [14]
Personally meaningful projects align exceptionally well with Scratch in autonomous learning environments.
“Personally meaningful projects, which allow children to freely choose their themes and forms of implementation, align exceptionally well with Scratch [51] in autonomous learning environments [14].”
From ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12 (2024) · cites this paper as [14]
Storytelling enhances creative capacities when learning CT concepts.
“Further studies, as indicated in [14, 62], have also validated the role of storytelling in enhancing creative capacities when learning CT concepts.”
From ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12 (2024) · cites this paper as [14]
Tailored approaches heighten levels of creativity.
“Echoing the idea of technology-enhanced creativity championed by Dietz et al. [14], we also have explored methods to enhance tool support for children's personalized projects in ChatScratch, acknowledging that such tailored approaches heighten levels of creativity.”
From ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12 (2024) · cites this paper as [14]
Voice interfaces assist children in maintaining attention and dismantle literacy barriers by eliminating the need for written text.
“They not only assist children in maintaining attention [45] but also dismantle literacy barriers by eliminating the need for written text [14].”
From ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12 (2024) · cites this paper as [14]
Autonomous learning without the structured guidance of a classroom is often less effective.
“Although these students can, in theory, avail themselves of publicly available resources online, research shows that autonomous learning without the structured guidance of a classroom is often less effective [14, 39, 51].”
From ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12 (2024) · cites this paper as [14]
Drawing is intuitive and straightforward for children compared to other digital literacy.
“we chose drawing as the medium for assets creation, recognizing its intuitive and straightforward nature for children in comparison to other digital literacy [14, 81].”
From ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12 (2024) · cites this paper as [14]
Children resonate more deeply with the sentiment 'This is something I created', fostering a stronger sense of ownership and achievement.
“They resonate more deeply with the sentiment, ' This is something I created ', fostering a stronger sense of ownership and achievement [14].”
From ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12 (2024) · cites this paper as [14]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
This paper’s reference list could not be split into single entries.
Wing, J. M. (2006) · Communications of the ACM · doi:10.1145/1118178.1118215 · Cited by 8,102 · see citing works
Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333 · see citing works
Barr, V., & Stephenson, C. (2011) · ACM Inroads · doi:10.1145/1929887.1929905 · Cited by 1,692 · see citing works
Defining Computational Thinking for Mathematics and Science Classrooms
Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L. E., & Wilensky, U. J. (2015) · Journal of Science Education and Technology · doi:10.1007/s10956-015-9581-5 · Cited by 1,625 · see citing works
The Scratch Programming Language and Environment
Maloney, J. H., Resnick, M., Rusk, N., Silverman, B., & Eastmond, E. (2010) · ACM Transactions on Computing Education · doi:10.1145/1868358.1868363 · Cited by 1,423 · see citing works
Putting Education in “Educational” Apps
Hirsh‐Pasek, K., Zosh, J. M., Golinkoff, R. M., Gray, J. H., Robb, M. B., & Kaufman, J. (2015) · Gothic.net · doi:10.1177/1529100615569721 · Cited by 1,163 · see citing works
Storytelling alice motivates middle school girls to learn computer programming
Kelleher, C. L., Pausch, R. F., & Kiesler, S. B. (2007) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/1240624.1240844 · Cited by 505 · see citing works
To block or not to block, that is the question
Weintrop, D., & Wilensky, U. J. (2015) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/2771839.2771860 · Cited by 426 · see citing works
Full STEAM Ahead: The Benefits of Integrating the Arts Into STEM
Land, M. H. (2013) · Procedia Computer Science · doi:10.1016/j.procs.2013.09.317 · Cited by 398 · see citing works
Coding as a Playground: Programming and Computational Thinking in the Early Childhood Classroom
Bers, M. U. (2017) · Cited by 293 · see citing works
Flannery, L. P., Silverman, B., Kazakoff, E. R., Bers, M. U., Bontá, P., & Resnick, M. (2013) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/2485760.2485785 · Cited by 256 · see citing works
Measuring Student Learning in Introductory Block-Based Programming
Grover, S., & Basu, S. (2017) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3017680.3017723 · Cited by 251 · see citing works
Block-based programming in computer science education
Weintrop, D. (2019) · Communications of the ACM · doi:10.1145/3341221 · Cited by 217 · see citing works
Patterns for How Users Overcome Obstacles in Voice User Interfaces
Myers, C., Furqan, A., Nebolsky, J., Caro, K., & Zhu, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173580 · Cited by 213 · cited 1× · see citing works
“in analyzing voice interface usage [42], we identifed categories of obstacles users faced and the types of tactics they employed to overcome them.”
MethodConway, M., Audia, S., Burnette, T., et al. (2000) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/332040.332481 · Cited by 212 · see citing works
Habits of programming in scratch
Meerbaum-Salant, O., Armoni, M., & Ben-Ari, M. (2011) · Proceedings of the 16th annual joint conference on Innovation and technology in computer science education · doi:10.1145/1999747.1999796 · Cited by 199 · see citing works
Dr. Scratch: a Web Tool to Automatically Evaluate Scratch Projects
Moreno-León, J., & Robles, G. (2015) · Proceedings of the Workshop in Primary and Secondary Computing Education · doi:10.1145/2818314.2818338 · Cited by 183 · see citing works
Proceedings of the 21st Annual Meeting of the Cognitive Science Society
Sturt, P., Branigan, H. P., & Matsumoto-Sturt, Y. (1999) · Cited by 178 · see citing works
Zhang, Z., Xu, Y., Wang, Y., Yao, B., Ritchie, D., Wu, T., Yu, M., Wang, D., & Li, T. J.-J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517479 · Cited by 174 · see citing works
Sullivan, A., Elkin, M., & Bers, M. U. (2015) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/2771839.2771868 · Cited by 145 · see citing works
Computational thinking in educational activities
Gouws, L. A., Bradshaw, K., & Wentworth, P. (2013) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2462476.2466518 · Cited by 137 · see citing works
Proceedings of the Interaction Design and Children Conference
(2020) · doi:10.1145/3392063 · Cited by 121 · see citing works
Hu, F., Zekelman, A., Horn, M., & Judd, F. (2015) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/2771839.2771866 · Cited by 118 · see citing works
Transferring Skills at Solving Word Problems from Computing to Algebra Through Bootstrap
Schanzer, E., Fisler, K., Krishnamurthi, S., & Felleisen, M. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677238 · Cited by 111 · see citing works
Comparing the Effectiveness of Online Learning Approaches on CS1 Learning Outcomes
Lee, M. J., & Ko, A. J. (2015) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2787622.2787709 · Cited by 109 · see citing works
Druin, A., Revett, I., Schwenn, T. P., Sumida, L., Wagner, R., Montemayor, J., Hendler, J., McAlister, B., Boltman, A., Fiterman, E., et al. (1999) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/302979.303103 · Cited by 102 · see citing works
Interactive storytelling systems for children: using technology to explore language and identity
Bers, M. U., & Cassell, J. (1999) · The Journal of Interactive Learning Research · Cited by 101 · see citing works
Koushik, V., Guinness, D., & Kane, S. K. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300722 · Cited by 91 · see citing works
Learner-Centered Design of Computing Education
Guzdial, M. (2016) · Synthesis lectures on human-centered informatics · doi:10.1007/978-3-031-02216-6 · Cited by 81 · see citing works
Exploring young children's engagement in joint reading with a conversational agent
Xu, Y., & Warschauer, M. (2020) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3392063.3394417 · Cited by 79 · see citing works
Children's storytelling and programming with robotic characters
Ryokai, K., Lee, M. J., & Lim-Breitbart, J. (2009) · ACM Conference on Creativity and Cognition (C&C) · doi:10.1145/1640233.1640240 · Cited by 68 · see citing works
Dietz, G., Le, J. K., Tamer, N., Han, J., Gweon, H., Murnane, E. L., & Landay, J. A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445039 · Cited by 67 · see citing works
Self-citationEvolution of Novice Programming Environments: The Structure Editors of Carnegie Mellon University
Miller, P. L., Pane, J. F., Meter, G., et al. (1994) · Interactive Learning Environments · doi:10.1080/1049482940040202 · Cited by 62 · see citing works
Young Children's Reading and Learning with Conversational Agents
Xu, Y., & Warschauer, M. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3299035 · Cited by 60 · see citing works
Evaluating Speech-Based Smart Devices Using New Usability Heuristics
Wei, Z., & Landay, J. A. (2018) · IEEE Pervasive Computing · doi:10.1109/mprv.2018.022511249 · Cited by 53 · see citing works
Self-citationPortelance, D. J., & Bers, M. U. (2015) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/2771839.2771894 · Cited by 52 · see citing works
DiSalvo, B. J. (2014) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2014.112 · Cited by 44 · see citing works
(2017) · Understanding innovation · doi:10.1007/978-3-319-60967-6 · Cited by 41 · see citing works
Xu, Y., Vigil, V., Bustamante, A. S., & Warschauer, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502050 · Cited by 40 · see citing works
Interacting with structure-oriented editors
Minör, S. (1992) · International Journal of Man-Machine Studies · doi:10.1016/0020-7373(92)90002-3 · Cited by 33 · see citing works
Soleimani, A., Herro, D., & Green, K. E. (2019) · International Journal of Child-Computer Interaction · doi:10.1016/j.ijcci.2019.01.002 · Cited by 31 · see citing works
Kyza, E. Α., Georgiou, Y., Agesilaou, A., & Souropetsis, M. (2021) · Journal of Educational Computing Research · doi:10.1177/07356331211027387 · Cited by 24 · see citing works
Design and Pilot Testing of Subgoal Labeled Worked Examples for Five Core Concepts in CS1
Margulieux, L. E., Morrison, B. B., & Decker, A. (2019) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3304221.3319756 · Cited by 21 · see citing works
Sniffing Through Millions of Blocks for Bad Smells
Techapalokul, P. (2017) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3017680.3022450 · Cited by 19 · see citing works
Bring the Page to Life: Engaging Rural Students in Computer Science Using Alice
Fasy, B. T., Hancock, S. A., Komlos, B. Z., et al. (2020) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3341525.3387367 · Cited by 15 · see citing works
How block-based languages support novices: A Framework for categorizing block-based affordances
Weintrop, D., & Wilensky, U. J. (2017) · Journal of Visual Languages and Sentient Systems · doi:10.18293/vlss2017-011 · Cited by 13 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Chen, L., Xiao, S., Chen, Y., Song, Y., Wu, R., & Sun, L. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642229 · Cited by 60 · see citing works
Liu, D., Zhou, H., & An, P. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642852 · Cited by 38 · see citing works
- Authors
- Dietz, G., Tamer, N., Ly, C., Le, J. K., & Landay, J. A.
- Affiliations
- Stanford University (US)
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-16
- OpenAlex
- W4366547406
- Cited by
- 24
- References
- 46
- Citations by year
- 2026: 6 · 2025: 9 · 2024: 8 · 2023: 1
- FWCI
- 7.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Software › Spreadsheets and End-User Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Educational Robotics and Engineering
- Keywords
- computational thinking, children, storytelling, block-based programming, voice assistants, multimodal interfaces, tablet application, programming education, self-directed learning, code quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
