CoChat
Sign Up
Cited by 38Teaching and Learning ProgrammingEducational Games and GamificationGender and Technology in Education

Promoting Computational Thinking Skills in Non-Computer-Science Students: Gamifying Computational Notebooks to Increase Student Engagement

Open access
De Santo, Alessio, Farah, Juan Carlos, Martinez, Marc Lafuente, Moro, Arielle, Bergram, Kristoffer, Purohit, Aditya Kumar, Felber, Pascal, Gillet, Denis +1 more·2022·IEEE Transactions on Learning Technologies
DOI 10.1109/tlt.2022.3180588
Open this paper in CoChat — free accountOpen PDF

Abstract

Computational thinking (CT) skills are becoming increasingly relevant for future professionals across all domains, beyond computer science (CS). As such, an increasing number of bachelor's and master's programs outside of the CS discipline integrate CT courses within their study program. At the same time, tools such as notebooks and interactive apps designed to support the teaching of programming concepts are becoming ever more popular. However, in non-CS majors, CT might not be perceived as essential, and students might lack the motivation to engage with such tools in order to acquire solid CT skills. This article presents a field study conducted with 115 students during a full semester on a novel computational notebook environment. It evaluates computational notebooks and CT skills in an introductory course on information technology for first-year undergraduates in business and economics. A multidimensional evaluation approach makes use of pre- and post-test surveys, lectures, and self-directed laboratory sessions tracking analytics. Our findings suggest that, in the process of learning CT for non-CS students, engagement in active learning activities can be a stronger determinant of learning outcomes than initial knowledge. Furthermore, gamifying computational notebooks can serve as a strong driver of active learning engagement, even more so than initial motivational factors.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tlt.2022.3180588

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Incorporating computational concepts into learning activities can enhance student engagement.

    “For example, De Santo et al. ( 2022) suggest that by incorporating computational concepts into learning activities, CT can enhance student engagement.”

    From Incorporating Computational Thinking Into Virtual Laboratories to Enhance Learning Motivation, Engagement, and Higher‐Order Thinking Skills (2025) · cites this paper as De Santo et al. 2022

    The Discussion (page unavailable) says the study taught computational thinking (CT) through notebook-based activities and found that gamification increased engagement with those notebooks. The Results section on RQ3 likewise reports greater engagement among students receiving gamified feedback. These passages support increased engagement in a CT learning activity, but attribute the increase to gamified feedback, not to incorporating computational concepts themselves. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • In active learning, learning outcomes are primarily a function of student engagement rather than the level of prior knowledge.

    “In active learning, learning outcomes are primarily a function of student engagement rather than the level of prior knowledge (De Santo et al. 2022;Sung et al. 2021).”

    From Incorporating Computational Thinking Into Virtual Laboratories to Enhance Learning Motivation, Engagement, and Higher‐Order Thinking Skills (2025) · cites this paper as De Santo et al. 2022

    In “Computational Notebooks in a Distance Learning Context” (page unavailable), the paper reports that more active students had higher pass rates and that laboratory performance was a stronger predictor of learning outcomes than initial knowledge. But laboratory performance is not the same as engagement, and the paper says initial knowledge was also strongly linked to outcomes. Its findings in one non-CS course do not establish the claim as a general rule for active learning. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 61

  1. J. C. Neubert et al., "The assessment of 21st century skills in industrial and organizational psychology: Complex and collaborative problem sol- ving," Ind. Org. Psychol., vol. 8, no. 2, pp. 238-268, 2015.
  2. D. Barr, J. Harrison, and L. Conery, "Computational thinking: A digital age skill for everyone," Learn. Leading Technol., vol. 38, no. 6, pp. 20-23, 2011.
  3. A. Yadav, N. Zhou, C. Mayfield, S. Hambrusch, and J. T. Korb, "Introducing computational thinking in education courses," in Proc. 42nd ACM Tech. Symp. Comput. Sci. Educ., 2011, pp. 465-470.
  4. A. Ju skevi cien_ e and V. Dagien _ E, "Computational thinking relationship with digital competence," Informat. Educ., vol. 17, no. 2, pp. 265-284, 2018.
  5. A. Dom ınguez, J. Saenz-de Navarrete, L. De-Marcos, L. Fern andez-Sanz, C. Pag es, and J.-J. Mart ınez-Herr aiz, "Gamifying learning experiences: Prac- tical implications and outcomes," Comput. Educ., vol. 63, pp. 380-392, 2013.
  6. L. Leifheit, "The role of self-concept and motivation within the "computational thinking" approach to early computer science education," Ph.D. dissertation, Dept. Informat., Eberhard Karls Univ. T€ ubingen, T€ ubingen, Germany, 2021.
  7. J. C. Farah, A. Moro, K. Bergram, A. K. Purohit, D. Gillet, and A. Holzer, "Bringing computational thinking to non-STEM undergradu- ates through an integrated notebook application," in Proc. 15th Eur. Conf. Technol.-Enhanced Learn., 2020, pp. 1-14.
  8. E. Eriksson et al., "Widening the scope of Fablearn research: Integrating computational thinking, design and making," in Proc. FabLearn Eur. Conf., 2019, pp. 1-9.
  9. D. Gillet, A. Vozniuk, M. J. Rodr ıguez-Triana, and A. Holzer, "Agile, versatile, and comprehensive social media platform for creating, shar- ing, exploiting, and archiving personal learning spaces, artifacts, and traces," in Proc. World Eng. Educ. Forum, 2016, pp. 1-12.
  10. J. Wing, "Research notebook: Computational thinking-What and why," Link Mag., vol. 6, pp. 20-23, 2011.
  11. C. Wang, J. Shen, and J. Chao, "Integrating computational thinking in STEM education: A literature review," Int. J. Sci. Math. Educ., pp. 1-24, 2021.
  12. C. Lu, R. Macdonald, B. Odell, V. Kokhan, C. Demmans Epp, and M. Cutumisu, "A scoping review of computational thinking assessments in higher education," J. Comput. Higher Educ., pp. 1-46, 2022.
  13. R. DiYanni and A. Borst, "Active learning," in The Craft of College Teaching. Princeton, NJ, USA: Princeton Univ. Press, 2020, pp. 42-60.
  14. L. Johnson, S. Adams Becker, M. Cummins, V. Estrada, A. Freeman, and H. Ludgate, "NMC Horizon Report: 2013 Higher Education edition," New Media Consortium, Austin, TX, USA, 2013.
  15. M. Oliver and K. Trigwell, "Can 'Blended Learning' be redeemed," E- Learn. Digit. Media, vol. 2, no. 1, pp. 17-26, 2005.
  16. M. J. Rodr ıguez-Triana et al., "Monitoring, awareness and reflection in blended technology enhanced learning: A systematic review," Int. J. Technol. Enhanced Learn., vol. 9, nos. 2/3, pp. 126-150, 2017.
  17. S. Van Goidsenhoven, D. Bogdanova, G. Deeva, S. vanden Broucke, J. De Weerdt, and M. Snoeck, "Predicting student success in a blended learning environment," in Proc. 10th Int. Conf. Learn. Anal. Knowl., 2020, pp. 17-25.
  18. R. Collopy and J. M. Arnold, "To blend or not to blend: Online-only and blended learning environments," Issues Teacher Educ., vol. 18, no. 2, pp. 85-101, 2009.
  19. Q. Chu, X. Yu, Y. Jiang, and H. Wang, "Data analysis of blended learn- ing in Python programming," in Proc. Int. Conf. Algorithms Archit. Par- allel Process., 2018, pp. 209-217.
  20. S. Kross and P. J. Guo, "Practitioners teaching data science in industry and academia: Expectations, workflows, and challenges," in Proc. CHI Conf. Hum. Factors Comput. Syst., 2019, pp. 1-14.
  21. A. Y. Wang, A. Mittal, C. Brooks, and S. Oney, "How data scientists use computational notebooks for real-time collaboration," in Proc. ACM Human-Comput. Interact., vol. 3, 2019, Art. no. 39.
  22. K. O'Hara, D. Blank, and J. Marshall, "Computational notebooks for AI education," in Proc. 28th Int. Flairs Conf., 2015, pp. 263-268.
  23. J. M. Perkel, "Why Jupyter is Data Scientists' Computational notebook of choice," Nature, vol. 563, no. 7732, pp. 145-147, 2018.
  24. A. Radenski, ""Python First": A lab-based digital introduction to com- puter science," SIGCSE Bull., vol. 38, no. 3, pp. 197-201, 2006.
  25. A. Cardoso, J. Leitão, and C. Teixeira, "Using the Jupyter notebook as a tool to support the teaching and learning processes in engineering courses," in Proc. Int. Conf. Interact. Collab. Learn., 2018, pp. 227-236.
  26. M. Zastre, "Jupyter notebook in CS1: An experience report," in Proc. Western Can. Conf. Comput. Educ., 2019, pp. 1-6.
  27. S. H. Edwards, D. S. Tilden, and A. Allevato, "Pythy: Improving the introductory python programming experience," in Proc. 45th ACM Tech. Symp. Comput. Sci. Educ., 2014, pp. 641-646.
  28. P. J. Guo, "Online python tutor: Embeddable web-based program visual- ization for CS education," in Proc. 44th ACM Tech. Symp. Comput. Sci. Educ., 2013, pp. 579-584.
  29. P. Jupyter et al., "nbgrader: A tool for creating and grading assignments in the Jupyter notebook," J. Open Source Educ., vol. 2, no. 16, 2019, Art. no. 32.
  30. A. Rule, A. Tabard, and J. D. Hollan, "Exploration and explanation in computational notebooks," in Proc. CHI Conf. Hum. Factors Comput. Syst., 2018, pp. 1-12.
  31. M. Borowski, J. Zagermann, C. N. Klokmose, H. Reiterer, and R. R€ adle, "Exploring the benefits and barriers of using computational notebooks for collaborative programming assignments," in Proc. 51st ACM Tech. Symp. Comput. Sci. Educ., 2020, pp. 468-474.
  32. M. Gagn e and E. L. Deci, "The history of self-determination theory in psychology and management," in The Oxford Handbook of Work Engagement, Motivation, and Self-Determination Theory. Oxford, U.K.: Oxford Univ. Press, 2014.
  33. R. J. Vallerand, "A hierarchical model of intrinsic and extrinsic motiva- tion in sport and exercise," Adv. Motivation Sport Exercise, vol. 2, pp. 263-319, 2001.
  34. E. L. Deci and R. M. Ryan, "A motivational approach to self: Integration in personality," in Nebraska Symposium on Motivation, 1990: Perspectives on Motivation, R. A. Dienstbier, Ed. Lincoln, USA: Univ. Nebraska Press, 1991, pp. 237-288.
  35. I. Kotini and S. Tzelepi, "A gamification-based framework for develop- ing learning activities of computational thinking," in Gamification in Education and Business. Berlin, Germany: Springer, 2015, pp. 219-252.
  36. D. L. Butler and P. H. Winne, "Feedback and self-regulated learning: A theoretical synthesis," Rev. Educ. Res., vol. 65, no. 3, pp. 245-281, 1995.
  37. L.-K. Lee, T.-K. Cheung, L.-T. Ho, W.-H. Yiu, and N.-I. Wu, "Learning computational thinking through gamification and collaborative learning," in Proc. Int. Conf. Blended Learn., 2019, pp. 339-349.
  38. C. W. Tan, P.-D. Yu, and L. Lin, "Teaching computational thinking using mathematics gamification in computer science game tournaments," in Computational Thinking Education. Singapore: Springer, 2019, pp. 167-181.
  39. F. Pires, F. M. M. Lima, R. Melo, J. R. S. Bernardo, and R. de Freitas, "Gamification and engagement: Development of computational thinking and the implications in mathematical learning," in Proc. IEEE 19th Int. Conf. Adv. Learn. Technol., 2019, vol. 2161, pp. 362-366.
  40. L. Gouws, K. Bradshaw, and P. Wentworth, "First year student perfor- mance in a test for computational thinking," in Proc. South Afr. Inst. Comput. Sci. Inf. Technol. Conf., 2013, pp. 271-277.
  41. V. Roadrangka, R. H. Yeany, and M. J. Padilla, "The construction and vali- dation of group assessment of logical thinking (GALT)," in Proc. Nat. Assoc. Res. Sci. Teach. Annu. Meeting, 1983, pp. 148-154.
  42. B. Kim, T. Kim, and J. Kim, "Paper-and-pencil programming strategy toward computational thinking for non-majors: Design your solution," J. Educ. Comput. Res., vol. 49, no. 4, pp. 437-459, 2013.
  43. M. Lafuente, O. L evêque, I. Ben ıtez, C. Hardebolle, and J. Dehler-Zufferey, "Assessing computational thinking: Development and validation of the algo- rithmic thinking test for adults," J. Educ. Comput. Res., pp. 1905-1914, 2022.
  44. O. Korkmaz, R. C ¸akir, and M. Y. € Ozden, "A validity and reliability study of the computational thinking scales (CTS)," Comput. Hum. Behav., vol. 72, pp. 558-569, 2017.
  45. M. Ya gcı, "A valid and reliable tool for examining computational think- ing skills," Educ. Inf. Technol., vol. 24, no. 1, pp. 929-951, 2019.
  46. S. Herzog and N. A. Bowman, Validity and Limitations of College Stu- dent Self-Report Data: New Directions for Institutional Research, vol. 110. Hoboken, NJ, USA: Wiley, 2011.
  47. T. T. Yuen and K. A. Robbins, "A qualitative study of students' compu- tational thinking skills in a data-driven computing class," ACM Trans. Comput. Educ., vol. 14, no. 4, 2014, Art. no. 27.
  48. S. Atmatzidou and S. Demetriadis, "Advancing students' computa- tional thinking skills through educational robotics: A study on age and gender relevant differences," Robot. Auton. Syst., vol. 75, pp. 661-670, 2016.
  49. Y.-H. Lai, S.-Y. Chen, C.-F. Lai, Y.-C. Chang, and Y.-S. Su, "Study on enhancing AIoT computational thinking skills by plot image-based VR," Interact. Learn. Environ., vol. 29, pp. 482-495, 2021.
  50. J. D. Hunter, "Matplotlib: A 2D graphics environment," Comput. Sci. Eng., vol. 9, no. 3, pp. 90-95, 2007.
  51. A. L. Strauss, Qualitative Analysis for Social Scientists. Cambridge, U.K.: Cambridge Univ. Press, 1987.
  52. J. Brooke, "SUS: A 'quick and dirty' usability scale," Usability Eval. Ind., vol. 189, 1996.
  53. J. Hair, C. L. Hollingsworth, A. B. Randolph, and A. Y. L. Chong, "An updated and expanded assessment of PLS-SEM in information systems research," Ind. Manage. Data Syst., vol. 117, pp. 442-458, 2017.
  54. K. A. Bollen, "Latent variables in psychology and the social sciences," Annu. Rev. Psychol., vol. 53, pp. 605-634, 2002.
  55. N. Gillet, R. J. Vallerand, M.-A. K. Lafreniere, and J. S. Bureau, "The mediating role of positive and negative affect in the situational motiva- tion-performance relationship," Motivation Emotion, vol. 37, no. 3, pp. 465-479, 2013.
  56. I. J. Quitadamo and M. J. Kurtz, "Learning to improve: Using writing to increase critical thinking performance in general education biology," CBE Life Sci. Educ., vol. 6, no. 2, pp. 140-154, 2007.
  57. O. A. Pilkington, "Active learning for an online composition classroom: Blogging as an enhancement of online curriculum," J. Educ. Technol. Syst., vol. 47, no. 2, pp. 213-226, 2018.
  58. Y. R. Tausczik and J. W. Pennebaker, "The psychological meaning of words: LIWC and computerized text analysis methods," J. Lang. Soc. Psychol., vol. 29, no. 1, pp. 24-54, 2010.
  59. Juan Carlos Farah received the B.A. degree in eco- nomics from Harvard University, Cambridge, MA, USA, in 2008, and the M.Sc. degree in computing from Imperial College London, London, U.K., in 2015. He is currently working toward the Ph.D. degree in robotics, control, and intelligent systems with the Ecole Polytech- nique F ed erale de Lausanne, Lausanne, Switzerland. His current research interests include human-com- puter interaction and the perception of anthropomorphic traits in intelligent conversational agents.
  60. Arielle Moro received the Ph.D. degree in informa- tion systems from the University of Lausanne, Lau- sanne, Switzerland, in 2018. She is currently a Postdoctoral Fellow with the Uni- versity of Neuchâtel, Neuchâtel, Switzerland. Her research thesis explored the tradeoff between predict- ing mobility and preserving location data privacy. Her research interests include analyzing human behavior in various domains (e.g., mobility, sustainability, and learning) using machine learning techniques.
  61. Dr. Gillet is also an Associate Editor for IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES and International Journal of Emerging Technologies in Learning.

Cited by

Sorted by citations

Authors
De Santo, A., Farah, J. C., Martinez, M. L., Moro, A., Bergram, K., Purohit, A. K., Felber, P., Gillet, D., & Holzer, A.
Affiliations
  • University of Neuchâtel, Neuchâtel, Switzerland
  • École Polytechnique Fédérale de Lausanne (CH)
  • Ecole Polytechnique Federale de Lausanne, Ecublens, Switzerland
  • University of Neuchâtel (CH)
ORCID
Venue
IEEE Transactions on Learning Technologies
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2022-06-01
Volume
15
Issue
3
Pages
392-405
ISSN
  • 1939-1382 (ISSN-L)
  • 2372-0050
DOI
10.1109/tlt.2022.3180588
OpenAlex
W4285260634
Cited by
38
References
7
Citations by year
2026: 8 · 2025: 9 · 2024: 15 · 2023: 6
FWCI
5.80
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Learning Technologies (publishedVersion, cc-by)
  • ArODES (HES-SO (https://www.hes-so.ch/)) (submittedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
Funders
Swissuniversities
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Social Sciences › Psychology › Developmental and Educational Psychology › Educational Games and Gamification
  • Social Sciences › Gender Studies › Gender and Technology in Education
Keywords
computational thinking, computational notebooks, gamification, non-computer majors, student engagement, active learning, programming education, economics and business students, learning analytics, student motivation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review