CoChat
Sign Up
Cited by 190

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.

Would ChatGPT-facilitated programming mode impact college students’ programming behaviors, performances, and perceptions? An empirical study

Sun, Dan, Boudouaia, Azzeddine, Zhu, Chengcong, Li, Yan·2024·International Journal of Educational Technology in Higher Education
DOI 10.1186/s41239-024-00446-5
Open this paper in CoChat — free account

Abstract

ChatGPT, an AI-based chatbot with automatic code generation abilities, has shown its promise in improving the quality of programming education by providing learners with opportunities to better understand the principles of programming. However, limited empirical studies have explored the impact of ChatGPT on learners’ programming processes. This study employed a quasi-experimental design to explore the possible impact of ChatGPT-facilitated programming mode on college students’ programming behaviors, performances, and perceptions. 82 college students were randomly divided into two classes. One class employed ChatGPT-facilitated programming (CFP) practice and the other class utilized self-directed programming (SDP) mode. Mixed methods were utilized to collect multidimensional data. Data analysis uncovered some intriguing results. Firstly, students in the CFP mode had more frequent behaviors of debugging and receiving error messages, as well as pasting console messages on the website and reading feedback. At the same time, students in the CFP mode had more frequent behaviors of copying and pasting codes from ChatGPT and debugging, as well as pasting codes to ChatGPT and reading feedback from ChatGPT. Secondly, CFP practice would improve college students’ programming performance, while the results indicated that there was no statistically significant difference between the students in CFP mode and the SDP mode. Thirdly, student interviews revealed three highly concerned themes from students' user experience about ChatGPT: the services offered by ChatGPT, the stages of ChatGPT usage, and experience with ChatGPT. Finally, college students’ perceptions toward ChatGPT significantly changed after CFP practice, including its perceived usefulness, perceived ease of use, and intention to use. Based on these findings, the study proposes implications for future instructional design and the development of AI-powered tools like ChatGPT.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s41239-024-00446-5

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. Bau, D., Gray, J., Kelleher, C., Sheldon, J., & Turbak, F. (2017). Learnable programming: Blocks and beyond. Communications of the ACM, 60(6), 72–80. https://doi.org/10.48550/arXiv.1705.09413 · DOI
  2. Chen, B., Resendes, M., Chai, C. S., & Hong, H. Y. (2017). Two tales of time: Uncovering the significance of sequential patterns among contribution types in knowledge-building discourse. Interactive Learning Environments, 25(2), 162–175. https://doi.org/10.1080/10494820.2016.1276081 · DOI
  3. Chen, E., Huang, R., Chen, H. S., Tseng, Y. H., & Li, L. Y. (2023). GPTutor: A ChatGPTpowered programming tool for code explanation. arXiv preprint arXiv:2305.01863. · DOI
  4. Cheng, G. (2019). Exploring factors influencing the acceptance of visual programming environment among boys and girls in primary schools. Computers in Human Behavior, 92, 361–372. https://doi.org/10.1016/j.chb.2018.11.043 · DOI
  5. Cohen, J. (1988). The effect size (pp. 77–83). Routledge.
  6. Cohen, L., Manion, L., & Morrison, K. (2013). Research methods in education. Routledge · DOI
  7. Elkhodr, M., Gide, E., Wu, R., & Darwish, O. (2023). ICT students’ perceptions towards chatgpt: An experimental reflective lab analysis. STEM Education, 3(2), 70–88. https://doi.org/10.3934/steme.2023006 · DOI
  8. Faraone, S. V., & Dorfman, D. D. (1987). Lag sequential analysis: Robust statistical methods. Psychological Bulletin, 101(2), 312–323. https://doi.org/10.1037/0033-2909.101.2.312 · DOI
  9. Firaina, R., & Sulisworo, D. (2023). Exploring the usage of ChatGPT in higher education: Frequency and impact on productivity. Buletin Edukasi Indonesia, 2(01), 39–46. https://doi.org/10.56741/bei.v2i01.310 · DOI
  10. Gervasi, O., Murgante, B., Misra, S., Garau, C., Blečić, I., Taniar, D., Apduhan, B. O., Rocha, A. C., Tarantino, A. C., & E., & Torre, C. M. (2021). Inference engines performance in reasoning tasks for intelligent tutoring systems. Computational science and its applications (pp. 471–482). Springer International Publishing AG. https://doi.org/10.1007/978-3-030-86960-1_33 · DOI
  11. Ghatrifi, M. O., Amairi, J. S., & Thottoli, M. M. (2023). Surfing the technology wave: An international perspective on enhancing teaching and learning in accounting. Computers and Education. Artificial Intelligence, 4, 100144. https://doi.org/10.1016/j.caeai.2023.100144 · DOI
  12. Iqbal, N., Ahmed, H., & Azhar, K. A. (2022). Exploring teachers’ attitudes towards using Chat GPT. Global Journal for Management and Administrative Sciences, 3(4), 97–111. https://doi.org/10.46568/gjmas.v3i4.163 · DOI · Paper page
  13. Jancheski, M. (2017). Improving teaching and learning computer programming in schools through educational software. Olympiads in Informatics, 11(1), 55–75. https://doi.org/10.15388/ioi.2017.05 · DOI
  14. Khan, I., Al Sadiri, A., Ahmad, A. R., & Jabeur, N. (2019). Tracking student performance in introductory programming by means of machine learning. In Proceedings of MEC international conference on big data and smart city (ICBDSC) (pp.1–6). · DOI
  15. Klasnja-Milićević, A., Vesin, B., Ivanović, M., Budimac, Z., & Jain, L. C. (2016). Case study: Design and implementation of programming tutoring system. Springer International Publishing AG.
  16. Limna, P., Kraiwanit, T., Jangjarat, K., Klayklung, P., & Chocksathaporn, P. (2023). The use of chatgpt in the digital era: Perspectives on Chatbot Implementation. Journal of Applied Learning & Teaching, 6(1), 64–74. https://doi.org/10.37074/jalt.2023.6.1.32 · DOI
  17. Looi, C. K., How, M. L., Wu, L. K., Seow, P., & Liu, L. (2018). Analysis of linkages between an unplugged activity and the development of computational thinking. Computer Science Education, 28(3), 255–279. https://doi.org/10.1080/08993408.2018.1533297 · DOI
  18. Lu, O. H. T., Huang, J. C. H., Huang, A. Y. Q., & Yang, S. J. H. (2017). Applying learning analytics for improving students engagement and learning outcomes in an MOOCs enabled collaborative programming course. Interactive Learning Environments, 25(2), 220–234. https://doi.org/10.1080/10494820.2016.1278391 · DOI
  19. Malik, S. I., Ashfque, M. W., Tawafak, R. M., Al-Farsi, G., Ahmad Usmani, N., & Hamza Khudayer, B. (2022). A chatbot to facilitate student learning in a programming 1 course: A gendered analysis. International Journal of Virtual and Personal Learning Environments, 12(1), 1–20. https://doi.org/10.4018/IJVPLE.310007 · DOI
  20. Nurbekova, Z., Tolganbaiuly, T., Nurbekov, B., Sagimbayeva, A., & Kazhiakparova, Z. (2020). Project-based learning technology: An example in programming microcontrollers. International Journal of Emerging Technologies in Learning, 15(11), 218–227. https://doi.org/10.3991/ijet.v15i11.13267 · DOI
  21. Peng, R., Hu, Q., & Kouider, B. (2023). Teachers’ acceptance of online teaching and emotional labor in the EFL context. Sustainability, 15(18), 13893–13893. https://doi.org/10.3390/su151813893 · DOI
  22. Qureshi, B. (2023). Exploring the use of chatgpt as a tool for learning and assessment in undergraduate computer science curriculum: Opportunities and challenges. arXiv preprint arXiv:2304.11214. · arXiv
  23. Ray, B., Posnett, D., Filkov, V., & Devanbu, P. (2014). A large scale study of programming languages and code quality in github. In Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering (pp. 155–165). · DOI
  24. Sánchez, R. A., & Hueros, A. D. (2010). Motivational factors that influence the acceptance of Moodle using TAM. Computers in Human Behavior, 26(6), 1632–1640. https://doi.org/10.1016/j.chb.2010.06.011 · DOI
  25. Shen, G., Yang, S., Huang, Z., Yu, Y., & Li, X. (2023). The prediction of programming performance using student profiles. Education and Information Technologies, 28(1), 725–740. https://doi.org/10.1007/s10639-022-11146-w · DOI
  26. Sivasakthi, M. (2017). Classification and prediction based data mining algorithms to predict students' introductory programming performance. In Proceedings of international conference on inventive computing and informatics (ICICI) (pp. 346–350). · DOI
  27. Skalka, J., Drlik, M., Benko, L., Kapusta, J., Pino, D., Rodríguez, J. C., Smyrnova-Trybulska, E., Stolinska, A., Svec, P., & Turcinek, P. (2021). Conceptual framework for programming skills development based on microlearning and automated source code evaluation in virtual learning environment. Sustainability, 13(6), 3293. https://doi.org/10.3390/su13063293 · DOI
  28. Stehle, S. M., & Peters-Burton, E. E. (2019). Developing student 21st century skills in selected exemplary inclusive STEM high schools. International Journal of STEM Education, 6(1), 1–15. https://doi.org/10.1186/s40594-019-0192-1 · DOI
  29. Sun, D., Ouyang, F., Li, Y., & Zhu, C. (2021b). Comparing learners’ knowledge, behaviors, and attitudes between two instructional modes of computer programming in secondary education. International Journal of STEM Education, 8(1), 54–54. https://doi.org/10.1186/s40594-021-00311-1 · DOI
  30. Sun, D., Ouyang, F., Li, Y., & Chen, H. (2021c). Three contrasting pairs’ collaborative programming processes in China’s secondary education. Journal of Educational Computing Research, 59(4), 740–762. https://doi.org/10.1177/0735633120973430 · DOI
  31. Sun, L., Hu, L., & Zhou, D. (2021d). Which way of design programming activities is more effective to promote K-12 students’ computational thinking skills?A meta-analysis. Journal of Computer Assisted Learning. https://doi.org/10.1111/jcal.12545 · DOI
  32. Sun, D., Xu, F., & Li, Y. (2021a). Using learning analytics in understanding students’ behavior in block-based and text-based programming modality. In 2021 tenth international conference of educational innovation through technology (EITT). https://doi.org/10.1109/eitt53287.2021.00060 · DOI
  33. Teo, T., Luan, W. S., & Sing, C. C. (2008). A cross-cultural examination of the intention to use technology between Singaporean and Malaysian pre-service teachers: An application of the technology acceptance model (TAM). Educational Technology and Society, 11(4), 265–280.
  34. Thongkoo, K., Daungcharone, K., & Thanyaphongphat, J. (2020). Students’ acceptance of digital learning tools in programming education course using technology acceptance model. IEEE Xplore. https://doi.org/10.1109/ECTIDAMTNCON48261.2020.9090771 · DOI
  35. Tian, H., Lu, W., Li, T. O., Tang, X., Cheung, S. C., Klein, J., & Bissyand ́e, T. F. (2023). Is ChatGPT the ultimate programming an assistant–How far is it?. arXiv preprint arXiv: 2304.11938. · arXiv
  36. Tom, M. (2015). Five cs framework: A student-centered approach for teaching programming courses to students with diverse disciplinary background. Journal of Learning Design, 8(1), 21–27. https://doi.org/10.5204/jld.v8i1.193 · DOI
  37. Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management Science, 46(2), 186–204. https://doi.org/10.1287/mnsc.46.2.186.11926 · DOI · Paper page
  38. Wang, H., Tlili, A., Huang, R., Cai, Z., Li, M., Cheng, Z., Yang, D., Li, M., Zhu, X., & Fei, C. (2023). Examining the applications of intelligent tutoring systems in real educational contexts: A systematic literature review from the social experiment perspective. Education and Information Technologies, 28(7), 9113–9148. https://doi.org/10.1007/s10639-022-11555-x · DOI
  39. Xie, Y., Boudouaia, A., Xu, J., AL-Qadri, A. H., Khattala, A., Li, Y., & Aung, Y. M. (2023). A study on teachers’ continuance intention to use technology in English instruction in western China junior secondary schools. Sustainability, 15(5), 4307. https://doi.org/10.3390/su15054307 · DOI
  40. Yang, F.-Y., & Tsai, C.-C. (2008). Investigating university student preferences and beliefs about learning in the web-based context. Computers and Education, 50, 1284–1303. · DOI
  41. Yilmaz, R., & Yilmaz, F. G. K. (2023b). The effect of generative artificial intelligence (AI)-based tool use on students’ computational thinking skills, programming self-efficacy and motivation. Computers and Education. Artificial Intelligence, 4, 100147. https://doi.org/10.1016/j.caeai.2023.100147 · DOI · Paper page
  42. Yilmaz, R., & Yilmaz, F. G. K. (2023c). The effect of generative artificial intelligence (AI) based tool use on students’ computational thinking skills, programming self-efficacy and motivation. Computers and Education: Artificial Intelligence, 4, 100147.

Cited by

Sorted by citations

Metadata

Authors
Sun, D., Boudouaia, A., Zhu, C., & Li, Y.
Venue
International Journal of Educational Technology in Higher Education
Publisher
Springer Science and Business Media LLC
Published
2024-02-22
Volume
21
Issue
1
ISSN
2365-9440
DOI
10.1186/s41239-024-00446-5
Cited by
190
References
10
Language
en
Licence
Funders
National Natural Science Foundation of China (62307011, 61977057)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review