What factors will affect the effectiveness of using ChatGPT to solve programming problems? A quasi-experimental study
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
The emergence of ChatGPT has sparked new expectations for AI-empowered educational transformation. However, it remains unknown which factors affect its effectiveness in empowering learners to solve programming problems. Therefore, this study employed a quasi-experimental research design and used Python graphing in programming education as an example to investigate the factors influencing the effectiveness of learners in applying ChatGPT to problem-solving. Findings: AI literacy significantly influences learners’ effectiveness in using ChatGPT to solve problems, with AI awareness and AI usage being key factors. The knowledge base of programming language significantly affects learners’ effectiveness in applying ChatGPT to solve programming problems. Learners’ cognitive level of ChatGPT significantly influences their effectiveness in applying ChatGPT to problem-solving, while usage intention does not have a significant impact. However, learners’ intention to use ChatGPT significantly improves after application. Based on these findings, this study proposes that in the process of empowering education with Artificial Intelligence Generated Content (AIGC) products, the focus on learners should shift from cultivating their AI usage to AI literacy, laying the foundation for empowering learning with AIGC products. It is suggested to shift from mastering specific knowledge to graph-based rules as a method for empowering learning with AIGC products. Additionally, the focus should shift from enhancing learners’ intention to use the technology to strengthen their technological awareness, thereby creating practical pathways for empowering learning with AIGC products.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1×
“…(Surameery and Shakor, 2023; Frieder et al., 2023), as well as its impact on fi nancial education (Dowling and Lucey, 2023), medical education (Kung et al., 2023; Gilson et al., 2022), second-language education (Yan, 2023), engineering education (Qadir, 2022), and programming education…”
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
Exploring the impact of artificial intelligence on teaching and learning in higher education
Popenici, S. A. D., & Kerr, S. (2017) · Research and Practice in Technology Enhanced Learning · doi:10.1186/s41039-017-0062-8 · Cited by 2,019 · cited 1×
“With the exponential development of information technology, the integration of emerging technologies with AI as the core of educational reforms has become an unstoppable trend (Popenici and Kerr, 2017).”
What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00237-x · Cited by 2,003 · cited 1×
“…and internal mechanisms, educational application scenarios and future challenges, educational value and limitations, ethical issues, and potential risks (Zhai, 2022; Cooper, 2023; Kohnke et al., 2023; Humphry and Fuller, 2023; Su and Yang, 2023; Bauer et al., 2023; Tlili et al., 2023).”
A SWOT analysis of ChatGPT: Implications for educational practice and research
Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2195846 · Cited by 1,384
Examining Science Education in ChatGPT: An Exploratory Study of Generative Artificial Intelligence
Cooper, G. (2023) · Journal of Science Education and Technology · doi:10.1007/s10956-023-10039-y · Cited by 1,263 · cited 1×
“…and internal mechanisms, educational application scenarios and future challenges, educational value and limitations, ethical issues, and potential risks (Zhai, 2022; Cooper, 2023; Kohnke et al., 2023; Humphry and Fuller, 2023; Su and Yang, 2023; Bauer et al., 2023; Tlili et al., 2023).”
Ethical principles for artificial intelligence in education
Nguyen, A., Ngo, H. N., Hong, Y., Dang, B., & Nguyen, B.-P. T. (2023) · Education and Information Technologies · doi:10.1007/s10639-022-11316-w · Cited by 1,228
ChatGPT for Education and Research: Opportunities, Threats, and Strategies
Rahman, M. M., & Watanobe, Y. (2023) · Applied Sciences · doi:10.3390/app13095783 · Cited by 1,080 · cited 2×
“…of learners learning programming with the assistance of ChatGPT, such as the opportunities for scientists and engineers to learn programming using AI tools like ChatGPT (Guo, 2023), and the methods and limitations of ChatGPT in enhancing learners ’ programming skills (Rahman and Watanobe, 2023).”
Wang, B., Rau, P.-L. P., & Yuan, T. (2023) · Behaviour & Information Technology · doi:10.1080/0144929x.2022.2072768 · Cited by 1,036
Cited by
Wang, C., Wang, H., Li, Y., Dai, J., Gu, X., & Yu, T. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2383033 · Cited by 353
Self-citationHow ChatGPT impacts student engagement from a systematic review and meta-analysis study
Heung, Y. M. E., & Chiu, T. K. F. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100361 · Cited by 229
Sun, L., & Zhou, L. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241277937 · Cited by 129
“Also, participants who used the ChatGPTaid had more positive attitudes and found the task easier (Jing et al., 2024).”
Yang, T.-C., Hsu, Y.-C., & Wu, J.-Y. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2025.2450659 · Cited by 80
García-Alonso, E. M., León-Mejía, A. C., Sánchez-Cabrero, R., & Guzmán-Ordaz, R. (2024) · Behavioral Sciences · doi:10.3390/bs14070612 · Cited by 53
Metadata
- Authors
- Jing, Y., Wang, H., Chen, X., & Wang, C.
- Venue
- Humanities and Social Sciences Communications
- Published
- 2024-02-26
- Volume
- 11
- Issue
- 1
- ISSN
- 2055-1045 (ISSN-L)
- 2662-9992
- OpenAlex
- W4392154075
- Cited by
- 73
- References
- 32
- FWCI
- 28.29
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: What factors will affect the effectiveness of using ChatGPT to solve programming problems? A quasi-experimental study. https://app.cochat.ai/papers/10.1057/s41599-024-02751-w
- The paper
- Jing, Y., Wang, H., Chen, X., & Wang, C. (2024). What factors will affect the effectiveness of using ChatGPT to solve programming problems? A quasi-experimental study. Humanities and Social Sciences Communications. https://doi.org/10.1057/s41599-024-02751-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.