The Role of Students’ Higher-Order Thinking Skills in the Relationship between Academic Achievements and Machine Learning Using Generative AI Chatbots
Open accessAbstract
Students’ perspectives on using generative artificial intelligence (AI) chatbots and machine learning are crucial in shaping the design, development, and implementation of their learning projects across various disciplines. Cognitive thinking, a key aspect of AI-related machine learning, aims to replicate human intelligence and behavior. However, the relation between cognitive thinking and knowledge acquisition is often overlooked. This cross-sectional study empirically examines the relationship between academic achievement and students’ attitudes toward machine learning, particularly through the use of generative AI chatbots. It specifically focuses on the role of higher-order thinking skills—such as problem-solving, critical thinking, and creativity—as both mediators and moderators in this relationship. A total of four hundred sixteen undergraduate students (n=416) from diverse academic backgrounds voluntarily took part in a project, in which they designed and developed generative AI chatbots in media and information literacy courses. The findings indicate that creativity mediated the relationship between academic achievements and attitudes toward machine learning, but its moderating impact was not significant. Problem-solving and critical thinking did not show significant mediating effects on attitudes toward machine learning, while they showed significant moderating effects in the connection between academic performance and attitudes toward machine learning. This study contributes by elucidating the interrelationships between students’ higher-order thinking skills, academic performance, and attitudes on the use of AI and machine learning technologies. By highlighting the mediating role of creativity and the moderating effects of problem-solving and critical thinking, this study offers a deeper understanding of how these skills shape students’ perceptions of AI.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Cornell Critical Thinking Test Level Z was one of the few standardized assessments employed, serving as a benchmark to quantify CT skills.
“The Cornell Critical Thinking Test Level Z was one of the few standardized assessments employed, serving as a benchmark to quantify CT skills [59,70].”
From A Systematic Literature Review on the Pedagogical Implications and Impact of GenAI on Students' Critical Thinking (2026) · cites this paper as [70]
The Higher-order thinking skills section (page unavailable) describes an adapted eleven-item questionnaire, including critical-thinking items, as the assessment this study employed. It does not identify the Cornell Critical Thinking Test Level Z as an instrument used or a benchmark; a search for its name found no passage in our copy. (LLM, conf=0.78)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredOther tools such as Gemini, Ernie Bot, and multi-tool workflows were less common.
“Other tools such as Gemini, Ernie Bot, and multi-tool workflows were less common [61,70], and a few studies developed custom GPT-based systems ( [50,59,84,89,90]).”
From A Systematic Literature Review on the Pedagogical Implications and Impact of GenAI on Students' Critical Thinking (2026) · cites this paper as [70]
The Introduction (page unavailable) names Gemini as an example of a generative AI chatbot. It does not compare how commonly tools were used, and the retrieved passages do not address Ernie Bot or multi-tool workflows; those parts of the attributed claim are unsupported. (LLM, conf=0.78)
Experimental — accuracy not yet measuredPellas examined associations between attitudes toward machine learning and performance outcomes, reporting relationships with problem-solving and CT measures.
“Pellas [70] examined associations between attitudes toward machine learning and performance outcomes, reporting relationships with problem-solving and CT measures.”
From A Systematic Literature Review on the Pedagogical Implications and Impact of GenAI on Students' Critical Thinking (2026) · cites this paper as [70]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- Cohen, J. (2013). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge.
- Jaccard, J., & Turrisi, R. (2003). Interaction effects in multiple regression (No. 72). Sage Publications.
- Lee, Y. (2019). An analysis of the influence of block-type programming language-based artificial intelligence education on the learner's attitude in artificial intelligence. Journal of The Korean Association of Information Education, 23(2), 189-196. https://m.earticle.net/Article/A353137
- Ma, H. H. (2011). Research: Quantitative. In M. A. Runco & S. R. Pritzker (Eds.), Encyclopedia of creativity (2nd ed., pp. 304-312). Elsevier.
Cited by
Sorted by citations
Allison, J., Hwang, G., Mayer, R. E., Πέλλας, Ν., Karnalim, O., de Freitas, S., Ng, O., Huang, Y., Hooshyar, D., Seidman, R. H., et al. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251359964 · Cited by 71 · see citing works
“Moreover, the fi eld lacks rigorous studies on whether AI-generated materials meet the high pedagogical standards of humanities instruction, where nu-ance and interpretive depth are paramount (Graziosi et al., 2023; Pellas, 2025).”
Pellas, N. (2025) · Education Sciences · doi:10.3390/educsci15010102 · Cited by 56 · see citing works
Song, D., Zhang, P., Zhu, Y., Qi, S., Yang, Y., Gong, L., & Zhou, L. (2025) · Nurse Education in Practice · doi:10.1016/j.nepr.2025.104549 · Cited by 45 · see citing works
A Systematic Review of User Attitudes Toward GenAI: Influencing Factors and Industry Perspectives
Chen, J., Wei, X., Xie, Q., Hu, A., Qiao, Y., Wan, R., & Liu, Y. (2025) · Journal of Intelligence · doi:10.3390/jintelligence13070078 · Cited by 36 · see citing works
“…(Franceschelli and Musolesi 2024) Among them, higher-order thinking skills (HOTS) list creativity along with critical thinking and problem-solving abilities as important components of human intelligence, and study the interrelationship between their attitudes towards the use of GenAI (Pellas 2025).”
InfluentialBalart, T., Díaz, B., & Shryock, K. (2026) · Algorithms · doi:10.3390/a19030179 · Cited by 15 · see citing works
“Other tools such as Gemini, Ernie Bot, and multi-tool workflows were less common [61,70], and a few studies developed custom GPT-based systems ([50,59,84,89,90]).”
Chiu, J., & Lien, H. (2025) · Sustainability · doi:10.3390/su17125437 · Cited by 6 · see citing works
Cong‐Lem, N., & THUY-DUNG, N. T. (2026) · European Journal of Psychology of Education · doi:10.1007/s10212-026-01089-y · Cited by 5 · see citing works
“A notable trend within this category is the frequent adaptation of the Hwang et al. (2018) Higher-order Thinking Skills Scale, which was utilised by Lee et al. (2024), Lu et al. (2025), and Pellas (2024) to measure CT along-side problem-solving and creativity.”
Liu, X., Yang, J., Zhao, W., & Wang, T. (2026) · Behavioral Sciences · doi:10.3390/bs16040488 · Cited by 4 · see citing works
“Both problem-solving and critical thinking represent essential higher-order cognitive competencies in the digital age (Pellas, 2025).”
Πέλλας, Ν. (2025) · Multimodal Technologies and Interaction · doi:10.3390/mti9070065 · Cited by 5 · see citing works
Elearning adoption among millennials: behavioural intentions and the extended UTAUT model
Hasan, A., Raghuwanshi, S., Harsora, H., Kumar, P., Gupta, V., & Singh, A. S. (2025) · Discover Education · doi:10.1007/s44217-025-00813-y · Cited by 5 · see citing works
“Such peculiarities precondition the implementation of empirical studies necessitated by special localisation of global behavioural and technological adoption models, including UTAUT, to the Indian context of higher learning [21, 62, 82].”
Xia, L., Dong, Y. H., & Teo, W. (2026) · NeuroImage · doi:10.1016/j.neuroimage.2026.121796 · Cited by 2 · see citing works
Hani, I. B., Hujran, O., & Anwar, M. (2026) · Journal of Decision System · doi:10.1080/12460125.2026.2652347 · Cited by 2 · see citing works
Large Language Models in Graduate Information Science Education: Benefits and Challenges
Xiang, S., Deng, H., Liu, J., & Wu, J. (2025) · International Conference on Computer Science and Technologies in Education (CSTE) · doi:10.1109/cste64638.2025.11092151 · Cited by 1 · see citing works
“[17].”
How Critical Thinking Shapes Student Engagement with AI Tools to Foster Self-Directed Learning
Singhachotsukpat, R., Thiangtham, K., Wilepana, C., Yaemsawat, N., Sisaeng, P., Songkram, N., & Khlaisang, J. (2025) · International Conference on Educational Communications and Technology (ICTAECT) · doi:10.1109/ictaect67351.2025.11391283 · Cited by 3 · see citing works
“Pellas [13] positions CT as a moderating force, ensuring that AI-supported learning translates into reflective rather than passive adoption.”
Ibrahim, A. I., & Iqbal, S. (2025) · International Journal of Environmental Sciences · doi:10.64252/bgewk868
“Pellas (2024) further found that while creativity mediated improved performance, GAI did not significantly aid critical thinking or long-term retention.”
Jecha, J., Liang, Y., Han, J., & Nassor, H. (2026) · Research Square · doi:10.21203/rs.3.rs-9461812/v1 · Cited by 1 · see citing works
- Authors
- Pellas, N.
- Venue
- Research and Practice in Technology Enhanced Learning
- Published
- 2024-12-10
- Volume
- 20
- Pages
- 036
- ISSN
- 1793-2068 (ISSN-L)
- 1793-7078
- OpenAlex
- W4405175596
- Cited by
- 16
- References
- 0
- Citations by year
- 2026: 6 · 2025: 10
- FWCI
- 6.03
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Research and Practice in Technology Enhanced Learning (publishedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- higher-order thinking skills, academic performance, generative AI chatbots, machine learning, student attitudes toward AI, creativity, problem solving, critical thinking, mediating effect, moderating effect, undergraduate students, media and information literacy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
