Facilitator or hindrance? The impact of AI on university students' higher-order thinking skills in complex problem solving
Open accessThe DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index; 6 of the 15 sources its claims cite were checked for support.
Abstract
Artificial Intelligence (AI) is increasingly permeating education, enhancing teaching efficiency and enriching personalized learning experiences. However, integrating AI without compromising students' independent thinking remains a challenge. Using ChatGPT-4o as an example, this study employed a within-subject experimental design to examine AI’s impact on university students' higher-order thinking skills (HOTS) and task outcomes in complex problem-solving. Based on the design problem context, the research constructed diverse task modules encompassing complex problem-solving processes, collected multidimensional data from 40 students, and analyzed using epistemic network analysis and consensus assessment techniques. The results indicate that AI slightly enhances the originality and usefulness of design sketches and significantly increases the number of HOTS used and the diversity of their interconnections. The study further categorizes students by their degree of AI influence, revealing that those more influenced by AI incorporate more personal thinking, use more HOTS, and exhibit richer interconnections among these skills. This research provides empirical evidence on AI's role in fostering HOTS and provides guidance on the integration of AI to enhance students' independent thinking in education.
Overview
CoChat Summary
TL;DR
In a within-subject study with 40 design students using ChatGPT-4o for complex design problem solving, AI slightly improved sketch originality and usefulness but significantly increased the number and interconnectedness of higher-order thinking skills (HOTS). Students more influenced by AI showed more HOTS and more complex HOTS networks, though direct adoption of AI ideas was common.
Key findings
- Sketch scores improved significantly but modestly from Re-sketch to Fi-sketch: originality increased from M = 2.80 (SD = 0.91) to M = 3.38 (SD = 0.75), Cohen’s d = 0.793, t = −5.018, p < 0.001; usefulness increased from M = 3.05 (SD = 0.88) to M = 3.78 (SD = 0.79), Cohen’s d = 1.119, t = −7.079, p < 0.001.
- Participants generated 310 components total; Fi-sketch components increased by 110.89% over Re-sketches. Among newly added components, 61.35% were direct adoption of AI ideas, 22.09% adaptation, 9.82% recreation, and 6.75% iteration.
- Participants exhibited 3329 HOTS; critical thinking was used most and creative thinking least, with “evaluation” highest and “analogy” lowest. Of 40 paired HOTS comparisons, 18 were significant (p < 0.05), and usage was higher after AI in all significant instances except problem-solving thinking between phase2 and phase5.
- High-impact participants (top 10 by z-scored sketch-score change) used adaptation relationships more than low-impact participants (M = 1.80, SD = 1.32 vs. M = 0.60, SD = 0.52; d = 1.200, t = −2.683, p = 0.020) and showed significantly more critical thinking, creative thinking, and sub-HOTS A3, A6, B2, B4, and C2.
Methods
- Within-subject experimental design with 40 design freshmen/sophomores (28 females, 12 males; M age = 19.28, SD = 0.67) from visual communication design, digital media art design, and architectural design. Five phases involved a Pr-sketch, Re-sketch, ChatGPT-4o interaction, and Fi-sketch; participants used think-aloud throughout, with screen recording and an average task time of 33.80 min (SD = 6.05).
- Sketch data were rated by three blind design teachers using Consensus Assessment Technique on 7-point originality and usefulness scales; ICC was 0.75 for originality and 0.76 for usefulness. Hairdryer components were coded as Airflow Barrel, Handle, and Power, and Re-sketch-to-Fi-sketch component relationships were classified as direct adoption, adaptation, iteration, recreation, or reduction.
- Think-aloud data (22.53 h audio) were coded with a HOTS scheme covering critical, creative, and problem-solving thinking and 17 sub-HOTS, achieving Kappa = 0.71. Analyses used paired t-tests, Mann-Whitney U tests, and Epistemic Network Analysis (ENA).
Limitations
- The paper states that it focused only on students solving complex problems independently or with AI, without considering educators’ role in natural educational settings; future research should explore AI use within teaching environments.
- The paper states that it examined only design problems as complex problems, limiting generalizability, and that the sample was small and relatively homogeneous; future work should involve larger-scale studies and varied readiness, cultural backgrounds, or AI familiarity.
- The paper also notes that AI models are continuously updated, so generation logic and presentation may change. Evident from the text: the study used a single task object (hairdryer), a novice sample, and ChatGPT-4o only, and the fixed within-subject phase order is not described as counterbalanced, so practice or order effects are not ruled out.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
6 Valid · 0 Not valid · 8 Not checked · 119 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, the immediacy of AI-generated concepts may compress the design concept generation phase, which is crucial for breakthrough innovation in design processes (Wadinambiarachchi et al., 2024).”
What the sources say
- Partly supportedcites Wadinambiarachchi et al., 2024 sourceExperimental — accuracy not yet measured
The Introduction says AI tools can produce visuals quickly, and Discussion §5.3 reports that, within a fixed-time task, using an AI image generator left less time for sketching ideas. The paper does not establish that immediacy compresses the design concept generation phase or assess that phase’s role in breakthrough innovation. Page numbers are unavailable in our copy. (LLM, conf=0.83)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results showed a significant difference in the adaptation relationship (Cohen's d = 1.200, t = -2.683, p = 0.020), with high-impact participants (M = 1.80, SD = 1.32) using this relationship more frequently than low-impact participants (M = 0.60, SD = 0.52).”
plainstates an associationn = 10 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From phase1 to phase3, evaluation shifted from the focal point to a node, forming a more complex cognitive network.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In phase1, evaluation was closely linked to inference, analysis, and explanation, with critical thinking as the core skill.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In all instances, the usage frequency of HOTS was higher after using AI (phase3 and phase5) than before (phase1 and phase2).”
- Also stated in: § Discussion · p. 16 · plain
“Additionally, the number of HOTS used increased significantly, stimulating connections between different categories.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Discussion · p. 16 · plain
71 more claims
Create a free account to see all 133 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 36
- Probably not a substantial claim. 27
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Appendix · p. 23
“| | High-impact participants (n = 10) | High-impact participants (n = 10) | Lower | Upper | | | |”
§ Appendix · p. 23
“Table 6 T-test results of HOTS frequency between high-impact and low-impact participants”
Citations
Cites
New Era of Artificial Intelligence in Education: Towards a Sustainable Multifaceted Revolution
Kamalov, F., Santandreu Calonge, D., & Gurrib, I. (2023) · Sustainability · doi:10.3390/su151612451 · Cited by 1,089 · cited 1×
“The deepening application of AI in education has significantly altered students’ thinking patterns and cognitive processes (Kamalov et al., 2023).”
Journal for Educational Research Online
(2021) · Journal for Educational Research Online / Journal für Bildungsforschung Online · doi:10.31244/jero · Cited by 887
How to Assess Higher-Order Thinking Skills in Your Classroom
Brookhart, S. M. (2010) · Cited by 844
Strzelecki, A. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2023.2209881 · Cited by 807 · cited 1×
“Although AI is widely adopted, its dynamics and role in higher education are still poorly understood, with limited empirical studies on the factors driving its use among university students (Strzelecki, 2023).”
Purposely Teaching for the Promotion of Higher-order Thinking Skills: A Case of Critical Thinking
Miri, B., David, B.-C., & Uri, Z. (2007) · Research in Science Education · doi:10.1007/s11165-006-9029-2 · Cited by 667 · cited 2×
“Researchers emphasize the importance of critical and creative thinking within HOT, particularly the central role of critical thinking (Miri et al., 2007).”
Higher Order Thinking Skills and Low-Achieving Students: Are They Mutually Exclusive?
Zohar, A., & Dori, Y. J. (2003) · Journal of the Learning Sciences · doi:10.1207/s15327809jls1202_1 · Cited by 631
Artificial intelligence and the changing sources of competitive advantage
Krakowski, S., Luger, J., & Raisch, S. (2023) · Strategic Management Journal · doi:10.1002/smj.3387 · Cited by 594 · cited 1×
“It enables artificial agents to perform cognitive functions like decision-making and problem-solving, tasks once exclusive to humans (Krakowski et al., 2023).”
Design Problem Solving: A Task Analysis
Chandrasekaran, B. (1990) · The AI Magazine · doi:10.1609/aimag.v11i4.857 · Cited by 568
Humans and Technology: Forms of Conjoined Agency in Organizations
Murray, A., Rhymer, J., & Sirmon, D. G. (2021) · Academy of Management Review · doi:10.5465/amr.2019.0186 · Cited by 518 · cited 1×
“Free from human cognitive limitations, these agents often outperform humans in information retrieval, decision-making, and prediction (Murray et al., 2021).”
An approach to the analysis of design protocols
Gero, J. S., & Mc Neill, T. (1998) · Design Studies · doi:10.1016/s0142-694x(97)00015-x · Cited by 512 · cited 1×
“This grading method ensured that the final data remained objective and independent (Gero & Neill, 1998).”
Research works printed but not linked: 2
- Ahmad, K., Qadir, J., Al-Fuqaha, A., Iqbal, W., El-Hassan, A., Benhaddou, D., & Ayyash, M. (2020). Artificial intelligence in education: A panoramic review. https://doi.org/10.35542/osf. io/zvu2n. · doi:10.35542/osf
- Yulianti, P. D., Yandhini, S., Sari, A. D. P., Herawani, I., & Oktarini, I. (2024). The influence of AI on students’ mind patterns. BICC Proceedings, 2, 183–186. https://doi.org/10.3983/bicc.v1i1.125 · doi:10.3983/bicc.v1i1.125
Cited by
Zhao, Y., Yue, Y., Sun, Z., Jiang, Q., & Li, G. (2025) · Journal of Intelligence · doi:10.3390/jintelligence13120160 · Cited by 37
Adaptive and Personalized Learning in Higher Education: An Artificial Intelligence-Based Approach
Hernández-Herrera, J. R., Ortiz-Bejar, J., & Ortiz-Bejar, J. (2026) · Education Sciences · doi:10.3390/educsci16010109 · Cited by 30
ChatGPT in education: A systematic review of its impact on critical thinking skills and dispositions
Guo, Y., Huang, L., Zhang, C., Li, Q., & Chen, M. (2025) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2025.102106 · Cited by 26
Whitty, M. T. (2025) · Electronics · doi:10.3390/electronics14204104 · Cited by 6
Chea, P., & Deng, F. (2026) · Innovative Higher Education · doi:10.1007/s10755-026-09886-5 · Cited by 6
Metadata
- Authors
- Du, X., Du, M., Zhou, Z., & Bai, Y.
- Affiliations
- Hunan Normal University (CN)
- Sichuan University (CN)
- Hunan University (CN)
- Corresponding author
- Du, Xing
- ORCID
- Du, Xing
- Venue
- International Journal of Educational Technology in Higher Education
- Published
- 2025-07-04
- Volume
- 22
- Issue
- 1
- ISSN
- 2365-9440 (ISSN-L)
- OpenAlex
- W4411986533
- Cited by
- 44
- References
- 65
- Citations by year
- 2026: 38 · 2025: 6
- FWCI
- 45.39
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- International Journal of Educational Technology in Higher Education (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Education Department of Hunan Province (202401000055)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
- Social Sciences › Education › Problem and Project Based Learning
- Keywords
- artificial intelligence in education, higher-order thinking skills, complex problem solving, GPT-4o, university students, independent thinking, design sketches, epistemic network analysis, personalized learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Facilitator or hindrance? The impact of AI on university students' higher-order thinking skills in complex problem solving. https://app.cochat.ai/papers/10.1186/s41239-025-00534-0
- The paper
- Du, X., Du, M., Zhou, Z., & Bai, Y. (2025). Facilitator or hindrance? The impact of AI on university students' higher-order thinking skills in complex problem solving. International Journal of Educational Technology in Higher Education. https://doi.org/10.1186/s41239-025-00534-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.