CoChat

Facilitator or hindrance? The impact of AI on university students' higher-order thinking skills in complex problem solving

Open access
Du, Xing, Du, Mingcheng, Zhou, Zihan, Bai, Yiming · 2025 · International Journal of Educational Technology in Higher Education

The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index; 6 of the 15 sources its claims cite were checked for support.

Abstract

Artificial Intelligence (AI) is increasingly permeating education, enhancing teaching efficiency and enriching personalized learning experiences. However, integrating AI without compromising students' independent thinking remains a challenge. Using ChatGPT-4o as an example, this study employed a within-subject experimental design to examine AI’s impact on university students' higher-order thinking skills (HOTS) and task outcomes in complex problem-solving. Based on the design problem context, the research constructed diverse task modules encompassing complex problem-solving processes, collected multidimensional data from 40 students, and analyzed using epistemic network analysis and consensus assessment techniques. The results indicate that AI slightly enhances the originality and usefulness of design sketches and significantly increases the number of HOTS used and the diversity of their interconnections. The study further categorizes students by their degree of AI influence, revealing that those more influenced by AI incorporate more personal thinking, use more HOTS, and exhibit richer interconnections among these skills. This research provides empirical evidence on AI's role in fostering HOTS and provides guidance on the integration of AI to enhance students' independent thinking in education.

Overview

CoChat Summary

TL;DR

In a within-subject study with 40 design students using ChatGPT-4o for complex design problem solving, AI slightly improved sketch originality and usefulness but significantly increased the number and interconnectedness of higher-order thinking skills (HOTS). Students more influenced by AI showed more HOTS and more complex HOTS networks, though direct adoption of AI ideas was common.

Key findings

  • Sketch scores improved significantly but modestly from Re-sketch to Fi-sketch: originality increased from M = 2.80 (SD = 0.91) to M = 3.38 (SD = 0.75), Cohen’s d = 0.793, t = −5.018, p < 0.001; usefulness increased from M = 3.05 (SD = 0.88) to M = 3.78 (SD = 0.79), Cohen’s d = 1.119, t = −7.079, p < 0.001.
  • Participants generated 310 components total; Fi-sketch components increased by 110.89% over Re-sketches. Among newly added components, 61.35% were direct adoption of AI ideas, 22.09% adaptation, 9.82% recreation, and 6.75% iteration.
  • Participants exhibited 3329 HOTS; critical thinking was used most and creative thinking least, with “evaluation” highest and “analogy” lowest. Of 40 paired HOTS comparisons, 18 were significant (p < 0.05), and usage was higher after AI in all significant instances except problem-solving thinking between phase2 and phase5.
  • High-impact participants (top 10 by z-scored sketch-score change) used adaptation relationships more than low-impact participants (M = 1.80, SD = 1.32 vs. M = 0.60, SD = 0.52; d = 1.200, t = −2.683, p = 0.020) and showed significantly more critical thinking, creative thinking, and sub-HOTS A3, A6, B2, B4, and C2.

Methods

  • Within-subject experimental design with 40 design freshmen/sophomores (28 females, 12 males; M age = 19.28, SD = 0.67) from visual communication design, digital media art design, and architectural design. Five phases involved a Pr-sketch, Re-sketch, ChatGPT-4o interaction, and Fi-sketch; participants used think-aloud throughout, with screen recording and an average task time of 33.80 min (SD = 6.05).
  • Sketch data were rated by three blind design teachers using Consensus Assessment Technique on 7-point originality and usefulness scales; ICC was 0.75 for originality and 0.76 for usefulness. Hairdryer components were coded as Airflow Barrel, Handle, and Power, and Re-sketch-to-Fi-sketch component relationships were classified as direct adoption, adaptation, iteration, recreation, or reduction.
  • Think-aloud data (22.53 h audio) were coded with a HOTS scheme covering critical, creative, and problem-solving thinking and 17 sub-HOTS, achieving Kappa = 0.71. Analyses used paired t-tests, Mann-Whitney U tests, and Epistemic Network Analysis (ENA).

Limitations

  • The paper states that it focused only on students solving complex problems independently or with AI, without considering educators’ role in natural educational settings; future research should explore AI use within teaching environments.
  • The paper states that it examined only design problems as complex problems, limiting generalizability, and that the sample was small and relatively homogeneous; future work should involve larger-scale studies and varied readiness, cultural backgrounds, or AI familiarity.
  • The paper also notes that AI models are continuously updated, so generation logic and presentation may change. Evident from the text: the study used a single task object (hairdryer), a novice sample, and ChatGPT-4o only, and the fixed within-subject phase order is not described as counterbalanced, so practice or order effects are not ruled out.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s41239-025-00534-0

Cites

73 of 73 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
From the PDF
Fig. 2 Research questions and data analysis
Fig. 2 Research questions and data analysisFrom the PDF
Image 3
From the PDF
Image 4
From the PDF
Fig. 5 HOTS usage frequency
Fig. 5 HOTS usage frequencyFrom the PDF
Fig. 6 HOTS paths of each phase
Fig. 6 HOTS paths of each phaseFrom the PDF
Fig. 7 ENA of phase1and phase3
Fig. 7 ENA of phase1and phase3From the PDF
Fig. 8 ENA of phase2 and phase5
Fig. 8 ENA of phase2 and phase5From the PDF
Fig. 9 Component relationships between Re-sketches and Fi-sketches for high-impact and low-impact participants
Fig. 9 Component relationships between Re-sketches and Fi-sketches for high-impact and low-impact participantsFrom the PDF
Fig. 10 ENA of high-impact participants and low-impact participants
Fig. 10 ENA of high-impact participants and low-impact participantsFrom the PDF

Verification

Claims the paper makes

6 Valid · 0 Not valid · 8 Not checked · 119 Own results and interpretations

71 more claims

Create a free account to see all 133 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 36
  • Probably not a substantial claim. 27
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 73 references

Research works printed but not linked: 2

  1. Ahmad, K., Qadir, J., Al-Fuqaha, A., Iqbal, W., El-Hassan, A., Benhaddou, D., & Ayyash, M. (2020). Artificial intelligence in education: A panoramic review. https://doi.org/10.35542/osf. io/zvu2n. · doi:10.35542/osf
  2. Yulianti, P. D., Yandhini, S., Sari, A. D. P., Herawani, I., & Oktarini, I. (2024). The influence of AI on students’ mind patterns. BICC Proceedings, 2, 183–186. https://doi.org/10.3983/bicc.v1i1.125 · doi:10.3983/bicc.v1i1.125

Metadata

Authors
Du, X., Du, M., Zhou, Z., & Bai, Y.
Affiliations
  • Hunan Normal University (CN)
  • Sichuan University (CN)
  • Hunan University (CN)
Corresponding author
Du, Xing
Venue
International Journal of Educational Technology in Higher Education
Published
2025-07-04
Volume
22
Issue
1
ISSN
2365-9440 (ISSN-L)
OpenAlex
W4411986533
Cited by
44
References
65
Citations by year
2026: 38 · 2025: 6
FWCI
45.39
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Educational Technology in Higher Education (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Education Department of Hunan Province (202401000055)
Topics
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
  • Social Sciences › Education › Problem and Project Based Learning
Keywords
artificial intelligence in education, higher-order thinking skills, complex problem solving, GPT-4o, university students, independent thinking, design sketches, epistemic network analysis, personalized learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Facilitator or hindrance? The impact of AI on university students' higher-order thinking skills in complex problem solving. https://app.cochat.ai/papers/10.1186/s41239-025-00534-0
The paper
Du, X., Du, M., Zhou, Z., & Bai, Y. (2025). Facilitator or hindrance? The impact of AI on university students' higher-order thinking skills in complex problem solving. International Journal of Educational Technology in Higher Education. https://doi.org/10.1186/s41239-025-00534-0

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.