The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 12 of the 31 sources its claims cite were checked for support.
Exploring the Impact of AI Tools on Cognitive Skills: A Comparative Analysis
Open accessAbstract
This study evaluates the impact of Generative AI (Artificial Intelligence) algorithms on human decision making in complex problem-solving tasks. Rather than assessing the algorithms in isolation, we focus on how their use shapes three critical cognitive components of decision making: analytical thinking, creative thinking, and systems thinking. In an experimental setting, student participants were randomly assigned to solve management consulting cases either with or without access to an AI algorithm. Their solutions were evaluated using a structured rubric that captures sub-skills across the three cognitive dimensions. The results of this exploratory study reveal that AI-supported participants achieved stronger outcomes in logical reasoning, structuring, and problem definition, but showed weaknesses in novel idea generation, multidisciplinary integration, and critical rejection of unsupported conclusions. These findings highlight how algorithmic support influences the configuration of human cognitive processes in decision making.
Overview
CoChat Summary
TL;DR
This exploratory experiment with 16 master’s/doctoral students and recent graduates found that ChatGPT (GPT-4) support improved some analytical and systems-thinking sub-skills but did not uniformly improve creativity or critical evaluation. Benefits depended on collaborative use; copy-pasting or minimal engagement often yielded weak or inconsistent outcomes.
Key findings
- AI-supported participants achieved stronger outcomes in logical reasoning, structuring, and problem definition, but showed weaknesses in novel idea generation, multidisciplinary integration, and critical rejection of unsupported conclusions. In Task 1, GPT users scored higher in Logical Thinking (+29%), Decision Making (+13%), Alternative Approach (+25%), and Hypothesis Formulation (+13%), while lower in Argument Evaluation (−25%) and Questioning (−20%); in Task 2, Argument Evaluation was +25% for GPT users but Alternative Approach and Hypothesis Formulation were −21% and −50%.
- Systems thinking: Task 1 GPT users outperformed non-users in Holistically and Parts (+50%), How Concepts Work Together (+25%), and Identify Patterns Over Time (+21%); Task 2 showed lower/no differences for these, though Systems Boundaries Definition was +38% for GPT users. Creative thinking: Task 2 Problem and Goals Definition and Representation was approximately 60% higher for GPT users; See the Big Picture was +38% in Task 1 but −13% in Task 2; Novelty +13%, Appropriateness −13%; average creativity difference approximately 7%.
- The most frequent configuration was negative on all three skills (0-0-0), with five participants (two GPT users, three non-users). Collaborative GPT use often correlated with stronger sub-skills; copy-pasting tended lower, e.g., P_09 71% mean score vs P_01 26% and P_08 31%. GPT users finished 15–20 min earlier on average; post-survey rated AI usefulness 60–65%, with Creative Thinking slightly lower.
Methods
- Experimental design: 16 participants (master’s/doctoral students and recent graduates in business administration/IT at a Finnish university; average age 31.6; 8 female/8 male) randomly assigned to solve one of two management consulting cases with ChatGPT (GPT-4) or without AI, up to 1.5 h; screen recordings and pre/post surveys collected.
- Rubric: three cognitive skills—analytical thinking (11 sub-skills), creative thinking (8), systems thinking (10)—each sub-skill scored 1–3 by three co-authors, majority vote/consensus. Mixed-methods analysis: descriptive group-level statistics, individual sub-skill review, screen-recording behavioral analysis, notes, and post-experiment survey.
Limitations
- Stated: definitions and measurement criteria should be refined; some sub-skills (e.g., Analogical Thinking, Mental Modeling) may need different assessment methods; small sample limits generalizability and robustness of statistical comparisons.
- Evident: sample of 16 is exploratory and not statistically generalizable; results varied by task and were not consistent across both tasks; one GPT-user participant who opted not to use the tool was excluded from calculations; some sub-skill cells were empty when evaluators did not agree, and rubric scoring is subjective.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 1 Not valid · 14 Not checked · 143 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent studies and reports indicate the presence of a skill gap across different professions [1,2]”
§ Exploring the Impact of AI Tools on Cognitive Skills: A Comparative Analysis > 1. Introduction
What the sources say
- Partly supportedcites [1] Musazade, N. (2023, January 25–28). Tools and technologies utilized in data-related positions: An empirical study of job advertisements. Proceedings of the 36th Bled eConference—Digital Economy and Society: The Balancing Act for Digital Innovation in Times of Instabili, Bled, Slovenia. 1(BLED 2023—Proceedings):155-170. sourceExperimental — accuracy not yet measured
The Introduction (p. 3) says skill gaps create recruitment challenges, and the Conclusions (p. 13) refer to an existing skill gap. But the study examines a limited set of data-related professions and says assessing workers’ possessed skills is future research; it does not establish a skill gap across professions generally. (LLM, conf=0.82)
- Not checked — no full text of the sourcecites [2] Do, 2022, Hard Skill Gap between University Education and the Robotic Industry, J. Comput. Inf. Syst. source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As shown in a recent literature review [41,42], the number of participants in similar studies can range from five to several hundreds of participants.”
§ Exploring the Impact of AI Tools on Cognitive Skills: A Comparative Analysis > 3.3. Participants
What the sources say
- Partly supportedcites [41] Holzner, N., Maier, S., and Feuerriegel, S. (2025). Generative AI and Creativity: A Systematic Literature Review and Meta-Analysis. arXiv. sourceExperimental — accuracy not yet measured
The study table in the indexed copy (section 2.3 Statistical Analysis; page unavailable) lists participant counts including 10, 310, and 562, supporting the claim that similar studies can have several hundred participants. The copy does not establish the stated lower endpoint of five participants; searches for five-person samples and sample-size ranges did not find it. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [42] Umer, H. (2025, October 03). Generative Artificial Intelligence’s Impact on Education: A Selected Review of Experimental/Interventional Studies. 2025. Available online: http://dx.doi.org/10.2139/ssrn.5084325. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Experiments are particularly effective in isolating variables and testing specific interventions, such as the inclusion of AI tools [38].”
§ Exploring the Impact of AI Tools on Cognitive Skills: A Comparative Analysis > 3.2. Experimental Design and Tasks
What the sources say
- Could not verifycites [38] Orrù, G., Piarulli, A., Conversano, C., and Gemignani, A. (2023). Human-like problem-solving abilities in large language models using ChatGPT. Front. Artif. Intell., 6. sourceExperimental — accuracy not yet measured
The Introduction describes an evaluation of ChatGPT’s verbal problem-solving, and Materials and Methods describes the tasks administered to it. Neither passage addresses whether experiments are particularly effective at isolating variables or testing interventions such as adding AI tools. Searches for those methodological claims found no supporting passage in our copy. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In particular, Novelty and Appropriateness stood out as sub-skills that most benefit from a collaborative rather than purely Generative AI interaction-an observation consistent with Rick et al. [30], who found that human-AI co-creation fosters more creative solutions.”
§ Exploring the Impact of AI Tools on Cognitive Skills: A Comparative Analysis > 5.2. Performance in Creative Thinking
What the sources say
- Partly supportedcites [30] Rick, S.R., Giacomelli, G., Wen, H., Laubacher, R.J., Taubenslag, N., Heyman, J.L., Knicker, M.S., Jeddi, Y., Maier, H., and Dwyer, S. (2023). Supermind Ideator: Exploring generative AI to support creative problem-solving. arXiv. sourceExperimental — accuracy not yet measured
Rick et al.'s paper supports the collaborative/augmentation framing the citation leans on — §2.2 Generative AI argues that letting generative AI 'rapidly generate ideas for humans to consider ... can thus improve the probability that designers will find better solutions faster', and §5 EVALUATION records participants organically realizing 'the power of augmenting human abilities rather than replacing them'. But the paper does not report a finding that human-AI co-creation yields more creative solutions: §2.2 only says it 'seems plausible' and §7 Future Work states that quantitative evaluations of 'the speed and quality of the ideas they generate are currently underway', while §5 confirms the study was formative, collecting UX insights and professional-applicability measures rather than novelty/appropriateness or any creativity outcome. The unsupported part is therefore the empirical 'found ... more creative solutions' claim (and any implication that Novelty/Appropriateness sub-skills were measured there). (LLM, conf=0.72)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Consistent with prior research underscoring the importance of critical engagement with AI outputs [37], among participants who relied on simple copy-paste strategies, Participants 1 and 4 (i.e., P_01 and P_04 in Appendix B) encountered multiple challenges.”
§ Exploring the Impact of AI Tools on Cognitive Skills: A Comparative Analysis > 5.4. Main Observed Barriers to Problem Solving
What the sources say
- Supportedcites [37] Bellettini, C., Lodi, M., Lonati, V., Monga, M., and Morpurgo, A. (2023, January 21–23). Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3. Proceedings of the 15th International Conference on Computer Supported Education, Prague, Czech Republic. source
The citation is attached to the prior-research point about critically engaging with AI outputs, not to the citing authors’ observations about Participants 1 and 4. In Conclusions (p. 10), the source says DaVinci’s authoritative style can be deceptive and requires critical thinking to identify errors. (LLM, conf=0.94)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
93 more claims
Create a free account to see all 169 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 34
- It may state two separate claims as one. 44
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Exploring the Impact of AI Tools on Cognitive Skills: A Comparative Analysis > 3.2. Experimental Design and Tasks
“Each participant worked on a business consulting case adapted from Cosentino [40]”
§ Exploring the Impact of AI Tools on Cognitive Skills: A Comparative Analysis > References
“40. Orrù, G.; Piarulli, A.; Conversano, C.; Gemignani, A. Human-like problem-solving abilities in large language models using ChatGPT. Front. Artif. Intell. 2023, 6, 1199350. [CrossRef]”
Citations
Cites
Executive Perceptions of the Top 10 Soft Skills Needed in Today’s Workplace
Robles, M. M. (2012) · Business Communication Quarterly · doi:10.1177/1080569912460400 · Cited by 1,613 · cited 1×
“For example, as highlighted by Robles [12], business executives define a “positive attitude” as one of the top soft skills they expect employees to embody.”
Dell’Acqua, F., McFowland, E., Mollick, E., Lifshitz, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2026) · Organization Science · doi:10.1287/orsc.2025.21838 · Cited by 1,189
Dell'Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4573321 · Cited by 794 · cited 1×
“The problem-solving tasks were designed to resemble typical cases frequently encountered in the management consulting profession, a domain that has been a focus of some of the earliest studies assessing the impact of Generative AI [8].”
ChatGPT improves creative problem-solving performance in university students: An experimental study
Urban, M., Děchtěrenko, F., Lukavský, J., Hrabalová, V., Svacha, F., Brom, C., & Urban, K. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105031 · Cited by 396
Hitsuwari, J., Ueda, Y., Yun, W., & Nomura, M. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107502 · Cited by 267 · cited 1×
“The findings also align with [45], indicating the high potential of human–AI collaboration in creativity, while distinguishing different dimensions of creativity.”
Soft skills, do we know what we are talking about?
Zapata, S. I. M., Román-Calderón, J. P., Robledo‐Ardila, C., & Jaramillo-Serna, M. A. (2021) · Review of Managerial Science · doi:10.1007/s11846-021-00474-9 · Cited by 250 · cited 1×
“In this research, we rely on the contribution of Marin-Zapata et al. [10], who review the relevant literature on soft skill-related studies.”
“Soft Skills”: A Phrase in Search of Meaning
Matteson, M. L., Anderson, L., & Boyden, C. (2016) · portal: Libraries and the Academy · doi:10.1353/pla.2016.0009 · Cited by 240 · cited 5×
“Matteson et al. [11] discuss definitions of soft skills, distinguishing them from related terms and concepts.”
InfluentialHuman-like problem-solving abilities in large language models using ChatGPT
Orrù, G., Piarulli, A., Conversano, C., & Gemignani, A. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1199350 · Cited by 170 · cited 2×
“Experiments are particularly effective in isolating variables and testing specific interventions, such as the inclusion of AI tools [38].”
Generative AI and Creativity: A Systematic Literature Review and Meta-Analysis
Holzner, N., Maier, S., & Feuerriegel, S. (2025) · arXiv · doi:10.48550/arxiv.2505.17241 · Cited by 143
Zhaı, X., Nyaaba, M., & Ma, W. (2024) · Science & Education · doi:10.1007/s11191-024-00496-1 · Cited by 136 · cited 1×
“Zhai et al. [34] compared the performance of GPT on cognitively demanding scientific assignments with that of school students.”
Research works printed but not linked: 2
- Sternberg, R.J. (1994). Creativity. Thinking and Problem Solving, Elsevier. [2nd ed.].
- Cosentino, M. (2013). Case in Point: Complete Case Interview Preparation, Burgee Press.
Cited by
A MODEL FOR FORMATION COGNITIVE SKILLS FOR CYBERSECURITY SPECIALISTS
Shevchenko, S., Zhdanovа, Y., & Spasiteleva, S. (2025) · Cybersecurity Education Science Technique · doi:10.28925/2663-4023.2025.30.973
Cognitwin: A Neuro-Digital Synthesis for Predictive Decision Intelligence in Smart Ecosystems
Helen, F. R., Sudhakar, K., Kodamanchili, R., Bansal, M., Sanku, N. K., & Maguluri, L. P. (2026) · International Conference on Artificial Intelligence and Machine Learning Applications Theme: Healthcare and Internet of Things (AIMLA) · doi:10.1109/aimla67915.2026.11522513
“Conventional digital twin models tend to concentrate on data-driven replication only without the ability to think, evolve as well as make context-oriented decisions, particularly in dynamically changing environments where uncertainties, contradictory objectives, and moral implications exist [3] [4].”
Sultana, N., Haque, A., & Rupa, R. A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01469-9 · Cited by 2
“Moreover, there lacks studies that assess the impact of AI on cognitive and PS skills of students, which frames the major gap in the literature [39, 59].”
Apeadido, S., Owusu-Fordjour, C., & Mahama, A. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02113-2
“The literature suggests that AI chatbots serve as externalised cognitive tools capable of supporting complex, multi-step tasks through Socratic-style questioning sequences and guided prompts that facilitate analysis, synthesis, and evidence-based reasoning [59, 85, 90].”
Metadata
- Authors
- Musazade, N., Mezei, J., & Wang, X.
- Affiliations
- Åbo Akademi University (FI)
- Corresponding author
- Mezei, József
- Venue
- Algorithms
- Publisher
- MDPI AG
- Published
- 2025-10-07
- Volume
- 18
- Issue
- 10
- Pages
- 631
- ISSN
- 1999-4893 (ISSN-L)
- OpenAlex
- W4414917618
- Cited by
- 10
- References
- 38
- Citations by year
- 2026: 9 · 2025: 1
- FWCI
- 2.57
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Algorithms (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Åbo Akademi University Research Portal (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Social Sciences › Education › Artificial Intelligence in Education
- Social Sciences › Decision Sciences › Management Science and Operations Research › Complex Systems and Decision Making
- Keywords
- generative AI, human-AI collaboration, decision-making, cognitive skills, analytical thinking, creative thinking, systems thinking, problem solving, logical reasoning, idea generation, multidisciplinary integration, critical evaluation of AI outputs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Exploring the Impact of AI Tools on Cognitive Skills: A Comparative Analysis. https://app.cochat.ai/papers/10.3390/a18100631
- The paper
- Musazade, N., Mezei, J., & Wang, X. (2025). Exploring the Impact of AI Tools on Cognitive Skills: A Comparative Analysis. Algorithms. https://doi.org/10.3390/a18100631
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.