CoChat

From Offloading to Engagement: An Experimental Study on Structured Prompting and Critical Reasoning with Generative AI

Open access
Gerlich, Michael · 2025 · Data

The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index; 8 of the 46 sources its claims cite were checked for support.

Abstract

The rapid adoption of generative AI raises questions not only about its transformative potential but also about its cognitive and societal risks. This study contributes to the debate by presenting cross-country experimental data (n = 150; Germany, Switzerland, United Kingdom) on how individuals engage with generative AI under different conditions: human-only, human + AI (unguided), human + AI (guided with structured prompting), and AI-only benchmarks. Across 450 evaluated responses, critical reasoning was assessed via expert rubric ratings, while perceived reflective engagement was captured through self-report indices. Results show that unguided AI use fosters cognitive offloading without improving reasoning quality, whereas structured prompting significantly reduces offloading and enhances both critical reasoning and reflective engagement. Mediation and latent class analyses reveal that guided AI use supports deeper human involvement and mitigates demographic disparities in performance. Beyond theoretical contributions, this study offers practical implications for business and society. As organisations integrate AI into workflows, unstructured use risks undermining workforce decision making and critical engagement. Structured prompting, by contrast, provides a scalable and low-cost governance tool that fosters responsible adoption, supports equitable access to technological benefits, and aligns with societal calls for human-centric AI. These findings highlight the dual nature of generative AI as an amplifier of human capability when scaffolded and an impediment to critical thinking when adopted passively.

Overview

CoChat Summary

TL;DR

In a cross-country experiment (n = 150; Germany, Switzerland, United Kingdom), structured prompting for ChatGPT use improved critical reasoning and reflective engagement, while unguided AI use fostered cognitive offloading without improving reasoning quality. Guided AI use significantly outperformed all other conditions, and mediation/latent class analyses linked lower offloading and higher reflective engagement to better performance.

Key findings

  • One-way ANOVA showed significant differences in total argument quality across the four conditions, F(3, 447) = 111.75, p < 0.0001, η² = 0.43. Tukey comparisons showed guided Human + AI significantly outperformed all other groups; guided vs. Human Alone mean difference = 7.80, p < 0.0001; unguided Human + AI vs. Human Alone = 2.56, p < 0.0001; AI Alone vs. Human Alone = 4.98, p = 0.5508 (not significant). Unguided Human + AI did not significantly outperform AI Alone.
  • Cognitive offloading was negatively correlated with perceived reflective engagement (r = −0.6602, p < 0.001), education (r = −0.6165, p < 0.001), and performance (r = −0.3166, p < 0.001). Perceived reflective engagement was positively correlated with performance (r = 0.3937, p < 0.001). Offloading and perceived difficulty were not significantly correlated (r = 0.0078, p = 0.87).
  • Mediation analysis found guided prompting increased perceived reflective engagement (β = 0.84, p < 0.001), which improved performance (β = 1.12, p < 0.001), with partial mediation. Sequential mediation found education predicted lower offloading (β = −0.73, p < 0.001), lower offloading predicted higher reflective engagement (β = −0.61, p < 0.001), and reflective engagement predicted performance (β = 1.12, p < 0.001).
  • Latent class analysis identified two classes: Class 0 “AI-dependent thinkers” (n = 77; offloading = 4.47, reflective engagement = 2.87) and Class 1 “reflective thinkers” (n = 73; offloading = 3.83, reflective engagement = 4.49). Education had a significant multivariate effect on post-task cognition (Wilks’ Lambda = 0.1626, F = 144.34, p < 0.0001), while country did not (p = 0.076).

Methods

  • Controlled multi-condition experiment with n = 150 participants (50 each in Germany, Switzerland, and the United Kingdom), recruited from academic institutions, professional workshops, and AI-focused conferences. Four conditions: human-only, AI-only, human + AI unguided, and human + AI guided. The three human-involved conditions were within-subjects and counterbalanced; AI-only was a fixed ChatGPT benchmark. The task was “What are the advantages and disadvantages of democracy?”
  • Measures included expert rubric ratings of critical thinking across five dimensions (6-point scale, total 6–30; Krippendorff’s α ≥ 0.70), a post-task questionnaire on cognitive offloading, perceived reflective engagement, and perceived difficulty (6-point Likert), and 30 semi-structured interviews (10 per country).
  • Analysis used ANOVA/Tukey, MANOVA, correlations, mediation and sequential mediation, latent class analysis, multiple regression, and thematic analysis, conducted in R 4.3.1 and Python 3.11, with bootstrapping (5000 samples) for robustness.

Limitations

  • Stated: participants were not randomly sampled from the general population; recruitment through universities, workshops, and AI conferences may introduce self-selection bias toward more technologically engaged or reflective individuals, limiting generalisability. The accessible democracy prompt may not replicate high-stakes, domain-specific tasks. Scoring involved subjective judgement despite acceptable inter-rater reliability, and self-report measures of offloading and reflective engagement may have limited introspective access and were not a pre-validated psychometric scale. Demographic subgroup analyses were exploratory due to reduced cell sizes, and the study focused on short-term effects only.
  • Stated: future research is needed across domains such as healthcare, finance, or law, and longitudinally to assess cumulative skill degradation or cognitive resilience.
  • Evident from the text: the AI-only condition was a fixed benchmark generated separately rather than a randomized participant condition, limiting direct causal comparison. Some reported statistics appear inconsistent—e.g., the text states perceived reflective engagement had β = 1.12, p < 0.0001 in the regression, while Table 9 lists 0.3163, p = 0.6679; the text also says age, offloading, and perceived difficulty did not remain significant, while Table 9 and Section 3.4.2 report age and perceived difficulty as significant.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/data10110172

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

8 Valid · 0 Not valid · 28 Not checked · 212 Own results and interpretations

133 more claims

Create a free account to see all 248 claims and to verify the 4 references still open.

See all claims — free account
  • Probably not a substantial claim. 52
  • It may state two separate claims as one. 78
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 9 · the rest aren’t loaded

4 more findings

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Show all 14 references

Research works printed but not linked: 11

  1. Baasch, 2023, Who owns the thought? Epistemic responsibility in human–AI collaboration, Ethics Inf. Technol., 25, 39
  2. Rahimi, 2022, Automation and metacognition: Rethinking learner agency in AI-supported learning environments, Br. J. Educ. Technol., 53, 1124
  3. Schmid, 2023, Predictive text and learning performance: When writing feels easier, do we think less?, Comput. Educ., 191, 104649
  4. Liu, 2024, Cognitive Load in AI-Supported Learning: Rethinking Scaffolding through Prompt Design, Learn. Instr., 85, 101782
  5. Schluter, 2021, Prompt Literacy and Human Autonomy: Rethinking Competence in the Age of Language Models, AI Ethics, 2, 601
  6. Facione, P.A. (1990). Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction (The Delphi Report), The California Academic Press for the American Philosophical Association.
  7. Halpern, D.F. (2013). Thought and Knowledge: An Introduction to Critical Thinking, Psychology Press. [5th ed.].
  8. Mata, 2012, Cognitive aging and decision making, Trends Cogn. Sci., 16, 261
  9. Yeo, 2023, The Persuasive Power of AI: Perceived Credibility, Bias Blindness, and Political Reasoning, AI Soc., 38, 433
  10. Dwivedi, 2023, Generative AI in practice: How knowledge workers use ChatGPT, Int. J. Inf. Manag., 71, 102653
  11. Leufer, 2022, Trustworthy AI in the Workplace: Between Functional Assistance and Cognitive Delegation, AI Ethics, 2, 439

Cited by

Metadata

Authors
Gerlich, M.
Affiliations
SBS Swiss Business School (CH)
Corresponding author
Gerlich, Michael
Venue
Data
Publisher
MDPI AG
Published
2025-10-30
Volume
10
Issue
11
Pages
172
ISSN
2306-5729 (ISSN-L)
OpenAlex
W4415720048
Cited by
40
References
7
Citations by year
2026: 40
FWCI
41.61
Type
journal article
Language
en
Open access
Locations
  • Data (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
generative AI, structured prompting, cognitive offloading, critical reasoning, reflective engagement, human-AI collaboration, AI governance, human-centered AI, enterprise AI adoption, cross-national comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.