CoChat
Sign Up
Cited by 28Artificial Intelligence in EducationScience Education and PedagogyEducation and Technology Integration

Can Students Make STEM Progress With the Large Language Models (LLMs)? An Empirical Study of LLMs Integration Within Middle School Science and Engineering Practice

Guo, Qing, Zhen, Junwen, Wu, Fenglin, He, Yanting, Qiao, Cuilan·2025·Journal of Educational Computing Research
DOI 10.1177/07356331241312365
Open this paper in CoChat — free account

Abstract

The rapid development of large language models (LLMs) presented opportunities for the transformation of science and STEM education. Research on LLMs was in the exploratory phase, characterized by discussions and observations rather than empirical investigations. This study presented a framework for incorporating LLMs into Science and Engineering Practice (SEP), utilizing a case study on submarine construction, followed by a four-week quasi-experimental validation. The research employed conditional cluster sampling, selecting two homogeneous natural classes from a middle school in China to serve as the experimental and control groups. The key experimental variable was the inclusion of LLMs in the SEP project. Various validated and self-developed assessment tools were used to measure students’ STEM learning outcomes. Statistical analyses, including pre- and post-test paired comparisons within classes and ANCOVA for between-class differences, were performed to evaluate the effects of LLM integration. The results showed that students participating in SEP integrated with LLMs significantly improved their mastery of scientific knowledge, attitudes towards science, perceived usefulness of technology, understanding of engineering, computational thinking skills, and problem-solving abilities. In contrast, students participating in traditional SEP exhibited weaker knowledge acquisition, differences in understanding engineering concepts, and lack of development in computational thinking and problem-solving skills. This study was a pioneering effort in integrating LLMs into science education and provided a framework and case reference for the deeper application of LLMs in the future.

Integrity signals

Registry

Crossref · 10.1177/07356331241312365

Cites

57 of 57 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Evidence spans STEM and the humanities, including gains in understanding physics and programming, as well as improved reading and poetry interpretation.

    “Evidence spans STEM and the humanities, including gains in understanding physics [91,101,104,147] and programming [95,148,150], as well as improved reading and poetry interpretation [72,75,81,116].”

    From A Systematic Review of Generative AI in K–12: Mapping Goals, Activities, Roles, and Outcomes via the 3P Model (2025) · cites this paper as [147]

  • AI provides strong scaffolds for core practices, markedly improving the quality and conventionality of learner outputs-for example, enhancing grammatical accuracy and argumentative coherence in writing, and supporting problem-solving and experimental design in science and mathematics-thereby helping learners meet baseline practice demands more efficiently.

    “First, AI provides strong scaffolds for core practices, markedly improving the quality and conventionality of learner outputs-for example, enhancing grammatical accuracy and argumentative coherence in writing [86,87,145], and supporting problem-solving and experimental design in science and mathematics [77,88,93,103,147]-thereby helping learners meet baseline practice demands more efficiently.”

    From A Systematic Review of Generative AI in K–12: Mapping Goals, Activities, Roles, and Outcomes via the 3P Model (2025) · cites this paper as [147]

  • Roles are reshaped: learners move from passive consumers of knowledge to active creators of learning experiences, strengthening autonomy and agency.

    “Roles are also reshaped: learners move from passive consumers of knowledge to active creators of learning experiences [116], strengthening autonomy and agency [85,147].”

    From A Systematic Review of Generative AI in K–12: Mapping Goals, Activities, Roles, and Outcomes via the 3P Model (2025) · cites this paper as [147]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Guo, Q., Zhen, J., Wu, F., He, Y., & Qiao, C.
Affiliations
Central China Normal University (CN)
Corresponding author
Qiao, Cuilan
ORCID
Venue
Journal of Educational Computing Research
Publisher
SAGE Publications
Published
2025-01-06
Volume
63
Issue
2
Pages
372-405
ISSN
  • 0735-6331 (ISSN-L)
  • 1541-4140
DOI
10.1177/07356331241312365
OpenAlex
W4406105230
Cited by
28
References
55
Citations by year
2026: 19 · 2025: 9
FWCI
28.89
Type
journal article
Language
en
Open access
closed
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
Journal of Educational Computing Research (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Social Sciences › Education › Science Education and Pedagogy
  • Social Sciences › Education › Education and Technology Integration
Keywords
large language models, junior high school science, STEM education, computational thinking, problem-solving skills, attitudes toward science

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review