The impact of AI in physics education: a comprehensive review from GCSE to university levels
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
With the rapid evolution of artificial intelligence (AI), its potential implications for higher education have become a focal point of interest. This study delves into the capabilities of AI in physics education and offers actionable AI policy recommendations. Using openAI’s flagship gpt-3.5-turbo large language model (LLM), we assessed its ability to answer 1337 physics exam questions spanning general certificate of secondary education (GCSE), A-Level, and introductory university curricula. We employed various AI prompting techniques: Zero Shot, in context learning, and confirmatory checking, which merges chain of thought reasoning with reflection. The proficiency of gpt-3.5-turbo varied across academic levels: it scored an average of 83.4% on GCSE, 63.8% on A-Level, and 37.4% on university-level questions, with an overall average of 59.9% using the most effective prompting technique. In a separate test, the LLM’s accuracy on 5000 mathematical operations was found to be 45.2%. When evaluated as a marking tool, the LLM’s concordance with human markers averaged at 50.8%, with notable inaccuracies in marking straightforward questions, like multiple-choice. Given these results, our recommendations underscore caution: while current LLMs can consistently perform well on physics questions at earlier educational stages, their efficacy diminishes with advanced content and complex calculations. LLM outputs often showcase novel methods not in the syllabus, excessive verbosity, and miscalculations in basic arithmetic. This suggests that at university, there’s no substantial threat from LLMs for non-invigilated physics questions. However, given the LLMs’ considerable proficiency in writing physics essays and coding abilities, non-invigilated examinations of these skills in physics are highly vulnerable to automated completion by LLMs. This vulnerability also extends to pysics questions pitched at lower academic levels. It is thus recommended that educators be transparent about LLM capabilities with their students, while emphasizing caution against overreliance on their output due to its tendency to sound plausible but be incorrect.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Baidoo-Anu, D., & Owusu Ansah, L. (2023) · doi:10.2139/ssrn.4337484 · Cited by 3,004 · cited 2×
“] Marks: [1] Solution: [C] Guidance: [] Answer: [C]”
BackgroundLarge language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.9 · Cited by 1,806
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
ChatGPT for Language Teaching and Learning
Kohnke, L., Moorhouse, B. L., & Zou, D. (2023) · RELC Journal · doi:10.1177/00336882231162868 · Cited by 1,371 · cited 1×
“A growing body of research suggests the importance of equipping students with skills and experience to interact effectively with AI [6, 11, 12].”
BackgroundExamining Science Education in ChatGPT: An Exploratory Study of Generative Artificial Intelligence
Cooper, G. (2023) · Journal of Science Education and Technology · doi:10.1007/s10956-023-10039-y · Cited by 1,263 · cited 1×
“Echoing the concerns raised by [30], the incorporation of AI into the educational realm warrants a balanced mix of skepticism and meticulous scrutiny.”
BackgroundReflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
Cited by
Krupp, L., Steinert, S., Kiefer-Emmanouilidis, M., Avila, K. E., Lukowicz, P., Kuhn, J., Küchemann, S., & Karolus, J. (2024) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/faia240195 · Cited by 82
Revell, T., Yeadon, W., Cahilly-Bretzin, G., Clarke, I., Manning, G., Jones, J., Mulley, C., Pascual, R. J., Bradley, N., Thomas, D., et al. (2024) · International Journal for Educational Integrity · doi:10.1007/s40979-024-00161-8 · Cited by 51
“While past technological innovations were adopted gradually within education (Scherer and Teo 2019), the emergence of ChatGPT suggests a significant shift in the realm of essay writing as a mode of assessment (Yeadon et al. 2024).”
BackgroundResultSelf-citationAI chatbots: A disguised enemy for academic integrity?
Niloy, A. C., Hafiz, R., Hossain, B. M., Gulmeher, F., Sultana, N., Islam, K. F., Bushra, F., Islam, S., Hoque, S. I., Rahman, M., et al. (2024) · International Journal of Educational Research Open · doi:10.1016/j.ijedro.2024.100396 · Cited by 26
Metadata
- Authors
- Yeadon, W., & Hardy, T.
- Venue
- Physics Education
- Publisher
- IOP Publishing
- Published
- 2024-02-06
- Volume
- 59
- Issue
- 2
- Pages
- 025010
- ISSN
- 0031-9120 (ISSN-L)
- 1361-6552
- OpenAlex
- W4391575364
- Cited by
- 78
- References
- 25
- FWCI
- 29.08
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The impact of AI in physics education: a comprehensive review from GCSE to university levels. https://app.cochat.ai/papers/10.1088/1361-6552/ad1fa2
- The paper
- Yeadon, W., & Hardy, T. (2024). The impact of AI in physics education: a comprehensive review from GCSE to university levels. Physics Education. https://doi.org/10.1088/1361-6552/ad1fa2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.