The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
Advances in apparent conceptual physics reasoning in GPT-4
Open accessAbstract
ChatGPT is built on a large language model trained on an enormous corpus of human text to emulate human conversation. Despite lacking any explicit programming regarding the laws of physics, recent work has demonstrated that GPT-3.5 could pass an introductory physics course at some nominal level and register something close to a minimal understanding of Newtonian Mechanics on the Force Concept Inventory. This work replicates those results and also demonstrates that the latest version, GPT-4, has reached a much higher mark in the latter context. Indeed, its responses come quite close to perfectly demonstrating expert-level competence, with a few very notable exceptions and limitations. We briefly comment on the implications of this for the future of physics education and pedagogy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, LLMs have demonstrated high proficiency in solving these types of problems (OpenAI et al, 2024; West, 2023)”
From Scaffold or Crutch? Examining College Students' Use and Views of Generative AI Tools for STEM Education (2025) · cites this paper as West, 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Research works printed but not linked: 25
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
- Jonathan H Choi, Kristin E Hickman, Amy Monahan, and Daniel Schwarcz. Chatgpt goes to law school. Avail- able at SSRN, 2023.
- Michael Bommarito II and Daniel Martin Katz. Gpt takes the bar exam. arXiv preprint arXiv:2212.14402, 2022. · arXiv
- Eva AM van Dis, Johan Bollen, Willem Zuidema, Robert van Rooij, and Claudi L Bockting. Chatgpt: five priori- ties for research. Nature, 614(7947):224-226, 2023.
- Gerd Kortemeyer. Could an artificial-intelligence agent pass an introductory physics course? arXiv preprint arXiv:2301.12127, 2023. · arXiv
- OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
- David Hestenes, Malcolm Wells, and Gregg Swackhamer. Force concept inventory. The physics teacher, 30(3):141- 158, 1992.
- Colin G West. Ai and the fci: Can chatgpt project an understanding of introductory physics? arXiv preprint arXiv:2303.01067, 2023. · arXiv
- Carl Wieman and Katherine Perkins. Transforming physics education. Physics today, 58(11):36, 2005.
- Andrew Elby. Helping physics students learn how to learn. American Journal of Physics, 69(S1):S54-S64, 2001.
- Jonathan Tuminaro and Edward F Redish. Understand- ing students' poor performance on mathematical problem solving in physics. In AIP Conference Proceedings, vol- ume 720, pages 113-116. American Institute of Physics, 2004.
- David Hestenes and Ibrahim Halloun. Interpreting the force concept inventory: A response to march 1995 critique by huffman and heller. The physics teacher, 33(8):502-502, 1995.
- Richard R Hake. Interactive-engagement versus tradi- tional methods: A six-thousand-student survey of me- chanics test data for introductory physics courses. Amer- ican journal of Physics, 66(1):64-74, 1998.
- Joshua Von Korff, Benjamin Archibeque, K Alison Gomez, Tyrel Heckendorf, Sarah B McKagan, Eleanor C Sayre, Edward W Schenk, Chase Shepherd, and Lane Sorell. Secondary analysis of teaching methods in intro- ductory physics: A 50 k-student study. American Journal of physics, 84(12):969-974, 2016.
- Maja Planinic, Lana Ivanjek, and Ana Susac. Rasch model based analysis of the force concept inventory. Physical Review Special Topics-Physics Education Re- search, 6(1):010103, 2010.
- Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. · arXiv
- Fredrik Jeppsson, Jesper Haglund, and Tamer G Amin. Varying use of conceptual metaphors across levels of ex- pertise in thermodynamics. International Journal of Sci- ence Education, 37(5-6):780-805, 2015.
- Patrick B Kohl and Noah D Finkelstein. Patterns of multiple representation use by experts and novices during physics problem solving. Physical review special topics- Physics education research, 4(1):010111, 2008.
- Stuart Armstrong, Kaj Sotala, and Seán S Ó h Éigeartaigh. The errors, insights and lessons of famous AI predictions-and what they mean for the future. Journal of Experimental & Theoretical Artificial Intelligence, 26(3):317-342, 2014.
- Nick Bostrom. Superintelligence: Paths, Dangers, Strate- gies. Oxford University Press, Oxford, 2014.
- Debby RE Cotton, Peter A Cotton, and J Reuben Ship- way. Chatting and cheating: Ensuring academic integrity in the era of chatgpt. Innovations in Education and Teaching International, pages 1-12, 2023.
- Marvin L Minsky. Why people think computers can't. AI magazine, 3(4):3-3, 1982.
- Robert Brandom. Articulating reasons: An introduction to inferentialism. Harvard University Press, 2009.
- Ruben Noorloos, Sam Taylor, Arthur Bakker, and Jan Derry. An inferentialist alternative to constructivism in mathematics education. North American Chapter of the International Group for the Psychology of Mathematics Education, 2014.
Cited by
Sorted by citations
Wang, K. D., Wu, Z., Tufts, L., Wieman, C., Salehi, S., & Haber, N. (2025) · 2025 IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016406 · Cited by 28
Metadata
- Authors
- West, C. G.
- Affiliations
- University of Colorado Boulder (US)
- Corresponding author
- West, Colin G.
- ORCID
- West, Colin G.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2303.17012
- OpenAlex
- W4361865432
- Cited by
- 16
- References
- 1
- Citations by year
- 2025: 4 · 2024: 5 · 2023: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Education › Science Education and Pedagogy
- Social Sciences › Education › Artificial Intelligence in Education
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- GPT-4, large language models, physics reasoning, Force Concept Inventory, classical mechanics, ChatGPT, physics education, artificial intelligence in education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
