Could an artificial-intelligence agent pass an introductory physics course?
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Massive pretrained language models have garnered attention and controversy due to their ability to generate humanlike responses: Attention due to their frequent indistinguishability from human-generated phraseology and narratives and controversy due to the fact that their convincingly presented arguments and facts are frequently simply false.Just how humanlike are these responses when it comes to dialogues about physics, in particular about the standard content of introductory physics courses?This case study explores that question by having ChatGPT, the preeminent language model in 2023, work through representative assessment content of an actual calculus-based physics course and grading the responses in the same way human responses would be graded.As it turns out, ChatGPT would narrowly pass this course while exhibiting many of the preconceptions and errors of a beginning learner.A discussion of possible consequences for teaching, testing, and physics education research is provided as a possible starter for more detailed studies and curricular efforts in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 1×
“Frequently, it is hard not to imagine an army of gig-economy workers behind the scenes of ChatGPT answering to the prompts, so the system would definitely pass the Turing Test most of the time [30], but for better or worse, sometimes it still fails in a way that only computers do — it does not have any metacognition,…”
BackgroundHake, R. R. (1998) · American Journal of Physics · doi:10.1119/1.18809 · Cited by 6,072 · cited 1×
“In the original course, the Force Concept Inventory was administered as a pre-/post-test in order to calculate gains [17].”
MethodCategorization and Representation of Physics Problems by Experts and Novices*
Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981) · Cognitive Science · doi:10.1207/s15516709cog0502_2 · Cited by 5,248 · cited 1×
“As can be seen, these surface features do not matter, so in that respect, ChatGPT does not act like a novice [19] (however, the reality is not quite as straightforward as this expert-novice distinction [20]).”
BackgroundHestenes, D., Wells, M., & Swackhamer, G. (1992) · The Physics Teacher · doi:10.1119/1.2343497 · Cited by 3,235 · cited 1×
“Several errors are related to “impetus” [21]: more than once did ChatGTP assume that an object immediately moves in the direction of an applied force, independent of initial movement (answering 8A, 9B, and 21B) and even that it returns to the original movement when the force is no longer applied (answering 23A).”
BackgroundPeer Instruction: Ten years of experience and results
Crouch, C. H., & Mazur, E. (2001) · American Journal of Physics · doi:10.1119/1.1374249 · Cited by 2,682 · cited 2×
“Unfortunately, this flies in the face of much of physics education research that favors frequent formative assessment [8, 25, 36, 37] and spaced repetition [38, 39], and it is much in contrast to the work environments our students will find.”
BackgroundResultFunctions, Graphs, and Graphing: Tasks, Learning, and Teaching
Leinhardt, G., Zaslavsky, O., & Stein, M. K. (1990) · Review of Educational Research · doi:10.3102/00346543060001001 · Cited by 1,103
Replication and Analysis of Ebbinghaus’ Forgetting Curve
Murre, J. M. J., & Dros, J. (2015) · PLOS ONE · doi:10.1371/journal.pone.0120644 · Cited by 777 · cited 1×
“Unfortunately, this flies in the face of much of physics education research that favors frequent formative assessment [8, 25, 36, 37] and spaced repetition [38, 39], and it is much in contrast to the work environments our students will find.”
ResultPromoting Student Metacognition
Tanner, K. D. (2012) · CBE—Life Sciences Education · doi:10.1187/cbe.12-03-0033 · Cited by 618
Physics Exams that Promote Collaborative Learning
Wieman, C. E., Rieger, G. W., & Heiner, C. E. (2014) · The Physics Teacher · doi:10.1119/1.4849159 · Cited by 67
Research works printed but not linked: 39
- 10.1007/s10956-016-9659-8 · DOI
- 10.1103/PhysRevPhysEducRes.16.020123 · DOI
- 10.1119/1.10458 · DOI
- 10.1119/1.17764 · DOI · Paper page
- 10.1103/PhysRevSTPER.3.020108 · DOI · Paper page
- 10.1119/1.4937964 · DOI
- 10.1109/MC.2022.3151419 · DOI
- 10.1119/1.2835046 · DOI
- 10.1119/1.4922256 · DOI
- 10.1007/s10956-016-9613-9 · DOI
- 10.1088/1361-6404/aad511 · DOI
- 10.1119/1.2798359 · doi:10.1119/1.2798359
- 10.1119/1.4719112 · DOI
- 10.1119/1.2344278 · DOI · Paper page
- 10.1119/1.12989 · DOI · Paper page
- 10.1103/PhysRevSTPER.8.020116 · DOI
- 10.1119/1.4928358 · doi:10.1119/1.4928358
- A. M. Pascarella, 2004, Proceedings of the NARST 2004 Annual Meeting, Vancouver, BC, Canada
- 10.1109/5992.877397 · doi:10.1109/5992.877397
- 10.1016/j.artint.2005.10.009 · DOI · Paper page
- 10.1063/1.4789648 · doi:10.1063/1.4789648
- 10.1103/PhysRevPhysEducRes.18.020109 · DOI
- L. P. Fettig, 1976, NACTA J., 20, 15
- 10.3102/00346543050001071 · doi:10.3102/00346543050001071
- 10.1119/1.1804662 · DOI
- 10.1007/s10648-011-9191-6 · DOI · Paper page
- H. Ebbinghaus, 1885, Über das Gedächtnis: Untersuchungen zur experimentellen Psychologie
- 10.1119/1.16667 · DOI · Paper page
- 10.1119/1.3119150 · DOI · Paper page
- 10.1103/PhysRevPhysEducRes.15.010136 · DOI
- 10.1119/1.17003 · DOI · Paper page
- 10.1088/0031-9120/42/3/003 · doi:10.1088/0031-9120/42/3/003
- 10.1088/1361-6552/aac05e · doi:10.1088/1361-6552/aac05e
- A. S. Theobold, 2021, J. Stat. Data Sci. Educ., 29, 156, 10.1080/26939169.2021.1914527 · DOI · Paper page
- 10.1119/1.2340150 · doi:10.1119/1.2340150
- 10.1002/andp.19053220607 · DOI · Paper page
- D. L. Nelson, 2003, J. Exp. Psychol., 29, 42
- 10.1063/1.2177021 · DOI
- 10.1119/1.2186684 · DOI
Cited by
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01852 · Cited by 1,172
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855
“Kortemeyer et al. [44] (FCI) to evaluate ChatGPT’s accuracy in answering physics concept problems related to kinematics and Newtonian mechanics in the first semester of college physics.”
BackgroundA Categorical Archive of ChatGPT Failures
Borji, A. (2023) · arXiv · doi:10.48550/arxiv.2302.03494 · Cited by 514
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514
A large-scale comparison of human-written versus ChatGPT-generated essays
Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-45644-9 · Cited by 338
“For example, Kortemeyer 32 conducts a case study to assess how well ChatGPT-3 would perform in a physics class, simulating the tasks that students need to complete as part of the course: answer multiple-choice questions, do homework assignments, ask questions during a lesson, complete programming exercises, and write…”
InfluentialKestin, G., Miller, K., Klales, A., Milbourne, T., & Ponti, G. (2025) · Scientific Reports · doi:10.1038/s41598-025-97652-6 · Cited by 214
AI Tutoring Outperforms Active Learning
Kestin*, G., Miller*, K., Klales, A., Milbourne, T., & Ponti, G. (2024) · doi:10.21203/rs.3.rs-4243877/v1 · Cited by 214
Alneyadi, S., & Wardat, Y. (2023) · Contemporary Educational Technology · doi:10.30935/cedtech/13417 · Cited by 142
ChatGPT in Teaching and Learning: A Systematic Review
Ali, D., Fatemi, Y., Boskabadi, E., Nikfar, M., Ugwuoke, J., & Ali, H. (2024) · Education Sciences · doi:10.3390/educsci14060643 · Cited by 146
Can ChatGPT Pass High School Exams on English Language Comprehension?
de Winter, J. C. F. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00372-z · Cited by 115
Wan, T., & Chen, Z. (2024) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.20.010152 · Cited by 109
Educational data augmentation in physics education research using ChatGPT
Kieser, F., Wulff, P., Kuhn, J., & Küchemann, S. (2023) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.19.020150 · Cited by 108
“Probing accuracy of ChatGPT on physics contents and the FCI was performed by multiple researchers [33, 34, 41].”
BackgroundMethodResultInfluentialChervenak, J., Lieman, H., Blanco-Breindel, M., & Jindal, S. (2023) · Fertility and Sterility · doi:10.1016/j.fertnstert.2023.05.151 · Cited by 87
Performance of ChatGPT on the test of understanding graphs in kinematics
Polverini, G., & Gregorcic, B. (2024) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.20.010109 · Cited by 70
Toward AI grading of student problem solutions in introductory physics: A feasibility study
Kortemeyer, G. (2023) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.19.020163 · Cited by 59
“A previous version, GPT-3, could already pass an introductory physics course at a nominal level [6], and there are indications that GPT-4 performs even better on physics concepts [7].”
BackgroundSelf-citationChatGPT as a tool for honing teachers’ Socratic dialogue skills
Gregorcic, B., Polverini, G., & Sarlah, A. (2024) · Physics Education · doi:10.1088/1361-6552/ad3d21 · Cited by 36
Making a revolution in physics learning in high schools with ChatGPT: A case study in UAE
Alarbi, K., Halaweh, M., Tairab, H., Alsalhi, N. R., Annamalai, N., & Aldarmaki, F. (2024) · Eurasia Journal of Mathematics, Science and Technology Education · doi:10.29333/ejmste/14983 · Cited by 28
Guo, Q., Zhen, J., Wu, F., He, Y., & Qiao, C. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331241312365 · Cited by 28
Wang, K. D., Wu, Z., Tufts, L., Wieman, C., Salehi, S., & Haber, N. (2025) · 2025 IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016406 · Cited by 28
Using artificial-intelligence tools to make LaTeX content accessible to blind readers
Kortemeyer, G. (2023) · TUGboat · doi:10.47397/tb/44-3/tb138kortemeyer-accessibility · Cited by 14
“In this exploratory study, GPT-4 [16] is used to translate the La-TeX source to HTML while narrating the formulas and graphics in plain language.”
MethodSelf-citation
Metadata
- Authors
- Kortemeyer, G.
- Venue
- Physical Review Physics Education Research
- Publisher
- American Physical Society (APS)
- Published
- 2023-05-11
- Volume
- 19
- Issue
- 1
- ISSN
- 2469-9896 (ISSN-L)
- OpenAlex
- W4377855098
- Cited by
- 175
- References
- 9
- FWCI
- 41.12
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.