Could an artificial-intelligence agent pass an introductory physics course?
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Massive pretrained language models have garnered attention and controversy due to their ability to generate humanlike responses: Attention due to their frequent indistinguishability from human-generated phraseology and narratives and controversy due to the fact that their convincingly presented arguments and facts are frequently simply false.Just how humanlike are these responses when it comes to dialogues about physics, in particular about the standard content of introductory physics courses?This case study explores that question by having ChatGPT, the preeminent language model in 2023, work through representative assessment content of an actual calculus-based physics course and grading the responses in the same way human responses would be graded.As it turns out, ChatGPT would narrowly pass this course while exhibiting many of the preconceptions and errors of a beginning learner.A discussion of possible consequences for teaching, testing, and physics education research is provided as a possible starter for more detailed studies and curricular efforts in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 1×
“Frequently, it is hard not to imagine an army of gig-economy workers behind the scenes of ChatGPT answering to the prompts, so the system would definitely pass the Turing Test most of the time [30], but for better or worse, sometimes it still fails in a way that only computers do — it does not have any metacognition,…”
BackgroundHake, R. R. (1998) · American Journal of Physics · doi:10.1119/1.18809 · Cited by 6,072 · cited 1×
“In the original course, the Force Concept Inventory was administered as a pre-/post-test in order to calculate gains [17].”
MethodCategorization and Representation of Physics Problems by Experts and Novices*
Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981) · Cognitive Science · doi:10.1207/s15516709cog0502_2 · Cited by 5,248 · cited 1×
“As can be seen, these surface features do not matter, so in that respect, ChatGPT does not act like a novice [19] (however, the reality is not quite as straightforward as this expert-novice distinction [20]).”
BackgroundHestenes, D., Wells, M., & Swackhamer, G. (1992) · The Physics Teacher · doi:10.1119/1.2343497 · Cited by 3,235 · cited 1×
“Several errors are related to “impetus” [21]: more than once did ChatGTP assume that an object immediately moves in the direction of an applied force, independent of initial movement (answering 8A, 9B, and 21B) and even that it returns to the original movement when the force is no longer applied (answering 23A).”
BackgroundPeer Instruction: Ten years of experience and results
Crouch, C. H., & Mazur, E. (2001) · American Journal of Physics · doi:10.1119/1.1374249 · Cited by 2,682 · cited 2×
“Unfortunately, this flies in the face of much of physics education research that favors frequent formative assessment [8, 25, 36, 37] and spaced repetition [38, 39], and it is much in contrast to the work environments our students will find.”
BackgroundResultFunctions, Graphs, and Graphing: Tasks, Learning, and Teaching
Leinhardt, G., Zaslavsky, O., & Stein, M. K. (1990) · Review of Educational Research · doi:10.3102/00346543060001001 · Cited by 1,103
Replication and Analysis of Ebbinghaus’ Forgetting Curve
Murre, J. M. J., & Dros, J. (2015) · PLOS ONE · doi:10.1371/journal.pone.0120644 · Cited by 777 · cited 1×
“Unfortunately, this flies in the face of much of physics education research that favors frequent formative assessment [8, 25, 36, 37] and spaced repetition [38, 39], and it is much in contrast to the work environments our students will find.”
ResultPromoting Student Metacognition
Tanner, K. D. (2012) · CBE—Life Sciences Education · doi:10.1187/cbe.12-03-0033 · Cited by 618
Physics Exams that Promote Collaborative Learning
Wieman, C. E., Rieger, G. W., & Heiner, C. E. (2014) · The Physics Teacher · doi:10.1119/1.4849159 · Cited by 67
Research works printed but not linked: 39
- 10.1007/s10956-016-9659-8 · DOI
- 10.1103/PhysRevPhysEducRes.16.020123 · DOI
- 10.1119/1.10458 · DOI
- 10.1119/1.17764 · DOI · Paper page
- 10.1103/PhysRevSTPER.3.020108 · DOI · Paper page
- 10.1119/1.4937964 · DOI
- 10.1109/MC.2022.3151419 · DOI
- 10.1119/1.2835046 · DOI
- 10.1119/1.4922256 · DOI
- 10.1007/s10956-016-9613-9 · DOI
- 10.1088/1361-6404/aad511 · DOI
- 10.1119/1.2798359 · doi:10.1119/1.2798359
- 10.1119/1.4719112 · DOI
- 10.1119/1.2344278 · DOI · Paper page
- 10.1119/1.12989 · DOI · Paper page
- 10.1103/PhysRevSTPER.8.020116 · DOI
- 10.1119/1.4928358 · doi:10.1119/1.4928358
- A. M. Pascarella, 2004, Proceedings of the NARST 2004 Annual Meeting, Vancouver, BC, Canada
- 10.1109/5992.877397 · doi:10.1109/5992.877397
- 10.1016/j.artint.2005.10.009 · DOI · Paper page
- 10.1063/1.4789648 · doi:10.1063/1.4789648
- 10.1103/PhysRevPhysEducRes.18.020109 · DOI
- L. P. Fettig, 1976, NACTA J., 20, 15
- 10.3102/00346543050001071 · doi:10.3102/00346543050001071
- 10.1119/1.1804662 · DOI
- 10.1007/s10648-011-9191-6 · DOI · Paper page
- H. Ebbinghaus, 1885, Über das Gedächtnis: Untersuchungen zur experimentellen Psychologie
- 10.1119/1.16667 · DOI · Paper page
- 10.1119/1.3119150 · DOI · Paper page
- 10.1103/PhysRevPhysEducRes.15.010136 · DOI
- 10.1119/1.17003 · DOI · Paper page
- 10.1088/0031-9120/42/3/003 · doi:10.1088/0031-9120/42/3/003
- 10.1088/1361-6552/aac05e · doi:10.1088/1361-6552/aac05e
- A. S. Theobold, 2021, J. Stat. Data Sci. Educ., 29, 156, 10.1080/26939169.2021.1914527 · DOI · Paper page
- 10.1119/1.2340150 · doi:10.1119/1.2340150
- 10.1002/andp.19053220607 · DOI · Paper page
- D. L. Nelson, 2003, J. Exp. Psychol., 29, 42
- 10.1063/1.2177021 · DOI
- 10.1119/1.2186684 · DOI
Cited by
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01852 · Cited by 1,172
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855
“Kortemeyer et al. [44] (FCI) to evaluate ChatGPT’s accuracy in answering physics concept problems related to kinematics and Newtonian mechanics in the first semester of college physics.”
BackgroundA Categorical Archive of ChatGPT Failures
Borji, A. (2023) · arXiv · doi:10.48550/arxiv.2302.03494 · Cited by 514
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514
A large-scale comparison of human-written versus ChatGPT-generated essays
Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-45644-9 · Cited by 338
“For example, Kortemeyer 32 conducts a case study to assess how well ChatGPT-3 would perform in a physics class, simulating the tasks that students need to complete as part of the course: answer multiple-choice questions, do homework assignments, ask questions during a lesson, complete programming exercises, and write…”
Influential
Metadata
- Authors
- Kortemeyer, G.
- Venue
- Physical Review Physics Education Research
- Publisher
- American Physical Society (APS)
- Published
- 2023-05-11
- Volume
- 19
- Issue
- 1
- ISSN
- 2469-9896 (ISSN-L)
- OpenAlex
- W4377855098
- Cited by
- 175
- References
- 9
- FWCI
- 41.12
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.