How do physics students evaluate artificial intelligence responses on comprehension questions? A study on the perceived scientific accuracy and linguistic quality of ChatGPT
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
This study aimed at evaluating how students perceive the linguistic quality and scientific accuracy of ChatGPT responses to physics comprehension questions.A total of 102 first-and second-year physics students were confronted with three questions of progressing difficulty from introductory mechanics (rolling motion, waves, and fluid dynamics).Each question was presented with four different responses.All responses were attributed to ChatGPT, but in reality, one sample solution was created by the researchers.All ChatGPT responses obtained in this study were wrong, imprecise, incomplete, or misleading.We found little differences in the perceived linguistic quality between ChatGPT responses and the sample solution.However, the students rated the overall scientific accuracy of the responses significantly differently, with the sample solution being rated best for the questions of low and medium difficulty.The discrepancy between the sample solution and the ChatGPT responses increased with the level of selfassessed knowledge of the question content.For the question of highest difficulty (fluid dynamics) that was unknown to most students, a ChatGPT response was rated just as good as the sample solution.Thus, this study provides data on the students' perception of ChatGPT responses and the factors influencing their perception.The results highlight the need for careful evaluation of ChatGPT responses both by instructors and students, particularly regarding scientific accuracy.Therefore, future research could explore the potential of similar "spot the bot" activities in physics education to foster students' critical thinking skills.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 1×
“The first chatbot that was introduced to the public was ELIZA [10] already in 1966.”
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Chatterjee, S., & Bhattacharjee, K. K. (2020) · Education and Information Technologies · doi:10.1007/s10639-020-10159-7 · Cited by 747
Chocarro, R., Cortiñas, M., & Marcos-Matás, G. (2023) · Educational Studies · doi:10.1080/03055698.2020.1850426 · Cited by 343
Evaluating Academic Answers Generated Using ChatGPT
Fergus, S., Botha, M., & Ostovar, M. (2023) · Journal of Chemical Education · doi:10.1021/acs.jchemed.3c00087 · Cited by 300
Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics
Holmes, J., Liu, Z., Zhang, L., Ding, Y., Sio, T. T., McGee, L. A., Ashman, J. B., Li, X., Liu, T., Shen, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01938 · Cited by 190
ChatGPT in physics education: A pilot study on easy-to-implement activities
Bitzenbauer, P. (2023) · Contemporary Educational Technology · doi:10.30935/cedtech/13176 · Cited by 181 · cited 1×
“In a recent pilot-study [24], two activities with ChatGPT regarding the critical thinking skills of students were implemented in the physics classroom.”
ChatGPT and the frustrated Socrates
Gregorcic, B., & Pendrill, A.-M. (2023) · Physics Education · doi:10.1088/1361-6552/acc299 · Cited by 125
Cited by
An expectancy value theory (EVT) based instrument for measuring student perceptions of generative AI
Chan, C. K. Y., & Zhou, W. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00284-4 · Cited by 192
ChatGPT in Teaching and Learning: A Systematic Review
Ali, D., Fatemi, Y., Boskabadi, E., Nikfar, M., Ugwuoke, J., & Ali, H. (2024) · Education Sciences · doi:10.3390/educsci14060643 · Cited by 146
Akiba, D., & Fraboni, M. C. (2023) · Education Sciences · doi:10.3390/educsci13090885 · Cited by 115
Wan, T., & Chen, Z. (2024) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.20.010152 · Cited by 109
Maxwell, D., Oyarzun, B., Kim, S., & Bong, J. Y. (2025) · TechTrends · doi:10.1007/s11528-025-01109-6 · Cited by 80
Metadata
- Authors
- Dahlkemper, M. N., Lahme, S. Z., & Klein, P.
- Venue
- Physical Review Physics Education Research
- Publisher
- American Physical Society (APS)
- Published
- 2023-06-23
- Volume
- 19
- Issue
- 1
- ISSN
- 2469-9896 (ISSN-L)
- OpenAlex
- W4381805978
- Cited by
- 100
- References
- 8
- FWCI
- 23.50
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.