CoChat

How do physics students evaluate artificial intelligence responses on comprehension questions? A study on the perceived scientific accuracy and linguistic quality of ChatGPT

Open access
Dahlkemper, Merten Nikolay, Lahme, Simon Zacharias, Klein, Pascal · 2023 · Physical Review Physics Education Research
Cited by 100

The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.

Abstract

This study aimed at evaluating how students perceive the linguistic quality and scientific accuracy of ChatGPT responses to physics comprehension questions.A total of 102 first-and second-year physics students were confronted with three questions of progressing difficulty from introductory mechanics (rolling motion, waves, and fluid dynamics).Each question was presented with four different responses.All responses were attributed to ChatGPT, but in reality, one sample solution was created by the researchers.All ChatGPT responses obtained in this study were wrong, imprecise, incomplete, or misleading.We found little differences in the perceived linguistic quality between ChatGPT responses and the sample solution.However, the students rated the overall scientific accuracy of the responses significantly differently, with the sample solution being rated best for the questions of low and medium difficulty.The discrepancy between the sample solution and the ChatGPT responses increased with the level of selfassessed knowledge of the question content.For the question of highest difficulty (fluid dynamics) that was unknown to most students, a ChatGPT response was rated just as good as the sample solution.Thus, this study provides data on the students' perception of ChatGPT responses and the factors influencing their perception.The results highlight the need for careful evaluation of ChatGPT responses both by instructors and students, particularly regarding scientific accuracy.Therefore, future research could explore the potential of similar "spot the bot" activities in physics education to foster students' critical thinking skills.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1103/physrevphyseducres.19.010142

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Metadata

Authors
Dahlkemper, M. N., Lahme, S. Z., & Klein, P.
Venue
Physical Review Physics Education Research
Published
2023-06-23
Volume
19
Issue
1
ISSN
2469-9896 (ISSN-L)
OpenAlex
W4381805978
Cited by
100
References
8
FWCI
23.50
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.