CoChat
Sign Up
Cited by 125Artificial Intelligence in EducationIntelligent Tutoring Systems and Adaptive LearningArtificial Intelligence in Healthcare and Education

ChatGPT and the frustrated Socrates

Open access
Gregorcic, Bor, Pendrill, Ann-Marie·2023·Physics Education
DOI 10.1088/1361-6552/acc299
Open this paper in CoChat — free accountOpen PDF

Abstract

Abstract We present a case study of a conversation between ourselves and an artificial intelligence-based chatbot ChatGPT. We asked the chatbot to respond to a basic physics question that will be familiar to most physics teachers: ‘A teddy bear is thrown into the air. What is its acceleration in the highest point?’ The chatbot’s responses, while linguistically quite advanced, were unreliable in their correctness and often full of contradictions. We then attempted to engage in Socratic dialogue with the chatbot to resolve the errors and contradictions, but with little success. We found that ChatGPT is not yet good enough to be used as a cheating tool for physics students or as a physics tutor. However, we found it quite reliable in generating incorrect responses on which physics teachers could train assessment of student responses.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1088/1361-6552/acc299

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Failures of LLMs in physical commonsense reasoning include lack of knowledge about object attributes, spatial relationships, simple physical laws, and object affordance.

    “Failures of LLMs include lack of knowledge about object attributes (e.g., size, weight, softness) (Wang et al., 2023c; Liu et al., 2022b; Shu et al., 2023; Kondo et al., 2023), spatial relationships (e.g., above, inside, next to) (Liu et al., 2022b; Shu et al., 2023; Kondo et al., 2023), simple physical laws (e.g., gravity, motion, and force) (Gregorcic & Pendrill, 2023), and object affordance”

    From Large Language Model Reasoning Failures (2026) · cites this paper as Gregorcic & Pendrill, 2023

    The section headings “The Socratic dialogue” (§3) and “Multiple choice question and student difficulties” (§4) identify contradictions in ChatGPT’s answers about net force and motion, supporting the narrower point that it can struggle with basic physics questions. The available passages do not establish a general lack of knowledge of simple physical laws, including gravity, motion, and force. The citation is attached to the physical-laws clause, not the separate claims about object attributes, spatial relationships, or affordance. (LLM, conf=0.70)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Gregorcic, B., & Pendrill, A.-M.
Affiliations
  • Uppsala University (SE)
  • Lund University (SE)
Corresponding author
Gregorcic, Bor
ORCID
Venue
Physics Education
Publisher
IOP Publishing
Published
2023-03-22
Volume
58
Issue
3
Pages
035021
ISSN
  • 0031-9120 (ISSN-L)
  • 1361-6552
DOI
10.1088/1361-6552/acc299
OpenAlex
W4360846784
Cited by
125
References
9
Citations by year
2026: 18 · 2025: 42 · 2024: 37 · 2023: 28
FWCI
29.34
Type
journal article
Language
en
Open access
Licence
Locations
  • Physics Education (publishedVersion, cc-by)
  • Uppsala University Publications (Uppsala University) (submittedVersion, other-oa)
  • Lund University Publications (Lund University) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
ChatGPT, large language models, Socratic dialogue, physics tutoring, academic dishonesty, incorrect responses, physics misconceptions, chatbot reliability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review