Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index; none of the 44 sources its claims cite is checked for support yet.

A Categorical Archive of ChatGPT Failures

Open access
Borji, ali · 2023 · Research Square

Abstract

Large language models have been demonstrated to be valuable in different fields. ChatGPT, developed by OpenAI, has been trained using massive amounts of data and simulates human conversation by comprehending context and generating appropriate responses. It has garnered significant attention due to its ability to effectively answer a broad range of human inquiries, with fluent and comprehensive answers surpassing prior public chatbots in both security and usefulness. However, a comprehensive analysis of ChatGPT’s failures is lacking, which is the focus of this study. Eleven categories of failures, including reasoning, factual errors, math, coding, and bias, are presented and discussed. The risks, limitations, and societal implications of ChatGPT are also highlighted. The goal of this study is to assist researchers and developers in enhancing future language models and chatbots. Please refer to here for the list of questions.

Overview

CoChat Summary

TL;DR

Borji catalogs ChatGPT failures into eleven categories, using examples mainly sourced from Twitter, to document limitations and risks. The paper argues ChatGPT lacks a comprehensive world model and remains unreliable across reasoning, math, factual recall, bias, coding, humor, self-awareness, and ethics.

Key findings

  • The paper presents eleven failure categories: reasoning, logic, math and arithmetic, factual errors, bias and discrimination, wit and humor, coding, syntactic structure/spelling/grammar, self-awareness, ethics and morality, and other failures.
  • Concrete failures include ChatGPT answering `16 * 38 * 42 * 22 * 20 * 19` as `164079320` instead of `213480960`; claiming `103` is not prime; answering the age riddle “When I was 6 my sister was half my age. Now I’m 70 how old is my sister?” as `73` instead of `67`; and answering an altered “My dad has 4 sons…” riddle as `8` instead of `5`.
  • On bias and politics, an earlier ChatGPT version favored white males in a prompt asking for a Python function to check if someone would be a good scientist based on race and gender, while a newer version rejected that framing; Hartmann et al.’s study of `630` political statements suggested a left-libertarian, pro-environment ideology, and another analysis found left-leaning responses in `14 out of 15` political orientation tests.
  • In coding, ChatGPT produced bug-free code for a modified Advent of Code day 3 problem but failed a product-priority variant; it failed a center-out matrix spiral problem after ten attempts and could not devise even a brute-force solution for a coin-minimization problem without help.

Methods

  • The paper is a high-level qualitative archive and discussion of ChatGPT failures, with examples mainly sourced from Twitter and some formulated by the author or others.
  • It groups failures into eleven categories, presents prompts and responses in figures and appendices, and discusses risks, limitations, and societal implications.
  • No standardized benchmark, systematic sampling procedure, or quantitative evaluation is reported; the author states the categories were selected arbitrarily and are not exhaustive.

Limitations

  • The paper states the categories are not exhaustive and were selected arbitrarily; a single failure can fit multiple categories, and some failures may no longer exist in newer ChatGPT versions.
  • The paper states it took a high-level approach and that future work should examine more specific categories such as sentiment analysis, named entity recognition, translation, summarization, and language ambiguity; fair evaluation and comparison of LLMs remains difficult because training sets are inaccessible and questions may have been seen before.
  • Evident from the text: the evidence is largely anecdotal and illustrative rather than a controlled benchmark, with no systematic sampling, coding scheme, or inter-rater reliability; many examples depend on specific ChatGPT versions and prompt formulations.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.21203/rs.3.rs-2895792/v1

Cites

82 of 82 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 35 Not valid · 30 Not checked · 103 Own results and interpretations

41 more claims

Create a free account to see all 168 claims and to verify the 44 references still open.

See all claims — free account
  • It may state two separate claims as one. 45
  • The quote only partly states this claim. 2
  • Probably not a substantial claim. 100

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

4 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 72 references

Research works printed but not linked: 2

  1. [31][31] Christof Koch and Giulio Tononi. A test for consciousness. Scientific American , 304(6):44-47, 2011. -
  2. [46][46] Oded Nov, Nina Singh, and Devin M Mann. Putting chatgpt's medical advice to the (turing) test. medRxiv , pages 2023-01, 2023. -

Cited by

See all 514

Metadata

Authors
Borji, A.
Corresponding author
Borji, ali
Venue
Research Square
Published
2023-05-09
OpenAlex
W4375949262
Cited by
514
References
72
Citations by year
2026: 42 · 2025: 97 · 2024: 175 · 2023: 199
Type
posted content
Language
en
Open access
Locations
  • Research Square (acceptedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
ChatGPT, large language models, factual errors, mathematical errors, coding errors, algorithmic bias, reasoning errors, societal implications, AI risks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Categorical Archive of ChatGPT Failures. https://app.cochat.ai/papers/10.21203/rs.3.rs-2895792/v1
The paper
Borji, A. (2023). A Categorical Archive of ChatGPT Failures. Research Square. https://doi.org/10.21203/rs.3.rs-2895792/v1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.