Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 95 of the 95 references it cites were found in a registry or an index; 9 of the 52 sources its claims cite were checked for support.

Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries

Open access
Eulerich, Marc, Sanatizadeh, Aida, Vakilzadeh, Hamid, Wood, David A. · 2024 · Review of Accounting Studies

Abstract

ChatGPT frequently appears in the media, with many predicting significant disruptions, especially in the fields of accounting and auditing. Yet research has demonstrated relatively poor performance of ChatGPT on student assessment questions. We extend this research to examine whether more recent ChatGPT models and capabilities can pass major accounting certification exams including the Certified Public Accountant (CPA), Certified Management Accountant (CMA), Certified Internal Auditor (CIA), and Enrolled Agent (EA) certification exams. We find that the ChatGPT 3.5 model cannot pass any exam (average score across all assessments of 53.1%). However, with additional enhancements, ChatGPT can pass all sections of each tested exam: moving to the ChatGPT 4 model improved scores by an average of 16.5%, providing 10-shot training improved scores an additional 6.6%, and allowing the model to use reasoning and acting (e.g., allow ChatGPT to use a calculator and other resources) improved scores an additional 8.9%. After all these improvements, ChatGPT passed all exams with an average score of 85.1%. This high performance indicates that ChatGPT has sufficient capabilities to disrupt the accounting and auditing industries, which we discuss in detail. This research provides practical insights for accounting professionals, investors, and stakeholders on how to adapt and mitigate the potential harms of this technology in accounting and auditing firms.

Overview

CoChat Summary

TL;DR

ChatGPT 3.5 could not pass any major accounting certification exam (average 53.1%), but ChatGPT 4 with 10-shot training and ReAct agents passed all tested sections with an average score of 85.1%. The authors argue this performance indicates ChatGPT has sufficient capability to disrupt accounting and auditing, while cautioning that hype and limitations remain.

Key findings

  • ChatGPT 3.5 averaged 53.1% across all tested exam sections and did not pass any section; scores ranged from 37.3% on EA Individuals to 68.0% on CIA Part 3.
  • Moving to ChatGPT 4 improved scores by an average of 16.5%, allowing the model to pass 5 sections, including all CIA sections, but it still did not fully pass the CPA, CMA, or EA exams.
  • Adding 10-shot training improved scores by an additional 6.6%, letting the model pass both CMA sections; adding ReAct (reasoning and acting, e.g., calculator and web search) improved scores by an additional 8.9%, so ChatGPT passed all sections of each exam with an overall average of 85.1%.
  • ReAct had a more pronounced effect on application questions than on remembering-and-understanding questions; a sensitivity test with older models found validation token accuracy increased about 6% up to roughly 4,000 training examples, with possible overfitting beyond that.

Methods

  • The authors tested OpenAI’s GPT-3.5-turbo-0301 and GPT-4-0314 models on multiple-choice questions from CPA, CMA, CIA, and EA practice exams, excluding questions with images/tables and workout-type questions.
  • Sample sizes varied by exam and testing phase; Table 1 reports CPA 2,000/1,200/200, CMA 300, CIA 150, and EA 150 questions across ChatGPT 3.5 zero-shot, ChatGPT 4 zero-shot, and ChatGPT 4 10-shot conditions.
  • They compared percentage correct against pass thresholds (CPA/CIA/EA 75%, CMA 70%; EA uses a scaled 75% benchmark), set temperature to zero, used 10-shot prompting, and added ReAct agents with calculator and web-search tools plus chain-of-thought reasoning.

Limitations

  • Stated: the study omits questions requiring greater cognitive ability, such as interpreting situations/contexts and visualizations; it tests practice exams rather than actual exams; and it does not test actual accounting tasks such as bank reconciliations, tax preparation, or closing the books.
  • Evident: sample sizes differ by exam and phase because of available review material and the higher cost of advanced models; testing is limited to multiple-choice questions without images or tables.
  • Evident: fine-tuning was not available for GPT-3.5 and GPT-4 at the time, and the discussion sections include normative statements, conjectures, and authors’ experiences rather than empirical tests.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11142-024-09833-9

Cites

95 of 95 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

7 Valid · 5 Not valid · 43 Not checked · 95 Own results and interpretations

38 more claims

Create a free account to see all 150 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 88
  • It may state two separate claims as one. 42
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 69 references

Research works printed but not linked: 5

  1. Appelbaum, D., and R. A. Nehmer. 2017b. The Coming Disruption of Drones, Robots, and Bots: How Will It Affect CPAs and Accounting Practice? CPA Journal 87, no. 6. https://www.cpajournal.com/2017/07/03/coming-disruption-drones-robots-bots/. Accessed 8 Apr 2024.
  2. PWC. 2023a. PwC US Makes $1 Billion Investment to Expand and Scale AI Capabilities. https://www.pwc.com/us/en/about-us/newsroom/press-releases/pwc-us-makes-billion-investment-in-ai-capabilities.html. Accessed 8 Apr 2024.
  3. PWC. 2023b. PwC Announces Strategic Alliance with Harvey, Positioning PwC’s Legal Business Solutions at the Forefront of Legal Generative AI. https://www.pwc.com/gx/en/news-room/press-releases/2023/pwc-announces-strategic-alliance-with-harvey-positioning-pwcs-legal-business-solutions-at-the-forefront-of-legal-generative-ai.html. Accessed 8 Apr 2024.
  4. Rozario, A., and C. Zhang. 2023. The Effect of Artificial Intelligence on the Accuracy of Management Earnings Forecasts. SSRN Electronic Journal. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3850823. Accessed 8 Apr 2024.
  5. Sanatizadeh, A., Y. Lu, K. Zhao, and Y. Hu. 2023. Exploring the Effect of Large Language Models on Knowledge Seeking and Contribution in Online Knowledge Exchange Platforms. SSRN Electronic Journal. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4459729. Accessed 8 Apr 2024.

Cited by

See all 195

Metadata

Authors
Eulerich, M., Sanatizadeh, A., Vakilzadeh, H., & Wood, D. A.
Affiliations
  • University of Duisburg-Essen (DE)
  • Northern Illinois University (US)
  • University of Wisconsin–Whitewater (US)
  • Brigham Young University (US)
Corresponding author
Eulerich, Marc
Venue
Review of Accounting Studies
Published
2024-06-27
Volume
29
Issue
3
Pages
2318-2349
ISSN
  • 1380-6653 (ISSN-L)
  • 1573-7136
OpenAlex
W4400091704
Cited by
195
References
69
Citations by year
2026: 55 · 2025: 64 · 2024: 33
FWCI
43.52
Type
journal article
Language
en
Open access
Licence
Locations
  • Review of Accounting Studies (publishedVersion, cc-by)
  • SSRN Electronic Journal (submittedVersion)
  • Econstor (Econstor) (submittedVersion, cc-by)
  • ScholarsArchive (Brigham Young University) (submittedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • Minerva Access (University of Melbourne) (submittedVersion, cc-by)
Funders
Universität Duisburg-Essen
Topics
  • Social Sciences › Business, Management and Accounting › Management Information Systems › Robotic Process Automation Applications
  • Social Sciences › Business, Management and Accounting › Accounting › Accounting Education and Careers
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
ChatGPT, large language models, auditing, disruptive technologies, CPA exam, ChatGPT-4, accounting education, future of accounting profession

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries. https://app.cochat.ai/papers/10.1007/s11142-024-09833-9
The paper
Eulerich, M., Sanatizadeh, A., Vakilzadeh, H., & Wood, D. A. (2024). Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries. Review of Accounting Studies. https://doi.org/10.1007/s11142-024-09833-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.