The DOI matches its Crossref record; 95 of the 95 references it cites were found in a registry or an index; 9 of the 52 sources its claims cite were checked for support.
Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries
Open accessAbstract
ChatGPT frequently appears in the media, with many predicting significant disruptions, especially in the fields of accounting and auditing. Yet research has demonstrated relatively poor performance of ChatGPT on student assessment questions. We extend this research to examine whether more recent ChatGPT models and capabilities can pass major accounting certification exams including the Certified Public Accountant (CPA), Certified Management Accountant (CMA), Certified Internal Auditor (CIA), and Enrolled Agent (EA) certification exams. We find that the ChatGPT 3.5 model cannot pass any exam (average score across all assessments of 53.1%). However, with additional enhancements, ChatGPT can pass all sections of each tested exam: moving to the ChatGPT 4 model improved scores by an average of 16.5%, providing 10-shot training improved scores an additional 6.6%, and allowing the model to use reasoning and acting (e.g., allow ChatGPT to use a calculator and other resources) improved scores an additional 8.9%. After all these improvements, ChatGPT passed all exams with an average score of 85.1%. This high performance indicates that ChatGPT has sufficient capabilities to disrupt the accounting and auditing industries, which we discuss in detail. This research provides practical insights for accounting professionals, investors, and stakeholders on how to adapt and mitigate the potential harms of this technology in accounting and auditing firms.
Overview
CoChat Summary
TL;DR
ChatGPT 3.5 could not pass any major accounting certification exam (average 53.1%), but ChatGPT 4 with 10-shot training and ReAct agents passed all tested sections with an average score of 85.1%. The authors argue this performance indicates ChatGPT has sufficient capability to disrupt accounting and auditing, while cautioning that hype and limitations remain.
Key findings
- ChatGPT 3.5 averaged 53.1% across all tested exam sections and did not pass any section; scores ranged from 37.3% on EA Individuals to 68.0% on CIA Part 3.
- Moving to ChatGPT 4 improved scores by an average of 16.5%, allowing the model to pass 5 sections, including all CIA sections, but it still did not fully pass the CPA, CMA, or EA exams.
- Adding 10-shot training improved scores by an additional 6.6%, letting the model pass both CMA sections; adding ReAct (reasoning and acting, e.g., calculator and web search) improved scores by an additional 8.9%, so ChatGPT passed all sections of each exam with an overall average of 85.1%.
- ReAct had a more pronounced effect on application questions than on remembering-and-understanding questions; a sensitivity test with older models found validation token accuracy increased about 6% up to roughly 4,000 training examples, with possible overfitting beyond that.
Methods
- The authors tested OpenAI’s GPT-3.5-turbo-0301 and GPT-4-0314 models on multiple-choice questions from CPA, CMA, CIA, and EA practice exams, excluding questions with images/tables and workout-type questions.
- Sample sizes varied by exam and testing phase; Table 1 reports CPA 2,000/1,200/200, CMA 300, CIA 150, and EA 150 questions across ChatGPT 3.5 zero-shot, ChatGPT 4 zero-shot, and ChatGPT 4 10-shot conditions.
- They compared percentage correct against pass thresholds (CPA/CIA/EA 75%, CMA 70%; EA uses a scaled 75% benchmark), set temperature to zero, used 10-shot prompting, and added ReAct agents with calculator and web-search tools plus chain-of-thought reasoning.
Limitations
- Stated: the study omits questions requiring greater cognitive ability, such as interpreting situations/contexts and visualizations; it tests practice exams rather than actual exams; and it does not test actual accounting tasks such as bank reconciliations, tax preparation, or closing the books.
- Evident: sample sizes differ by exam and phase because of available review material and the higher cost of advanced models; testing is limited to multiple-choice questions without images or tables.
- Evident: fine-tuning was not available for GPT-3.5 and GPT-4 at the time, and the discussion sections include normative statements, conjectures, and authors’ experiences rather than empirical tests.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
7 Valid · 5 Not valid · 43 Not checked · 95 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“They found that a one-standard-deviation increase in AI investments is associated with a reduction in accounting employees that reaches 3.6% after three years and 7.1% after four years.”
What the sources say
- Supportedcites Fedyk et al. (2022) source
The “Growth accounting employees” section reports that a one-standard-deviation increase in the share of AI workers over the prior three years corresponds to 3.6% fewer accounting employees three years later and 7.1% fewer four years later. The Introduction reports the same figures for the total number of accounting employees at the firm. The claim’s “AI investments” paraphrases the paper’s measure of the share of AI workers. (LLM, conf=0.94)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“Our results show that the original ChatGPT 3.5 model performs like the Wood et al. (2023) tests using accounting assessment questions. The average across all parts of exams was 53.1%, compared to Wood et al.'s (2023) overall average of 55%.”
What the sources say
- Not checked — no full text of the sourcecites Wood et al. (2023) source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 3 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“allowing the model to use reasoning and acting (e.g., allow ChatGPT to use a calculator and other resources) improved scores an additional 8.9%”
- Also stated in: § 1 Introduction · p. 4 · plain
“allowing ChatGPT to react and reason improves scores by an additional 8.9%”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 6 assumptions not shown here.
- Also stated in: § 1 Introduction · p. 4 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“moving to the ChatGPT 4 model improved scores by an average of 16.5%”
- Also stated in: § 1 Introduction · p. 4 · boosted
“Using the new ChatGPT 4 model markedly improves scores by an average of 16.5%.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 6 assumptions not shown here.
- Also stated in: § 1 Introduction · p. 4 · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After all these improvements, ChatGPT passed all exams with an average score of 85.1%.”
- Also stated in: § 1 Introduction · p. 4 · plain
“The results are that ChatGPT 4, with few-shot training and the ability to react and reason, results in an overall average score of 85.1% across all content tested.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 6 assumptions not shown here.
- Also stated in: § 1 Introduction · p. 4 · plain
38 more claims
Create a free account to see all 150 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 88
- It may state two separate claims as one. 42
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI models have long passed the profession's certification exams: in 2023, GPT-4 with 10-shot prompting averaged 85.1% across the CPA, CMA, CIA, and Enrolled Agent exams (Eulerich et al., 2024).”
From APEX-Accounting (2026) · cites this paper as Eulerich et al., 2024
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3 Methodology · p. 13
“Given our sample sizes are all above 150 for each section of each exam, this choice is unlikely to bias our results.”
§ 4 Results · p. 14
“| CIA | 75% | 150 | 150 | 150 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 1 Introduction · p. 4
“We take a random sample of 150 to 300 questions for each part of each exam for these tests.”
§ 4 Results · p. 14
“| CPA | 75% | 2,000 | 1,200 | 200 |”
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Generalizing from a Few Examples: A Survey on Few-shot Learning
Wang, Y., Yao, Q., Kwok, J. T., & Ni, L. M. (2021) · ACM Computing Surveys · doi:10.1145/3386252 · Cited by 3,165 · cited 1×
“Few-shot training is a method in which the model is provided a few examples before submitting questions for testing (Wang et al. 2020).”
GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 2,330
Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence
Noy, S., & Zhang, W. (2023) · doi:10.2139/ssrn.4375283 · Cited by 2,157
Experimental evidence on the productivity effects of generative artificial intelligence
Noy, S., & Zhang, W. (2023) · Science · doi:10.1126/science.adh2586 · Cited by 2,150
Dell’Acqua, F., McFowland, E., Mollick, E., Lifshitz, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2026) · Organization Science · doi:10.1287/orsc.2025.21838 · Cited by 1,189
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
Toward Blockchain-Based Accounting and Assurance
Dai, J., & Vasarhelyi, M. A. (2017) · Journal of Information Systems · doi:10.2308/isys-51804 · Cited by 881 · cited 4×
“…offers an array of possible benefits in these sectors, such as enhanced transparency, data immutability, smart controls, and transactional security as well as close-to-real time audits based on the increased auditability (e.g., Dai and Vasarhelyi 2017; Nordgren et al. 2019; Kwilinski 2019).”
InfluentialDell'Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4573321 · Cited by 794
Research works printed but not linked: 5
- Appelbaum, D., and R. A. Nehmer. 2017b. The Coming Disruption of Drones, Robots, and Bots: How Will It Affect CPAs and Accounting Practice? CPA Journal 87, no. 6. https://www.cpajournal.com/2017/07/03/coming-disruption-drones-robots-bots/. Accessed 8 Apr 2024.
- PWC. 2023a. PwC US Makes $1 Billion Investment to Expand and Scale AI Capabilities. https://www.pwc.com/us/en/about-us/newsroom/press-releases/pwc-us-makes-billion-investment-in-ai-capabilities.html. Accessed 8 Apr 2024.
- PWC. 2023b. PwC Announces Strategic Alliance with Harvey, Positioning PwC’s Legal Business Solutions at the Forefront of Legal Generative AI. https://www.pwc.com/gx/en/news-room/press-releases/2023/pwc-announces-strategic-alliance-with-harvey-positioning-pwcs-legal-business-solutions-at-the-forefront-of-legal-generative-ai.html. Accessed 8 Apr 2024.
- Rozario, A., and C. Zhang. 2023. The Effect of Artificial Intelligence on the Accuracy of Management Earnings Forecasts. SSRN Electronic Journal. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3850823. Accessed 8 Apr 2024.
- Sanatizadeh, A., Y. Lu, K. Zhao, and Y. Hu. 2023. Exploring the Effect of Large Language Models on Knowledge Seeking and Contribution in Online Knowledge Exchange Platforms. SSRN Electronic Journal. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4459729. Accessed 8 Apr 2024.
Cited by
Pursnani, V., Sermet, Y., Kurt, M., & Demir, I. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100183 · Cited by 106
Li, H., Xi, J., Hsu, C. H. C., Yu, B. X. B., & Zheng, X. (2025) · Tourism Management · doi:10.1016/j.tourman.2025.105179 · Cited by 101
The impact of generative AI on information processing: Evidence from the ban of ChatGPT in Italy
Bertomeu, J., Lin, Y., Liu, Y., & Ni, Z. (2025) · Journal of Accounting and Economics · doi:10.1016/j.jacceco.2025.101782 · Cited by 89
Zhang, S., Wang, H., & Yi, X. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757594 · Cited by 36
Generative AI in Financial Reporting
Blankespoor, E., deHaan, E., & Li, Q. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70050 · Cited by 53
Metadata
- Authors
- Eulerich, M., Sanatizadeh, A., Vakilzadeh, H., & Wood, D. A.
- Affiliations
- University of Duisburg-Essen (DE)
- Northern Illinois University (US)
- University of Wisconsin–Whitewater (US)
- Brigham Young University (US)
- Corresponding author
- Eulerich, Marc
- Venue
- Review of Accounting Studies
- Published
- 2024-06-27
- Volume
- 29
- Issue
- 3
- Pages
- 2318-2349
- ISSN
- 1380-6653 (ISSN-L)
- 1573-7136
- OpenAlex
- W4400091704
- Cited by
- 195
- References
- 69
- Citations by year
- 2026: 55 · 2025: 64 · 2024: 33
- FWCI
- 43.52
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Review of Accounting Studies (publishedVersion, cc-by)
- SSRN Electronic Journal (submittedVersion)
- Econstor (Econstor) (submittedVersion, cc-by)
- ScholarsArchive (Brigham Young University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- Funders
- Universität Duisburg-Essen
- Topics
- Social Sciences › Business, Management and Accounting › Management Information Systems › Robotic Process Automation Applications
- Social Sciences › Business, Management and Accounting › Accounting › Accounting Education and Careers
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- ChatGPT, large language models, auditing, disruptive technologies, CPA exam, ChatGPT-4, accounting education, future of accounting profession
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries. https://app.cochat.ai/papers/10.1007/s11142-024-09833-9
- The paper
- Eulerich, M., Sanatizadeh, A., Vakilzadeh, H., & Wood, D. A. (2024). Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries. Review of Accounting Studies. https://doi.org/10.1007/s11142-024-09833-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.