CoChat

GEE! Grammar Error Explanation with Large Language Models

Open access
Song, Yixiao, Krishna, Kalpesh, Bhatt, Rajesh, Gimpel, Kevin, Iyyer, Mohit · 2024 · Findings of the Association for Computational Linguistics: NAACL 2024

The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index; none of the 38 sources its claims cite is checked for support yet.

Abstract

Existing grammatical error correction tools do not provide natural language explanations of the errors that they correct in user-written text.However, such explanations are essential for helping users learn the language by gaining a deeper understanding of its grammatical rules (DeKeyser, 2003;Ellis et al., 2006).To address this gap, we propose the task of grammar error explanation, where a system needs to provide one-sentence explanations for each grammatical error in a pair of erroneous and corrected sentences.The task is not easily solved by prompting LLMs: we find that, using one-shot prompting, GPT-4 only explains 40.6% of the errors and does not even attempt to explain 39.8% of the errors.Since LLMs struggle to identify grammar errors, we develop a two-step pipeline that leverages fine-tuned and prompted large language models to perform structured atomic token edit extraction, followed by prompting GPT-4 to explain each edit.We evaluate our pipeline on German, Chinese, and English grammar error correction data.Our atomic edit extraction achieves an F1 of 0.93 on German, 0.91 on Chinese, and 0.891 on English.Human evaluation of generated explanations reveals that 93.9% of German errors, 96.4% of Chinese errors, and 92.20% of English errors are correctly detected and explained.To encourage further research, we open-source our data and code. 1

Overview

CoChat Summary

TL;DR

The paper proposes grammar error explanation (GEE), where a system generates one-sentence explanations for each grammatical error given an erroneous sentence and its corrected version. It develops a two-step LLM pipeline—atomic token edit extraction followed by GPT-4 explanation—and reports high edit-extraction F1 and human-judged explanation correctness in German, Chinese, and English.

Key findings

  • One-shot GPT-4 is insufficient for GEE: it detects only 60.2% of true errors and correctly explains 67.5% of detected errors (40.6% of all errors; it does not attempt 39.8% of errors). It mentions 120 edits vs. 90 gold edits, with recall 0.602, precision 0.675, and F1 0.636. Providing gold atomic edits raises error-coverage F1 to 0.968 and 82% of true errors receive an appropriate explanation.
  • Atomic edit extraction: fine-tuned GPT-3.5-turbo is best for German (recall 0.923, precision 0.939, F1 0.931) and English (F1 0.891); prompted GPT-4 is best for Chinese (average recall 0.888, precision 0.924, F1 0.906).
  • Human evaluation of GEE outputs finds 93.9% of German, 96.4% of Chinese, and 92.20% of English explanations correct. German teacher agreement is 89.6%; Chinese teacher agreement is 92.9%.
  • Remaining issues include GPT-4 producing overly general error types, insufficient context for some preposition errors, and unclear word-choice contrasts in Chinese.

Methods

  • Task definition: input is a pair \(X_{\text{error}}\), \(X_{\text{correct}}\); output is one-sentence natural-language explanation per atomic edit, with edits typed as insert, delete, replace, or relocate.
  • Pipeline: Step 1 extracts atomic edits by prompting Claude-2, GPT-3.5-turbo, and GPT-4 and by fine-tuning Llama2-7B and GPT-3.5-turbo; Step 2 appends extracted edits to the sentence pair and prompts GPT-4 with language-specific few-shot prompts to generate an explanation and error type.
  • Data and evaluation: German uses Falko and Merlin (1122 GEE sentence pairs; 500 fine-tune and 50 test pairs with 1598 and 186 gold edits); Chinese uses CGED2017 (496 fine-tune, 53 test, 970 GEE pairs); English uses W&I+LOCNESS from BEA-2019 (512 fine-tune, 57 test, 93 GEE pairs). Edit extraction is evaluated with precision/recall/F1; GEE outputs are evaluated by human teachers for German and Chinese and manually by the authors for English.

Limitations

  • The paper states the system only considers sentence-level inputs, so error types such as word choice and coreference may benefit from larger context.
  • The paper states the Chinese data come from the HSK test, so covered topics are limited and do not include learners from all proficiency levels; error types may not be representative.
  • The paper states that, due to time limitations, English language teachers were not hired to evaluate English GEE outputs; the text also makes evident that English GEE was evaluated by the authors rather than independent teachers.
  • The paper notes the relocation edit type reduces model performance because models tend to label relocated tokens as deletion plus insertion.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2024.findings-naacl.49

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
CC-BY · From the PDF
Figure 2 The prompt used for generating German grammar error explanation given an input defined in Section 2. The prompt consists of: (1) task description, (2) generic explanation template, (3) few-shot examples, and (4) current input. The full prompts for German and Chinese are in Appendix E.
Figure 2 The prompt used for generating German grammar error explanation given an input defined in Section 2. The prompt consists of: (1) task description, (2) generic explanation template, (3) few-shot examples, and (4) current input. The full prompts for German and Chinese are in Appendix E.CC-BY · From the PDF
Figure 3 The sentence length distribution of the data in 2017 CGED training set Rao et al. (2020). Most of the sentences have less than 50 tokens. For the bars that are invisible in the plot, we add the numbers to them.
Figure 3 The sentence length distribution of the data in 2017 CGED training set Rao et al. (2020). Most of the sentences have less than 50 tokens. For the bars that are invisible in the plot, we add the numbers to them.CC-BY · From the PDF
Figure 4 A screenshot of the interface presented to the annotators for explanation evaluation.
Figure 4 A screenshot of the interface presented to the annotators for explanation evaluation.CC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 2 Not valid · 25 Not checked · 131 Own results and interpretations

86 more claims

Create a free account to see all 158 claims and to verify the 27 references still open.

See all claims — free account
  • It may state two separate claims as one. 25
  • The quote only partly states this claim. 7
  • Probably not a substantial claim. 50

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 5

Citations

Cites

Show all 35 references

Cited by

See all 62

Metadata

Authors
Song, Y., Krishna, K., Bhatt, R., Gimpel, K., & Iyyer, M.
Affiliations
University of Massachusetts Amherst (US)
Venue
Findings of the Association for Computational Linguistics: NAACL 2024
Published
2024
Pages
754-781
OpenAlex
W4401042780
Cited by
62
References
32
Citations by year
2026: 25 · 2025: 28 · 2024: 9
FWCI
5.03
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (2202506, 2312949)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
grammatical error correction, large language models, GPT-4, error detection, language learning, multilingual grammar, Chinese grammar, German grammar, English grammar

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: GEE! Grammar Error Explanation with Large Language Models. https://app.cochat.ai/papers/10.18653/v1/2024.findings-naacl.49
The paper
Song, Y., Krishna, K., Bhatt, R., Gimpel, K., & Iyyer, M. (2024). GEE! Grammar Error Explanation with Large Language Models. Findings of the Association for Computational Linguistics: NAACL 2024. https://doi.org/10.18653/v1/2024.findings-naacl.49

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.