CoChat

DICO-JALF v.1.0: Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision

Open access

The DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index; none of the 53 sources its claims cite is checked for support yet.

Abstract

There is a growing body of research in using AI for corrective feedback in foreign language teaching. However, few studies have specifically addressed the accuracy of AI analysis in learner corpus research. This study aims to create an AI-annotated corpus whose data were obtained from learners of Japanese as a Foreign Language (JFL) in Indonesia with human supervision; branded it as DICO-JALF v.1.0. The aim is to measure to what extent ChatGPT accurately annotates errors. A task was first administered to collect corpus data and metadata to build the corpus. The corpus was error-annotated using ChatGPT 4.0. Human annotators manually supervised the accuracy of AI-generated annotations. Regarding errors committed by learners, it is observed that incorrect lexical choices and forms dominate the cause of errors, while underuse and overuse are minimal. It can be concluded that ChatGPT demonstrated an average accuracy of 70% correct identification of errors. Regarding error rate, the verb is the category where errors are most frequent, which maybe driven by its conjugation, a feature absent in Indonesian, the L1 of the students. This suggests that Indonesian learners' acquisition of Japanese verbs needs greater emphasis. As compared to other similar studies, this is relatively low. However, it can be argued that one factor determining the accuracy of ChatGPT annotations, or any other LLM-based tool, is the complexity of the annotation scheme they adhere to. The corpus have been made available for download. The annotations shall be readable by a corpus query system that reads XML tags. This corpus serves as a foundational resource for future research on AI-assisted error analysis in JFL learning contexts in Indonesia.

Overview

CoChat Summary

TL;DR

This paper introduces DICO-JALF v.1.0, described as the first error-annotated learner corpus of Japanese as a foreign language written by Indonesian students, with ChatGPT 4.0 error annotation and human supervision. It reports that ChatGPT’s annotations averaged 70.31% accuracy and that verb errors were the most frequent learner error category.

Key findings

  • DICO-JALF v.1.0 was created in RAW and ANNOTATED XML versions: RAW contains all 325 essays (70,070 tokens), while ANNOTATED contains the 107-essay sample (19,223 tokens) with ChatGPT error annotation and human evaluation; both are freely available.
  • ChatGPT 4.0’s annotation accuracy was 84.76% for correct segments, 60.44% for erroneous segments, 65.69% for error-type tags, and 70.33% for error-cause tags, with a mean of 70.31% (SD=9.07, Mdn=68.01, IQR=14.48). The paper notes this is lower than Yu et al.’s >90% for English pragmatic annotation.
  • Verb-related errors were prominent: two of the top-three validated error tags were VERB and VERB_INFL; among verb errors, wrong forms accounted for 58%, incorrect verb choice 38%, and overuse 21%. The paper attributes this partly to Japanese verb conjugation, a feature absent in Indonesian, the learners’ L1.
  • Error-type label likelihood varied: PART had the lowest error likelihood (23.26%, i.e., 76.74% chance correct), while PRON (84.21%) and AUX (83.58%) had higher error likelihoods; the abstract states incorrect lexical choices and forms dominate error causes, with underuse and overuse minimal.

Methods

  • Data were collected from approximately 300 students in years 1–3 (2021–2023) in the Department of Japanese Language, Universitas Diponegoro, in June–July 2024. Students wrote a 300–600 character essay on “my hobby” in 75 minutes without AI writing assistance or dictionaries, then completed a metadata questionnaire; 3 essays were excluded. A stratified random sample of 30% per year yielded 107 essays (around 33% of the population; 35/36/36 by year; average characters 372/410/522).
  • Error annotation used ChatGPT 4.0 with prompts to identify incorrect segments and assign error types and cause tags, using a two-layer tagset combining Koyama et al.’s error types with a modified Pavlovic error-cause tagset. Human reviewers evaluated four parameters—Correct Segment (CS), Incorrect Segment (IS), Error Type (ET), and Error Cause (EC)—with Boolean T/F values; accuracy was defined as the proportion of TRUE values among all values.
  • The corpus was converted to XML and indexed in Sketch Engine with MeCab POS tagging. Analysis measured accuracy per parameter and overall mean, the likelihood of each error-type label being correct, and CQL searchability by error tags and human evaluation.

Limitations

  • The paper states that data come from only one university and that the corpus is smaller than IJAS, Natsume, or NAIST lang 8; it calls this a “sensible starting point” and plans future expansion. It also states the sample was around 30% of essays due to time and financial constraints, and that only one essay topic (“my hobby”) was used.
  • The paper states ChatGPT was not completely accurate, sometimes assigned tags beyond the required tagset despite prompts, and that the default ChatGPT temperature was preserved. It suggests annotation-scheme complexity and less Japanese training data may affect accuracy.
  • Evident from the text: no inter-annotator agreement or reliability statistic is reported for the human reviewers; the paper reports only Boolean evaluations by human reviewers. The single-university, single-topic sample also limits generalizability, which the paper acknowledges by saying representativeness may be improved.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.25077/ar.12.3.274-288.2025

Cites

33 of 34 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
CC-BY-NC · From the PDF
Fig 2 ChatGPT's Installed prompt [Japanese Essay Analyzer] and its output sample
Fig 2 ChatGPT's Installed prompt [Japanese Essay Analyzer] and its output sampleCC-BY-NC · From the PDF
Fig 4 Concordance lines from <ER_TYPE EVAL='TRUE'/> with VERB restriction in ER_TYPE Text Types in DICO-JALF's SE (Generated by Concordance tool)
Fig 4 Concordance lines from <ER_TYPE EVAL='TRUE'/> with VERB restriction in ER_TYPE Text Types in DICO-JALF's SE (Generated by Concordance tool)CC-BY-NC · From the PDF
Fig 5 Setting the visibility of XML attribute-values in DICO-JALF's SE (Generated by Concordance tool)
Fig 5 Setting the visibility of XML attribute-values in DICO-JALF's SE (Generated by Concordance tool)CC-BY-NC · From the PDF
Fig 6 Concordance lines from <ER_TYPE> with visible error type in DICO-JALF's SE (Generated by Concordance tool)
Fig 6 Concordance lines from <ER_TYPE> with visible error type in DICO-JALF's SE (Generated by Concordance tool)CC-BY-NC · From the PDF
Fig 7 Distribution of error tags
Fig 7 Distribution of error tagsCC-BY-NC · From the PDF
Fig 8 Distribution of error tags for all verbs errors
Fig 8 Distribution of error tags for all verbs errorsCC-BY-NC · From the PDF

Verification

Claims the paper makes

0 Valid · 5 Not valid · 36 Not checked · 87 Own results and interpretations

45 more claims

Create a free account to see all 128 claims and to verify the 36 references still open.

See all claims — free account
  • Probably not a substantial claim. 60
  • It may state two separate claims as one. 30

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

3 of 3

Citations

Cites

Show all 34 references

Cited by

See all 2

Metadata

Authors
Prihantoro, P., Ishikawa, S., Liu, T., Fadli, Z. A., Rini, E. I. H. A. N., & Kepirianto, C.
Affiliations
  • Diponegoro University (ID)
  • Kobe University (JP)
  • Xi’an Jiaotong-Liverpool University (CN)
Venue
Jurnal Arbitrer
Published
2025-09-30
Volume
12
Issue
3
Pages
274-288
ISSN
  • 2339-1162 (ISSN-L)
  • 2550-1011
OpenAlex
W4414674840
Cited by
2
References
31
Citations by year
2026: 1 · 2025: 1
FWCI
1.29
Type
journal article
Language
en
Open access
Locations
  • JURNAL ARBITRER (publishedVersion, cc-by-nc)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Psychology › Developmental and Educational Psychology › Second Language Acquisition and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
Keywords
Japanese as a foreign language, learner corpus, ChatGPT, human oversight, student errors, lexical errors, verb conjugation, Indonesian learners

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: DICO-JALF v.1.0: Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision. https://app.cochat.ai/papers/10.25077/ar.12.3.274-288.2025
The paper
Prihantoro, P., Ishikawa, S., Liu, T., Fadli, Z. A., Rini, E. I. H. A. N., & Kepirianto, C. (2025). DICO-JALF v.1.0: Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision. Jurnal Arbitrer. https://doi.org/10.25077/ar.12.3.274-288.2025

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.