DICO-JALF v.1.0: Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision
Open accessThe DOI matches its Crossref record; 33 of the 34 references it cites were found in a registry or an index; none of the 53 sources its claims cite is checked for support yet.
Abstract
There is a growing body of research in using AI for corrective feedback in foreign language teaching. However, few studies have specifically addressed the accuracy of AI analysis in learner corpus research. This study aims to create an AI-annotated corpus whose data were obtained from learners of Japanese as a Foreign Language (JFL) in Indonesia with human supervision; branded it as DICO-JALF v.1.0. The aim is to measure to what extent ChatGPT accurately annotates errors. A task was first administered to collect corpus data and metadata to build the corpus. The corpus was error-annotated using ChatGPT 4.0. Human annotators manually supervised the accuracy of AI-generated annotations. Regarding errors committed by learners, it is observed that incorrect lexical choices and forms dominate the cause of errors, while underuse and overuse are minimal. It can be concluded that ChatGPT demonstrated an average accuracy of 70% correct identification of errors. Regarding error rate, the verb is the category where errors are most frequent, which maybe driven by its conjugation, a feature absent in Indonesian, the L1 of the students. This suggests that Indonesian learners' acquisition of Japanese verbs needs greater emphasis. As compared to other similar studies, this is relatively low. However, it can be argued that one factor determining the accuracy of ChatGPT annotations, or any other LLM-based tool, is the complexity of the annotation scheme they adhere to. The corpus have been made available for download. The annotations shall be readable by a corpus query system that reads XML tags. This corpus serves as a foundational resource for future research on AI-assisted error analysis in JFL learning contexts in Indonesia.
Overview
CoChat Summary
TL;DR
This paper introduces DICO-JALF v.1.0, described as the first error-annotated learner corpus of Japanese as a foreign language written by Indonesian students, with ChatGPT 4.0 error annotation and human supervision. It reports that ChatGPT’s annotations averaged 70.31% accuracy and that verb errors were the most frequent learner error category.
Key findings
- DICO-JALF v.1.0 was created in RAW and ANNOTATED XML versions: RAW contains all 325 essays (70,070 tokens), while ANNOTATED contains the 107-essay sample (19,223 tokens) with ChatGPT error annotation and human evaluation; both are freely available.
- ChatGPT 4.0’s annotation accuracy was 84.76% for correct segments, 60.44% for erroneous segments, 65.69% for error-type tags, and 70.33% for error-cause tags, with a mean of 70.31% (SD=9.07, Mdn=68.01, IQR=14.48). The paper notes this is lower than Yu et al.’s >90% for English pragmatic annotation.
- Verb-related errors were prominent: two of the top-three validated error tags were VERB and VERB_INFL; among verb errors, wrong forms accounted for 58%, incorrect verb choice 38%, and overuse 21%. The paper attributes this partly to Japanese verb conjugation, a feature absent in Indonesian, the learners’ L1.
- Error-type label likelihood varied: PART had the lowest error likelihood (23.26%, i.e., 76.74% chance correct), while PRON (84.21%) and AUX (83.58%) had higher error likelihoods; the abstract states incorrect lexical choices and forms dominate error causes, with underuse and overuse minimal.
Methods
- Data were collected from approximately 300 students in years 1–3 (2021–2023) in the Department of Japanese Language, Universitas Diponegoro, in June–July 2024. Students wrote a 300–600 character essay on “my hobby” in 75 minutes without AI writing assistance or dictionaries, then completed a metadata questionnaire; 3 essays were excluded. A stratified random sample of 30% per year yielded 107 essays (around 33% of the population; 35/36/36 by year; average characters 372/410/522).
- Error annotation used ChatGPT 4.0 with prompts to identify incorrect segments and assign error types and cause tags, using a two-layer tagset combining Koyama et al.’s error types with a modified Pavlovic error-cause tagset. Human reviewers evaluated four parameters—Correct Segment (CS), Incorrect Segment (IS), Error Type (ET), and Error Cause (EC)—with Boolean T/F values; accuracy was defined as the proportion of TRUE values among all values.
- The corpus was converted to XML and indexed in Sketch Engine with MeCab POS tagging. Analysis measured accuracy per parameter and overall mean, the likelihood of each error-type label being correct, and CQL searchability by error tags and human evaluation.
Limitations
- The paper states that data come from only one university and that the corpus is smaller than IJAS, Natsume, or NAIST lang 8; it calls this a “sensible starting point” and plans future expansion. It also states the sample was around 30% of essays due to time and financial constraints, and that only one essay topic (“my hobby”) was used.
- The paper states ChatGPT was not completely accurate, sometimes assigned tags beyond the required tagset despite prompts, and that the default ChatGPT temperature was preserved. It suggests annotation-scheme complexity and less Japanese training data may affect accuracy.
- Evident from the text: no inter-annotator agreement or reliability statistic is reported for the human reviewers; the paper reports only Boolean evaluations by human reviewers. The single-university, single-topic sample also limits generalizability, which the paper acknowledges by saying representativeness may be improved.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

![Fig 2 ChatGPT's Installed prompt [Japanese Essay Analyzer] and its output sample](https://figures.cochat.ai/paper-figures/7dc9736d6b174d24890fa16e68ddd6c4/dico-jalf-v-1-0-diponegoro-corpus-of-japanese-learners-as-a-foreign-language-in-figure-2.png)





Verification
Claims the paper makes
0 Valid · 5 Not valid · 36 Not checked · 87 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The corpus is free to access.”
- Also stated in: § Objective 1 Architecture · plain
“These versions are freely available for download 10 and use in users' preferred corpus query systems.”
§ Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision > ()LTTLEMIX-no-ga-ichiban-suki-desu. ()LITTLEMIX-GEN-PS-most-like-COP. '(I) really like little mix'
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Objective 1 Architecture · plain
None of the sources cited for this claim have been checked yet.
In the paper
“If specifically compared to other learner corpora in Japanese, such as the International Corpus of Japanese as a Second Language (IJAS), Natsume (Nishina et al., 2014), the learner corpus created for Imamura et al.'s (Imamura et al., 2014) experiment, or NAIST lang 8 (Kasahara et al., 2011), this corpus is smaller.”
§ Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision > Objective 1 Architecture
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The corpus created corpus fulfils the first aim of this research, which is to create an errorannotated corpus from Indonesian students learning Japanese as a foreign language at various levels.”
§ Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision > Objective 1 Architecture
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The RAW version includes all essays (325), while the ANNOTATED version includes a sample (107 9 ), as shown in Table 6.”
§ Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision > Objective 1 Architecture
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As POS tags and metadata are present, users can perform POS-based searches with metadata restrictions, as shown in Figure 3.”
§ Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision > Objective 1 Architecture
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
45 more claims
Create a free account to see all 128 claims and to verify the 36 references still open.
- Probably not a substantial claim. 60
- It may state two separate claims as one. 30
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Table 4. Quick summary of the samplefoot_4 No Information Year 1 Year 2 Year 3 1 Essay 35 36 36 2 Average characters 372 410 522 3 JLPT/No-JLPT 5/30 15/20 30/5 4 Male/Female 15/20 13/22 9/26 5 Studied Japanese before college (Y/N) 6/29 2/33 20/10”
§ Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision > II. METHODS
Reported 36 · recomputed 30
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Table 4. Quick summary of the samplefoot_4 No Information Year 1 Year 2 Year 3 1 Essay 35 36 36 2 Average characters 372 410 522 3 JLPT/No-JLPT 5/30 15/20 30/5”
§ Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision > II. METHODS
Reported 36 · recomputed 35
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Table 4. Quick summary of the samplefoot_4 No Information Year 1 Year 2 Year 3 1 Essay 35 36 36 2 Average characters 372 410 522 3 JLPT/No-JLPT 5/30 15/20 30/5 4 Male/Female 15/20 13/22 9/26”
§ Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision > II. METHODS
Reported 36 · recomputed 35
Citations
Cites
Karataş, F., Abedi, F. Y., Ozek Gunyel, F., Karadeniz, D., & Kuzgun, Y. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12574-6 · Cited by 252
Massey, P. A., Montgomery, C., & Zhang, A. S. (2023) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-23-00396 · Cited by 190
Yu, D., Li, L., Su, H., & Fuoli, M. (2024) · International Journal of Corpus Linguistics · doi:10.1075/ijcl.23087.yu · Cited by 153
Tyson, J. (2023) · Journal of Chemical Education · doi:10.1021/acs.jchemed.3c00361 · Cited by 87
Capturing L2 Accuracy Developmental Patterns: Insights From an Error‐Tagged EFL Learner Corpus
THEWISSEN, J. (2013) · The Modern Language Journal · doi:10.1111/j.1540-4781.2012.01422.x · Cited by 69
Pan, M., Guo, K., & Lai, C. (2025) · RELC Journal · doi:10.1177/00336882241264030 · Cited by 35
What can corpora tell us about language learning?
Pérez-Paredes, P., & Mark, G. (2022) · Routledge eBooks · doi:10.4324/9780367076399-22 · Cited by 25
Yang, J. C., & Akahori, K. (1998) · CALICO Journal · doi:10.1558/cj.v15i1-3.47-66 · Cited by 21
(2023) · Advances in library and information science (ALIS) book series · doi:10.4018/978-1-6684-7693-2 · Cited by 15
Heintz, K., Roh, Y., & Lee, J. (2022) · Science Editing · doi:10.6087/kcse.261 · Cited by 12
Cited by
A corpus-based comparison of EFL, ESL, and ENS lexical bundles in the ICNALE corpus
Prihantoro, P., & Ishikawa, S. (2025) · Indonesian Journal of Applied Linguistics · doi:10.17509/xq43nf49 · Cited by 2
Self-citationRisdiani, R., Yuwono, U., Halimi, S. S., & Suganda, S. P. (2026) · JURNAL ARBITRER · doi:10.25077/ar.13.3.325-338.2026
Metadata
- Authors
- Prihantoro, P., Ishikawa, S., Liu, T., Fadli, Z. A., Rini, E. I. H. A. N., & Kepirianto, C.
- Affiliations
- Diponegoro University (ID)
- Kobe University (JP)
- Xi’an Jiaotong-Liverpool University (CN)
- Venue
- Jurnal Arbitrer
- Publisher
- Universitas Andalas
- Published
- 2025-09-30
- Volume
- 12
- Issue
- 3
- Pages
- 274-288
- ISSN
- 2339-1162 (ISSN-L)
- 2550-1011
- OpenAlex
- W4414674840
- Cited by
- 2
- References
- 31
- Citations by year
- 2026: 1 · 2025: 1
- FWCI
- 1.29
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- JURNAL ARBITRER (publishedVersion, cc-by-nc)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Second Language Acquisition and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
- Keywords
- Japanese as a foreign language, learner corpus, ChatGPT, human oversight, student errors, lexical errors, verb conjugation, Indonesian learners
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: DICO-JALF v.1.0: Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision. https://app.cochat.ai/papers/10.25077/ar.12.3.274-288.2025
- The paper
- Prihantoro, P., Ishikawa, S., Liu, T., Fadli, Z. A., Rini, E. I. H. A. N., & Kepirianto, C. (2025). DICO-JALF v.1.0: Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision. Jurnal Arbitrer. https://doi.org/10.25077/ar.12.3.274-288.2025
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.