CoChat

Exploring the Potential of Generative AI for Korean Speaking Assessment : Focusing on Multimodal-based Voice Interaction

Open access
The Korean Association of General Education, Jang, Jiyeong, Park, Sooyeon · 2025 · Korean Journal of General Education

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index; none of the 7 sources its claims cite is checked for support yet.

Abstract

This study explores the potential and limitations of GPT-4o, a multimodal generative AI equipped with real-time voice interaction, in generating responses to Korean speaking proficiency test tasks. To this end, the study collected voice-based responses to six TOPIK speaking tasks generated by GPT-4o and analyzed them both quantitatively and qualitatively based on the evaluations of 18 Korean language education experts. The results indicate that GPT-4o demonstrates high performance in grammatical accuracy and logical structure but reveals clear limitations in prosodic features such as intonation, speech rate, and pragmatic appropriateness. In particular, the use of overly advanced vocabulary and unnatural delivery was repeatedly pointed out as misaligned with actual learner proficiency levels. Nevertheless, the study found that GPT-4o holds potential as a supplementary tool for speaking test preparation, especially for modeling ideal responses, providing self-directed feedback, and enabling repeated practice in autonomous learning contexts. This research underscores the applicability of multimodal AI in Korean language education and suggests the need for future studies that compare various multimodal models and examine response patterns under more complex input conditions combining visual, contextual, and interactive elements.

Overview

CoChat Summary

TL;DR

This study examined GPT-4o’s multimodal voice interaction as a respondent to six TOPIK speaking tasks, with 18 Korean-language education experts scoring 18 AI-generated voice responses. GPT-4o produced generally high-scoring, grammatically accurate and logically structured responses, but showed limitations in naturalness, prosody, pragmatic appropriateness, and level-appropriate vocabulary.

Key findings

  • Across six item types, GPT-4o scored highest on item 5 “자료 해석하기” (M=4.86, SD=0.09) and item 6 “의견 제시하기” (M=4.83, SD=0.08), and lowest on item 2 “그림 보고 역할극 수행하기” (M=4.53, SD=0.17). Repeated-measures ANOVA found significant item-type differences for items 3, 4, and 5 (e.g., item 4 F=34.22, p=.0), borderline for item 1 (p=.0861), and non-significant for items 2 and 6 (p=.4503, p=.9853).
  • By scoring criterion, task performance/discourse construction (criteria 1–3) averaged 4.72–4.78, and language use (criteria 4–6) mostly ≥4.70; accuracy (criterion 6) had the highest mean (4.79, SD=0.18) but was borderline (p=.0795). Delivery/other criteria were lower: pronunciation/intonation (criterion 7) M=4.61, SD=0.21; proficiency-level judgment (criterion 9) M=4.66, SD=0.19. Eight of nine criteria showed significant item-type differences (p<.05), with criteria 1–5 at p<.01.
  • Expert qualitative feedback praised grammatical accuracy and logical structure, especially for advanced items 5 and 6, but repeatedly noted fast speech, artificial intonation, insufficient breath control, pronunciation errors (e.g., 한옥→한욱, 갈등→칼등), and advanced vocabulary/structures beyond the target learner level. Reliability was high: Cronbach’s α ≥ .96 for all evaluation items, and scorer deviations were within 1 point, mostly ≤0.2.

Methods

  • Design: GPT-4o was treated as an AI test-taker using its multimodal real-time voice interaction via smartphone ChatGPT app; six official TOPIK speaking item types were used, with three responses per item (18 total). Voice, image, and text inputs were provided as appropriate; prompts specified role, proficiency level, response timing, and sentence limits.
  • Sample/data: 18 Korean-language education experts (15 PhD holders, 3 PhD candidates; mean 10.5 years experience, range 5–18; experienced in TOPIK/SKA item writing/scoring or related research) independently scored the 18 AI voice responses using nine 5-point criteria plus one open-ended item, based on TOPIK speaking evaluation elements (NIIED, 2023), via TeacherMade online scoring.
  • Analysis: Descriptive statistics, repeated-measures ANOVA (SPSS 28.0) for item-type and criterion differences, Cronbach’s α for inter-rater reliability, and qualitative summary of expert feedback.

Limitations

  • The paper states it studied only one model, GPT-4o; it calls for future comparison of various multimodal AI models, tracking changes with technology, examining how prompts change responses, and analyzing complex input environments combining voice, images, and contextual variables.
  • Evident from the text: the study used a small stimulus set (6 items × 3 responses = 18 AI responses) and no human learner baseline, so findings are tied to this prompt design and scoring setup; the authors also note that current TTS still cannot fully implement human-like suprasegmental features such as subtle intonation, stress, and pauses.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.46392/kjge.2025.19.4.205

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 0 Not valid · 5 Not checked · 51 Own results and interpretations

11 more claims

Create a free account to see all 56 claims and to verify the 6 references still open.

See all claims — free account
  • It may state two separate claims as one. 33
  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 13

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 11 references

Research works printed but not linked: 12

  1. Baek, 2023, Design direction of AI-based KFL writing automatic scoring model, 129
  2. Choi, 2025, Development and validation of an evaluation tool for generative AI-based Korean writing feedback, 227
  3. Kim, 2024a, Developing and exploring the effectiveness of a ChatGPT-based TOPIK speaking practice model as a learning feedback tool, 1
  4. Kim, 2024b, Development and application of a ChatGPT-based feedback model for TOPIK speaking assessment, 1
  5. Kim, 2024c, Designing and validating feedback prompts for question 5 in the TOPIK speaking assessment:A case study using generative AI, 161
  6. Kwon, 2025, A study on the quality evaluation of AI-based Korean writing automatic feedback, 205
  7. Kwon, 2023, Exploring automatic feedback methods for writing instruction, 73
  8. Lee, 2025, Exploring the possibility of Korean writing assessment using LLMs:A comparison of GPT-4o and GPT-4.5, 281
  9. Lee, 2023, Feature selection for automatic Korean writing scoring:Focusing on content and structure, 433
  10. Lee, 2025, A study on the applicability of ChatGPT for Korean writing feedback, 527
  11. Park, 2002, Analysis of variance and repeated measures data
  12. Park, 2024, AI knowledge for non-majors

Metadata

Authors
The Korean Association of General Education, Jang, J., & Park, S.
Affiliations
Ewha Womans University (KR)
Venue
Korean Journal of General Education
Published
2025-08-31
Volume
19
Issue
4
Pages
205-216
OpenAlex
W4414030662
Cited by
1
References
10
Citations by year
2026: 1
FWCI
0.58
Type
journal article
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc/4.0/ (tdm)
Topics
  • Social Sciences › Education › Artificial Intelligence in Education
  • Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
GPT-4o, multimodal generative AI, TOPIK, voice interaction, prosodic features, pragmatic appropriateness, speaking test preparation, self-directed learning, Korean language education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Exploring the Potential of Generative AI for Korean Speaking Assessment : Focusing on Multimodal-based Voice Interaction. https://app.cochat.ai/papers/10.46392/kjge.2025.19.4.205
The paper
The Korean Association of General Education, Jang, J., & Park, S. (2025). Exploring the Potential of Generative AI for Korean Speaking Assessment : Focusing on Multimodal-based Voice Interaction. Korean Journal of General Education. https://doi.org/10.46392/kjge.2025.19.4.205

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.