Exploring the Potential of Generative AI for Korean Speaking Assessment : Focusing on Multimodal-based Voice Interaction
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index; none of the 7 sources its claims cite is checked for support yet.
Abstract
This study explores the potential and limitations of GPT-4o, a multimodal generative AI equipped with real-time voice interaction, in generating responses to Korean speaking proficiency test tasks. To this end, the study collected voice-based responses to six TOPIK speaking tasks generated by GPT-4o and analyzed them both quantitatively and qualitatively based on the evaluations of 18 Korean language education experts. The results indicate that GPT-4o demonstrates high performance in grammatical accuracy and logical structure but reveals clear limitations in prosodic features such as intonation, speech rate, and pragmatic appropriateness. In particular, the use of overly advanced vocabulary and unnatural delivery was repeatedly pointed out as misaligned with actual learner proficiency levels. Nevertheless, the study found that GPT-4o holds potential as a supplementary tool for speaking test preparation, especially for modeling ideal responses, providing self-directed feedback, and enabling repeated practice in autonomous learning contexts. This research underscores the applicability of multimodal AI in Korean language education and suggests the need for future studies that compare various multimodal models and examine response patterns under more complex input conditions combining visual, contextual, and interactive elements.
Overview
CoChat Summary
TL;DR
This study examined GPT-4o’s multimodal voice interaction as a respondent to six TOPIK speaking tasks, with 18 Korean-language education experts scoring 18 AI-generated voice responses. GPT-4o produced generally high-scoring, grammatically accurate and logically structured responses, but showed limitations in naturalness, prosody, pragmatic appropriateness, and level-appropriate vocabulary.
Key findings
- Across six item types, GPT-4o scored highest on item 5 “자료 해석하기” (M=4.86, SD=0.09) and item 6 “의견 제시하기” (M=4.83, SD=0.08), and lowest on item 2 “그림 보고 역할극 수행하기” (M=4.53, SD=0.17). Repeated-measures ANOVA found significant item-type differences for items 3, 4, and 5 (e.g., item 4 F=34.22, p=.0), borderline for item 1 (p=.0861), and non-significant for items 2 and 6 (p=.4503, p=.9853).
- By scoring criterion, task performance/discourse construction (criteria 1–3) averaged 4.72–4.78, and language use (criteria 4–6) mostly ≥4.70; accuracy (criterion 6) had the highest mean (4.79, SD=0.18) but was borderline (p=.0795). Delivery/other criteria were lower: pronunciation/intonation (criterion 7) M=4.61, SD=0.21; proficiency-level judgment (criterion 9) M=4.66, SD=0.19. Eight of nine criteria showed significant item-type differences (p<.05), with criteria 1–5 at p<.01.
- Expert qualitative feedback praised grammatical accuracy and logical structure, especially for advanced items 5 and 6, but repeatedly noted fast speech, artificial intonation, insufficient breath control, pronunciation errors (e.g., 한옥→한욱, 갈등→칼등), and advanced vocabulary/structures beyond the target learner level. Reliability was high: Cronbach’s α ≥ .96 for all evaluation items, and scorer deviations were within 1 point, mostly ≤0.2.
Methods
- Design: GPT-4o was treated as an AI test-taker using its multimodal real-time voice interaction via smartphone ChatGPT app; six official TOPIK speaking item types were used, with three responses per item (18 total). Voice, image, and text inputs were provided as appropriate; prompts specified role, proficiency level, response timing, and sentence limits.
- Sample/data: 18 Korean-language education experts (15 PhD holders, 3 PhD candidates; mean 10.5 years experience, range 5–18; experienced in TOPIK/SKA item writing/scoring or related research) independently scored the 18 AI voice responses using nine 5-point criteria plus one open-ended item, based on TOPIK speaking evaluation elements (NIIED, 2023), via TeacherMade online scoring.
- Analysis: Descriptive statistics, repeated-measures ANOVA (SPSS 28.0) for item-type and criterion differences, Cronbach’s α for inter-rater reliability, and qualitative summary of expert feedback.
Limitations
- The paper states it studied only one model, GPT-4o; it calls for future comparison of various multimodal AI models, tracking changes with technology, examining how prompts change responses, and analyzing complex input environments combining voice, images, and contextual variables.
- Evident from the text: the study used a small stimulus set (6 items × 3 responses = 18 AI responses) and no human learner baseline, so findings are tied to this prompt design and scoring setup; the authors also note that current TTS still cannot fully implement human-like suprasegmental features such as subtle intonation, stress, and pauses.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 5 Not checked · 51 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“이와 같은 일관성을 보다 명확히 검증하기 위해 Cronbach's α 계수를 산출한 결과, 모든 평가 항목에서 α 값이 0.96 이상으로 나타났다.”
§ 한국어 말하기 평가 대비를 위한 생성형 AI의 활용 가능성 탐색 > 3.3.3. 발화 전달력
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“반복측정 분산분석 결과, 3, 4, 5번은 는 모두 유의수준 p<.05 이하로 통계적으로 유의미한 차이를 나타냈다.”
§ 한국어 말하기 평가 대비를 위한 생성형 AI의 활용 가능성 탐색 > 3.3.3. 발화 전달력
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“또한 표준편차 역시 2번 문항에서 채점자 간 점수 차가 비교적 크게 나타났음을 알 수 있다.”
§ 한국어 말하기 평가 대비를 위한 생성형 AI의 활용 가능성 탐색 > 3.3.3. 발화 전달력
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“채점 기준별로 살펴보았을 때도 GPT-4o가 생성한 말하 기 응답에 대해 전문가들이 부여한 평균 점수는 전반적으 로 높은 수준을 보였다.”
§ 한국어 말하기 평가 대비를 위한 생성형 AI의 활용 가능성 탐색 > 3.3.3. 발화 전달력
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“한편, 발화 전달력 및 기타 질문(기준 7-9) 항목에서는 평균 점수가 다른 항목에 비해 다소 낮은 경향을 보였다.”
§ 한국어 말하기 평가 대비를 위한 생성형 AI의 활용 가능성 탐색 > 3.3.3. 발화 전달력
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
11 more claims
Create a free account to see all 56 claims and to verify the 6 references still open.
- It may state two separate claims as one. 33
- The quote only partly states this claim. 1
- Probably not a substantial claim. 13
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 한국어 말하기 평가 대비를 위한 생성형 AI의 활용 가능성 탐색 > 3.3.3. 발화 전달력
“이들 항목은 표준편차가 0.20 이하로 낮게 나타나”
§ 한국어 말하기 평가 대비를 위한 생성형 AI의 활용 가능성 탐색 > 3.3.3. 발화 전달력
“기준 1 4.76 0.21 5.23 0.0021 유의함”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 한국어 말하기 평가 대비를 위한 생성형 AI의 활용 가능성 탐색 > 3.3.3. 발화 전달력
“'과제 수행 및 담화 구성(기준 1-3)'에 해당하는 항목들이 평균 4.72-4.78로 가장 높은(”
§ 한국어 말하기 평가 대비를 위한 생성형 AI의 활용 가능성 탐색 > 3.3.3. 발화 전달력
“기준 4 4.79 0.15 3.94 0.0082 유의함 기준 5 4.78 0.17 3.25 0.0186 유의함 기준 6 4.79 0.18 2.12 0.0795 경계”
Citations
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 31,709
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Mayer, R. E. (2009) · doi:10.1017/cbo9780511811678 · Cited by 7,800
Mayer, R. (2020) · doi:10.1017/9781316941355 · Cited by 7,777
Ten myths of multimodal interaction
Oviatt, S. (1999) · Communications of the ACM · doi:10.1145/319382.319398 · Cited by 689
Lee, J., Jung, J.-K., & Kim, H. (2023) · 국제어문 · doi:10.31147/iall.99.16 · Cited by 3
The Design Direction for an Automated AI-based KFL Essay Scoring Model
Baek, J. P. (2023) · Urimal Society · doi:10.35902/urm.2023.73.129 · Cited by 2
Exploring Automated Feedback Methods to Teach and Learn Writing
Kwon, T., & Park, H. C. (2023) · Korean Language Education · doi:10.29401/kle.181.3 · Cited by 1
Yang, S., & Younghwan, B. (2023) · Baedalmal · doi:10.52636/kl.73.5 · Cited by 1
Kim, J.-Y. (2024) · The Journal of Yeongju Language & Literature · doi:10.30774/yjll.2024.10.58.355 · Cited by 1
Research works printed but not linked: 12
- Baek, 2023, Design direction of AI-based KFL writing automatic scoring model, 129
- Choi, 2025, Development and validation of an evaluation tool for generative AI-based Korean writing feedback, 227
- Kim, 2024a, Developing and exploring the effectiveness of a ChatGPT-based TOPIK speaking practice model as a learning feedback tool, 1
- Kim, 2024b, Development and application of a ChatGPT-based feedback model for TOPIK speaking assessment, 1
- Kim, 2024c, Designing and validating feedback prompts for question 5 in the TOPIK speaking assessment:A case study using generative AI, 161
- Kwon, 2025, A study on the quality evaluation of AI-based Korean writing automatic feedback, 205
- Kwon, 2023, Exploring automatic feedback methods for writing instruction, 73
- Lee, 2025, Exploring the possibility of Korean writing assessment using LLMs:A comparison of GPT-4o and GPT-4.5, 281
- Lee, 2023, Feature selection for automatic Korean writing scoring:Focusing on content and structure, 433
- Lee, 2025, A study on the applicability of ChatGPT for Korean writing feedback, 527
- Park, 2002, Analysis of variance and repeated measures data
- Park, 2024, AI knowledge for non-majors
Cited by
AI-Generated Voice, Synthetic Speech, and Voice Cloning: Scoping Review with ☸️SAIMSARA
SAIMSARA (2026) · SAIMSARA Journal · doi:10.62487/saimsara3635922a
Metadata
- Authors
- The Korean Association of General Education, Jang, J., & Park, S.
- Affiliations
- Ewha Womans University (KR)
- ORCID
- Park, Sooyeon
- Venue
- Korean Journal of General Education
- Published
- 2025-08-31
- Volume
- 19
- Issue
- 4
- Pages
- 205-216
- OpenAlex
- W4414030662
- Cited by
- 1
- References
- 10
- Citations by year
- 2026: 1
- FWCI
- 0.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- creativecommons.org/licenses/by-nc/4.0/ (tdm)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- GPT-4o, multimodal generative AI, TOPIK, voice interaction, prosodic features, pragmatic appropriateness, speaking test preparation, self-directed learning, Korean language education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Exploring the Potential of Generative AI for Korean Speaking Assessment : Focusing on Multimodal-based Voice Interaction. https://app.cochat.ai/papers/10.46392/kjge.2025.19.4.205
- The paper
- The Korean Association of General Education, Jang, J., & Park, S. (2025). Exploring the Potential of Generative AI for Korean Speaking Assessment : Focusing on Multimodal-based Voice Interaction. Korean Journal of General Education. https://doi.org/10.46392/kjge.2025.19.4.205
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.