CoChat
Sign Up
Cited by 183Artificial Intelligence in EducationEnglish Language Learning and TeachingEFL/ESL Teaching and Learning

Assessing the Usability of ChatGPT for Formal English Language Learning

Open access
Shaikh, Sarang, Yayilgan, Sule Yildirim, Klimova, Blanka, Pikhart, Marcel·2023·European Journal of Investigation in Health, Psychology and Education
DOI 10.3390/ejihpe13090140PMC PMC10528190
Open this paper in CoChat — free accountOpen PDF

Abstract

Recently, the emerging technologies have been constantly shaping the education domain, especially the use of artificial intelligence (AI) for language learning, which has attracted significant attention. Many of the AI tools are being used for learning foreign languages, in both formal and informal ways. There are many studies that have explored the potential of the recent technology “ChatGPT” for education and learning languages, but none of the existing studies have conducted any exploratory study for assessing the usability of ChatGPT. This paper conducts an assessment for usability of ChatGPT for formal English language learning. The study uses a standard questionnaire-based approach to ask participants about their feedback for usefulness and effectiveness of ChatGPT. The participants were asked for their feedback after performing series of tasks related to formal English language learning with ChatGPT. A variety of student participants were selected for this study with diverse English language proficiency levels, education levels, and nationalities. The quantitative analysis of the participant responses shed light on their experience with regards to the usability of ChatGPT for performing different English language learning tasks such as conversation, writing, grammar, and vocabulary. The findings from this study are quite promising and indicate that ChatGPT is an effective tool to be used for formal English language learning. Overall, this study contributes to the fast-growing research domain on using emerging technologies for formal English language learning by conducting in-depth assessment of usability for ChatGPT in formal English language learning.

AI snapshot

Written from the paper’s full text.

TL;DR

This exploratory study assessed ChatGPT’s usability for formal English language learning with 10 NTNU students who completed four language tasks and post-task USE/SUS questionnaires. Participants rated ChatGPT favorably across conversation, writing, grammar, and vocabulary, and usability scores indicated positive perceptions, especially ease of use.

Key findings

  • Mean task-performance ratings out of 5 were 3.90 for conversation, 4.10 for writing, 4.20 for grammar, and 4.30 for vocabulary; minimum/maximum ratings were mostly 2–5, except grammar had a minimum of 3.
  • Mean usability scores were 29.90/50 for SUS, 30.80/40 for usefulness, 43.30/55 for ease of use, 17.50/20 for ease of learning, and 29.70/35 for satisfaction.
  • Ease of use was the most popular USE aspect; participants perceived ChatGPT as user-friendly, accessible, and very usable, with positive satisfaction.
  • The 31–34 age group had the highest average scores across tasks and usability aspects; female participants rated vocabulary, grammar, and writing higher, while male participants rated conversation higher.

Methods

  • Ten participants were recruited from students at NTNU, Gjøvik; the sample was 80% male and 20% female, ages 26–38 with a mean age of 30, and was evenly split between master’s and postgraduate education levels.
  • In a controlled meeting-room setting, participants used their own laptops and email accounts to complete four 15-minute modules with ChatGPT—conversation, writing, grammar, and vocabulary—for a total of 1 hour, then completed a post-task questionnaire.
  • The questionnaire included demographics, self-rated English language skills, the 30-item USE questionnaire, the 10-item SUS questionnaire, and task-satisfaction ratings; analysis used descriptive statistics including mean, standard deviation, skewness, and kurtosis, linked to task performance, English skill level, and demographics.

Limitations

  • The paper states that the evaluation was conducted only by a very specific group of university students who study computer science, so their views may differ from students majoring in English as a second language, and it calls for a similar study in a different cultural context where English as a second language is not at a high level.
  • Evident from the text: the sample was very small (10 participants), 80% male, and self-selected; exposure was short (1 hour), outcomes were self-reported, and there was no control/comparison group or objective measure of language gains, limiting generalizability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/ejihpe13090140

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 10 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    For the writing task, both education level groups have the same average rating score of 4 out of 5.

    “Finally, for the writing task performed with ChatGPT, both education level groups have the same average rating score of 4 out of 5.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Speaking skill has the lowest average scores in all language skills for both master's and postgraduate groups.

    “The lowest average scores in all the language skills seems to be for the speaking skill for both groups.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The male group dominated the writing skill level with an average score of 3.87, compared to the female group.

    “The remaining writing skill level is dominated by the male group as compared to the female group with an average score of 3.87.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' self-ratings accurately reflect their English language skill levels.
  • Findingown data

    For the conversation task, the male group has a higher average performance rating than the female group.

    “For the conversation task, the male group has more average performance rating as compared to the female group.”
    Quote in the paperNot a substantial claim
  • Findingown data

    For the conversation and vocabulary tasks, the master's group has higher average rating scores ranging from 4.4 to 4.6 compared to the postgraduate group.

    “For the conversation and vocabulary tasks, the participants with master’s have higher average rating scores ranging from 4.4 to 4.6 as compared to the postgraduate group.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 39
  • Classified as probably not a substantial claim. 36

  • May combine two claims.

    Findingown data

    Age group 35-38 has the lowest average performance ratings, with all task categories having the same average rating of 4.

    “The lowest average scores’ age group is the 35–38, which shows slightly different patterns in the average scores. All the task categories have the same average performance rating of 4.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Mean self-rated English skill for Writing was 3.80 out of 5.

    “The mean score of the participants’ English language skill level for each skill was found to be 3.50 (Speaking), 3.70 (Listening), 4.30 (Reading), and 3.80 (Writing) out of 5.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The SUS aspect had overall skewness of 0.15, showing positive skewness but almost symmetrical.

    “The SUS aspect has overall skewness of 0.15, which shows a positive skewness but almost equal to symmetrical.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Participant age distribution: 60% in 26-30, 30% in 31-34, 10% in 35-38.

    “Most of the participants lie within the 26–30 age group (60%), followed by the 31–34 age group (30%) and the 35–38 age group (10%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Age group 26-30 contains 60% of participants and has maximum average performance rating for vocabulary (4.17), followed by grammar (4.17), writing (4), and conversation (3.83).

    “The age group 26–30, which contains 60% of the participants, also has the maximum average score for the vocabulary (4.17) task followed by the grammar (4.17), writing (4), and conversation (3.83) tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The female group has higher average ChatGPT performance ratings than the male group for vocabulary, grammar, and writing, with average scores of 4.5 for all three.

    “The female group is higher in terms of average performance rating from ChatGPT for three task categories—vocabulary, grammar, and writing—with average scores of 4.5 for all the task categories.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The lowest average rating score was from the Computer Science department for the conversation task.

    “However, the lowest average rating score was from the participants of the department of Computer Science for the conversation task performed with ChatGPT.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Other usability aspects had negative skewness: -0.52 for ease of use (moderate), -1.12 for usefulness, -1.00 for ease of learning, and -1.07 for satisfaction (high).

    “For the rest of the aspects, the skewness is negative skewed with −0.52 (ease of use) as moderate and −1.12 (usefulness), −1.00 (ease of learning), and −1.07 (satisfaction) as high skewed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The kurtosis distribution for all five usability aspects tends more towards a flatter curve compared to normal or high peak curves.

    “The kurtosis distribution for all the five aspects of the usability tends to be more towards the flatter curve as compared to the normal or high peak curves.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Age group 31-34 has the highest average performance ratings for all task categories, followed by 26-30 and 35-38.

    “The age group 31–34 has the highest average scores of performance rating for all the task categories followed by the 26–30, 35–38 age groups, respectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In age group 31-34, the highest average score is ease of use (49.7/55), followed by usefulness (32/40), satisfaction (31.7/35), SUS (31.3/50), and ease of learning (18.3/20).

    “In the age group 31–34, the highest average score of the participants is for the ease of use aspect (49.7 out of 55) followed by usefulness (32 out of 40), satisfaction (31.7 out of 35), SUS score (31.3 out of 50), and ease of learning (18.3 out of 20) aspects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Mean self-rated English skill for Listening was 3.70 out of 5.

    “The mean score of the participants’ English language skill level for each skill was found to be 3.50 (Speaking), 3.70 (Listening), 4.30 (Reading), and 3.80 (Writing) out of 5.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Both gender groups have the same average ease of learning score of 17.5/20.

    “For the ease of learning aspect, both gender groups have the same average score of 17.5 out of 20.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The mean ChatGPT performance rating was 3.90 for conversation, 4.10 for writing, 4.20 for grammar, and 4.30 for vocabulary out of 5.

    “The mean score for ChatGPT performance on each of the task categories was found to be 3.90 (Conversation), 4.10 (Writing), 4.20 (Grammar), and 4.30 (Vocabulary) out of 5 as identified from the participants ratings.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Participant age ranged from 26 to 38 years, with a mean age of 30 years.

    “The age distribution for the participants is ranging from 26 to 38 years, with a mean age value of 30 years.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    For the ease of learning aspect, participants with the highest rating of 5 for all tasks also gave the highest rating score for this usability aspect.

    “Next, for the ease of learning aspect, the participants with highest rating of 5 for all the tasks performed with ChatGPT have also given the highest rating score for this usability aspect.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the age group 26-30, the maximum average score was in reading (4), followed by writing (3.67), listening (3.50), and speaking (3.33).

    “The age-group 26–30, which consists of the 60% of the participants, also has the maximum average score for reading (4) followed by the writing (3.67), listening (3.50), and speaking (3.33).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' self-ratings accurately reflect their English language skill levels.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Mean self-rated English skill for Speaking was 3.50 out of 5.

    “The mean score of the participants’ English language skill level for each skill was found to be 3.50 (Speaking), 3.70 (Listening), 4.30 (Reading), and 3.80 (Writing) out of 5.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The minimum and maximum performance ratings for all tasks were 2 and 5, except the grammar task minimum was 3.

    “Furthermore, the minimum and maximum rating for the performance for all the tasks was found to be 2 and 5, respectively, with an exception where the minimum value for the grammar task was found to be 3.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The distribution of speaking skill level was completely symmetric (skewness 0), indicating a normal distribution.

    “The skewness distribution of the speaking level is completely symmetric as the value is 0, which shows a normal distribution of participants for the rating scale from 1 to 5.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Nationality distribution: 25% Norwegian, followed by participants from Italy, Georgia, Nepal, and Pakistan.

    “Out of total 10, 25% of the participants were Norwegians, followed by participants from Italy, Georgia, Nepal, and Pakistan.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Age group 26-30 contains 60% of participants and has the highest average score for the ease of use aspect.

    “The age-group 26–30, which contains 60% of the participants, also has the highest average score for the ease of use aspect.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Department distribution: 50% Computer Science, 20% Civil Engineering, 20% Information Security, 10% Construction Engineering.

    “Most of the participants/students belong to the Computer Science department (50%), followed by the Civil Engineering, Information Security departments (20% each), and the Construction Engineering department (10%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For age group 31-34, the highest average performance rating is vocabulary (5), followed by grammar (4.33), writing (4.33), and conversation (4).

    “For the age group 31–34, the highest score for participants is in the vocabulary task (5) followed by the grammar (4.33), writing (4.33), and conversation (4) tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the age group 31-34, the highest score was in reading (5), followed by writing (4.33), listening (4.33), and speaking (4).

    “For the age group 31–34, the highest score for participants is in reading (5) followed by writing (4.33), listening (4.33), and speaking (4).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' self-ratings accurately reflect their English language skill levels.

  • May combine two claims.

    Findingown data

    The age group 31-34 had the highest average scores of all English language skill levels, followed by the 26-30 and 35-38 age groups, respectively.

    “The age group 31–34 has the highest average scores of all the English language skill levels followed by the 26–30, 35–38 age groups, respectively.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' self-ratings accurately reflect their English language skill levels.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Standard deviations of self-rated English skills were 0.84 for speaking, 0.94 for listening, 0.94 for reading, and 0.91 for writing.

    “Overall, the SD was found to be 0.84, 0.94, 0.94, and 0.91 for speaking, listening, reading, and writing language skills, respectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Age group 31-34 has the highest average scores for all five usability aspects, followed by 26-30 and 35-38.

    “The age group 31–34 has the highest average scores for all the five aspects followed by the 26–30 and 35–38 age-groups, respectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The standard deviations for task ratings were 0.99 for conversation, 0.99 for writing, 0.63 for grammar, and 0.94 for vocabulary.

    “Overall, the SD was found to be 0.99, 0.99, 0.63, and 0.94 for conversation, writing, grammar, and vocabulary, respectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Mean self-rated English skill for Reading was 4.30 out of 5.

    “The mean score of the participants’ English language skill level for each skill was found to be 3.50 (Speaking), 3.70 (Listening), 4.30 (Reading), and 3.80 (Writing) out of 5.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The female group dominated the average scores for speaking, listening, and reading with average scores of 3.5, 4, and 4.5, respectively, compared to the male group.

    “The female group is dominating the average scores for three English language skills, speaking, listening, and reading with an average score of 3.5, 4, and 4.5, respectively as compared to the male group.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' self-ratings accurately reflect their English language skill levels.

  • May combine two claims.

    Findingown data

    Overall mean usability feedback scores were 29.90/50 for SUS, 30.80/40 for usefulness, 43.30/55 for ease of use, 17.50/20 for ease of learning, and 29.70/35 for satisfaction.

    “The overall mean of sum of participants’ feedback scores for using ChatGPT in terms of five aspects were found to be 29.90 out of 50 for SUS, 30.80 out of 40 for usefulness, 43.30 out of 55 for ease of use, 17.50 out of 20 for ease of learning, and 29.70 out of 35 for satisfaction.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Summing Likert-scale item scores for each participant yields a meaningful base score for each usability aspect.

  • May combine two claims.

    Findingown data

    The skewness values for task ratings were -0.61 for conversation, -1.08 for writing, -0.13 for grammar, and -1.71 for vocabulary, and do not show a normal distribution among participants.

    “The skewness distribution of all the task categories, conversation (−0.61), writing (−1.08), grammar (−0.13) and vocabulary (−1.71), do not show a normal distribution among the participants.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The standard deviations for usability aspects were 3.00 for SUS, 7.60 for ease of use, 5.35 for usefulness, 2.36 for ease of learning, and 3.88 for satisfaction.

    “The SD was found to be 3.00, 7.60, 5.35, 2.36, and 3.88 for SUS, ease of use, usefulness, ease of learning, and satisfaction, respectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Education level distribution: 50% masters, 50% postgraduate.

    “In the current education level distribution, the number of participants remained equal between masters (50%) and postgraduate (50%) education levels.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the age group 35-38, the reading level had the highest average score (4) and the rest of the English language skill levels had equal scores (3).

    “The age group that has the lowest average scores is the 35–38 group where the reading level has the highest average score (4) and rest of the English language skills levels have equal scores (3).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' self-ratings accurately reflect their English language skill levels.

  • May combine two claims.

    Findingown data

    The male group has higher average scores than the female group for SUS, usefulness, and satisfaction: 30.4/50, 31.3/40, and 29.9/35, respectively.

    “The male group is higher in terms of average score for three aspects e.g., SUS, usefulness, and satisfaction with overall scores of 30.4 out of 50, 31.3 out of 40, and 29.9 out of 35, respectively as compared to the female group.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The sample population consisted of 80% male and 20% female participants.

    “In Figure 3, the sample population consisted of 80% male participants and 20% female participants.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Minimum and maximum self-rated English skill level for all four skills were 2 and 5, respectively.

    “Furthermore, the minimum and maximum language skill level for all four skills was found to be 2 and 5, respectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The research process was reviewed and approved by the Committee for Research Ethics of the University of Hradec Kralove, Czech Republic, no. 4/2023.

    “The research process conducted for this study was carefully reviewed and approved by the Committee for Research Ethics of the University of Hradec Kralove, Czech Republic, no. 4/2023.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The tasks were designed to cover a range of language learning skills and to understand the usability of ChatGPT.

    “These tasks were designed to cover a range of language learning skills and to understand the usability of ChatGPT.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The USE questionnaire is a common tool for evaluating usability of a technical system, consisting of 30 statements on a 1-5 Likert scale, and its goal is to assess users' perceptions and experience of overall usability.

    “The USE questionnaire (https://garyperlman.com/quest/quest.cgi?form=USE) (accessed on 1 August 2023) is a common tool that is used for the evaluation of usability of a technical system. It consists of a series of 30 statements or items for which users respond on a Likert scale from 1 to 5 by showing their disagreement or agreement.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [40] Gao, M., Kortum, P., and Oswald, F. (2018, January 1–5). Psychometric evaluation of the use (usefulness, satisfaction, and ease of use) questionnaire for reliability and validity. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Philadelphia, PA, USA. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data

    The 3.53 kurtosis value indicates higher peaked concentration than normal, indicating gathering of a higher part of data points around the overall mean.

    “The value 3.53 shows a higher peaked concentration as compared to the normal distribution indicating a gathering of a higher part of data points around the overall mean.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Effectiveness of chatbots for language learning highly depends on their usability, efficiency, and effectiveness.

    “However, effectiveness of the chatbots for language learning highly depends on their usability, efficiency, and effectiveness [10].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10] Ren, 2019, Evaluation techniques for chatbot usability: A systematic mapping study, Int. J. Softw. Eng. Knowl. Eng. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    No existing study has conducted an exploratory study assessing the usability of ChatGPT.

    “none of the existing studies have conducted any exploratory study for assessing the usability of ChatGPT”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The -0.15 kurtosis value indicates a relatively flat peak/flatter distribution than normal, indicating a lighter-tailed distribution.

    “The value −0.15 shows the relatively flat peak/flatter distribution than normal distribution indicating a lighter-tailed distribution.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    All required GDPR standards regarding privacy and data protection were strictly followed.

    “All required GDPR standards regarding privacy and data protection were strictly followed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The peak of distribution for writing and grammar is neither too sharp nor too flat compared to the normal distribution.

    “This shows that the peak of distribution is neither too sharp nor too flat as compared to the normal distribution.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Data were collected confidentially to protect participant privacy and anonymity.

    “The data were collected in a confidential manner to protect participant privacy and anonymity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    No existing study has performed an exploratory study/work with the actual use of ChatGPT for language learning.

    “Although existing state-of-the-art studies have discussed the role of ChatGPT for language learning and its implications, none of them have performed any exploratory study/work with the actual use of ChatGPT for language learning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    Participants showed a moderately high level of English language competence.

    “This shows an indication of moderately high level of English language competence of the participants.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Participants' self-ratings accurately reflect their English language skill levels.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    No existing study assesses the usability of ChatGPT for language learning either formal or informal.

    “Furthermore, there exists no such study that assesses the usability of ChatGPT for language learning either formal or informal.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    The recruitment of 10 participants corresponds to the well-established usability testing methodology and follows the rule of only 5 users.

    “A total of 10 participants were recruited for this study, which corresponds to the well-established methodology, namely usability testing described in the Introductory part, and which follows the rule of only 5 users.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    The 10 participants provide a sufficient sample for the study purpose.

    “The 10 participants provide a sufficient sample for our study purpose”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The effectiveness, user experiences, satisfaction, and user engagement with emerging language learning technologies are still an open problem.

    “However, despite of the many advancements for using emerging technologies in language learning, the effectiveness, user experiences, satisfaction, and user engagement with these technologies are still an open problem.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The SUS questionnaire is a commonly used tool for measuring perceived usability, consisting of ten statements on a 1-5 Likert scale, and gives a standardized and reliable measure for assessing usability and satisfaction.

    “The SUS questionnaire (https://measuringu.com/sus/) (accessed on 1 August 2023) is again a commonly used tool for measuring the “perceived usability” of a technical system. Again, it consists of a series of ten statements or items for which participants respond on a Likert scale from 1 to 5 by showing their agreement or disagreement.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [40] Gao, M., Kortum, P., and Oswald, F. (2018, January 1–5). Psychometric evaluation of the use (usefulness, satisfaction, and ease of use) questionnaire for reliability and validity. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Philadelphia, PA, USA. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Because task rating skewness values are negative and left-skewed, most participants rated closer to 5 than 1.

    “However, since the values are more negative and left-skewed, then the majority of the participants are closer to the higher end of the scale (closer to 5) as compared to the lower end of the scale (close to 1).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The evaluation was conducted only by a very specific group of university students who study computer science; their view can be somewhat different from university students who study English as a second language as their major.

    “There are several limitations of this study related to the fact that the evaluation was conducted only by a very specific group of university students who study computer science; therefore, their view of the topic can be somewhat different from university students who, for example, study English as a second language as their major.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Students can ask questions, share thoughts, and discuss various topics, which reflects real-life dialogues and makes them motivated to learn a foreign language.

    “Students can ask questions, share their thoughts, and discuss various topics, which reflects real-life dialogues and makes them motivated to learn a foreign language (cf. [28,29]).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [28] Jeon, J., Lee, S., and Choi, S. (2023). A systematic review of research on speech-recognition chatbots for language learning: Implications for future directions in the era of large language models. Interact. Learn. Environ., 1–19. source

      Not checked in the free run.

    • cites [29] Jeon, J. (2022). Exploring AI chatbot affordances in the EFL classroom: Young learners’ experiences and perspectives. Comput. Assist. Lang. Learn., 1–26. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Using ChatGPT for English learning has big potential because learners admitted it can expand vocabulary, enhance grammatical and syntactical structures, and improve written and conversational skills.

    “In addition, as the results of this study show, using ChatGPT for English learning has a big potential since learners admitted that ChatGPT can expand their vocabulary, enhance their grammatical and syntactical structures, and thus improve their written and conversational skills [28,29,41].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [28] Jeon, J., Lee, S., and Choi, S. (2023). A systematic review of research on speech-recognition chatbots for language learning: Implications for future directions in the era of large language models. Interact. Learn. Environ., 1–19. source

      Not checked in the free run.

    • cites [29] Jeon, J. (2022). Exploring AI chatbot affordances in the EFL classroom: Young learners’ experiences and perspectives. Comput. Assist. Lang. Learn., 1–26. source

      Not checked in the free run.

    • cites [41] Barrot, 2023, Using ChatGPT for second language writing: Pitfalls and potentials, Assess. Writ. source

      Not checked in the free run.

    • Rests on an assumption: Participants' self-reported ratings accurately reflect ChatGPT's actual performance and usability for the tasks.

  • May combine two claims.

    Premisecitation

    ChatGPT can evaluate students' tasks and translate texts from one language to another.

    “Moreover, ChatGPT can evaluate students’ tasks and can translate texts from one language to another (cf. [42]).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [42] Montenegro-Rueda, M., Fernández-Cerero, J., Fernández-Batanero, J.M., and López-Meneses, E. (2023). Impact of the Implementation of ChatGPT in Education: A Systematic Review. Computers, 12. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Overall, ChatGPT can be used for developing learners' language skills, scaffolding the learning process by providing feedback, and acting as a support tool for practicing a foreign language.

    “Overall, ChatGPT can be used for developing learners’ language skills, scaffolding the learning process by providing feedback to students on their language use and performance. It seems to act as a support tool for practicing a foreign language (cf. [43]).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [43] Bin-Hady, W.R.A., Al-Kadi, A., Hazaea, A., and Ali, J.K.M. (2023). Exploring the dimensions of ChatGPT in English language learning: A global perspective. Library Hi Tech., ahead-of-print. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Participants perceived ChatGPT as user-friendly and easily accessible for language learners.

    “This means that the participants perceived ChatGPT as user-friendly and easily accessible for the language learners.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Participants' self-reported ratings accurately reflect ChatGPT's actual performance and usability for the tasks.
63 of 63

  • The paper relies on this without support.

    Participants' self-reported feedback accurately reflects the usability and effectiveness of ChatGPT for formal English language learning.

    “We developed a post-task questionnaire to ask participants for their feedback/experience regarding the usability of ChatGPT and their overall experience during the 1 h of interaction with the tool.”

  • The paper relies on this without support.

    The tasks performed in the study are representative of formal English language learning tasks.

    “These tasks were designed to cover a range of language learning skills and to understand the usability of ChatGPT.”

  • The paper relies on this without support.

    The sample of 10 students from NTNU is representative of formal English language learners in general.

    “A total of 10 participants were recruited for this study, which corresponds to the well-established methodology, namely usability testing described in the Introductory part, and which follows the rule of only 5 users.”

  • The paper relies on this without support.

    ChatGPT's performance during the study is typical of its general performance.

    “The participants interacted with the ChatGPT tool using their own laptop device and email account.”

  • The paper relies on this without support.

    Participants' self-ratings accurately reflect their English language skill levels.

    “The English language skill level of the participants was acquired in the post-experiment questionnaire form where each of the participants rated himself/herself on a scale from 1 to 5 (1 being minimum and 5 being maximum skill level).”

  • The paper relies on this without support.

    Participants' self-reported ratings accurately reflect ChatGPT's actual performance and usability for the tasks.

    “The ratings for these tasks were acquired in the post-experiment questionnaire form where each of the participants rated their experience with each of the tasks performed (i.e., conversation, writing, grammar, and vocabulary) with ChatGPT.”

  • The paper relies on this without support.

    The USE and SUS questionnaires are valid instruments for assessing ChatGPT's usability for formal English language learning.

    “Since these are the standard questionnaires to assess the usability of any technical system; hence analyzing the participants’ feedback help to understand the usability of ChatGPT for our specific study.”

  • The paper relies on this without support.

    Summing Likert-scale item scores for each participant yields a meaningful base score for each usability aspect.

    “For the SUS questionnaire, we sum up the feedback score given for each of the questions for all 10 questions. This summation is calculated against each individual participant response and used as the base feedback score.”
8 of 8

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Female speaking average cannot exceed the male average

    Minor

    The female speaking average and the overall speaking average are both 3.5. With both genders represented, the male average must also be 3.5, contrary to the claim that the female group has a higher speaking average.

  • Ease-of-use maximum changes from 55 to 50

    Minor

    The paper reports ease of use as a score out of 55, but describes a conversation-task ease-of-use score as out of 50.

  • Education-level vocabulary scores cannot produce the reported overall score

    Minor

    The postgraduate vocabulary average is stated as 4.4, while the masters average is stated to be higher. Their combined average therefore cannot be the reported 4.30.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 9

  1. Herrera, P. (2022). English and Students with Limited or Interrupted Formal Education, Springer.
  2. Malyuga, 2012, Professional language in formal and business style, Glob. J. Hum. Soc. Sci., 12, 7
  3. Bajorek, 2017, L2 pronunciation in CALL: The unrealized potential of Rosetta stone, Duolingo, Babbel, and mango languages, Issues Trends Educ. Technol., 5, 24
  4. Nushi, 2018, Babbel: A mobile language learning app, TESL Report., 51, 13
  5. Shortt, 2023, Gamification in mobile-assisted language learning: A systematic review of Duolingo literature from public release of 2012 to early 2020, Comput. Assist. Lang. Learn., 36, 517, 10.1080/09588221.2021.1933540 · DOI
  6. Reknasari, A., and Putro, N. (2020). Teacher Education and Professional Development in Industry 4.0, CRC Press.
  7. Cardoso, 2022, Google Translate as a tool for self-directed language learning, Lang. Learn. Technol., 26, 1
  8. Prathyusha, 2020, Role of gamification in language learning, Int. J. Res. Anal. Rev. (IJRAR), 7, 577
  9. Hobert, S. (2019, January 15–18). Say hello to ‘coding tutor’! design and evaluation of a chatbot-based learning system supporting students to learn to program. Proceedings of the Fortieth International Conference on Information Systems, Munich, Germany.

Cited by

Sorted by citations

Authors
Shaikh, S., Yayilgan, S. Y., Klimova, B., & Pikhart, M.
Affiliations
  • Norwegian University of Science and Technology (NO)
  • Department of Applied Linguistics, Faculty of Informatics and Management, University of Hradec Kralove, Rokitanskeho 62, 500 03 Hradec Kralove, Czech Republic
  • University of Hradec Králové (CZ)
Corresponding author
Pikhart, Marcel
ORCID
Venue
European Journal of Investigation in Health, Psychology and Education
Publisher
MDPI AG
Published
2023-09-21
Volume
13
Issue
9
Pages
1937-1960
ISSN
  • 2174-8144 (ISSN-L)
  • 2254-9625
DOI
10.3390/ejihpe13090140
PMC
PMC10528190
OpenAlex
W4386959594
Semantic Scholar
32295bf70f128711ff49bea15261bfd1d0e474d6
Cited by
183
References
33
Citations by year
2026: 38 · 2025: 74 · 2024: 62 · 2023: 9
FWCI
42.87
Type
journal article
Language
en
Open access
Licence
Locations
  • European Journal of Investigation in Health Psychology and Education (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Dialnet (Universidad de la Rioja) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Information Systems › English Language Learning and Teaching
  • Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
Keywords
ChatGPT, usability evaluation, large language models in education, AI-assisted language learning, student feedback, English writing, grammar learning, vocabulary acquisition, conversation practice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review