Exploring the Feedback Quality of an Automated Writing Evaluation System Pigai
Open accessAbstract
The study made an exploration of the feedback quality of an Automated Writing Evaluation system (AWE) Pigai, which has been widely applied in English teaching and learning in China. The study not only focused on the diagnostic precision of the feedback but also investigated the students’ perceptions of the feedback use in their daily writing practices. Taking 104 university students’ final exam essays as the research materials, the paired sample t-test was conducted to compare the mean number of errors identified by Pigai and professional teachers. It was found that Pigai feedback could not so well diagnose the essays as the human feedback given by the experienced teachers, however, it was quite competent in identifying lexical errors. The analysis of students’ perceptions indicated that most students thought Pigai feedback was multi-functional, but it was inadequate in identifying the collocation errors and giving suggestions in syntactic use. The implications and limitations of the study were discussed at the end of the paper.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT could be used to grade assignments and provide feedback to students in real time.
“ChatGPT could be used to grade assignments and provide feedback to students in realtime, allowing for a more efficient and personalised learning experience (J.Gao, 2021;Roscoe et al., 2017;Zawacki-Richter et al., 2019).”
From Chatting and cheating: Ensuring academic integrity in the era of ChatGPT (2024) · cites this paper as J.Gao, 2021
Section 1.1 describes Pigai automatically scoring English essays and giving students specific corrective feedback; the Introduction also describes automated writing evaluation as efficient. The paper does not study ChatGPT or establish that ChatGPT can grade assignments and provide real-time feedback for a more personalised learning experience. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0
Attali, Y., & Burstein, J. C. (2004) · ETS Research Report Series · doi:10.1002/j.2333-8504.2004.tb01972.x · Cited by 630 · see citing works
(2006) · The Journal of Technology, Learning and Assessment (Boston College) · Cited by 424 · see citing works
Chen, C.-F. E., & Cheng, W.-Y. E. C. (2008) · Language Learning & Technology · doi:10.64152/10125/44145 · Cited by 304 · see citing works
Automated Writing Assessment in the Classroom
Warschauer, M., & Grimes, D. (2008) · Pedagogies: An International Journal · doi:10.1080/15544800701771580 · Cited by 243 · see citing works
Automated Grammatical Error Detection for Language Learners
Leacock, C., Chodorow, M. S., Gamon, M., & Tetreault, J. (2010) · Synthesis lectures on human language technologies · doi:10.2200/s00275ed1v01y201006hlt009 · Cited by 214 · see citing works
Dikli, S., & Bleyle, S. (2014) · Assessing Writing · doi:10.1016/j.asw.2014.03.006 · Cited by 193 · see citing works
Complementing human judgment of essays written by English language learners with e-rater ® scoring
Enright, M. K., & Quinlan, T. (2010) · Language Testing · doi:10.1177/0265532210363144 · Cited by 114 · see citing works
Using automated writing evaluation to reduce grammar errors in writing
Liao, H.-C. (2015) · ELT Journal · doi:10.1093/elt/ccv058 · Cited by 81 · see citing works
Automated Essay Evaluation for English Language Learners:A Case Study of<i>MY Access</i>
Hoang, G. T. L., & Kunnan, A. J. (2016) · Language Assessment Quarterly · doi:10.1080/15434303.2016.1230121 · Cited by 68 · see citing works
Reilly, E. D., Stafford, R. E., Williams, K. M., & Corliss, S. B. (2014) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v15i5.1857 · Cited by 55 · see citing works
Students' Experiences with an Automated Essay Scorer.
Scharber, C., Dexter, S. L., & Riedel, E. (2008) · The Journal of Technology, Learning and Assessment (Boston College) · Cited by 26 · see citing works
A Study on the Application of Automatic Scoring and Feedback System in College English Writing
Xiao-xiao, L. (2018) · International Journal of Emerging Technologies in Learning (iJET) · doi:10.3991/ijet.v13i03.8386 · Cited by 21 · see citing works
An Evaluation of Online and Automated English Writing Assistants: Collocations and Idioms Checkers
Grami, G. M. A. (2020) · International Journal of Emerging Technologies in Learning (iJET) · doi:10.3991/ijet.v15i04.11782 · Cited by 20 · see citing works
THE IMPACT OF SAMPLING APPROACH ON POPULATION INVARIANCE IN AUTOMATED SCORING OF ESSAYS
Zhang, M. (2013) · ETS Research Report Series · doi:10.1002/j.2333-8504.2013.tb02325.x · Cited by 18 · see citing works
A Survey of EFL College Learners’ Perceptions of an On-Line Writing Program
He, H. (2016) · International Journal of Emerging Technologies in Learning (iJET) · doi:10.3991/ijet.v11i04.5459 · Cited by 10 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT
Cotton, D., Cotton, P., & Shipway, J. R. (2023) · doi:10.35542/osf.io/mrz8h · Cited by 3,079 · see citing works
Chatting and cheating: Ensuring academic integrity in the era of ChatGPT
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2190148 · Cited by 3,087 · see citing works
Ali, O., Murray, P. A., Momin, M., Dwivedi, Y. K., & Malik, T. (2024) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.123076 · Cited by 317 · see citing works
Kohnke, L., Moorhouse, B. L., & Zou, D. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100156 · Cited by 309 · see citing works
Educational Applications of the ChatGPT AI System: A Systematic Review Research
İPEK, Z. H., Gözüm, A. İ. C., Papadakis, S., & Kalogiannakis, M. (2023) · Educational Process International Journal · doi:10.22521/edupij.2023.123.2 · Cited by 209 · see citing works
Jiang, R. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1049401 · Cited by 181 · see citing works
“A large and growing body of literature has suggested that AI can benefit language teaching and learning (Gao, 2021; Pikhart, 2021; Klimova et al., 2022), and ameliorate the quality of online EFL learning especially during the COVID-19 pandemic (Zitouni, 2022).”
BackgroundMethodInfluentialWang, Y., Derakhshan, A., & Ghiasvand, F. (2025) · System · doi:10.1016/j.system.2025.103791 · Cited by 199 · see citing works
Yang, L., & Zhao, S. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108337 · Cited by 164 · see citing works
Ding, L., & Zou, D. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12402-3 · Cited by 122 · see citing works
Gustilo, L., Ong, E., & Lapinid, M. R. (2024) · International Journal for Educational Integrity · doi:10.1007/s40979-024-00153-8 · Cited by 89 · see citing works
The effectiveness of automated writing evaluation in EFL/ESL writing: a three-level meta-analysis
Ngo, T. T.-N., Chen, H. H.-J., & Lai, K. K.-W. (2022) · Interactive Learning Environments · doi:10.1080/10494820.2022.2096642 · Cited by 108 · see citing works
The knowledge and innovation challenges of ChatGPT: A scoping review
Ali, O., Murray, P., Momin, M., & Al–Anzi, F. S. (2023) · Technology in Society · doi:10.1016/j.techsoc.2023.102402 · Cited by 79 · see citing works
Du, Y., & Gao, H. (2022) · Education and Information Technologies · doi:10.1007/s10639-022-11001-y · Cited by 89 · see citing works
The impact of integrating ChatGPT with teachers’ feedback on EFL writing skills
Asadi, M., Ebadi, S., & Mohammadi, L. (2025) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2025.101766 · Cited by 101 · see citing works
Yi, M., & Chen, M. (2024) · BMC Psychology · doi:10.1186/s40359-024-02248-w · Cited by 88 · see citing works
Utilizing Artificial Intelligence Technologies in Saudi EFL Tertiary Level Classrooms
AbdAlgane, M., & Othman, K. A. J. (2023) · Journal of Intercultural Communication · doi:10.36923/jicc.v23i1.124 · Cited by 72 · see citing works
Enhancing students’ learning experience in mathematics class through ChatGPT
Gouia‐Zarrad, R., & Gunn, C. L. (2024) · International Electronic Journal of Mathematics Education · doi:10.29333/iejme/14614 · Cited by 62 · see citing works
Chinese University Students’ Attitudes and Perceptions in Learning English Using ChatGPT
Liu, B. (2023) · International Journal of Education and Humanities · doi:10.58557/(ijeh).v3i2.145 · Cited by 40 · see citing works
Ding, L., Zou, D., & Kohnke, L. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13775-3 · Cited by 50 · see citing works
Unpacking the Factors Shaping TESOL Teachers' GenAI Literacy From an Ecological Perspective
Yang, L., & Derakhshan, A. (2026) · TESOL Quarterly · doi:10.1002/tesq.70095 · Cited by 29 · see citing works
George-Reyes, C. E., Vilhunen, E., Martínez, R. A., & López-Caudana, E. O. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1378564 · Cited by 28 · see citing works
Hoàng, N. H. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13451-6 · Cited by 23 · see citing works
Yu-peng, L., & Yu, Z. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03861-1 · Cited by 23 · see citing works
An, X., Bai, J., Li, Y., Zhou, Y., Chai, C. S., & Hong, J. (2025) · The Asia-Pacific Education Researcher · doi:10.1007/s40299-025-01045-2 · Cited by 21 · see citing works
Automated Writing Evaluation in EFL Contexts
Aldosemani, T. I., Assalahi, H. M., Lhothali, A., & Albsisi, M. (2023) · International Journal of Computer-Assisted Language Learning and Teaching · doi:10.4018/ijcallt.329962 · Cited by 27 · see citing works
Predictors of middle school students’ perceptions of automated writing evaluation
Wilson, J., Zhang, F., Palermo, C., Cordero, T. M. C., Myers, M. C., Eacker, H., Potter, A., & Coles, J. (2023) · Computers & Education · doi:10.1016/j.compedu.2023.104985 · Cited by 17 · see citing works
Advancing autonomy through lifelong learning: a survey of autonomous intelligent systems
Zhu, D., Bu, Q., Zhu, Z., Zhang, Y., & Wang, Z. (2024) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2024.1385778 · Cited by 14 · see citing works
Li, W., Kau, K., & SHIUNG, Y.-J. (2023) · SAGE Open · doi:10.1177/21582440231209087 · Cited by 11 · see citing works
Krange, I., Segaran, M., Gamlem, S. M., Moltudal, S., & Engeness, I. (2023) · Interaction Design and Architecture(s) · doi:10.55612/s-5002-059-001 · Cited by 7 · see citing works
Zhu, S., Li, Q. H., Yao, Y., Li, J., & Zhu, X. (2025) · Assessing Writing · doi:10.1016/j.asw.2025.100960 · Cited by 12 · see citing works
Yang, G., Zhou, W., Zhou, H., Li, J., Chen, X., & Tu, Y. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11905-3 · Cited by 8 · see citing works
Students’ perceptions of automated writing evaluation in autonomous learning
Wang, X. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13378-y · Cited by 7 · see citing works
Zhao, J., Li, Y., & Feng, W. (2023) · International Journal of Emerging Technologies in Learning (iJET) · doi:10.3991/ijet.v18i04.38241 · Cited by 7 · see citing works
Ilic, P., & Carr, N. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343337 · Cited by 6 · see citing works
How AI shapes self-regulated learning in foreign language education
Hou, F., & Zhou, C. (2025) · Discover Computing · doi:10.1007/s10791-025-09801-0 · Cited by 13 · see citing works
Beyond error correction: lower-proficiency EFL learners’ engagement with Grammarly
Murtisari, E. T., Januardi, J. I., Bonar, G., & Kurniawan, D. (2025) · Language Awareness · doi:10.1080/09658416.2025.2585004 · Cited by 5 · see citing works
Impact of prompt sophistication on ChatGPT’s output for automated written corrective feedback
Luo, N., Wang, Y., Zhang, Z. (., Zhou, Y., & Zhao, R. (2025) · ReCALL · doi:10.1017/s095834402510044x · Cited by 5 · see citing works
Marciniak, R., & Baksa, M. (2024) · Educatio · doi:10.1556/2063.32.2023.4.4 · Cited by 6 · see citing works
English Writing Feedback Based on Online Automatic Evaluation in the Era of Big Data
Li, J. (2022) · Mobile Information Systems · doi:10.1155/2022/9884273 · Cited by 5 · see citing works
RetractedDeveloping an Artificial Neural Network-based model for predicting EFL achievement level
Bouzayenne, A., & Riadh, H. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20295-8 · Cited by 5 · see citing works
Li, L. L., Mao, G., Zhang, L., & Ying, J. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13849-2 · Cited by 4 · see citing works
Osunbunmi, I. S., Cutler, S. L., Dansu, V., Brijmohan, Y., Bamidele, B., Udosen, A. N., Arinze, L., Oje, A. V., Moyaki, D., Hicks, M., et al. (2024) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--47041 · Cited by 6 · see citing works
Segaran, M. K., Engeness, I., & Gamlem, S. M. (2025) · Computer Assisted Language Learning · doi:10.1080/09588221.2025.2539154 · Cited by 4 · see citing works
Automated Writing Evaluation: Users’ Perception and Expectations
UiTM Cawangan Terengganu, D. C., & Mat, A. C. (2024) · International Journal of Information and Education Technology · doi:10.18178/ijiet.2024.14.2.2039 · Cited by 4 · see citing works
Review of Feedback in Automated Essay ScoringPreprint
Jong, Y.-J., Kim, Y.-J., & Ri, O.-C. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4462105 · Cited by 4 · see citing works
Inteligencia artificial e innovación educativa
Dopazo, P. (2023) · REVISTA ELETRÔNICA PESQUISEDUCA · doi:10.58422/repesq.2023.e1521 · Cited by 3 · see citing works
Cai, B., He, Z., Fu, H. C., Zheng, Y., & Song, Y. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3547321 · Cited by 4 · see citing works
Muththamizh Selvi, S. I., Thennavan, S., Kumar, S. K., Hameed, N. S., Vijayakumar, S., & Raj, I. I. (2025) · International Conference on Intelligent Systems, Advanced Computing and Communication (ISACC) · doi:10.1109/isacc65211.2025.10969303 · Cited by 2 · see citing works
Mariappan, R., Tan, K. H., Yang, J., Jian, C., & Chang, P. K. (2022) · Sustainability · doi:10.3390/su142315649 · Cited by 2 · see citing works
Ethically Utilizing GenAI Tools to Alleviate Challenges in Conventional Feedback Provision
Zainurrahman, Z., Purnawarman, P., & Muslim, A. B. (2024) · Journal of Academic Ethics · doi:10.1007/s10805-024-09578-9 · Cited by 3 · see citing works
Mesterséges intelligencia, civil szervezetek és a felsőoktatás
Kőkuti, T., & Maczó, E. C. (2024) · Civil Szemle · doi:10.62560/csz.2024.05.7 · Cited by 2 · see citing works
El impacto de la Inteligencia Artificial generativa: clínica jurídica e innovación docente
Fraguío, M. P. D. (2024) · Revista Jurídica de Investigación e Innovación Educativa (REJIE Nueva Época) · doi:10.24310/rejie.30.2024.18080 · Cited by 1 · see citing works
AI-Driven Personalization in English as a Foreign Language (EFL)
Nwanakwaugwu, A. C., Obi-Akwari, O. E., Ojaide, E., Emakporuena, D., Kwakpovwe, P., & Igbokwe, M. M. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-0969-9.ch003 · Cited by 1 · see citing works
Bibliometrically and systematically analyzing automated writing evaluation for English learning
Wu, T., & Yu, Z. (2023) · Forum for Linguistic Studies · doi:10.59400/fls.v5i3.1907 · Cited by 1 · see citing works
Artificial Intelligence Language Models: The Path to Development or Regression for Education?
Gonçalves, B. F., & Gonçalves, V. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-44131-8_6
Effectiveness and moderating factors of computer-mediated feedback in L2 writing: A meta-analysis
Park, S. (2025) · International Journal of English Language and Literature Studies · doi:10.55493/5019.v14i1.5328
Using ChatGPT as an AWE tool: quality, precision, and accuracy of the feedback
Hernández, A. P. (2025) · Bellaterra Journal of Teaching & Learning Language & Literature · doi:10.5565/rev/jtl3.1338
Exploring English as a Foreign Language Teachers’ Competencies in using AI for Teaching and Learning
Hussain, A. G. (2026) · Cihan University-Erbil Journal of Humanities and Social Sciences · doi:10.24086/cuejhss.v10n1y2026.pp48-55
Boudouaia, A., Khattala, A., & Bennouioua, H. (2026) · Journal of Computing in Higher Education · doi:10.1007/s12528-026-09491-7 · Cited by 1 · see citing works
The application of deep learning in the automatic scoring of English writing
Dong, Y. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01146-x
Generative Artificial Intelligence (GenAI) in Language Education: Current Trends and Perspectives
Bui, T. T. T., & Hoa, C. T. H. (2026) · Studies in computational intelligence · doi:10.1007/978-3-032-18392-7_1
Differences between human and AI scoring: A meta-analysis of english language assessments
Li, B., Qunhan, X., & Mao, C. (2026) · Scientific Reports · doi:10.1038/s41598-026-48053-w
Zhang, D., Hong, X., & Qi, Y. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101121 · Cited by 2 · see citing works
Literature Review: Analysing the Factors Affecting College Students’ Digital Language Tests
Zhang, J., & Razak, R. A. (2026) · Africa Education Review · doi:10.1080/18146627.2026.2659587
From Human Raters to AI: A Critical Review of Research on ChatGPT in Writing Assessment
Wang, Y.-M., Harmer, P., & Xue, C. (2026) · Futurity Education · doi:10.57125/fed.2026.06.12
Giri, U., Sharma, P. B., Lamoria, J., & Pradhan, T. (2026) · Springer eBooks · doi:10.1007/978-981-92-0960-6_8
Enhancing English Essay Evaluation Through Topic-Aware Sentiment Analysis with Gated Recurrent Units
Wang, Z. (2025) · International Conference on Computer Science and Artificial Intelligence Applications (CSAIA) · doi:10.1109/csaia65930.2025.11623812
PEUMO: An AI-based writing assessment and support tool for engineering report writing in Spanish
Canales, C. C., & René Venegas, V. (2026) · Assessing Writing · doi:10.1016/j.asw.2026.101106
Ma, X., Wang, D., & Chen, G. (2026) · The Journal of Educational Research · doi:10.1080/00220671.2026.2710208
- Authors
- Gao, J.
- Affiliations
- Zhejiang University (CN)
- Corresponding author
- Gao, Jianmin
- ORCID
- Gao, Jianmin
- Venue
- International Journal of Emerging Technologies in Learning (iJET)
- Published
- 2021-06-04
- Volume
- 16
- Issue
- 11
- Pages
- 322
- ISSN
- 1863-0383 (ISSN-L)
- OpenAlex
- W3166986560
- Cited by
- 68
- References
- 15
- Citations by year
- 2026: 12 · 2025: 19 · 2024: 15 · 2023: 17 · 2022: 5
- FWCI
- 36.54
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- International Journal of Emerging Technologies in Learning (iJET) (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › EFL/ESL Teaching and Learning
- Physical Sciences › Computer Science › Information Systems › English Language Learning and Teaching
- Social Sciences › Psychology › Developmental and Educational Psychology › Second Language Acquisition and Learning
- Keywords
- automated writing evaluation, feedback quality, diagnostic precision, student perceptions, English writing, lexical errors, collocation errors, syntactic errors, university students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
