CoChat
Sign Up
Cited by 247Natural Language Processing TechniquesEducational Technology and AssessmentAdvanced Text Analysis Techniques

Automated Essay Scoring: A Survey of the State of the Art

Open access
Ke, Zixuan, Ng, Vincent·2019·Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence
DOI 10.24963/ijcai.2019/879
Open this paper in CoChat — free accountOpen PDF

Abstract

Despite being investigated for over 50 years, the task of automated essay scoring is far from being solved. Nevertheless, it continues to draw a lot of attention in the natural language processing community in part because of its commercial and educational values as well as the associated research challenges. This paper presents an overview of the major milestones made in automated essay scoring research since its inception.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.24963/ijcai.2019/879

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Trained rater panels do not converge to a single true score, and agreement is raised through training and adjudication rather than existing a priori.

    “trained rater panels do not converge to a single true score, and agreement is raised through training and adjudication rather than existing a priori.”

    From A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays (2026) · cites this paper as Ke and Ng, 2019

    Section 4 (Evaluation; page unavailable) describes an agreement metric for evaluating automated essay scoring, and Section 6 (Concluding Remarks; page unavailable) discusses annotated corpora. Neither addresses whether trained rater panels converge on a true score or whether training and adjudication raise their agreement. Searches for rater training, inter-rater reliability, and adjudication found no passage supporting those claims. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 61

  1. Dimitrios Alikaniotis, Helen Yan- nakoudakis, and Marek Rei. Automatic text scoring using neural networks. In Proc. of ACL, pages 715-725, 2016.
  2. Evelin Amorim, Marcia Canc ¸ado, and Adriano Veloso. Automated essay scoring in the presence of biased ratings. In Proc. of NAACL-HLT, pages 229-237, 2018.
  3. Attali and Burstein, 2006] Yigal Attali and Jill Burstein. Automated essay scoring with e-rater R v.2. The Journal of Technology, Learning and Assessment, 4(3), 2006.
  4. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. The Berkeley FrameNet project. In Proc. of COLING/ACL, pages 86-90, 1998.
  5. Barzilay and Lapata, 2008] Regina Barzilay and Mirella Lapata. Modeling local coherence: An entity-based ap- proach. Computational Linguistics, 34(1):1-34, 2008.
  6. Jill Burstein, Martin Chodorow, and Claudia Leacock. Automated essay evaluation: The Crite- rion online writing service. AI Magazine, 25(3):27, 2004.
  7. Jill Burstein, Joel Tetreault, and Slava Andreyev. Using entity-based features to model coherence in student essays. In Proc. of NAACL-HLT, pages 681- 684, 2010.
  8. Winston Carlile, Nishant Gurrapadi, Zixuan Ke, and Vincent Ng. Give me more feedback: An- notating argument persuasiveness and related attributes in student essays. In Proc. of ACL, pages 621-631, 2018.
  9. Hongbo Chen and Ben He. Automated essay scoring by maximizing human-machine agreement. In Proc. of EMNLP, pages 1741-1752, 2013.
  10. Yen-Yu Chen, Chien-Liang Liu, Chia- Hoang Lee, and Tao-Hsing Chang. An unsupervised au- tomated essay-scoring system. IEEE Intelligent systems, 25(5):61-67, 2010.
  11. Collobert and Weston, 2008] Ronan Collobert and Jason Weston. A unified architecture for natural language pro- cessing: Deep neural networks with multitask learning. In Proc. of ICML, pages 160-167, 2008.
  12. Madalina Cozma, Andrei Butnaru, and Radu Tudor Ionescu. Automated essay scoring with string kernels and word embeddings. In Proc. of ACL, pages 503-509, 2018.
  13. Scott Crossley, Laura K. Allen, Er- ica L. Snow, and Danielle S. McNamara. Pssst... textual features... there is more to automatic essay scoring than just you! In Proc. of ICLAK, pages 203-207, 2015.
  14. Ronan Cummins, Meng Zhang, and Ted Briscoe. Constrained multi-task learning for auto- mated essay scoring. In Proc. of ACL, pages 789-799, 2016.
  15. Daumé III, 2007] Hal Daumé III. Frustratingly easy domain adaptation. In Proc. of ACL, pages 256-263, 2007.
  16. Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In Proc. of NAACL-HLT, pages 4171-4186, 2019.
  17. Fei Dong and Yue Zhang. Auto- matic features for essay scoring -An empirical study. In Proc. of EMNLP, pages 1072-1077, 2016.
  18. Fei Dong, Yue Zhang, and Jie Yang. Attention-based recurrent convolutional neural network for automatic essay scoring. In Proc. of CoNLL, pages 153-162, 2017.
  19. Noura Farra, Swapna Somasundaran, and Jill Burstein. Scoring persuasive essays using opinions and their targets. In Proc. of the BEA Workshop, pages 64-74, 2015.
  20. Faulkner, 2014] Adam Faulkner. Automated classification of stance in student essays: An approach using stance tar- get information and the Wikipedia link-based measure. In Proc. of FLAIRS, 2014.
  21. Debanjan Ghosh, Aquila Khanam, Yubo Han, and Smaranda Muresan. Coarse-grained argu- mentation features for scoring persuasive essays. In Proc. of ACL, pages 549-554, 2016.
  22. Barbara J. Grosz, Scott Weinstein, and Aravind K. Joshi. Centering: A framework for modeling the local coherence of discourse. Computational linguis- tics, 21(2):203-225, 1995.
  23. Derrick Higgins, Jill Burstein, Daniel Marcu, and Claudia Gentile. Evaluating multiple aspects of coherence in student essays. In Proc. of HLT-NAACL, pages 185-192, 2004.
  24. Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.
  25. Andrea Horbach, Dirk Scholten- Akoun, Yuning Ding, and Torsten Zesch. Fine-grained es- say scoring of a complex writing task for native speakers. In Proc. of the BEA workshop, pages 357-366, 2017.
  26. Cancan Jin, Ben He, Kai Hui, and Le Sun. TDNN: A two-stage deep neural network for prompt- independent automated essay scoring. In Proc. of ACL, pages 1088-1097, 2018.
  27. Ke et al., 2018] Zixuan Ke, Winston Carlile, Nishant Gurra- padi, and Vincent Ng. Learning to give feedback: Model- ing attributes affecting argument persuasiveness in student essays. In Proc. of IJCAI, pages 4130-4136, 2018.
  28. Klebanov and Flor, 2013] Beata Beigman Klebanov and Michael Flor. Word association profiles and their use for automated scoring of essays. In Proc. of ACL, pages 1148- 1158, 2013.
  29. Beata Beigman Klebanov, Nitin Madnani, and Jill Burstein. Using pivot-based paraphras- ing and sentiment profiles to improve a subjectivity lexi- con for essay data. Transactions of the ACL, 1:99-110, 2013.
  30. Beata Beigman Klebanov, Michael Flor, and Binod Gyawali. Topicality-based indices for es- say scoring. In Proc. of the BEA Workshop, pages 63-72, 2016.
  31. Thomas K. Landauer, Darrell La- ham, and Peter W. Foltz. Automated scoring and anno- tation of essays with the Intelligent Essay Assessor TM ˙In Automated Essay Scoring: A Cross-Disciplinary Perspec- tive, pages 87-112.
  32. Claudia Leacock, Martin Chodorow, Michael Gamon, and Joel R. Tetreault. Automated Gram- matical Error Detection for Language Learners. Morgan & Claypool Publishers, 2nd edition, 2014. Synthesis Lec- tures on Human Language Technologies.
  33. Louis and Higgins, 2010] Annie Louis and Derrick Hig- gins. Off-topic essay detection using short prompt texts. In Proc. of the BEA Workshop, pages 92-95, 2010.
  34. William C. Mann and San- dra A. Thompson. Rhetorical Structure Theory: To- ward a functional theory of text organization. Text- Interdisciplinary Journal for the Study of Discourse, 8(3):243-281, 1988.
  35. Miltsakaki and Kukich, 2004] Eleni Miltsakaki and Karen Kukich. Evaluation of text coherence for electronic es- say scoring systems. Natural Language Engineering, 10(1):25-55, 2004.
  36. Morris and Hirst, 1991] Jane Morris and Graeme Hirst. Lexical cohesion computed by thesaural relations as an in- dicator of the structure of text. Computational Linguistics, 17(1):21-48, 1991.
  37. Huy V. Nguyen and Diane J. Litman. Argument mining for improving the automated scoring of persuasive essays. In Proc. of AAAI, pages 5892-5899, 2018.
  38. Robert Östling, André Smolentzov, Björn Tyrefors Hinnerich, and Erik Höglin. Automated essay scoring for Swedish. In Proc. of the BEA Workshop, pages 42-47, 2013.
  39. Ellis B Page. The imminence of... grading es- says by computer. The Phi Delta Kappan, 47(5):238-243, 1966.
  40. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global vectors for word representation. In Proc. of EMNLP, pages 1532-1543, 2014.
  41. Persing and Ng, 2013] Isaac Persing and Vincent Ng. Mod- eling thesis clarity in student essays. In Proc. of ACL, pages 260-269, 2013.
  42. Persing and Ng, 2014] Isaac Persing and Vincent Ng. Mod- eling prompt adherence in student essays. In Proc. of ACL, pages 1534-1543, 2014.
  43. Persing and Ng, 2015] Isaac Persing and Vincent Ng. Mod- eling argument strength in student essays. In Proc. of ACL- IJCNLP, pages 543-552, 2015.
  44. Persing et al., 2010] Isaac Persing, Alan Davis, and Vincent Ng. Modeling organization in student essays. In Proc. of EMNLP, pages 229-239, 2010.
  45. Peter Phandi, Kian Ming A Chai, and Hwee Tou Ng. Flexible domain adaptation for automated essay scoring using correlated linear regression. In Proc. of EMNLP, pages 431-439, 2015.
  46. Lawrence M. Rudner and Tahung Liang. Automated essay scoring using Bayes' Theorem. The Journal of Technology, Learning and Assessment, 1(2), 2002.
  47. Mark D. Shermis and Jill C. Burstein. Handbook of Automated Essay Evaluation: Cur- rent Applications and New Directions. Routledge, New York, NY, 2013.
  48. Swapna Somasundaran, Jill Burstein, and Martin Chodorow. Lexical chaining for measuring discourse coherence quality in test-taker essays. In Proc. of COLING, pages 950-961, 2014.
  49. Stab and Gurevych, 2014] Christian Stab and Iryna Gurevych. Annotating argument components and rela- tions in persuasive essays. In Proc. of COLING, pages 1501-1510, 2014.
  50. Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Proc. of NIPS, pages 3104-3112, 2014.
  51. Taghipour and Ng, 2016] Kaveh Taghipour and Hwee Tou Ng. A neural approach to automated essay scoring. In Proc. of EMNLP, pages 1882-1891, 2016.
  52. Yi Tay, Minh C. Phan, Luu Anh Tuan, and Siu Cheung Hui. SkipFlow: Incorporating neural coher- ence features for end-to-end automatic text scoring. In Proc. of AAAI, pages 5948-5955, 2018.
  53. Sowmya Vajjala. Automated assessment of non-native learner essays: Investigating the role of linguis- tic features. International Journal of Artificial Intelligence in Education, 28(1):79-105, 2018.
  54. Henning Wachsmuth, Khalid Al Khatib, and Benno Stein. Using argument mining to assess the argumentation quality of essays. In Proc. of COLING, pages 1680-1691, 2016.
  55. Yucheng Wang, Zhongyu Wei, Yaqian Zhou, and Xuanjing Huang. Automatic essay scoring in- corporating rating schema via reinforcement learning. In Proc. of EMNLP, pages 791-797, 2018.
  56. Yannakoudakis and Briscoe, 2012] Helen Yannakoudakis and Ted Briscoe. Modeling coherence in ESOL learner texts. In Proc. of the BEA workshop, pages 33-43, 2012.
  57. Yannakoudakis and Cummins, 2015] Helen Yannakoudakis and Ronan Cummins. Evaluating the performance of au- tomated text scoring systems. In Proceedings of the BEA Workshop, pages 213-223, 2015.
  58. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. A new dataset and method for automatically grading ESOL texts. In Proc. of ACL, pages 180-189, 2011.
  59. Torsten Zesch, Michael Wojatzki, and Dirk Scholten-Akoun. Task-independent features for au- tomated essay grading. In Proc. of the BEA Workshop, pages 224-232, 2015.
  60. Fan Zhang, Homa B. Hashemi, Rebecca Hwa, and Diane Litman. A corpus of annotated revisions for studying argumentative writing. In Proc. of ACL, pages 1568-1578, 2017.
  61. Zupanc and Bosnic, 2016] Kaja Zupanc and Zoran Bosnic. Advances in the field of automated essay evaluation. In- formatica, 39(4), 2016.

Cited by

Sorted by citations

Authors
Ke, Z., & Ng, V.
Affiliations
The University of Texas at Dallas (US)
Corresponding author
Ke, Zixuan
ORCID
Ng, Vincent
Venue
Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence
Publisher
International Joint Conferences on Artificial Intelligence Organization
Published
2019-08
Pages
6300-6308
DOI
10.24963/ijcai.2019/879
OpenAlex
W2964413085
Cited by
247
References
0
Citations by year
2026: 35 · 2025: 41 · 2024: 38 · 2023: 56 · 2022: 35 · 2021: 30 · 2020: 12
FWCI
7.37
Type
conference paper
Language
en
Open access
Locations
International Joint Conference on Artificial Intelligence (publishedVersion)
Funders
National Science Foundation (1528037, 1848608)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
Keywords
automated essay scoring, natural language processing, writing assessment, essay scoring, research milestones, commercial applications, educational technology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review