The DOI matches its Crossref record; 67 of the 70 references it cites were found in a registry or an index.
Automated essay scoring: Psychometric guidelines and practices
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. E., & Cohen, J. (1989) · Technometrics · doi:10.2307/1270020 · Cited by 99,603
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,417
Fleiss, J. L., & Cohen, J. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300309 · Cited by 3,354
MacArthur, C. A., Graham, S. W., & Fitzgerald, J. (2005) · Cited by 2,327
Shanahan, C. R. (2011) · Cited by 866
AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0
Attali, Y., & Burstein, J. C. (2004) · ETS Research Report Series · doi:10.1002/j.2333-8504.2004.tb01972.x · Cited by 641
Rating the ratings: Assessing the psychometric quality of rating data.
Saal, F. E., Downey, R. G., & Lahey, M. A. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.88.2.413 · Cited by 545
Shermis, M. D., & Burstein, J. C. (2003) · Routledge eBooks · doi:10.4324/9781410606860 · Cited by 512
Shanahan, T. (2015) · The Elementary School Journal · doi:10.1086/681130 · Cited by 469
(1966) · Phi Delta Kappan · Cited by 449
A framework for evaluation and use of automated scoring
Williamson, D. M., Xi, X., & Breyer, F. J. (2012) · Educational Measurement: Issues and Practice · doi:10.1111/j.1745-3992.2011.00223.x · Cited by 430
Self-citation(2006) · The Journal of Technology, Learning and Assessment (Boston College) · Cited by 424
C-rater: Automated Scoring of Short-Answer Questions
Leacock, C., & Chodorow, M. S. (2003) · Computers and the Humanities · doi:10.1023/a:1025779619903 · Cited by 384
Rearticulating Writing Assessment for Teaching and Learning
Huot, B. (2003) · Utah State University Press eBooks · doi:10.2307/j.ctt46nx5z · Cited by 355
Automated writing evaluation: Defining the classroom research agenda
Warschauer, M., & Ware, P. (2006) · Language Teaching Research · doi:10.1191/1362168806lr190oa · Cited by 337
An Overview of Current Research on Automated Essay Grading
Valenti, S., Neri, F., & Cucchiarelli, A. (2003) · Journal of Information Technology Education Research · doi:10.28945/331 · Cited by 313
On the relation between automated essay scoring and modern views of the writing construct
Deane, P. (2012) · Assessing Writing · doi:10.1016/j.asw.2012.10.002 · Cited by 288
Decision Making while Rating ESL/EFL Writing Tasks: A Descriptive Framework
Cumming, A. H., Kantor, R. N., & Powers, D. E. (2002) · Modern Language Journal · doi:10.1111/1540-4781.00137 · Cited by 243
Automated Scoring of Complex Tasks in Computer-Based Testing
(2006) · Routledge eBooks · doi:10.4324/9780415963572 · Cited by 229
Measuring the Impact of Judge Severity on Examination Scores
Lunz, M. E., Wright, B. D., & Linacre, J. M. (1990) · Applied Measurement in Education · doi:10.1207/s15324818ame0304_3 · Cited by 225
Comparison of Human and Machine Scoring of Essays: Differences by Gender, Ethnicity, and Country
Bridgeman, B., Trapani, C. S., & Attali, Y. (2012) · Applied Measurement in Education · doi:10.1080/08957347.2012.635502 · Cited by 160
The E-rater® Scoring Engine: Automated Essay Scoring With Natural Language Processing
Shermis, M. D., & Burstein, J. C. (2003) · Routledge eBooks · doi:10.4324/9781410606860-16 · Cited by 158
Weigle, S. C. (1999) · Assessing Writing · doi:10.1016/s1075-2935(00)00010-6 · Cited by 157
Automated Scoring and Annotation of Essays with the Intelligent Essay Assessor™
Shermis, M. D., & Burstein, J. C. (2003) · Routledge eBooks · doi:10.4324/9781410606860-15 · Cited by 153
Towards robust computerised marking of free-text responses
Mitchell, T. M., Russell, T., Broomhead, P., & Aldridge, N. (2002) · Loughborough University Research Repository (Loughborough University) · Cited by 151
Page, E. B. (2003) · Cited by 148
Assessing Second Language Writing in Academic Contexts
Leki, I., & Hamp‐Lyons, L. (1993) · College Composition and Communication · doi:10.2307/358997 · Cited by 147
Stumping e-rater:challenging the validity of automated essay scoring
Powers, D. E., Burstein, J. C., Chodorow, M. S., Fowles, M. E., & Kukich, K. (2002) · Computers in Human Behavior · doi:10.1016/s0747-5632(01)00052-8 · Cited by 146
Validity and Automad Scoring: It's Not Only the Scoring
Bennett, R. E., & Bejar, I. I. (1998) · Educational Measurement Issues and Practice · doi:10.1111/j.1745-3992.1998.tb00631.x · Cited by 145
The use of the computer in analyzing student essays
Page, E. B. (1968) · International Review of Education · doi:10.1007/bf01419938 · Cited by 137
On a Scale: A Social History of Writing Assessment in America
Elliot, N. (2005) · Cited by 135
A Review of Strategies for Validating Computer-Automated Scoring
Yang, Y., Buckendahl, C. W., Juszkiewicz, P. J., & Bhola, D. S. (2002) · Applied Measurement in Education · doi:10.1207/s15324818ame1504_04 · Cited by 127
Leckie, G., & Baird, J. (2011) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2011.00152.x · Cited by 111
An Evaluation of IntelliMetric™ Essay Scoring System
Rudner, L. M., García, V. A., & Welch, C. J. (2006) · Open Access Journals at BC (Boston College) · Cited by 101
Understanding Scoring Reliability: Experiments in Calibrating Essay Readers
Braun, H. I. (1988) · Journal of Educational Statistics · doi:10.2307/1164948 · Cited by 98
Cognitive Differences in Proficient and Nonproficient Essay Scorers
Wolfe, E. W., KAO, C.-W., & Ranney, M. A. (1998) · Written Communication · doi:10.1177/0741088398015004002 · Cited by 94
Davis, M. T. (1988) · The Journal of Higher Education · doi:10.1080/00221546.1988.11778315 · Cited by 94
What Happens When Machines Read Our Students' Writing?
Herrington, A. J., & Moran, C. (2001) · College English · doi:10.2307/378891 · Cited by 81
Validity Issues for Performance-Based Tests Scored With Computer-Automated Scoring Systems
Clauser, B. E., Kane, M. T., & Swanson, D. B. (2002) · Applied Measurement in Education · doi:10.1207/s15324818ame1504_05 · Cited by 80
Automated Scoring Technologies and the Rising Influence of Error
Cheville, J. (2004) · The English Journal · doi:10.2307/4128980 · Cited by 72
VALIDITY AND AUTOMATED SCORING: IT'S NOT ONLY THE SCORING
Bennett, R. E., & Bejar, I. I. (1997) · ETS Research Report Series · doi:10.1002/j.2333-8504.1997.tb01734.x · Cited by 69
The SRI EduSpeak System: Recognition and Pronunciation Scoring for Language Learning
Franco, H., Abrash, V., Precoda, K., Bratt, H., Rao, R., Butzberger, J., Rossier, R., & Cesari, F. (2007) · Cited by 67
Rater effects: Ego engagement in rater decision-making
Wiseman, C. S. (2012) · Assessing Writing · doi:10.1016/j.asw.2011.12.001 · Cited by 66
Callear, D., Jerrams-Smith, J., & Soh, V. C.-M. (2001) · Loughborough University Institutional Repository (Loughborough University) · Cited by 64
Performance of a Generic Approach in Automated Essay Scoring
Attali, Y., Bridgeman, B., & Trapani, C. S. (2010) · The Journal of Technology, Learning and Assessment (Boston College) · Cited by 58
‘Mental Model’ Comparison of Automated and Human Scoring
Williamson, D. M., Bejar, I. I., & Hone, A. S. (1999) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1999.tb00552.x · Cited by 58
Self-citationCOMPARING THE VALIDITY OF AUTOMATED AND HUMAN SCORING OF ESSAYS
Powers, D. E., Burstein, J. C., Chodorow, M. S., Fowles, M. E., & Kukich, K. (2002) · Journal of Educational Computing Research · doi:10.2190/cx92-7wkv-n7wc-jl0a · Cited by 49
CCCC Position Statement on Teaching, Learning, and Assessing Writing in Digital Environments
(2004) · College Composition and Communication · doi:10.58680/ccc20042788 · Cited by 46
Towards automatic scoring of non-native spontaneous speech
Zechner, K., & Bejar, I. I. (2006) · doi:10.3115/1220835.1220863 · Cited by 44
Two Experiments on Automatic Scoring of Spoken Language Proficiency
Bernstein, J., de Jong, J. H. A. L., Pisoni, D. B., & Townshend, B. (2000) · Cited by 43
Reducing Errors Due to the Use of Judges
Rudner, L. M. (2020) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/w4a1-cb66 · Cited by 41
Hamp‐Lyons, L. (2007) · Assessing Writing · doi:10.1016/j.asw.2007.05.002 · Cited by 41
Hales, L. W., & Tokar, E. (1975) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1975.tb01014.x · Cited by 40
Sukkarieh, J. Z., & Pulman, S. (2005) · Cited by 38
Ramineni, C., Trapani, C. S., Williamson, D. M., Davey, T., & Bridgeman, B. (2012) · ETS Research Report Series · Cited by 34
Self-citationValidating automated essay scoring for online writing placement
Ramineni, C. (2012) · Assessing Writing · doi:10.1016/j.asw.2012.10.005 · Cited by 31
Self-citationMoving the Field Forward: Some Thoughts on Validity and Automated Scoring
(2006) · Routledge eBooks · doi:10.4324/9780415963572-15 · Cited by 28
A human-computer collaborative approach to the marking of free text answers
Sargeant, J., Wood, M. M., & Anderson, S. M. (2004) · Loughborough University Research Repository (Loughborough University) · Cited by 26
THE USE OF MODEL ESSAYS TO REDUCE CONTEXT EFFECTS IN ESSAY SCORING
HUGHES, D. C., & Keeling, B. (1984) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1984.tb01034.x · Cited by 26
Automated Scoring of Spontaneous Speech Using SpeechRater? v1.0. Research Report. ETS RR-08-62.
Xi, X., Higgins, D., Zechner, K., & Williamson, D. M. (2008) · ETS Research Report Series · Cited by 24
Self-citationEvaluating the Construct-Coverage of the e-rater[R] Scoring Engine. Research Report. ETS RR-09-01.
Quinlan, T., Higgins, D., & Wolff, S. (2009) · TOEFL iBT research reports · Cited by 22
Automated Scoring of Constructed-Response Literacy and Mathematics Items
Elliot, R. (2011) · Cited by 19
A Regression-Based Procedure for Automated Scoring of a Complex Medical Performance Assessment
(2006) · Routledge eBooks · doi:10.4324/9780415963572-7 · Cited by 18
Ramineni, C., Trapani, C. S., Williamson, D. M., Davey, T., & Bridgeman, B. (2012) · ETS Research Report Series · Cited by 15
Self-citationNEW SAT® WRITING PROMPT STUDY: ANALYSES OF GROUP IMPACT AND RELIABILITY
Breland, H. M., Kubota, M. Y., Nickerson, K., Trapani, C. S., & Walker, M. E. (2004) · ETS Research Report Series · doi:10.1002/j.2333-8504.2004.tb01930.x · Cited by 14
How important is content in the ratings of essay assessments?
Shermis, M. D., Shneyderman, A., & Attali, Y. (2008) · Assessment in Education Principles Policy and Practice · doi:10.1080/09695940701876219 · Cited by 11
Testing and Assessing Mathematical Skills by a Script Based System
Risse, T. (2007) · Cited by 9
Flagged first, then by citations
3 of the 70 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Pearson (2009, March). PTE academic automated scoring. Retrieved from http://www.pearsonpte.com/SiteCollectionDocuments/AutomatedScoringUS.pdf
- Stalnaker, 1936, The problem of the English examination, Educational Record, 17, 35
- Williamson, 2010, Automated Scoring for the Assessment of Common Core Standards
Cited by
Sorted by citations
Baker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993
A hierarchical classification approach to automated essay scoring
McNamara, D. S., Crossley, S. A., Roscoe, R. D., Allen, L. K., & Dai, J. (2014) · Assessing Writing · doi:10.1016/j.asw.2014.09.002 · Cited by 243
Shermis, M. D. (2014) · Assessing Writing · doi:10.1016/j.asw.2013.04.001 · Cited by 227
Stop! Grammar time: University students’ perceptions of the automated feedback program Grammarly
ONeill, R., & Russell, A. (2019) · Australasian Journal of Educational Technology · doi:10.14742/ajet.3795 · Cited by 180
Graduate Theses and Dissertations
Davis, M., Davis, K. J., & Dunagan, M. M. (2012) · Elsevier eBooks · doi:10.1016/b978-0-12-384727-0.00006-9 · Cited by 165
Metadata
- Authors
- Ramineni, C., & Williamson, D. M.
- Affiliations
- Educational Testing Service (US)
- Corresponding author
- Ramineni, Chaitanya
- Venue
- Assessing Writing
- Publisher
- Elsevier BV
- Published
- 2013-01
- Volume
- 18
- Issue
- 1
- Pages
- 25-39
- ISSN
- 1075-2935 (ISSN-L)
- OpenAlex
- W2088070094
- Cited by
- 127
- References
- 70
- Citations by year
- 2026: 11 · 2025: 15 · 2024: 26 · 2023: 6 · 2022: 7 · 2021: 7 · 2020: 7 · 2019: 9 · 2018: 10 · 2017: 7 · 2016: 6 · 2015: 4 · 2014: 4 · 2013: 4
- FWCI
- 13.89
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Assessing Writing (publishedVersion)
- Funders
- Educational Testing Service
- Topics
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Education › Student Assessment and Feedback
- Keywords
- automated essay scoring, e-rater, psychometric assessment, writing assessment, large-scale assessment, performance expectations, Educational Testing Service
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
