Reliability of repeated grading of essay type examinations.
The DOI matches its Crossref record.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“and that, after analysis of reliability coefficients, assignment of scores amounted to “little better than sheer guesses” (Eells, 1930).”
From Teaching More by Grading Less (or Differently) (2014) · cites this paper as Eells, 1930
In the citing paper
“He concluded that “variability of grading is about as great in the same individual as in groups of different individuals””
From Teaching More by Grading Less (or Differently) (2014) · cites this paper as Eells, 1930
Citations
Cites
Still being checked.
Cited by
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496
A Century of Grading Research: Meaning and Value in the Most Common Educational Measure
Brookhart, S. M., Guskey, T. R., Bowers, A. J., McMillan, J. H., Smith, J. K., Smith, L. F., Stevens, M. T., & Welsh, M. E. (2016) · Review of Educational Research · doi:10.3102/0034654316672069 · Cited by 386
Teaching More by Grading Less (or Differently)
Schinske, J., & Tanner, K. (2014) · CBE—Life Sciences Education · doi:10.1187/cbe.cbe-14-03-0054 · Cited by 326
Examinations and higher education: a survey of the literature
Cox, R. (1967) · Higher Education Quarterly · doi:10.1111/j.1468-2273.1967.tb00244.x · Cited by 82
Laming, D. (1990) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640749008401220 · Cited by 76
The Effects of Tests on Students and Schools
Kirkland, M. C. (1971) · Review of Educational Research · doi:10.3102/00346543041004303 · Cited by 87
Reliability, fairness, objectivity and other inappropriate goals in peer review
Bailar, J. C. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065705 · Cited by 66
Hoskens, M., & Wilson, M. (2001) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2001.tb01119.x · Cited by 52
“In line with earlier research on the statewide tests administered by the CDE (Wilson & Case, 2000; Wilson & Wang, 1995) and research by others from as early as the 1930s (e.g., Ashburn, 1938; Eells, 1930) significant variability between raters was found in their essay scoring.”
MethodAnalytic or holistic? A study about how to increase the agreement in teachers’ grading
Jönsson, A., Balan, A., & Hartell, E. (2021) · Assessment in Education Principles Policy and Practice · doi:10.1080/0969594x.2021.1884041 · Cited by 58
Does the need for agreement among reviewers inhibit the publication controversial findings?
Armstrong, J. S., & Hubbard, R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065699 · Cited by 41
Practices of grading: an ethnographic study of educational assessment
Kalthoff, H. (2013) · Ethnography & Education · doi:10.1080/17457823.2013.766436 · Cited by 32
Unreliability of Marking: further evidence and a possible explanation
Branthwaite, A., Trueman, M., & Berrisford, T. (1981) · Educational Review · doi:10.1080/0013191810330105 · Cited by 26
The predictive validity of peer review: A neglected issue
Bornstein, R. F. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065717 · Cited by 26
Marking University Examinations: Some Lessons from Psychophysics
Laming, D. (2004) · Psychology Learning & Teaching · doi:10.2304/plat.2003.3.2.89 · Cited by 23
When nonreliability of reviews indicates solid science
Eckberg, D. L. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065791 · Cited by 24
New-type or objective tests: a summary of recent investigations.
Lee, J. M., & Symonds, P. M. (1933) · Journal of Educational Psychology · doi:10.1037/h0072226 · Cited by 19
Reflections on the peer review process
Marsh, H. W., & Ball, S. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065912 · Cited by 18
Analytic or Holistic: A Study of Agreement Between Different Grading Models
Jönsson, A., & Balan, A. (2018) · Practical assessment, research & evaluation · doi:10.7275/mg59-xq60 · Cited by 21
Confusion between reviewer reliability and wise editorial and funding decisions
Kiesler, C. A. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065869 · Cited by 16
Reflections from the peer review mirror
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00066024 · Cited by 16
Hargens, L. L. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065857 · Cited by 12
Gugiu, M. R., Gugiu, P. C., & Baldus, R. (2012) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v8i19.362 · Cited by 20
Do we really want more “reliable” reviewers?
KRAEMER, H. C. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065870 · Cited by 14
Disagreement among journal reviewers: No cause for undue alarm
Stricker, L. J. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065985 · Cited by 12
What should be done improve reviewing?
Crandall, R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065766 · Cited by 11
Is unreliability in peer review harmful?
Roediger, H. L., III (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065936 · Cited by 9
Beurteilungsübereinstimmung bei Mathematikarbeiten?
Birkel, P. (2005) · Journal für Mathematik-Didaktik · doi:10.1007/bf03339005 · Cited by 11
Why is the reliability of peer review so low?
Laming, D. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065882 · Cited by 8
Schönemann, P. H. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065973 · Cited by 7
Peer review: An unflattering picture
Adams, K. M. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065687 · Cited by 5
Journal availability and the quality of published research
Fletcher, J. M. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065808 · Cited by 5
Equitable and Culturally Relevant Grading Practices
Brookhart, S. M., & Chism, D. T. (2025) · Routledge eBooks · doi:10.4324/9781003392217-21 · Cited by 5
On forecasting validity and finessing reliability
Gilmore, J. B. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065821 · Cited by 4
Toward openness and fairness in the review process
Rourke, B. P. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0006595x · Cited by 4
Peer review: Explicit criteria and training can help
Delcomyn, F. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065778 · Cited by 3
Stasz, C. (1976) · Teaching Sociology · doi:10.2307/1317088 · Cited by 4
Some indices of the reliability of peer review
Rosenthal, R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065948 · Cited by 3
Should the blinded lead the blinded?
Lock, S. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065894 · Cited by 3
Chairman's action: The importance of executive decisions in peer review
Tyrer, P. J. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065997 · Cited by 4
What to do about peer review: Is the cure worse than the disease?
Zentall, T. R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00066012 · Cited by 3
Justice, efficiency and epistemology in the peer review of scientific manuscripts
Mahoney, M. J. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065900 · Cited by 3
Peer review is not enough: Editors must work with librarians to ensure access to research
Fuller, S. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0006581x · Cited by 2
Unreliable peer review: Causes and cures of human misery
Colman, A. M. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065742 · Cited by 3
The process of peer review: Unanswered questions
Nelson, L. D. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065924 · Cited by 2
Criterion reliability: A study of folkways and mores of the psychological profession.
Sydiaha, D. (1964) · The Canadian Psychologist · doi:10.1037/h0083010 · Cited by 2
Evaluating scholarly works: How many reviewers? How much anonymity?
Cone, J. D. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065754 · Cited by 2
Consensus and the reliability of peer-review evaluations
Cole, S. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065730 · Cited by 4
A Half Century of Research on Essay Testing
Yeasmeen, N., & Barker, D. G. (1973) · Improving College and University Teaching · doi:10.1080/00193089.1973.10533348 · Cited by 1
Wasserman, G. S. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00066000 · Cited by 1
Educational psychology—some general aspects of learning and teaching.
Spence, R. B., & Duncan, R. (1931) · Psychological Bulletin · doi:10.1037/h0075709
Chapter IV: Recent Developments in the Written Essay Examination
Weidemann, C. C. (1935) · Review of Educational Research · doi:10.3102/00346543005005484
Does group discussion contribute reliability of complex judgments?
Cohen, P. R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065729
Now that we know how low the reliability is, what shall we do?
Salzinger, K. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065961
Is there an alternative to peer review?
Greene, R. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065845
Replication, reliability and peer review: A case study
Gorman, M. E. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065833
Different rates of agreement on acceptance and rejection: A statistical artifact?
Demorest, M. E. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0006578x
Swenson, G. (2020) · STARS (University of Central Florida)
Chapter VIII: Foreign Languages
(1931) · Review of Educational Research · doi:10.3102/00346543001005371
(1940) · The Journal of Higher Education · doi:10.1080/00221546.1940.11773740
The Evaluation of Performance in the Study of Engineering
Kalthoff, H., & Engert, K. (2025) · Human Studies · doi:10.1007/s10746-025-09824-y
Marked: School Grades and the Quantified Life
Rohde, N. (2026) · LSE Press eBooks · doi:10.31389/lsepress.mar
Metadata
- Authors
- Eells, W. C.
- Corresponding author
- Eells, Walter Crosby
- Venue
- Journal of Educational Psychology
- Published
- 1930-01
- Volume
- 21
- Issue
- 1
- Pages
- 48-52
- ISSN
- 0022-0663 (ISSN-L)
- 1939-2176
- OpenAlex
- W2087091712
- Cited by
- 59
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1 · 2021: 1 · 2020: 1 · 2016: 1 · 2014: 1 · 2013: 1 · 2012: 1
- FWCI
- 3.96
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Educational Psychology (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Social Sciences › Education › Student Assessment and Feedback
- Keywords
- essay tests, rating reliability, inter-rater reliability, essay scoring
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.