CoChat
Sign Up
Cited by 103Discourse Analysis in Language StudiesSecond Language Acquisition and LearningText Readability and Simplification

Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing

Crossley, Scott A., McNamara, Danielle S.·2011·International Journal of Continuing Engineering Education and Life-Long Learning
DOI 10.1504/ijceell.2011.040197
Open this paper in CoChat — free account

Abstract

This article reviews recent studies in which human judgements of essay quality are assessed using Coh-Metrix, an automated text analysis tool. The goal of these studies is to better understand the relationship between linguistic features of essays and human judgements of writing quality. Coh-Metrix reports on a wide range of linguistic features, affording analyses of writing at various levels of text structure, including surface, text-base, and situation model levels. Recent studies have examined linguistic features of essay quality related to co-reference, connectives, syntactic complexity, lexical diversity, spatiality, temporality, and lexical characteristics. These studies have analysed essays written by both first language and second language writers. The results support the notion that human judgements of essay quality are best predicted by linguistic indices that correlate with measures of language sophistication such as lexical diversity, word frequency, and syntactic complexity. In contrast, human judgements of essay quality are not strongly predicted by linguistic indices related to cohesion. Overall, the studies portray high quality writing as containing more complex language that may not facilitate text comprehension.

Integrity signals

Registry

Crossref · 10.1504/ijceell.2011.040197

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The participants may find it challenging to write a high-quality composition where low frequency vocabulary is needed.

    “Thus, the participants may find it challenging to write a high-quality composition where low frequency vocabulary is needed (Crossley & McNamara, 2011;Olinghouse & Wilson, 2013).”

    From The Vocabulary Knowledge of Saudi EFL Tertiary Students (2019) · cites this paper as Crossley & McNamara, 2011

  • There is a correlation between employing low frequency vocabulary and high writing scores.

    “A correlation was found between employing low frequency vocabulary and high writing scores (Crossley & McNamara, 2011).”

    From The Vocabulary Knowledge of Saudi EFL Tertiary Students (2019) · cites this paper as Crossley & McNamara, 2011

  • There is a high correlation between vocabulary knowledge and writing skill.

    “High correlation was found between vocabulary knowledge and writing skill in a number of studies (e.g.Yu, 2010; Crossley & McNamara, 2011;Olinghouse & Wilson, 2013).”

    From The Vocabulary Knowledge of Saudi EFL Tertiary Students (2019) · cites this paper as Crossley & McNamara, 2011

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. Abbott, 1993, Journal of Educational Psychology, 85, 478, 10.1037/0022-0663.85.3.478 · DOI
  2. Abbott, 2010, Journal of Educational Psychology, 102, 281, 10.1037/a0019318 · DOI
  3. Berninger, 1991, Journal of School Psychology, 29, 57, 10.1016/0022-4405(91)90016-K · DOI
  4. Berninger, 1992, Reading and Writing: An Interdisciplinary Journal, 4, 257, 10.1007/BF01027151 · DOI
  5. Berninger, 2010, Journal of Psycholinguistic Research, 39, 141, 10.1007/s10936-009-9130-6 · DOI
  6. Charniak, 2000, Proceedings of the First Conference on North American Chapter of the Association for Computational Linguistics, 132
  7. Coltheart, 1981, Quarterly Journal of Experimental Psychology, 33, 497, 10.1080/14640748108400805 · DOI
  8. Connor, 1987, TESaL Quarterly, 21, 677, 10.2307/3586989 · DOI
  9. Connor, 1990, Research in the Teaching of English, 24, 67, 10.58680/rte199015501 · DOI
  10. Crismore, 1993, Written Communication, 10, 39, 10.1177/0741088393010001002 · DOI
  11. Crossley, 2010, Proceedings of the 32nd Annual Conference of the Cognitive Science Society, 984
  12. Crossley, 2008, TESaL Quarterly, 42, 475, 10.1002/j.1545-7249.2008.tb00142.x · DOI
  13. Crossley, 2010b, Language Learning, 60, 573, 10.1111/j.1467-9922.2010.00568.x · DOI
  14. Crowhurst, 1990, Canadian Journal of Education, 15, 155, 10.2307/1495373 · DOI
  15. Davies, 1974, The Edinburgh Course in Applied Linguistics: Techniques in Applied Linguistics, 155
  16. Douglas, 1981, Learning to Read in Different Languages, 33
  17. Engber, 1995, Journal of Second Language Writing, 4, 139, 10.1016/1060-3743(95)90004-7 · DOI
  18. Ferris, 1994, TESaL Quarterly, 28, 14
  19. Foltz, 2007, Handbook of Latent Semantic Analysis, 167
  20. Graesser, 2003, Rethinking Reading Comprehension, 82
  21. Graesser, 2004, Behavior Research Methods, Instruments, & Computers, 36, 193, 10.3758/BF03195564 · DOI
  22. Grant, 2000, Journal of Second Language Writing, 9, 123, 10.1016/S1060-3743(00)00019-9 · DOI
  23. Gumperz, 1984, Coherence in Spoken and Written Discourse, 3
  24. Haberlandt, 1985, Journal of Experimental Psychology: General, 114, 357, 10.1037/0096-3445.114.3.357 · DOI
  25. Jarvis, 2002, Studies in Second Language Acquisition, 24, 387, 10.1017/S0272263102003029 · DOI
  26. Jarvis, 2003, Journal of Second Language Writing, 12, 377, 10.1016/j.jslw.2003.09.001 · DOI
  27. Jenkins, 2004, Scientific Studies in Reading, 8, 125, 10.1207/s1532799xssr0802_2 · DOI
  28. Just, 1992, Psychological Review, 99, 122, 10.1037/0033-295X.99.1.122 · DOI · Paper page
  29. King, 1979, Research in the Teaching of English, 13, 243, 10.58680/rte201117862 · DOI
  30. Kintsch, 1978, Psychological Review, 85, 363, 10.1037/0033-295X.85.5.363 · DOI · Paper page
  31. Longo, 1994, Technical Communication, 41, 348
  32. Louwerse, 2001, Cognitive Linguistics, 12, 291, 10.1515/cogl.2002.005 · DOI
  33. McCutchen, 1986, Journal of Memory and Language, 25, 431, 10.1016/0749-596X(86)90036-7 · DOI
  34. McCutchen, 1982, Text, 2, 113, 10.1515/text.1.1982.2.1-3.113 · DOI
  35. McNamara, 2001, Canadian Journal of Experimental Psychology, 55, 51, 10.1037/h0087352 · DOI · Paper page
  36. McNamara, 2010, Written Communication, 27, 57, 10.1177/0741088309351547 · DOI · Paper page
  37. McNamara, 1996, Cognition and Instruction, 14, 1, 10.1207/s1532690xci1401_1 · DOI · Paper page
  38. a"Reilly, 2007, Discourse Processes, 43, 121, 10.1080/01638530709336895 · DOI
  39. Pearson, Reading Research Quarterly, 10, 155, 10.2307/747180 · DOI
  40. Perfetti, 2005a, The Science of Reading: A Handbook, 227, 10.1002/9780470757642.ch13 · DOI
  41. Perfetti, 2005b, Journal of Experimental Psychology: Learning Memory and Cognition, 31, 1281, 10.1037/0278-7393.31.6.1281 · DOI
  42. Rashotte, 1985, Reading Research Quarterly, 20, 180, 10.1598/RRQ.20.2.4 · DOI
  43. Reid, 1986, Technology and Language Testing, 167
  44. Reid, 1990, Second Language Writing: Research Insights for the Classroom, 191, 10.1017/CBO9781139524551.017 · DOI
  45. Reid, 1992, Journal of Second Language Writing, 1, 79, 10.1016/1060-3743(92)90010-M · DOI
  46. Reppen, 1994, TESaL Journal, 4, 32
  47. van de Kopple, 1985, College Composition and Communication, 36, 82, 10.2307/357609 · DOI
  48. Wilson, 1988, Behavior Research Methods, Instruments, and Computers, 20, 6, 10.3758/BF03202594 · DOI

Cited by

Sorted by citations

Authors
Crossley, S. A., & McNamara, D. S.
Affiliations
  • Georgia State University (US)
  • University of Memphis (US)
ORCID
Venue
International Journal of Continuing Engineering Education and Life-Long Learning
Publisher
Inderscience Publishers
Published
2011
Volume
21
Issue
2/3
Pages
170
ISSN
  • 1560-4624 (ISSN-L)
  • 1741-5055
DOI
10.1504/ijceell.2011.040197
OpenAlex
W2008455588
Semantic Scholar
b5dfb3a92c55f6ef50c8442e91828568240ab31c
Cited by
103
References
2
Citations by year
2026: 13 · 2025: 4 · 2024: 3 · 2023: 4 · 2022: 5 · 2021: 7 · 2020: 7 · 2019: 6 · 2018: 9 · 2017: 9 · 2016: 8 · 2015: 6 · 2014: 9 · 2013: 5 · 2012: 6
FWCI
130.71
Type
journal article
Language
en
Open access
closed
Locations
International Journal of Continuing Engineering Education and Life-Long Learning (publishedVersion)
Topics
  • Social Sciences › Arts and Humanities › Literature and Literary Theory › Discourse Analysis in Language Studies
  • Social Sciences › Psychology › Developmental and Educational Psychology › Second Language Acquisition and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Text Readability and Simplification
Keywords
essay quality, Coh-Metrix, linguistic features, first language writing, second language writing, lexical diversity, syntactic complexity, coreference, cohesion, connectives, text comprehension, situation model, word frequency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review