CoChat

Assessing Text-Based Writing of Low-Skilled College Students

Open access
Perin, Dolores, Lauterbach, Mark · 2018 · International Journal of Artificial Intelligence in Education
Cited by 65

The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.

Abstract

The problem of poor writing skills at the postsecondary level is a large and troubling one. This study investigated the writing skills of low-skilled adults attending college developmental education courses by determining whether variables from an automated scoring system were predictive of human scores on writing quality rubrics. The human-scored measures were a holistic quality rating for a persuasive essay and an analytic quality score for a written summary. Both writing samples were based on text on psychology and sociology topics related to content taught at the introductory undergraduate level. The study is a modified replication of McNamara et al. ( Written Communication, 27 (1), 57–86 2010 ), who identified several Coh-Metrix variables from five linguistic classes that reliably predicted group membership (high versus low proficiency) using human quality scores on persuasive essays written by average-achieving college students. When discriminant analyses and ANOVAs failed to replicate the McNamara et al. ( Written Communication, 27 (1), 57–86 2010 ) findings, the current study proceeded to analyze all of the variables in the five Coh-Metrix classes. This larger analysis identified 10 variables that predicted human-scored writing proficiency. Essay and summary scores were predicted by different automated variables. Implications for instruction and future use of automated scoring to understand the writing of low-skilled adults are discussed.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s40593-016-0122-z

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 9 references

Research works printed but not linked: 59

  1. Abdi, 2016, An automated summarization assessment algorithm for identifying summarizing strategies, PloS One, 11, 1, 10.1371/journal.pone.0145809 · DOI
  2. Acker, 2008, Preparing high school students for college-level writing: using an e-portfolio to support a successful transition, The Journal of General Education, 57, 1, 10.2307/27798087 · doi:10.2307/27798087
  3. Brace, N., Kemp, R., & Snelgar, R. (2012). SPSS for psychologists: a guide to data analysis using (5th ed.). New York, NY: Routledge.
  4. Bridgeman, 1984, Survey of academic writing tasks, Written Communication, 1, 247, 10.1177/0741088384001002004 · DOI · Paper page
  5. Brown, 1983, Macrorules for summarizing texts: the development of expertise, Journal of Verbal Learning and Verbal Behavior, 22, 1, 10.1016/S0022-5371(83)80002-4 · DOI · Paper page
  6. Carretti, 2016, Oral and written expression in children with reading comprehension difficulties, Journal of Learning Disabilities, 49, 65, 10.1177/0022219414528539 · DOI
  7. Cohen, A. M., Brawer, F. B., & Kisker, C. B. (2013). The American community college (6th ed.). Boston, MA: Wiley.
  8. Comer, 2016, Adventuring into MOOC writing assessment: challenges, results, and possibilities, College Composition and Communication, 67, 318, 10.58680/ccc201628063 · DOI
  9. Crossley, 2009, Computational assessment of lexical differences in L1 and L2 writing, Journal of Second Language Writing, 18, 119, 10.1016/j.jslw.2009.02.002 · DOI · Paper page
  10. Crossley, 2011, The development of writing proficiency as a function of grade level: a linguistic analysis, Written Communication, 28, 282, 10.1177/0741088311410188 · DOI · Paper page
  11. Crossley, 2012, Predicting the proficiency level of language learners using lexical indices, Language Testing, 29, 243, 10.1177/0265532211419331 · DOI · Paper page
  12. Crossley, 2016, The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality, Journal of Second Language Writing, 32, 1, 10.1016/j.jslw.2016.01.003 · DOI · Paper page
  13. Danzak, 2011, The integration of lexical, syntactic, and discourse features in bilingual adolescents’ writing: an exploratory approach, Language, Speech, and Hearing Services in Schools, 42, 491, 10.1044/0161-1461(2011/10-0063) · doi:10.1044/0161-1461(2011/10-0063
  14. De La Paz, 2005, Effects of historical reasoning instruction and writing strategy mastery in culturally and academically diverse middle school classrooms, Journal of Educational Psychology, 97, 139, 10.1037/0022-0663.97.2.139 · DOI · Paper page
  15. De La Paz, 2012, Adolescents’ disciplinary use of evidence, argumentative strategies, and organizational structure in writing about historical controversies, Written Communication, 29, 412, 10.1177/0741088312461591 · DOI
  16. Deane, 2013, On the relation between automated essay scoring and modern views of the writing construct, Assessing Writing, 18, 7, 10.1016/j.asw.2012.10.002 · DOI · Paper page
  17. Deane, 2010, What automated analyses of corpora can tell us about students’ writing skills, Journal of Writing Research, 2, 151, 10.17239/jowr-2010.02.02.4 · DOI
  18. Elliot, 2012, Placement of students into first-year writing courses, Research in the Teaching of English, 46, 285, 10.58680/rte201218457 · DOI
  19. Ferretti, 2000, The effects of an elaborated goal on the persuasive writing of students with learning disabilities and their normally achieving peers, Journal of Educational Psychology, 92, 694, 10.1037/0022-0663.92.4.694 · doi:10.1037/0022-0663.92.4.694 · Paper page
  20. Ferretti, 2007, Improving the argumentative writing of students with learning disabilities: descriptive and normative considerations, Reading and Writing Quarterly, 23, 267, 10.1080/10573560701277740 · DOI
  21. Ferretti, 2009, Do goals affect the structure of students’ argumentative writing strategies?, Journal of Educational Psychology, 101, 577, 10.1037/a0014702 · DOI · Paper page
  22. Golder, 1994, Argumentative text writing: developmental trends, Discourse Processes, 18, 187, 10.1080/01638539409544891 · DOI · Paper page
  23. Graesser, 2004, Coh-Metrix: analysis of text on cohesion and language, Behavioral Research Methods, Instruments and Computers, 36, 193, 10.3758/BF03195564 · DOI · Paper page
  24. Graham, 1999, Handwriting and spelling instruction for students with learning disabilities: a review, Learning Disability Quarterly, 22, 78, 10.2307/1511268 · DOI · Paper page
  25. Graham, 2014, Assessing the writing achievement of young struggling writers: application of generalizability theory, Learning Disability Quarterly (online first).
  26. Holtzman, 2005, Computerized assessment of dental student writing skills, Journal of Dental Education, 69, 285, 10.1002/j.0022-0337.2005.69.2.tb03915.x · doi:10.1002/j.0022-0337.2005.69.2.tb03915.x
  27. Hughes, K. L., & Scott-Clayton, J. (2011). Assessing developmental assessment in community colleges. Retrieved from CCRC Working Paper No. 19. New York. · DOI
  28. Kiuhara, 2012, The effectiveness of teaching 10th-grade students STOP, AIMS, and DARE for planning and drafting persuasive text, Exceptional Children, 78, 335, 10.1177/001440291207800305 · DOI
  29. Klobucar, 2013, Automated scoring in context: rapid assessment for placed students, Assessing Writing, 18, 62, 10.1016/j.asw.2012.10.001 · DOI
  30. MacArthur, 2009, Strategy instruction in writing for adult literacy learners, Reading and Writing: An Interdisciplinary Journal, 22, 1021, 10.1007/s11145-008-9142-x · DOI
  31. MacArthur, 2010, Instruction in a strategy for compare-contrast writing, Exceptional Children, 76, 438, 10.1177/001440291007600404 · DOI
  32. MacArthur, 2013, Self-regulated strategy instruction in developmental writing: a design research project, Community College Review, 41, 176, 10.1177/0091552113484580 · DOI · Paper page
  33. MacArthur, 2016, A multicomponent measure of writing motivation with basic college writers, Learning Disability Quarterly, 39, 31, 10.1177/0731948715583115 · DOI · Paper page
  34. Magliano, 2012, Computer-based assessment of student-constructed responses, Behavior Research Methods (Online), 44, 608, 10.3758/s13428-012-0211-3 · DOI
  35. Mason, 2013, Knowledge, writing, and language outcomes for a reading comprehension and writing intervention, Reading and Writing: An Interdisciplinary Journal, 26, 1133, 10.1007/s11145-012-9409-0 · DOI
  36. Mateos, 2008, Reading and writing to learn in secondary education: online processing activity and written products in summarizing and synthesizing tasks, Reading and Writing: An Interdisciplinary Journal, 21, 675, 10.1007/s11145-007-9086-6 · DOI
  37. McCarthy, 2007, Vocd: a theoretical and empirical evaluation, Language Testing, 24, 459, 10.1177/0265532207080767 · DOI · Paper page
  38. McNamara, 2013, Natural language processing in an intelligent writing strategy tutoring system, Behavior Research Methods (Online), 45, 499, 10.3758/s13428-012-0258-1 · DOI · Paper page
  39. McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. New York, NY: Cambridge University Press. · DOI · Paper page
  40. Miller, 2015, Evaluating undergraduate nursing students’ self-efficacy and competence in writing: effects of a writing intensive intervention, Nurse Education in Practice, 15, 174, 10.1016/j.nepr.2014.12.002 · DOI
  41. Newell, 2011, Teaching and learning: argumentative reading and writing: a review of research, Reading Research Quarterly, 46, 273, 10.1598/RRQ.46.3.4 · DOI · Paper page
  42. O’Neill, 2012, Creating the framework for success in postsecondary writing, College English, 74, 520, 10.58680/ce201220310 · DOI
  43. Olinghouse, 2008, Student- and instruction-level predictors of narrative writing in third-grade students, Reading and Writing: An Interdisciplinary Journal, 21, 3, 10.1007/s11145-007-9062-1 · DOI
  44. Olinghouse, 2009, The relationship between measures of vocabulary and narrative writing quality in second- and fourth-grade students, Reading and Writing: An Interdisciplinary Journal, 22, 545, 10.1007/s11145-008-9124-z · DOI · Paper page
  45. Parsad, B., & Lewis, L. (2003). Remedial education at degree-granting postsecondary institutions in fall 2000: Statistical analysis report (NCES 2004–010). Washington D.C.: U.S. Department of Education, National Center for Education Statistics. Retrieved from http://nces.ed.gov/pubs2004/2004010.pdf. · DOI
  46. Perelman, L. (2013). Critique of Mark D. Shermis & Ben Hamner, “Contrasting state-of-the-art automated scoring of essays: analysis” Journal of Writing Assessment, 6(1), not paginated. Available at http://www.journalofwritingassessment.org/article.php?article=69.
  47. Perin, 1993, Relationship between literacy gains and length of stay in basic education program for health care workers, Adult Basic Education, 3, 171
  48. Perin, 2003, The academic writing of community college remedial students: text and learner variables, Higher Education, 45, 19, 10.1023/A:1021237532056 · DOI
  49. Perin, 2013, A contextualized curricular supplement for developmental reading and writing, Journal of College Reading and Learning, 43, 8, 10.1080/10790195.2013.10850365 · DOI
  50. Ramineni, 2013, Validating automated essay scoring for online writing placement, Assessing Writing, 18, 40, 10.1016/j.asw.2012.10.005 · DOI
  51. Reilly, 2014, Evaluating the validity and applicability of automated essay scoring in two massive open online courses, International Review of Research in Open and Distance Learning, 15, 83, 10.19173/irrodl.v15i5.1857 · DOI
  52. Sampson, 2011, Argument-driven inquiry as a way to help students learn how to participate in scientific argumentation and craft written arguments: an exploratory study, Science Education, 95, 217, 10.1002/sce.20421 · DOI · Paper page
  53. Weigle, 2013, English language learners and automated scoring of essays: critical considerations, Assessing Writing, 18, 85, 10.1016/j.asw.2012.10.006 · DOI
  54. Westby, 2010, Summarizing expository texts, Topics in Language Disorders, 30, 275, 10.1097/TLD.0b013e3181ff5a88 · DOI
  55. Williamson, 2008, A text readability continuum for postsecondary readiness, Journal of Advanced Academics, 19, 602, 10.4219/jaa-2008-832 · DOI
  56. Wilson, 2016, Comparing the accuracy of different scoring methods for identifying sixth graders at risk of failing a state writing assessment, Assessing Writing, 27, 11, 10.1016/j.asw.2015.06.003 · DOI
  57. Wissinger, 2015, Effects of critical discussions on middle school students’ written historical arguments, Journal of Educational Psychology (online first).
  58. Wolfe, 2011, Argumentation across the curriculum, Written Communication, 28, 193, 10.1177/0741088311399236 · DOI · Paper page
  59. Zhang, 2015, A Coh-Metrix study of writings by majors of mechanic engineering in the vocational college, Theory and Practice in Language Studies, 5, 1929, 10.17507/tpls.0509.23 · doi:10.17507/tpls.0509.23

Cited by

See all 65

Metadata

Authors
Perin, D., & Lauterbach, M.
Venue
International Journal of Artificial Intelligence in Education
Published
2016-11-08
Volume
28
Issue
1
Pages
56-78
ISSN
  • 1560-4292 (ISSN-L)
  • 1560-4306
OpenAlex
W2552445905
Cited by
65
References
9
FWCI
2.21
Type
journal article
Language
en
Open access
hybrid
Licence
Funders
Bill and Melinda Gates Foundation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Assessing Text-Based Writing of Low-Skilled College Students. https://app.cochat.ai/papers/10.1007/s40593-016-0122-z
The paper
Perin, D., & Lauterbach, M. (2018). Assessing Text-Based Writing of Low-Skilled College Students. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-016-0122-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.