Assessing Text-Based Writing of Low-Skilled College Students
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
The problem of poor writing skills at the postsecondary level is a large and troubling one. This study investigated the writing skills of low-skilled adults attending college developmental education courses by determining whether variables from an automated scoring system were predictive of human scores on writing quality rubrics. The human-scored measures were a holistic quality rating for a persuasive essay and an analytic quality score for a written summary. Both writing samples were based on text on psychology and sociology topics related to content taught at the introductory undergraduate level. The study is a modified replication of McNamara et al. ( Written Communication, 27 (1), 57–86 2010 ), who identified several Coh-Metrix variables from five linguistic classes that reliably predicted group membership (high versus low proficiency) using human quality scores on persuasive essays written by average-achieving college students. When discriminant analyses and ANOVAs failed to replicate the McNamara et al. ( Written Communication, 27 (1), 57–86 2010 ) findings, the current study proceeded to analyze all of the variables in the five Coh-Metrix classes. This larger analysis identified 10 variables that predicted human-scored writing proficiency. Essay and summary scores were predicted by different automated variables. Implications for instruction and future use of automated scoring to understand the writing of low-skilled adults are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Referral, enrollment, and completion in developmental education sequences in community colleges
Bailey, T., Jeong, D. W., & Cho, S.-W. (2010) · Economics of Education Review · doi:10.1016/j.econedurev.2009.09.002 · Cited by 845 · cited 3×
“, low-skilled adults who, despite having completed secondary education, were underprepared for the college curriculum (Bailey et al. 2010).”
BackgroundInfluentialLinguistic Features of Writing Quality
McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010) · Written Communication · doi:10.1177/0741088309351547 · Cited by 570
Predicting second language writing proficiency: the roles of cohesion and linguistic sophistication
Crossley, S. A., & McNamara, D. S. (2012) · Journal of Research in Reading · doi:10.1111/j.1467-9817.2010.01449.x · Cited by 331 · cited 2×
“The Btypetoken ratio^ measure divides the number of unique words, i.e., types, in the writing sample, by the total word count, i.e. tokens (Crossley and McNamara 2011) as a measure of vocabulary diversity.”
BackgroundPromoting Argument-Counterargument Integration in Students' Writing
Nussbaum, E. M., & Schraw, G. (2007) · The Journal of Experimental Education · doi:10.3200/jexe.76.1.59-92 · Cited by 279
The relationship between vocabulary and writing quality in three genres
Olinghouse, N. G., & Wilson, J. (2013) · Reading and Writing · doi:10.1007/s11145-012-9392-5 · Cited by 263
What High-Achieving Low-Income Students Know About College
Hoxby, C. M., & Turner, S. (2015) · American Economic Review · doi:10.1257/aer.p20151027 · Cited by 239
Argumentation Schema and the Myside Bias in Written Argumentation
Wolfe, C. R., Britt, M. A., & Butler, J. A. (2009) · Written Communication · doi:10.1177/0741088309333019 · Cited by 173
The Effects of Vocabulary Breadth and Depth on English Reading
Li, M., & Kirby, J. R. (2014) · Applied Linguistics · doi:10.1093/applin/amu007 · Cited by 172 · cited 1×
“Summarizing information from text requires both reading comprehension and writing skills (Mateos et al. 2008; Shanahan 2016) and in fact, summary-writing is often used as a measure of reading comprehension (Li and Kirby 2016; Mason et al. 2013).”
BackgroundSelf-regulated strategy instruction in college developmental writing
MacArthur, C. A., Philippakos, Z. A., & Ianetta, M. (2015) · Journal of Educational Psychology · doi:10.1037/edu0000011 · Cited by 135 · cited 8×
“A seven-point holistic scale (based on MacArthur et al. 2015) was used for the essay and a 16-point analytic scale (based on Westby et al. 2010) for the summary.”
BackgroundMethodInfluential
Research works printed but not linked: 59
- Abdi, 2016, An automated summarization assessment algorithm for identifying summarizing strategies, PloS One, 11, 1, 10.1371/journal.pone.0145809 · DOI
- Acker, 2008, Preparing high school students for college-level writing: using an e-portfolio to support a successful transition, The Journal of General Education, 57, 1, 10.2307/27798087 · doi:10.2307/27798087
- Brace, N., Kemp, R., & Snelgar, R. (2012). SPSS for psychologists: a guide to data analysis using (5th ed.). New York, NY: Routledge.
- Bridgeman, 1984, Survey of academic writing tasks, Written Communication, 1, 247, 10.1177/0741088384001002004 · DOI · Paper page
- Brown, 1983, Macrorules for summarizing texts: the development of expertise, Journal of Verbal Learning and Verbal Behavior, 22, 1, 10.1016/S0022-5371(83)80002-4 · DOI · Paper page
- Carretti, 2016, Oral and written expression in children with reading comprehension difficulties, Journal of Learning Disabilities, 49, 65, 10.1177/0022219414528539 · DOI
- Cohen, A. M., Brawer, F. B., & Kisker, C. B. (2013). The American community college (6th ed.). Boston, MA: Wiley.
- Comer, 2016, Adventuring into MOOC writing assessment: challenges, results, and possibilities, College Composition and Communication, 67, 318, 10.58680/ccc201628063 · DOI
- Crossley, 2009, Computational assessment of lexical differences in L1 and L2 writing, Journal of Second Language Writing, 18, 119, 10.1016/j.jslw.2009.02.002 · DOI · Paper page
- Crossley, 2011, The development of writing proficiency as a function of grade level: a linguistic analysis, Written Communication, 28, 282, 10.1177/0741088311410188 · DOI · Paper page
- Crossley, 2012, Predicting the proficiency level of language learners using lexical indices, Language Testing, 29, 243, 10.1177/0265532211419331 · DOI · Paper page
- Crossley, 2016, The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality, Journal of Second Language Writing, 32, 1, 10.1016/j.jslw.2016.01.003 · DOI · Paper page
- Danzak, 2011, The integration of lexical, syntactic, and discourse features in bilingual adolescents’ writing: an exploratory approach, Language, Speech, and Hearing Services in Schools, 42, 491, 10.1044/0161-1461(2011/10-0063) · doi:10.1044/0161-1461(2011/10-0063
- De La Paz, 2005, Effects of historical reasoning instruction and writing strategy mastery in culturally and academically diverse middle school classrooms, Journal of Educational Psychology, 97, 139, 10.1037/0022-0663.97.2.139 · DOI · Paper page
- De La Paz, 2012, Adolescents’ disciplinary use of evidence, argumentative strategies, and organizational structure in writing about historical controversies, Written Communication, 29, 412, 10.1177/0741088312461591 · DOI
- Deane, 2013, On the relation between automated essay scoring and modern views of the writing construct, Assessing Writing, 18, 7, 10.1016/j.asw.2012.10.002 · DOI · Paper page
- Deane, 2010, What automated analyses of corpora can tell us about students’ writing skills, Journal of Writing Research, 2, 151, 10.17239/jowr-2010.02.02.4 · DOI
- Elliot, 2012, Placement of students into first-year writing courses, Research in the Teaching of English, 46, 285, 10.58680/rte201218457 · DOI
- Ferretti, 2000, The effects of an elaborated goal on the persuasive writing of students with learning disabilities and their normally achieving peers, Journal of Educational Psychology, 92, 694, 10.1037/0022-0663.92.4.694 · doi:10.1037/0022-0663.92.4.694 · Paper page
- Ferretti, 2007, Improving the argumentative writing of students with learning disabilities: descriptive and normative considerations, Reading and Writing Quarterly, 23, 267, 10.1080/10573560701277740 · DOI
- Ferretti, 2009, Do goals affect the structure of students’ argumentative writing strategies?, Journal of Educational Psychology, 101, 577, 10.1037/a0014702 · DOI · Paper page
- Golder, 1994, Argumentative text writing: developmental trends, Discourse Processes, 18, 187, 10.1080/01638539409544891 · DOI · Paper page
- Graesser, 2004, Coh-Metrix: analysis of text on cohesion and language, Behavioral Research Methods, Instruments and Computers, 36, 193, 10.3758/BF03195564 · DOI · Paper page
- Graham, 1999, Handwriting and spelling instruction for students with learning disabilities: a review, Learning Disability Quarterly, 22, 78, 10.2307/1511268 · DOI · Paper page
- Graham, 2014, Assessing the writing achievement of young struggling writers: application of generalizability theory, Learning Disability Quarterly (online first).
- Holtzman, 2005, Computerized assessment of dental student writing skills, Journal of Dental Education, 69, 285, 10.1002/j.0022-0337.2005.69.2.tb03915.x · doi:10.1002/j.0022-0337.2005.69.2.tb03915.x
- Hughes, K. L., & Scott-Clayton, J. (2011). Assessing developmental assessment in community colleges. Retrieved from CCRC Working Paper No. 19. New York. · DOI
- Kiuhara, 2012, The effectiveness of teaching 10th-grade students STOP, AIMS, and DARE for planning and drafting persuasive text, Exceptional Children, 78, 335, 10.1177/001440291207800305 · DOI
- Klobucar, 2013, Automated scoring in context: rapid assessment for placed students, Assessing Writing, 18, 62, 10.1016/j.asw.2012.10.001 · DOI
- MacArthur, 2009, Strategy instruction in writing for adult literacy learners, Reading and Writing: An Interdisciplinary Journal, 22, 1021, 10.1007/s11145-008-9142-x · DOI
- MacArthur, 2010, Instruction in a strategy for compare-contrast writing, Exceptional Children, 76, 438, 10.1177/001440291007600404 · DOI
- MacArthur, 2013, Self-regulated strategy instruction in developmental writing: a design research project, Community College Review, 41, 176, 10.1177/0091552113484580 · DOI · Paper page
- MacArthur, 2016, A multicomponent measure of writing motivation with basic college writers, Learning Disability Quarterly, 39, 31, 10.1177/0731948715583115 · DOI · Paper page
- Magliano, 2012, Computer-based assessment of student-constructed responses, Behavior Research Methods (Online), 44, 608, 10.3758/s13428-012-0211-3 · DOI
- Mason, 2013, Knowledge, writing, and language outcomes for a reading comprehension and writing intervention, Reading and Writing: An Interdisciplinary Journal, 26, 1133, 10.1007/s11145-012-9409-0 · DOI
- Mateos, 2008, Reading and writing to learn in secondary education: online processing activity and written products in summarizing and synthesizing tasks, Reading and Writing: An Interdisciplinary Journal, 21, 675, 10.1007/s11145-007-9086-6 · DOI
- McCarthy, 2007, Vocd: a theoretical and empirical evaluation, Language Testing, 24, 459, 10.1177/0265532207080767 · DOI · Paper page
- McNamara, 2013, Natural language processing in an intelligent writing strategy tutoring system, Behavior Research Methods (Online), 45, 499, 10.3758/s13428-012-0258-1 · DOI · Paper page
- McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. New York, NY: Cambridge University Press. · DOI · Paper page
- Miller, 2015, Evaluating undergraduate nursing students’ self-efficacy and competence in writing: effects of a writing intensive intervention, Nurse Education in Practice, 15, 174, 10.1016/j.nepr.2014.12.002 · DOI
- Newell, 2011, Teaching and learning: argumentative reading and writing: a review of research, Reading Research Quarterly, 46, 273, 10.1598/RRQ.46.3.4 · DOI · Paper page
- O’Neill, 2012, Creating the framework for success in postsecondary writing, College English, 74, 520, 10.58680/ce201220310 · DOI
- Olinghouse, 2008, Student- and instruction-level predictors of narrative writing in third-grade students, Reading and Writing: An Interdisciplinary Journal, 21, 3, 10.1007/s11145-007-9062-1 · DOI
- Olinghouse, 2009, The relationship between measures of vocabulary and narrative writing quality in second- and fourth-grade students, Reading and Writing: An Interdisciplinary Journal, 22, 545, 10.1007/s11145-008-9124-z · DOI · Paper page
- Parsad, B., & Lewis, L. (2003). Remedial education at degree-granting postsecondary institutions in fall 2000: Statistical analysis report (NCES 2004–010). Washington D.C.: U.S. Department of Education, National Center for Education Statistics. Retrieved from http://nces.ed.gov/pubs2004/2004010.pdf. · DOI
- Perelman, L. (2013). Critique of Mark D. Shermis & Ben Hamner, “Contrasting state-of-the-art automated scoring of essays: analysis” Journal of Writing Assessment, 6(1), not paginated. Available at http://www.journalofwritingassessment.org/article.php?article=69.
- Perin, 1993, Relationship between literacy gains and length of stay in basic education program for health care workers, Adult Basic Education, 3, 171
- Perin, 2003, The academic writing of community college remedial students: text and learner variables, Higher Education, 45, 19, 10.1023/A:1021237532056 · DOI
- Perin, 2013, A contextualized curricular supplement for developmental reading and writing, Journal of College Reading and Learning, 43, 8, 10.1080/10790195.2013.10850365 · DOI
- Ramineni, 2013, Validating automated essay scoring for online writing placement, Assessing Writing, 18, 40, 10.1016/j.asw.2012.10.005 · DOI
- Reilly, 2014, Evaluating the validity and applicability of automated essay scoring in two massive open online courses, International Review of Research in Open and Distance Learning, 15, 83, 10.19173/irrodl.v15i5.1857 · DOI
- Sampson, 2011, Argument-driven inquiry as a way to help students learn how to participate in scientific argumentation and craft written arguments: an exploratory study, Science Education, 95, 217, 10.1002/sce.20421 · DOI · Paper page
- Weigle, 2013, English language learners and automated scoring of essays: critical considerations, Assessing Writing, 18, 85, 10.1016/j.asw.2012.10.006 · DOI
- Westby, 2010, Summarizing expository texts, Topics in Language Disorders, 30, 275, 10.1097/TLD.0b013e3181ff5a88 · DOI
- Williamson, 2008, A text readability continuum for postsecondary readiness, Journal of Advanced Academics, 19, 602, 10.4219/jaa-2008-832 · DOI
- Wilson, 2016, Comparing the accuracy of different scoring methods for identifying sixth graders at risk of failing a state writing assessment, Assessing Writing, 27, 11, 10.1016/j.asw.2015.06.003 · DOI
- Wissinger, 2015, Effects of critical discussions on middle school students’ written historical arguments, Journal of Educational Psychology (online first).
- Wolfe, 2011, Argumentation across the curriculum, Written Communication, 28, 193, 10.1177/0741088311399236 · DOI · Paper page
- Zhang, 2015, A Coh-Metrix study of writings by majors of mechanic engineering in the vocational college, Theory and Practice in Language Studies, 5, 1929, 10.17507/tpls.0509.23 · doi:10.17507/tpls.0509.23
Cited by
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-019-0171-0 · Cited by 7,242
The impact of artificial intelligence on learner–instructor interaction in online learning
Seo, K., Tang, J., Roll, I., Fels, S., & Yoon, D. (2021) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-021-00292-9 · Cited by 788
Empowering Educators to Be AI-Ready
Luckin, R., Cukurova, M., Kent, C., & du Boulay, B. (2022) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2022.100076 · Cited by 422
“Automated grading systems can be found in a range of disciplines, such as biology, medicine, business studies and English as a Second Language (ESL), mostly in use in higher education contexts (Perin & Lauterbach, 2018).”
BackgroundAutomated feedback and writing: a multi-level meta-analysis of effects on students' performance
Fleckenstein, J., Liebenow, L. W., & Meyer, J. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1162454 · Cited by 145
“Various studies have demonstrated the quality of AWE assessment (Shermis, 2014; Perin and Lauterbach, 2018; Rupp et al., 2019; Zawacki-Richter et al., 2019).”
BackgroundAnalysis of the scientific literature's abstract writing style and citations
Hu, H., Wang, D., & Deng, S. (2021) · Online Information Review · doi:10.1108/oir-05-2020-0188 · Cited by 16
Metadata
- Authors
- Perin, D., & Lauterbach, M.
- Venue
- International Journal of Artificial Intelligence in Education
- Published
- 2016-11-08
- Volume
- 28
- Issue
- 1
- Pages
- 56-78
- ISSN
- 1560-4292 (ISSN-L)
- 1560-4306
- OpenAlex
- W2552445905
- Cited by
- 65
- References
- 9
- FWCI
- 2.21
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0/ (vor, 2982 d)
- Funders
- Bill and Melinda Gates Foundation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Assessing Text-Based Writing of Low-Skilled College Students. https://app.cochat.ai/papers/10.1007/s40593-016-0122-z
- The paper
- Perin, D., & Lauterbach, M. (2018). Assessing Text-Based Writing of Low-Skilled College Students. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-016-0122-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.