Exploring the Feasibility of Using Writing Process Features to Assess Text Production Skills
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
In this report, we examine the feasibility of characterizing writing performance using process features derived from a keystroke log. Using data derived from a set ofCBAL™writing assessments, we examine the following research questions: (a) How stable are the keystroke timing and process features across testing occasions? (b) How consistent are the patterns of feature–human correlation across genres and topics? (c) How accurately can we predict human ratings on writing fundamentals using a combination of the keystroke timing and process features, and what are the contributions of each feature to the reliable variance in the human ratings? (d) If we train a predictive model on one prompt, how well do its predictions generalize to the other prompts of the same or different genre? The results of the study indicate that keystroke log features vary considerably in stability across testing occasions and display somewhat different patterns of feature–human correlation across genres and topics. However, using the most stable features, we can obtain moderate to strong prediction of human essay scores, and those models generalize reasonably well across prompts though more strongly within than across writing genres.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The variable selection was based on previous studies (Aben, Van den Broek, Vandermeulen, Van Steendam, & Rijlaarsdam, 2017;Deane & Zhang, 2015;Leijten, Bernolet, et al., 2019;Van Waes & Leijten, 2015;Vandermeulen, De Maeyer, et al., 2020;Vandermeulen, Van Steendam, van den Broek, & Rijlaarsdam, 2020;Zhang & Deane, 2015).”
From Reporting Writing Process Feedback in the Classroom. Using Keystroke Logging Data to Reflect on Writing Processes (2020) · cites this paper as Deane & Zhang, 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A capacity theory of writing: Working memory in composition
McCutchen, D. (1996) · Educational Psychology Review · doi:10.1007/bf01464076 · Cited by 654 · cited 1×
“Unskilled writers may be restricted by limitations in a subprocess, such as transcription fluency, which may limit capacity to learn/exercise other writing skills (Berninger, 1996; Flower & Hayes, 1981; Kellogg, 2001, 2008; McCutchen, 1996, 2000).”
BackgroundKeystroke Logging in Writing Research: Using Inputlog to Analyze and Visualize Writing Processes
Leijten, M., & Van Waes, L. (2013) · Written Communication · doi:10.1177/0741088313491692 · Cited by 513
Competition for working memory among writing processes
Kellogg, R. T. (2001) · The American Journal of Psychology · doi:10.2307/1423513 · Cited by 199
Beauvais, C., Olive, T., & Passerault, J.-M. (2011) · Journal of Educational Psychology · doi:10.1037/a0022545 · Cited by 159
Keystroke Analysis: Reflections on Procedures and Measures
Baaijen, V. M., Galbraith, D., & de Glopper, K. (2012) · Written Communication · doi:10.1177/0741088312451108 · Cited by 124
Research works printed but not linked: 37
- 10.1037/a0019318 · DOI · Paper page
- 10.1007/978-94-010-0804-4 · DOI · Paper page
- 10.4135/9780857021069.n3 · DOI
- Almond R. Deane P. Quinlan T. Wagner M. &Sydorenko T.(2012).A preliminary analysis of keystroke log data from a timed writing task(Research Report No. RR‐12‐23). Princeton NJ: Educational Testing Service.http://dx.doi.org/10.1002/j.2333-8504.2012.tb02305.x · DOI
- Alves R. A., 2012, Past, present, and future contributions of cognitive writing research to cognitive psychology, 389
- Andersson B., 2006, Combining keystroke logging with eye‐tracking, Writing and Digital Media, 17, 45
- Bennett R. E.(2011).CBAL:Results from piloting innovative K–12 assessments(Research Report No. RR‐11‐23). Princeton NJ: Educational Testing Service.http://dx.doi.org/10.1002/j.2333-8504.2011.tb02259.x · DOI
- 10.1007/978-1-4020-9964-9_3 · DOI · Paper page
- Bereiter C., 1987, The psychology of written composition
- Berninger V. W., 1996, Reading and writing acquisition: A developmental neuropsychological perspective
- 10.2307/1511269 · DOI · Paper page
- 10.1207/s1532690xci1202_2 · DOI · Paper page
- 10.1177/0741088303253572 · DOI
- 10.1177/0741088312451109 · DOI
- Deane P.(2014).Using writing process and product features to assess writing quality and explore how those features relate to other literacy tasks(Research Report No. RR‐14‐03). Princeton NJ: Educational Testing Service.http://dx.doi.org/10.1002/ets2.12002 · DOI
- 10.17239/jowr-2010.02.02.4 · DOI
- Deane P. Quinlan T. &Kostin I.(2011).Automated scoring within a developmental cognitive model of writing proficiency(Research Report No. RR‐11‐16). Princeton NJ: Educational Testing Service.http://dx.doi.org/10.1002/j.2333-8504.2011.tb02252.x · DOI
- 10.2307/356600 · doi:10.2307/356600 · Paper page
- 10.1037/0022-0663.89.1.170 · DOI · Paper page
- 10.1177/0741088312451260 · DOI · Paper page
- Hayes J. R., 1980, Cognitive processes in writing, 3
- Jakobsen A. L., 2006, Studies in writing: Vol. 18. Computer keystroke logging and writing, 95, 10.1163/9780080460932_007 · DOI
- Kaufer D. S., 1986, Composing written sentences, Research in the Teaching of English, 20, 121, 10.58680/rte198615612 · DOI · Paper page
- 10.17239/jowr-2008.01.01.1 · DOI · Paper page
- Leijten M. Hoste V. Van Waes L. Macken L. &Van Horenbeeck E.(2012 April). From character to word level: Enabling the linguistic analyses of Inputlog process data. In M. Piotrowski C. Mahlow & R. Dale (Eds.) Proceedings of the Second Workshop on Computational Linguistics and Writing (CLW 2012).Linguistic and Cognitive Aspects of Document Creation and Document Engineering(pp. 1–8). Stroudsburg PA: Association for Computational Linguistics.
- Leijten M., 2006, Studies in writing: Vol. 18. Computer key‐stroke logging and writing: Methods and applications, 73, 10.1163/9780080460932_006 · DOI
- 10.1037/a0031391 · DOI · Paper page
- 10.1207/S15326985EP3501_3 · DOI · Paper page
- 10.1177/136216880000400203 · DOI
- Perl S., 1979, The composing processes of unskilled college writers, Research in the Teaching of English, 13, 317, 10.58680/rte201117867 · DOI · Paper page
- Strömqvist S., 2006, Studies in writing: Vol. 18. Computer keystroke‐logging and writing, 45, 10.1163/9780080460932_005 · DOI
- 10.1007/s11409-011-9072-x · DOI
- Torrance M., 2006, Handbook of writing research, 67
- 10.1080/01443410120090777 · DOI
- Waes L., 2006, Logging writing processes with Inputlog, Writing and Digital Media, 17, 158
- Wengelin Å., 2006, Studies in writing: Vol. 18. Computer keystroke‐logging and writing, 107, 10.1163/9780080460932_008 · DOI · Paper page
- Zhang M. &Deane P.(in press).Process features in writing:Internal structure and incremental value over product features. ETS Research Report Series.
Cited by
Vandermeulen, N., Leijten, M., & Van Waes, L. (2020) · Journal of Writing Research · doi:10.17239/jowr-2020.12.01.05 · Cited by 87
Modifying AI, Enhancing Essays: How Active Engagement with Generative AI Boosts Writing Quality
Yang, K., Raković, M., Liang, Z., Yan, L., Zeng, Z., Fan, Y., Gašević, D., & Chen, G. (2025) · Proceedings of the 15th International Learning Analytics and Knowledge Conference · doi:10.1145/3706468.3706544 · Cited by 26
Metadata
- Authors
- Deane, P., & Zhang, M.
- Affiliations
- Educational Testing Service Princeton NJ
- Venue
- ETS Research Report Series
- Publisher
- Wiley
- Published
- 2015-07-08
- Volume
- 2015
- Issue
- 2
- Pages
- 1-16
- ISSN
- 2330-8516
- Cited by
- 29
- References
- 5
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Exploring the Feasibility of Using Writing Process Features to Assess Text Production Skills. https://app.cochat.ai/papers/10.1002/ets2.12071
- The paper
- Deane, P., & Zhang, M. (2015). Exploring the Feasibility of Using Writing Process Features to Assess Text Production Skills. ETS Research Report Series. https://doi.org/10.1002/ets2.12071
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.