CoChat

Exploring the Feasibility of Using Writing Process Features to Assess Text Production Skills

Deane, Paul, Zhang, Mo · 2015 · ETS Research Report Series
Cited by 29

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Abstract

In this report, we examine the feasibility of characterizing writing performance using process features derived from a keystroke log. Using data derived from a set ofCBAL™writing assessments, we examine the following research questions: (a) How stable are the keystroke timing and process features across testing occasions? (b) How consistent are the patterns of feature–human correlation across genres and topics? (c) How accurately can we predict human ratings on writing fundamentals using a combination of the keystroke timing and process features, and what are the contributions of each feature to the reliable variance in the human ratings? (d) If we train a predictive model on one prompt, how well do its predictions generalize to the other prompts of the same or different genre? The results of the study indicate that keystroke log features vary considerably in stability across testing occasions and display somewhat different patterns of feature–human correlation across genres and topics. However, using the most stable features, we can obtain moderate to strong prediction of human essay scores, and those models generalize reasonably well across prompts though more strongly within than across writing genres.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/ets2.12071

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 37

  1. 10.1037/a0019318 · DOI · Paper page
  2. 10.1007/978-94-010-0804-4 · DOI · Paper page
  3. 10.4135/9780857021069.n3 · DOI
  4. Almond R. Deane P. Quinlan T. Wagner M. &Sydorenko T.(2012).A preliminary analysis of keystroke log data from a timed writing task(Research Report No. RR‐12‐23). Princeton NJ: Educational Testing Service.http://dx.doi.org/10.1002/j.2333-8504.2012.tb02305.x · DOI
  5. Alves R. A., 2012, Past, present, and future contributions of cognitive writing research to cognitive psychology, 389
  6. Andersson B., 2006, Combining keystroke logging with eye‐tracking, Writing and Digital Media, 17, 45
  7. Bennett R. E.(2011).CBAL:Results from piloting innovative K–12 assessments(Research Report No. RR‐11‐23). Princeton NJ: Educational Testing Service.http://dx.doi.org/10.1002/j.2333-8504.2011.tb02259.x · DOI
  8. 10.1007/978-1-4020-9964-9_3 · DOI · Paper page
  9. Bereiter C., 1987, The psychology of written composition
  10. Berninger V. W., 1996, Reading and writing acquisition: A developmental neuropsychological perspective
  11. 10.2307/1511269 · DOI · Paper page
  12. 10.1207/s1532690xci1202_2 · DOI · Paper page
  13. 10.1177/0741088303253572 · DOI
  14. 10.1177/0741088312451109 · DOI
  15. Deane P.(2014).Using writing process and product features to assess writing quality and explore how those features relate to other literacy tasks(Research Report No. RR‐14‐03). Princeton NJ: Educational Testing Service.http://dx.doi.org/10.1002/ets2.12002 · DOI
  16. 10.17239/jowr-2010.02.02.4 · DOI
  17. Deane P. Quinlan T. &Kostin I.(2011).Automated scoring within a developmental cognitive model of writing proficiency(Research Report No. RR‐11‐16). Princeton NJ: Educational Testing Service.http://dx.doi.org/10.1002/j.2333-8504.2011.tb02252.x · DOI
  18. 10.2307/356600 · doi:10.2307/356600 · Paper page
  19. 10.1037/0022-0663.89.1.170 · DOI · Paper page
  20. 10.1177/0741088312451260 · DOI · Paper page
  21. Hayes J. R., 1980, Cognitive processes in writing, 3
  22. Jakobsen A. L., 2006, Studies in writing: Vol. 18. Computer keystroke logging and writing, 95, 10.1163/9780080460932_007 · DOI
  23. Kaufer D. S., 1986, Composing written sentences, Research in the Teaching of English, 20, 121, 10.58680/rte198615612 · DOI · Paper page
  24. 10.17239/jowr-2008.01.01.1 · DOI · Paper page
  25. Leijten M. Hoste V. Van Waes L. Macken L. &Van Horenbeeck E.(2012 April). From character to word level: Enabling the linguistic analyses of Inputlog process data. In M. Piotrowski C. Mahlow & R. Dale (Eds.) Proceedings of the Second Workshop on Computational Linguistics and Writing (CLW 2012).Linguistic and Cognitive Aspects of Document Creation and Document Engineering(pp. 1–8). Stroudsburg PA: Association for Computational Linguistics.
  26. Leijten M., 2006, Studies in writing: Vol. 18. Computer key‐stroke logging and writing: Methods and applications, 73, 10.1163/9780080460932_006 · DOI
  27. 10.1037/a0031391 · DOI · Paper page
  28. 10.1207/S15326985EP3501_3 · DOI · Paper page
  29. 10.1177/136216880000400203 · DOI
  30. Perl S., 1979, The composing processes of unskilled college writers, Research in the Teaching of English, 13, 317, 10.58680/rte201117867 · DOI · Paper page
  31. Strömqvist S., 2006, Studies in writing: Vol. 18. Computer keystroke‐logging and writing, 45, 10.1163/9780080460932_005 · DOI
  32. 10.1007/s11409-011-9072-x · DOI
  33. Torrance M., 2006, Handbook of writing research, 67
  34. 10.1080/01443410120090777 · DOI
  35. Waes L., 2006, Logging writing processes with Inputlog, Writing and Digital Media, 17, 158
  36. Wengelin Å., 2006, Studies in writing: Vol. 18. Computer keystroke‐logging and writing, 107, 10.1163/9780080460932_008 · DOI · Paper page
  37. Zhang M. &Deane P.(in press).Process features in writing:Internal structure and incremental value over product features. ETS Research Report Series.

Cited by

Metadata

Authors
Deane, P., & Zhang, M.
Affiliations
Educational Testing Service Princeton NJ
Venue
ETS Research Report Series
Publisher
Wiley
Published
2015-07-08
Volume
2015
Issue
2
Pages
1-16
ISSN
2330-8516
Cited by
29
References
5
Language
en

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.