Does size matter? Authorship attribution, small samples, big problem
The DOI matches its Crossref record; 33 of the 35 references it cites were found in a registry or an index.
Abstract
The aim of this study is to find such a minimal size of text samples for authorship attribution that would provide stable results independent of random noise. A few controlled tests for different sample lengths, languages, and genres are discussed and compared. Depending on the corpus used, the minimal sample length varied from 2,500 words (Latin prose) to 5,000 or so words (in most cases, including English, German, Polish, and Hungarian novels). Another observation is connected with the method of sampling: contrary to common sense, randomly excerpted ‘bags of words’ turned out to be much more effective than the classical solution, i.e. using original sequences of words (‘passages’) of desired size. Although the tests have been performed using the Delta method (Burrows, J.F. (2002). ‘Delta’: a measure of stylistic difference and a guide to likely authorship. Literary and Linguistic Computing, 17(3): 267–87) applied to the most frequent words, some additional experiments have been conducted for support vector machines and k-NN applied to most frequent words, character 3-grams, character 4-grams, and parts-of-speech-tag 3-grams. Despite significant differences in overall attributive success rate between particular methods and/or style markers, the minimal amount of textual data needed for reliable authorship attribution turned out to be method-independent.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It is worth knowing that text length is an important factor for a reliable outcome to be provided. When employing texts with a mean length clearly below 5,000 words, reliable authorship attribution is rather difficult to achieve (Eder 2015; Savoy 2018).”
From Is Luke the author of a Gospel and the Acts of the Apostles? (2026) · cites this paper as Eder 2015
In the citing paper
“LLMs are also effective with shorter texts, reducing the necessity for long inputs to derive meaningful representations [Eder, 2015].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Eder, 2015
In the citing paper
“Short or low-quality texts, such as social media posts, are often unreliable, complicating accurate attribution [Eder, 2015;Theophilo et al., 2021].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Eder, 2015
Citations
Cites
A survey of modern authorship attribution methods
Stamatatos, E. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21001 · Cited by 1,176
Delta: A Measure of Stylistic Difference and a Guide to Likely Authorship
Burrows, J. (2002) · Literary and Linguistic Computing · doi:10.1093/llc/17.3.267 · Cited by 627 · cited 1×
“These new samples were analyzed using the classical Delta method as developed by Burrows (2002); the percentage of attributive success was regarded as a measure of effectiveness of the current sample length.”
MethodComputational methods in authorship attribution
Koppel, M., Schler, J., & Argamon, S. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20961 · Cited by 528
How Variable May a Constant be? Measures of Lexical Richness in Perspective
Tweedie, F. J., & Baayen, R. H. (1998) · Computers and the Humanities · doi:10.1023/a:1001749303137 · Cited by 423
Computation into criticism : a study of Jane Austen's novels and an experiment in method
Burrows, J. F. (1987) · Medical Entomology and Zoology · Cited by 279
The State of Authorship Attribution Studies: Some Problems and Solutions
Rudman, J. (1997) · Computers and the Humanities · doi:10.1023/a:1001018624850 · Cited by 243
Attributing Authorship: An Introduction
Love, H. (2002) · doi:10.1017/cbo9780511483165 · Cited by 235
Bigrams of Syntactic Labels for Authorship Discrimination of Short Texts
Hirst, G., & Feiguina, O. (2007) · Literary and Linguistic Computing · doi:10.1093/llc/fqm023 · Cited by 227
A comparative study of machine learning methods for authorship attribution
Jockers, M. L., & Witten, D. M. (2010) · Literary and Linguistic Computing · doi:10.1093/llc/fqq001 · Cited by 188
Attributing authorship an introduction
Love, H. (2002) · Cited by 181
Interpreting Burrows's Delta: Geometric and Probabilistic Foundations
Argamon, S. (2007) · Literary and Linguistic Computing · doi:10.1093/llc/fqn003 · Cited by 174
Effective and Scalable Authorship Attribution Using Function Words
Zhao, Y., & Zobel, J. (2005) · Lecture Notes in Computer Science · doi:10.1007/11562382_14 · Cited by 160
The effect of author set size and data size in authorship attribution
Luyckx, K., & Daelemans, W. (2011) · Literary and Linguistic Computing · doi:10.1093/llc/fqq013 · Cited by 157
A Controlled-corpus Experiment in Authorship Identification by Cross-entropy
Juola, P. (2005) · Literary and Linguistic Computing · doi:10.1093/llc/fqi024 · Cited by 146
Author identification: Using text sampling to handle the class imbalance problem
Stamatatos, E. (2007) · Information Processing & Management · doi:10.1016/j.ipm.2007.05.012 · Cited by 137
Style-markers in authorship attribution : a cross-language study of the authorial fingerprint
Eder, M. (2011) · Jagiellonian University Repository (Jagiellonian University) · Cited by 134
Self-citationDeeper Delta across genres and languages: do we really need the most frequent words?
Rybicki, J., & Eder, M. (2011) · Literary and Linguistic Computing · doi:10.1093/llc/fqr031 · Cited by 119
Self-citationJockers, M. L., Witten, D. M., & Criddle, C. S. (2008) · Literary and Linguistic Computing · doi:10.1093/llc/fqn040 · Cited by 84
Multivariate Analysis and the Study of Style Variation
Hoover, D. L. (2003) · Literary and Linguistic Computing · doi:10.1093/llc/18.4.341 · Cited by 77
Statistical Stylistics and Authorship Attribution: an Empirical Investigation
Hoover, D. L. (2001) · Literary and Linguistic Computing · doi:10.1093/llc/16.4.421 · Cited by 72 · cited 1×
“This effect might be due to the obvious differences in word distribution between narrative and dialogue parts in novels (cf. Hoover, 2001); however, the same effect was equally strong for poetry (Latin and English) and non-literary prose (Latin).”
BackgroundMind your corpus: systematic errors in authorship attribution
Eder, M. (2013) · Literary and Linguistic Computing · doi:10.1093/llc/fqt039 · Cited by 71
Self-citationStylistic Analysis and Authorship Studies
Craig, H. (2007) · A Companion to Digital Humanities · doi:10.1002/9780470999875.ch20 · Cited by 66
Digital Humanities 2016: Conference Abstracts
Stokes, P. A., Brookes, S., Noël, G., Davies, J. R., Webber, T., Broun, D., Taylor, A., & Tucker, J. (2016) · Cited by 65
Eder, M., & Rybicki, J. (2012) · Literary and Linguistic Computing · doi:10.1093/llc/fqs036 · Cited by 59
Self-citationA Widow and her Soldier: Stylometry and the American Civil War
Holmes, D. I., Gordon, L. J., & Wilson, C. (2001) · Literary and Linguistic Computing · doi:10.1093/llc/16.4.403 · Cited by 57
How to Handle Small Samples: Bootstrap and Bayesian Methods in the Analysis of Linguistic Change
Hinneburg, A., Mannila, H., Kaislaniemi, S., Nevalainen, T., & Raumolin‐Brunberg, H. (2006) · Literary and Linguistic Computing · doi:10.1093/llc/fqm006 · Cited by 55
Scalability Issues in Authorship Attribution
Luyckx, K. (2011) · Cited by 51
Non-traditional Authorship Attribution Studies in the Historia Augusta: Some Caveats
Rudman, J. (1998) · Literary and Linguistic Computing · doi:10.1093/llc/13.3.151 · Cited by 45
Cherry Picking in Nontraditional Authorship Attribution Studies
Rudman, J. (2003) · CHANCE · doi:10.1080/09332480.2003.10554845 · Cited by 38
Rybicki, J. (2005) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqh051 · Cited by 35
Digital Humanities 2023: Book of Abstracts
Baillot, A., Tasovac, T., Scholger, W., & Vogeler, G. (2023) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.5281/zenodo.7961821 · Cited by 20
Reynolds, N., Schaalje, G. B., & Hilton, J. (2012) · Literary and Linguistic Computing · doi:10.1093/llc/fqs020 · Cited by 11
Goldsmith's contributions to the 'Critical Review': a supplement
Dixon, P., & Mannion, D. (2012) · Literary and Linguistic Computing · doi:10.1093/llc/fqs021 · Cited by 4
Research works printed but not linked: 4
- Eder, 2009, PCA, Delta, JGAAP and Polish poetry of the 16th and the 17th centuries: who wrote the dirty stuff?, 242
- Mikros, 2009, Content words in authorship attribution: an evaluation of stylometric features in a literary corpus, 61
- Mautner, 1976, Maxim(e)s, sentences, Fragmente, Aphorismen, 399
- Rybicki, 2008, Does size matter? A re-examination of a time-proven method, 184
Cited by
Authorship Attribution for Social Media Forensics
Rocha, A., Scheirer, W. J., Forstall, C. W., Cavalcante, T., Theophilo, A., Shen, B., Carvalho, A. R. B., & Stamatatos, E. (2017) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2016.2603960 · Cited by 222
“How small can we go before the signal becomes lost in the noise? Eder examined this problem in a method-independent manner [49], and determined that for modern languages, the minimal stable sample size is 5,000 words.”
BackgroundVisualization in stylometry: Cluster analysis using networks
Eder, M. (2015) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqv061 · Cited by 143
“…scalability issues in stylometry (Luyckx, 2010), a paper discussing the short sample effect and its impact on authorship attribution reliability (Eder, 2015), an experiment using intensive corpus re-composition to test whether the attribution accuracy depends on particular constellation of…”
BackgroundMethodSelf-citationHill, M. J., & Hengchen, S. (2019) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqz024 · Cited by 114
Wright, D. (2017) · International Journal of Corpus Linguistics · doi:10.1075/ijcl.22.2.03wri · Cited by 112
“Other authorship studies that have systematically reduced the amount of data used in authorship experiments have also reported drops in success rates (e.g. Grant 2007, Luyckx & Daelemans 2011, Eder 2015).”
BackgroundThe Rowling Case: A Proposed Standard Analytic Protocol for Authorship Questions
Juola, P. (2015) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqv040 · Cited by 133
“Similarly, the question of ‘sufficiency’ of data is based, ultimately, on questions of sample size and representativeness (Eder, 2010), and the case-specific application hinges in part upon whether the analysis has been cherry-picked or represents a practice that has been shown to be, in general,…”
Background
Metadata
- Authors
- Eder, M.
- Affiliations
- Uniwersytet Komisji Edukacji Narodowej w Krakowie (PL)
- Corresponding author
- Eder, Maciej
- ORCID
- Eder, Maciej
- Venue
- Digital Scholarship in the Humanities
- Publisher
- Oxford University Press (OUP)
- Published
- 2013-11-14
- Volume
- 30
- Issue
- 2
- Pages
- 167-182
- ISSN
- 2055-7671 (ISSN-L)
- 2055-768X
- OpenAlex
- W1970948046
- Cited by
- 214
- References
- 35
- Citations by year
- 2026: 10 · 2025: 18 · 2024: 13 · 2023: 20 · 2022: 14 · 2021: 26 · 2020: 27 · 2019: 19 · 2018: 20 · 2017: 13 · 2016: 14 · 2015: 6 · 2014: 5 · 2013: 2
- FWCI
- 9.95
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Digital Scholarship in the Humanities (publishedVersion)
- DH
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Digital Humanities and Scholarship
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- authorship attribution, sample size, text length, stylistic analysis, delta method, most frequent words, character n-grams, support vector machine, k-nearest neighbors, random sampling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.