The Topic Confusion Task: A Novel Evaluation Scenario for Authorship Attribution
Open accessAbstract
Authorship attribution is the problem of identifying the most plausible author of an anonymous text from a set of candidate authors.Researchers have investigated same-topic and cross-topic scenarios of authorship attribution, which differ according to whether new, unseen topics are used in the testing phase.However, neither scenario allows us to explain whether errors are caused by a failure to capture authorship writing style or by a topic shift.Motivated by this, we propose the topic confusion task where we switch the author-topic configuration between the training and testing sets.This setup allows us to distinguish two types of errors: those caused by the topic shift and those caused by the features' inability to capture the writing styles.We show that stylometric features with part-of-speech tags are the least susceptible to topic variations.We further show that combining them with other features leads to significantly lower topic confusion and higher attribution accuracy.Finally, we show that pretrained language models such as BERT and RoBERTa perform poorly on this task and are surpassed by simple features such as word-level n-grams.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Attribution performance tends to drop when known and query texts differ in topic or genre.
“Attribution performance tends to drop when known and query texts differ in topic or genre [Altakrori et al., 2021].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Altakrori et al., 2021
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,171 · see citing works
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · see citing works
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
A survey of modern authorship attribution methods
Stamatatos, E. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21001 · Cited by 1,176 · see citing works
Juola, P. (2008) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000005 · Cited by 505 · see citing works
Mosteller, F., & Wallace, D. L. (1963) · Journal of the American Statistical Association · doi:10.1080/01621459.1963.10500849 · Cited by 466 · see citing works
The Characteristic Curves of Composition
Mendenhall, T. C. (1887) · Science · doi:10.1126/science.ns-9.214s.237 · Cited by 407 · see citing works
Authorship Attribution for Social Media Forensics
Rocha, A., Scheirer, W. J., Forstall, C. W., Cavalcante, T., Theophilo, A., Shen, B., Carvalho, A. R. B., & Stamatatos, E. (2017) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2016.2603960 · Cited by 222 · see citing works
N-Gram Feature Selection for Authorship Identification
Houvardas, J., & Stamatatos, E. (2006) · Lecture notes in computer science · doi:10.1007/11861461_10 · Cited by 201 · see citing works
Not All Character N-grams Are Created Equal: A Study in Authorship Attribution
Sapkota, U., Bethard, S. J., Montes, M., & Solorio, T. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1010 · Cited by 159 · see citing works
The effect of author set size and data size in authorship attribution
Luyckx, K., & Daelemans, W. M. P. (2010) · Literary and Linguistic Computing · doi:10.1093/llc/fqq013 · Cited by 157 · see citing works
On the Robustness of Authorship Attribution Based on Character N -gram Features
Stamatatos, E. (2013) · Journal of law and policy · Cited by 140 · see citing works
A novel approach of mining write-prints for authorship attribution in e-mail forensics
Iqbal, F., Hadjidj, R., Fung, B. C. M., & Debbabi, M. (2008) · Digital Investigation · doi:10.1016/j.diin.2008.05.001 · Cited by 128 · see citing works
Self-citationIdentifying Authorship by Byte-Level N-Grams: The Source Code Author Profile (SCAP) Method.
Frantzeskou, G., Stamatatos, E., Gritzalis, S., Chaski, C. E., & Howald, B. S. (2007) · International journal of digital evidence · Cited by 118 · see citing works
A unified data mining solution for authorship analysis in anonymous textual communications
Iqbal, F., Binsalleeh, H., Fung, B. C. M., & Debbabi, M. (2011) · Information Sciences · doi:10.1016/j.ins.2011.03.006 · Cited by 102 · see citing works
Self-citationAuthorship Attribution Using Text Distortion
Stamatatos, E. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1107 · Cited by 98 · see citing works
‘twazn me!!! ;(’ Automatic Authorship Analysis of Micro-Blogging Messages
Sousa‐Silva, R., Laboreiro, G., Sarmento, L., Grant, T., Oliveira, E., & Maia, B. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22327-3_16 · Cited by 80 · see citing works
Visualizing Authorship for Identification
Abbasi, A., & Chen, H. (2006) · Lecture notes in computer science · doi:10.1007/11760146_6 · Cited by 71 · see citing works
The Characteristic Curves of Composition
Eddy, H. T. (1887) · Science · doi:10.1126/science.ns-9.216.297.a · Cited by 63 · see citing works
Cross-Domain Authorship Attribution Using Pre-trained Language Models
Barlas, G., & Stamatatos, E. (2020) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-49161-1_22 · Cited by 54 · see citing works
Masking topic‐related information to enhance authorship attribution
Stamatatos, E. (2017) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23968 · Cited by 51 · see citing works
Syntax Encoding with Application in Authorship Attribution
Zhang, R., Hu, Z., Guo, H., & Mao, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1294 · Cited by 50 · see citing works
Cross-Topic Authorship Attribution: Will Out-Of-Topic Data Help?
Sapkota, U., Solorio, T., Bethard, S. J., & Rosso, P. (2015) · Cited by 34 · see citing works
Cross-Language Authorship Attribution
Bogdanova, D., & Lazaridou, A. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4eyemy7dowat · Cited by 28 · see citing works
A Visualizable Evidence-Driven Approach for Authorship Attribution
Ding, S. H. H., Fung, B. C. M., & Debbabi, M. (2015) · ACM Transactions on Information and System Security · doi:10.1145/2699910 · Cited by 26 · see citing works
Self-citationAltakrori, M. H., Iqbal, F., Fung, B. C. M., Ding, S. H. H., & Tubaishat, A. (2018) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3236391 · Cited by 24 · see citing works
Self-citationPerson Identification from Text and Speech Genre Samples
Goldstein-Stewart, J., Winder, R., & Sabin, R. E. (2009) · Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics on - EACL '09 · doi:10.3115/1609067.1609104 · Cited by 22 · see citing works
What represents “style” in authorship attribution?
Sundararajan, K., & Woodard, D. L. (2018) · International Conference on Computational Linguistics · Cited by 22 · see citing works
Authorship Attribution of IRC Messages Using Inverse Author Frequency
Layton, R., McCombie, S. J., & Watters, P. (2012) · Cybercrime and Trustworthy Computing Workshop (CTC) · doi:10.1109/ctc.2012.11 · Cited by 20 · see citing works
An Ensemble Approach to Cross-Domain Authorship Attribution
Custódio, J. E., & Paraboni, I. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-28577-7_17 · Cited by 18 · see citing works
A transfer learning approach to cross-domain authorship attribution
Barlas, G., & Stamatatos, E. (2021) · Evolving Systems · doi:10.1007/s12530-021-09377-2 · Cited by 15 · see citing works
Is the contextual information relevant in text clustering by compression?
Granados, A., Camacho, D., & Rodrı́guez, F. B. (2012) · Expert Systems with Applications · doi:10.1016/j.eswa.2012.01.215 · Cited by 11 · see citing works
Learning Stylometric Representations for Authorship Analysis
Ding, S. H. H., Fung, B. C. M., Iqbal, F., & Cheung, W. K. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2766189 · Cited by 5 · see citing works
Self-citationMode Effects’ Challenge to Authorship Attribution
Wang, H., Riddell, A. B., & Juola, P. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.97 · Cited by 4 · see citing works
Flagged first, then by citations
1 of the 40 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges
Huang, B., Chen, C., & Shu, K. (2025) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3715073.3715076 · Cited by 42 · see citing works
“Attribution performance tends to drop when known and query texts differ in topic or genre (Altakrori et al., 2021).”
Understanding writing style in social media with a supervised contrastively pre-trained transformer
Huertas‐Tato, J., Martín, A. G., & Camacho, D. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111867 · Cited by 15 · see citing works
Evaluating Synthetic Data Generation from User Generated Text
Chim, J., Ive, J., & Liakata, M. (2024) · Computational Linguistics · doi:10.1162/coli_a_00540 · Cited by 10 · see citing works
Tyo, J., Dhingra, B., & Lipton, Z. C. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.43 · Cited by 6 · see citing works
A Comparison of Several AI Techniques for Authorship Attribution on Romanian Texts
Avram, S.-M., & Oltean, M. (2022) · Mathematics · doi:10.3390/math10234589 · Cited by 5 · see citing works
Addressing Topic Leakage in Cross-Topic Evaluation for Authorship Verification
Sawatphol, J., Udomcharoenchaikit, C., & Nutanong, S. Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00709 · Cited by 5 · see citing works
Altakrori, M. H., Scialom, T., Fung, B. C. M., & Cheung, J. C. K. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.153 · Cited by 2 · see citing works
Self-citationImproving Long-Text Authorship Verification via Model Selection and Data Tuning
Nguyen, T., Dagli, C. K., Alperin, K., VanDam, C., & Singer, E. (2023) · Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL) · doi:10.18653/v1/2023.latechclfl-1.4 · Cited by 1 · see citing works
Learning the Style via Mixed SN-Grams: An Evaluation in Authorship Attribution
Posadas-Durán, J. P., Ríos-Toledo, G., Velázquez-Lozada, E., Osuna-Coutiño, J. A. D. J., Pérez‐Patricio, M., & Pech-May, F. (2025) · AI · doi:10.3390/ai6050104 · Cited by 1 · see citing works
Authorship Analysis with Machine Learning
Ahmed, W., Javed, A. R., Jalil, Z., & Iqbal, F. (2022) · Springer eBooks · doi:10.1007/978-1-4899-7502-7_986-1
The robustness of Japanese stylometric features to topic variability in authorship analysis
Zaitsu, W., Jin, M., Ishihara, S., Tsuge, S., & Inaba, M. (2026) · Behaviormetrika · doi:10.1007/s41237-026-00319-3
Ismakaeva, I. D., Senina, A. V., & Shapovalov, M. S. (2026) · Историческая информатика · doi:10.7256/2585-7797.2026.3.81047
- Authors
- Altakrori, M., Cheung, J. C. K., & Fung, B. C. M.
- Affiliations
- Mila - Quebec Artificial Intelligence Institute (CA)
- McGill University (CA)
- ORCID
- Fung, Benjamin C. M.
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2021
- Published
- 2021
- Pages
- 4242-4256
- OpenAlex
- W3214222557
- Cited by
- 11
- References
- 40
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 3 · 2023: 2 · 2022: 3
- FWCI
- 0.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- Funders
- Canada Research Chairs
- Natural Sciences and Engineering Research Council of Canada (rgpin-2018-03872, RGPIN 2018)
- Fonds de recherche du Québec – Nature et technologies
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › User Authentication and Security Systems
- Keywords
- authorship attribution, topic shift, stylometry, part-of-speech features, writing style, pre-trained language models, BERT, RoBERTa, word n-grams
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review