CoChat
Sign Up
PreprintCited by 5Authorship Attribution and ProfilingTopic ModelingMental Health via Writing

PART: Pre-trained Authorship Representation Transformer

Open access
Huertas-Tato, Javier, Martin, Alejandro, Camacho, David·2022·arXiv
DOI 10.48550/arxiv.2209.15373arXiv 2209.15373
Open this paper in CoChat — free account

Abstract

Authors writing documents imprint identifying information within their texts: vocabulary, registry, punctuation, misspellings, or even emoji usage. Previous works use hand-crafted features or classification tasks to train their authorship models, leading to poor performance on out-of-domain authors. Using stylometric representations is more suitable, but this by itself is an open research challenge. In this paper, we propose PART, a contrastively trained model fit to learn \textbf{authorship embeddings} instead of semantics. We train our model on ~1.5M texts belonging to 1162 literature authors, 17287 blog posters and 135 corporate email accounts; a heterogeneous set with identifiable writing styles. We evaluate the model on current challenges, achieving competitive performance. We also evaluate our model on test splits of the datasets achieving zero-shot 72.39\% accuracy when bounded to 250 authors, a 54\% and 56\% higher than RoBERTa embeddings. We qualitatively assess the representations with different data visualizations on the available datasets, observing features such as gender, age, or occupation of the author.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2209.15373

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Authorship attribution methods are typically divided into classification-based methods for a small set of candidate authors and similarity-based ranking methods for larger numbers of authors.

    “Authorship attribution methods are typically divided into classification-based methods for a small set of candidate authors and similarity-based ranking methods for larger numbers of authors [Rivera-Soto et al., 2021;Huertas-Tato et al., 2022].”

    From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Huertas-Tato et al., 2022

    The paper describes classifier-based authorship approaches in §2.1 (p. 3) and evaluates its own similarity-based attribution from a pool of authors in §5.2 (p. 8). It does not say that methods are typically divided into these two categories or associate classification with small candidate sets and similarity-based ranking with larger ones. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Contrastive learning is often used with pre-trained language models to enhance stylistic representation.

    “Contrastive learning [Khosla et al., 2020] is often used with pre-trained language models to enhance stylistic representation by maximizing similarity between texts written by the same authors and minimizing it with texts from different authors [Huertas-Tato et al., 2022].”

    From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Huertas-Tato et al., 2022

    Section 2.3 (p. 5) says the authors pair pre-trained transformers with contrastive learning to build representations of writing style. The Introduction (p. 2) says texts by the same author should have higher similarity than texts by different authors. The paper supports this approach, but its discussion of recent efforts does not establish that it is often used for stylistic representation. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Models like BERT and RoBERTa have shown limitations in cross-domain tasks, and adapting models to new domains remains difficult due to factors like dataset size variability and writing styles.

    “Models like BERT and RoBERTa have shown limitations in cross-domain tasks [Barlas and Stamatatos, 2020b;Huertas-Tato et al., 2022], and adapting models to new domains remains difficult due to factors like dataset size variability and writing styles.”

    From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Huertas-Tato et al., 2022

    Section 2.1 (p. 3) says adaptation to new domains is usually hard and identifies variation in document size and style; Section 5.2 (p. 8) shows the RoBERTa baseline underperforming other models on out-of-domain authorship tasks. The paper does not establish a cross-domain limitation for BERT specifically, and its discussion of varying size concerns documents rather than explicitly identifying variability in dataset size as a cause of adaptation difficulty. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 61

  1. [1][1] Efstathios Stamatatos. A survey of modern authorship attribution methods. Journal of the American Society for information Science and Technology , 60(3):538-556, 2009. -
  2. [2][2] Moshe Koppel, Jonathan Schler, and Shlomo Argamon. Computational methods in authorship attribution. Journal of the American Society for information Science and Technology , 60(1):9-26, 2009. -
  3. [3][3] Andrea Nini. Authorship profiling in a forensic context . PhD thesis, Aston University, 2015. -
  4. [4][4] Sergio Muñoz and Carlos A. Iglesias. A text classification approach to detect psychological stress combining a lexicon-based feature framework with distributional representations. Information Processing & Management , 59(5):103011, September 2022. -
  5. [5][5] Mudit Bhargava, Pulkit Mehndiratta, and Krishna Asawa. Stylometric analysis for authorship attribution on twitter. In International Conference on Big Data Analytics , pages 37-47. Springer, 2013. -
  6. [6][6] Robert Layton, Paul Watters, and Richard Dazeley. Authorship attribution for twitter in 140 characters or less. In 2010 Second Cybercrime and Trustworthy Computing Workshop , pages 1-8. IEEE, 2010. -
  7. [7][7] Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. A stylometric inquiry into hyperpartisan and fake news. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 231-240, 2018. -
  8. [8][8] Janek Bevendorff, Bilal Ghanem, Anastasia Giachanou, Mike Kestemont, Enrique Manjavacas, Ilia Markov, Maximilian Mayerl, Martin Potthast, Francisco Rangel, Paolo Rosso, et al. Overview of pan 2020: Authorship verification, celebrity profiling, profiling fake news spreaders on twitter, and style change detection. In International Conference of the Cross-Language Evaluation Forum for European Languages , pages 372-383. Springer, 2020. -
  9. [9][9] Shlomo Argamon, Moshe Koppel, James W Pennebaker, and Jonathan Schler. Automatically profiling the author of an anonymous text. Communications of the ACM , 52(2):119-123, 2009. -
  10. [10][10] Helena Gómez-Adorno, Juan-Pablo Posadas-Durán, Grigori Sidorov, and David Pinto. Document embeddings learned on various types of n-grams for cross-topic authorship attribution. Computing , 100(7):741-756, 2018. -
  11. [11][11] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT , pages 4171-4186, 2019. -
  12. [12][12] Maël Fabien, Esaú Villatoro-Tello, Petr Motlicek, and Shantipriya Parida. Bertaa: Bert fine-tuning for authorship attribution. In Proceedings of the 17th International Conference on Natural Language Processing (ICON) , pages 127-137, 2020. -
  13. [13][13] Georgios Barlas and Efstathios Stamatatos. Cross-domain authorship attribution using pre-trained language models. In IFIP International Conference on Artificial Intelligence Applications and Innovations , pages 255-266. Springer, 2020. -
  14. [15][15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021. -
  15. [16][16] Efstathios Stamatatos. A survey of modern authorship attribution methods. Journal of the American Society for information Science and Technology , 60(3):538-556, 2009. -
  16. [18][18] Kalaivani Sundararajan and Damon Woodard. What represents 'style' in authorship attribution? In Proceedings of the 27th International Conference on Computational Linguistics , pages 2814-2822, 2018. -
  17. [19][19] Yunita Sari, Mark Stevenson, and Andreas Vlachos. Topic or style? exploring the most useful features for authorship attribution. In Proceedings of the 27th International Conference on Computational Linguistics , pages 343-353, 2018. -
  18. [21][21] Zhou Ai, Zhang Yijia, Wei Hao, and Lu Mingyu. Lda-transformer model in chinese poetry authorship attribution. In China Conference on Information Retrieval , pages 59-73. Springer, 2021. -
  19. [22][22] Anastasia Fedotova, Aleksandr Romanov, Anna Kurtukova, and Alexander Shelupanov. Authorship attribution of social media and literary russian-language texts using machine learning methods and feature selection. Future Internet , 14(1):4, 2022. -
  20. [24][24] Benjamin Murauer and Günther Specht. Developing a benchmark for reducing data bias in authorship attribution. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems , pages 179-188, 2021. -
  21. [25][25] Julian Hitschler, Esther van den Berg, and Ines Rehbein. Authorship Attribution with Convolutional Neural Networks and POS-Eliding. ACL Anthology , pages 53-58, September 2017. -
  22. [26][26] Robert Deibel and Denise Löfflad. Style change detection on real-world data using an lstm-powered attribution algorithm. In CLEF (Working Notes) , pages 1899-1909, 2021. -
  23. [27][27] Jagna Nieuwazny, Karol Nowakowski, Michal Ptaszynski, and Fumito Masui. Can you fool AI by doing a 180? - A case study on authorship analysis of texts by Arata Osada. Information Processing & Management , 58(5):102644, September 2021. -
  24. [28][28] Emilio Villa-Cueva, Ivan González-Franco, Fernando Sanchez-Vega, and Adrián Pastor López-Monroy. Nlp-cimat at politices 2022: Politibeto, a domain-adapted transformer for multi-class political author profiling. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022). CEUR Workshop Proceedings, CEUR-WS, A Coruna, Spain , 2022. -
  25. [32][32] Mihailo Škori´ c, Ranka Stankovi´ c, Milica Ikoni´ c Neši´ c, Joanna Byszuk, and Maciej Eder. Parallel stylometric document embeddings with deep learning based language models in literary authorship attribution. Mathematics , 10(5), 2022. -
  26. [33][33] Jacob Tyo, Bhuwan Dhingra, and Zachary C Lipton. Siamese bert for authorship verification. In CLEF (Working Notes) , pages 2169-2177, 2021. -
  27. [34][34] Romain Futrzynski. Author classification as pre-training for pairwise authorship verification. In CLEF (Working Notes) , pages 1945-1952, 2021. -
  28. [35][35] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3982-3992, 2019. -
  29. [36][36] Ipek Baris Schlicht and Angel Felipe Magnossão de Paula. Unified and multilingual author profiling for detecting haters. arXiv preprint arXiv:2109.09233 , 2021. - · arXiv
  30. [37][37] Tarun Kumar, S Gowtham, and Udit Kr Chakraborty. Comparing word embeddings on authorship identification. In Applied Soft Computing , pages 177-194. Apple Academic Press, 2022. -
  31. [38][38] Enzo Terreau, Antoine Gourru, and Julien Velcin. Writing style author embedding evaluation. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems , pages 84-93, 2021. -
  32. [39][39] Maryam Najafi and Ehsan Tavan. Text-to-text transformer in authorship verification via stylistic and semantical analysis. In Proceedings of the CLEF , 2022. -
  33. [40][40] Ting Chen and Yizhou Sun. Task-guided and path-augmented heterogeneous network embedding for author identification. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining , WSDM'17, page 295-304, New York, NY, USA, 2017. Association for Computing Machinery. -
  34. [41][41] Zhiqiang Hu, Roy Ka-Wei Lee, Lei Wang, Ee-peng Lim, and Bo Dai. Deepstyle: User style embedding for authorship attribution of short texts. In Xin Wang, Rui Zhang, Young-Koo Lee, Le Sun, and Yang-Sae Moon, editors, Web and Big Data , pages 221-229, Cham, 2020. Springer International Publishing. -
  35. [42][42] Fereshteh Jafariakinabad and Kien A. Hua. A self-supervised representation learning of sentence structure for authorship attribution. ACM Trans. Knowl. Discov. Data , 16(4), jan 2022. -
  36. [43][43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. -
  37. [45][45] Kristy Choi, Curtis Hawthorne, Ian Simon, Monica Dinculescu, and Jesse Engel. Encoding musical style with transformer autoencoders. In International Conference on Machine Learning , pages 1899-1908. PMLR, 2020. -
  38. [46][46] Vaibhavi Kalgutkar, Ratinder Kaur, Hugo Gonzalez, Natalia Stakhanova, and Alina Matyukhina. Code Authorship Attribution: Methods and Challenges. ACM Comput. Surv. , 52(1):1-36, February 2019. -
  39. [47][47] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019. - · arXiv
  40. [48][48] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365 , February 2018. - · arXiv
  41. [49][49] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. -
  42. [50][50] Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 55-65, 2019. -
  43. [51][51] Chanchal Suman, Ayush Raj, Sriparna Saha, and Pushpak Bhattacharyya. Authorship attribution of microtext using capsule networks. IEEE Transactions on Computational Social Systems , 2021. -
  44. [52][52] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21(140):1-67, 2020. -
  45. [53][53] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. -
  46. [54][54] Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6894-6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. -
  47. [55][55] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems , 33:18661-18673, 2020. -
  48. [56][56] Xuming Hu, Li Lin, Lijie Wen, et al. Pair-level supervised contrastive learning for natural language inference. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 8237-8241. IEEE, 2022. -
  49. [57][57] Rafael A Rivera-Soto, Olivia Elizabeth Miano, Juanita Ordonez, Barry Y Chen, Aleem Khan, Marcus Bishop, and Nicholas Andrews. Learning universal authorship representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 913-919, 2021. -
  50. [58][58] Tempestt Neal, Kalaivani Sundararajan, Aneez Fatima, Yiming Yan, Yingfei Xiang, and Damon Woodard. Surveying stylometry techniques and applications. ACM Computing Surveys (CSuR) , 50(6):1-36, 2017. -
  51. [59][59] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018. - · arXiv
  52. [60][60] Kihyuk Sohn. Improved Deep Metric Learning with Multi-class N-pair Loss Objective. Advances in Neural Information Processing Systems , 29, 2016. -
  53. [61][61] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020. -
  54. [62][62] Martin Gerlach and Francesc Font-Clos. A standardized project gutenberg corpus for statistical analysis of natural language and quantitative linguistics. Entropy , 22(1):126, 2020. -
  55. [63][63] Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James W Pennebaker. Effects of age and gender on blogging. In AAAI spring symposium: Computational approaches to analyzing weblogs , volume 6, pages 199-205, 2006. -
  56. [64][64] Jitesh Shetty and Jafar Adibi. The enron email dataset database schema and brief statistical report. Information sciences institute technical report, University of Southern California , 4(1):120-128, 2004. -
  57. [65][65] Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing. arXiv , November 2021. -
  58. [66][66] Anna Wegmann, Marijn Schraagen, and Dong Nguyen. Same author or just same topic? towards contentindependent style representations. In Proceedings of the 7th Workshop on Representation Learning for NLP , pages 249-268, Dublin, Ireland, May 2022. Association for Computational Linguistics. -
  59. [72][72] Janek Bevendorff, BERTa Chulvi, Gretel Liz De La Peña Sarracén, Mike Kestemont, Enrique Manjavacas, Ilia Markov, Maximilian Mayerl, Martin Potthast, Francisco Rangel, Paolo Rosso, et al. Overview of pan 2021: Authorship verification, profiling hate speech spreaders on twitter, and style change detection. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: 12th International Conference of the CLEF Association, CLEF 2021, Virtual Event, September 21-24, 2021, Proceedings 12 , pages 419-431. Springer, 2021. -
  60. [73][73] Janek Bevendorff, Berta Chulvi, Elisabetta Fersini, Annina Heini, Mike Kestemont, Krzysztof Kredens, Maximilian Mayerl, Reynier Ortega-Bueno, Piotr P˛ ezik, Martin Potthast, et al. Overview of pan 2022: Authorship verification, profiling irony and stereotype spreaders, and style change detection. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: 13th International Conference of the CLEF Association, CLEF 2022, Bologna, Italy, September 5-8, 2022, Proceedings , pages 382-394. Springer, 2022. -
  61. [74][74] Janek Bevendorff, Mara Chinea-Ríos, Marc Franco-Salvador, Annina Heini, Erik Körner, Krzysztof Kredens, Maximilian Mayerl, Piotr P˛ ezik, Martin Potthast, Francisco Rangel, et al. Overview of pan 2023: Authorship verification, multi-author writing style analysis, profiling cryptocurrency influencers, and trigger detection. In Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part III , pages 518-526. Springer, 2023.

Cited by

Sorted by citations

Related

Authors
Huertas-Tato, J., Martin, A., & Camacho, D.
ORCID
Camacho, David
Venue
arXiv
Publisher
arXiv
Published
2022
DOI
10.48550/arxiv.2209.15373
arXiv
2209.15373
OpenAlex
W4300980259
Cited by
4
References
0
Citations by year
2025: 1 · 2024: 1 · 2023: 2
FWCI
0.33
Type
journal article
Language
en
Open access
Licence
Locations
  • Hispana (acceptedVersion, cc-by-nc)
  • arXiv (Cornell University) (submittedVersion)
  • Archivo Digital UPM (Universidad Politécnica de Madrid) (submittedVersion, cc-by-nc)
  • arXiv (Cornell University)
Funders
  • European Commission (13039/501100011033, 2020‐EU‐IA‐0252, 501100011033)
  • Universidad Politécnica de Madrid
  • Ministerio de Ciencia e Innovación (MCIN/AEI/10, .13039/501100011033, 10.13039, / AEI/10, PID2020-117263GB-100, 501100011033, PID2020, PLEC2021-007681, MCIN /AEI /10.13039)
  • Agencia Estatal de Investigación (501100011033, 13039, .13039/501100011033, 10.13039, AEI/10)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Psychology › Social Psychology › Mental Health via Writing
Keywords
authorship attribution, stylometry, contrastive learning, pre-trained transformer models, author demographics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review