Learning Interpretable Style Embeddings via Prompting LLMs
Open accessAbstract
Style representation learning builds content-independent representations of author style in text. Stylometry, the analysis of style in text, is often performed by expert forensic linguists and no large dataset of stylometric annotations exists for training. Current style representation learning uses neural methods to disentangle style from content to create style vectors, however, these approaches result in uninterpretable representations, complicating their usage in downstream applications like authorship attribution where auditing and explainability is critical. In this work, we use prompting to perform stylometry on a large number of texts to create a synthetic dataset and train human-interpretable style representations we call LISA embeddings. We release our synthetic stylometry dataset and our interpretable style models as resources.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Traditionally, LLMs have been employed mainly for auxiliary tasks such as feature extraction and data annotation.
“Traditionally, LLMs have been employed mainly for auxiliary tasks such as feature extraction and data annotation [Patel et al., 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Patel et al., 2023
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · see citing works
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · Neural Information Processing Systems · doi:10.48550/arxiv.1509.01626 · Cited by 7,552 · see citing works
The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods
Tausczik, Y. R., & Pennebaker, J. W. (2010) · Journal of Language and Social Psychology · doi:10.1177/0261927x09351676 · Cited by 5,774 · see citing works
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 3,710 · see citing works
Supervised Contrastive Learning
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., & Krishnan, D. (2020) · arXiv · doi:10.48550/arxiv.2004.11362 · Cited by 3,569 · see citing works
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
Pang, B., & Lee, L. (2005) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1219840.1219855 · Cited by 2,153 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“As large language models scale and improve, however, we believe this work could benefit from increasing coherency and decreasing hallucination in the annotations (Kaplan et al., 2020).”
BackgroundDo Deep Nets Really Need to be Deep?Preprint
Ba, J. L., & Caruana, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6184 · Cited by 1,592 · see citing works
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · see citing works
SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation
Hill, F., Reichart, R., & Korhonen, A. (2015) · Computational Linguistics · doi:10.1162/coli_a_00237 · Cited by 1,179 · see citing works
A survey of modern authorship attribution methods
Stamatatos, E. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21001 · Cited by 1,176 · see citing works
Neural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 984 · see citing works
GoEmotions: A Dataset of Fine-Grained Emotions
Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., & Ravi, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.372 · Cited by 873 · see citing works
Style Transfer from Non-Parallel Text by Cross-AlignmentPreprint
Shen, T., Lei, T., Barzilay, R., & Jaakkola, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09655 · Cited by 616 · see citing works
Computational methods in authorship attribution
Koppel, M., Schler, J., & Argamon, S. (2009) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20961 · Cited by 528 · see citing works
Style Transfer in Text: Exploration and Evaluation
Fu, Z., Tan, X., Peng, N., Zhao, D., & Yan, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11330 · Cited by 468 · see citing works
Mosteller, F., & Wallace, D. L. (1963) · Journal of the American Statistical Association · doi:10.1080/01621459.1963.10500849 · Cited by 466 · see citing works
Hate Speech Dataset from a White Supremacy Forum
de Gibert, O., Pérez, N., Pablos, A. G., & Cuadros, M. (2018) · Workshop on Abusive Language Online (ALW) · doi:10.18653/v1/w18-5102 · Cited by 428 · see citing works
Style transfer through back-translation
Prabhumoye, S., Tsvetkov, Y., Salakhutdinov, R., & Black, A. W. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1080 · Cited by 348 · see citing works
CARER: Contextualized Affect Representations for Emotion Recognition
Saravia, E., Liu, H.-C. T., Huang, Y.-H., Wu, J., & Chen, Y.-S. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1404 · Cited by 337 · see citing works
Rao, S., & Tetreault, J. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1012 · Cited by 327 · see citing works
Holmes, D. I. (1994) · Computers and the Humanities · doi:10.1007/bf01830689 · Cited by 272 · see citing works
Disentangled Representation Learning for Non-Parallel Text Style Transfer
John, V., Mou, L., Bahuleyan, H., & Olga Vechtomova, A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1041 · Cited by 266 · see citing works
Large Language Models Can Self-Improve
Huang, J., Gu, S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.67 · Cited by 230 · see citing works
Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation
Dai, N., Liang, J., Qiu, X., & Huang, X. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1601 · Cited by 203 · see citing works
Simple English Wikipedia: A New Text Simplification Task
Cöster, W., & Kauchak, D. (2011) · Cited by 181 · see citing works
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
Honovich, O., Scialom, T., Levy, O., & Schick, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.806 · Cited by 160 · see citing works
Automatically Neutralizing Subjective Bias in Text
Pryzant, R., Martinez, R. D., Dass, N., Kurohashi, S., Jurafsky, D., & Yang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5385 · Cited by 155 · see citing works
An Empirical Analysis of Formality in Online Communication
Pavlick, E., & Tetreault, J. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00083 · Cited by 144 · see citing works
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022) · arXiv · doi:10.48550/arxiv.2212.10560 · Cited by 118 · see citing works
Alva-Manchego, F. E., Martin, L., Bordes, A., Scarton, C., Sagot, B., & Specia, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.424 · Cited by 105 · see citing works
Groenwold, S., Ou, L.-H., Parekh, A., Honnavalli, S., Levy, S., Mirza, D., & Wang, W. Y. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2020.emnlp-main.473 · Cited by 100 · see citing works
Personalized Machine Translation: Preserving Original Author Traits
Rabinovich, E., Patel, R. N., Mirkin, S., Specia, L., & Wintner, S. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1101 · Cited by 95 · see citing works
The Linguist on the Witness Stand: Forensic Linguistics in American Courts
Tiersma, P., & Solan, L. M. (2002) · Language · doi:10.1353/lan.2002.0135 · Cited by 73 · see citing works
Android apps and user feedback: a dataset for software evolution and quality improvement
Grano, G., Sorbo, A. D., Mercaldo, F., Visaggio, C. A., Canfora, G., & Panichella, S. (2017) · doi:10.1145/3121264.3121266 · Cited by 64 · see citing works
A Study of Style in Machine Translation: Controlling the Formality of Machine Translation Output
Niu, X., Martindale, M. J., & Carpuat, M. J. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1299 · Cited by 56 · see citing works
Text Style Transfer via Learning Style Instance Supported Latent Space
Yi, X., Liu, Z., Li, W., & Sun, M. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/526 · Cited by 53 · see citing works
Domain Adaptive Text Style Transfer
Li, D., Zhang, Y., Gan, Z., Cheng, Y., Brockett, C., Dolan, B., & Sun, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1325 · Cited by 50 · see citing works
Similarity Learning for Authorship Verification in Social Media
Boenninghoff, B., Nickel, R. M., Zeiler, S., & Kolossa, D. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683405 · Cited by 40 · see citing works
TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling
Riley, P., Constant, N., Guo, M., Kumar, G., Uthus, D., & Parekh, Z. (2021) · doi:10.18653/v1/2021.acl-long.293 · Cited by 35 · see citing works
BLiMP: the benchmark of linguistic minimal pairs for English
Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00321 · Cited by 28 · see citing works
Same Author or Just Same Topic? Towards Content-Independent Style Representations
Wegmann, A., Schraagen, M., & Nguyen, D. (2022) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2022.repl4nlp-1.26 · Cited by 26 · see citing works
The Importance of Suppressing Domain Style in Authorship AnalysisPreprint
Bischoff, S., Deckers, N., Schliebs, M., Thies, B., Hagen, M., Stamatatos, E., Stein, B., & Potthast, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.14714 · Cited by 25 · see citing works
Learning Universal Authorship Representations
Rivera-Soto, R. A., Miano, O. E., Ordonez, J., Chen, B. Y., Khan, A., Bishop, M., & Andrews, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.70 · Cited by 25 · see citing works
Learning Invariant Representations of Social Media Users
Andrews, N., & Bishop, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1178 · Cited by 19 · see citing works
From Shakespeare to Twitter: What are Language Styles all about?
Xu, W. H. (2017) · doi:10.18653/v1/w17-4901 · Cited by 17 · see citing works
Zhu, J. Y., & Jurgens, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.25 · Cited by 16 · see citing works
Representation learning of writing style
Hay, J., Doan, B.-L., Popineau, F., & Elhara, O. A. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.30 · Cited by 14 · see citing works
A Deep Metric Learning Approach to Account Linking
Khan, A., Fleming, E., Schofield, N., Bishop, M., & Andrews, N. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.415 · Cited by 14 · see citing works
Atwell, K., Hassan, S., & Alikhani, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.08207 · Cited by 14 · see citing works
EncT5: Fine-tuning T5 Encoder for Non-autoregressive TasksPreprint
Liu, F., Shakeri, S., Yu, H., & Li, J. (2021) · arXiv (Cornell University) · Cited by 11 · see citing works
Style versus Content: A distinction without a (learnable) difference?
Jafaritazehjani, S., Lecorvé, G., Lolive, D., & Kelleher, J. D. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.197 · Cited by 10 · see citing works
Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework
Wegmann, A., & Nguyen, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.569 · Cited by 8 · see citing works
Low-Resource Authorship Style Transfer: Can Non-Famous Authors Be Imitated?Preprint
Patel, A., Andrews, N., & Callison-Burch, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08986 · Cited by 3 · see citing works
Self-citationZhu, K., Tian, Z., Wei, J., Luo, R., Song, Y., & Mao, X. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2s6u4uh53bf2 · Cited by 2 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Cited by
Sorted by citations
Combating misinformation in the age of LLMs: Opportunities and challenges
Chen, C., & Shu, K. (2024) · AI Magazine · doi:10.1002/aaai.12188 · Cited by 148 · see citing works
Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges
Huang, B., Chen, C., & Shu, K. (2025) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3715073.3715076 · Cited by 42 · see citing works
“Traditionally, LLMs have been employed mainly for auxiliary tasks such as feature extraction and data annotation (Patel et al., 2023).”
Stylometry recognizes human and LLM-generated texts in short samples
Przystalski, K., Argasiński, J. K., Grabska-Gradzińska, I., & Ochab, J. K. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129001 · Cited by 24 · see citing works
You, W., & Lowd, D. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00020 · Cited by 2 · see citing works
Morphosyntactic Annotation in Literary Stylometry
Gorman, R. J. (2024) · Information · doi:10.3390/info15040211 · Cited by 1 · see citing works
Better Call Claude: Can LLMs Detect Changes of Writing Style?
Römisch, J., Gorovaia, S., Halchynska, M., Schmidt, G., & Yamshchikov, I. P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04354-2_4 · Cited by 1 · see citing works
Identification and Verification of Authorship Using Machine and Deep Learning Methods
Ovasapyan, T. D., Tolgorenko, E. M., & Zubkov, E. A. (2024) · Automatic Control and Computer Sciences · doi:10.3103/s0146411624700925
Sayoud, H. (2025) · International Conference on Computer Technology Applications (ICCTA) · doi:10.1109/iccta65425.2025.11166592
Analysis of Factors Contributing to Conversational Derailment Using Text Style Embeddings
Nonaka, K., & Yoshida, M. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.09.371
Schlicht, J. A., van Woudenberg, T. J., Blok, D. J., Simoski, B., & Buijzen, M. (2026) · Appetite · doi:10.1016/j.appet.2026.108773
From Chat to Academia: Calibrating Formality in Low-Resource Languages
Efendioglu, S., & Pehlivan, H. (2026) · IEEE Access · doi:10.1109/access.2026.3652343
- Authors
- Patel, A., Rao, D., Kothary, A., McKeown, K., & Callison-Burch, C.
- Affiliations
- University of Pennsylvania (US)
- Columbia University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 15270-15290
- arXiv
- 2305.12696
- OpenAlex
- W4389518719
- Cited by
- 10
- References
- 63
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 3
- FWCI
- 1.77
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1928631)
- Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750)
- Intelligence Advanced Research Projects Activity
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Keywords
- stylometry, interpretable embeddings, authorship attribution, explainability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
