Cited by 7Authorship Attribution and ProfilingSpeech Recognition and SynthesisNatural Language Processing Techniques

HANSEN: Human and AI Spoken Text Benchmark for Authorship Analysis

Open access
Tripto, Nafis, Uchendu, Adaku, Le, Thai, Setzu, Mattia, Giannotti, Fosca, Lee, Dongwon·2023·Findings of the Association for Computational Linguistics: EMNLP 2023

Abstract

Authorship Analysis, also known as stylometry, has been an essential aspect of Natural Language Processing (NLP) for a long time. Likewise, the recent advancement of Large Language Models (LLMs) has made authorship analysis increasingly crucial for distinguishing between human-written and AI-generated texts. However, these authorship analysis tasks have primarily been focused on written texts, not considering spoken texts. Thus, we introduce the largest benchmark for spoken texts - HANSEN (Human ANd ai Spoken tExt beNchmark). HANSEN encompasses meticulous curation of existing speech datasets accompanied by transcripts, alongside the creation of novel AI-generated spoken text datasets. Together, it comprises 17 human datasets, and AI-generated spoken texts created using 3 prominent LLMs: ChatGPT, PaLM2, and Vicuna13B. To evaluate and demonstrate the utility of HANSEN, we perform Authorship Attribution (AA) & Author Verification (AV) on human-spoken datasets and conducted Human vs. AI spoken text detection using state-of-the-art (SOTA) models. While SOTA methods, such as, character ngram or Transformer-based model, exhibit similar AA & AV performance in human-spoken datasets compared to written ones, there is much room for improvement in AI-generated spoken text detection. The HANSEN benchmark is available at: https://huggingface.co/datasets/HANSEN-REPO/HANSEN.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2023.findings-emnlp.916

Cites

69 of 69 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing detection methods are mostly binary classifiers and cannot distinguish complex LLM-generated content such as mixed human-LLM input.

    “current methods often focus on binary classification, failing to address the complexities of realworld scenarios like human-LLM collaboration.”

    From Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement (2025) · cites this paper as [52]

    The Introduction (p. 2) says substantial research focuses on distinguishing human from AI-generated text, and Related Work (p. 3) describes detectors trained on human and AI texts. This supports the point that detection commonly addresses a human-versus-AI distinction, but the paper does not say that most methods are binary classifiers or that they fail on mixed human–LLM content or collaboration. (LLM, conf=0.79)

    Experimental — accuracy not yet measured
  • Existing supervised classifiers frequently overfit to particular domains, perform poorly on out-of-distribution data, and lose capability on unseen datasets from other domains.

    “However, these classifiers often overfit to specific domains, leading to poor performance on out-of-distribution data [12,51], and their capabilities significantly degrade when applied to unseen datasets from different domains [32].”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 10

  1. Ahmed Abbasi and Hsinchun Chen. 2008. Writeprints: Astylometric approach to identity-level identification and similarity detection in cyberspace. ACM Transactions on Information Systems (TOIS) , 26(2):1-29.
  2. Gillian Brown, Gillian D Brown, George Yule, Gillian R Brown, and Brown Gillian. 1983. Discourse analysis . Cambridge university press.
  3. BNC Consortium et al. 2007. British national corpus. Oxford Text Archive Core Collection .
  4. Howard Giles. 1973. Accent mobility: A model and some data. Anthropological linguistics , pages 87105.
  5. Hans Van Halteren. 2007. Author verification by linguistic profiling: An exploration of the parameter space. ACM Transactions on Speech and Language Processing (TSLP) , 4(1):1-17.
  6. Robbie Love, Claire Dembry, Andrew Hardie, Vaclav Brezina, and Tony McEnery. 2017. The spoken bnc2014: Designing and building a spoken corpus of everyday conversations. International Journal of Corpus Linguistics , 22(3):319-344.
  7. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems , 34:4816-4828.
  8. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
  9. Melvin I Urofsky. 2001. The oyez project: Us supreme court multimedia database. The Journal of American History , 88(2):753.
  10. Jason S Wrench, James C McCroskey, and Virginia P Richmond. 2008. Human communication in everyday life: Explanations and applications . Pearson Education.

Cited by

Sorted by citations

Authors
Tripto, N., Uchendu, A., Le, T., Setzu, M., Giannotti, F., & Lee, D.
Affiliations
  • Pennsylvania State University (US)
  • University of Mississippi (US)
  • Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" (IT)
Venue
Findings of the Association for Computational Linguistics: EMNLP 2023
Published
2023
Pages
13706-13724
OpenAlex
W4389518717
Cited by
6
References
69
Citations by year
2026: 1 · 2025: 3 · 2024: 1 · 2023: 1
FWCI
1.07
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, cc-by-nc-sa)
  • Scuola Normale Superiore di Pisa (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (1820609, 1934782, 2114824, 2131144)
  • European Commission (654024, IR0000013, HORIZON2020, PE00000013, 101079043, 871042, Spoke 1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
authorship analysis, stylometry, spoken text, AI-generated text detection, authorship attribution, author verification, large language models, speech dataset

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review