CoChat
Sign Up
Cited by 8Authorship Attribution and ProfilingNatural Language Processing TechniquesDigital Media Forensic Detection

Detecting AI-Generated Sentences in Human-AI Collaborative Hybrid Texts: Challenges, Strategies, and Insights

Open access
Zeng, Zijie, Liu, Shiqi, Sha, Lele, Li, Zhuang, Yang, Kaixun, Liu, Sannyuya, Gasevic, Dragan, Chen, Guangliang·2024·Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence
DOI 10.24963/ijcai.2024/835
Open this paper in CoChat — free accountOpen PDF

Abstract

This study explores the challenge of sentence-level AI-generated text detection within human-AI collaborative hybrid texts (abbreviated as hybrid texts). Existing studies of AI-generated text detection for hybrid texts often rely on synthetic datasets. These typically involve hybrid texts with a limited number of boundaries, e.g., single-boundary hybrid texts that begin with human-written content and end with machine-generated continuations. We contend that studies of detecting AI-generated content within hybrid texts should cover different types of hybrid texts generated in realistic settings to better inform real-world applications. Therefore, our study utilizes the CoAuthor dataset, which includes diverse, realistic hybrid texts generated through the collaboration between human writers and an intelligent writing system in multi-turn interactions. We adopt a two-step, segmentation-based pipeline: (i) detect segments within a given hybrid text where each segment contains sentences of consistent authorship, and (ii) classify the authorship of each identified segment. Our empirical findings highlight (1) detecting AI-generated sentences in hybrid texts is overall a challenging task because (1.1) human writers' selecting and even editing AI-generated sentences based on personal preferences adds difficulty in identifying the authorship of segments; (1.2) the frequent change of authorship between neighboring sentences within the hybrid text creates difficulties for segment detectors in identifying authorship-consistent segments; (1.3) the short length of text segments within hybrid texts provides limited stylistic cues for reliable authorship determination; (2) before embarking on the detection process, it is beneficial to assess the average length of segments within the hybrid text. This assessment aids in deciding whether (2.1) to employ a text segmentation-based strategy for hybrid texts with longer segments, or (2.2) to adopt a direct sentence-by-sentence classification strategy for those with shorter segments.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.24963/ijcai.2024/835

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Zeng et al. introduced a two-step segmentation pipeline with a segment detection module (e.g., Transformer 2, SegFormer) and a segment classification module (e.g., BERT, RoBERTa).

    “Zeng et al. [79] introduced a twostep segmentation-based pipeline to deal with AHT.They first used a segment detection module to divide a hybrid text into segments, each of which comes from a single authorship.The segment detection module can use Transformer 2 [87], SegFormer [88], and any other text segmentation model.Then, a segment classification module was used to classify the segments”

    From Enhancing the Robustness of AI-Generated Text Detectors: A Survey (2025) · cites this paper as [79]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. [2]2. Haitao Bai, Pinghui Wang, Ruofei Zhang, and Zhou Su. Segformer: a topic segmentation model with controllable range of attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12545-12552, 2023.
  2. [5]5. Daniel Buschek, Martin Zürn, and Malin Eiband. The impact of multiple parallel phrase sug- gestions on email input and composition behaviour of na- tive and non-native english writers. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-13, 2021.
  3. [7]7. Elizabeth Clark, Tal August, Sofia Ser- rano, Nikita Haduong, Suchin Gururangan, and Noah A Smith. All that's 'human'is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing, pages 7282-7296, 2021.
  4. [11]11. Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Callison-Burch. Real or fake text?: Investigating human ability to detect bound- aries between human-written and machine-generated text. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 37, pages 12763-12771, 2023.
  5. [13]13. Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maur- izio Tesconi. Tweepfake: About detecting deepfake tweets. Plos one, 16(5):e0251415, 2021.
  6. [15]15. Iacopo Ghinassi, Lin Wang, Chris Newell, and Matthew Purver. Lessons learnt from lin- ear text segmentation: a fair comparison of architectural and sentence encoding strategies for successful segmenta- tion. In Ruslan Mitkov and Galia Angelova, editors, Pro- ceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 408- 418, Varna, Bulgaria, September 2023. INCOMA Ltd., Shoumen, Bulgaria.
  7. [17]17. Zheng Gong, Shiwei Tong, Han Wu, Qi Liu, Hanqing Tao, Wei Huang, and Runlong Yu. Tipster: A topic-guided language model for topic-aware text segmentation. In International Conference on Database Systems for Advanced Applications, pages 213- 221. Springer, 2022.
  8. [24]24. Beizhe Hu, Qiang Sheng, Juan Cao, Yuhui Shi, Yang Li, Danding Wang, and Peng Qi. Bad actor, good advisor: Exploring the role of large language models in fake news detection. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, Vancouver, Canada, February 2024.
  9. [26]26. Ganesh Jawahar, Muhammad Abdul- Mageed, and VS Laks Lakshmanan. Automatic detection of machine generated text: A critical survey. In Proceed- ings of the 28th International Conference on Computa- tional Linguistics, pages 2296-2309, 2020.
  10. [28]28. Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.
  11. [29]29. Koike et al., 2024] Ryuto Koike, Masahiro Kaneko, and Naoaki Okazaki. Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, Vancouver, Canada, February 2024.
  12. [31]31. Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant. Text seg- mentation as a supervised learning task. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 469-473, 2018.
  13. [33]33. Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In Pro- ceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1-19, 2022.
  14. [35]35. Raymond Li, Wen Xiao, Linzi Xing, Lan- jun Wang, Gabriel Murray, and Giuseppe Carenini. Hu- man guided exploitation of interpretable attention patterns in summarization and topic segmentation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10189-10204, 2022.
  15. [38]38. Kelvin Lo, Yuan Jin, Weicong Tan, Ming Liu, Lan Du, and Wray Buntine. Transformer over pre- trained transformer for neural text segmentation with en- hanced topic coherence. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3334- 3340, 2021.
  16. [40]40. Michal Lukasik, Boris Dadachev, Kishore Papineni, and Gonc ¸alo Simões. Text segmenta- tion by cross segment attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 4707-4716, 2020.
  17. [44]44. Daniel Martens and Walid Maalej. Towards understanding and detecting fake reviews in app stores. Empirical Software Engineering, 24(6):3316-3355, 2019.
  18. [46]46. Eric Mitchell, Yoonho Lee, Alexan- der Khazatsky, Christopher D. Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text de- tection using probability curvature. In Proceedings of the 40th International Conference on Machine Learning, ICML, 2023.
  19. [47]47. Pagnoni et al., 2022] Artidoro Pagnoni, Martin Graciarena, and Yulia Tsvetkov. Threat scenarios and best practices to detect neural fake news. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 1233-1249, 2022.
  20. [51]51. Domenic Rosati. SynSciPass: detecting ap- propriate uses of scientific text generation. In Arman Co- han, Guy Feigenblat, Dayne Freitag, Tirthankar Ghosal, Drahomira Herrmannova, Petr Knoth, Kyle Lo, Philipp Mayr, Michal Shmueli-Scheuer, Anita de Waard, and Lucy Lu Wang, editors, Proceedings of the Third Work- shop on Scholarly Document Processing, pages 214-222, Gyeongju, Republic of Korea, October 2022. Association for Computational Linguistics.
  21. [55]55. Swapna Somasundaran et al. Two-level transformer and auxiliary coherence modeling for improved text segmentation. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7797-7804, 2020.
  22. [57]57. Grace H Sun and Stephanie H Hoelscher. The chatgpt storm and what faculty can do. Nurse Educator, 48(3):119-124, 2023.
  23. [59]59. Michael Tepper, Daniel Capurro, Fei Xia, Lucy Vanderwende, and Meliha Yetisgen-Yildiz. Sta- tistical section segmentation in free-text clinical records. In Lrec, pages 2001-2008, 2012.
  24. [60]60. Wang et al., 2018] Jenq-Haur Wang, Ting-Wei Liu, Xiong Luo, and Long Wang. An lstm approach to short text sen- timent classification with word embeddings. In Proceed- ings of the 30th conference on computational linguistics and speech processing (ROCLING 2018), pages 214-223, 2018.
  25. [61]61. Wang et al., 2021] Yaqing Wang, Song Wang, Quanming Yao, and Dejing Dou. Hierarchical heterogeneous graph representation learning for short text classification. In Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3091-3101, 2021.
  26. [63]63. Pengyu Wang, Linyang Li, Ke Ren, Bo- tian Jiang, Dong Zhang, and Xipeng Qiu. SeqXGPT: Sentence-level AI-generated text detection. In Proceed- ings of the 2023 Conference on Empirical Methods in Nat- ural Language Processing, pages 1144-1156, Singapore, Dec 2023. Association for Computational Linguistics.
  27. [64]64. Xia and Wang, 2023] Jinxiong Xia and Houfeng Wang. A sequence-to-sequence approach with mixed pointers to topic segmentation and segment labeling. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining, pages 2683-2693, 2023.
  28. [66]66. Jinxiong Xia, Cao Liu, Jiansong Chen, Yuchen Li, Fan Yang, Xunliang Cai, Guanglu Wan, and Houfeng Wang. Dialogue topic segmentation via paral- lel extraction network with neighbor smoothing. In Pro- ceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2126-2131, 2022.
  29. [68]68. Linzi Xing, Brad Hackinen, Giuseppe Carenini, and Francesco Trebbi. Improving context mod- eling in neural topic segmentation. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Associ- ation for Computational Linguistics and the 10th Interna- tional Joint Conference on Natural Language Processing, pages 626-636, 2020.
  30. [70]70. Hai Yu, Chong Deng, Qinglin Zhang, Ji- aqing Liu, Qian Chen, and Wen Wang. Improving long document topic segmentation models with enhanced co- herence modeling. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5592-5605, 2023.
  31. [72]72. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019.
  32. [74]74. Zijie Zeng, Lele Sha, Yuheng Li, Kaixun Yang, Dragan Gašević, and Guanliang Chen. Towards au- tomatic boundary detection for human-ai collaborative hy- brid essay in education. Proceedings of the 38th AAAI Conference on Artificial Intelligence, 2024.

Cited by

Sorted by citations

Related

Authors
Zeng, Z., Liu, S., Sha, L., Li, Z., Yang, K., Liu, S., Gasevic, D., & Chen, G.
Affiliations
  • Monash University (AU)
  • Central China Normal University (CN)
ORCID
Venue
Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence
Publisher
International Joint Conferences on Artificial Intelligence Organization
Published
2024-08
Pages
7545-7553
DOI
10.24963/ijcai.2024/835
OpenAlex
W4401023530
Cited by
8
References
4
Citations by year
2026: 2 · 2025: 6
FWCI
2.31
Type
conference paper
Language
en
Open access
Locations
  • International Joint Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
Keywords
AI-generated text detection, hybrid texts, human-AI collaboration, authorship detection, text segmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review