Is ChatGPT Involved in Texts? Measure the Polish Ratio to Detect ChatGPT-Generated Text
Open accessAbstract
The remarkable capabilities of large-scale language models, such as ChatGPT, in text generation have impressed readers and spurred researchers to devise detectors to mitigate potential risks, including misinformation, phishing, and academic dishonesty. Despite this, most previous studies have been predominantly geared towards creating detectors that differentiate between purely ChatGPT-generated texts and human-authored texts. This approach, however, fails to work on discerning texts generated through human-machine collaboration, such as ChatGPT-polished texts. Addressing this gap, we introduce a novel dataset termed HPPT (ChatGPT-polished academic abstracts), facilitating the construction of more robust detectors. It diverges from extant corpora by comprising pairs of human-written and ChatGPT-polished abstracts instead of purely ChatGPT-generated texts. Additionally, we propose the "Polish Ratio" method, an innovative measure of the degree of modification made by ChatGPT compared to the original human-written text. It provides a mechanism to measure the degree of ChatGPT influence in the resulting text. Our experimental results show our proposed model has better robustness on the HPPT dataset and two existing datasets (HC3 and CDB). Furthermore, the "Polish Ratio" we proposed offers a more comprehensive explanation by quantifying the degree of ChatGPT involvement.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Yang, Jiang, and Li (2023); Shi et al. (2023); He et al. (2023) conducted adversarial data augmentation on LLM-generated text, and findings indicated that models trained on meticulously augmented data exhibited commendable robustness against potential attacks.
“Yang, Jiang, and Li (2023); Shi et al. (2023); He et al. (2023) conducted the adversarial data augmentation process on LLM-generated text, the findings of which indicated that models trained on meticulously augmented data exhibited commendable robustness against potential attacks.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Yang, Jiang, and Li 2023
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredRecent research has addressed robustness against specific attacks but often neglects potential threats posed by other attack forms.
“While recent research, such as Yang, Jiang, and Li (2023), has addressed robustness against specific attacks, it often neglects potential threats posed by other attack forms.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Yang, Jiang, and Li (2023)
Section 3.1 says existing detection datasets focus on purely ChatGPT-generated text and ignore ChatGPT-polished text; the Error Analysis also shows a baseline detector failing on a polishing attack. This supports the narrower point that a particular form of altered text was overlooked, but the paper does not establish the broader claim that recent research addresses robustness against specific attacks while often neglecting other attack forms generally. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
Cited by 16
Research works printed but not linked: 18
- [1][1] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- [2][2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
- [5][5] Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597, 2023. · arXiv
- [6][6] Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.11156, 2023. · arXiv
- [8][8] Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. The science of detecting llm-generated texts. arXiv preprint arXiv:2303.07205, 2023. · arXiv
- [9][9] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305, 2023. · arXiv
- [10][10] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint arXiv:2301.10226, 2023. 11 Is ChatGPT Involved in Texts? Measure the Polish Ratio to Detect ChatGPT-Generated Text · arXiv
- [13][13] Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116, Florence, Italy, July 2019. Association for Computational Linguistics.
- [14][14] Hasan Mesut Meral, Bülent Sankur, A Sumru Özsoy, Tunga Güngör, and Emre Sevinç. Natural language watermarking via morphosyntactic alterations. Computer Speech & Language, 23(1):107–125, 2009.
- [15][15] Marco Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 97–101, San Diego, California, June 2016. Association for Computational Linguistics.
- [16][16] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765–4774. Curran Associates, Inc., 2017.
- [17][17] Hohyun Hwang and Younghoon Lee. Semi-supervised learning based on auto-generated lexicon using XAI in sentiment analysis. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 593–600, Held Online, September 2021. INCOMA Ltd.
- [18][18] Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. In EMNLP. Association for Computational Linguistics, 2019.
- [19][19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. · arXiv
- [20][20] Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. Gpt detectors are biased against non-native english writers. arXiv preprint arXiv:2304.02819, 2023. · arXiv
- [21][21] Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203, 2019. · arXiv
- [22][22] Yudong Li, Yuqing Zhang, Zhe Zhao, Linlin Shen, Weijie Liu, Weiquan Mao, and Hui Zhang. CSL: A large-scale Chinese scientific literature dataset. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3917–3923, Gyeongju, Republic of Korea, October 2022. International Committee on Compu- tational Linguistics.
- [23][23] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 12 · arXiv
- Authors
- Yang, L., Jiang, F., & Li, H.
- Affiliations
- Chinese University of Hong Kong (HK)
- University of Science and Technology of China (CN)
- ORCID
- Li, Haizhou
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Volume
- 13
- Issue
- 2
- ISSN
- 2048-7703 (ISSN-L)
- arXiv
- 2307.11380
- OpenAlex
- W4391758837
- Cited by
- 16
- References
- 0
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 3 · 2023: 1
- FWCI
- 2.22
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- APSIPA Transactions on Signal and Information Processing (publishedVersion, cc-by-nc)
- arXiv (Cornell University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- Shenzhen Research Institute of Big Data
- Shenzhen Research Institute, City University of Hong Kong
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- ChatGPT, large language models, textual authenticity, AI-generated text detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
