How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection
Open accessAbstract
The introduction of ChatGPT has garnered widespread attention in both academic and industrial communities. ChatGPT is able to respond effectively to a wide range of human questions, providing fluent and comprehensive answers that significantly surpass previous public chatbots in terms of security and usefulness. On one hand, people are curious about how ChatGPT is able to achieve such strength and how far it is from human experts. On the other hand, people are starting to worry about the potential negative impacts that large language models (LLMs) like ChatGPT could have on society, such as fake news, plagiarism, and social security issues. In this work, we collected tens of thousands of comparison responses from both human experts and ChatGPT, with questions ranging from open-domain, financial, medical, legal, and psychological areas. We call the collected dataset the Human ChatGPT Comparison Corpus (HC3). Based on the HC3 dataset, we study the characteristics of ChatGPT's responses, the differences and gaps from human experts, and future directions for LLMs. We conducted comprehensive human evaluations and linguistic analyses of ChatGPT-generated content compared with that of humans, where many interesting results are revealed. After that, we conduct extensive experiments on how to effectively detect whether a certain text is generated by ChatGPT or humans. We build three different detection systems, explore several key factors that influence their effectiveness, and evaluate them in different scenarios. The dataset, code, and models are all publicly available at https://github.com/Hello-SimpleAI/chatgpt-comparison-detection.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MPT Chat is based on MPT 7B and fine-tuned on ShareGPT, Alpaca, Anthropic HH, HC3, and Evol-Instruct.
“MPT Chat is based on MPT 7B 15 fine-tuned on the ShareGPT data, the Alpaca data, Anthropic HH, HC3 (Guo et al., 2023), and Evol-Instruct.”
From FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023) · cites this paper as Guo et al., 2023
LLMs beat traditional text generation methods in producing human-like text.
“First, LLMs beat traditional text generation methods in producing human-like text (Jakesch et al., 2023;Guo et al., 2023;Clark et al., 2021).”
From Anatomy of an AI-powered malicious social botnet (2024) · cites this paper as Guo et al., 2023
No benchmark dataset is specified for AIGC detection; researchers have been developing custom data on their own.
“Firstly, no benchmark dataset is specified for the problem.Researchers have been developing custom data on their own [15,89].”
From Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity (2024) · cites this paper as [89]
Some papers claim very high results in AIGC detection using specific custom datasets, but the dataset is not open-sourced.
“Some papers claim very high results in AIGC detection using specific custom datasets, but the dataset is not open-sourced [89].”
From Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity (2024) · cites this paper as [89]
The AI-generated text part in open-source AIGC detection datasets is built using different prompting strategies applied to OpenAI models using the OpenAI API service.
“On the other hand, the AI-generated text part in those open-source datasets is built using different prompting strategies applied to OpenAI models using the OpenAI API service [89,90].”
From Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity (2024) · cites this paper as [89]
The human-written part of most open-source AIGC detection datasets was built using three major techniques: directly taking from other open-source datasets, using an available corpus such as in-class or homework exercises, TOEFL writing tasks, and GRE writing tasks, and manually collecting observations from online sources.
“The human-written part of most open-source AIGC detection datasets was built using three major techniques.First is directly taking from other open-source datasets such as the wikiHow text dataset [88].This technique was used to build the GPT-Sentinel dataset [89].Second is using an available corpus such as in-class or homework exercises, Test of English as a Foreign Language (TOEFL) writing tasks,”
From Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity (2024) · cites this paper as [89]
GPT-Sentinel was a pioneer work based on binary classification using transformer-based networks like Roberta and T5, achieving over 97% accuracy.
“GPT-Sentinel was another pioneer work based on binary classification using transformer-based networks like the Robustly Optimized BERT Pretraining Approach (Roberta) and Text-to-Text Transfer Transformer (T5) achieving over 97% accuracy [89].”
From Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity (2024) · cites this paper as [89]
LLM-generated texts are typically less emotional and more objective than human-written texts.
“LLM-generated texts are typically less emotional and more objective than human-written texts [Guo et al., 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Guo et al., 2023
Human-authored texts tend to be more coherent and exhibit greater lexical diversity compared to LLM-generated texts.
“Human-authored texts tend to be more coherent and exhibit greater lexical diversity compared to LLM-generated texts [Guo et al., 2023;Dugan et al., 2023;Zhang et al., 2024].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Guo et al., 2023
This ability to adapt makes attribution challenging, especially as LLMs rapidly evolve and their outputs change significantly over time.
“This ability to adapt makes attribution challenging, especially as LLMs rapidly evolve and their outputs change significantly over time [Guo et al., 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Guo et al., 2023
LLM-generated texts typically exhibit consistent syntax and a broader vocabulary.
“In contrast, LLMgenerated texts typically exhibit consistent syntax and a broader vocabulary [Guo et al., 2023].”
From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Guo et al., 2023
The maintainers proposed to incorporate indicators for identifying possible GenAI usage and suggested tool support.
“In addition, the maintainers proposed to incorporate indicators for identifying possible GenAI usage and suggested tool support, for example, a ChatGPT detector published on HuggingFace [21].”
From Self-Admitted GenAI Usage in Open-Source Software (2026) · cites this paper as [21]
Cited by
Sorted by citations
ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.9 · Cited by 1,806 · see citing works
ChatGPT: Jack of all trades, master of none
Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101861 · Cited by 758 · see citing works
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · see citing works
The role of ChatGPT in higher education: Benefits, challenges, and future research directions
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.29 · Cited by 561 · see citing works
Testing of detection tools for AI-generated text
Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023) · International Journal for Educational Integrity · doi:10.1007/s40979-023-00146-z · Cited by 582 · see citing works
“Firstly, several researchers has studied human abilities to detect machine-generated texts (e.g. Guo et al., 2023; Ippolito et al., 2020; Ma et al., 2023).”
BackgroundBoscardin, C. K., Gin, B., Golde, P. B., & Hauer, K. E. (2024) · Academic Medicine · doi:10.1097/acm.0000000000005439 · Cited by 533 · see citing works
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514 · see citing works
Benchmarking Large Language Models in Retrieval-Augmented Generation
Chen, J., Lin, H., Han, X., & Sun, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29728 · Cited by 454 · see citing works
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407 · see citing works
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337 · see citing works
“Guo et al. (2023) conducts linguistic analyses of ChatGPT’s writing against human experts and found that ChatGPT responses are strictly focused on the given question, more formal, objective, and less emotional.”
BackgroundResultA large-scale comparison of human-written versus ChatGPT-generated essays
Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-45644-9 · Cited by 338 · see citing works
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Lai, V. D., Ngo, N. T., Veyseh, A. P. B., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. H. (2023) · arXiv · doi:10.48550/arxiv.2304.05613 · Cited by 233 · see citing works
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212 · see citing works
Early applications of ChatGPT in medical practice, education and research
Sedaghat, S. (2023) · Clinical Medicine · doi:10.7861/clinmed.2023-0078 · Cited by 163 · see citing works
Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642596 · Cited by 160 · see citing works
A Survey on the Detection and Impacts of Deepfakes in Visual, Audio, and Textual Formats
Mubarak, R., Alsboui, T., Alshaikh, O., Inuwa-Dutse, I., Khan, S., & Parkinson, S. (2023) · IEEE Access · doi:10.1109/access.2023.3344653 · Cited by 132 · see citing works
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.2.12 · Cited by 132 · see citing works
Beyond human expertise: the promise and limitations of ChatGPT in suicide risk assessment
Elyoseph, Z., & Levkovich, I. (2023) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2023.1213141 · Cited by 116 · see citing works
The Science of Detecting LLM-Generated Text
Tang, R., Chuang, Y.-N., & Hu, X. (2024) · Communications of the ACM · doi:10.1145/3624725 · Cited by 125 · see citing works
Walters, W. H. (2023) · Open Information Science · doi:10.1515/opis-2022-0158 · Cited by 109 · see citing works
Roos, J., Kasapovic, A., Jansen, T., & Kaczmarczyk, R. (2023) · JMIR Medical Education · doi:10.2196/46482 · Cited by 98 · see citing works
Cheong, I., Xia, K., Feng, K. J. K., Chen, Q. Z., & Zhang, A. X. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3659048 · Cited by 91 · see citing works
MAGE: Machine-generated Text Detection in the Wild
Li, Y., Li, Q., Cui, L., Bi, W., Wang, Z., Wang, L., Yang, L., Shi, S., & Zhang, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.3 · Cited by 79 · see citing works
Self-citationClassification of human- and AI-generated texts: Investigating features for ChatGPT
Mindner, L., Schlippe, T., & Schaaff, K. (2023) · arXiv · doi:10.48550/arxiv.2308.05341 · Cited by 73
Impact of a Digital Scribe System on Clinical Documentation Time and Quality: Usability Study
van Buchem, M. M., Kant, I. M. J., King, L., Kazmaier, J., Steyerberg, E. W., & Bauer, M. P. (2024) · JMIR AI · doi:10.2196/60020 · Cited by 70 · see citing works
Anatomy of an AI-powered malicious social botnet
Yang, K., & Menczer, F. (2024) · Journal of Quantitative Description: Digital Media · doi:10.51685/jqd.2024.icwsm.7 · Cited by 71 · see citing works
Deepfake: definitions, performance metrics and standards, datasets, and a meta-review
Altuncu, E., Franqueira, V. N. L., & Li, S. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1400024 · Cited by 65 · see citing works
“Human ChatGPT comparison corpus (Guo et al., 2023a,b): This bilingual (i.e., English and Chinese) dataset involves questions as well as answers to them provided by humans and ChatGPT to enable a comparison between human and ChatGPT answers to the same questions.”
BackgroundThe False Promise of Imitating Proprietary LLMs
Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., & Song, D. (2023) · arXiv · doi:10.48550/arxiv.2305.15717 · Cited by 48 · see citing works
“• HC3 (Guo et al., 2023): we use the ChatGPT responses from the English Human-ChatGPT Comparison Corpus.”
MethodAuthorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges
Huang, B., Chen, C., & Shu, K. (2025) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3715073.3715076 · Cited by 42 · see citing works
“Recently, many commercial platforms have also integrated “Humanizer” functions, reflecting an emerging detector-evasion ecosystem in which detection and text rewriting tools co-evolve.”
InfluentialTesting of Detection Tools for AI-Generated Text
Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023) · arXiv · doi:10.48550/arxiv.2306.15666 · Cited by 32 · see citing works
ModelShield: Adaptive and Robust Watermark Against Model Extraction Attack
Pang, K., Qi, T., Wu, C., Bai, M., Jiang, M., & Huang, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3530691 · Cited by 29 · see citing works
Self-citationZhang, M., & Crosthwaite, P. (2026) · International Review of Applied Linguistics in Language Teaching · doi:10.1515/iral-2024-0196 · Cited by 26 · see citing works
Can Large Language Models Beat Wall Street? Unveiling the Potential of AI in Stock Selection
Fatouros, G., Metaxas, K., Soldatos, J., & Kyriazis, D. (2024) · doi:10.2139/ssrn.4693849 · Cited by 25 · see citing works
M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection
Wang, Y., Mansurov, J., Ivanov, P., Su, J., Shelmanov, A., Tsvigun, A., Mohammed Afzal, O., Mahmoud, T., Puccetti, G., Arnold, T., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.218 · Cited by 25 · see citing works
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
Hans, A., Schwarzschild, A., Cherepanova, V., Kazemi, H., Saha, A., Goldblum, M., Geiping, J., & Goldstein, T. (2024) · arXiv · doi:10.48550/arxiv.2401.12070 · Cited by 24 · see citing works
M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection
Wang, Y., Mansurov, J., Ivanov, P., Su, J., Shelmanov, A., Tsvigun, A., Whitehouse, C., Afzal, O. M., Mahmoud, T., Sasaki, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.14902 · Cited by 19 · see citing works
“For example, Guo et al. (2023) detect whether a certain text (English and Chinese) is generated by ChatGPT or is human-written across several domains, Shijaku and Can-hasi (2023) identified whether TOEFL essays were human-written and generated by ChatGPT over a mini set (126 essays for each).”
BackgroundMethodYang, Z., Feng, Z., Huo, R., Lin, H., Zheng, H., Nie, R., & Chen, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126694 · Cited by 17 · see citing works
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · arXiv · doi:10.48550/arxiv.2305.14251 · Cited by 14 · see citing works
“MPT Chat is based on MPT 7B 15 fine-tuned on the ShareGPT data, the Alpaca data, Anthropic HH, HC3 (Guo et al., 2023), and Evol-Instruct.”
MethodNaing, S. Z. S., & Udomwong, P. (2024) · Data Intelligence · doi:10.1162/dint_a_00250 · Cited by 15 · see citing works
OLMo: Accelerating the Science of Language Models
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.00838 · Cited by 10 · see citing works
Pudasaini, S., Miralles-Pechuán, L., Lillis, D., & Salvador, M. L. (2024) · arXiv · doi:10.48550/arxiv.2407.13105 · Cited by 8 · see citing works
Enhancing the Robustness of AI-Generated Text Detectors: A Survey
Liu, X., Li, Y., & Li, K. (2025) · Mathematics · doi:10.3390/math13132145 · Cited by 10 · see citing works
National Institute of Standards and Technology (US) (2024) · doi:10.6028/nist.ai.100-4 · Cited by 9 · see citing works
Teachers vs. Chat GPT: identifying and naming acts of online sexual abuse
Dolev-Cohen, M., & Henry, N. (2025) · Information, Communication & Society · doi:10.1080/1369118x.2025.2492585 · Cited by 8 · see citing works
“It understands and interprets simple and complex user queries and responds in natural language (Alberts et al., 2023; Guo et al., 2023).”
Zhang, M., & Zhang, J. (2026) · Applied Linguistics · doi:10.1093/applin/amaf032 · Cited by 7 · see citing works
Charting the Future of AI in Project-Based Learning: A Co-Design Exploration with Students
Zheng, C., Yuan, K., Guo, B., Mogavi, R. H., Peng, Z., Ma, S., & Ma, X. (2024) · arXiv · doi:10.48550/arxiv.2401.14915 · Cited by 70 · see citing works
Self-citationHANSEN: Human and AI Spoken Text Benchmark for Authorship Analysis
Tripto, N., Uchendu, A., Le, T., Setzu, M., Giannotti, F., & Lee, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.916 · Cited by 6 · see citing works
Cheng, Z., Zhou, L., Jiang, F., Wang, B., & Li, H. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714770 · Cited by 3 · see citing works
Self-Admitted GenAI Usage in Open-Source Software
Xiao, T., Fan, Y., Calefato, F., Treude, C., Kula, R. G., Hata, H., & Baltes, S. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3681886 · Cited by 1 · see citing works
Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2023) · arXiv · doi:10.48550/arxiv.2308.02312
- Authors
- Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., & Wu, Y.
- Affiliations
- Shanghai University of Finance and Economics (CN)
- Harbin Institute of Technology (CN)
- Beijing Language and Culture University (CN)
- Xidian University (CN)
- Queen's University (CA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2301.07597
- OpenAlex
- W4317553041
- Cited by
- 318
- References
- 0
- Citations by year
- 2026: 15 · 2025: 65 · 2024: 108 · 2023: 128 · 2021: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Shanghai University of Finance and Economics
- Beijing Language and Culture University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Keywords
- ChatGPT, large language models, human-machine comparison, financial issues, medical questions, legal issues, fake news, plagiarism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
