AI vs. Human -- Differentiation Analysis of Scientific Content Generation
Open accessThe DOI matches its OpenAlex record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Recent neural language models have taken a significant step forward in producing remarkably controllable, fluent, and grammatical text. Although studies have found that AI-generated text is not distinguishable from human-written text for crowd-sourcing workers, there still exist errors in AI-generated text which are even subtler and harder to spot. We primarily focus on the scenario in which scientific AI writing assistant is deeply involved. First, we construct a feature description framework to distinguish between AI-generated text and human-written text from syntax, semantics, and pragmatics based on the human evaluation. Then we utilize the features, i.e., writing style, coherence, consistency, and argument logistics, from the proposed framework to analyze two types of content. Finally, we adopt several publicly available methods to investigate the gap of between AI-generated scientific text and human-written scientific text by AI-generated scientific text detection models. The results suggest that while AI has the potential to generate scientific content that is as accurate as human-written content, there is still a gap in terms of depth and overall quality. The AI-generated scientific content is more likely to contain errors in factual issues. We find that there exists a "writing style" gap between AI-generated scientific text and human-written scientific text. Based on the analysis result, we summarize a series of model-agnostic and distribution-agnostic features for detection tasks in other domains. Findings in this paper contribute to guiding the optimization of AI models to produce high-quality content and addressing related ethical and security concerns.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Generative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218 · cited 5×
“Black et al., (2021) combine source-domain data with in-domain labeled data to solve the problem of detecting GPT-2-generated technical research text. During the Scholarly Document Processing workshop at COLING 2022, Kashnitsky et al., (2022) proposed the task and corresponding dataset on the detection of automaticall…”
BackgroundMethodInfluentialLanguage Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
Deepfake Video Detection Using Recurrent Neural Networks
Guera, D., & Delp, E. J. (2018) · 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2018.8639163 · Cited by 1,285
Rasmy, L., Xiang, Y., Xie, Z., Tao, C., & Zhi, D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00455-y · Cited by 886
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · cited 1×
“Similar to other LLMs, we find that the ChatGPT suffers from the hallucination problem, i.e., “reference hallucination”, in generated scientific text (Bang et al., 2023).”
BackgroundGalactica: A Large Language Model for Science
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R. (2022) · arXiv · doi:10.48550/arxiv.2211.09085 · Cited by 605
Köbis, N., & Mossink, L. D. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106553 · Cited by 406
All the News That’s Fit to Fabricate: AI-Generated Text as a Tool of Media Misinformation
Kreps, S., McCain, R. M., & Brundage, M. (2022) · Journal of Experimental Political Science · doi:10.1017/xps.2020.37 · Cited by 360 · cited 1×
“[30] carried out experiments to study the influence for opinions on foreign policy of AI-generated texts.”
BackgroundAI-generated characters for supporting personalized learning and well-being
Pataranutaporn, P., Danry, V., Leong, J., Punpongsanon, P., Novy, D., Maes, P., & Sra, M. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00417-9 · Cited by 358 · cited 1×
“Early computer-generated content can be easily distinguished by the general public from human-generated content (Pataranutaporn et al., 2021).”
BackgroundHuman heuristics for AI-generated language are flawed
Jakesch, M., Hancock, J. T., & Naaman, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208839120 · Cited by 342
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324
The Epistemic Threat of Deepfakes
Fallis, D. (2021) · Philosophy & Technology · doi:10.1007/s13347-020-00419-2 · Cited by 266 · cited 2×
“Deepfakes provide information that sounds right but is not truthful, which can lead people to acquire false beliefs [18].”
BackgroundMachine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E., Japkowicz, N., & Viktor, H. (2022) · arXiv · doi:10.48550/arxiv.2210.07321 · Cited by 196 · cited 2×
“Machine-generated text or AI-generated text is a natural language text generated, rewritten or expanded by a machine or algorithm (Crothers et al., 2022).”
BackgroundMachine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E. N., Japkowicz, N., & Viktor, H. L. (2023) · IEEE Access · doi:10.1109/access.2023.3294090 · Cited by 196 · cited 2×
“Machine-generated text or AI-generated text is a natural language text generated, rewritten or expanded by a machine or algorithm (Crothers et al., 2022).”
BackgroundLabbé, C., & Labbé, D. (2013) · Scientometrics · doi:10.1007/s11192-012-0781-y · Cited by 132
Lee, J., & Shin, S. Y. (2022) · Media Psychology · doi:10.1080/15213269.2021.2007489 · Cited by 89 · cited 1×
“Deepfake[37], researchers have conducted extensive research from the technical, cognitive perspective, and social perspectives[18, 21, 26, 33, 60].”
BackgroundDugan, L., Ippolito, D., Kirubarajan, A., Shi, S., & Callison-Burch, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26501 · Cited by 54
‘Tortured phrases’ give away fabricated research papers
Else, H. (2021) · Nature · doi:10.1038/d41586-021-02134-0 · Cited by 53 · cited 1×
“The NLG models with strong text generation ability can be used to facilitate plagiarism[12, 17].”
BackgroundHundreds of gibberish papers still lurk in the scientific literature
Van Noorden, R. (2021) · Nature · doi:10.1038/d41586-021-01436-7 · Cited by 16 · cited 1×
“Since the release of SCIgen in 2005, researchers have found that fake papers generated by SCIgen were published by academic paper publishers such as Springer and IEEE (Van Noorden, 2014, 2021).”
Background
Cited by
Testing of detection tools for AI-generated text
Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023) · International Journal for Educational Integrity · doi:10.1007/s40979-023-00146-z · Cited by 582
“Firstly, several researchers has studied human abilities to detect machine-generated texts (e.g. Guo et al., 2023; Ippolito et al., 2020; Ma et al., 2023).”
BackgroundCasal, J. E., & Kessler, M. (2023) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2023.100068 · Cited by 261
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Wong, D. F., & Chao, L. S. (2023) · arXiv · doi:10.48550/arxiv.2310.14724 · Cited by 212
“…responses, certain signifiers, for instance, the explicit statement “I am an AI language model”, may facilitate human adjudication, but such examples are less Recent studies [40, 41, 42, 43] have highlighted significant disparities between human-written and LLM-generated text, such as ChatGPT.”
BackgroundBetter by You, better than Me? ChatGPT-3 as writing assistance in students’ essays.
Bašić, Ž., Banovac, A., Kružić, I., & Jerković, I. (2023) · doi:10.35542/osf.io/n5m7s · Cited by 189
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212
Metadata
- Authors
- Ma, Y., Liu, J., Fan, Y., Cheng, Q., Huang, Y., Lu, W., & Liu, X.
- Venue
- arXiv (Cornell University)
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2301.10416
- OpenAlex
- W4318239126
- Cited by
- 68
- References
- 22
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.