The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Generative AI models should include detection mechanisms as a condition for public release
Abstract
The new wave of ‘foundation models’—general-purpose generative AI models, for production of text (e.g., ChatGPT) or images (e.g., MidJourney)—represent a dramatic advance in the state of the art for AI. But their use also introduces a range of new risks, which has prompted an ongoing conversation about possible regulatory mechanisms. Here we propose a specific principle that should be incorporated into legislation: that any organization developing a foundation model intended for public use must demonstrate a reliable detection mechanism for the content it generates, as a condition of its public release. The detection mechanism should be made publicly available in a tool that allows users to query, for an arbitrary item of content, whether the item was generated (wholly or partly) by the model. In this paper, we argue that this requirement is technically feasible and would play an important role in reducing certain risks from new AI models in many domains. We also outline a number of options for the tool’s design, and summarize a number of points where further input from policymakers and researchers would be required.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“while automatically recognizing their AI provenance [2, 15] will likely become an uphill struggle as generative AI tools become more sophisticated and therefore produce output that will be increasingly difficult to detect.”
From False authorship: an explorative case study around an AI-generated article published under my name (2025) · cites this paper as [15]
In the citing paper
“Requiring model developers to release detection tools for their models’ generated text [15] is likely to be ineffective, due to the existence of open-source models that can be easily adjusted and run even on a laptop.”
From False authorship: an explorative case study around an AI-generated article published under my name (2025) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1×
“…is a citizen receiving the text as professional advice from her doctor or lawyer, she may want to know how thoroughly it has been checked for errors, given the known problems of errors in FM output (e.g., Ji et al., 2023) and overreliance on FM output by human operators (e.g., Wang et al., 2023).”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
Ethical Considerations of Using ChatGPT in Health Care
Wang, C., Liu, S., Yang, H., Guo, J., Wu, Y., & Liu, J. (2023) · Journal of Medical Internet Research · doi:10.2196/48009 · Cited by 541
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2301.11305 · Cited by 151
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
Krishna, K., Song, Y., Karpinska, M., Wieting, J., & Iyyer, M. (2023) · arXiv · doi:10.48550/arxiv.2303.13408 · Cited by 150 · cited 1×
“This method was recently demonstrated for text by Krishna et al. (2023). plagiarism detector is essentially an information retrieval (IR) device: the companies at the fore-front of FM content generation also have huge expertise in this area, and would be very well placed to provide detectors of…”
Method(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.2.12 · Cited by 132 · cited 1×
“There are many tools that attempt to distinguish AI-generated from non-AI-generated content, both for text (e.g., Chaka, 2023) and images (e.g., Stroebel et al., 2023).”
Background
Research works printed but not linked: 19
- Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, p., Tat Lee, Y., Li, Y., Lundberg, S., Nori, H., Palangi, H., Tulio Ribeiro, M., Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712. · arXiv
- Gao, S. and Gao, A. (2023). On the Origin of LLMs: An Evolutionary Tree and Graph for 15,821 Large Language Models. https://arxiv.org/pdf/2307.09793.pdf · arXiv
- Goldwasser, S., & Rothblum, G. N. (2007). On best-possible obfuscation. In Theory of Cryptography: 4th Theory of Cryptography Conference, TCC 2007, Proceedings 4 (pp. 194–213). Springer · DOI
- Hefti, A., & Lareida, J. (2021). Competitive attention, superstars and the long tail. University of Zurich, Department of Economics, Working Paper, (383). · DOI
- Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. · arXiv
- Jakesch, M., Hancock, J. T., & Naaman, M. (2023). Human heuristics for AI-generated language are flawed. Proceedings of the National Academy of Sciences, 120(11), e2208839120. · DOI · Paper page
- Jiang, Z., Zhang, J., & Gong, N. Z. (2023). Evading watermark based detection of AI-Generated content. arXiv preprint arXiv:2305.03807. · DOI
- Kirchenbauer, J., Geiping, J., Wen, Y., Shu, M., Saifullah, K., Kong, K., Fernando, K., Saha, A., Goldblum, M., Goldstein, T. (2023). On the reliability of watermarks for large language models. arXiv preprint arXiv:2306.04634. · arXiv
- Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023). A watermark for large language models. arXiv preprint arXiv:2301.10226. · arXiv
- Messer, K. D., Costanigro, M., & Kaiser, H. M. (2017). Labeling food processes: The good, the bad and the ugly. Applied Economic Perspectives and Policy, 39(3), 407–427. · DOI
- Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2023). Auditing large language models: a three-layered approach. arXiv preprint arXiv:2302.08500. · DOI · Paper page
- OpenAI (2023a). GPT-4 Technical Report. arXiv:2303.08774v2. · arXiv
- Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023). Can AI-Generated Text be Reliably Detected? arXiv preprint arXiv:2303.11156. · arXiv
- Shi, Z., Wang, Y., Yin, F., Chen, X., Chang, K. W., & Hsieh, C. J. (2023). Red teaming language model detectors with language models. arXiv preprint arXiv:2305.19713. · arXiv
- Stroebel, L., Llewellyn, M., Hartley, T., Ip, T. S., & Ahmed, M. (2023). A systematic literature review on the effectiveness of deepfake detection techniques. Journal of Cyber Security Technology, 7(2), 83–113. · DOI · Paper page
- Tamkin, A., Brundage, M., Clark, J., & Ganguli, D. (2021). Understanding the capabilities, limitations, and societal impact of large language models. arXiv preprint arXiv:2102.02503. · arXiv
- Tulchinskii, E., Kuznetsov, K., Kushnareva, L., Cherniavskii, D., Barannikov, S., Piontkovskaya, I., Nikolenko, S., Burnaev, E. (2023). Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts. arXiv preprint arXiv:2306.04723. · arXiv
- Waltzer, T., Cox, R. L., & Heyman, G. D. (2023). Testing the ability of teachers and students to differentiate between essays generated by ChatGPT and high school students. Human Behavior and Emerging Technologies. https://doi.org/10.1155/2023/1923981 · DOI · Paper page
- Zhao, Y., Pang, T., Du, C., Yang, X., Cheung, N. M., & Lin, M. (2023). A recipe for watermarking diffusion models. arXiv preprint arXiv:2303.10137 · arXiv
Cited by
Sorted by citations
Mapping the ethics of generative AI: a comprehensive scoping review
Hagendorff, T. (2024) · Minds and Machines · doi:10.1007/s11023-024-09694-w · Cited by 141
False authorship: an explorative case study around an AI-generated article published under my name
Spinellis, D. (2025) · Research Integrity and Peer Review · doi:10.1186/s41073-025-00165-z · Cited by 37
Metadata
- Authors
- Knott, A., Pedreschi, D., Chatila, R., Chakraborti, T., Leavy, S., Baeza-Yates, R., Eyers, D., Trotman, A., Teal, P. D., Biecek, P., Russell, S., & Bengio, Y.
- Venue
- Ethics and Information Technology
- Published
- 2023-10-28
- Volume
- 25
- Issue
- 4
- ISSN
- 1388-1957
- 1572-8439
- Cited by
- 34
- References
- 7
- Language
- en
- Funders
- Victoria University of Wellington
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Generative AI models should include detection mechanisms as a condition for public release. https://app.cochat.ai/papers/10.1007/s10676-023-09728-4
- The paper
- Knott, A., Pedreschi, D., Chatila, R., Chakraborti, T., Leavy, S., Baeza-Yates, R., Eyers, D., Trotman, A., Teal, P. D., Biecek, P., Russell, S., & Bengio, Y. (2023). Generative AI models should include detection mechanisms as a condition for public release. Ethics and Information Technology. https://doi.org/10.1007/s10676-023-09728-4
