CoChat
Sign Up
PreprintCited by 87Authorship Attribution and ProfilingAcademic integrity and plagiarismEthics and Social Impacts of AI

Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews

Open access
Liang, Weixin, Izzo, Zachary, Zhang, Yaohui, Lepp, Haley, Cao, Hancheng, Zhao, Xuandong, Chen, Lingjiao, Ye, Haotian +4 more·2024·arXiv
DOI 10.48550/arxiv.2403.07183arXiv 2403.07183
Open this paper in CoChat — free accountOpen PDF

Abstract

We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM). Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently examine real-world LLM-use at the corpus level. We apply this approach to a case study of scientific peer review in AI conferences that took place after the release of ChatGPT: ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023. Our results suggest that between 6.5% and 16.9% of text submitted as peer reviews to these conferences could have been substantially modified by LLMs, i.e. beyond spell-checking or minor writing updates. The circumstances in which generated text occurs offer insight into user behavior: the estimated fraction of LLM-generated text is higher in reviews which report lower confidence, were submitted close to the deadline, and from reviewers who are less likely to respond to author rebuttals. We also observe corpus-level trends in generated text which may be too subtle to detect at the individual level, and discuss the implications of such trends on peer review. We call for future interdisciplinary work to examine how LLM use is changing our information and knowledge practices.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.07183

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 23 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Human capability to discern AI-generated text from human-written content barely exceeds that of a random classifier.

    “Human capability to discern AI-generated text from human-written content barely exceeds that of a random classifier (Gehrmann et al., 2019;Else, 2023;Clark et al., 2021), heightening the risk that unsubstantiated generated text can masquerade as authoritative, evidence-based writing.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Gehrmann et al., 2019 source
    • cites Else, 2023 source
    • cites Clark et al., 2021 source

      Section 2.4 (Results; page not available in our copy) reports 49.9% accuracy for distinguishing GPT-3 text from human writing, effectively equal to 50% random chance. Accuracy was 57.9% for GPT-2 and significantly above chance, so the claim overgeneralizes a result that depends on the model tested. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Premisecitation

    There is growing evidence that the introduction of LLM content in information ecosystems can contribute to output homogenization.

    “There is growing evidence that the introduction of LLM content in information ecosystems can contribute to to output homogenization (Liu et al., 2024;Bommasani et al., 2022;Kleinberg & Raghavan, 2021).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Liu et al., 2024 source
    • cites Bommasani et al., 2022 source

      Section 5.1 (p. 10) reports that sharing a language foundation model can contribute to homogeneous outcomes, depending on how the model is adapted. But the paper measures whether decision systems produce the same outcomes for individuals or groups; it does not study whether introducing LLM-generated content into information ecosystems homogenizes that content. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Kleinberg & Raghavan, 2021 source
  • Findingown data

    Low self-reported confidence in reviews is associated with an increase of ChatGPT usage.

    “Low Confidence Correlation: Low self-reported confidence in reviews are associated with an increase of ChatGPT usage (Section § 4.7, Figure 11).”
    • Also stated in: § Abstract · hedged
      “the estimated fraction of LLM-generated text is higher in reviews which report lower confidence, were submitted close to the deadline, and from reviewers who are less likely to respond to author rebuttals.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Documents in the target corpus are generated from the mixture distribution (1-α)P + αQ, where P is the distribution of human-written documents and Q is the distribution of AI-generated documents.
    • The authors state this as an assumption: We have access to a collection of reviews known to contain only human-authored text, along with the associated review questions and the reviewed papers.
    • The authors state this as an assumption: Each document is represented as a set of token occurrences rather than a list of token counts.
    • The authors state this as an assumption: Documents in the target corpus are drawn independently from the mixture distribution.
    • The authors state this as an assumption: The likelihood of a document can be estimated as the product of token occurrence probabilities (and their complements) under the human and AI distributions.
    • Rests on an assumption: Token occurrences within a document are independent.
  • Findingown data

    Reviews containing scholarly citations are less likely to be AI modified or generated than those lacking such citations.

    “Reference Effect: Reviews containing scholarly citations are less likely to be AI modified or generated than those lacking such citations (Section § 4.7, Figure 8).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Documents in the target corpus are generated from the mixture distribution (1-α)P + αQ, where P is the distribution of human-written documents and Q is the distribution of AI-generated documents.
    • The authors state this as an assumption: We have access to a collection of reviews known to contain only human-authored text, along with the associated review questions and the reviewed papers.
    • The authors state this as an assumption: Each document is represented as a set of token occurrences rather than a list of token counts.
    • The authors state this as an assumption: Documents in the target corpus are drawn independently from the mixture distribution.
    • The authors state this as an assumption: The likelihood of a document can be estimated as the product of token occurrence probabilities (and their complements) under the human and AI distributions.
    • Rests on an assumption: Token occurrences within a document are independent.
  • Findingown data

    Estimated ChatGPT usage in reviews spikes significantly within 3 days of review deadlines.

    “Deadline Effect: Estimated ChatGPT usage in reviews spikes significantly within 3 days of review deadlines (Section § 4.7, Figure 7).”
    • Also stated in: § Abstract · hedged
      “the estimated fraction of LLM-generated text is higher in reviews which report lower confidence, were submitted close to the deadline, and from reviewers who are less likely to respond to author rebuttals.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Documents in the target corpus are generated from the mixture distribution (1-α)P + αQ, where P is the distribution of human-written documents and Q is the distribution of AI-generated documents.
    • The authors state this as an assumption: We have access to a collection of reviews known to contain only human-authored text, along with the associated review questions and the reviewed papers.
    • The authors state this as an assumption: Each document is represented as a set of token occurrences rather than a list of token counts.
    • The authors state this as an assumption: Documents in the target corpus are drawn independently from the mixture distribution.
    • The authors state this as an assumption: The likelihood of a document can be estimated as the product of token occurrence probabilities (and their complements) under the human and AI distributions.
    • Rests on an assumption: Token occurrences within a document are independent.
  • Classified as probably not a substantial claim. 31
  • May combine two claims. 29
  • The quote only partly states this claim. 10

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    GPT-4 substantially outperforms other LLMs, including Bard, in the reviewing of scientific papers or proposals.

    “Recent studies have also found that GPT-4 substantially outperforms other LLMs, including Bard, in the reviewing of scientific papers or proposals (Liu & Shah, 2023).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Liu & Shah, 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent studies have found that ChatGPT substantially outperforms other LLMs, including Bard, in the reviewing of scientific papers or proposals.

    “However, recent studies have found that ChatGPT substantially outperforms other LLMs, including Bard, in the reviewing of scientific papers or proposals (Liang et al., 2023c;Liu & Shah, 2023).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Liang et al., 2023c

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Liu & Shah, 2023 source

      The Introduction and §3.2 report that GPT-4 identified errors in short papers while Bard and the other tested models did not. Section 5.1 says GPT-4 was used through ChatGPT for an abstract-comparison task. The comparative result supports a narrower claim about GPT-4's error detection, not ChatGPT's general superiority in reviewing scientific papers or proposals. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Stylistic words are the most impacted during alignment fine-tuning.

    “This remotely aligns with the findings in the literature (Lin et al., 2023), which indicate that stylistic words are the most impacted during alignment fine-tuning.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Lin et al., 2023 source

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    The use of peer review data for research purposes raises important ethical considerations around reviewer consent, data licensing, and responsible use.

    “The use of peer review data for research purposes raises important ethical considerations around reviewer consent, data licensing, and responsible use (Dycke et al., 2023).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Dycke et al., 2023 source

  • May combine two claims.

    Findingown data

    A similar trend of increased frequency of certain tokens is found in NeurIPS but not in Nature Portfolio journals.

    “We find a similar trend in NeurIPS but not in Nature Portfolio journals.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Documents in the target corpus are generated from the mixture distribution (1-α)P + αQ, where P is the distribution of human-written documents and Q is the distribution of AI-generated documents.
    • The authors state this as an assumption: We have access to a collection of reviews known to contain only human-authored text, along with the associated review questions and the reviewed papers.
    • The authors state this as an assumption: Each document is represented as a set of token occurrences rather than a list of token counts.
    • The authors state this as an assumption: Documents in the target corpus are drawn independently from the mixture distribution.
    • The authors state this as an assumption: The likelihood of a document can be estimated as the product of token occurrence probabilities (and their complements) under the human and AI distributions.
    • Rests on an assumption: Token occurrences within a document are independent.

  • The quote only partly states this claim.

    Findingown data

    The same qualitative trend of α estimates pre-ChatGPT close to 0 and a sharp increase after ChatGPT release is observed when using verbs.

    “The same qualitative trend is observed: α estimates pre-ChatGPT are close to 0, and there is a sharp increase after the release of ChatGPT.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    There was a significant increase in AI-generated sentences after the release of ChatGPT for the ML venues, but not for Nature.

    “We find that there was a significant increase in AI-generated sentences after the release of ChatGPT for the ML venues, but not for Nature(Appendix D.2).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Documents in the target corpus are generated from the mixture distribution (1-α)P + αQ, where P is the distribution of human-written documents and Q is the distribution of AI-generated documents.
    • The authors state this as an assumption: We have access to a collection of reviews known to contain only human-authored text, along with the associated review questions and the reviewed papers.
    • The authors state this as an assumption: Each document is represented as a set of token occurrences rather than a list of token counts.
    • The authors state this as an assumption: Documents in the target corpus are drawn independently from the mixture distribution.
    • The authors state this as an assumption: The likelihood of a document can be estimated as the product of token occurrence probabilities (and their complements) under the human and AI distributions.
    • Rests on an assumption: Token occurrences within a document are independent.

  • May combine two claims.

    Findingown data

    With the exception of the BERT-based method, the predictions made by all of the classifiers remain nearly constant across all α levels, leading to poor performance for all of them.

    “With the exception of the BERT-based method, the predictions made by all of the classifiers remain nearly constant across all α levels, leading to poor performance for all of them.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The α estimates pre-ChatGPT are close to 0, and there is a sharp increase after the release of ChatGPT.

    “α estimates pre-ChatGPT are close to 0, and there is a sharp increase after the release of ChatGPT.”
    • Also stated in: § D.5. Results with Adverbs · plain
      “The same qualitative trend is observed: α estimates pre-ChatGPT are close to 0, and there is a sharp increase after the release of ChatGPT.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § documents in which token t appears total # documents in the corpus

    Both document-level and sentence-level units performed well in validation.

    “We validated both approaches using either a document or a sentence as the unit of x, and both performed well (Appendix D.8).”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Documents in the target corpus are generated from the mixture distribution (1-α)P + αQ, where P is the distribution of human-written documents and Q is the distribution of AI-generated documents.
    • The authors state this as an assumption: We have access to a collection of reviews known to contain only human-authored text, along with the associated review questions and the reviewed papers.
    • The authors state this as an assumption: Each document is represented as a set of token occurrences rather than a list of token counts.
    • The authors state this as an assumption: Documents in the target corpus are drawn independently from the mixture distribution.
    • The authors state this as an assumption: The likelihood of a document can be estimated as the product of token occurrence probabilities (and their complements) under the human and AI distributions.
    • Rests on an assumption: Token occurrences within a document are independent.

  • The quote only partly states this claim.

    Findingown data

    The algorithm's estimated α closely matches the ground-truth α when human peer reviews are mixed with LLM-expanded feedback at various ground-truth levels of α.

    “The α estimated by our algorithm closely matches the ground truth α.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). The quote only partly states this claim.

    Findingown data

    The same qualitative trend of α estimates pre-ChatGPT close to 0 and a sharp increase after ChatGPT release is observed using the model trained on reviews generated by GPT-3.5.

    “The same qualitative trend is observed: α estimates pre-ChatGPT are close to 0, and there is a sharp increase after the release of ChatGPT.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    While BERT's estimates for α seem at least positively correlated with the correct α value, the error in the estimate is still large compared to the high accuracy obtained by the paper's method.

    “While BERT's estimates for α seem at least positively correlated with the correct α value, the error in the estimate is still large compared to the high accuracy obtained by our method (see Figure 3 and Table 5).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § documents in which token t appears total # documents in the corpus

    Sentence-level unit performed slightly better than document-level for estimates.

    “We used a sentence as our main unit for estimates, as sentences perform slightly better.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Documents in the target corpus are generated from the mixture distribution (1-α)P + αQ, where P is the distribution of human-written documents and Q is the distribution of AI-generated documents.
    • The authors state this as an assumption: We have access to a collection of reviews known to contain only human-authored text, along with the associated review questions and the reviewed papers.
    • The authors state this as an assumption: Each document is represented as a set of token occurrences rather than a list of token counts.
    • The authors state this as an assumption: Documents in the target corpus are drawn independently from the mixture distribution.
    • The authors state this as an assumption: The likelihood of a document can be estimated as the product of token occurrence probabilities (and their complements) under the human and AI distributions.
    • Rests on an assumption: Token occurrences within a document are independent.

  • May combine two claims.

    Findingown data

    Applying the method to peer reviews simply edited by ChatGPT for typos and grammar yields a slight increase in estimated α, much smaller than the effect size seen in the real review corpus.

    “While there is a slight increase in the estimated α, it is much smaller than the effect size seen in the real review corpus in the previous section (denoted with dashed lines in the figure).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Documents in the target corpus are generated from the mixture distribution (1-α)P + αQ, where P is the distribution of human-written documents and Q is the distribution of AI-generated documents.
    • The authors state this as an assumption: We have access to a collection of reviews known to contain only human-authored text, along with the associated review questions and the reviewed papers.
    • The authors state this as an assumption: Each document is represented as a set of token occurrences rather than a list of token counts.
    • The authors state this as an assumption: Documents in the target corpus are drawn independently from the mixture distribution.
    • The authors state this as an assumption: The likelihood of a document can be estimated as the product of token occurrence probabilities (and their complements) under the human and AI distributions.
    • Rests on an assumption: Token occurrences within a document are independent.

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). The quote only partly states this claim.

    Findingnone given

    The same qualitative trend of α estimates pre-ChatGPT close to 0 and a sharp increase after ChatGPT release is observed when using nouns.

    “The same qualitative trend is observed: α estimates pre-ChatGPT are close to 0, and there is a sharp increase after the release of ChatGPT.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The model trained exclusively on ICLR data from 2018 to 2022 has a prediction error below 1.8% for NeurIPS '22 and under 2.4% for CoRL '22 across various ground truth α.

    “The prediction error remains below 1.8% across various ground truth α for NeurIPS '22 and under 2.4% for CoRL '22 (Figure 3, Supp.Table 5).”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Documents in the target corpus are generated from the mixture distribution (1-α)P + αQ, where P is the distribution of human-written documents and Q is the distribution of AI-generated documents.
    • The authors state this as an assumption: We have access to a collection of reviews known to contain only human-authored text, along with the associated review questions and the reviewed papers.
    • The authors state this as an assumption: Each document is represented as a set of token occurrences rather than a list of token counts.
    • The authors state this as an assumption: Documents in the target corpus are drawn independently from the mixture distribution.
    • The authors state this as an assumption: The likelihood of a document can be estimated as the product of token occurrence probabilities (and their complements) under the human and AI distributions.
    • Rests on an assumption: Token occurrences within a document are independent.

  • May combine two claims.

    Findingown data

    Two off-the-shelf AI text detectors (RADAR and Deepfake Text Detect) predict that either almost all (RADAR) or none (Deepfake) of the text are AI-generated, regardless of the true α level.

    “Two off-the-shelf classifiers predict that either almost all (RADAR) or none (Deepfake) of the text are AI-generated, regardless of the true α level.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Using adverbs, the same qualitative trend is observed: α estimates pre-ChatGPT are close to 0, and there is a sharp increase after the release of ChatGPT.

    “The same qualitative trend is observed: α estimates pre-ChatGPT are close to 0, and there is a sharp increase after the release of ChatGPT.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Using the model trained at the document level, the same qualitative trend is observed: α estimates pre-ChatGPT are close to 0, and there is a sharp increase after the release of ChatGPT.

    “The same qualitative trend is observed: α estimates pre-ChatGPT are close to 0, and there is a sharp increase after the release of ChatGPT.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    The framework predicts the proportion of AI data (alpha) very well when trained on LLaMA-2 Chat (70B) and Claude-2.1 generated data.

    “As shown in the two results tables, our framework predicts the proportion of AI data (i.e., alpha) very well.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    BERT estimates are correlated with the true α, but the estimates are still far off.

    “The BERT estimates are correlated with the true α, but the estimates are still far off.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Nature portfolio reviews in the dataset comprise an average of 37.03 sentences each.

    “Each review in the Nature dataset comprises an average of 37.03 sentences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Performance degrades slightly when verbs are used compared to using adjectives.

    “The performance degrades slightly as compared to using adjectives.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    A similar trend of shifted adjective frequency is observed in NeurIPS but not in Nature Portfolio journals.

    “We find a similar trend in NeurIPS but not in Nature Portfolio journals.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The method is robust to changes in the prompt and still exhibits accurate and stable performance.

    “The method is robust to changes in the prompt and still exhibits accurate and stable performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Performance degrades when nouns are used compared to using adjectives.

    “The performance degrades as compared to using adjectives.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Machine learning conference reviews in the dataset contain an average of 25.94 sentences each.

    “Each review contains an average of 25.94 sentences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Performance degrades when adverbs are used compared to using adjectives.

    “The performance degrades compared to using adjectives.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Factors such as field-specific writing styles and the prevalence of AI use could influence the effectiveness of the paper's approach.

    “Factors such as field-specific writing styles and the prevalence of AI use could influence the effectiveness of our approach.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Study design and conductown data

    The paper's method achieves high accuracy compared to BERT-based and other off-the-shelf classifiers.

    “While BERT's estimates for α seem at least positively correlated with the correct α value, the error in the estimate is still large compared to the high accuracy obtained by our method (see Figure 3 and Table 5).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    ChatGPT dominates the generative AI market, with 76% of global internet traffic in the category; Bing AI follows with 16%, Bard with 7%, and Claude with 1%.

    “According to a comprehensive analysis by FlexOS in early 2024, ChatGPT dominates the generative AI market, with 76% of global internet traffic in the category.Bing AI follows with 16%, Bard with 7%, and Claude with 1% (Van Rossum, 2024).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Van Rossum, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ChatGPT-generated medical abstracts may frequently bypass AI-detectors and experts.

    “In scientific research, for example, studies have found that ChatGPT-generated medical abstracts may frequently bypass AI-detectors and experts (Else, 2023;Gao et al., 2022).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Else, 2023 source
    • cites Gao et al., 2022 source

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The poor performance of off-the-shelf classifiers may be due to a distribution shift between the data used to train the classifier (likely general text scraped from the internet) and text found in conference reviews.

    “This may be due to a distribution shift between the data used to train the classifier (likely general text scraped from the internet) vs. text found in conference reviews.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ChatGPT dominates the generative AI market with 76% of global internet traffic in the category.

    “While our study focused on ChatGPT, which dominates the generative AI market with 76% of global internet traffic in the category (Van Rossum, 2024), we acknowledge that there are other diverse LLMs used for generating or rephrasing text.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Van Rossum, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Further experimentation may be required to fully establish the framework's generalizability to an even wider range of fields and publication venues.

    “While this demonstrates the applicability of our framework beyond these domains, further experimentation may be required to fully establish its generalizability to an even wider range of fields and publication venues.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Study design and conductown data

    The paper's framework is applicable beyond major ML conferences and Nature Family Journals, spanning 15 distinct journals across medicine, biology, chemistry, and environmental sciences.

    “Our findings are primarily based on datasets from major ML conferences (ICLR, NeurIPS, CoRL, EMNLP) and Nature Family Journals spanning 15 distinct journals across different disciplines such as medicine, biology, chemistry, and environmental sciences.While this demonstrates the applicability of our framework beyond these domains, further experimentation may be required to fully establish its gener”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The model trained with only GPT-3.5 data provides consistent estimation results and findings, and demonstrates the ability to generalize, accurately detecting GPT-4 as well.

    “For example, the model trained with only GPT-3.5 data provides consistent estimation results and findings, and demonstrates the ability to generalize, accurately detecting GPT-4 as well (see Supp.Table 28 and 29).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The document-level method tends to slightly over-estimate the true α and shows a slight degradation in performance compared to the sentence-level approach.

    “There is a slight degradation in performance compared to the sentence-level approach, and the method tends to slightly over-estimate the true α.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The prompting techniques used to simulate revising, expanding, paraphrasing, and proofreading review texts may not capture the full range of techniques used by reviewers or AI assistants.

    “The prompts we employed were designed based on our understanding of common practices, but they may not capture the full range of techniques used by reviewers or AI assistants.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The paper's framework's effectiveness may vary depending on the specific LLM used, and future practitioners should select the LLM that most closely mirrors the language model likely used to generate their target corpus.

    “However, we acknowledge that our framework's effectiveness may vary depending on the specific LLM used, and future practitioners should select the LLM that most closely mirrors the language model likely used to generate their target corpus, reflecting actual usage patterns at the time of creation.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Study design and conductown data

    The paper's method is robust to the design choice of analyzing at the document level rather than the sentence level, though with slight degradation and over-estimation.

    “We can perform the same analysis on entire documents (i.e., complete reviews) to check the robustness of our method to this design choice.”
    • Also stated in: § D.8. Results on Document-Level Analysis · plain
      “The results of the document-level analysis are similar to that at the sentence level.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The framework operates at the population level and only outputs aggregate statistics, without performing inference on individual reviews.

    “A key strength of our proposed framework is that it operates at the population level and only outputs aggregate statistics, without the need to perform inference on individual reviews.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Study design and conductown data

    The paper's method is robust to the choice of using adjectives versus adverbs, verbs, or nouns, with adjectives exhibiting greater stability.

    “For our results in the main paper, we only considered adjectives for the space of all possible tokens.We found this vocabulary choice to exhibit greater stability than using other parts of speech such as adverbs, verbs, nouns, or all possible tokens.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Substantial shifts in the non-native speaker population over time could still impact the accuracy of the paper's estimates.

    “we recognize that substantial shifts in the non-native speaker population over time could still impact the accuracy of our estimates (Liang et al., 2023a).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Liang et al., 2023a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The prompting techniques should be interpreted as a best-effort approximation rather than a definitive representation of how AI is used for review text modifications.

    “We emphasize that these techniques should be interpreted as a best-effort approximation rather than a definitive representation of how AI is used for review text modifications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Temporal distribution shift in token frequencies due to changes in topics, reviewers, etc., introduces an additional source of error.

    “as does the temporal distribution shift in token frequencies due to, e.g., changes in topics, reviewers, etc.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The approximations made to the review generating process to make estimation of the review likelihood tractable introduce an additional source of error.

    “In addition, the approximations made to the review generating process in Section § 3 in order to make estimation of the review likelihood tractable introduce an additional source of error, as does the temporal distribution shift in token frequencies due to, e.g., changes in topics, reviewers, etc.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    All the data used in this study are publicly available.

    “All the data are publicly available.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Authors and peer reviewers agree to make their reviews public upon submission.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The paper does not wish to pass a value judgement or claim that the use of AI tools for review papers is necessarily bad or good.

    “We emphasize here that we do not wish to pass a value judgement or claim that the use of AI tools for review papers is necessarily bad or good.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    It is possible that a reviewer may sketch out several bullet points related to the paper and use ChatGPT to formulate these bullet points into paragraphs, and in this case the estimated α can be high.

    “For example, it is possible that a reviewer may sketch out several bullet points related to the paper and uses ChatGPT to formulate these bullet points into paragraphs.In this case, it is possible for the estimated α to be high; indeed our results in Appendix 4.6 is consistent with this mode of using LLM to substantially modify and flesh out reviews.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    The paper does not claim (nor believe) that many reviewers are using ChatGPT to write entire reviews outright.

    “We also do not claim (nor do we believe) that many reviewers are using ChatGPT to write entire reviews outright.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Operating at the population level and outputting only aggregate statistics helps protect reviewer anonymity and mitigates de-anonymization risk based on writing style.

    “This helps protect the anonymity of reviewers and mitigates the risk of de-anonymization based on writing style, which is an important consideration when working with peer review data.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The paper's method is robust to the use of alternative LLMs such as GPT-3.5, with the model trained on GPT-3.5 data providing consistent estimation results and accurately detecting GPT-4.

    “For example, the model trained with only GPT-3.5 data provides consistent estimation results and findings, and demonstrates the ability to generalize, accurately detecting GPT-4 as well (see Supp.Table 28 and 29).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationnone given

    The paper's data and analyses can serve as a foundation for constructive discussions and further research by the community, ultimately contributing to the development of robust guidelines and best practices for the ethical use of generative AI.

    “We believe that our data and analyses can serve as a foundation for constructive discussions and further research by the community, ultimately contributing to the development of robust guidelines and best practices for the ethical use of generative AI.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductcitation

    The paper's validation experiments used real reviews from prior years, which included a significant fraction of non-native speaker-written texts, and the results remained accurate.

    “Although our validation experiments used real reviews from prior years, which included a significant fraction of nonnative speaker-written texts, and our results remained accurate, we recognize that substantial shifts in the non-native speaker population over time could still impact the accuracy of our estimates (Liang et al., 2023a).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Liang et al., 2023a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The authors accessed publicly available, anonymous peer review data through the public OpenReview API and complied with OpenReview's terms of use.

    “In our work, we accessed this publicly available, anonymous peer review data through the public OpenReview API and confirm that we have complied with their terms of use.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Authors and peer reviewers agree to make their reviews public upon submission.
57 of 57

  • The authors state this as an assumption.

    Documents in the target corpus are generated from the mixture distribution (1-α)P + αQ, where P is the distribution of human-written documents and Q is the distribution of AI-generated documents.

    “We assume that the documents in the target corpus are generated from the mixture distribution (1 -α)P + αQ”

  • The authors state this as an assumption.

    We have access to a collection of reviews known to contain only human-authored text, along with the associated review questions and the reviewed papers.

    “Specifically, we assume that we have access to a collection of reviews which are known to contain only human-authored text, along with the associated review questions and the reviewed papers.”

  • The authors state this as an assumption.

    Each document is represented as a set of token occurrences rather than a list of token counts.

    “We represent each document x i as a list of occurrences (i.e., a set) of tokens rather than a list of token counts.”

  • The authors state this as an assumption.

    Documents in the target corpus are drawn independently from the mixture distribution.

    “Given a collection of n documents {x i } n i=1 drawn independently from the mixture (1)”

  • The authors state this as an assumption.

    The likelihood of a document can be estimated as the product of token occurrence probabilities (and their complements) under the human and AI distributions.

    “Using the notation t ∈ x to denote that token t occurs in document x, we can then estimate P via P (x i ) = t∈x p(t) × t̸ ∈x (1 -p(t))”

  • The paper relies on this without support.

    Token occurrences within a document are independent.

    “P (x i ) = t∈x p(t) × t̸ ∈x (1 -p(t))”

  • The paper relies on this without support.

    The historical documents used to build the human distribution P are representative of the human-written texts in the target corpus.

    “The method makes use of historical documents.The prompts from this historical corpus are then fed into an LLM (or LLMs) to produce a corresponding corpus of AI-generated texts.”

  • The paper relies on this without support.

    The LLM-generated corpus Q is representative of the AI-generated texts actually present in the target corpus.

    “The prompts from this historical corpus are then fed into an LLM (or LLMs) to produce a corresponding corpus of AI-generated texts.The written and AI-generated corpora are then used to estimate the distributions of AIgenerated vs. written texts in a mixed corpus.”

  • The authors state this as an assumption.

    Each sentence in a review is drawn from either the human distribution P or the AI-generated distribution Q, rather than from a mixture or hybrid process.

    “Our results in the main paper analyzed the data at a sentence level.That is, we assumed that each sentence in a review was drawn from the mixture model (1), and estimated the fraction α of sentences which were AI generated.”

  • The authors state this as an assumption.

    The estimation of the human and AI distributions P and Q is perfect for the theoretical sample-size analysis.

    “Furthermore, suppose we have n papers from the mixture, and the estimation of P and Q is perfect.”

  • The authors state this as an assumption.

    The occurrence of the string “et al.” is a valid proxy for scholarly citations in reviews.

    “To test this, we use the occurrence of the string “et al.” as a proxy for scholarly citations in reviews.”

  • The paper relies on this without support.

    The OpenAI text embeddings and centroid-based classification of reviews as convergent or divergent capture meaningful homogenization of feedback.

    “For each paper, we obtained the OpenAI's textembeddings for all reviews, followed by the calculation of their centroid (average).Among the assigned reviews, the one with its embedding closest to the centroid is labeled as convergent, and the one farthest as divergent.”

  • The authors state this as an assumption.

    The prompts used to simulate AI-assisted review modification are a reasonable approximation of how reviewers actually use LLMs.

    “The prompts we employed were designed based on our understanding of common practices, but they may not capture the full range of techniques used by reviewers or AI assistants.”

  • The paper relies on this without support.

    The temporal distribution of token frequencies is stable enough that historical documents can be used to estimate current human-written text distributions.

    “as does the temporal distribution shift in token frequencies due to, e.g., changes in topics, reviewers, etc.”

  • The paper relies on this without support.

    The non-native speaker population in the validation data is representative of the target corpus's non-native speaker population.

    “Although our validation experiments used real reviews from prior years, which included a significant fraction of nonnative speaker-written texts, and our results remained accurate, we recognize that substantial shifts in the non-native speaker population over time could still impact the accuracy of our estimates (Liang et al., 2023a).”

  • The paper relies on this without support.

    The mixture model's likelihood approximation is sufficiently accurate for estimating α.

    “In addition, the approximations made to the review generating process in Section § 3 in order to make estimation of the review likelihood tractable introduce an additional source of error, as does the temporal distribution shift in token frequencies due to, e.g., changes in topics, reviewers, etc.”

  • The paper relies on this without support.

    The GPT-4 snapshot from June 13th, 2023 (gpt-4-0613) was the exact version of ChatGPT available during the peer review process of ICLR 2024, NeurIPS 2023, EMNLP 2023, and CoRL 2023.

    “We used the snapshot of GPT-4 from June 13th, 2023 (gpt-4-0613), for our experiments because this is the exact version of ChatGPT that was available during the peer review process of ICLR 2024, NeurIPS 2023, EMNLP 2023, and CoRL 2023.”

  • The paper relies on this without support.

    Authors and peer reviewers agree to make their reviews public upon submission.

    “Authors and peer reviewers agree to make their reviews public upon submission.”

  • The paper relies on this without support.

    The Nature family journals have a reputation for publishing high-quality, impactful research across multiple disciplines.

    “We chose to focus on the Nature family journals for our baseline dataset due to their reputation for publishing high-quality, impactful research across multiple disciplines.”
19 of 19

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • This finding complements and extends previous research that found widespread adoption across academic researchers.

    “This finding complements and extends our previous research that found widespread adoption across academic researchers.12”

    From The widespread adoption of large language model-assisted writing across society (2025) · cites this paper as [12]

    Section 4.4 estimates that 6.5%–16.9% of review text at several ML conferences may have been substantially modified by LLMs, supporting use in that academic setting. It reports no significant increase for Nature Portfolio reviews and does not measure adoption across academic researchers generally, so the claim is broader than the finding. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • As validated in a prior paper, the distributional LLM quantification framework has demonstrated robustness, transparency, and low cost, allowing tracking of adoption trajectories and key demographic and organizational factors driving LLM integration.

    “As validated in a prior paper, the framework has demonstrated robustness, transparency, and low cost,12,37,40 allowing us to track adoption trajectories and key demographic and organizational factors driving LLM integration.”

    From The widespread adoption of large language model-assisted writing across society (2025) · cites this paper as [12]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • A broader analysis of conference peer-review data reveals that the estimated fraction of LLM-generated reviews spikes as deadlines approach.

    “A broader analysis of conference peer-review data reveals that the estimated fraction of LLM-generated reviews spikes as deadlines approach 4”

    From Fine-Grained Detection of AI-Generated Writing in the Biomedical Literature (2026) · cites this paper as [4]

    Section 4.7 (Figure 7) reports a deadline effect in AI conference reviews, but describes it as a “small but consistent increase” for reviews submitted within three days of the deadline. “Spikes” overstates the reported magnitude. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Large-scale comparisons have shown that LLM-generated comments can overlap substantially with human reviewer feedback in both journal and ICLR papers, and that many researchers perceive such feedback as helpful.

    “Large-scale comparisons have shown that LLM-generated comments can overlap substantially with human reviewer feedback in both journal and ICLR papers, and that many researchers perceive such feedback as helpful (Liang, Zhang, et al., 2024).”

    From Beyond Human-Likeness: Mapping the Scientific Critique Profiles of LLMs and Human Reviewers (2026) · cites this paper as Liang, Zhang, et al., 2024

    The Homogenization Effect analysis (Figure 10; page unavailable) finds that reviews more similar to other reviews of the same paper tend to have a higher estimated proportion of LLM-generated text, offering limited support for overlap in conference reviews. It does not establish substantial overlap between LLM-generated and human feedback in both journal and ICLR papers, or report that researchers found the feedback helpful. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • An estimated 10.6% of reviewers at ICLR 2024 used LLMs to assist in drafting their reviews.

    “with an estimated 10.6% of reviewers at ICLR 2024 using LLMs for this purpose [15, 16].”

    From Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 (2025) · cites this paper as [16]

    Section 4.4 reports an estimated α of 10.6% for ICLR after ChatGPT’s release. But it defines α as the fraction of review text substantially modified by LLMs, not the fraction of reviewers who used them; the Abstract makes the same distinction. (LLM, conf=0.92)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • The ICLR 2024 estimate changes in the proofreading figure

    Minor

    Figure 5 labels the real-review ICLR 2024 estimate as 10.8%, while the main results table reports 10.6% for that conference and year.

  • NeurIPS validation error exceeds its stated bound

    Minor

    The text says the NeurIPS error remains below 1.8%, but Table 5 reports an error of 1.9% for one mixture.

  • An absolute value is bounded by a negative number

    Minor

    The proof asserts that |L′′(α)| is at most −κ, even though κ is explicitly positive. An absolute value cannot satisfy that inequality.

3 of 3

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 6

  1. [4]4. Badaskar, S., Agarwal, S., and Arora, S. Identifying Real or Fake Articles: Towards better Language Mod- eling. In International Joint Conference on Natural Language Processing, 2008. URL https://api. semanticscholar.org/CorpusID:4324753.
  2. [17]17. Christin, A. What data can do: A typology of mechanisms. International Journal of Communication, 14:20, 2020.
  3. [43]43. Lavergne, T., Urvoy, T., and Yvon, F. Detect- ing Fake Content with Relative Entropy Scoring. 2008. URL https://api.semanticscholar. org/CorpusID:12098535.
  4. [47]47. Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. arXiv preprint arXiv:2310.01783, 2023c. · arXiv
  5. [57]57. Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., and Finn, C. DetectGPT: Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305, 2023b. · arXiv
  6. [68]68. Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., and Wang, J. Release Strategies and the Social Impacts of Language Models. ArXiv, abs/1908.09203, 2019b. URL https://api.semanticscholar. org/CorpusID:201666234. · arXiv

Cited by

Sorted by citations

Related

Authors
Liang, W., Izzo, Z., Zhang, Y., Lepp, H., Cao, H., Zhao, X., Chen, L., Ye, H., Liu, S., Huang, Z., McFarland, D. A., & Zou, J. Y.
Affiliations
  • Stanford University (US)
  • NEC (United States) (US)
  • University of California, Santa Barbara (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2403.07183
arXiv
2403.07183
OpenAlex
W4392781833
Cited by
73
References
0
Citations by year
2026: 8 · 2025: 36 · 2024: 29
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (2244804, 1942926, CAREER, 2022435)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
AI-generated text detection, large language models, ChatGPT, peer review, maximum likelihood estimation, scholarly publishing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review