The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index; 9 of the 22 sources its claims cite were checked for support.
Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review
Open accessPublished version of a preprint
Abstract
BACKGROUND: The emergence of systems based on large language models (LLMs) such as OpenAI's ChatGPT has created a range of discussions in scholarly circles. Since LLMs generate grammatically correct and mostly relevant (yet sometimes outright wrong, irrelevant or biased) outputs in response to provided prompts, using them in various writing tasks including writing peer review reports could result in improved productivity. Given the significance of peer reviews in the existing scholarly publication landscape, exploring challenges and opportunities of using LLMs in peer review seems urgent. After the generation of the first scholarly outputs with LLMs, we anticipate that peer review reports too would be generated with the help of these systems. However, there are currently no guidelines on how these systems should be used in review tasks. METHODS: To investigate the potential impact of using LLMs on the peer review process, we used five core themes within discussions about peer review suggested by Tennant and Ross-Hellauer. These include 1) reviewers' role, 2) editors' role, 3) functions and quality of peer reviews, 4) reproducibility, and 5) the social and epistemic functions of peer reviews. We provide a small-scale exploration of ChatGPT's performance regarding identified issues. RESULTS: LLMs have the potential to substantially alter the role of both peer reviewers and editors. Through supporting both actors in efficiently writing constructive reports or decision letters, LLMs can facilitate higher quality review and address issues of review shortage. However, the fundamental opacity of LLMs' training data, inner workings, data handling, and development processes raise concerns about potential biases, confidentiality and the reproducibility of review reports. Additionally, as editorial work has a prominent function in defining and shaping epistemic communities, as well as negotiating normative frameworks within such communities, partly outsourcing this work to LLMs might have unforeseen consequences for social and epistemic relations within academia. Regarding performance, we identified major enhancements in a short period and expect LLMs to continue developing. CONCLUSIONS: We believe that LLMs are likely to have a profound impact on academia and scholarly communication. While potentially beneficial to the scholarly communication system, many uncertainties remain and their use is not without risks. In particular, concerns about the amplification of existing biases and inequalities in access to appropriate infrastructure warrant further attention. For the moment, we recommend that if LLMs are used to write scholarly reviews and decision letters, reviewers and editors should disclose their use and accept full responsibility for data security and confidentiality, and their reports' accuracy, tone, reasoning and originality.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper argues that LLMs such as ChatGPT can support peer review by helping reviewers and editors write reports and decision letters and by addressing reviewer shortage, but they raise serious risks around bias, confidentiality, reproducibility, and social/epistemic impacts. It recommends disclosure, human responsibility, training, and caution pending further evidence.
Key findings
- Using five core themes from Tennant and Ross-Hellauer, the authors argue LLMs can support reviewers’ quality-improvement and gatekeeping roles and editors’ tasks such as reviewer search, initial screening, and writing decision letters, potentially expanding reviewer pools and reducing reviewer fatigue.
- Concerns include opacity of LLM training data, inner workings, and data handling; possible bias amplification related to geography, race, or class; and threats to confidentiality and reproducibility. As an example, ChatGPT generated a cynical review targeting China in December 2022, while a January 2023 response to a similar prompt was different and mentioned limitations including a “small sample size (only 214 patients).”
- Accessibility inequalities are noted: ChatGPT was unavailable in countries such as Iran, China, Russia, Venezuela, and Ukraine, and ChatGPT PLUS was released at $20/month, with future business models potentially worsening access inequalities.
- Recommendations include disclosing LLM use, accepting full responsibility for reports’ accuracy, tone, reasoning, and originality, avoiding sensitive/protected data unless safeguards exist, verifying outputs, training researchers, and developing journal/platform policies.
Methods
- Conceptual/commentary essay using five core themes within discussions about peer review: reviewers’ role, editors’ role, functions and quality of peer reviews, reproducibility, and social and epistemic functions.
- Small-scale exploration of ChatGPT’s performance through authors’ prompted interactions, including asking for a review of a COVID-19 preprint; supplementary material includes interactions with ChatGPT, ChatGPT PLUS, and Google’s BARD.
- Main analysis is qualitative discussion of opportunities, risks, and recommendations, not a systematic empirical study.
Limitations
- Authors state that the essay only discussed review of journal articles and not other object types like grants, used examples from ChatGPT, and was constrained by limitations of the used framework.
- Authors note that rapid LLM development challenged some assertions during review, including the launch of GPT-4 as a successor to GPT-3.5 used in their examples.
- Evident from the text: there is no systematic sample, coding, or quantitative evaluation; examples are anecdotal, and the authors report that identical queries three weeks apart and minor prompt variations yielded dissimilar responses, making reproducibility fragile.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
54 Own interpretation · 7 Supported · 4 No full text of the sources · 3 Sources not found · 3 Not checked yet · 3 Own result · 3 No basis given
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, such automated reviewer selection mechanisms should be implemented with caution as sub-optimal implementation can lead to undesirable consequences [16].”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of editors in peer review
What the sources say
- Supportedcites [16] Horbach S, Ochsner, Michael, Kaltenbrunner W. Reflections on guest editing a Frontiers journal. LeidenMadtrics. 2022 [cited 2023 Jan 31]. Available from: https://www.leidenmadtrics.nl/articles/reflections-on-guest-editing-a-frontiers-journal source
In the discussion of peer-review organization (page not provided), the authors describe imperfect algorithmic reviewer matches and quasi-automated invitations. They report that this process sometimes assigned reviewers they considered unqualified and could erode academics’ willingness to review, supporting the caution in the citing sentence. (LLM, conf=0.93)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Currently, AI tools are being developed to support automated detection of problematic research, despite diverse reactions from editorial staff regarding the desirability and usefulness of such tools [20].”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
What the sources say
- Supportedcites [20] Hesselmann F. A tale of two academic communities: digital imaginaries of automatic screening tools in editorial practice. Minerva. 2023 Jan 11 [cited 2023 Feb 14]; Available from: https://doi.org/10.1007/s11024-022-09484-7 source
The Introduction describes algorithms that screen papers for research-integrity problems, and “Automatic Screening Tools in Publishers’ Policies” explains their role in flagging errors before publication. “Automatic Screening Tools in Editors’ Accounts” reports differing views among surveyed editors about wider use. The paper also quotes a publishing-ethics document referring to AI and automation tools being developed. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“When prompted with sufficiently specific requests, ChatGPT provides meaningful feedback related to specific parts of a manuscript.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
What the sources say
- Not matchedcites [6] OpenAI ChatGPT. 2023. Response to Query Made by Mohammad Hosseini, February 13, 2023, 10:31am CST.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, OpenAI’s Classifier (released in January 2023) does not always succeed in identifying text that is generated by OpenAI’s ChatGPT (upon using as input ChatGPT generated text in December 2022 mentioned in Sect. 1, the classifier notes “unclear if it is AI-generated”, see the supplementary document).”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For example, running an identical query with ChatGPT three weeks apart, resulted in dissimilar conversations.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments with these systems show that small variations in prompts result in significantly different responses, and sometimes, repeating the same prompt to a system yields randomly dissimilar responses.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments with these systems show that small variations in prompts result in significantly different responses, and sometimes, repeating the same prompt to a system yields randomly dissimilar responses.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“address issues of review shortage”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Abstract > Results
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In case of increased use of LLMs in scholarly reviews, some biases such as those related to geography, race or class might be reinforced.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This capability could enable researchers (native and non-native speakers of supported languages) to write more reviews in a short time.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 56
- It may state two separate claims as one. 21
Every source the paper cites for this claim was checked and supports it.
In the paper
“a lack of incentives for reviewers has been introduced as a hurdle to rigorous testing or validation of studies under review [5]”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
What the sources say
- Supportedcites [5] Tennant JP, Ross-Hellauer T. The limitations to our understanding of peer review. Res Integr Peer Rev. 2020;5(1):6. source
The ‘Role of reviewers in peer review’ section explicitly describes a general lack of rewards or incentives that could motivate reviewers to undertake rigorous testing or replications. This supports the citing sentence’s characterization of that lack as a hurdle. (LLM, conf=0.94)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Partly, this is already done [3, 4] but future LLMs could enhance these applications.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of editors in peer review
What the sources say
- Supportedcites [3] Schulz R, Barnett A, Bernard R, Brown NJL, Byrne JA, Eckmann P, et al. Is the future of peer review automated? BMC Res Notes. 2022;15(1):203. source
The Introduction says automated screening in academic publishing is not new and that publishers have used plagiarism-detection tools for more than a decade. It also reports that some publishers are experimenting with automated checks for statistical reporting errors and other compliance-related items. The citation is attached to the claim that this is already being done, not to the authors’ suggestion about future LLMs. (LLM, conf=0.93)
- Supportedcites [4] Weissgerber T, Riedel N, Kilicoglu H, Labbé C, Eckmann P, ter Riet G, et al. Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility? Nat Med. 2021;27(1):6–7. source
The paper’s main text reports that six automated tools screened new medRxiv and bioRxiv COVID-19 preprints daily and flagged common reporting problems. This supports the citation’s claim that automated screening and compliance checks are already being done; the statement about future LLMs is the citing authors’ projection, not a finding attributed to this source. (LLM, conf=0.92)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The feasibility and usefulness of this approach has already been demonstrated in academic contexts [7].”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
What the sources say
- Supportedcites [7] Pividori M, Greene CS. A publishing infrastructure for AI-assisted academic authoring. bioRxiv; 2023 [cited 2023 Jan 31]. p. 2023.01.21.525030. Available from: https://www.biorxiv.org/content/10.1101/2023.01.21.525030v1 source
The Introduction describes an LLM-assisted manuscript-revision workflow tested on three scholarly manuscripts and reports improvements in writing style. The Conclusions say the workflow was implemented and that many paragraphs were improved, while noting that the evaluation was subjective. This supports the claim of demonstrated feasibility and usefulness in academic writing. (LLM, conf=0.91)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Currently, several journals have attempted to develop guidelines [28, 29].”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Social and epistemic impacts of peer review
What the sources say
- Supportedcites [28] Hosseini M, Rasmussen LM, Resnik DB. Using AI to write scholarly publications. Account Res. 2023;0(0):1–9. source
The Abstract states that many biomedical journals have developed guidelines for using computer programs to process digital images. It also describes Accountability in Research’s draft policy for AI-generated text. These passages support the cited sentence’s general statement that several journals have attempted to develop guidelines; the source does not say that several journals had developed AI-writing guidelines specifically. (LLM, conf=0.87)
- Supportedcites [29] Thorp HH. ChatGPT is fun, but not an author. Science. 2023;379(6630):313–313. source
The passage on Science editorial policies says the Science family of journals was updating its license and Editorial Policies to address AI-generated text and images. That supports this citation as an example of journals developing guidelines. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Apart from contributing to the quality of manuscripts and filtering out poor or problematic science and improving “arguments and gaps in logic” in a collegial and constructive manner [22], peer review also has important social functions.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Social and epistemic impacts of peer review
What the sources say
- Supportedcites [22] Woods HB, Brumberg J, Kaltenbrunner W, Pinfield S, Waltman L. An overview of innovations in the external peer review of journal manuscripts. Wellcome Open Res. 2023;7:82. https://doi.org/10.12688/wellcomeopenres.17715.2. source
In Results > Details of included studies (page not provided), the review describes peer review’s collegial, constructive purpose as improving arguments and gaps in logic, alongside its gatekeeping function. Results > Description of innovations in peer review (page not provided) describes review against a research-quality threshold, supporting the claim’s filtering and quality-assessment points. The cited paper reports these functions in its synthesis of earlier reviews. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Currently, ChatGPT does not seem very capable of performing this task”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of editors in peer review
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, we note that the system is very sensitive to minor differences in prompts.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the introduction of novel LLMs has the potential to move automation much further”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Background
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We believe that LLMs are likely to have a profound impact on academia and scholarly communication.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Abstract > Conclusions
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When using LLMs to review unpublished manuscripts, however, the LLM may inadvertently incorporate data or ideas to its training data or generate responses that may disclose authors' identity (in closed peer-review processes).”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“freeing up reviewers’ time can improve productivity”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The quality of reviewed work could be improved when more feedback is delivered in a timely fashion”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Since LLMs can support reviewers to write better reviews and submit their report more quickly, editors would likely have access to a larger and potentially more diverse and efficient pool of candidate reviewers.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of editors in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“It is possible that LLMs could help researchers conduct repetitive or tedious writing tasks more efficiently (e.g., editing manuscripts to minimize the use of jargon, correcting grammar and spelling mistakes, or changing the voice from passive to active) [8] thereby improving the likelihood to have more time for testing or assessing studies more rigorously.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
What the sources say
- Not checked — no full text of the sourcecites [8] Stokel-Walker C, Van Noorden R. What ChatGPT and generative AI mean for science. Nature. 2023;614(7947):214–6. source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLMs have the potential to substantially alter the role of both peer reviewers and editors.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Abstract > Results
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but because of certain shortcomings of AI, we know that it can provide biased views”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLMs can facilitate higher quality review”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Abstract > Results
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even if a basic version would remain freely available, it is possible that more sophisticated versions with better functionality would become available to researchers/universities who can/will afford it.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of editors in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In essence, these systems are necessarily conservative, favoring the status-quo and potentially skewed approaches already present in current and past discourse.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, while ChatGPT is currently freely available, it is unclear what business model will be chosen by its future investors, thereby introducing further accessibility inequalities”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of editors in peer review
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, LLMs could be used in initial screening of manuscripts, for instance to assess fit with journal scope or general quality.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of editors in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As this stage arguably involves little original contribution from the side of the editor, it is an obvious part of the editorial process that LLMs, even in their current state, can already contribute to.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of editors in peer review
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Rests on an assumption: The final stage of editorial work is a core task of editors and potentially takes up a significant amount of their time.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLMs could either automatically perform triage (e.g., initial quality checks to filter or flag problematic research), or support editorial staff to perform such inspections more efficiently.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of editors in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Through supporting both actors in efficiently writing constructive reports or decision letters”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Abstract > Results
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“partly outsourcing this work to LLMs might have unforeseen consequences for social and epistemic relations within academia”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Abstract > Results
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Editorial work has a prominent function in defining and shaping epistemic communities and negotiating normative frameworks within such communities.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Since LLMs generate grammatically correct and mostly relevant (yet sometimes outright wrong, irrelevant or biased) outputs in response to provided prompts, using them in various writing tasks including writing peer review reports could result in improved productivity.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Abstract > Background
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
- Rests on an assumption: LLMs generate grammatically correct and mostly relevant, but sometimes wrong, irrelevant or biased, outputs in response to provided prompts.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As LLMs are trained on existing data by (mostly) biased humans, it is unclear whether or how these systems can mitigate existing biases, with the risk of in fact reproducing or amplifying them unless human agents are educated and aware of possible ways of addressing these biases.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: LLMs are trained on existing data by mostly biased humans.
In the paper
“Editors struggle to find sufficiently qualified reviewers and maintain reasonable turnaround times for their journals [14].”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of editors in peer review
What the sources say
- Not checked — no full text of the sourcecites [14] Dance A. Stop the peer-review treadmill. I want to get off. Nature. 2023;614(7948):581–3. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, LLMs could assist editors in writing final decision letters and summarizing individual review reports.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of editors in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Since it is possible that LLMs may store inputted text, using them to review manuscripts that contain sensitive data could raise concerns about data privacy and unauthorized sharing of confidential information.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“It might be true that systems such as ChatGPT seem to have been trained well in avoiding judgmental views and, for the most part, do a good job in refraining from using abusive, racist or homophobic language [9]”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
What the sources say
- Not checked yetcites [9] Gault M. Conservatives Are Panicking About AI Bias, Think ChatGPT Has Gone “Woke”. Vice. 2023 [cited 2023 Feb 14]. Available from: https://www.vice.com/en/article/93a4qe/conservatives-panicking-about-ai-bias-years-too-late-think-chatgpt-has-gone-woke
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another advantage of using LLMs pertains to their ability to adopt unique styles or constructs (e.g., literary, philosophical, temporal, etc.). When writing reviews this could provide a chance to write reasonably consistent (not to be mistaken with accurate, useful or relevant) yet context-tailored reviews.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We believe LLMs could contribute to editors’ tasks in peer review by supporting the search for suitable reviewers, the initial screening of manuscripts, and the write-up of final decision letters from individual review reports.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of editors in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“For example, in terms of usefulness, given the significance of providing a solution (on how to resolve highlighted problems) in peer review reports [18, 19], and the fact that human reviewers might not always be motivated to do this, LLMs could complement human skills to improve the usefulness of review reports.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Functionality and quality of peer review
What the sources say
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
- Rests on an assumption: Human reviewers might not always be motivated to provide solutions in peer review reports.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even in preprint servers where there are practically no editors, LLMs could enhance automated reviews to address the concern commonly voiced regarding preprints, i.e., that such unreviewed papers may disseminate substandard quality research or unvetted knowledge.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of editors in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the fundamental opacity of LLMs’ training data, inner workings, data handling, and development processes raise concerns about potential biases, confidentiality and the reproducibility of review reports.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Abstract > Results
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, continuous improvements in LLMs in responding to prompts and avoiding the use of pejorative or disrespectful words could help make review reports more constructive and result in reduced outright harsh or jarring reviews.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Roles of reviewers in peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, unlike researchers who might only be fluent in a handful of languages, LLMs are likely to access sources of knowledge regardless of language.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Functionality and quality of peer review
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but in principle, LLMs can improve researchers’ capabilities to provide more constructive feedback”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Functionality and quality of peer review
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Collective publication outlets in general, and the peer review process in particular, are prime mechanisms that define and help shape epistemic communities [23, 24].”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Social and epistemic impacts of peer review
What the sources say
- Not checked yetcites [23] Cronin B. The hand of science: academic writing and its rewards. Lanham: Scarecrow Press; 2005.
- Not checked yetcites [24] Hinz A. Exploring the Human Dimension of Peer Review. De Gruyter Conversations. 2021 [cited 2023 Feb 14]. Available from: https://blog.degruyter.com/exploring-the-human-dimension-of-peer-review/
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Clearly, when a system is developing this fast, its results cannot be reproducible and the basis on which to responsibly rely on the system becomes fragile.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The peer review process is also a way to shape and negotiate normative frameworks within such communities, for example regarding what is to be considered ‘good’ science, what methods and questions are appropriate and relevant, and what means of communication are most suitable [25].”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Social and epistemic impacts of peer review
What the sources say
- Not checked — no full text of the sourcecites [25] Strang D, Siler K. From ‘just the facts’ to ‘more theory and methods, please’: The evolution of the research article in Administrative Science Quarterly, 1956–2008. Soc Stud Sci. 2017;47(4):528–55. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Partly due to the inherent nature of self-learning AI systems and partly due to the commercial interests of their developers, it is unclear how exactly the LLMs learn, how they have been trained and using what data.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We are likely at the very beginning of an era in which LLMs and future models will have a significant impact on many parts of society, including academia and scholarly communication.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Conclusion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This tension between keeping LLMs up to date and ensuring reproducibility is likely to confront metascience experts with major challenges.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Particularly when integrated in search engines, to the extent that they could find information about various involved actors, such tools are likely to be capable of detecting (undisclosed) conflicts of interest or related issues.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If used responsibly, such capabilities could improve the validity and rigor of reviews.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Functionality and quality of peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“LLMs could improve rigor and validity of peer-reviews because they can access and have the capacity to analyze a larger pool of previously published articles and review reports.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Functionality and quality of peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even though the quality of human-written review reports has often been critiqued [30] and calls for more training in peer review have been voiced [31], the introduction of LLMs might further exacerbate challenges around skills development and enhancement.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Social and epistemic impacts of peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Rests on an assumption: Writing good-quality and useful reviews is a skill that researchers acquire by practice.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is clear though, that they learn rapidly (although this feature too, could be a double-edged sword because they may learn impulses that humans have not had time to reflect on, or learn insights that are not deemed desirable or correct in hindsight).”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Due to this opacity, it is unclear how LLMs will develop, why they perform in the way they do and what they do with inputted texts in terms of storing, sharing or using these for training purposes.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Indeed, LLMs are still in early stages of their development and for the moment seem only suitable to improve the first draft of a review instead of writing a review from scratch.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Recommendations
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“one can imagine that a selective feeding of content, or not providing offered limitations or used references might result in biased or outright erroneous reviews.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Recommendations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“peer review is fundamentally built around the notion of the scientific ‘peer’ and it derives its legitimacy from this notion [26, 27].”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Social and epistemic impacts of peer review
What the sources say
- Not matchedcites [26] Biagioli M. From book censorship to academic peer review. Emerg J Study Media Compos Cult. 2002;12(1):11–45.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites [27] Shapin S. A social history of truth: civility and science in seventeenth-century England. 1st ed. Chicago: University of Chicago Press; 1995. p. 512.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand though, if these systems would not develop this quickly, their analysis might be out of date.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Given the recent exponential expansion of the corpus of scholarly publications and human limitations to read and analyze these in order to remain up to date, LLMs that are trained to be unbiased and neutral could significantly enhance researchers' capabilities to write better reviews.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Functionality and quality of peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, this example shows that while LLMs could develop fast, it is unclear why they developed in ways they did and how they will develop in the future.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Peer review and reproducibility
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, LLMs could pose threats to post-publication peer-review systems (such as PubPeer) that allow anonymous comments by enabling ill-intentioned actors to, for example, flood the scene and start smear campaigns against competitors.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Functionality and quality of peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the flipside, LLMs might exacerbate existing challenges of the peer review system such as fake peer reviews as they allow fraudsters to create more unique and well-written reviews.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Functionality and quality of peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using LLMs to write review reports, either partly or in full, could obviously impact such initiatives, necessitating strict regulations on the acknowledgement of the use of LLMs in review, similar to the use of LLMs in original articles.”
§ Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review > Social and epistemic impacts of peer review
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
LLMs generate grammatically correct and mostly relevant, but sometimes wrong, irrelevant or biased, outputs in response to provided prompts.
“Since LLMs generate grammatically correct and mostly relevant (yet sometimes outright wrong, irrelevant or biased) outputs in response to provided prompts”
The paper relies on this without support.
There are currently no guidelines on how LLMs should be used in review tasks.
“However, there are currently no guidelines on how these systems should be used in review tasks.”
The paper relies on this without support.
Reviewers are expected to provide neutral, unbiased and constructive feedback and recommendations to editors about publishability.
“Reviewers are expected to provide neutral, unbiased and constructive feedback when they agree to review a manuscript. In addition, they are usually expected to provide recommendations to the editor about the publishability of a paper.”
The paper relies on this without support.
When authors submit their work, they trust reviewers will not scoop or disclose their ideas.
“When authors submit their work for publication, they trust that reviewers will not scoop or disclose their ideas to others, thereby protecting their ideas and intellectual property.”
The paper relies on this without support.
Editorial work has a prominent function in defining and shaping epistemic communities and negotiating normative frameworks within such communities.
“as editorial work has a prominent function in defining and shaping epistemic communities, as well as negotiating normative frameworks within such communities”
The paper relies on this without support.
The final stage of editorial work is a core task of editors and potentially takes up a significant amount of their time.
“This final stage of editorial work, integrating gate-keeping and quality improvement functions of peer review, is a core task of editors and one that potentially takes up a significant amount of their time.”
The paper relies on this without support.
Human reviewers might not always be motivated to provide solutions in peer review reports.
“the fact that human reviewers might not always be motivated to do this”
The paper relies on this without support.
LLMs are trained on existing data by mostly biased humans.
“As LLMs are trained on existing data by (mostly) biased humans”
The paper relies on this without support.
ChatGPT is just one example of a host of LLMs and was not specifically designed or trained for peer review purposes.
“We are aware though, that ChatGPT is just one example of a host of LLMs and that it was not specifically designed or trained for peer review purposes.”
The paper relies on this without support.
Writing good-quality and useful reviews is a skill that researchers acquire by practice.
“In addition, writing good-quality and useful reviews is a skill that researchers acquire by practice.”
The authors state this as an assumption.
Peer reviews improve research quality or catch egregious mistakes.
“The assumption that peer reviews improve research quality or catch egregious mistakes, puts significant trust in the reproducibility and reliability of peer review reports.”
The paper relies on this without support.
ChatGPT (currently) does not accept inputs as long as an entire manuscript’s length.
“since it (currently) does not accept inputs as long as an entire manuscript’s length”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Around 2023, models began to process full manuscripts and produce credible end-to-end reviews (Robertson, 2023;Hosseini and Horbach, 2023).”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Hosseini and Horbach, 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the proliferation of AI-generated text, including that containing hallucinations, compels the scientific community to reevaluate ethical standards and peer-review processes”
From A Meta-Synthesis of Factual Accuracy and Citation Hallucination in LLM Academic Assistants (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This enables LLMs to generate suggestions based on both the manuscript's content and the reviewer's initial comments, thereby supporting the production of more insightful, comprehensive, and analytically robust reviews (Hosseini & Horbach, 2023).”
From Large language models for automated scholarly paper review: A survey (2025) · cites this paper as Hosseini & Horbach, 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since many LLMs are continuously fine-tuned or updated using user-provided inputs, there exists a risk that unpublished research materials submitted for scholarly review could become part of future training corpora, thereby compromising confidentiality (Hosseini & Horbach, 2023).”
From Large language models for automated scholarly paper review: A survey (2025) · cites this paper as Hosseini & Horbach, 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For instance, for many academic publishing venues, the number of submitted papers outpaces the size of the reviewing workforce, putting reviewer time and review quality in tension [144].”
From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [144]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Third, LLMs can play an important role in helping researchers who are not highly proficient in English (the lingua franca for most top journals) to write and edit their papers, or review others’ manuscripts (Hosseini and Horbach, 2023), which could promote equity in science (Berdejo-Espinola and Amano, 2023).”
From The ethics of disclosing the use of artificial intelligence tools in writing scholarly manuscripts (2023) · cites this paper as Hosseini and Horbach, 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although LLMs can make some glaring mistakes, are susceptible to bias, and may even fabricate facts or citations (Hosseini et al., 2023), these flaws should not be held against them because human researchers might make similar errors.”
From The ethics of disclosing the use of artificial intelligence tools in writing scholarly manuscripts (2023) · cites this paper as Hosseini et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While Accountability in Research, JAMA, and Nature decided to adopt or pursue policies that allow using LLMs under conditions that promote transparency, accountability, fair assignment of credit, and honesty, the editors of Science highlighted ethical problems created by LLMs and banned their use:”
From The ethics of disclosing the use of artificial intelligence tools in writing scholarly manuscripts (2023) · cites this paper as Hosseini et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since LLMs are constantly learning (or in the event of plugging them to the internet, receive new data), responses to the same question few days or weeks apart could be different as was shown recently(Hosseini and Horbach, 2023).”
From The ethics of disclosing the use of artificial intelligence tools in writing scholarly manuscripts (2023) · cites this paper as Hosseini and Horbach, 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We interpreted this disagreement as journals’ different stances on how to protect the confidentiality of manuscripts [8, 61].”
From Generative artificial intelligence (GAI) usage guidelines for scholarly publishing: a cross-sectional study of medical journals (2025) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“In addition to traditional editorial processes, LLMs can also be applied to preprint screening, which often lacks formal editorial oversight, to help prevent the dissemination of low-quality or misleading research (Hosseini & Horbach, 2023).”
From Large language models for automated scholarly paper review: A survey (2025) · cites this paper as Hosseini & Horbach, 2023
The cited source says what the claim attributes to it.
In the citing paper
“These biases, which can include prejudices related to factors such as social class, race, and geography, are inherent in LLMs and may be reproduced or even amplified during their use (Hosseini & Horbach, 2023).”
From Large language models for automated scholarly paper review: A survey (2025) · cites this paper as Hosseini & Horbach, 2023
The cited source says what the claim attributes to it.
In the citing paper
“These biases include a preference for positive outcomes, varying levels of tolerance or intolerance toward authors with specific demographic profiles, or from certain institutions or countries (Hosseini & Horbach, 2023).”
From Large language models for automated scholarly paper review: A survey (2025) · cites this paper as Hosseini & Horbach, 2023
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, LLMs can be useful in writing, reviewing and editing text, and promote equity in science.”
From The ethics of disclosing the use of artificial intelligence tools in writing scholarly manuscripts (2023) · cites this paper as Hosseini and Horbach, 2023
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“with an estimated 10.6% of reviewers at ICLR 2024 using LLMs for this purpose [15, 16].”
From Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 (2025) · cites this paper as [15]
Experimental — accuracy not yet measuredIn the citing paper
“Because commercial AI tools cannot guarantee the confidentiality of submitted data, most funding agencies (and many journals) place limitations on the use of AI by reviewers or ban it entirely [85].”
From Autonomous artificial intelligence, scientific research, and human values (2026) · cites this paper as [85]
In the citing paper
“Even in the case of generative AI models where some filtering has happened, changing the inputted prompt may simply confuse and push a system to generate biased content anyway [98].”
From The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool (2025) · cites this paper as [98]
In the citing paper
“given that content generated by AI systems is not always reproducible [98], identifying and adopting measures to reduce errors is extremely complicated.”
From The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool (2025) · cites this paper as [98]
Citations
Cites
ChatGPT is fun, but not an author
Thorp, H. H. (2023) · Science · doi:10.1126/science.adg7879 · Cited by 1,437 · cited 1×
“Currently, several journals have attempted to develop guidelines [28, 29].”
BackgroundGao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2022) · doi:10.1101/2022.12.23.521610 · Cited by 1,099
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099
Peer Review: A Flawed Process at the Heart of Science and Journals
Smith, R. (2006) · Journal of the Royal Society of Medicine · doi:10.1177/014107680609900414 · Cited by 880
What ChatGPT and generative AI mean for science
Stokel-Walker, C., & Van Noorden, R. (2023) · Nature · doi:10.1038/d41586-023-00340-6 · Cited by 828
The role of AI in drug discovery: challenges, opportunities, and strategies
Blanco-González, A., Cabezón, A., Seco-González, A., Conde-Torres, D., Antelo-Riveiro, P., Piñeiro, Á., & Garcia-Fandino, R. (2023) · Pharmaceuticals · doi:10.3390/ph16060891 · Cited by 690 · cited 1×
“For instance, some researchers used it to write a paper,[1] others used it to generate academic abstracts,[2] and many more have probably tried it to support them in diverse other tasks.”
BackgroundThe limitations to our understanding of peer review
Tennant, J. P., & Ross-Hellauer, T. (2020) · Research Integrity and Peer Review · doi:10.1186/s41073-020-00092-1 · Cited by 384
A social history of truth: civility and science in seventeenth-century England
(1994) · Choice Reviews Online · doi:10.5860/choice.32-2103 · Cited by 375 · cited 1×
“Whether as an individual or in a collaborative process (“the process where reviewers, editors and other contributors pool their comments to offer one set of consolidated recommendations for authors to address”) [22] peer review is fundamentally built around the notion of the scientific ‘peer’ and it derives its legiti…”
BackgroundUsing AI to write scholarly publications
Hosseini, M., Rasmussen, L. M., & Resnik, D. B. (2024) · Accountability in Research · doi:10.1080/08989621.2023.2168535 · Cited by 286 · cited 1×
“Currently, several journals have attempted to develop guidelines [28, 29].”
BackgroundSelf-citationThe hand of science: academic writing and its rewards
(2005) · Choice Reviews Online · doi:10.5860/choice.43-0269 · Cited by 206
Research works printed but not linked: 1
- Biagioli M. From book censorship to academic peer review. Emerg J Study Media Compos Cult. 2002;12(1):11–45.
Cited by
Sorted by citations
Items 101–200 of 269
Luo, X., Wang, B., Shi, Q., Wang, Z., Lai, H., Liu, H., Qin, Y., Chen, F., Song, X., Ge, L., et al. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111903 · Cited by 9
Geosystems risk and uncertainty: The application of ChatGPT with targeted prompting
Mahjour, S. K., Soltanmohammadi, R., Heidaryan, E., & Faroughi, S. A. (2024) · Geoenergy Science and Engineering · doi:10.1016/j.geoen.2024.212889 · Cited by 8
No reward without responsibility: Focus on peer review reports
da Silva, J. A. T., & Daly, T. P. (2024) · Ethics Medicine and Public Health · doi:10.1016/j.jemep.2024.101033 · Cited by 10
Kim, J., Koo, B., Nam, M., Jang, K., Lee, J., Chung, M., & Song, Y. B. (2025) · Applied Sciences · doi:10.3390/app15063355 · Cited by 9
ChatGPT in medical literature – a concise review and SWOT analysisPreprint
Gödde, D., Nöhl, S., Wolf, C., Rupert, Y., Rimkus, L., Ehlers, J. P., Breuckmann, F., & Sellmann, T. (2023) · medRxiv · doi:10.1101/2023.05.06.23289608 · Cited by 8
“Interestingly, however, a publication speed, if ascertainable, of 16 days (4-83) was significantly faster than described in other studies on biomedical journals.”
ResultA Review of Using Chatgpt for Scientific Manuscript Writing
Nafea, A. A., AL-Ani, M. M., Khalaf, M. A., & Alsumaidaie, M. S. I. (2024) · Babylonian Journal of Artificial Intelligence · doi:10.58496/bjai/2024/002 · Cited by 10
Methodological foundations for artificial intelligence ‐driven survey question generation
Mburu, T., Rong, K., McColley, C. J., & Werth, A. (2025) · Journal of Engineering Education · doi:10.1002/jee.70012 · Cited by 14
“Further, AI-generated prompts can vary in tone, relevance, or clarity, especially when poorly designed (Hosseini & Horbach, 2023; Johri et al., 2023; Menekse, 2023; Miao & Holmes, 2023).”
Miao, W., Zang, H., Liu, Q., Zheng, T., Zhou, Y., Liu, C., Yang, N., Zhang, H., Zhang, Y., Zhang, Y., et al. (2025) · BMC Medical Ethics · doi:10.1186/s12910-025-01288-0 · Cited by 7
“Reviewers and authors must disclose their technology use and are responsible for data security, confidentiality, and report accuracy, logic, and originality [31].”
How public involvement can improve the science of AI
Matias, J. N., & Price, M. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2421111122 · Cited by 13
Updated Guidance for Author Use of AI in Medical Publication
Flanagin, A., Perlis, R. H., & Bibbins-Domingo, K. B. (2026) · JAMA · doi:10.1001/jama.2026.16613 · Cited by 16
Disclosing use of Artificial Intelligence: Promoting transparency in publishing
Koul, P. A. (2023) · Lung India · doi:10.4103/lungindia.lungindia_370_23 · Cited by 7
“While some journals such as Accountability in Research, [5] Journal of the American Medical Association (JAMA) [6] and Nature [7] decided to adopt or pursue policies that allow using LLMs under conditions that promote transparency, accountability, fair assignment of credit and honesty, the editors…”
BackgroundRole of generative artificial intelligence in publishing. What is acceptable, what is not.
Wong, R. K. (2023) · Journal of ExtraCorporeal Technology · doi:10.1051/ject/2023033 · Cited by 7
“Other concerns that Hosseini and Horbach (2023) identified fault “the fundamental opacity of LLM’s training data, inner workings, data handling, and development processes” which can result in “potential biases and the reproducibility of review reports” [3].”
BackgroundA bibliometric analysis of literature published on ChatGPT and GPT
Safdar, M., Siddique, N., Gulzar, A., Adil, S. A., Yasin, H., & Khan, M. A. (2024) · Global Knowledge, Memory and Communication · doi:10.1108/gkmc-08-2023-0304 · Cited by 7
Hosseini, M., & Horbach, S. P. J. M. (2023) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00136-2 · Cited by 7
Self-citationCan Artificial Intelligence Be My Coauthor?
Scerbo, M. W. (2023) · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · doi:10.1097/sih.0000000000000743 · Cited by 7
Winarti, H., Kholid, M. R., & Zakiyah, Z. (2025) · Cetta: Jurnal Ilmu Pendidikan · doi:10.37329/cetta.v8i2.4076 · Cited by 9
“This careful approach demonstrates an understanding that while AI is helpful, it should not dictate the final outcome of their work (Hosseini and Horbach, 2023).”
Ali, S. I., & Shaban, M. (2025) · International Nursing Review · doi:10.1111/inr.70100 · Cited by 6
Guidelines needed for the use of AI in the preparation or review of IRB, IBC, and IACUC applications
Hosseini, M., Eisenman, D., Riddle, J., Pyle, S., Peters, A., Cobb, N. L., & Holmes, K. (2026) · Accountability in Research · doi:10.1080/08989621.2025.2612564 · Cited by 7
“…studies; (b) AI searches are not always reproducible and could yield different results depending on the prompt (or chain of prompts) used (Hosseini and Horbach 2023); (c) AI’s (sometimes) inappropriate cultural assumptions and social orientations, based on limited and/or biased training…”
Self-citationAI and the Future of Publishing: Not Detection, but Answerability
Floridi, L. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01148-8 · Cited by 6
“Hosseini and Horbach warn that careless deployment of eLLMs risks amplifying past review bias rather than correcting it (Hosseini & Horbach, 2023).”
Tang, B. L. (2023) · EXCLI journal · doi:10.17179/excli2023-6435 · Cited by 16
Aliyeva, A., & Sarı, E. (2023) · Cureus · doi:10.7759/cureus.48366 · Cited by 7
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12
Hidden Prompts in Manuscripts Exploit AI-Assisted Peer Review
Lin, Z. (2026) · Communications of the ACM · doi:10.1145/3779116 · Cited by 7
“3 This burden creates incentives for shortcuts that compromise review integrity.”
Mohamed, A. A., Rajendran, S., Colome, D., Sargent, E. C., Schirmer, C. M., Vessell, M., Lucke-Wold, B. P., Sharma, A., Adogwa, O., & Pirris, S. M. (2025) · Neurosurgical Review · doi:10.1007/s10143-025-03793-7 · Cited by 5
Navigating the complexities of artificial intelligence in scientific writing: a dual perspective
Levin, G., Piedimonte, S., & Zand, B. (2024) · International Journal of Gynecological Cancer · doi:10.1136/ijgc-2024-005691 · Cited by 7
“35 This may potentially lead to erroneous decisions by editorial boards.”
Wang, X., Dai, C., & Bao, L. (2025) · Information Discovery and Delivery · doi:10.1108/idd-05-2024-0072 · Cited by 5
Medical ChatGPT – A systematic Meta-ReviewPreprint
Egger, J., Sallam, M., Luijten, G., Gsaxner, C., Pepe, A., Kleesiek, J., Puladi, B., & Li, J. (2024) · medRxiv · doi:10.1101/2024.04.02.24304716 · Cited by 5
The problem of fairness in tools for algorithmic fairness
Sun, X.-Y., Ye, B., & Xia, B.-H. (2024) · AI and Ethics · doi:10.1007/s43681-024-00533-3 · Cited by 4
Promotion of medical research: Stance in AI-generated and scholar-written highlights
Zou, H., & Fan, Y. (2025) · Journal of English for Academic Purposes · doi:10.1016/j.jeap.2025.101593 · Cited by 6
Yang, Z., Zhou, X., Jiang, Y., Zhang, X., Gao, Q., Lu, Y., & Yang, A. (2025) · Publications · doi:10.3390/publications14010001 · Cited by 5
“This over-trust could lead to a “rubber-stamping” culture, where the AI’s lack of deep contextual understanding or potential training biases are overlooked (Hosseini & Horbach, 2023).”
LLMs in Peer Review—How Publishing Policies Must Advance
Leung, T. I. (2026) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.52042 · Cited by 5
When AI reviews science: Can we trust the referee?
Wang, J., Liu, Y., Xu, H., Hu, K., Di, S., Ni, W., Yue, L., Zhang, M.-L., Ren, K., & Chen, L. (2026) · The Innovation Informatics · doi:10.59717/j.xinn-inform.2026.100030 · Cited by 7
“Despite attempts at algorithmic mitigation, these solutions face significant limitations 135,136 , often due to deep-seated structural issues that make the bias difficult to eradicate without effective oversight 137,138 .”
Challenges in the Peer-review process
Mishra, U. (2025) · Journal of Indonesian Management · doi:10.53697/jim.v5i1.2570 · Cited by 3
“Reports of https://penerbitadm.pubmedia.id/index.php/JIM biased decisions, lack of accountability, and gatekeeping practices that strangle new or unconventional ideas have surfaced, prompting calls for greater transparency and the exploration of alternative review models (Hosseini & Horbach, 2023).”
Linardon, J., Thomas, J. J., Crow, S. J., Ghaderi, A., Hilbert, A., Klump, K. L., Wade, T. D., Walsh, B. T., & Weissman, R. S. (2025) · International Journal of Eating Disorders · doi:10.1002/eat.24543 · Cited by 7
Cen, M. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-13175-z · Cited by 7
“This transformation is supported by institutional backing and the development of open-access infrastructure, which are vital for promoting the sustainability of these initiatives (Hosseini & Horbach, 2023).”
Generative Artificial Intelligence in Surgical Publishing
Li, B., Kayssi, A., & McLean, L. J. (2025) · JAMA Surgery · doi:10.1001/jamasurg.2024.6446 · Cited by 4
Zhou, X., Su, G., Zhu, L., & Liu, Y. (2025) · BMC Nursing · doi:10.1186/s12912-025-04102-9 · Cited by 4
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
Thakkar, N., Yuksekgonul, M., Silberg, J., Garg, A., Peng, N., Sha, F., Yu, R., Vondrick, C., & Zou, J. (2025) · arXiv · doi:10.48550/arxiv.2504.09737 · Cited by 4
“Reviewers are increasingly turning to LLMs to assist in drafting their reviews, with an estimated 10.6% of reviewers at ICLR 2024 using LLMs for this purpose [15, 16].”
InfluentialCan generative AI facilitate the research process? It’s complicated
Kingsley, D. (2023) · College & Research Libraries News · doi:10.5860/crln.84.9.342 · Cited by 6
Design Knowledge in AI: Navigating Temporality and Continuity
Cho, H., Seo, J. A., Jung, H., Cheon, E., Seo, W., Lupetti, M. L., Pierce, J., & Dove, G. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715668.3734175 · Cited by 3
Lei, F., Du, L., Wang, W., Dong, M., & Liu, X. (2025) · Journal of Information Science · doi:10.1177/01655515251359771 · Cited by 3
Wang, X., Liu, B., & Li, X. (2025) · Journal of Information Science · doi:10.1177/01655515251362399 · Cited by 3
Is ChatGPT Trustworthy Enough? A Review
Zhou, G., Liu, Y., Yan, Z., & Erol Gelenbe, A. (2024) · IEEE Consumer Electronics Magazine · doi:10.1109/mce.2024.3485257 · Cited by 3
“Hosseini and Horbach 28 discussed potential biases in peer review processes when using ChatGPT, emphasizing its amplifi-cation of existing biases.”
When and how to disclose AI use in academic peer review
Masters, K., & Cleland, J. (2026) · Medical Teacher · doi:10.1080/0142159x.2025.2609741 · Cited by 3
Caputo, J., Ibrahim, A. H., Davis, D. W., Stone, W. J., Garver, M. J., & Johnson, S. L. (2025) · Kinesiology Review · doi:10.1123/kr.2025-0010 · Cited by 3
Co-Reviewer: can AI review like a human? An agentic framework for LLM-human alignment in peer review
Bharti, P. K., Dalal, V., & Panchal, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05557-6 · Cited by 2
Large Language Models as Peer Reviewers: Prompt Sensitivity and Model-Dependent Reproducibility
Erturk, S. M., & Durmaz, M. (2026) · Academic Radiology · doi:10.1016/j.acra.2026.04.046 · Cited by 2
AI in Scholarly Communications and Academic Research
Tzanova, S. (2024) · Advances in library and information science (ALIS) book series · doi:10.4018/979-8-3693-3053-1.ch002 · Cited by 3
Artificial Intelligence and Publishing Ethics: A Narrative Review and SWOT Analysis
Gurnal, P., & Rana, L. (2025) · Cureus · doi:10.7759/cureus.84098 · Cited by 4
“…authors to take onus to maintain the integrity of the content generated by AI models and tools for reproduced and recreated material and images [26,28-32, Regulatory agencies have devised policies for the use of various analysis software and recommended authors to follow EQUATOR guidelines, and…”
InfluentialBernstein, E., Ramsamooj, A., Millar, K. L., & Lum, Z. C. (2025) · JMIR AI · doi:10.2196/68603 · Cited by 3
Buitrago, J., Campos, E. M., & Romero, C. L. V. (2025) · Investigación Bibliotecológica Archivonomía Bibliotecología e Información · doi:10.22201/iibi.24488321xe.2025.104.59032 · Cited by 8
Teixeira, A. L. D. S. (2025) · Research Integrity and Peer Review · doi:10.1186/s41073-025-00182-y · Cited by 5
“…have been explored for several tasks, including (but not limited to) accelerating the time to secure reviewers, providing initial quality control of submitted manuscripts, detecting plagiarism, generating review reports and recommendations, and assessing the quality of peer reviews [15, 24–28].”
Ng, J. Y., Lee, M. S., Gupta, M. M., Tan, J., Adapa, K., Gupta, P. K., Li, S., Mehta, D. H., Ring, M. R., Shridhar, M., et al. (2025) · International Journal of Ayurveda Research · doi:10.4103/ijar.ijar_293_25 · Cited by 2
Charbonneau, D. H., & Zhang, M. (2026) · Accountability in Research · doi:10.1080/08989621.2026.2632083 · Cited by 7
The New Editorial Gatekeepers: Understanding LLM-Based Interfaces, Their Benefits, Risks and Design
Floridi, L. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01068-7 · Cited by 4
“This proactive filtering saves reviewers’ time and efforts (Hosseini and Horbach 2023), strengthens research integrity and could prevent embarrassing later retractions.”
Gruszczyński, D., Nijakowski, K., Halupczok-Żyła, J., Ruchała, M., Walkowiak, J., & Sawicka‐Gutaj, N. (2026) · Journal of Medical Science · doi:10.20883/medical.e1571 · Cited by 3
ChatGPT-4 as a Tool for Reviewing Academic Books in Spanish
Berrezueta-Guzmán, J., Malache-Silva, L., & Krusche, S. (2023) · Lecture notes in educational technology · doi:10.1007/978-981-99-7353-8_29 · Cited by 3
Algorithmic power and scientific knowledge
Signorini, I. (2023) · Language Culture and Society · doi:10.1075/lcs.00044.sig · Cited by 2
Navigating Scientific Peer Review with ChatGPT: Ally or Adversary?
MohanaSundaram, A. (2024) · Advanced Pharmaceutical Bulletin · doi:10.34172/apb.2024.053 · Cited by 2
Reflections on a mentored group peer review process
Auby, H., Grundy, L. S., Huffman, S., Cantilina, K., Gavitte, S. B., Kaczynski, S., Penyai, M., & Koretsky, M. (2024) · Journal of Engineering Education · doi:10.1002/jee.20616 · Cited by 2
Atalan, A., Keskin, A., & Özer, S. (2024) · International Journal of Healthcare Management · doi:10.1080/20479700.2024.2444102 · Cited by 4
“ChatGPT can support staff performance evaluation and feedback processes [94].”
Changing peer review practices: transforming roles and future challenges
Horbach, S. P. J. M., & Overgaard, S. (2025) · Acta Orthopaedica · doi:10.2340/17453674.2025.44353 · Cited by 2
Self-citationDoes ChatGPT Add Value to University Students’ Academic Performance?
Kaur, P., Jawaid, A., Qi, R. O. J., Rokman, R. D. B. C. M., Awai, S. S. A., Liew, S. S., & Yunus, S. N. B. M. (2023) · International Journal of Membrane Science and Technology · doi:10.15379/ijmst.v10i3.2583 · Cited by 2
“ChatGPT can help editors finish time-consuming or repetitive jobs (such fixing grammar mistakes) and prevent them from having preconceived notions about certain articles (Hosseini et al., 2023).”
BackgroundChatGPT's Influence on Journal Impact Factors: A Paradigm ShiftPreprint
Ahmed, S. K., & Ahmed, S. K. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4467193 · Cited by 2
Krausman, P. R., & Knipps, A. C. S. (2023) · Journal of Wildlife Management · doi:10.1002/jwmg.22537 · Cited by 1
An Algorithm to Support the Review Process of a Scientific Research Project Proposal Using ChatGPT
Rashidov, A. (2024) · International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT) · doi:10.1109/ismsit63511.2024.10757273 · Cited by 1
Low, D. A., Halvorsen, P. H., & Hedrick, S. (2025) · Medical Physics · doi:10.1002/mp.17657 · Cited by 1
Public Versus Academic Discourse on ChatGPT in Health Care: Mixed Methods Study
Baxter, P., Li, M., Wei, J., & Koizumi, N. (2025) · JMIR Infodemiology · doi:10.2196/64509 · Cited by 2
Sun, C. (2025) · Ceramics - Silikaty · doi:10.13168/cs.2025.0023 · Cited by 1
Detecting the knowledge structure of generative artificial intelligence research
Cheng, X. S., Wu, P., & Liang, L. (2025) · Journal of Information Science · doi:10.1177/01655515251353188 · Cited by 1
GenAI Embedded in Activities of Academic Research: Experiences and Lessons from HCI Studies
de Alencar, T. S., Ciscon, L., Pereira, G., Macedo, M. P., Martinelli, S., Choma, J., & Zaina, L. A. M. (2025) · Simpósio Brasileiro de Fatores Humanos em Sistemas Computacionais (IHC) · doi:10.5753/ihc.2025.10800 · Cited by 1
“…report practical application of GenAI for particular tasks, such as coding for qualitative analysis [Xiao et al. 2023], reviewing academic articles [Hosseini e Horbach 2023], literature review [Alshami et al. 2023, Fabiano et al. 2024, Burger et al. 2023], and quantitative analysis [Perkins e Roe…”
InfluentialSebo, P., Nie, B., & Wang, T. (2025) · F1000Research · doi:10.12688/f1000research.173647.1 · Cited by 4
A Comparative Study on Large Language Models with an Overview of Their Issues and Challenges
Kumar, A., & Sood, M. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-5969-2_13 · Cited by 1
Harnessing generative AI to drive responsible business research and accelerate social impact
Steingard, D. S., Reibstein, D., & Normandin, M. (2025) · Journal of Social Impact in Business Research · doi:10.1108/jsibr-11-2024-0051 · Cited by 1
Sharma, N. K., & Sarode, S. C. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01013-9 · Cited by 2
Ibrahim, S. A., & Mahmoud, Z. A. Z. (2026) · Humanities and Social Sciences Communications · doi:10.1057/s41599-026-06956-z · Cited by 5
Safeguarding the role of humans in conservation science in the age of AI
Bates, A. E., Maas, B., Adams, V. M., Benson, J. F., DellaSala, D. A., Ghoddousi, A., Goswami, V. R., Lee, J. S. H., Nori, J., Rinaudo, V., et al. (2026) · Biological Conservation · doi:10.1016/j.biocon.2026.111893 · Cited by 2
Peykani, P., Charles, V., Emrouznejad, A., Ghanidel, S., & Sabour, D. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10720-6 · Cited by 1
Sanpablo, A. I. P., Ruiz, M. D. C. A., Peñaloza, A. M., Rodríguez‐Reyes, G., Quiñones-Urióstegui, I., & Anaya-Campos, L. (2023) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-031-46933-6_14 · Cited by 1
The Peer Review Process - a Tutorial
Pedersen, M. R. V., Rusandu, A., Sanderud, A., et al. (2023) · Radiography Open · doi:10.7577/radopen.5702 · Cited by 1
Ortopedi ve travmatoloji araştırmalarında yapay zekâ uygulamaları: Doğal dil işleme potansiyeli
Köse, Ö., Yapar, D., & Boz, M. (2024) · Türk Ortopedi ve Travmatoloji Birliği Derneği · doi:10.5578/totbid.dergisi.2024.13 · Cited by 1
“[34] Başlık Önerme ve Gözden Geçirme Bilimsel bir makalenin başlığı, makalenin daha kolay anlaşılmasına, indekslenmesine ve aranmasına yardımcı olmaktadır.”
ChatGPT and Its Ethical Implications on Libraries, Other Institutions, and Society
Holland, B. J. (2023) · Advances in library and information science (ALIS) book series · doi:10.4018/979-8-3693-2841-5.ch014 · Cited by 1
Lievens, J. (2024) · INTED proceedings · doi:10.21125/inted.2024.0394 · Cited by 2
Multilingualism in Scholarly Communication: How Far Can Technology Take Us and What Else Can We Do?
Bowker, L. (2024) · Journal of Electronic Publishing · doi:10.3998/jep.6262 · Cited by 1
Status of peer review guidelines in international surgical journals: A cross‐sectional survey
Dong, M., Wang, W., Liu, X., Lei, F., & Luo, Y. (2024) · Learned Publishing · doi:10.1002/leap.1624 · Cited by 1
“Furthermore, the guidelines should also clearly outline the content and methods of peer reviewers using LLMs to assist in writing review reports in a transparent way (Hosseini & Horbach, 2023).”
Artificial intelligence in medical publishing
Ørstavik, R. (2024) · Tidsskrift for Den norske legeforening · doi:10.4045/tidsskr.24.0514 · Cited by 1
Enhancing Peer Fairness via Data-Driven Analysis for Outlier Detection
Di, Z., Zhang, J., Tan, W., & Sun, X. (2024) · Electronics · doi:10.3390/electronics13234735 · Cited by 1
The need for ethical guidelines in mathematical research in the time of generative AI
Pantsar, M. (2025) · AI and Ethics · doi:10.1007/s43681-025-00660-5 · Cited by 1
“Consequently, many have argued for introducing clear guide - lines for the use of AI in scientific research and publishing [7, 47 ].”
Singh, N., Neubronner, S., Kanayan, S., Illanes, S. E., Choolani, M. A., & Kemp, M. W. (2025) · Singapore Medical Journal · doi:10.4103/singaporemedj.smj-2024-173 · Cited by 1
A systematic review on the role of ChatGPT on student learning
Iqbal, J. (2024) · Integration of Mankind and Society · doi:10.56770/ims2024123 · Cited by 1
Fast, Furious and Dubious? MDPI and the Depth of Peer Review ReportsPreprint
Maddi, A., & Boukacem‐Zeghmouri, C. (2023) · Research Square · doi:10.21203/rs.3.rs-3027724/v1 · Cited by 1
Generative AI-assisted Peer Review in Medical Publications: Opportunities Or Trap (Preprint)Preprint
Li, Z., Shen, C., Cao, F., Xiang, C. J., Fei, Y., & Liu, J. (2024) · doi:10.2196/preprints.66022 · Cited by 1
Harnessing Generative AI to Drive Responsible Business Research and Accelerate Social ImpactPreprint
Steingard, D. S., Reibstein, D. J., & Normandin, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5016963 · Cited by 1
Invisible Text Injection: The Trojan Horse of AI-Assisted Medical Peer ReviewPreprint
Choi, B., Jun, T. J., Sung, J. W., et al. (2025) · medRxiv · doi:10.1101/2025.07.24.25332148 · Cited by 1
Retraction trends in Indonesian research: misconduct, AI risks, and policy gaps
Samala, A. D., Ferdian, F., Fiandra, Y. A., Hanafiah, M. H., Rawas, S., & Asad, M. M. (2025) · Quality & Quantity · doi:10.1007/s11135-025-02453-8 · Cited by 4
A Meta-Synthesis of Factual Accuracy and Citation Hallucination in LLM Academic Assistants
Anantama, R., Bachtiar, M. I., & Rahman, Z. (2026) · Bulletin of Data Science · doi:10.47065/bulletinds.v5i3.10443
“[18] emphasize that the proliferation of AI-generated text, including that containing hallucinations, compels the scientific community to re-evaluate ethical standards and peer-review processes to maintain trust in scientific literature and avoid unintentional biases.”
Lupon, E., Bérard, Q., Migaud, H., Clavert, P., & Micicoi, G. (2026) · Orthopaedics & Traumatology: Surgery & Research · doi:10.1016/j.otsr.2026.104801
ChatGPT és más nagy nyelvi modellek (LLM-ek) biztonsági kérdései, szervezeti és társadalmi hatásai
Nagy, P. (2024) · Rendvédelem · doi:10.53793/rv.2024.2.1 · Cited by 1
Lucas, F. M., Mackie, I. J., d’Onofrio, G., & Frater, J. L. (2024) · International Journal of Laboratory Hematology · doi:10.1111/ijlh.14285
“30 One area where chatbots are likely to add benefit to the medical literature is by improving the overall quality of medical writing.”
BackgroundFrom advancements to ethics: Assessing ChatGPT’s role in writing research paper
Gupta, V., Anamika, F., Parikh, K., Patel, M. A., Jain, R., & Jain, R. (2024) · Turkish Journal of Internal Medicine · doi:10.46310/tjim.1398093
Metadata
- Authors
- Hosseini, M., & Horbach, S. P. J. M.
- Affiliations
- Northwestern University (US)
- Aarhus University (DK)
- Corresponding author
- Hosseini, Mohammad
- Venue
- Research Integrity and Peer Review
- Published
- 2023-05-18
- Volume
- 8
- Issue
- 1
- Pages
- 4-4
- ISSN
- 2058-8615 (ISSN-L)
- PMC
- PMC10191680
- OpenAlex
- W4376866708
- Semantic Scholar
- 4670942a6332554dfaea91cebee6d64d2e4ae233
- Cited by
- 328
- References
- 27
- Citations by year
- 2026: 115 · 2025: 88 · 2024: 75 · 2023: 50
- FWCI
- 104.06
- Type
- journal article
- Language
- en
- Update notices
- correction (2023-07-10)
- Open access
- gold
- doi.org
- Locations
- Research Integrity and Peer Review (publishedVersion, cc-by)
- Research Square (submittedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Center for Advancing Translational Sciences (UL1TR001422)
- Topics
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- large language models, ChatGPT, peer review, reviewer fatigue, review bias, editorial work, reproducibility, epistemic communities, data security and confidentiality, scholarly communication, AI ethics in academia, AI disclosure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-16, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-16 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
