ChatGPT cites the most-cited articles and journals, relying solely on Google Scholar's citation counts. As a result, AI may amplify the Matthew Effect in environmental science
Open accessAbstract
ChatGPT (GPT) has become one of the most talked-about innovations in recent years, with over 100 million users worldwide. However, there is still limited knowledge about the sources of information GPT utilizes. As a result, we carried out a study focusing on the sources of information within the field of environmental science. In our study, we asked GPT to identify the ten most significant subdisciplines within the field of environmental science. We then asked it to compose a scientific review article on each subdiscipline, including 25 references. We proceeded to analyze these references, focusing on factors such as the number of citations, publication date, and the journal in which the work was published. Our findings indicate that GPT tends to cite highly-cited publications in environmental science, with a median citation count of 1184.5. It also exhibits a preference for older publications, with a median publication year of 2010, and predominantly refers to well-respected journals in the field, with Nature being the most cited journal by GPT. Interestingly, our findings suggest that GPT seems to exclusively rely on citation count data from Google Scholar for the works it cites, rather than utilizing citation information from other scientific databases such as Web of Science or Scopus. In conclusion, our study suggests that Google Scholar citations play a significant role as a predictor for mentioning a study in GPT-generated content. This finding reinforces the dominance of Google Scholar among scientific databases and perpetuates the Matthew Effect in science, where the rich get richer in terms of citations. With many scholars already utilizing GPT for literature review purposes, we can anticipate further disparities and an expanding gap between lesser-cited and highly-cited publications.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The traditional safeguards of scholarly publishing are poorly calibrated for this class of error.
“suggesting that the traditional safeguards of scholarly publishing are poorly calibrated for this class of error [5].”
From How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations (2026) · cites this paper as [5]
The Conclusion says traditional academic publishing methods are becoming obsolete, and the Introduction describes differing journal policies on ChatGPT. Neither passage assesses whether publishing safeguards can detect or prevent the particular class of error the citing sentence refers to; the Discussion of GPT’s inconsistent references does not make that connection either. (LLM, conf=0.76)
Experimental — accuracy not yet measuredRecent audits of conference proceedings have found hallucinated references embedded in peer-reviewed submissions that multiple expert reviewers failed to flag.
“Recent audits of conference proceedings have found hallucinated references embedded in peer-reviewed submissions that multiple expert reviewers failed to flag”
From How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations (2026) · cites this paper as [5]
The Introduction (page unavailable) discusses concerns about AI-generated academic writing, and the section on research problems (page unavailable) says GPT provides references somewhat randomly. Neither reports an audit of conference proceedings or hallucinated references in peer-reviewed submissions that multiple reviewers failed to flag. Searches for those details found no supporting passage in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 3
- Li, E. Y., & Chen, Y. (2019).The Matthew effect in scientific funding.Proceedings of the National Academy of Sciences,
- 116(16), 7684-7689.9Petersen, A. M., Fortunato, S., Pan, R. K., Kaski, K., Penner, O.,Rungi, A., ... & Barabási, A. L. (2018).Reputation and impact in academic careers.Proceedings of the National Academy of Sciences, 111(43), 15316-15321.8Merton, R. K. (1968).The Matthew effect in science: The reward and communication systems of science are considered.Science, 159(3810), 56-63.7 https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=allintitle%3AChatGPT&btnG=&oq=al6 https://en.wikipedia.org/wiki/ChatGPT 5 Brainard, Jeffrey (February 22, 2023)."As scientists explore AI-written text, journals hammer out policies".Science.doi:10.1126/science.adh2937 4 Mitchell, Alex (December 26, 2022)."Students using ChatGPT to cheat, professor warns".The New York Post. 3 Stokel-Walker, Chris (December 9, 2022)."AI bot ChatGPT writes smart essays -should professors worry?".Nature.doi:10.1038/d41586-022-04397-7 2 Stokel-Walker, Chris (January 18, 2023)."ChatGPT listed as author on research papers: many scientists disapprove".Nature.613 (7945): 620-621. 1 Bushard, Brian (January 10, 2023)."Fake Scientific Abstracts Written By ChatGPT Fooled Scientists, Study Finds".Forbes.Archived from the original on February 3, 2023. · DOI
- Thompson, N., & Hanley, D. (2018).Science is shaped by Wikipedia: evidence from a randomized control trial.13 Wang, J., Veugelers, R., & Stephan, P. (2013).Bias against novelty in science: A cautionary tale for users of bibliometric indicators.NBER Working Paper No. 19135.12Bol, T., de Vaan, M., & van deRijt, A. (2018).The Matthew effect in science funding.Proceedings of the National Academy of Sciences, 115(19), 4887-4890.11Allison,P. D., & Stewart, J. A. (1974).Productivity differences among scientists: Evidence for accumulative advantage.American Sociological Review, 39(4), 596-606.
Cited by
Sorted by citations
Naser, M. (2026) · arXiv · doi:10.48550/arxiv.2603.03299 · Cited by 1 · see citing works
“Recent audits of conference proceedings have found hallucinated references embedded in peer-reviewed submissions that multiple expert reviewers failed to flag, suggesting that the traditional safeguards of scholarly publishing are poorly calibrated for this class of error [5].”
- Authors
- Petiska, E.
- Corresponding author
- Petiska, Eduard
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.06794
- OpenAlex
- W4366196961
- Cited by
- 2
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- ChatGPT, citation bias, Matthew effect, Google Scholar, citation counts, environmental science, highly cited papers, scientific databases, journal prestige
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
