The DOI matches its Crossref record; 90 of the 90 references it cites were found in a registry or an index; 12 of the 43 sources its claims cite were checked for support.
How should the advancement of large language models affect the practice of science?
Open accessAbstract
Large language models (LLMs) are being increasingly incorporated into scientific workflows. However, we have yet to fully grasp the implications of this integration. How should the advancement of large language models affect the practice of science? For this opinion piece, we have invited four diverse groups of scientists to reflect on this query, sharing their perspectives and engaging in debate. Schulz et al. make the argument that working with LLMs is not fundamentally different from working with human collaborators, while Bender et al. argue that LLMs are often misused and overhyped, and that their limitations warrant a focus on more specialized, easily interpretable tools. Marelli et al. emphasize the importance of transparent attribution and responsible use of LLMs. Finally, Botvinick and Gershman advocate that humans should retain responsibility for determining the scientific roadmap. To facilitate the discussion, the four perspectives are complemented with a response from each group. By putting these different perspectives in conversation, we aim to bring attention to important considerations within the academic community regarding the adoption of LLMs and their impact on both current and future scientific practices.
Overview
CoChat Summary
TL;DR
This PNAS opinion piece convenes four groups of scientists to debate how LLM advancement should affect scientific practice, with each perspective followed by responses from the other groups. The contributors disagree sharply on whether LLMs are collaborator-like aids, overhyped and often inappropriate tools, or technologies requiring principles of transparency, accountability, and fairness while reserving core scientific judgment for humans.
Key findings
- Schulz et al. argue that working with LLMs is “not fundamentally different from working with other collaborators” and that existing rules for scientific practice suffice; they advocate open-source models for reproducibility.
- Bender et al. argue LLMs are “often mischaracterized, misused, and overhyped,” that they are models of word form distributions rather than meaning, and that bespoke, interpretable tools are preferable for most scientific tasks.
- Marelli et al. call for principles rather than strict regulation, emphasizing transparency, accountability, and fairness; they cite Liang et al.’s finding that “more than half (57.4%) of the users found GPT-4 generated feedback helpful or very helpful and 82.4% found it more beneficial than feedback from at least some human reviewers.”
- Botvinick and Gershman argue that two core aspects of science should remain human: the normative aspect of choosing problems and the epistemic aspect of human understanding.
Methods
- Opinion piece format: four invited groups of scientists each provide a perspective on how LLM advancement should affect science, followed by a response from each group and a concluding general discussion.
- No new empirical data are collected; the article synthesizes arguments and cites prior studies, including examples such as Meta’s Galactica being taken offline after three days and Yax et al.’s failed replication of proprietary LLM reasoning results after three months.
- The discussion is organized around applications in navigating, doing, and communicating science, with cross-responses among the four contributor groups.
Limitations
- The paper states that it is an opinion piece and that the invited perspectives were selected to cover a broad spectrum, not to provide a systematic review or empirical test.
- The authors note that it remains unclear whether benchmarks can be established for scientific workflows and whether benchmark success would translate to meaningful scientific output.
- Bender et al. state that there are “no published user studies that quantify just how much effort this checking process is, nor how accurately researchers can carry it out, especially while working under pressure”; more generally, the text makes evident that the perspectives rely on argument and cited examples rather than new controlled evidence.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 7 Not valid · 20 Not checked · 28 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Finding answers to this question is urgent as LLMs are already starting to permeate the academic landscape (10–17).”
§ How should the advancement of large language models affect the practice of science? > Abstract
What the sources say
- Not checked yetcites [10] R. Peres, M. Schreier, D. Schweidel, A. Sorescu, On ChatGPT and beyond: How generative artificial intelligence may affect research, teaching, and practice. Int. J. Res. Mark. 40, 269– 275 (2023). source
A check is scheduled.
- Supportedcites [11] B. D. Lund, T. Wang, Chatting about ChatGPT: how may AI and GPT impact academia and libraries? Libr. Hi Tech News 40, 26– 29 (2023). source
The GPT and ChatGPT section (pp. 4–5) describes ChatGPT as a public tool capable of producing scholarly essays and articles. The paper also uses ChatGPT to discuss academic research and library work (p. 5), supporting the broad claim that LLMs were beginning to enter academia. (LLM, conf=0.84)
- Not checked yetcites [12] E. L. Hill-Yardin, M. R. Hutchinson, R. Laycock, S. J. Spencer, A Chat(GPT) about the future of scientific publishing. Brain Behav. Immun. 110, 152– 154 (2023). 36868432 source
A check is scheduled.
- Supportedcites [13] H. Zheng, H. Zhan, ChatGPT in scientific writing: A cautionary tale. Am. J. Med. 136, 725– 726.e6 (2023). 36906169 source
The opening passage (labeled Abstract in our copy) describes ChatGPT’s growing popularity, reports journal articles written using it, and explicitly refers to its increasing use in scientific writing. That supports the broad claim that LLMs are beginning to enter academic work. (LLM, conf=0.91)
- Supportedcites [14] B. D. Lund, ChatGPT and a new academic reality: Artificial intelligence-written research papers and the ethics of the large language models in scholarly publishing. J. Assoc. Inf. Sci. Technol. 74, 570– 581 (2023). source
The Authorship, Copyright, and Plagiarism section says the ethics of using ChatGPT to generate scientific articles have already been widely discussed. The Citation Practices section also discusses researchers using ChatGPT to assist with citations. Together, these support the broad claim that LLMs are beginning to enter academic work; no page numbers are available in our copy. (LLM, conf=0.86)
- Not checked yetcites [15] A. Birhane, A. Kasirzadeh, D. Leslie, S. Wachter, Science in the age of large language models. Nat. Rev. Phys. 5, 277– 280 (2023). source
A check is scheduled.
- Supportedcites [16] B. Fecher, M. Hebing, M. Laufer, J. Pohle, F. Sofsky, Friend or foe? Exploring the implications of large language models on the science system. arXiv [Preprint] (2023). https://arxiv.org/abs/2306.09928 (Accessed 19 February 2024). source
Section 4.2 reports that most surveyed experts already use or intend to use LLMs in their work, supporting the claim that LLMs are starting to enter academic practice. The paper cautions that its sample may overrepresent technology-proficient experts. (LLM, conf=0.88)
- Not checked yetcites [17] C. Stokel-Walker, R. Van Noorden, What ChatGPT and generative AI mean for science. Nature 614, 214–216 (2023). 36747115 source
A check is scheduled.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not checked yet
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Likewise, this has been observed by Yax et al. (40) who tested the reasoning capabilities of LLMs, and found that the results of proprietary LLMs, i.e., ChatGPT and GPT4, could not be replicated three months after the initial experiments.”
§ How should the advancement of large language models affect the practice of science? > Perspective—LLMs: More Like a Human Collaborator than a Software Tool > Contributors.
What the sources say
- Partly supportedcites [40] N. Yax, H. Anlló, S Palminteri, Studying and improving reasoning in humans and machines. Commun. Psychol. 2, 51 (2024). PMC11332180 39242743 sourceExperimental — accuracy not yet measured
The Results (Test-retest reliability) report significant changes in ChatGPT’s and GPT4’s task performance between March and June 2023. But the Discussion (Limitations) says the experiments generally reached the same overall conclusion at both time points—that the models outperformed humans. Thus, saying the results ‘could not be replicated’ overstates what changed. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Also, LLMs can access a broader range of literature than any individual researcher could, potentially offering valuable support for literature analysis and hypothesis generation (59), with a reach that, biases aside, goes beyond one’s research specialization.”
§ How should the advancement of large language models affect the practice of science? > Perspective—LLMs in Scientific Practice: A Matter of Principles, Not Just Regulations > Contributors.
What the sources say
- Could not verifycites [59] J. Jumper, Highly accurate protein structure prediction with alphafold. Nature 596, 583– 589 (2021). 34265844 PMC8371605 sourceExperimental — accuracy not yet measured
The Related work section discusses attention-based methods for interpreting protein sequences, while Methods > Inputs and data sources describes protein-sequence databases; neither addresses researchers’ access to scientific literature. Searches for LLM literature access, literature analysis, and hypothesis generation found no passage supporting the attributed claim. Page numbers were not available in our copy. (LLM, conf=0.75)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“It has been, for instance, repeatedly demonstrated that even the most capable models at present fail at basic arithmetic problems such as multiplying two four-digit numbers (22) when directly asked for the answer as opposed to writing code.”
§ How should the advancement of large language models affect the practice of science? > Abstract
What the sources say
- Supportedcites [22] K. Arkoudas, GPT-4 can’t reason. arXiv [Preprint] (2023). https://arxiv.org/abs/2308.03762 (Accessed 4 July 2024). source
Section 3.1 (Simple Arithmetic) states directly that "GPT-4 is still unable to reliably perform elementary arithmetic operations such as addition and multiplication," and demonstrates it on a four-digit-by-four-digit product: asked to select two random numbers between 1381 and 1453 and report the product, GPT-4 answers 1405 * 1421 = 1996025, and the author notes "the correct answer is 1405 • 1421 = 1996505." The prompt used there is exactly the "directly asked for the answer" mode the citing sentence describes (report the result, no code), and the same section records the concession that GPT-4 "can't add, multiply or divide" with computational gaps supposedly fillable by plug-ins/agents — the tool-using alternative. GPT-4 is the most capable model at the time of writing, so the claim's scope is met. (LLM, conf=0.85)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“our inability to understand why they make the errors they do or when they will make them impairs our ability to understand their limits, especially on the edges of knowledge, where their training corpus is arguably less robust (83, 84).”
§ How should the advancement of large language models affect the practice of science? > Response by Marco Marelli, Adina Roskies, Balazs Aczel, Colin Allen, Dirk Wulff, and Qiong Zhang
What the sources say
- Supportedcites [83] H. Zhao, Explainability for large language models: A survey. ACM Trans. Intell. Syst. Technol. 15, 1–38 (2023). source
The Introduction (page not provided) says LLM complexity makes their decisions harder to interpret and obstructs debugging and diagnosis. Section 4.2.2, “Explaining Hallucination” (page not provided), links errors to insufficient relevant data, difficulty learning long-tail knowledge, and difficulty identifying factual knowledge boundaries. Together, these support the claim’s point about understanding errors and limits at the edges of knowledge. (LLM, conf=0.88)
- Not checked yetcites [84] H. Luo, L. Specia, From understanding to utilization: A survey on explainability for large language models. arXiv [Preprint] (2024). https://arxiv.org/abs/2401.12874 (Accessed 19 February 2024).
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
22 more claims
Create a free account to see all 63 claims and to verify the 31 references still open.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 14
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ How should the advancement of large language models affect the practice of science? > Response by Eric Schulz, Daniel Schad, Marcel Binz, Stephan Alaniz, Ven Popov, and Zeynep Akata
“For example, Marelli et al. write that “we should not blindly trust or rely on LLMs, but rather use them as a complement to our own expertise and judgment,””
§ How should the advancement of large language models affect the practice of science? > Perspective—LLMs: More Like a Human Collaborator than a Software Tool > Contributors.
“That means that we should not blindly trust or rely on LLMs, but rather use them as a complement to our own expertise and judgment.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ How should the advancement of large language models affect the practice of science? > Perspective—LLMs: More Like a Human Collaborator than a Software Tool > Contributors.
“Following a recent call for action to the European Parliament (43)”
§ How should the advancement of large language models affect the practice of science? > References
“43. Towards a transparent AI future: The call for less regulatory hurdles on open-source AI in Europe (2003).”
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Self-citationThe Principles of Scientific Management.
Woolf, L. S., Taylor, F. W., Scott, W. D., & Goldmark, J. (1912) · The Economic Journal · doi:10.2307/2221959 · Cited by 5,256
Jurafsky, D. S., & Martin, J. (2000) · Cited by 4,132
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Research works printed but not linked: 3
- S. Narayan S. B. Cohen M. Lapata Ranking sentences for extractive summarization with reinforcement learning. arXiv [Preprint] (2018). https://arxiv.org/abs/1802.08636 (Accessed 19 February 2024). · arXiv
- 10.1073/pnas.2305016120 · DOI · Paper page
- C. Flaherty The peer review crisis (2023). https://www.insidehighered.com/news/2022/06/13/peer-review-crisis-creates-problems-journals-and-scholars. Accessed 30 October 2023.
Cited by
Generalization bias in large language model summarization of scientific research
Peters, U., & Chin-Yee, B. (2025) · Royal Society Open Science · doi:10.1098/rsos.241776 · Cited by 126
“While LLMs hold substantial potential as tools for scientific summarization [3,5], their use carries significant risks, as they may oversimplify or exaggerate scientific findings [12], which can lead to large-scale misunderstandings of science.”
InfluentialThe benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110
Self-citationThe simulation of judgment in LLMs
Loru, E., Nudo, J., Di Marco, N., Santirocchi, A., Atzeni, R., Cinelli, M., Cestari, V., Rossi-Arnaud, C., & Quattrociocchi, W. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2518443122 · Cited by 57
“Large Language Models (LLMs) are increasingly embedded in workflows involving classification, evaluation, recommendation, and decision support [1].”
Quaquebeke, N. V., Tonidandel, S., & Banks, G. C. (2025) · The Leadership Quarterly · doi:10.1016/j.leaqua.2025.101895 · Cited by 57
Empowering Generalist Material Intelligence with Large Language Models
Yuan, W., Chen, G., Wang, Z., & You, F. (2025) · Advanced Materials · doi:10.1002/adma.202502771 · Cited by 54
Metadata
- Authors
- Binz, M., Alaniz, S., Roskies, A., Aczel, B., Bergstrom, C. T., Allen, C., Schad, D., Wulff, D., West, J. D., Zhang, Q., Shiffrin, R. M., Gershman, S. J., Popov, V., Bender, E. M., Marelli, M., Botvinick, M. M., Akata, Z., & Schulz, E.
- Affiliations
- Max Planck Institute for Biological Cybernetics (DE)
- Munich Center for Machine Learning (DE)
- University of California, Santa Barbara (US)
- Institute of Psychology, Eötvös Loránd University
- University of Washington (US)
- HMU Health and Medical University (DE)
- University of Basel (CH)
- Rutgers, The State University of New Jersey (US)
- Indiana University Bloomington (US)
- Harvard University (US)
- University of Zurich (CH)
- University of Milano-Bicocca (IT)
- Google DeepMind (United Kingdom) (GB)
- Corresponding author
- Binz, Marcel
- Alaniz, Stephan
- Bender, Emily M.
- Marelli, Marco
- Botvinick, Matthew M.
- Akata, Zeynep
- Schulz, Eric
- ORCID
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2025-01-27
- Volume
- 122
- Issue
- 5
- Pages
- e2401227121-e2401227121
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC11804466
- OpenAlex
- W4406855515
- Semantic Scholar
- bff18c97402aeb5cdb0244e2239ff0fa21c1590e
- Cited by
- 74
- References
- 68
- Citations by year
- 2026: 42 · 2025: 31
- FWCI
- 29.28
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- BOA (University of Milano-Bicocca) (submittedVersion, other-oa)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
- Funders
- Deutsche Forschungsgemeinschaft (206/4-1, 390727645)
- Bundesministerium für Bildung und Forschung (01IS18039A, FKZ: 01IS18039A)
- H2020 European Research Council (853489)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- large language models, scientific practice, scientific workflows, LLM misuse, human-AI collaboration, responsible AI use
- MeSH terms
- Large Language Models, Diffusion of Innovation, Humans, Language, Models, Theoretical, Science
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How should the advancement of large language models affect the practice of science? https://app.cochat.ai/papers/10.1073/pnas.2401227121
- The paper
- Binz, M., Alaniz, S., Roskies, A., Aczel, B., Bergstrom, C. T., Allen, C., Schad, D., Wulff, D., West, J. D., Zhang, Q., Shiffrin, R. M., Gershman, S. J., Popov, V., Bender, E. M., Marelli, M., Botvinick, M. M., Akata, Z., & Schulz, E. (2025). How should the advancement of large language models affect the practice of science? Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2401227121
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.