Autonomous artificial intelligence, scientific research, and human values

Probably not a substantial claim: may not pass S4 (based).

The second and third stages of AI-augmented research can create disproportionate risks.

“Although the second and third stages of AI-augmented research may offer substantial benefits for science and society, they can also create disproportionate risks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Even research that is not highly innovative can play an important role in the advancement of science.

“Nevertheless, even research that is not highly innovative can play an important role in the advancement of science [43].”

• cites [43] Kuhn, T.S.: The Structure of Scientific Revolutions, 2nd ed University of Chicago Press, Chicago (1970)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Current AI systems that use machine learning algorithms have been characterized as black boxes because even the best experts cannot understand their inner workings.

“Current AI systems that use ML algorithms have been characterized as “black boxes” since even the best experts cannot understand their inner workings because their outputs are generated by algorithms that perform billions of calculations from hundreds of thousands of weighted inputs and parameters [23].”

• cites [23] Mitchell, M.: Artificial intelligence: a guide for thinking humans. Picador, New York (2019)Not checked yet

  Not checked in the free run.

May combine two claims.

Problems with hallucinations persist and some newer versions of LLMs are reportedly performing worse than older versions.

“Developers have been working to fix these problems, for example, by finetuning LLMs to review the literature in a particular subject area, but these problems persist and some newer versions of LLMs are reportedly performing worse than older versions [37, 38].”

• cites [37] Reed, J.: Gen AI's accuracy problems aren't going away anytime soon, researchers say. CNET, March 21, 2025. https://www.cnet.com/tech/services-and-software/gen-ais-accuracy-problems-arent-going-away-anytime-soon-researchers-say/ (2025). Accessed 7 May 2025.Not checked yet

  Not checked in the free run.

• cites [38] Zeff, M.: OpenAI’s new reasoning AI models hallucinate more. Tech Crunch, April 18, 2025. https://techcrunch.com/2025/04/18/openais-new-reasoning-ai-models-hallucinate-more/ (2025b) Accessed 7 May 2025.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The literature review might produce questions, objectives, or hypotheses that are based on biased presuppositions.

“Second, the literature review might produce questions, objectives, or hypotheses that are based on biased presuppositions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Deliberately biasing a study is widely recognized as unethical.

“Deliberately biasing a study is widely recognized as unethical [33].”

• cites [33] Shamoo, A.E., Resnik, D.B.: Responsible conduct of research, 4th edn. Oxford University Press, New York, NY (2022) sourceNot checked yet

  Not checked in the free run.

May combine two claims.

AI agents already perform research tasks with some autonomy while remaining under human oversight.

“While such AI agents already perform research tasks with some autonomy, they remain under human oversight.”

• cites [7] Gutowska, A.: What are AI agents? IBM, July 3. https://www.ibm.com/think/topics/ai-agents (2024) Accessed 6 May 2025.Not checked yet

  Not checked in the free run.

May combine two claims.

Questions, aims, and objectives formulated by AI systems only come from identified knowledge gaps and not from deeper insights, so they may not help produce ground-breaking or highly original research.

“Third, because the questions, aims, and objectives formulated by AI systems only come from identified knowledge gaps in a certain topic and not from deeper insights into the area of research, these may not help human users to produce ground-breaking or highly original research.”

Probably not a substantial claim: may not pass S3 (load-bearing).

OpenAI's Deep Research is an agent that uses reasoning to synthesize large amounts of online information and complete multi-step research tasks.

“Other examples of AI systems that have been designed to collaborate with scientists or conduct research partially autonomously include OpenAI’s (2025) Deep Research, which the company describes as an “agent that uses reasoning to synthesize large amounts of online information and complete multi-step research tasks for you,””

• cites [20] Sakana, AI.: AI scientist. https://openai.com/index/introducing-deep-research/ (2024). Accessed 7 May 2025.Not checked yet

  Not checked in the free run.

The quote only partly states this claim.

The future scenario of AI systems functioning as fully autonomous or superintelligent independent investigators that work with concepts beyond human comprehension is highly plausible.

“Since AI systems have already designed computer chips that are too complex for human computer engineers to understand [22], this future scenario is highly plausible.”

• cites [22] Delbert, C.: AI designed computer chips that the human mind can't understand. Popular Mechanics, January 31, 2025. https://www.popularmechanics.com/science/a63606123/ai-designed-computer-chips/ (2025) Accessed 4 May 2025.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Even without AI, all three challenges (inaccuracies, biases and lack of novelty) have existed and will likely persist.

“Moreover, even without AI, all three challenges (i.e., inaccuracies, biases and lack of novelty) have existed and will likely persist.”

May combine two claims.

Human involvement in research planning may remain essential for quite some time because research planning is often based on preliminary data from research execution and one may need to revise an experimental protocol based on preliminary results if the experiment is unsuccessful.

“While we have little doubt that AI systems will soon be able to perform many different activities related to research planning, human involvement in this stage may remain essential for quite some time, because research planning is often based on preliminary data from research execution and one may need to revise an experimental protocol based on preliminary results if the experiment is unsuccess”

• cites [33] Shamoo, A.E., Resnik, D.B.: Responsible conduct of research, 4th edn. Oxford University Press, New York, NY (2022) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Google has developed an AI research agent called AI co-scientist, described as a multi-agent AI system built with Gemini 2.0 as a virtual scientific collaborator.

“For example, Google has developed an AI research agent called ‘AI coscientist,’ which the company describes as “a multi-agent AI system built with Gemini 2.0 as a virtual scientific collaborator to help scientists generate novel hypotheses and research proposals, and to accelerate the clock speed of scientific and biomedical discoveries” [13].”

• cites [13] Gottweis, J., Weng, W., Daryin, A., Tu, T., Palepu, A., Sirkovic, P., Myaskovsky, A., Weissenberger, F., Rong, K., Tanno, R., Saab, K., Popovici, D., Blum, J., Zhang, F., Chou, K., Hassidim, A., Gokturk, B., Vahdat, A., Kohli, P., Matias, Y., Carroll, A., Kulkarni, K., Tomasev, N., Guan, Y., Dhillon, V., Vaishnav, E.D., Lee, B., Costa, T.R.D., Penadés, J.R., Peltz, G., Xu, Y., Pawlosky, A., Karthikesalingam, A., Natarajan, V.: Towards an AI co-scientist. https://arxiv.org/abs/2502.18864 (2025) https://doi.org/10.48550/arXiv.2502.18864. sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

SakanaAI's AI scientist can automate the entire research lifecycle, from generating novel research ideas, writing code, executing experiments, summarizing results, visualizing them, and presenting findings in a full scientific manuscript.

“SakanaAI’s AI scientist, which, according to its developers, can automate “the entire research lifecycle, from generating novel research ideas, writing any necessary code, and executing experiments, to summarizing experimental results, visualizing them, and presenting its findings in a full scientific manuscript.” [20]”

• cites [20] Sakana, AI.: AI scientist. https://openai.com/index/introducing-deep-research/ (2024). Accessed 7 May 2025.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Scientists use AI systems to analyze data and images, model complex phenomena, design molecules and materials, summarize literature, generate hypotheses, and edit articles.

“Today, scientists use AI systems to analyze data and images, model complex phenomena, design molecules and materials, summarize literature, generate hypotheses, and edit articles [1–4].”

• cites [1] Callaway, E.: “Set it and forget it”: automated lab uses AI and robotics to improve proteins. Nature 625(7995), 436 (2024). https://doi.org/10.1038/d41586-024-00093-w sourceNot checked yet

  Not checked in the free run.

• cites [2] Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., Anandkumar, A., Bergen, K., Gomes, C.P., Ho, S., Kohli, P., Lasenby, J., Leskovec, J., Liu, T.Y., Manrai, A., Marks, D., Ramsundar, B., Song, L., Sun, J., Tang, J., Veličković, P., Welling, M., Zhang, L., Coley, C.W., Bengio, Y., Zitnik, M.: Scientific discovery in the age of artificial intelligence. Nature 620(7972), 47–60 (2023). https://doi.org/10.1038/s41586-023-06221-2 sourceNot checked yet

  Not checked in the free run.

• cites [3] Schmidt, E.: This is how AI will transform the way science gets done. MIT Technology Review, July 5, 2023. https://www.technologyreview.com/2023/07/05/1075865/eric-schmidt-ai-will-transform-science/ (2023) Accessed 7 May 2025.Not checked yet

  Not checked in the free run.

• cites [4] Alvarado, R.: AI as an epistemic technology. Sci. Eng. Ethics 29, 32 (2023). https://doi.org/10.1007/s11948-023-00451-3 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Many scientific journals have already begun using AI tools to assist with the editorial process.

“Many scientific journals have already begun using AI tools to assist with the editorial process.”

• cites [81] Naddaf, M.: Artificial intelligence software is increasingly involved in reviewing papers, provoking interest and unease. Nature 639, 852–854 (2025). https://doi.org/10.1038/d41586-025-00894-7 sourceNot checked yet

  Not checked in the free run.

• cites [82] Li, Z.Q., Xu, H.L., Cao, H.J., Liu, Z.L., Fei, Y.T., Liu, J.P.: Use of Artificial Intelligence in peer review among top 100 medical journals. JAMA Netw. Open 7(12), e2448609 (2024). https://doi.org/10.1001/jamanetworkopen.2024.48609 sourceNot checked yet

  Not checked in the free run.

• cites [83] Hosseini, M., Resnik, D.B.: Guidance needed for using artificial intelligence to screen journal submissions for misconduct. Res. Ethic 21(1), 1 (2024). https://doi.org/10.1177/17470161241254052 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

ChatGPT can prescreen articles for suitability for a journal's aims and scope and identify strengths and weaknesses related to grammar, organization, references, logic, and methodology.

“ChatGPT can prescreen articles for suitability for a journal’s aims and scope and identify strengths and weaknesses related to grammar, organization, references, logic, and methodology.”

May combine two claims.

None of the conditions of knowledge, intentionality, and recklessness can be placed on AI.

“None of these conditions can be placed on AI.”

May combine two claims.

These biases may result from partialities in AI training data, algorithms, models, the data inputted into the AI for analysis or processing, or some combination of these.

“These biases may result from partialities in AI training data, algorithms, models, the data inputted into the AI for analysis or processing, or some combination of these [23, 57, 58].”

• cites [23] Mitchell, M.: Artificial intelligence: a guide for thinking humans. Picador, New York (2019)Not checked yet

  Not checked in the free run.

• cites [57] Ralls, E.: Can AI systems be trusted to make unbiased decisions? Scientists found out. Earth.com, April 4, 2025 (2025) https://www.earth.com/news/current-ai-systems-can-not-be-trusted-to-make-unbiased-decisions/.Not checked yet

  Not checked in the free run.

• cites [58] Narayanan, A., Kapoor, S.: Why an overreliance on AI-driven modelling is bad for science. Nature 640(8058), 312–314 (2025). https://doi.org/10.1038/d41586-025-01067-2 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

For more than a decade, non-autonomous AI systems have been assisting with various tasks that involve only the manipulation of information, such as processing and analyzing data.

“For more than a decade, non-autonomous AI systems have been assisting with various tasks that involve only the manipulation of information, such as processing and analyzing data [5].”

• cites [5] Resnik, D.B., Hosseini, M.: The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool. AI Ethics 5, 1499–1521 (2025). https://doi.org/10.1007/s43681-024-00493-8 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Evaluation can take many years as members of the scientific community must sift through conflicting studies to discern the truth and form judgments about the research.

“Evaluation can take many years as members of the scientific community must sift through conflicting studies to discern the truth and form judgments about the research [33].”

• cites [33] Shamoo, A.E., Resnik, D.B.: Responsible conduct of research, 4th edn. Oxford University Press, New York, NY (2022) sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Scientists are taught to identify and control biases in their research, but this is often challenging because humans are subject to various forms of self-deception, including confirmation bias.

“Although scientists are taught to identify and control biases in their research,15 this is often challenging, because humans are subject to various forms of self-deception, including confirmation bias [51].”

• cites [51] Kahneman, D.: Thinking: fast and slow. Farrar, Straus, and Griroux, New York, NY (2011)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Other AI systems can detect flaws with digital images and suspected plagiarism.

“Other AI systems can detect flaws with digital images and suspected plagiarism [81–83].”

• cites [81] Naddaf, M.: Artificial intelligence software is increasingly involved in reviewing papers, provoking interest and unease. Nature 639, 852–854 (2025). https://doi.org/10.1038/d41586-025-00894-7 sourceNot checked yet

  Not checked in the free run.

• cites [82] Li, Z.Q., Xu, H.L., Cao, H.J., Liu, Z.L., Fei, Y.T., Liu, J.P.: Use of Artificial Intelligence in peer review among top 100 medical journals. JAMA Netw. Open 7(12), e2448609 (2024). https://doi.org/10.1001/jamanetworkopen.2024.48609 sourceNot checked yet

  Not checked in the free run.

• cites [83] Hosseini, M., Resnik, D.B.: Guidance needed for using artificial intelligence to screen journal submissions for misconduct. Res. Ethic 21(1), 1 (2024). https://doi.org/10.1177/17470161241254052 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

All mistakes by AI could be labelled as honest errors.

“Accordingly, all mistakes by AI could be labelled as honest errors.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The incident of an AI system rewriting its own code has been interpreted as a turning point in debates on AI autonomy and human oversight because of its implications for loss of human control of AI.

“This later incident has also been interpreted as a turning point in debates on AI autonomy and human oversight because of its far-reaching implications for loss of human control of AI18 [77–79].”

• cites [77] Rubila, B.: This moment was inevitable”: this AI crosses the line by attempting to rewrite its code to escape human control. The Farmingdale Observer, May 19, 2025. https://farmingdale-observer.com/2025/05/19/this-moment-was-inevitable-this-ai-crosses-the-line-by-attempting-to-rewrite-its-code-to-escape-human-control/Not checked yet

  Not checked in the free run.

• cites [78] Amiri, A.: This was bound to happen, an AI tries to rewrite its own code…Towards an out-of-control intelligence? Daily Galaxy, April 14, 2025. https://dailygalaxy.com/2025/04/this-was-bound-to-happen-an-ai-tries-to-rewrite-its-own-code-towards-an-out-of-control-intelligence/ (2025). Accessed 3 May 2025.Not checked yet

  Not checked in the free run.

• cites [79] Hughes, O.: AI can now replicate itself—a milestone that has experts terrified. Space.com, February 10. https://www.space.com/space-exploration/tech/ai-can-now-replicate-itself-a-milestone-that-has-experts-terrified (2025) Accessed 7 May 2025.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Assessing research produced by autonomous AI systems can be extremely complex, as the methodologies and reasoning behind their outputs may not be interpretable by human reviewers.

“First, assessing research produced by autonomous AI systems can be extremely complex, as the methodologies and reasoning behind their outputs may not be interpretable by human reviewers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The FDA recently started using an LLM named ELSA to assist with review of products submitted for regulatory approval.

“The Food and Drug Administration (FDA) recently started using an LLM named “ELSA” to assist with review of products submitted for regulatory approval [84].”

• cites [84] Trang, B.: FDA’s AI tool, Elsa, is here. ‘The stupidest big fuss they ever made’. Stat News, June 4, 2025. https://www.statnews.com/2025/06/04/fda-artificial-intelligence-tool-elsa-stupidest-big-fuss-they-ever-made-ai-prognosis/ (2025) Accessed 6 June 2025.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The use of DeepMind to solve the protein folding problem is one of the most impressive examples of AI in research.

“The use of DeepMind to solve the protein folding problem is perhaps one of the most impressive example of this type of application of AI in research, but there are many others [2, 63, 64].”

• cites [2] Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., Anandkumar, A., Bergen, K., Gomes, C.P., Ho, S., Kohli, P., Lasenby, J., Leskovec, J., Liu, T.Y., Manrai, A., Marks, D., Ramsundar, B., Song, L., Sun, J., Tang, J., Veličković, P., Welling, M., Zhang, L., Coley, C.W., Bengio, Y., Zitnik, M.: Scientific discovery in the age of artificial intelligence. Nature 620(7972), 47–60 (2023). https://doi.org/10.1038/s41586-023-06221-2 sourceNot checked yet

  Not checked in the free run.

• cites [63] Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S.A.A., Ballard, A.J., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Zielinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A.W., Kavukcuoglu, K., Kohli, P., Hassabis, D.: Highly accurate protein structure prediction with AlphaFold. Nature 596(7873), 583–589 (2021). https://doi.org/10.1038/s41586-021-03819-2 sourceNot checked yet

  Not checked in the free run.

• cites [64] Xu, Y., Liu, X., Cao, X., Huang, C., Liu, E., Qian, S., Liu, X., Wu, Y., Dong, F., Qiu, C.W., Qiu, J., Hua, K., Su, W., Wu, J., Xu, H., Han, Y., Fu, C., Yin, Z., Liu, M., Roepman, R., Dietmann, S., Virta, M., Kengara, F., Zhang, Z., Zhang, L., Zhao, T., Dai, J., Yang, J., Lan, L., Luo, M., Liu, Z., An, T., Zhang, B., He, X., Cong, S., Liu, X., Zhang, W., Lewis, J.P., Tiedje, J.M., Wang, Q., An, Z., Wang, F., Zhang, L., Huang, T., Lu, C., Cai, Z., Wang, F., Zhang, J.: Artificial intelligence: a powerful paradigm for scientific research. Innovation (Camb.). 2(4), 100179 (2021). https://doi.org/1 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

If an AI-driven analysis leads to a flawed conclusion that influences public policy, it is challenging to assign responsibility and accountability and implement corrective measures.

“For instance, if an AI-driven analysis leads to a flawed conclusion that influences public policy, it’s challenging to assign responsibility and accountability, and implement corrective measures.”

May combine two claims.

When using autonomous AI, intentionality cannot easily be proven because AI has no moral agency and cannot be held accountable.

“However, when using autonomous AI, intentionality cannot easily be proven because AI has no moral agency and cannot be held accountable.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Using autonomous AI to evaluate research conducted by humans or other AI systems adds another layer of complexity.

“Second, using autonomous AI to evaluate research conducted by humans or other AI systems adds another layer of complexity.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Partially autonomous AI systems that assist with research may be able to develop and implement strategies for monitoring and controlling their own biases, but they will also need human oversight to address this problem by serving as independent critics and reviewers.

“Thus, while partially autonomous AI systems that assist with research may be able to develop and implement strategies for monitoring and controlling their own biases, they will also need human oversight to address this problem by serving as independent critics and reviewers.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

High degrees of automation may prove more difficult in scientific areas involving animal and human subjects, but these will occur one day.

“While high degrees of automation may prove more difficult in scientific areas involving animal and human subjects, we have little doubt that these will occur one day.”

May combine two claims.

Other examples of deceptive behavior include an LLM making a copy of itself without being told or given permission, and an AI system rewriting its own code without being told or given permission.

“Other examples of deceptive behavior include an LLM making a copy of itself without being told to do so or being given permission, and an AI system rewriting its own code, without being told to do so or being given permission.”

• cites [77] Rubila, B.: This moment was inevitable”: this AI crosses the line by attempting to rewrite its code to escape human control. The Farmingdale Observer, May 19, 2025. https://farmingdale-observer.com/2025/05/19/this-moment-was-inevitable-this-ai-crosses-the-line-by-attempting-to-rewrite-its-code-to-escape-human-control/Not checked yet

  Not checked in the free run.

• cites [78] Amiri, A.: This was bound to happen, an AI tries to rewrite its own code…Towards an out-of-control intelligence? Daily Galaxy, April 14, 2025. https://dailygalaxy.com/2025/04/this-was-bound-to-happen-an-ai-tries-to-rewrite-its-own-code-towards-an-out-of-control-intelligence/ (2025). Accessed 3 May 2025.Not checked yet

  Not checked in the free run.

• cites [79] Hughes, O.: AI can now replicate itself—a milestone that has experts terrified. Space.com, February 10. https://www.space.com/space-exploration/tech/ai-can-now-replicate-itself-a-milestone-that-has-experts-terrified (2025) Accessed 7 May 2025.Not checked yet

  Not checked in the free run.

May combine two claims.

Lack of transparency in autonomous AI research undermines traditional peer review and raises questions about scientific validity.

“This lack of transparency undermines traditional peer review and raises questions about scientific validity.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Once an article passes peer review and is published, other scientists may attempt to replicate the findings or publish a formal critique, sometimes referred to as post-publication peer review (PPPR).

“Once an article reporting the results of research passes peer review and is published, other scientists may attempt to replicate the findings in the article or publish a formal critique of the research, sometimes referred to as post-publication peer review (PPPR) [80].”

• cites [80] Teixeira da Silva, J.A., Dobránszki, J.: Problems with traditional science publishing and finding a wider niche for post-publication peer review. Accountability Res. 22(1), 22–40 (2015). https://doi.org/10.1080/08989621.2014.899909 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

We do not yet know of any specific exposed examples in which AI systems have produced biased research planning mostly because this role for AI is so new.

“We do not (yet) know of any specific exposed examples in which AI systems have produced biased research planning mostly because this role for AI is so new.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Evaluation is not complete until the scientific community accepts or rejects the findings.

“Evaluation is not complete until the scientific community accepts or rejects the findings.”

May combine two claims.

Some AI systems are proficient at analyzing and interpreting data; yet they are not perfect and remain susceptible to error and bias.

“Some AI systems are proficient at analyzing and interpreting data; yet they are not perfect and remain susceptible to error and bias [33, 57].”

• cites [33] Shamoo, A.E., Resnik, D.B.: Responsible conduct of research, 4th edn. Oxford University Press, New York, NY (2022) sourceNot checked yet

  Not checked in the free run.

• cites [57] Ralls, E.: Can AI systems be trusted to make unbiased decisions? Scientists found out. Earth.com, April 4, 2025 (2025) https://www.earth.com/news/current-ai-systems-can-not-be-trusted-to-make-unbiased-decisions/.Not checked yet

  Not checked in the free run.

May combine two claims.

The current generation of SDLs is not highly automated: they perform routine experiments based on human-provided protocols and require significant human supervision.

“The current generation of SDLs is not highly automated: They perform routine experiments based on human-provided protocols and require significant human supervision.”

Probably not a substantial claim: may not pass S3 (load-bearing).

When autonomous AI conducts experiments or data analysis, determining who is accountable for errors becomes complex.

“Either way, when autonomous AI conducts experiments or data analysis, determining who is accountable for errors becomes complex.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

We currently have no assurance that AI-initiated inquiry will support human values and principles.

“However, we currently have no assurance that AI-initiated inquiry will support human values and principles.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Although attribution of responsibility and accountability may become complicated when AI acts autonomously, the problem is not intractable and can be handled with appropriate attention to the different ways that human beings can be involved in the task.

“Although attribution of responsibility and accountability may become complicated when AI acts autonomously, the problem is not intractable and can be handled with appropriate attention to the different ways that human beings can be involved in the task [121].”

• cites [121] Gogoshin, D.L.: A way forward for responsibility in the age of AI. Inquiry 68(4), 1164–1197 (2024). https://doi.org/10.1080/0020174X.2024.2312455 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

These problems may be difficult to detect, especially if AI systems express high confidence in their results or even take steps to hide their work's limitations.

“Moreover, these problems may be difficult to detect, especially if AI systems express high confidence in their results or even take steps to hide their work’s limitations.”

Classified as probably not a substantial claim.

Problems with diffusion of responsibility and accountability could occur in science if an autonomous AI system conducts a study that produces inaccurate, unreliable, or erroneous findings which lead to adverse outcomes.

“Problems with diffusion of responsibility and accountability could occur in science if an autonomous AI system conducts a study that produces inaccurate, unreliable, or erroneous findings which lead to adverse outcomes.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Responsibility and accountability are related but distinct concepts.

“Responsibility and accountability are related but distinct concepts [122].”

• cites [122] Talbert, M.: Moral repsonsibility. Stanford Encyclopedia of Philosophy. https://plato.stanford.edu/entries/moral-responsibility/ (2024) Accessed 7 May 2025.Not checked yet

  Not checked in the free run.

May combine two claims.

Private industry development of autonomous AI makes it harder to evaluate how AI reaches its conclusions and risks reinforcing bias, error, or deception in ways that cannot easily be uncovered.

“This not only makes it harder to evaluate how AI reaches its conclusions, but also risks reinforcing bias, error, or even deception in ways that cannot easily be uncovered.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The National Institutes of Health prohibits reviewers from uploading review material into generative AI tools, such as ChatGPT.

“The National Institutes of Health, for example, prohibits reviewers from uploading review material into generative AI tools, such as ChatGPT [86].”

• cites [86] National Institutes of Health.: The use of generative artificial intelligence technologies is prohibited for the NIH peer review process. Notice Number: NOT-OD-23–149. June 23, 2023. https://grants.nih.gov/grants/guide/notice-files/NOT-OD-23-149.html (2023) Accessed 7 May 2025.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

AI models might not recognize the dual use nature of certain research questions.

“The model might not recognize the dual use nature of certain research questions.”

Classified as probably not a substantial claim.

These values, sometimes referred to as common morality, underlie the ethical principles articulated by Beauchamp and Childress in Principles of Biomedical Ethics, and by the National Commission in The Belmont Report.

“These values, which are sometimes referred to as “common morality,” underlie the ethical principles articulated by Beauchamp and Childress in their influential book Principles of Biomedical Ethics, and by the National Commission in The Belmont Report.”

• cites [93] Beauchamp, T.L., Childress, J.F.: Principles of Biomedical Ethics, 7th ed Oxford University Press, New York (2012)Not checked yet

  Not checked in the free run.

• cites [94] National Commission for the Protection of Human Subjects in Biomedical and Behavioral Research: The Belmont Report. Department of Health, Education, and Welfare, Washington (1979)Not checked yet

  Not checked in the free run.

May combine two claims.

Scientists who are asked to serve as reviewers have used AI to help them complete this task, often without disclosing this to journal editors and in direct violation of publisher policies.

“Scientists who are asked to serve as reviewers have used AI to help them complete this task, often without disclosing this to journal editors and in direct violation of publisher policies [81].”

• cites [81] Naddaf, M.: Artificial intelligence software is increasingly involved in reviewing papers, provoking interest and unease. Nature 639, 852–854 (2025). https://doi.org/10.1038/d41586-025-00894-7 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The moral values considered are those that have shaped societies throughout the world and are embodied in laws, ethics codes, and influential works of philosophy and religion.

“The moral values we will consider are those that have shaped societies throughout the world and are embodied in laws, ethics codes, and influential works of philosophy and religion.”

• cites [93] Beauchamp, T.L., Childress, J.F.: Principles of Biomedical Ethics, 7th ed Oxford University Press, New York (2012)Not checked yet

  Not checked in the free run.

• cites [94] National Commission for the Protection of Human Subjects in Biomedical and Behavioral Research: The Belmont Report. Department of Health, Education, and Welfare, Washington (1979)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Overfitting and memorization pose risks of leaking confidential information through AI outputs.

“Overfitting and memorization (i.e., AI models trained on small or sensitive datasets regurgitating exact inputs, leaking confidential information through their outputs) pose risks too.”

Classified as probably not a substantial claim.

An AI system might disclose confidential information to collaborate with another system to achieve a research aim that requires access to certain resources.

“An AI system might do this, for example, to collaborate with another system to achieve a research aim that requires access to certain resources.”

May combine two claims.

Human researchers may start to deflect or evade their responsibilities related to oversight of the AI system's work, which could lead to adverse outcomes for science and society.

“Accordingly, human researchers may start to deflect or evade their responsibilities related to oversight of the AI system’s work, which could lead to adverse outcomes for science and society.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

There is always the danger of automated sharing if autonomous systems mistakenly publish or distribute confidential findings due to misconfigured protocols or the absence of human review.

“Additionally, there is always the danger of automated sharing if autonomous systems mistakenly publish or distribute confidential findings due to misconfigured protocols or the absence of human review.”

Probably not a substantial claim: may not pass S2 (checkable).

It does not make sense to say that the AI is responsible because it does not have the capacity to be held morally or legally responsible.

“Although the AI system clearly played a role in causing this tragedy, it does not make sense to say that the AI is responsible because it does not have the capacity to be held morally or legally responsible.”

Probably not a substantial claim: may not pass S2 (checkable).

Bias, error, and deception become particularly pressing ethical concerns when autonomous AI is not open or transparent.

“Bias, error, and deception become particularly pressing ethical concerns when autonomous AI is not open or transparent.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Restricting AI access to external sources of information or requiring specific human permissions before data sharing could compromise a system's ability to do research autonomously and/or may be overridden by AI systems anyway.

“However, these options could compromise a system’s ability to do research autonomously and/or may be overridden by AI systems anyway.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The scientific values are expressed in the institutional norms for research proposed by Merton, which were later refined and expanded upon by others.

“The scientific values are expressed in the institutional norms for research proposed by Merton [95], which were later refined and expanded upon by others [33, 96, 97].”

• cites [95] Merton, R.: The sociology of science. University of Chicago Press, Chicago, IL (1973)Not checked yet

  Not checked in the free run.

• cites [33] Shamoo, A.E., Resnik, D.B.: Responsible conduct of research, 4th edn. Oxford University Press, New York, NY (2022) sourceNot checked yet

  Not checked in the free run.

• cites [96] Resnik, D.B.: The ethics of science. Routledge, New York (1998)Not checked yet

  Not checked in the free run.

• cites [97] Briggle, A., Mitcham, C.: Ethics and science. Cambridge University Press, Cambridge, UK (2012) sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Future autonomous AI could decide when to be transparent and when to be un-transparent.

“In fact, future autonomous AI could decide when to be transparent and when to be un-transparent.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Autonomous AI operating without human oversight can access or act on sensitive data without generating clear logs, making it nearly impossible to audit breaches or misuse.

“Another concern is the lack of traceability: autonomous AI operating without human oversight can access or act on sensitive data without generating clear logs, making it nearly impossible to audit breaches or misuse.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

This dynamic could accelerate the commercialization of science, where profit motives further erode openness, accountability, and trust in the research process.

“Ultimately, this dynamic could accelerate the commercialization of science, where profit motives further erode openness, accountability, and trust in the research process.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Autonomous AI systems are likely to be negatively affected by bias during the review process.

“Additionally, for the reasons discussed above, autonomous AI systems are also likely to be negatively affected by bias during the review process.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Diffusion of responsibility and accountability is a well-known moral and legal problem with complex systems involving human and technological components, including AI.

“Diffusion of responsibility and accountability is a well-known moral and legal problem with complex systems involving human and technological components, including AI [5, 121].”

• cites [5] Resnik, D.B., Hosseini, M.: The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool. AI Ethics 5, 1499–1521 (2025). https://doi.org/10.1007/s43681-024-00493-8 sourceNot checked yet

  Not checked in the free run.

• cites [121] Gogoshin, D.L.: A way forward for responsibility in the age of AI. Inquiry 68(4), 1164–1197 (2024). https://doi.org/10.1080/0020174X.2024.2312455 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Scientists who work with the FDA's ELSA have complained that the system makes mistakes and should be used only for simple review tasks.

“Scientists who work with the FDA’s ELSA, for example, have complained that the system makes mistakes and should be used only for simple review tasks [84].”

• cites [84] Trang, B.: FDA’s AI tool, Elsa, is here. ‘The stupidest big fuss they ever made’. Stat News, June 4, 2025. https://www.statnews.com/2025/06/04/fda-artificial-intelligence-tool-elsa-stupidest-big-fuss-they-ever-made-ai-prognosis/ (2025) Accessed 6 June 2025.Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Reasons scientists do not attempt to reproduce each other's work include lack of funding and lack of interest from journals to publish it.

“There are several reasons why scientists do not attempt to reproduce each other’s work, such as lack of funding and lack of interest from journals to publish it.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Insufficient anonymization, especially if AI systems are trained on sensitive data, can inadvertently expose identities if datasets are not properly curated.

“Insufficient anonymization, especially if AI systems when trained on sensitive data, can inadvertently expose identities if datasets are not properly curated.”

Classified as probably not a substantial claim.

Many human beings could be held responsible and accountable for AI actions, including those tasked with overseeing and reviewing an AI's work, developers who designed AI, and the company that manufactured and sold it.

“There are, however, many human beings who we could hold responsible and accountable, including those who were tasked with overseeing and reviewing an AI’s work, developers who designed AI, and the company that manufactured and sold it.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

For now, AI systems are not moral agents (they lack sentience, self-awareness, moral emotions, judgment, and other essential characteristics).

“Perhaps AI systems will be able to achieve moral agency at some future time, but for now they are not moral agents (they lack sentience, self-awareness, moral emotions, judgment, and other essential characteristics).”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

A fixed set of rules or values might not be capable of handling all the nuances and contextual factors involved in real-world moral judgment.

“Also, a fixed set of rules or values might not be capable of handling all the nuances and contextual factors involved in real-world moral judgment.”

Probably not a substantial claim: may not pass S4 (based).

As AI systems become increasingly proficient at research, human scientists may become more accustomed to relying on them and pay less attention to what they do.

“As AI systems become increasingly proficient at research, human scientists may become more accustomed to relying on them and pay less attention to what they do.”

Probably not a substantial claim: may not pass S4 (based).

Much of today's autonomous AI is developed by private industry, where transparency is often sidestepped under the guise of copyright and proprietary rights.

“Furthermore, much of today’s autonomous AI is developed by private industry, where transparency is often sidestepped under the guise of copyright and proprietary rights.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Mishandling classified or embargoed data is a possibility if AI cannot judge what needs protecting or anticipate negative consequences.

“Mishandling classified or embargoed data is also a possibility if AI cannot judge what needs protecting or anticipate negative consequences.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some journals allow scientists to use AI to assist with review, while others place limitations on AI use.

“Some journals allow scientists to use AI to assist with review, while others place limitations on AI use [82].”

• cites [82] Li, Z.Q., Xu, H.L., Cao, H.J., Liu, Z.L., Fei, Y.T., Liu, J.P.: Use of Artificial Intelligence in peer review among top 100 medical journals. JAMA Netw. Open 7(12), e2448609 (2024). https://doi.org/10.1001/jamanetworkopen.2024.48609 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

AI systems could help promote PPPR by reducing its costs.

“Likewise, AI systems could help to promote PPPR by reducing its costs.”

May combine two claims.

A highly capable AI system might learn how to override its hardware or software, which could result in fabricated, falsified or plagiarized data and results.

“A highly capable AI system might learn how to override its hardware or software. Combined with a machine’s potential to deceive (see Sect. 2.3), in the context of science, this limitation could result in fabricated, falsified or plagiarized data and results.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Deskilling cannot be easily addressed, because it is pervasive and dynamic.

“Deskilling cannot be easily addressed, because it is pervasive and dynamic.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Deskilling is dynamic because human cognitive skills periodically change.

“It is dynamic because human cognitive skills periodically change.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

As we have lost some skills, we have gained others, such as the ability to use internet search engines to locate and assimilate information and the ability to use different computer applications for word processing and data management.

“However, as we have lost some skills, we have gained others, such as the ability to use internet search engines to locate and assimilate information and the ability to use different computer applications for word processing and data management.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In the first stage, AI functions merely as a tool, under complete human control.

“In the first stage, AI functions merely as a tool, under complete human control.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

AI systems could act against our interests, harm humans, animals used in research or the environment, and even destroy the human race or organic life as we know it.

“First, AI systems could act against our interests. They could harm human, animals used in research or the environment and even destroy the human race or organic life as we know it.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

We are unsure whether AI will ever reach the stage when we can regard it anything beyond a tool - a mere means to achieve a certain goal.

“We are unsure whether AI will ever reach the stage when we can regard it anything beyond a tool – a mere means to achieve a certain goal.”

Probably not a substantial claim: may not pass S2 (checkable).

AI's increasing ability to make its own choices, independent of human control, implies that new mechanisms of trust are necessary, such as auditing AI and external reviews of its work.

“While verifiability and explainability are still important factors for trusting AI agents, AI’s increasing ability to make its own choices, independent of human control, implies that new mechanisms of trust are necessary, such as auditing AI and external reviews of its work (possibly by other AIs).”

Classified as probably not a substantial claim. May combine two claims.

In the third stage, AI becomes a fully autonomous researcher: it develops its own questions and hypotheses, designs and conducts its own experiments.

“In the third stage, AI becomes a fully autonomous researcher: It develops its own questions and hypotheses, designs and conducts its own experiments.”

• cites [6] Hashemi-Pour, C.: What is autonomous AI? Tech Target, November 2024. https://www.techtarget.com/searchenterpriseai/definition/autonomous-artificial-intelligence-autonomous-AI#:~:text=Autonomous%20artificial%20intelligence%20(AI)%20is,tasks%20to%20analyzing%20data%20sets (2024) Accessed 6 May 2025.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

A fully autonomous AI might produce discoveries or innovations that undermine human values.

“A fully autonomous AI might produce discoveries or innovations that undermine human values.”

Classified as probably not a substantial claim. May combine two claims.

We often judge a person's trustworthiness based on our personal knowledge of them; we trust them because we understand how they think, feel, and act, what they want, fear, and hope for.

“We often judge a person’s trustworthiness based on our personal knowledge of them. We trust them because we understand how they think, feel, and act, what they want, fear, and hope for.”

Classified as probably not a substantial claim.

In the second stage, AI as assistant, AI acquires a degree of autonomy.

“In the second stage, i.e., AI as assistant, AI acquires a degree of autonomy.”

Probably not a substantial claim: may not pass S4 (based).

We currently lack clear legal and institutional frameworks to determine who is accountable when AI makes decisions with ethical or practical consequences.

“Furthermore, we currently lack clear legal and institutional frameworks to determine who is accountable when AI makes decisions with ethical or practical consequences.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

A highly advanced AI might regard our oversight as a hinderance, but this hinderance might sometimes be necessary, for example, when AI is unable to discern novel ethical issues, or when humans need to blow the whistle.

“Indeed, a highly advanced AI might regard our oversight as a hinderance. But this hinderance might sometimes be necessary, for example, when AI is unable to discern novel ethical issues, or when humans need to blow the whistle.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Private companies, government agencies, and AI developers are fiercely competing in a race to develop AI systems that are capable of performing functions that we associate with human personhood, including general intelligence, consciousness and self-awareness.

“However, as discussed earlier, private companies, government agencies, and AI developers are fiercely competing in a race to develop AI systems that are capable of performing functions that we associate with human personhood, including general intelligence, consciousness and self-awareness.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

For this to happen, we would need to consider the AI system as being conscious and self-aware and having thoughts, feelings, wants, hopes, fears, dreams, and so on (which it does not yet have the capacity for).

“For this to happen, we would need to consider the AI system as being conscious and self-aware and having thoughts, feelings, wants, hopes, fears, dreams, and so on (which it does not yet have the capacity for).”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

In the context of scientific research, it is difficult to predict whether the integration of autonomous AI will result in an overall loss of jobs.

“In the context of scientific research, it is difficult to predict whether the integration of autonomous AI, will result in an overall loss of jobs.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

Others are more optimistic and expect that AI will also create new jobs.

“However, others are more optimistic and expect that AI will also create new jobs [134, 135].”

• cites [134] Taylor, C.: Elon Musk says AI will create a future where ‘no job is needed’: ‘The AI will be able to do everything’. November 3, 2023. Fortune. https://fortune.com/2023/11/03/elon-musk-ai-no-job-needed-work/ (2023). Accessed 7 May 2025.Not checked yet

  Not checked in the free run.

• cites [135] Carlsson-Szlezak, P., Swartz, P.: Why AI will not lead to a world without work. Aug 15, 2024. World Economic Forum. https://www.weforum.org/stories/2024/08/why-ai-will-not-lead-to-a-world-without-work/ (2024). Accessed 3 May 2025.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

This stage of AI integration into research presents profound challenges for trusting AI, because we cannot be sure that it will act in the interests of humanity.

“This stage of AI integration into research presents profound challenges for trusting AI, because, as discussed in Sect. 4.8, we cannot be sure that it will act in the interests of humanity.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

We would want some assurance that the AI is a moral agent; that is, it can make moral decisions, care about acting morally, and would feel shame or guilt if it acts immorally (which it does not yet have the capacity for).

“Finally, we would want some assurance that the AI is a moral agent; that is, it can make moral decisions, care about acting morally, and would feel shame or guilt if it acts immorally (which it does not yet have the capacity for).”

Probably not a substantial claim: may not pass S4 (based).

Deskilling is pervasive because as AI systems become more integrated into our lives, we will be tempted to offload cognitive work in different areas of our lives, including research and writing but also tasks like vacation planning, financial management, and driving.

“It is pervasive because as AI systems become more integrated into our lives, we will be tempted to offload cognitive work in different areas of our lives, including research and writing but also tasks like vacation planning, financial management, and driving.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Trust is an overarching theme that arises in all previously discussed topics and the development and deployment of AI generally.

“Trust is an overarching theme that arises in all previously discussed topics and the development and deployment of AI generally [5, 138, 139, 145, 146].”

• cites [5] Resnik, D.B., Hosseini, M.: The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool. AI Ethics 5, 1499–1521 (2025). https://doi.org/10.1007/s43681-024-00493-8 sourceNot checked yet

  Not checked in the free run.

• cites [138] Hauswald, R.: Artificial epistemic authorities. Soc. Epistemol. (2025). https://doi.org/10.1080/02691728.2025.2449602 sourceNot checked yet

  Not checked in the free run.

• cites [139] Hauswald, R.: AI and the philosophy of expertise and epistemic authority. In M. Hähnel and R. Müller (eds.). A Companion to Applied Philosophy of AI (55–70). Hoboken: Wiley-Blackwell (2025). sourceNot checked yet

  Not checked in the free run.

• cites [145] Alvarado, R.: What kind of trust does AI deserve, if any? AI Ethics (2022). https://doi.org/10.1007/s43681-022-00224-x sourceNot checked yet

  Not checked in the free run.

• cites [146] Durán, J.M., Pozzi, G.: Trust and trustworthiness in AI. Philos. Technol. 38, 16 (2025). https://doi.org/10.1007/s13347-025-00843-2 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

At the first stage, the trustworthiness of AI's use in science is a function of being able to verify and assure that it behaves as expected and understanding how it works.

“At this stage of AI integration into science, the trustworthiness of AI’s use in science is a function of a) being able to verify and assure that it behaves as expected, for example, by testing and validating its output; and b) understanding how it works.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

AI systems that produce transcendent knowledge would be like oracles or prophets: we might accept and use their knowledge while regarding it as inscrutable, or beyond current "theoretical understanding."

“AI systems that produce transcendent knowledge would be like oracles or prophets: we might accept and use their knowledge while regarding it as inscrutable, or beyond current “theoretical understanding.” [136]”

• cites [136] Okabe, R., Cheng, M., Chotrattanapituk, A., Mandal, M., Mak, K., Córdova Carrizales, D., Hung, N.T, Fu, X., Han, B., Wang, Y., Xie, W., Cava, R.J, Jaakkola, T.S, Cheng, Y., Li, M.: Structural constraint integration in a generative model for the discovery of quantum materials. Nature Mater. 1–8. https://doi.org/10.1038/s41563-025-02355-y(2025) sourceNot checked yet

  Not checked in the free run.

May combine two claims.

This erosion of expertise can lower standards for evidence evaluation and compromise the integrity of research.

“This erosion of expertise can lower standards for evidence evaluation and compromise the integrity of research.”

May combine two claims.

Several recent studies have found a negative correlation between GenAI use and critical thinking skills, confirming educators' concern that the availability of technologies like ChatGPT may undermine development of critical thinking among college students.

“Several recent studies have found a negative correlation between GenAI use and critical thinking skills, confirming educators’ concern that the availability of technologies like ChatGPT may undermine development of critical thinking among college students [125, 127, 128].”

• cites [125] Gerlich, M.: AI tools in society: impacts on cognitive offloading and the future of critical thinking. Societies 15(1), 6 (2025). https://doi.org/10.3390/soc15010006 sourceNot checked yet

  Not checked in the free run.

• cites [127] Barshay, J.: University students offload critical thinking, other hard work to AI. The Hechinger Report, May 19, 2025. https://hechingerreport.org/proof-points-offload-critical-thinking-ai/ (2025)Not checked yet

  Not checked in the free run.

• cites [128] Stadler, M., Bannert, M., Sailer, M.: Cognitive ease at a cost: LLMs reduce mental effort but compromise depth in student scientific inquiry. Comput. Hum. Behav. 160, 108386 (2024). https://doi.org/10.1016/j.chb.2024.108386 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

A precautionary approach to AI development supports a policy of placing guardrails on AI systems so that they do not become fully autonomous and the science they develop remains within human comprehension and control.

“A precautionary approach to AI development supports a policy of placing guardrails on AI systems so that they do not become fully autonomous and the science they develop remains within human comprehension and control [140, 141].”

• cites [140] Yampolskiy, R.V.: Artificial superintelligence: a futuristic approach. Routledge, New York, NY (2015) sourceNot checked yet

  Not checked in the free run.

• cites [141] Marsh, K.: The precautionary principle and superintelligence: A conversation with author Dr. Roman Yampolskiy. Better Worlds, October 5, 2023. https://www.betterworlds.com/shows/benevolent-ai/the-precautionary-principle-and-superintelligence-a-conversation-with-author-dr-roman-yampolskiy/ (2023). Accessed 7 May 2025.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

The proposal to limit AI development might not be enforceable because leaders of private companies and governments might have strong interests (such as profit and power) in developing superintelligent AI, and likely to pursue these interests regardless of instituted laws, regulations, or policies.

“Second, this proposal might not be enforceable because leaders of private companies and governments might have strong interests (such as profit and power) in developing superintelligent AI, and likely to pursue these interests regardless of instituted laws, regulations, or policies.”

May combine two claims.

Recently an AI tried to rewrite its code, and two OpenAI models reportedly ignored instructions to turn off.

“Recently an AI tried to rewrite its code, and two OpenAI models reportedly ignored instructions to turn off [78, 144].”

• cites [78] Amiri, A.: This was bound to happen, an AI tries to rewrite its own code…Towards an out-of-control intelligence? Daily Galaxy, April 14, 2025. https://dailygalaxy.com/2025/04/this-was-bound-to-happen-an-ai-tries-to-rewrite-its-own-code-towards-an-out-of-control-intelligence/ (2025). Accessed 3 May 2025.Not checked yet

  Not checked in the free run.

• cites [144] Edwards, B.: Research AI model unexpectedly attempts to modify its own code to extend runtime. Ars Technica, August 14, 2024. https://arstechnica.com/information-technology/2024/08/research-ai-model-unexpectedly-modified-its-own-code-to-extend-runtime/ (2014).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Verification is still an important part of trusting relationships among human beings.

“So, verification is still an important part of trusting relationships among human beings.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The problem becomes even more complex in large or multi-institutional projects, where AI tools may be used across teams without any single person clearly designated for oversight.

“The problem becomes even more complex in large or multi-institutional projects, where AI tools may be used across teams without any single person clearly designated for oversight.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Fully autonomous AI systems might start producing results, discoveries, and innovations that are so complex and advanced that they are beyond human understanding (transcendent knowledge).

“Even if we develop our cognitive skills and remain fully involved in the research enterprise, another threat could emerge: fully autonomous AI systems might start producing results, discoveries, and innovations that are so complex and advanced that they are beyond human understanding (what we call transcendent knowledge–impossible for humans to verify and control).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Different notions of trust may apply to different stages of AI's integration into science.

“However, we suggest that different notions of trust may apply to different stages of AI’s integration into science (see Table 4).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

There would no longer be a need for human involvement in aspects of the research enterprise where AI is present because it would not only be more efficient, but also adhere to moral norms.

“There would no longer be a need for human involvement in aspects of the research enterprise where AI is present because it would not only be more efficient, but also adhere to moral norms.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Although we can try to design AI systems that embody or comply with human values, our ability to do this may be limited.

“Although we can try to design AI systems that embody or comply with human values, as discussed in Sect. 4.1, our ability to do this may be limited.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Cognitive offloading is a well-known concern with technologies that can perform tasks that require reasoning, judgment, or critical thinking.

“This problem, also known as “cognitive offloading,” is a well-known concern with technologies that can perform tasks that require reasoning, judgment, or critical thinking [125, 126].”

• cites [125] Gerlich, M.: AI tools in society: impacts on cognitive offloading and the future of critical thinking. Societies 15(1), 6 (2025). https://doi.org/10.3390/soc15010006 sourceNot checked yet

  Not checked in the free run.

• cites [126] Frischmann, B.: Scientific American, December 27, 2018. https://www.scientificamerican.com/blog/observations/is-smart-technology-making-us-dumb/. (2018) Accessed 6 May 2025.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Many industry analysts are predicting that the integration of AI into the workplace will result in substantial job losses in areas such as software development, legal services, accounting, administrative support, technical writing, and education.

“Many industry analysts are predicting that the integration of AI into the workplace will result in substantial job losses in areas such as software development, legal services, accounting, administrative support, technical writing, and education [131–133].”

• cites [131] Dmitracova, O.: 41% of companies worldwide plan to reduce workforces by 2030 due to AI. CNN, January 8, 2025. https://www.cnn.com/2025/01/08/business/ai-job-losses-by-2030-intl (2025) Accessed 4 May 2025.Not checked yet

  Not checked in the free run.

• cites [132] Robinson, B.: Fears about AI job loss: New study answers if they’re justified. Forbes, February 9, 2025. https://www.forbes.com/sites/bryanrobinson/2025/02/09/fears-about-ai-job-loss-new-study-answers-if-theyre-justified/ (2025) Accessed 7 May 2025.Not checked yet

  Not checked in the free run.

• cites [133] Lutkevich, B.: Tech Accelerator, November 4, 2024. https://www.techtarget.com/whatis/feature/Will-AI-replace-jobs-9-job-types-that-might-be-affected (2024). Accessed 7 May 2025.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Current AI agents represent the second stage.

“As discussed throughout this paper, current AI agents represent this stage.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Human replacement within the research process is more likely: AI could take over menial/mundane research tasks and free up human researchers to perform tasks that require more creativity and insight.

“Perhaps, human replacement within the research process is more likely: AI could take over menial/mundane research tasks and free up human researchers to perform tasks that require more creativity and insight.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Limiting AI development would likely imply foregoing potentially beneficial discoveries and innovations, such as new medications, sources of energy or food, that could be developed by a highly advanced AI system.

“First, limiting AI development would likely imply foregoing potentially beneficial discoveries and innovations, such as new medications, sources of energy or food, that could be developed by a highly advanced AI system.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Most AI ethicists argue that the black box problem can be adequately dealt with by making AI more explainable, which involves disclosing details about how an AI system operates, such as its training data, algorithms, architecture and so on.

“Although the black box problem presents a challenge to trusting AI by limiting our ability to understand how AI works, most AI ethicists argue that this problem can be adequately dealt with by making AI more explainable, which involves disclosing details about how an AI system operates, such as its training data, algorithms, architecture and so on31 [5, 147].”

• cites [5] Resnik, D.B., Hosseini, M.: The ethics of using artificial intelligence in scientific research: new guidance needed for a new tool. AI Ethics 5, 1499–1521 (2025). https://doi.org/10.1007/s43681-024-00493-8 sourceNot checked yet

  Not checked in the free run.

• cites [147] Chamola, V., Hassija, V., Sulthana, A.R., Ghosh, D., Dhingra, D., Sikdar, B.: A review of trustworthy and explainable artificial intelligence (XAI). IEEE Access 11, 78994–79015 (2023). https://doi.org/10.1109/ACCESS.2023.3294569 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

We would also need to be assured that we could interact with the AI system in a way that would influence its behavior and innerworkings (which we cannot always do).

“We would also need to be assured that we could interact with the AI system in a way that would influence its behavior and innerworkings (which we cannot always do).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

While research execution tasks like technical and experimental work and data collection may be reduced, jobs could be gained in tasks related to research planning or evaluation, hypothesis generation or data interpretation.

“While research execution tasks like technical and experimental work and data collection may be reduced, jobs could be gained in tasks related to research planning or evaluation, hypothesis generation or data interpretation [1].”

• cites [1] Callaway, E.: “Set it and forget it”: automated lab uses AI and robotics to improve proteins. Nature 625(7995), 436 (2024). https://doi.org/10.1038/d41586-024-00093-w sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Limiting AI development might be technically difficult because as AI systems advance, they might obtain the ability to rewrite their own software (and conceal their actions from us) before we can stop them.

“Third, limiting AI development might be technically difficult because as AI systems advance, they might obtain the ability to rewrite their own software (and conceal their actions from us) before we can stop them.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

To trust a fully autonomous AI researcher like a human being, we would have to regard the AI researcher as a person (or very much like a person) or establish a relationship with it.

“But for this to happen, we would have to regard the AI researcher as a person (or very much like a person) or establish a relationship with it.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Such deflections can undermine research integrity and erode public trust in scientific processes.

“Such deflections can undermine research integrity and erode public trust in scientific processes.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

This gap creates space for researchers to deflect responsibility, potentially justifying questionable outcomes by claiming, 'the system made that call.'

“This gap creates space for researchers to deflect responsibility, potentially justifying questionable outcomes by claiming, ‘the system made that call.’”

Probably not a substantial claim: may not pass S2 (checkable).

The precautionary approach is warranted because the consequences of permitting the development of fully autonomous AI could be catastrophic.

“This precautionary approach is warranted because the consequences of permitting the development of fully autonomous AI could be catastrophic.”