CoChat
Sign Up
Cited by 24Scientific Computing and Data ManagementArtificial Intelligence ApplicationsMachine Learning in Materials Science

Toward Automated Simulation Research Workflow through LLM Prompt Engineering Design

Open access
Liu, Zhihan, Chai, Yubo, Li, Jianfeng·2025·Journal of Chemical Information and Modeling
DOI 10.1021/acs.jcim.4c01653
Open this paper in CoChat — free accountOpen PDF

Abstract

The advent of Large Language Models (LLMs) has created new opportunities for the automation of scientific research spanning both experimental processes and computational simulations. This study explores the feasibility of constructing an autonomous simulation agent (ASA) powered by LLMs through prompt engineering and automated program design to automate the entire simulation research process according to a human-provided research plan. This process includes experimental design, remote upload and simulation execution, data analysis, and report compilation. Using a well-studied simulation problem of polymer chain conformations as a test case, we assessed the long-task completion and reliability of ASAs powered by different LLMs, including GPT-4o, Claude-3.5, etc. Our findings revealed that ASA-GPT-4o achieved near-flawless execution on designated research missions, underscoring the potential of methods like ASA to achieve automation in simulation research processes to enhance research efficiency. The outlined automation can be iteratively performed for up to 20 cycles without human intervention, illustrating the potential of ASA for long-task workflow automation. Additionally, we discussed the intrinsic traits of ASA in managing extensive tasks, focusing on self-validation mechanisms, and the balance between local attention and global oversight.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper develops an autonomous simulation agent (ASA) using LLM prompt engineering and an automated program to execute end-to-end simulation research from a human-written research plan. In tests on polymer chain conformation and gravitational simulation problems, ASA-GPT-4o achieved near-flawless execution on designated missions, and the workflow could run up to 20 cycles without human intervention.

Key findings

  • ASA-GPT-4o achieved near-flawless execution on designated research missions; ASA-GPT-4o and ASA-GPT-4-Turbo excelled on RP 1 with near-perfect completion in simulation and report writing.
  • For RP 2 (remote upload and simulation), completion rates declined; ASA-Claude-3.5 had the highest completion rate, completing the entire mission including all required graphs in over two-thirds of trials, though it occasionally missed displaying some generated graphs in the report. ASA-GPT-4o, ASA-GPT-4-Turbo, and ASA-Qwen-2 also had some success over 20 trials.
  • For RP 3, ASA-GPT-4o, ASA-GPT-4-Turbo, ASA-Claude-3.5, and ASA-Gemini-1.5-Pro performed relatively well, but none provided a correct self-avoiding scaling due to incorrect sampling methods during simulation.
  • In RP 4, ASA-GPT-4-Turbo’s Primary AI executed 20 instances of RP 1, wrote 66 programs (including 26 error versions), conducted more than 20 simulations, generated 120 images, and wrote 20 research reports totaling over 5000 words without human intervention.

Methods

  • ASA uses an API automation program (AutoProg) for multi-round LLM interactions via API and local execution of LLM-written Python; the prompt flow breaks the research plan into subtasks, generates code, debugs errors, checks completion, and enforces a maximum debug attempt limit.
  • Test problems: polymer chain conformation (random walk and self-avoiding walk) with three RPs—RP 1 local random walk; RP 2 modify a provided program and run on a remote server; RP 3 random and self-avoiding walks. Additional gravitational/asteroid RPs S1/S2, RP 4 nested automation of 20 RP 1 runs, and brief-RP ASA were also used.
  • Evaluation: nine ASAs powered by different LLMs, 20 trials per RP 1–3, seven criteria per RP, scored with the Entropy Weight Method (EWM) and TOPSIS.

Limitations

  • The paper states limitations including maintaining a broad overview while focusing on specific details as needed, and information transmission accuracy; scientific conclusions drawn by ASA are not yet fully reliable, and its ability to generate entirely novel knowledge remains limited.
  • The paper reports common failure modes: overlooking mission requirements or taking shortcuts, lack of specialized domain knowledge (e.g., incorrect uniform sphere sampling, incorrect self-avoiding walk sampling), and error persistence/debugging loops.
  • Evident from the text: Manager-Executor mode failed on RP 2 due to precise information transmission; no agent produced correct self-avoiding scaling; results require further validation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1021/acs.jcim.4c01653

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 17 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    ASA-GPT-4o achieved near-flawless execution on designated research missions.

    “Our findings revealed that ASA-GPT-4o achieved near-flawless execution on designated research missions”
    Quote in the paperSubstantial claim
  • Findingnone given

    Some ASAs tend to overlook parts of the RP's requirements or

    “First, some ASAs tend to overlook parts of the RP's requirements or”
    Quote in the paperSubstantial claim
  • Findingown data

    Instructing the Manager AI to include all relevant information in the sub-RPs did not notably enhance the accuracy of the information transmission.

    “We attempted to instruct the Manager AI to include all relevant information in the sub-RPs, but this did not notably enhance the accuracy of the information transmission.”
    Quote in the paperNot a substantial claim
  • Findingown data

    ASA-GPT-4o, ASA-GPT-4-Turbo, and ASA-Qwen-2 demonstrated some success in solving complex missions over 20 trials.

    “ASA-GPT-4o, ASA-GPT-4-Turbo, and ASA-Qwen-2 also demonstrated some success in solving complex missions over 20 trials (Figure 3).”
    Quote in the paperNot a substantial claim
  • Findingown data

    Sub-RPs sometimes contained irrelevant information, potentially confusing the Executor AIs about the mission's focus.

    “Additionally, we observed instances where sub-RPs contained irrelevant information, potentially confusing the Executor AIs about the mission's focus.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 28
  • The quote only partly states this claim. 2
  • Classified as probably not a substantial claim. 29

  • May combine two claims.

    Findingown data

    ASA-Claude-3.5 achieved the highest completion rate on RP 2, successfully completing the entire mission, including generating all required graphs in over two-thirds of trials, though it occasionally missed displaying some generated graphs in the report.

    “ASA-Claude-3.5 achieved the highest completion rate, successfully completing the entire mission, including generating all required graphs in over two-thirds of trials, though it occasionally missed displaying some generated graphs in the report.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Using ASA-GPT-4-Turbo, the Primary AI successfully executed the 20 instances of RP 1, meticulously organizing all files into their respective folders and providing a brief summary after all experiments.

    “Using ASA-GPT-4-Turbo, we conducted tests on RP 4. The results (SI) strikingly demonstrate that the Primary AI successfully executed the 20 instances of RP 1, meticulously organizing all files into their respective folders and providing a brief summary after all experiments.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingreasoning

    ASA can execute all subsequent tasks without human intervention after submitting an RP txt file, including automatically uploading programs to a remote server and performing simulations.

    “simply submitting an RP txt file via the command”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Brief-RP ASA can address three distinct research questions, such as predicting polymer viscosity and investigating natural disaster frequency impact on grassland ecosystem.

    “Using brief-RP ASA, we addressed three distinct research questions (Figures S4-S6), such as predicting the viscosity of polymers based on a given data set using two machine learning methods and investigating the impact of natural disaster frequency on the dynamics of a simple grassland ecosystem.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Other models performed well in plotting graphs on RP 1 but lacked accuracy in establishing the correct physical models and deriving correct scaling values through simulations.

    “Other models performed well in plotting graphs but lacked accuracy in establishing the correct physical models and deriving correct scaling values through simulations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For RP 2, completion rates across ASAs significantly declined, with some agents consistently failing across multiple trials.

    “For RP 2, which involves remote upload and remote simulations, completion rates across ASAs significantly declined, with some agents consistently failing across multiple trials.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    ASA-GPT-4o presented experimental data in enriched formats, such as frequency distribution plots of end-to-end distances and interactive web-based charts.

    “Additionally, ASA-GPT-4o presented experimental data in enriched formats, such as frequency distribution plots of end-to-end distances and interactive web-based charts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For RP 1, the Manager AI effectively decomposes the mission into subtasks and successfully distributes them to Executor AIs, thereby completing RP 1.

    “The results (SI-data-2) revealed that for RP 1, the Manager AI effectively decomposes the mission into subtasks and successfully distributes them to Executor AIs, thereby completing RP 1.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    ASA-Claude-3.5 referenced accurate self-avoiding walk scaling relationships validated through theoretical derivations and experimental verifications, accompanied by brief analyses of simulation result deviations.

    “ASA-Claude-3.5 referenced accurate self-avoiding walk scaling relationships validated through theoretical derivations and experimental verifications, accompanied by brief analyses of simulation result deviations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ASA-Claude-3.5 and ASA-Qwen-2 performed well in simulation tasks on RP 1 but sometimes generated reports in response or as text files, failing to meet RP 1's requirements.

    “ASA-Claude-3.5 and ASA-Qwen-2 followed, performing well in simulation tasks but sometimes generating reports in response or as text files, failing to meet RP 1's requirements.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    ASA-GPT-4o, ASA-GPT-4-Turbo, ASA-Claude-3.5, and ASA-Gemini-1.5-Pro performed relatively well on RP 3.

    “ASA-GPT-4o, ASA-GPT-4-Turbo, ASA-Claude-3.5, and ASA-Gemini-1.5-Properformed relatively well.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Agents such as ASA-GPT-4o demonstrated high completion rates, a strong grasp of task requirements, and consistent alignment with the overall objectives across extensive tasks.

    “Agents such as ASA-GPT-4o demonstrated high completion rates, a strong grasp of task requirements, and consistent alignment with the overall objectives across extensive tasks.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    Securing that specific information is correctly interpreted and utilized remains problematic.

    “Securing that specific information is correctly interpreted and utilized remains problematic.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The performance exhibited in completing RP 4, which entailed a hierarchy of nested tasks, surpassed expectations in a significant manner.

    “Notably, the performance exhibited in completing RP 4, which entailed a hierarchy of nested tasks, surpassed expectations in a significant manner.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    Ensuring that ASA maintains a broad overview while focusing on specific details as needed poses a challenge.

    “( 1 Ensuring that ASA maintains a broad overview while focusing on specific details as needed poses a challenge;”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Achieving successful and precise mission completion requires a delicate balance between global oversight and local attention, two normally conflicting perspectives.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given

    When simulating a random walk, ASAs can often generate spatially random unit vectors but fail to ensure uniform distribution on the sphere.

    “Although ASAs can often generate spatially random unit vectors, they fail to ensure uniform distribution on the sphere.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ASA can solve gravitational simulation problem RP S2, automatically acquiring solar system parameters, coding trajectory simulation, and writing report.

    “By simply inputting RP S2 into the same ASA used for chain conformation problems, the ASA will automatically acquire the necessary parameters of the solar system, code the simulation program to calculate the asteroid's trajectory, and ultimately write the report.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    By submitting a research plan file using the command do prj1.txt, ASA will automatically execute all subsequent simulation tasks without any human intervention.

    “By submitting a research plan file using the command do prj1.txt,ASA will automatically execute all subsequent simulation tasks without any human intervention.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Findingnone given

    ASAs may overlook mission requirements, necessitating methods to prioritize critical information.

    “Mission Requirement Overlooking.ASAs may overlook mission requirements, necessitating methods to prioritize critical information”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given

    General LLMs lack specialized domain knowledge, requiring supplementary information in RPs.

    “General LLMs lack specialized domain knowledge, requiring supplementary information in RPs”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ASA occasionally skips steps in prolonged missions or fails to deliver comprehensive solutions for subtasks, indicating a lack of sufficient local attention.

    “However, as discussed above, ASA occasionally skips steps in prolonged missions or fails to deliver comprehensive solutions for subtasks, indicating a lack of sufficient local attention.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Multitier RP designs effectively handle more complex missions, though accurate information transmission remains challenging for such RPs.

    “Multitier RP designs effectively handle more complex missions, though accurate information transmission remains challenging for such RPs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    ASA, through prompt engineering to normalize LLM responses, is reliable when logic is well-designed.

    “ASA, through prompt engineering to normalize LLM responses, although it restricts the LLM's freedom, is reliable when logic is well-designed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Ensuring key outputs at each subtask step maintains coherence and facilitates continuation of subsequent tasks.

    “To maintain coherence, prompt the LLM to consider previous outputs, generate key results for each step and input key results into the proceeding round of dialogue, facilitating the continuation of subsequent tasks and allowing the agent to monitor overall progress.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    AI powered by LLMs can exhibit a degree of creativity and autonomy in simulation-based research, and ASA can be effectively applied to a wide range of research problems.

    “These examples illustrate that AI powered by large language models (LLMs) can exhibit a degree of creativity and autonomy in simulation-based research and that our ASA can be effectively applied to a wide range of research problems.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    By effectively structuring logic, ASA can oversee the LLM's output, enhancing the smoothness and reliability of the automated simulation research process.

    “By effectively structuring logic, ASA can oversee the LLM's output, enhancing the smoothness and reliability of the automated simulation research process (Figure 4).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Incorporating agent-led code review and debugging allows ASA to generate correct and task-compliant code within several dialogue rounds.

    “By integrating selfreview and debugging processes, ASA can generate correct and task-compliant code within several dialogue rounds.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    These issues affect mission completeness and can disrupt the entire workflow.

    “These issues affect mission completeness and can disrupt the entire workflow.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Despite varying completion rates among tested ASAs, their errors and failure points share common characteristics, reflecting the inherent logical features of the LLMs.

    “Despite varying completion rates among tested ASAs, their errors and failure points share common characteristics, reflecting the inherent logical features of the LLMs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    ASA achieves an autonomous research process guided by human-provided RPs through continuous interaction with the LLM according to a specific logical design.

    “ASA achieves an autonomous research process guided by humanprovided RPs through continuous interaction with the LLM according to a specific logical design.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Structuring agent responses for efficient code and task extraction improves code execution success rates and ensures smooth mission progression.

    “Prompt the LLM to produce structured outputs, enabling ASA to accurately extract Python code, subtasks, and progress indicators.This improves code execution success rates and ensures smooth mission progression.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    ASA's ability to generate entirely novel knowledge remains limited, and the reliability of its results requires further validation.

    “its ability to generate entirely novel knowledge remains limited, and the reliability of its results requires further validation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). The quote only partly states this claim.

    Interpretationreasoning

    Providing detailed information for complex operations like remote server connections improves mission success rates.

    “Providing sufficient information in RP can improve mission success rates.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    While LLMs can adapt responses based on short dialogue histories, this capability can also lead to persistent errors.

    “While this capability allows LLMs to adapt responses based on short dialogue histories, it can also lead to persistent errors.”
    • Also stated in: § ASA Scoring via EWM and TOPSIS · hedged
      “LLMs may persist in errors by relying too heavily on previous content, requiring memory management strategies in ASAs' development”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The success herein can be largely attributed to prompt engineering and automated program design.

    “The success herein can be largely attributed to prompt engineering and automated program design.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    These capabilities underscore the readiness of ASA to conduct long-term research missions.

    “These capabilities underscore the readiness of ASA to conduct long-term research missions”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The polymer physics and celestial simulation problems used are representative examples of simulation research tasks.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The comprehensive simulation research process, encompassing simulation coding, remote execution, data analysis, and report generation, represents an autonomous AI-conducted simulation research process.

    “This comprehensive simulation research process, encompassing simulation coding, remote execution, data analysis, and report generation, represents an autonomous AI-conducted simulation research process.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The Manager-Executor AI collaboration plan minimized redundant information in different AIs' dialogue history, enhancing the Manager AI's global awareness and the Executor AIs' local attention.

    “This approach minimized redundant information in different AIs' dialogue history, enhancing the Manager AI's global awareness and the Executor AIs' local attention.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Achieving successful and precise mission completion requires a delicate balance between global oversight and local attention, two normally conflicting perspectives.

  • May combine two claims.

    Premisecitation

    LLMs can generate corresponding code based on text descriptions, significantly simplifying programming tasks.

    “LLMs can also generate corresponding code based on text descriptions, significantly simplifying programming tasks.14,15”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [14] Xu, F. F.; Alon, U.; Neubig, G.; Hellendoorn, V. J. In A Systematic Evaluation of Large Language Models of Code, Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming; ACM Digital Library: New York, NY, USA, 2022; pp 1–10. source

      Not checked in the free run.

    • cites [15] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    LLMs excel in handling complex logic, code completion, and debugging, making them valuable assistants in software development and data analysis.

    “Research shows that LLMs excel in handling complex logic, code completion, and debugging, making them valuable assistants in software development and data analysis. [16][17]7][18][19][20]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16][17]7][18][19][20]

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    TOPSIS computes the geometric distance between each agent's performance and both ideal and anti-ideal solutions, facilitating comparative assessment of ASA efficacy.

    “TOPSIS, originally formulated by Hwang and Yoon, 58 was subsequently applied.This method computes the geometric distance between each agent's performance and both ideal and anti-ideal solutions, facilitating a comparative assessment of ASA efficacy.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [58] Hwang C. L., 1981 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The scaling relationship between mean square end-to-end distance and number of chain segments N has been theoretically derived by Flory, De Gennes, and others and validated through simulations and experiments.

    “This scaling relationship has been theoretically derived by Flory, De Gennes, and others 47,48 and validated through simulations and experiments.49,50”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [47] Flory P. J., 1969 source

      Not checked in the free run.

    • cites [48] De Gennes P. G., 1979

      Not checked in the free run.

    • cites [49] Madras N., 2013 source

      Not checked in the free run.

    • cites [50] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The ASA approach has potential to assist in exploring new questions.

    “and its potential to assist in exploring new questions.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Prompt engineering techniques such as Chain of Thought, ReAct, and Tree of Thoughts help structure the model's reasoning process, reduce errors, and improve the quality of generated results.

    “Techniques such as Chain of Thought, ReAct, and Tree of Thoughts help structure the model's reasoning process, reduce errors, and improve the quality of generated results.28,[41][42][43][44][45][46]”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites 28,[41][42][43][44][45][46]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Correct sampling for self-avoiding walks should use importance sampling to generate statistically significant conformations.

    “Correct sampling should use importance sampling to generate statistically significant conformations.54”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [54] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The scientific conclusions drawn by ASA are not yet fully reliable due to current limitations of LLMs.

    “Despite the current limitations of LLMs, which mean that the scientific conclusions drawn by ASA are not yet fully reliable”
    Quote in the paperSubstantial claim
46 of 46

  • The paper relies on this without support.

    After the research plan (RP) is determined, the subsequent steps of simulation research are largely standardized and present opportunities for automation.

    “After the RP is determined, the subsequent steps are largely standardized and present opportunities for automation.”

  • The paper relies on this without support.

    LLMs cannot guarantee the accuracy of generated code in a single attempt.

    “they cannot guarantee the accuracy of the generated code in a single attempt”

  • The paper relies on this without support.

    Due to the length and complexity of tasks described in the RP, LLMs may overlook subtasks.

    “due to the length and complexity of tasks described in the RP, LLMs may overlook subtasks”

  • The paper relies on this without support.

    LLMs do not have the ability to control local computers to execute programs.

    “as language models, LLMs do not have the ability to control local computers to execute programs”

  • The paper relies on this without support.

    This study represents the first exploration of an AI-driven, end-to-end simulation research process guided by an RP provided by humans.

    “this study represents the first exploration of an AIdriven, end-to-end simulation research process guided by an RP provided by humans”

  • The paper relies on this without support.

    The current model of studies on AI in scientific research mainly involves interaction between humans and AI.

    “the current model of these studies mainly involves interaction between humans and AI”

  • The paper relies on this without support.

    The mission of RP S2 is challenging even for graduates of physics departments.

    “This mission is challenging even for graduates of physics departments.”

  • The paper relies on this without support.

    The algorithm for simulating a random walk needs to generate unit vectors uniformly distributed on a sphere to ensure isotropy.

    “when simulating a random walk, the algorithm needs to generate unit vectors uniformly distributed on a sphere to ensure isotropy”

  • The paper relies on this without support.

    LLMs have strong generative capabilities but unpredictable outputs can complicate automation.

    “while the LLM has strong generative capabilities, its unpredictable outputs can complicate automation”

  • The paper relies on this without support.

    Achieving successful and precise mission completion requires a delicate balance between global oversight and local attention, two normally conflicting perspectives.

    “Achieving successful and precise mission completion requires a delicate balance between global oversight and local attention, two perspectives that are normally conflicting.”

  • The paper relies on this without support.

    The polymer physics and celestial simulation problems used are representative examples of simulation research tasks.

    “We used a well-recognized polymer physics simulation problem and a more challenging celestial simulation problem as representative examples.”

  • The paper relies on this without support.

    The criteria devised for RP 1-3 (diagram generation, report composition, etc.) adequately measure mission completion.

    “To evaluate the performance of ASAs on designated RPs, we devised specific criteria for RP 1-3, including diagram generation, report composition, etc.”
12 of 12

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A third group relied on more compact planner-executor or single-agent loops that repeatedly called tools and updated decisions from the observed outputs.

    “A third group relied on more compact planner-executor or single-agent loops that repeatedly called tools and updated decisions from the observed outputs [24,28,29,32-34,39,40].”

    From From Scientific Copilots to Tool-Grounded Autonomy: AI Agents in Simulation-Driven Materials Discovery (2026) · cites this paper as [24]

  • The remaining literature was distributed across general materials-workflow papers, atomistic simulation infrastructures, MOF and porous-material screening, and polymer simulation.

    “The remaining literature was distributed across general materials-workflow papers, atomistic simulation infrastructures, MOF and porous-material screening, and polymer simulation [17,21,22,24-26,32,35-42].”

    From From Scientific Copilots to Tool-Grounded Autonomy: AI Agents in Simulation-Driven Materials Discovery (2026) · cites this paper as [24]

  • In other papers, the workflow could be reconstructed only partially from supporting information, and prompt logs, failure traces, or decision histories remained incomplete.

    “In others, the workflow could be reconstructed only partially from supporting information, and prompt logs, failure traces, or decision histories remained incomplete [17,22,24,26,27,29,33,34].”

    From From Scientific Copilots to Tool-Grounded Autonomy: AI Agents in Simulation-Driven Materials Discovery (2026) · cites this paper as [24]

    The Additional information section lists supporting materials, including partial simulation results and AutoProg examples, but does not assess whether the workflow can be reconstructed from them. The Common Issues in Sequential Task and Error Loops in Debugging sections discuss agent failures, not incomplete prompt logs, failure traces, or decision histories. No page numbers are available in our copy. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Self-avoiding-walk sampling cites an LLM reference

    Minor

    The sampling claim cites reference 54, but reference 54 is titled "Language Models are Few-Shot Learners," not a work on self-avoiding-walk sampling.

  • TOPSIS citation points to a missing reference

    Minor

    The text cites Hwang and Yoon as reference 58, but the reference list ends at 57 and lists Hwang and Yoon under 57.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 8

  1. Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. In BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, 2019; pp 4171–4186.
  2. Raffel C., 2020, J. Mach. Learn. Res., 21, 1
  3. Chowdhery A., 2023, J. Mach. Learn. Res., 24, 1
  4. Ramesh, A.; Pavlov, M.; Goh, G.; Gray, S.; Voss, C.; Radford, A.; Chen, M.; Sutskever, I. In Zero-shot Text-to-image Generation, International Conference on Machine Learning, 2021; pp 8821–8831.
  5. Lozano A., 2024, Pac. Symp. Biocomput., 29, 8
  6. Yao S., 2023, Advances in Neural Information Processing Systems, 11809
  7. Wei, J.; Bosma, M.; Zhao, V. Y.; Guu, K.; Yu, A. W.; Lester, B.; Du, N.; Dai, A. M.; Le, Q. V. Finetuned Language Models are Zero-Shot Learners, The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25–29, 2022.
  8. Ravindran A., 1987, Operations Research Principles & Practice

Cited by

Sorted by citations

Related

Authors
Liu, Z., Chai, Y., & Li, J.
Affiliations
Fudan University (CN)
Corresponding author
Li, Jianfeng
ORCID
Venue
Journal of Chemical Information and Modeling
Publisher
American Chemical Society (ACS)
Published
2024-12-30
Volume
65
Issue
1
Pages
114-124
ISSN
  • 1549-9596 (ISSN-L)
  • 1549-960X
DOI
10.1021/acs.jcim.4c01653
OpenAlex
W4405922624
Semantic Scholar
3cf20d97cf0f9f7ea1ed87184f455a0e46e4cc2f
Cited by
24
References
50
Citations by year
2026: 16 · 2025: 8
FWCI
8.94
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Chemical Information and Modeling (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
  • National Key Research and Development Program of China (2023YFA0915300)
  • Scientific and Innovative Action Plan of Shanghai (24JD1400700)
  • National Natural Science Foundation of China (22373022, 52394272)
Topics
  • Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
  • Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
Keywords
large language models, prompt engineering, simulation workflow automation, chain conformation, GPT-4o, self-validation, report generation
MeSH terms
Automation, Computer Simulation, Humans, Programming Languages, Workflow

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review