Generative AI for self-adaptive systems: State of the art and research roadmap

May combine two claims.

The search results comprised 3 pieces from self-adaptive systems, 302 from software engineering, 5,061 from AI, 245 from HCI, and 228 from robotics.

“The breakdown is as follows: 3 from SAS [Li et al., 2024c;Nascimento et al., 2023;Sarda, 2023], 302 from SE, 5061 from AI, 245 from HCI, and 228 from Robotics.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In the generative AI relevance screening, 1,401 pieces were excluded, leaving 4,473 articles for further analysis.

“In this step, we excluded 1,401 pieces of literature, leaving a total of 4,473 articles remaining for the next analysis.”

May combine two claims.

Generative AI research is abundant in AI and software engineering but scarce in leading self-adaptive systems conferences and journals such as SEAMS, ACSOS, and TAAS.

“while GenAI-related research is abundant in fields such as AI and Software Engineering (SE), it is notably limited in leading conferences and journals on SASs like SEAMS, ACSOS, and TAAS.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The literature search returned 5,874 pieces of literature.

“As a result of our literature search, we obtained a total of 5,874 pieces of literature.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The final literature selection yielded 219 pieces of literature.

“As a result, we finally obtained 219 pieces of literature.”

Probably not a substantial claim: may not pass S4 (based).

To the authors' knowledge, no research directly uses LLMs to augment control theory or its direct applications.

“However, to date, to the best of our knowledge, there is currently no research on using LLMs to directly augment control theory or its direct applications.”

May combine two claims.

LLM use in swarm intelligence remains at an early stage; the review found only Pluhacek et al. (2023).

“The integration of LLMs into swarm intelligence is still nascent, with [Pluhacek et al., 2023] being the only study we found in our review.”

• cites Pluhacek et al., 2023Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A single publication can belong to more than one category.

“One piece of literature may be involved in multiple categories.”

May combine two claims.

Transformers have remained a major research area in AI and robotics, and language model use across fields grew quickly between 2023 and 2024.

“We can observe that Transformer has consistently been a prominent research topic in AI and robotics, while the application of language models in each research field shows rapid growth from 2023 to 2024.”

Classified as probably not a substantial claim. May combine two claims.

Large language models exhibit notable performance in understanding semantic meaning in text and in performing logical reasoning with text.

“Trained on extensive corpora of human language data, LLMs “naturally” exhibit remarkable performance in (i) understanding the semantic meaning within textual data and (ii) performing logical reasoning using text.”

May combine two claims.

Involving humans in the decision-making loop may provide considerable benefits and improve trustworthiness.

“integrating humans into the decision-making loop may offer considerable benefits [Cámara et al., 2015] and enhance trustworthiness [Calinescu et al., 2018;Weyns et al., 2023].”

• cites Cámara et al., 2015Not checked yet

  Not checked in the free run.

• cites Calinescu et al., 2018Not checked yet

  Not checked in the free run.

• cites Weyns et al., 2023Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The main drawback of diffusion models is their computational speed and cost, because they typically require hundreds of denoising iterations.

“The primary drawback of diffusion models is their computational speed and cost, as these models typically necessitate hundreds of iterations in the denoising process.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Self-adaptive systems were originally designed to operate automatically with minimal human intervention.

“Although SASs were initially designed to operate automatically with minimal human intervention [Kephart and Chess, 2003]”

• cites Kephart and Chess, 2003Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The Transformer architecture is better than its predecessors at handling long-range dependencies in text.

“Unlike its predecessors (RNNs), the Transformer excels in managing long-range dependencies within texts.”

• cites Vaswani et al., 2017Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

The study organizes potential benefits into two categories: enhancements to SAS autonomy through MAPE-K functions and improvements in human-SAS interaction in human-on-the-loop settings.

“we gather, filter, and analyze literature from four distinct research fields and organize them into two main categories to potential benefits: (i) enhancements to the autonomy of SASs centered around the specific functions of the MAPE-K feedback loop, and (ii) improvements in the interaction between humans and SASs within human-on-the-loop settings.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Two authors independently reviewed each article, and a third author resolved conflicts.

“To confirm the relevance, two authors independently reviewed each article, and a third author was involved to discuss and resolve conflicts.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Using Transformers to handle long-range dependencies and LLMs to reason can improve an SAS's awareness of itself and its context.

“Leveraging the Transformer's ability to handle long-range dependencies in data, as well as the reasoning capabilities of LLMs, has the potential to enhance self-and context-awareness of an SAS significantly.”

• AssumptionRests on an assumption: LLMs have reasoning capabilities and Transformers can handle long-range dependencies in data.

May combine two claims.

Prompt strategies for online log analysis achieve an average F1 score of 0.797 for log parsing and 0.412 for anomaly detection.

“The effectiveness of these strategies was evaluated through their performance in log parsing, achieving an average F1 score of 0.797, and in anomaly detection, with an average F1 score of 0.412.”

• cites Liu et al., 2024dNot checked yet

  Not checked in the free run.

May combine two claims.

Diffusion models have been extended to time series analysis and show significant performance in forecasting and imputation.

“The concept of “diffusion” has effectively extended to time series analysis, demonstrating significant performance in time series forecasting and imputation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Eighteen articles (5 from RE and 13 from ICRA) were unavailable on the platforms used and were excluded from the review.

“In total 18 articles (5 from RE and 13 from ICRA) were not available on these platforms and were not included in our review.”

May combine two claims.

Recent LLM developments mirror architecture-based adaptation principles and can be viewed as chain-of-thought with external tool calls.

“Recent developments in LLMs reflect similar principles, which can be considered as CoTs with external tool calls [Inaba et al., 2023].”

• cites Inaba et al., 2023Not checked yet

  Not checked in the free run.

May combine two claims.

LLM-based log parsing studies report an average accuracy of about 95%, which is higher than state-of-the-art parsers.

“The results in the above studies show a notable 95% average accuracy, significantly higher than state-of-the-art parsers.”

May combine two claims.

LLMs can automate reward function design and achieve higher performance and faster convergence than expert-designed reward functions.

“LLMs can automate the design of reward functions, demonstrating higher performance and faster convergence speed than expert-designed reward function [Kwon et al., 2023;Sun et al., 2024d;Xie et al., 2024;Yu et al., 2023a]”

• cites Kwon et al., 2023Not checked yet

  Not checked in the free run.

• cites Sun et al., 2024dNot checked yet

  Not checked in the free run.

• cites Xie et al., 2024Not checked yet

  Not checked in the free run.

May combine two claims.

Integrating LLMs into ML model construction can reduce manual effort and potentially improve model performance.

“Integrating LLMs into the ML model construction process can not only reduce the manual effort required in ML model construction but could also potentially improve the model's performance.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The effectiveness of LLMs within traditional game theory based on mathematical definitions is not yet fully established.

“Additionally, while traditional game theory still relies on mathematical definitions, the efficacy of LLMs within this conventional framework has yet to be fully ascertained.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Targeted prompt engineering that includes explicit desire and belief information could substantially improve LLM rationality.

“Enhancing LLMs' performance through targeted prompt engineering, such as incorporating explicit desire and belief information, could significantly improve their rationality.”

May combine two claims.

LLMs allow game-theoretic reasoning to be expressed in ordinary language rather than mathematical definitions, which widens its use to domains like social simulation.

“Leveraging the natural language and understanding capabilities of LLMs, game theory can now be “realized” directly through natural language instead of mathematical definitions, broadening its application scope to include areas like social simulation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

These planning studies can be seen as a fresh form of RL, where pre-training a Transformer substitutes for conventional value-function fitting or policy-gradient calculation.

“It is important to note that these studies can also be viewed as a new realization of RL, where Transformer pre-training is employed to replace traditional methods of fitting value functions or computing policy gradients.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

Guo et al. (2024b) reported that LLM agents naturally perform human-like actions, for example telling a leader agent about task progress.

“(c) agents spontaneously engage in activities that mimic human behaviors, such as reporting task progress to the leader agent.”

• cites Guo et al., 2024bNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Collective intelligence opens decision-making paths that could fit decentralized SASs especially well.

“This paradigm offers new decision-making avenues, which may be particularly suitable for decentralized SASs [Weyns et al., 2013].”

• cites Weyns et al., 2013Not checked yet

  Not checked in the free run.

May combine two claims.

When episodes fail, LLMs or people can diagnose why, reflect on the reasons, and feed those lessons into later planning.

“For failed experiences, LLMs or human analyses can identify the causes of failures, reflecting on these insights and integrating them into future planning cycles.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

LLMs' reasoning and broad knowledge make them potentially powerful generalized planners.

“Given the above background, LLMs' reasoning capabilities and broad knowledge further position them as potentially powerful, generalized planners.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Guo et al. (2024b) reported that LLM agents are able to choose a leader and change leadership as they communicate.

“(b) agents can elect their own leader and dynamically adjust leadership via communication;”

• cites Guo et al., 2024bNot checked yet

  Not checked in the free run.

May combine two claims.

MetaMorph trained Universal Controllers for exponentially morphable modular robots, demonstrating Transformer combinatorial generalization.

“MetaMorph [Gupta et al., 2022] trained Universal Controllers for exponentially morphable modular robots, demonstrating the Transformer's combinatorial generalization capabilities.”

• cites Gupta et al., 2022Not checked yet

  Not checked in the free run.

May combine two claims.

Zhang et al. (2023c) integrated Actor-Critic from RL into LLM multi-agent crowdsourcing, showing potential to reduce hallucinations and token usage costs.

“Zhang et al. [2023c] integrates the Actor-Critic concept from RL into LLM multi-agent crowdsourcing, highlighting its potential to cut hallucinations and reduce token usage costs.”

• cites Zhang et al., 2023cNot checked yet

  Not checked in the free run.

May combine two claims.

Decision Transformer uses a Transformer to predict optimal actions from returns, prior states, and prior actions, and it beat the best model-free offline RL methods at the time.

“It aligns with RL and trains a Transformer to output optimal actions based on expected returns (rewards), past states, and actions, achieving performance that surpassed the then state-of-the-art model-free offline RL methods.”

• cites Chen et al., 2021aNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Multi-Game DT, after training on broad varied data, reaches near-human play in as many as 46 Atari games.

“Multi-Game DT is trained on large, diverse datasets, enabling near-human performance in up to 46 Atari games.”

May combine two claims.

TIMAT pulls out temporal cues and treats multi-agent RL as a sequence model, which lets it plan for any number of agents.

“TIMAT [Kang et al., 2024] extracts temporal information and models Multi-Agent RL (MARL) as a sequential model, its advantage is its ability to plan for an arbitrary number of agents.”

• cites Kang et al., 2024Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Guo et al. (2024b) reported that smaller teams of LLM agents can be more efficient while spending less on communication.

“Guo et al. [2024b] evaluates the impact of designated leadership in LLM-agent organizations, demonstrating some interesting results include (a) in small teams, higher efficiency can be achieved with less communication cost;”

• cites Guo et al., 2024bNot checked yet

  Not checked in the free run.

May combine two claims.

Diffusion methods have seen wide use for motion planning in robotic arms and four-legged robots, and for continuous constraint solving.

“Subsequently, diffusion has been extensively applied in motion planning for robotic arms [Mishra and Chen, 2023;Pearce et al., 2023;Ze et al., 2024] and quadruped robots [Liu et al., 2024c], as well as continuous constraint solvers [Yang et al., 2023c].”

• cites Mishra and Chen, 2023Not checked yet

  Not checked in the free run.

• cites Pearce et al., 2023Not checked yet

  Not checked in the free run.

• cites Ze et al., 2024Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

When guarantees under uncertainty or control-based adaptation depend on rigorous mathematics, LLM translation may lower the cost of building models.

“For providing guarantees under uncertainty and control-based adaptation that rely on strict mathematical frameworks, LLMs' translation capabilities may have the potential to reduce the modeling costs associated with using these methods.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Diffusion models offer a planning scheme suited to high-dimensional problems with complex constraints.

“diffusion models provide a planning diagram tailored for high-dimensional and complex constraints.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

In learning for analysis and planning, LLMs and diffusion models might supply prior knowledge, which could boost model quality and cut training or planning expense.

“For the application of learning in analysis and planning, LLMs and Diffusion models could support generating prior knowledge, enhancing model performance and reducing training/planning costs.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Planning with Transformers is especially strong for offline RL and scaling, so it may fit offline learning and large adaptive systems.

“Transformer-based planning methods have strong advantages in offline RL and scalability, potentially suitable for offline learning (i.e., inability to interact with the real environment during training) and large-scale adaptive systems;”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Experience accumulation mirrors reflection after failure and evolution after success, offering lessons for lifelong learning and self-evolution in SASs.

“Experience accumulation shows a paradigm similar to self-reflection (for failed experiences) and self-evolution (for successful experiences), which can inform lifelong learning and self-evolution for SASs;”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

For adaptation driven by architecture or requirements, LLMs may help reason over unstructured or natural-language inputs, widening where they can be used.

“In architecture-based adaptation and requirement-driven adaptation, LLMs have potential to support reasoning based on natural language or unstructured data, potentially broadening their application scope.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Robotics work shows MLLMs can carry out provided plans even when the environment is uncertain.

“Additionally, studies in robotics still demonstrate the capability of MLLMs to successfully execute given plans in uncertain environments.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

LLMs' strong language comprehension could greatly improve human-on-the-loop setups in SASs.

“The advanced language understanding capabilities of LLMs undoubtedly offer significant potential to enhance HOTL configurations within SASs.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Transparency matters because it helps users grasp what the system does and builds trust.

“The second category focuses on transparency, which is critical for helping users understand system behavior and thereby enhancing trust.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Such models usually need extra tuning to match real situations, via human edits or LLM corrections driven by interaction feedback.

“However, these models often require further alignment with real-world scenarios, through manual adjustments or LLM-based corrections based on feedback from actual interactions.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

When the executor has to translate plans in complex cases, Transformers and LLMs may enable end-to-end conversion.

“Yet, for more complex cases where the executor needs to convert plans Transformers and LLMs have potential to support end-to-end transformation.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

For knowledge and runtime models, LLMs may bring two main advantages: building models from built-in knowledge and translating descriptions into DSML.

“LLMs offer two primary potential benefits in the realm of knowledge and runtime models.”

May combine two claims.

Gao et al. (2024) adapted an MLLM to grasp object properties such as material and fragility, raising execution success.

“For example, Gao et al. [2024] fine-tuned an MLLM to understand the physical properties of objects (e.g., material, fragility) to improve the success rate of execution.”

• cites Gao et al., 2024Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Because execution in SASs is frequently simple, LLMs bring only modest gains there.

“Considering that the implementation of execution in SASs is often straightforward, LLMs offer limited benefits.”

Classified as probably not a substantial claim.

To adapt in a more user-centered way, systems must capture users' changing preferences accurately while running.

“Accurately capturing users' dynamic preferences during operation is necessary for achieving better user-centered adaptation.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLM translation can turn natural-language or other descriptions into DSML, greatly lowering the cost of manual modeling.

“The second benefit involves the use of LLMs' translation capabilities to convert descriptions in natural language or other formats into DSML, thereby significantly reducing the costs associated with manual modeling.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Collaboration mainly means using human skill to fix system mistakes or joining human and system abilities for harder goals.

“The final category is collaboration, which is primarily about leveraging human expertise to correct system errors or combining the strengths of both humans and systems to achieve more complex goals.”

May combine two claims.

LLM-created personas are nearly impossible to tell apart from human-written ones and are rated similarly in quality and acceptance.

“An 11-participant user study [Schuller et al., 2024] has verified that personas generated by LLMs are virtually indistinguishable from those written by humans, exhibiting comparable quality and acceptance.”

• cites Schuller et al., 2024Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Gathering preferences is key to better user experience and tailored behavior, which raises satisfaction and trust.

“This process is essential for improving user experience and personalizing system behavior, thereby enhancing user satisfaction and trust.”

May combine two claims.

Work on explaining code and decision models is still early: code explanations usually address static features rather than runtime behavior, and decision-model explanations often rely on LLMs to explain the models directly.

“However, the exploration of explaining code and decision models remains preliminary; the former typically involves only static aspects rather than dynamic behaviors of the code, and the latter often uses LLMs to directly explain decision-making models.”

May combine two claims.

Liu et al. (2024d) reported that log explanations were rated highly for usefulness and readability by six experts with more than a decade of experience (mean 4.42/5), even though detection accuracy needed work (mean F1 0.412).

“Despite needing improvements in detection accuracy (F1 Ave = 0.412), the explanations received high ratings for usefulness and readability from experts (six experts with over ten years of work experience, Ave = 4.42/5).”

• cites Liu et al., 2024dNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

It is important to explain how decisions are made, particularly when opaque grey- or black-box modules make them.

“Explaining decision-making processes is essential, especially when they are executed by opaque, grey-or black-box modules.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

How explanations are visualized and interacted with matters greatly for making them easy to grasp.

“Beyond explanatory content, the methods of visualization and interaction are critical for ensuring explanations are easily understood.”

May combine two claims.

PlanCollabNL applies LLMs to estimate task costs from user language and encodes them as PDDL formulas.

“PlanCollabNL [Izquierdo et al., 2024], addressing human-robot collaboration, uses LLMs to infer the cost associated with specific user tasks from natural language inputs, such as “I have back pain today.” It translates such information as formulas in the PDDL, involving definitions of operation objects, (human) agents, and the numeric cost.”

• cites Izquierdo et al., 2024Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Describing what code does is one straightforward way to make a system more transparent.

“Explaining how a piece of code functions is a direct method to enhance system transparency.”

Classified as probably not a substantial claim.

How to handle clashes among multiple needs and preferences, such as cost versus efficiency, is still little studied in self-adaptation.

“However, potential conflicts between different needs and preferences in multi-objective settings, such as the tradeoff between non-functional properties like cost and efficiency, which are core to self-adaptation, is still lacking and needs further exploration.”

May combine two claims.

LLMs are common in data-visualization automation and may help build and adjust dashboards at runtime.

“Additionally, LLMs are widely used in the automation of data visualization, potentially supporting the automatic construction and runtime adjustment of dashboards [Arawjo et al., 2024;Chung et al., 2022;Ko et al., 2024;Reif et al., 2024].”

• cites Arawjo et al., 2024Not checked yet

  Not checked in the free run.

• cites Chung et al., 2022Not checked yet

  Not checked in the free run.

• cites Ko et al., 2024Not checked yet

  Not checked in the free run.

May combine two claims.

Transparency helps users follow how adaptive systems decide and spot mistakes in those decisions.

“This transparency is crucial as it allows users to better understand the decision-making process of adaptive systems, thereby enabling them to effectively identify errors in system decisions [Li et al., 2020b;Parra-Ullauri et al., 2022].”

• cites Li et al., 2020bNot checked yet

  Not checked in the free run.

• cites Parra-Ullauri et al., 2022Not checked yet

  Not checked in the free run.

May combine two claims.

LLMs show promise for explaining code, decision models, and logs, and for making visualizations clearer and more intuitive.

“LLMs have demonstrated potential in explaining code, decision models, and system logs, as well as in creating more intuitive and understandable visualizations.”

May combine two claims.

LLMs have been used in task allocation, cooperative behavior, and user correction, mainly to infer user intentions or behavioral patterns and to plan collaborative patterns.

“LLMs have been applied in task allocation, cooperative behavior, and user correction, where their main function is to infer users' intentions or behavioral patterns and plan collaborative patterns.”

May combine two claims.

Existing methods have mainly targeted design time, which limits their direct use at runtime.

“However, existing methods have primarily focused on design time, limiting their direct applicability to runtime settings.”

Probably not a substantial claim: may not pass S4 (based).

At design time, methods generally depend on historical data and knowledge from domain experts.

“At design time, methods generally rely on historical data and knowledge available from domain experts.”

Classified as probably not a substantial claim.

Runtime methods are concerned with adapting or modifying systems while they operate.

“Runtime methods on the other hand are concerned with adapting or modifying systems during operation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This versatility complicates the design and management of system observations and representations.

“However, this versatility also complicates the design and management of system observations and representations.”

May combine two claims.

LLMs perform better with data in HTML or XML than in JSON or Markdown, likely because structured tag repetition enhances attention in Transformers.

“studies have shown that LLMs perform better with data in HTML or XML formats than in JSON or Markdown, likely due to the structured repetition in tags such as <tr> and </tr>, which enhances attention mechanisms in Transformers [Sui et al., 2024].”

• cites Sui et al., 2024Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Runtime methods can use online operational data as a primary information source.

“Runtime methods on the other hand can leverage online operational data as a primary information source.”

Probably not a substantial claim: may not pass S2 (checkable).

At runtime, GenAI must take a more autonomous role to support timely decision-making for adaptation.

“to make timely decisions at runtime, GenAI needs to take on a more autonomous role supporting on-time decision-making for adapting the system to different contexts and environmental changes.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Languages that are more suitable for communication between LLMs are not necessarily human-readable.

“some studies report that languages more suitable for communication between LLMs (i.e., higher performance) are not necessarily human-readable [Deng et al., 2022].”

• cites Deng et al., 2022Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

LLM outputs are inherently probabilistic, so the same input can produce different outcomes.

“However, the output of LLMs is inherently probabilistic, i.e., LLMs might produce different outcomes even for the same input.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLMs, especially multimodal LLMs, can process and interpret data across diverse modalities and fields.

“LLMs, and MLLMs in particular have shown the ability to process and interpret data across diverse modalities and fields.”

May combine two claims.

LLMs often perform better when fine-tuned for specific domains, yielding smaller models with lower usage costs and faster response times.

“While LLMs have demonstrated generalized capabilities, they often see enhanced performance when fine-tuned for specific domains, resulting in smaller models with lower usage costs and faster response times [Liu et al., 2023a].”

• cites Liu et al., 2023aNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLMs have been used to partially or fully automate various aspects of software system realization.

“As widely discussed in the area of SE, LLMs have been utilized to (semi-)automate various aspects of realizing software systems.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

At design time, decision-making centers on stakeholders who may use GenAI as a supportive tool.

“At design time, decision-making is centered around stakeholders that may leverage GenAI as a supportive tool”

May combine two claims.

Industry has been developing many domain-specific LLMs; in China, out of 190 large models, only 45 are general-purpose and 145 are domain-specific.

“an industry analysis report [Nandu Digital Economy Governance Research Center, 2023] notes that out of 190 large models in China, only 45 are general-purpose LLMs, while 145 are tailored for specific domains.”

• cites Nandu Digital Economy Governance Research Center, 2023Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Two authors carried out the filtering together, and a third author settled disagreements.

“Additionally, the filtering process was collaboratively conducted by two authors, with a third author involved to resolve any discrepancies.”

• AssumptionRests on an assumption: Collaborative discussion among authors reduces bias in filtering and categorization.

Probably not a substantial claim: may not pass S3 (load-bearing).

To reduce subjectivity in categorization, two or more authors discussed the categories.

“To mitigate this, categorization was also based on discussions involving two or more authors.”

• AssumptionRests on an assumption: Collaborative discussion among authors reduces bias in filtering and categorization.

Probably not a substantial claim: may not pass S3 (load-bearing).

The rules for filtering were repeatedly adjusted and merged when the authors discussed ambiguous cases.

“Simultaneously, as discussions occur about the parts of ambiguity among the authors, the rule is accordingly refined and integrated several times.”

• AssumptionRests on an assumption: Collaborative discussion among authors reduces bias in filtering and categorization.

Probably not a substantial claim: may not pass S3 (load-bearing).

Decentralization of control in SASs, by dividing MAPE-K responsibilities across modules, is well-documented.

“The notion of decentralization of control in SASs, by dividing the responsibilities of the MAPE-K functions across modules, is well-documented [Weyns et al., 2013].”

• cites Weyns et al., 2013Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors simplified the discussion of certain topics to balance article length and content distribution.

“To maintain a balance between the length of the article and the distribution of content, we simplified the discussion of certain topics.”

• AssumptionRests on an assumption: Monitoring and certain planning techniques are not central research topics in SASs.
• AssumptionRests on an assumption: Detailed discussion of Transformer architecture variants and LLM prompt designs for different tasks offers less insight to readers.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

LLMs are used to inject knowledge or commonsense into reinforcement learning as reward functions and into search algorithms as heuristics.

“LLMs are employed to inject knowledge or commonsense into RL as reward functions and into search algorithms as heuristics.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

To counteract presentation bias, the authors list all literature in the main text as fully as possible and make the survey data publicly accessible.

“To counteract this, we mention all literature in the main text as comprehensively as possible (even if it leads to redundancy), and we make the survey data publicly accessible.”

• AssumptionRests on an assumption: Public access to the survey data lets readers examine the original papers for technical details.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The authors' interpretation of categorization may be subjective, and even with structured guidelines, sorting literature into categories can be swayed by personal viewpoints, affecting the neutrality of the analysis.

“The third threat involves the potential subjectivity in our interpretation of categorization.Even with structured guidelines, the classification of literature into categories can be influenced by individual perspectives, which may affect the neutrality of the analysis.”

May combine two claims.

The authors' literature search coverage was limited by reliance on specific title keywords and focus on leading conferences.

“The first threat to our research methodology is the limitation of our literature search coverage due to our reliance on specific title keywords to search for literature within particular conferences.”

May combine two claims.

The authors' approach to filtering literature for SAS relevance was based on specific rules that may reflect the authors' biases and affect the objectivity and accuracy of literature selection.

“The second threat stems from our method of filtering literature for relevance to SASs, which was based on specific rules.These rules may reflect the authors' biases, potentially impacting the objectivity and accuracy of our literature selection.”

Probably not a substantial claim: may not pass S3 (load-bearing).

To mitigate filtering bias, the authors used the MAPE-K feedback loop and the three main HOTL design principles to identify literature relevance to SASs.

“To mitigate these concerns, we first refer to the MAPE-K loop and the three key design principles for HOTL to identify the literature's relevance to SASs.”

• AssumptionRests on an assumption: The MAPE-K feedback loop and the three main HOTL design principles are enough to judge literature relevance to SASs.

Probably not a substantial claim: may not pass S3 (load-bearing).

To reduce search coverage limitations, the authors broadened their search to cover more conferences and keywords within the time available.

“To mitigate these limitations, we expanded our search to include as many conferences and keywords as possible within our limited timeframe, although a more comprehensive systematic literature review would be ideal to further address this issue.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Collective intelligence has shown promise for distributed planning in LLM multi-agent systems.

“the notion of collective intelligence has shown promise in distributed planning within LLM Multi-Agent Systems (LLM-MAS).”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

LLMs typically have a static context window, and inference time increases with context length.

“However, LLMs typically have a static context window (e.g., 16k tokens in GPT-3), and the LLMs' inference time also increases with the length of the context.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The scalability of LLM-MAS is currently limited to about five agents, and communication costs grow exponentially with the number of agents.

“Additionally, the scalability of LLM-MAS is currently limited (typically up to 5 agents), and as the number of agents grows, the communication costs increase exponentially.”

May combine two claims.

As SASs enter daily life and serve more general end-users, the assumption that humans are domain experts may not hold, introducing new uncertainties about users' knowledge and interaction capabilities.

“However, as SASs are penetrating daily life and serve more general end-users [Weyns et al., 2023], these assumptions may not hold, introducing new uncertainties related to the users' knowledge and interaction capabilities.”

• cites Weyns et al., 2023Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The authors' presentation of the literature may reflect their inherent biases.

“Such a presentation may reflect the authors' inherent biases.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Generally, the more contextually rich and relevant the information in a prompt, the higher the quality of the LLM's output.

“Generally, the more contextually rich and relevant the information provided in a prompt, the higher the quality of the LLM's output will be.”

May combine two claims.

LLM outputs mimic human reasoning, but this does not mean that neural-network-based LLMs truly reason as symbolic AI systems do.

“It is important to note that the output of LLMs simulates human reasoning processes, this does not imply that LLMs, which are based on neural networks, are truly reasoning in the way symbolic AI approaches do[Fedorenko et al., 2024].”

• cites Fedorenko et al., 2024Not checked yet

  Not checked in the free run.