Language Model Crossover: Variation through Few-Shot Prompting

May combine two claims.

Experiments in the paper highlight LMX's versatility in binary strings, style transfer of plain-text sentences, symbolic regression of mathematical expressions, generating images through text-to-image prompts, and generating Python code.

“LMX's versatility is highlighted in experiments with binary strings, style transfer of plain-text sentences, symbolic regression of mathematical expressions, generating images through prompts for a text-to-image model, and generating Python code.”

May combine two claims.

LMX performance improves as LLM capabilities increase, as validated by experiments in the paper.

“this operator should naturally improve as LLMs increase in capabilities (which experiments here validate)”

May combine two claims.

Offspring from the all-ones neighborhood had a significantly greater Hamming distance from the all-zeros string than from the all-ones string, and the reverse also held, with p<0.001.

“Indeed, offspring generated from the neighborhood of the all-ones string had significantly higher (Mann-Whitney U-test; 𝑝 < 0.001) hamming distance from the all-zeros string than the all-ones string (and vice-versa; see Figure 3a).”

May combine two claims.

In OneMax, LMX achieves higher mean fitness values than one-point crossover, with a Mann-Whitney p-value of 0.002.

“Though fewer solutions converge on the optima using LMX the classical recombination (16/20 vs. 20/20), mean values are higher (Mann-Whitney 𝑝 = 0.002).”

May combine two claims.

LMX using the GALACTICA LLM achieves overall higher fitness and lower expression size than gplearn.

“Figure 6 shows that LMX (using the GALACTICA LLM) achieves overall higher fitness and lower expression size than gplearn, and the choice of LLM appears to have a substantial impact, with the Pythia runs falling short of the others.”

May combine two claims.

The choice of LLM has a substantial impact on LMX performance, with Pythia runs falling short of the others.

“Figure 6 shows that LMX (using the GALACTICA LLM) achieves overall higher fitness and lower expression size than gplearn, and the choice of LLM appears to have a substantial impact, with the Pythia runs falling short of the others.”

May combine two claims. The quote only partly states this claim.

In one LMX run on the banana problem, the process quickly finds a core functional skeleton, then refines constants, and around generation 3000 makes further substantial improvements.

“Even after the process appears to have converged, around generation 3000 it discovers innovations leading to further substantial improvements.”

May combine two claims.

LMX avoids model bloat as it incrementally improves fitness, satisfying a key desirable property for SR.

“(b) LMX avoids model bloat as it incrementally improves fitness, thereby satisfying a key desirable property for SR.”

May combine two claims.

LMX produces competitive results and generates fit, parsimonious expressions.

“LMX produces competitive results, generating fit and parsimonous expressions.”

May combine two claims.

Both zero-shot and random human prompt baselines far underperform the evolutionary methods.

“The zero-shot baseline quickly stagnates, as it is unable to iteratively refine it's initial solutions; even randomly-selected human prompts eventually outperform it, as they have greater diversity; both these baselines far underperform the evolutionary methods.”

May combine two claims.

LMX performance on the SR problem is competitive with state-of-the-art methods, settling at an intermediate point along the Pareto front.

“Figure 7 shows that the performance of LMX on this problem is competitive with state-of-the-art methods [58], settling at an intermediate point along the Pareto front.”

• cites [58]Not checked yet

  Not checked in the free run.

May combine two claims.

One-point crossover is a strong baseline, with performance statistically similar to the no-header ablation.

“One-point crossover is a strong baseline, with performance statistically similar to the 'no header' ablation, supporting the idea that the ability to naturally incorporate natural language problem specifications is a key advantage of LMX over classical EAs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

At sufficient distance from the original sentence, evolution often produces repetitive, unrelated text.

“At sufficient distance from the original sentence, evolution often produces repetitive, unrelated text: e.g.”

May combine two claims.

Beginning each completion with the make_walker() function signature improves performance; experiments where the prompt did not end with the function signature performed worse.

“Note that we begin each completion with this function signature to improve performance (experiments where the LLM prompt did not end with the function signature performed worse; see appendix E).”

May combine two claims.

There is a clear trend in model size: the 6B model can create higher fitness and more diverse Sodaracers, with a slight trend toward improved proportion of valid programs with model scale.

“There is a clear trend in model size, showing that the 6B model can create higher fitness and more diverse Sodaracers, along with a slight trend towards an improved proportion of valid programs with model scale.”

May combine two claims.

When randomly generated parents are arranged in ascending order of the 'ones' count, the resulting offspring distribution exhibits a clear skew towards higher fitness, and vice versa.

“When randomly generated parents are arranged in ascending order of the 'ones' count, the resulting offspring distribution exhibits a clear skew towards higher fitness (left) and vice versa.”

May combine two claims.

LMX avoids model bloat as it incrementally improves fitness, satisfying a key desirable property for symbolic regression.

“LMX avoids model bloat as it incrementally improves fitness, thereby satisfying a key desirable property for SR.”

The quote only partly states this claim.

When Pythia is the LLM, performance is not as strong as with Galactica.

“As an ablation, when Pythia is the LLM, performance is not as strong.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The experiments focused on breadth rather than depth.

“The experiments focused on breadth rather than depth, and it is possible that with further effort LMX could enable state-of-the-art results, e.g., in symbolic regression.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All experiments in this paper except the Python Sodaracer experiment were conducted through Google Colab notebooks or on a single GPU.

“While LLMs are computationally expensive, all of the experiments in this paper (with exception of the Python Sodaracer experiment) were conducted either through Google Colab notebooks or on a single GPU; the code to run experiments is surprisingly compact, as the LMX method consists mainly of a simple LLM prompting strategy, and”

• AssumptionRests on an assumption: LLMs are computationally expensive.

May combine two claims.

LMX performs comparably to previously published state-of-the-art symbolic regression methods, falling on the Pareto front for the 'banana' problem.

“LMX performs comparably to previously published results from state-of-the-art SR methods[58], falling on the Pareto front for the 'banana' problem, suggesting that it is a promising approach to symbolic regression.”

• cites [58]Not checked yet

  Not checked in the free run.

May combine two claims.

As the parameter count of the LLM increases in the length-9 binary string domain, the percent of valid offspring and number of novel offspring also increase.

“As the parameter count (i.e., number of weights trained with SGD) of the LLM is increased in the length-9 binary string domain, the percent of valid offspring and number of novel offspring (out of at most 60) also increase.”

May combine two claims.

The number of novel genotypes generated on average from 20 applications of LMX reaches its maximum at four parents, while five parents tends to more often produce offspring that duplicate one of the parents exactly.

“The number of novel genotypes generated on average from 20 applications of LMX (which at 3 offspring per application can result in at most 60 offspring) to a random set of parents reaches its maximum at four parents (while five parents tends to more often produce offspring that duplicate one of the parents exactly).”

May combine two claims.

The experiments in this paper cover a broad range of applications including equations, plain-text sentences, images, and code, using open-source neural networks for variation, crossing modalities, and measuring fitness and diversity.

“The experiments in this paper tackle a broad range of potential applications, across equations, plain-text sentences, images, and code, leveraging the burgeoning ecosystem of open-source neural networks as means of generating variation, crossing modalities, and measuring both fitness and diversity.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Preliminary experiments showed some qualitative effect from applying LMX on individuals with similar embeddings.

“Another interesting question is whether examples fed into LMX could be chosen more deliberately (e.g.only crossingover similar individuals to get more nuanced variation); preliminary experiments showed some qualitative effect from applying LMX on individuals with similar embeddings (Section 4.3), but require further experimentation to validate.”

May combine two claims.

In the Homer Simpson quote sentiment task, LMX-Near performs better than LMX, filling a few extra niches nearer to the start sentence.

“LMX-Near performs better than LMX in this case, filling a few extra niches nearer to the start sentence.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The sampling temperature of 0.8 noticeably improved consistency of generated output compared to the default of 1.0 in initial tests.

“The sampling temperature was set to 0.8, which in initial tests was found to noticeably improve consistency of generated output compared to the default of 1.0.”

May combine two claims.

LMX generally benefits from more examples in its prompt, is able to produce reasonable variation, and often creates valid Sodarace mutations.

“LMX generally benefits from more examples in its prompt, is able to produce reasonable variation, and often creates valid Sodarace mutations, highlighting its promise for evolving code.”

May combine two claims.

For the Gore Vidal quote, LMX-Near outperforms LMX significantly, and both significantly outperform the control.

“LMX-Near outperforms LMX significantly, and both significantly outperform the control.”

May combine two claims.

When averaged over 15 randomly-generated parent sets, both the percent of valid offspring and number of novel offspring generally increase with model size.

“When averaged over 15 randomly-generated parent sets, both the percent of valid offspring and number of novel offspring generally increase with model size (Figure 2b).”

May combine two claims.

The zero-shot baseline quickly stagnates because it cannot iteratively refine its initial solutions; even Human Random solutions eventually outperform it due to greater diversity.

“The zero-shot baseline quickly stagnates, as it is unable to iteratively refine it's initial solutions; even Human Random solutions eventually outperform it, as they have greater diversity.”

May combine two claims.

LMX and LMX-Near do not perform significantly differently in increasing positive sentiment of the Simpsons quote, but both significantly outperform the control.

“LMX and LMX-Near do not perform significantly differently, but both significantly outperform the control.”

May combine two claims.

In the initial Sodaracers experiments, higher numbers of parents nearly always increases performance, and the 2B model performs the best, while the 6B model underperforms.

“Higher numbers of parents nearly always increases performance in this setting, and the 2B model performs the best (interestingly, the 6B model underperforms; the reason is unclear, but suggests follow-up work to better understand what LLMs best fit a given task).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The vast majority of wall-clock time is spent in the forward pass of the LLM.

“The vast majority of wall-clock time is spent in the forward pass of the LLM.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Running 5000 generations took around 100 hours.

“Running 5000 generations took around 100hrs.”

May combine two claims. The quote only partly states this claim.

The highest-fitness prompts and corresponding images of LMX for each color all include the word 'background' but vary in length and detailed content, highlighting LMX's ability to discover diverse, non-obvious solutions.

“The highest-fitness prompts and corresponding images of LMX for each color all include the word”

May combine two claims.

For the Woody Allen quote, LMX and LMX-Near do not perform significantly differently, but both significantly outperform the control.

“LMX and LMX-Near do not perform significantly differently, but both significantly outperform the control.”

May combine two claims.

One-point crossover is a strong baseline, with performance statistically similar to the LMX-No Header ablation.

“One-point crossover is a strong baseline, with performance statistically similar to the LMX -No Header ablation.”

May combine two claims.

The results indicate that LMX can produce quality results across domains, often by leveraging the ecosystem of pretrained models to quantify fitness or diversity or to cross modalities.

“The results highlight the potential of the method to produce quality results across domains, often by leveraging the broad ecosystem of pretrained models that can be easily combined in many ways to quantify fitness or diversity, or to cross modalities (i.e. from text to image).”

May combine two claims.

LMX provides a simple mechanism to evolve semantically rich text representations with few domain-specific tweaks and benefits from progress in language models.

“The promise of such language model crossover (which is simple to implement and can leverage many different open-source language models) is that it enables a simple mechanism to evolve semantically-rich text representations (with few domain-specific tweaks), and naturally benefits from current progress in language models.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs are behind many approaches achieving state-of-the-art results in NLP domains such as question answering, code generation, and few-shot classification.

“Large language models (LLMs; [9,13]) are behind many of the approaches achieving state-of-the-art results in natural language processing domains, such as question-answering [19,33,70], code-generation [17,66], and few-shot classification [13,99].”

• cites [9] sourceNot checked yet

  Not checked in the free run.

• cites [13] sourceNot checked yet

  Not checked in the free run.

• cites [19] sourceNot checked yet

  Not checked in the free run.

• cites [33] sourceNot checked yet

  Not checked in the free run.

• cites [70]Not checked yet

  Not checked in the free run.

• cites [17]Not checked yet

  Not checked in the free run.

• cites [66] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Performance at in-context learning improves with model scale, implying that methods relying on this capability will benefit from continuing progress in LLM training.

“Importantly, performance at in-context learning improves with model scale [15,124], implying that methods relying upon this capability will benefit from continuing progress in LLM training.”

• cites [15]Not checked yet

  Not checked in the free run.

• cites [124] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Variation is critical as a primary driver of exploration and exploitation of the search space.

“Such variation is critical as a primary driver of both exploration and exploitation of the search space [27].”

• cites [27]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The variation operator suggested by LMX is simple to implement.

“What is intriguing about this insight is that the variation operator it suggests is (1) simple to implement (i.e.concatenate a few text-based genotypes into a prompt, run it through an LLM, and extract a new genotype from its output; we release code implementing it accompanying this paper)”

Probably not a substantial claim: may not pass S4 (based).

Few-shot prompting works because the LLM can guess the pattern behind a few input/output pairs and generalize to a new target input at the end of the prompt.

“Few-shot prompting works because the LLM can”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Such an operator can be created through very few lines of code.

“Such an operator can be created through very few lines of code.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Autoencoders serve either to encode solutions into phenotypes matching a learned distribution or to generate new solutions sampled from that distribution.

“Autoencoders are used either as solution encodings which convert raw genotypes into phenotypes that align with the learned distribution; or as a mechanism for solution generation, with new solutions drawn directly from the established distribution.”

• cites [46] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Estimation of distribution approaches have employed autoencoders to define distributions over high-performing solutions found during search.

“EDA approaches have leveraged autoencoders [46] to define distributions based on high-performing solutions identified during the search process.”

• cites [46] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Advanced foundation models can be used directly, with distributions defined through strategic prompting, unlike traditional approaches that require training on solutions generated during search.

“Unlike traditional approaches that necessitate training models on solutions generated during the search, these advanced models can be directly leveraged, with distributions defined via strategic prompting.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In theory, LMX can be generically applied to most evolutionary algorithms by swapping it in as the genetic variation operator.

“In theory, however, LMX can be generically applied to most EAs, e.g.multi-objective EAs [23,29], evolutionary strategies [1,5], or in support of open-ended evolution [121], but simply swapping it in as the genetic variation operator.”

• cites [23]Not checked yet

  Not checked in the free run.

• cites [29] sourceNot checked yet

  Not checked in the free run.

• cites [1] sourceNot checked yet

  Not checked in the free run.

• cites [5] sourceNot checked yet

  Not checked in the free run.

• cites [121]Not checked yet

  Not checked in the free run.

May combine two claims.

For examples drawn from a distribution, the probability of an LLM's output given multiple examples exceeds the probability given one example, which exceeds the probability given no examples.

“Pr 𝜓 LLM 𝜙 𝑥 1 , . . . , 𝑥 𝑘 X > Pr 𝜓 LLM 𝜙 𝑥 1 X > Pr 𝜓 LLM 𝜙 X .(5)”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LMX has a theoretical relationship to estimation of distribution algorithms.

“although LMX does bear a theoretical relationship to EDAs, as explored in Section 5.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The 'banana' problem consists of 5300 samples and two input features.

“This black-box problem was originally derived from a popular ML benchmark in the KEEL data set repository [31]; it has 5300 samples and two input features 𝑥 1 , 𝑥 2 .”

• cites [31]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

One can choose among various off-the-shelf LLMs for LMX according to the characteristics of the problem.

“This choice also highlights how different off-the-shelf LLMs can be selected for LMX based on properties of the problem.”

May combine two claims.

LMX is able to evolve reasonable solutions, though the paper does not claim it is better than existing methods.

“Note that the claim here is not that LMX is better than these existing methods, but simply that it is able to evolve reasonable solutions.”

Classified as probably not a substantial claim.

Stable Diffusion was made deterministic by reseeding with a fixed PRNG seed before each image generation, so a given prompt always produces the same image.

“We make Stable Diffusion deterministic by reseeding with a fixed PRNG seed before each image is generated, so a given prompt always produces the same image.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The bias toward human-designed expressions is also a natural bias against model bloat.

“This bias towards human-designed expressions is also a natural bias against model bloat, since humans strive to design compact expressions.”

• AssumptionRests on an assumption: Humans strive to design compact expressions.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Stable Diffusion is a publicly available latent diffusion model that supports CLIP-guided text-to-image synthesis.

“Stable Diffusionfoot_3 is a publicly available latent diffusion model [95] that supports CLIP-guided [91] text-to-image synthesis.”

• cites [95] sourceNot checked yet

  Not checked in the free run.

• cites [91]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The excess color fitness functions correspond roughly to perceived image color and are well studied in agricultural image processing.

“These functions are easy to calculate, correspond roughly to perceived image color (e.g., they are well studied in agricultural image processing [75]), and provide a proof-of-concept where performance can be visually verified at a glance.”

• cites [75]Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Sodarace is a 2D simulation of robots with arbitrary morphology constructed from Python functions that output a dictionary specifying joints and muscles and their connections.

“Sodarace is a 2D simulation of robots with arbitrary morphology constructed from Python functions (the genotype) which output a dictionary specifying joints and muscles, and how they are connected.”

• cites [63]Not checked yet

  Not checked in the free run.

• cites [12]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Randomizing the order of seeds for each LMX application controls for variance from the order of programs in the prompt.

“We randomize the order of seeds for each application of LMX, to control for variance in results from the order of programs in the prompt.”

Probably not a substantial claim: may not pass S4 (based).

By concatenating parents in random order, the implicit signal to the LLM is that the list is unordered.

“By concatenating parents in a random order, the implicit signal to the LLM is that the list is unordered.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A standard GA generates children by sampling from an implicit conditional probability distribution induced by randomly sampling parents and applying a stochastic reproduction operator.

“In contrast, a standard GA generates children by sampling from an implicit conditional probability distribution 𝑝 𝑔 (𝑥 | [𝑥 1 , . . ., 𝑥 𝑀 ]) induced by the process of randomly sampling parents and applying a stochastic reproduction operator 𝑔 (e.g., a crossover operator; Eq. 4).”

Probably not a substantial claim: may not pass S4 (based).

The LLM's highest-probability action is to keep sampling objects from D* as it generates output.

“These parents are text-based objects that must have been sampled from some ground truth distribution 𝐷 * , and thus the LLM's highest-probability action is to keep sampling objects from 𝐷 * as it generates output.”

• AssumptionRests on an assumption: Parents are text-based objects that must have been sampled from some ground truth distribution D*.

Probably not a substantial claim: may not pass S3 (load-bearing).

An EDA constructs an explicit probabilistic distribution fit to the parent set and samples child solutions from it.

“An EDA constructs an explicit probabilistic distribution 𝐷 fit to the parent set {𝑥 1 , . . ., 𝑥 𝑀 }, and samples child solutions 𝑥 from 𝐷 [45,61].”

• cites [45] sourceNot checked yet

  Not checked in the free run.

• cites [61]Not checked yet

  Not checked in the free run.

May combine two claims.

As the number of parents increases, the significance of accidental order patterns diminishes and a well-trained LLM is more likely to perceive the order as random.

“The LLM may notice some accidental patterns in the order, but, as the number of parents increases, e.g., when LMX processes the full population as in the EDA above, the significance of such spurious patterns diminishes and a well-trained LLM is more likely to perceive the order as random.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Because a causal LLM is used, probabilities of each bit are not technically independent but conditional on previously generated bits and parent order.

“Note that, as we are using a causal LLM, probabilities of each bit are not technically independent, but rather conditional on the previously generated bits in the genome as well as on the order of the parents.”

May combine two claims.

As the LLM becomes a better in-context learner, it becomes better able to detect subtler patterns in randomly-ordered concatenated parents, and thus LMX approximates the ideal EDA distribution D*_EDA.

“As the LLM becomes a better and better in-context learner, it becomes better able to detect subtler patterns within a prompt of randomly-ordered concatenated parents, and thus LMX(𝑥 1 , . . . , 𝑥 𝑘 ) ≈ 𝐷 * EDA .(13)”

Probably not a substantial claim: may not pass S2 (checkable).

D_LM consists of distributions of objects found in sets that might appear in the universe of data from which the LLM's training dataset was drawn.

“In other words D LM consists of distributions of objects that are found in sets that might appear in the universe of data from which the dataset used to train the LLM was drawn.”

May combine two claims.

LMX occupies an intermediate level of explicitness: its conditional distribution is explicit in yielding token probability distributions and computable output sequence probabilities, but implicit because the internal workings are encoded opaquely in LLM parameters and activations.

“LMX occupies an intermediate level of explicitness: The conditional distribution induced by feeding the parent prompt into the LLM is explicit in that it yields a series of probability distributions over tokens, and the probability of any output sequence can be directly computed, but is Manuscript submitted to ACM implicit in the sense that the internal workings of the distribution are encoded opa”

The quote only partly states this claim.

Theorem 5.2: For any genetic operator g on candidate space X and epsilon > 0, if phi is injective and psi is surjective, then there exists an LLM such that for all parent sets X of g and children x, the probability difference between g and LMX is less than epsilon.

“Theorem 5.2 (weight-based universality).For any genetic operator 𝑔 on candidate space X and 𝜖 > 0, if 𝜙 : 2 X → 𝑉 * is injective, and 𝜓 : 𝑉 * → X is surjective, then ∃ an LLM s.t. for all parent sets 𝑋 of 𝑔 and”

May combine two claims.

The generative distributions in the experiments are limited to ones induced by simple concatenation of parents, but the underlying universality of LMX provides further explanation for its effectiveness as a generic operator across domains.

“The generative distributions in the experiments in this paper are limited to ones induced by simple concatenation of parents, but the underlying universality of the LMX method in general provides further explanation for how it can be an effective generic operator across such a wide range of domains, as was demonstrated in Section 4.”

May combine two claims.

The ability to achieve arbitrarily complex and diverse reproductive behavior within a single framework gives LMX a potential advantage over hand-designed genetic operators; in theory, LMX can represent all such operators.

“This ability to achieve arbitrarily complex and diverse reproductive behavior within a single framework gives LMX a potential advantage over genetic operators that are hand-designed for different tasks: In theory, LMX can represent all such operators (especially if they appear in the dataset used to train the LLM).”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The connection to EDAs may help explain why LMX is effective as an off-the-shelf genetic operator across a wide range of domains.

“Despite this confounding factor, it is clear that with a larger number of parents both approaches converge toward the same distribution -and this connection to EDAs may help to explain why LMX is effective as an off-the-shelf genetic operator across a wide range of domains.”

May combine two claims.

LMX can in principle sample from any conditional probability distribution, making it universal in the space of genetic operators, even with small parent sets.

“making it universal in the space of genetic operators, even with small parent sets.”

May combine two claims.

Existing fixed pre-trained LLMs with external memory, e.g., Flan-U-PaLM 540B, have been shown to implement universal Turing machines, implying universality can be achieved through effective prompting without altering LLM weights.

“However, when coupled with external memory, existing fixed pre-trained LLMs today, e.g., Flan-U-PaLM 540B [21], have been shown to implement universal Turing machines (UTMs) [38,104], implying that universality can be achieved through effective prompting schemes, without altering LLM weights:”

• cites [21]Not checked yet

  Not checked in the free run.

• cites [38]Not checked yet

  Not checked in the free run.

• cites [104] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

EDAs typically assume an unordered distribution, but the inherent input ordering in autoregressive models creates an opportunity for directing the operator's output.

“EDAs typically assume an unordered distribution, yet the inherent input ordering in autoregressive models creates a unique opportunity for directing the operator's output.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For relatively small search spaces, if LMX behaves unexpectedly, it may be difficult to diagnose the cause due to LLM opacity, whereas traditional EAs allow straightforward reasoning about genetic operator variation.

“However, even for relatively small search spaces, if LMX is not behaving as expected on a particular problem, it may be difficult to find the precise cause of the issue, due to the opacity of LLMs, whereas in traditional EAs it could be straightforward to reason about the kinds of variation a genetic operator is likely to produce.”

• AssumptionRests on an assumption: The opacity of LLMs makes it difficult to diagnose unexpected behavior.

Probably not a substantial claim: may not pass S3 (load-bearing).

LLM opacity can obscure undesirable biases in the model, which is especially concerning for applications with societal implications.

“This opacity could also obscure undesirable biases hidden in the model, which could be especially concerning if the application has societal implications.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Later work using an LMX-like approach to discover new solutions to mathematical problems also sorted parents by fitness before recombination.

“Later work, which used an LMX-like approach to discover new solutions to mathematical problems [96] also sorted parents by fitness before recombination.”

• cites [96]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

EAs have been shown to be effective in search spaces with millions and even billions of variables.

“EAs have been shown to be effective in search spaces with millions and even billions of variables [20,28].”

• cites [20]Not checked yet

  Not checked in the free run.

• cites [28] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

These results are suggestive of the versatility of this biasing technique; a variety of sorting strategies could be employed to cater to specific objectives.

“These results are suggestive of the versatility of this biasing technique; a variety of sorting strategies could be employed to cater to specific objectives.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Subsequent work following the arxiv release of the LMX paper performed extensive experimentation with different orderings of parents in more complex natural language domains and came to similar conclusions about the importance of ordering.

“Subsequent work following the arxiv release of the LMX paper [34,131] performed extensive experimentation with different orderings of parents in more complex natural language domains, and came to similar conclusions about the importance of ordering.”

• cites [34]Not checked yet

  Not checked in the free run.

• cites [131]Not checked yet

  Not checked in the free run.

May combine two claims.

Autoregressive models act as pattern completion engines and provide a pathway to guide sampling through directional cues.

“Acting as pattern completion engines, these models provide a pathway to guide the sampling process through directional cues.”

May combine two claims.

An ascending input prompt produces ascending output; consequently, an input arranged in ascending fitness order prompts the model to generate output following an ascending fitness trend.

“An ascending input prompt produces ascending output.Consequently, an input arranged in ascending fitness order prompts the model to generate output that follows an ascending fitness trend.”

May combine two claims.

The LLM can generate invalid outputs that cannot be parsed as solutions, whereas traditional EAs can formally guarantee validity through carefully designed genetic operators.

“There is also the issue that the LLM can generate”

• AssumptionRests on an assumption: Traditional EAs can formally guarantee validity through carefully designed genetic operators.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

For application domains far from the LLM's training set, the method of population initialization becomes particularly important.

“Relatedly, if an application domain is especially far from the LLM's training set, e.g., far from any kind of text found on the internet, then choosing a method of population initialization becomes particularly important.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For a given model size, there may be a trade-off between more generic models, which likely perform decently in most domains, and more specialized ones.

“If larger models are infeasible to run, for any given model size there may be a trade-off between more generic models, which are likely to perform decently in most domains, and more specialized ones.”

• AssumptionRests on an assumption: Larger LLMs generally have been trained on larger and broader data sets.

Probably not a substantial claim: may not pass S3 (load-bearing).

LMX's ability to suggest relevant variation in a domain is likely dependent on the LLM's training data, size, and training quality.

“One important direction is Manuscript submitted to ACM to explore the dependence of LMX's performance on qualities of the underlying LLM; the ability of LMX to suggest relevant variation in a particular domain is likely dependent on the LLM's training data (along with its size and how well it was trained).”

Probably not a substantial claim: may not pass S4 (based).

LMX alone does not address common population dynamics concerns of traditional EAs such as premature convergence, diversity maintenance, and exploration/exploitation pitfalls.

“Finally, LMX alone does not allow us to escape common population dynamics concerns that plague traditional EAs, such as premature convergence, diversity maintenance, and other exploration/exploitation pitfalls.”

Classified as probably not a substantial claim.

If the type of text being evolved is not well-represented in the LLM's training distribution, LMX relying on that LLM will likely perform poorly.

“For example, the expectation is that if the type of text chosen for evolution (e.g.code in a very new programming language) is not well-represented in the training distribution of a particular LLM, then LMX that relies on that LLM will likely perform poorly.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LMX could be used for continuous optimization, as suggested by its tuning of floating-point constants.

“LMX could even be used for continuous optimization, as suggested by how it tunes floating-point constants (Figure 5).”

Probably not a substantial claim: may not pass S2 (checkable).

LMX is a flexible and easy-to-use genetic operator that lets EA practitioners take advantage of recent large language models.

“As a flexible and easy-to-use genetic operator, LMX provides a way for EA practitioners to take advantage of the recent revolution in large language models.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The level of stochasticity in LMX can be controlled by the softmax temperature parameter, analogous to a mutation rate parameter in a traditional EA.

“For example, the level of stochasticity in LMX can be controlled by the softmax temperature parameter, which can be seen as analogous to a mutation rate parameter in a traditional EA.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The code to run experiments is surprisingly compact because LMX consists mainly of a simple LLM prompting strategy.

“While LLMs are computationally expensive, all of the experiments in this paper (with exception of the Python Sodaracer experiment) were conducted either through Google Colab notebooks or on a single GPU; the code to run experiments is surprisingly compact, as the LMX method consists mainly of a simple LLM prompting strategy, and”

Classified as probably not a substantial claim.

LMX provides a generic genetic operator that allows practitioners to quickly set up algorithms to search diverse spaces in a manner aligned with the generative nature of LLMs.

“LMX does not solve these grand challenges of EAs, rather it provides a generic genetic operator that allows practitioners to quickly set up algorithms to search diverse spaces in a manner aligned with the generative nature of LLMs, i.e. aligned with the way humans produce text.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This time could be reduced considerably through batching offspring generation, which is naturally parallelized.

“This could be reduced considerably through batching offspring generation, which is naturally parallelized.”

• AssumptionRests on an assumption: Batching offspring generation is naturally parallelized.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Evolution in general is a powerful and easy-to-implement way to quickly explore possibilities, and LMX is a promising and simple way of instantiating them.

“In conclusion, there are likely many creative ways to beneficially combine various models together that this paper leaves unexplored; evolution in general is a powerful and easy-to-implement way to quickly explore such possibilities, and LMX in particular is a promising and simple way of instantiating them.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Adding an instruction to the end of the LMX prompt that asks the model to combine the starting programs into a new program provides a minimal form of domain customisation for the language model.

“This provides some minimal domain customisation to the language model, and is reminiscent of prior work demonstrating that fine-tuning language models on tasks described as instructions can dramatically improve performance on unseen tasks [98,123].”

Probably not a substantial claim: may not pass S3 (load-bearing).

Prior work indicates that instruction-based fine-tuning of language models can greatly enhance their performance on tasks they have not seen.

“This provides some minimal domain customisation to the language model, and is reminiscent of prior work demonstrating that fine-tuning language models on tasks described as instructions can dramatically improve performance on unseen tasks [98,123].”

• cites [98]Not checked yet

  Not checked in the free run.

• cites [123]Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

In the binary string experiments, genomes were pre- and post-processed with '_' characters separating each bit to ensure each bit is a single token, avoiding unintended tokenizer effects.

“In the binary string experiments to avoid unintended effects from the tokenizer, rather than evolving strings like 0011 all genomes were pre and post processed and given to the language model with”

Classified as probably not a substantial claim. May combine two claims.

To control for parent-order effects, the study either averaged results over every possible permutation of parents or, in long evolution runs, randomly selected a permutation for each sample.

“To control for this we either averaged our results over every possible permutation of parents or (in long evolution runs) randomly selected from the set of permutations for each sample.”

Classified as probably not a substantial claim. May combine two claims.

The 1-second simulation time used to evaluate Sodaracers is sufficient to show meaningful locomotion and was selected to allow meaningful evolution under the study's computational limits.

“The fitness function was measured as the horizontal distance travelled by an instantiated robot after 1 second of simulation time; this abbreviated evaluation time is enough to show meaningful locomotion and was chosen to demonstrate meaningful evolution within our computational constraints.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the excess-red, excess-blue, and excess-green fitness functions computed on 8-bit integer RGB images at 512×512 resolution, the maximum possible fitness is 66,846,720, the value for a monochromatic image of the target color.

“Image fitness functions were computed using 8-bit integer RGB images; the maximum fitness for the excess-red, excess-blue, and excess-green fitness functions is therefore 512 • 512 • 255 = 66,846,720, which would be the fitness of a monochromatic image of the target color.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Pythia models used are trained by EleutherAI for ongoing research.

“These Pythia models are trained by EleutherAI for ongoing research [7].”

• cites [7]Not checked yet

  Not checked in the free run.