Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs

May combine two claims.

The general purpose OptoPrime is competitive with specialized optimizers for each domain, e.g., achieving 10% higher accuracy on BigBenchHard when optimizing a DSPy program compared to their hand-designed optimizer.

“We find that the general purpose OptoPrime is competitive with specialized optimizers for each domain, e.g., achieving 10% higher accuracy on BigBenchHard [21] when optimizing a DSPy [22] program compared to their hand-designed optimizer.”

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [22]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Trace is available as a Python library with an API inspired by PyTorch.

“Trace is available as a Python library with an API inspired by PyTorch [19].”

• cites [19]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The agent can quickly improve its performance and learn strategies that are increasingly complex.

“The agent can quickly improve its performance and learn strategies that are increasingly complex.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

There is no mention of the specific Battleship environment API, nor details on how the functions reason and act should behave or adapt in Fig. 2a.

“Remarkably, there is no mention of the specific Battleship environment API, nor details on how the functions reason and act should behave or adapt in Fig. 2a.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The design of bundle allows tracing most Python codes, except for those modifying the content of an object reference in place.

“The design of bundle allows tracing most Python codes, except for those modifying the content of an object reference in place.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Trace logs the source code of decorated methods.

“Currently, OptoPrime uses the docstring of operators in the execution trace to understand the operators; nonetheless, Trace logs also the source code of the decorated methods, which can also be used in the future to design optimizers that uses more details of the operators.”

May combine two claims.

For a graph with N nodes and maximum degree W, Algorithms 1 and 2 have time complexity O(W N^2 log N) and space complexity O(W N).

“Theorem 1.For a graph with N nodes and maximum degree W , Algorithms 1 and 2 have time complexity O(W N 2 log N ) and space complexity 8 O(W N ).”

May combine two claims.

OptoPrime optimizes most workflows reasonably well using just the traced graph and feedback, but it can run into issues when single feedback alone is not informative.

“OptoPrime optimizes most workflows reasonably well using just the traced graph and feedback, but it can run into issues when single feedback alone is not informative (e.g., the output feedback is rewards, but there is no description of how the rewards are generated).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Currently, OptoPrime uses the docstring of operators in the execution trace to understand the operators.

“Currently, OptoPrime uses the docstring of operators in the execution trace to understand the operators; nonetheless, Trace logs also the source code of the decorated methods, which can also be used in the future to design optimizers that uses more details of the operators.”

May combine two claims.

TextGrad can be easily implemented as an optimizer in Trace, and OptoPrime achieves similar or better performance than TextGrad while using much less computation time.

“We show that TextGrad can be easily implemented as an optimizer in Trace, and OptoPrime achieves similar or better performance than TextGrad while using much less computation time.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

This level of abstraction in our experiments strikes a good balance between the ease of documentation and the graph size.

“This level of abstraction in our experiments strikes a good balance between the ease of documentation and the graph size.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Stateful functions that modify their state in place cannot be represented as a DAG without modification.

“Also, Trace cannot convert all computational workflows into OPTO problems, e.g., stateful functions that modify their state in place cannot be represented as a DAG without modification, and distributed/parallel computing workflows are incompatible with the current implementation (though in theory Trace can run in an asynchronous way so long as the overall graph does not end up with cycles).”

Probably not a substantial claim: may not pass S3 (load-bearing).

OptoPrime has a basic memory module that tracks the past parameter-feedback pairs as in-context examples.

“For robustness, we have a basic memory module in OptoPrime, which tracks the past parameter-feedback pairs as in-context examples.”

May combine two claims.

Trace is quickly competitive with the SCATS heuristic, whereas OPRO is not.

“Trace is quickly competitive with the SCATS heuristic, whereas OPRO is not.”

• Also stated in: § I.4 Traffic Control · plain

  “Trace is quickly competitive with the SCATS heuristic, whereas OPRO is not.”

May combine two claims.

Trace is able to match the best-in-class Adam; without access to the full computational graph, the feedback-alone optimizer struggles to find x*.

“Trace is able to match the best-in-class Adam; on the other hand, without access to the full computational graph, the feedback-alone optimizer struggles to find x * (Figure 5a).”

May combine two claims.

TextGrad can be implemented as an optimizer in the Trace framework, but the reverse is not possible.

“In fact, we easily implemented TextGrad as an optimizer in the Trace framework, but the reverse is not possible (because TextGrad couples the infrastructure and the optimization algorithm together).”

Classified as probably not a substantial claim.

Trace provides a lightweight Python tool to implement the Trace Oracle of OPTO for optimizing computational workflows.

“Trace provides a lightweight Pythonfoot_2 tool to implement the Trace Oracle of OPTO for optimizing computational workflows.”

May combine two claims.

In the experiments, Trace + OptoPrime can effectively learn hyperparameters, faster than Bayesian optimization.

“Here in the experiments we show that Trace + OptoPrime also can effectively learn hyperparamters, faster than Bayesian optimization.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The current design of OptoPrime does not handle non-textual content in parameters, inputs, or feedbacks.

“While Trace can handle this feedback, the current design of OptoPrime does not handle non-textual content (either in parameters, inputs or feedbacks).”

Probably not a substantial claim: may not pass S3 (load-bearing).

OptoPrime represents constraints on parameters as part of the text description.

“Lastly, right now OptoPrime represents the constraints on parameters as part of the text description, but we have observed that LLMs do not always follow it.”

May combine two claims.

OptoPrime is not as computationally efficient and cannot scale as well as ADAM.

“While we show in Section 5.1, that OptoPrime can compete with ADAM in small problems, OptoPrime is not as computational efficient and cannot scale as well as ADAM; therefore, for large-scale numerical problems, Trace and OptoPrime with the current design does not replace classical AutoDiff.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Our experiments in Appendix I use workflows declared using both AutoGen and DSPy.

“In fact, our experiments in Appendix I use workflows declared using both AutoGen and DSPy.”

May combine two claims.

Updating the parameter using the minimal subgraph is trivial, whereas without it there are 2^k possibilities to check.

“Updating the parameter using the minimal subgraph is trivial, whereas without it there are 2 k possibilities to check.”

May combine two claims.

Trace consumes additional overhead compared to black-box methods; beyond the space and time complexity for running the traffic simulation, Trace additionally materializes the computation graph per iteration.

“Finally, Trace consumes additional overhead compared to black-box methods; beyond the space and time complexity for running the traffic simulation, Trace additionally materializes the computation graph per iteration.”

Classified as probably not a substantial claim. May combine two claims.

Trace builds a DAG as the workflow executes and updates the parameters.

“Trace then builds a DAG as the workflow executes and updates the parameters”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Such errors can happen frequently especially when codes are parameters, as during optimization codes not satisfying syntax or downstream API requirements can happen.

“Such errors can happen frequently especially when codes are parameters, as during optimization codes not satisfying syntax or downstream API requirements can happen.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This error handling mechanism has two convenient usages.

“We find that this error handling mechanism has two convenient usages.”

Classified as probably not a substantial claim. May combine two claims.

The exception node becomes the new output of the inputs to the decorated method and is the output of the truncated computational graph.

“This exception node becomes the new output of the inputs to the decorated method (since the original method raises an error) and is the output of the truncated computational graph.”

May combine two claims.

Messages in TraceExecutionError can be used as the feedback f in OPTO and propagated from the exception node to the parameters.

“Messages in TraceExecutionError can then be used as the feedback f in OPTO and propagated from the exception node to the parameters.”

May combine two claims.

OPRO is able to solve Reach at the start but its performance degraded over iterations and gets similar performance as OptoPrime without memory in Push.

“OPRO is able to solve Reach at the start but its performance degraded over iterations (this instability was observed in [13]) and gets similar performance as OptoPrime (without memory) in Push.”

• cites [13]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Trace is able to learn a complex control logic in dozens of interactions, not only working on the training initial conditions but also on the held-out testing ones.

“The experimental results here are quite impressive, showing that Trace is able to learn a complex control logic in a dozens of interactions, not only working on the training initial conditions but also on the held-out testing ones too.”

Classified as probably not a substantial claim.

The computational graph ends at where the execution error happens.

“The computational graph ends at where the execution error happens.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Even though DSPy's meta-prompt optimization is trained on each task individually, the output of LLM to the evaluation method is still not post-processed, resulting in low performances of these tasks.

“Even though DSPy's meta-prompt optimization is trained on each task individually, the output of LLM to the evaluation method is still not post-processed, resulting in low performances of these tasks.”

May combine two claims.

Trace optimizes both DSPy's original design and outperforms their own optimizer COPRO by 10% on algorithmic tasks.

“Trace optimizes both DSPy's original design and outperforms their own optimizer COPRO by 10% on algorithmic tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

All algorithms in Trace see the same graph, which is slightly different from the graph of TextGrad.

“All algorithms in Trace see the same graph, which is slightly different from the graph of TextGrad.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The real program and the traced execution graph are different.

“Note that the real program and the traced execution graph are different.”

Probably not a substantial claim: may not pass S4 (based).

Trace encapsulates all Python datatypes as node, including numbers, floats, strings, etc.

“Trace encapsulates all Python datatypes as node, including numbers, floats, strings, etc.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The problem horizon is set to 10 steps because expert policies implemented in LLF-Bench can solve these problems within 10 steps.

“We set the problem horizon to be 10 steps, as we find the expert policies implemented in LLF-Bench can solve these problems within 10 steps.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The numbers in the original paper cannot be reproduced exactly despite using the released TextGrad code.

“The numbers in the original paper cannot be reproduced exactly despite using the released TextGrad code.”

May combine two claims.

Trace automatically constructs a computation graph and represents the optimization problem as a debugging report.

“Trace automatically constructs a computation graph and represent the optimization problem as a debugging report.”

Classified as probably not a substantial claim.

The task name alone, except for Reach, is too vague even for humans to understand how to read the observation dictionary is related to the problem.

“We find this task name alone, except for Reach, is too vague even for humans to understand how to read the observation dictionary is related to the problem”

Probably not a substantial claim: may not pass S3 (load-bearing).

The current implementation of Trace is in Python.

“The current implementation is in Python.”

May combine two claims.

Propagating the execution trace subsumes back-propagation for differentiable workflows, and remains applicable even for non-differentiable workflows.

“We show that propagating the execution trace subsumes back-propagation for differentiable workflows, and remains applicable even for non-differentiable workflows.”

Probably not a substantial claim: may not pass S4 (based).

Trace, OPTO, and OptoPrime together provide the first tractable algorithm for optimizing general computational workflows end-to-end.

“Trace, OPTO, and OptoPrime together provide the first tractable algorithm for optimizing general computational workflows end-to-end.”

May combine two claims.

Computational workflow optimization is usually framed as a black-box or algorithm configuration problem, and is tackled by general techniques like Bayesian Optimization, Evolutionary Algorithms, Reinforcement Learning using scalar scores as feedback.

“Due to its complexity, computational workflow optimization is usually framed as a black-box [6] or algorithm configuration [7] problem, and is tackled by general techniques like Bayesian Optimization [8], Evolutionary Algorithms [9], Reinforcement Learning (RL) [10] using scalar scores as feedback.”

• cites [6]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [7]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [8]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [9]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [10]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

One observation of scalar feedback alone does not provide an improvement signal, so these algorithms are very inefficient when the parameter space is large (e.g. codes or natural language prompts).

“But one observation of scalar feedback alone does not provide an improvement signal, so these algorithms are very inefficient when the parameter space is large (e.g.codes or natural language prompts).”

May combine two claims.

Execution traces enable end-to-end generative optimization, because they provide the information needed to relate rich feedback to the parameters in general workflows.

“Our novel insight is that execution traces also enable end-to-end generative optimization, because they provide the information needed to relate rich feedback to the parameters in general workflows.”

May combine two claims.

Workflow execution traces are akin to back-propagated gradients in AutoDiff and can provide key information to interpret feedback for efficient optimization.

“We discover that workflow execution traces are akin to back-propagated gradients in AutoDiff and can provide key information to interpret feedback for efficient optimization.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Since MSP only involves merging priority queues of references, most actual computation happens in the forward pass (and also the optimizer's step method).

“Since MSP only involves merging priority queues of references, most actual computation happens in the forward pass (and also the optimizer's step method).”

Classified as probably not a substantial claim.

The OPTO framework captures an abstraction of rich feedback called trace feedback (the execution trace and the output feedback).

“The OPTO framework captures an abstraction of rich feedback, called trace feedback (the execution trace and the output feedback), but it requires specification of operator descriptions and output feedback source to guide the optimization effectively.”

May combine two claims.

Black-box optimization techniques can use numerical feedback to optimize heterogeneous parameters, though they are inefficient.

“Black-box optimization techniques (e.g.Reinforcement Learning) can use numerical feedback to optimize heterogenous parameters (e.g.codes, hyper-parameters as well as tensors), though they are inefficient.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Back-propagation has time and space complexities of O(W N d^2) and O(d) respectively, where d is the maximal dimension of tensors.

“By contrast, back-propagation has time and space complexities of O(W N d 2 ) and O(d) respectively, where d is the maximal dimension of tensors.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Despite the worst case complexity of MSP, in practice the difference is negligible.

“Despite the worst case complexity of MSP, in practice the difference is negligible.”

May combine two claims.

In the most general setting of computational graphs and feedback, the propagated feedback does not have a constant size and needs the full subgraph.

“The difference is because in the most general setting of computational graphs and feedback, the propagated feedback (no matter how it is represented) does not have a constant size and needs the full subgraph.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

OPTO can be more efficiently solvable than black-box problems only when the trace feedback provides information beyond reward signals; otherwise, information-wise, OPTO is no easier than black-box problems.

“OPTO can be more efficiently solvable than black-box problems only when the trace feedback provides information beyond reward signals; otherwise, information-wise, OPTO is no easier than black-box problems.”

Classified as probably not a substantial claim.

In OptoPrime, optimization is connected to an LLM's reasoning capability.

“In OptoPrime, we connect optimization to an LLM's reasoning capability.”

Classified as probably not a substantial claim.

AutoDiff for deep neural networks uses numerical feedback to optimize numerical parameters.

“In contrast, AutoDiff for deep neural networks uses numerical feedback (e.g.rewards or loss functions) to optimize numerical parameters (e.g.tensors).”

Probably not a substantial claim: may not pass S4 (based).

Through the OPTO framing, Trace separates the design of optimizers and domain-specific components so that general-purpose optimizers can be built that work across diverse domains.

“Through the OPTO framing, Trace separates the design of optimizers and domain-specific components so that general-purpose optimizers can be built that work across diverse domains.”

Probably not a substantial claim: may not pass S3 (load-bearing).

TextGrad shares the same goal of end-to-end optimizing AI workflows as Trace.

“After the submission of our work to NeurIPS, another AutoDiff-like framework, TextGrad [24], was released, which shares the same goal of end-to-end optimizing AI workflows as Trace.”

• cites [24]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

One core challenge of designing an LLM-based OPTO optimizer is how to represent the execution trace subgraph g to LLMs, in a way that LLMs can understand and reason about the downstream effects of parameter updates.

“One core challenge of designing an LLM-based OPTO optimizer is how to represent the execution trace subgraph g (which can involve various graph structures and heterogenous data) to LLMs, in a way that LLMs can understand and reason about the downstream effects of parameter updates.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

GP and PSO appear bad because 50 iterations are insufficient for their convergence; given enough iterations, both will eventually perform well.

“GP and PSO appear bad because 50 iterations are insufficient for their convergence; given enough iterations, both will eventually perform well.”

May combine two claims.

OPRO does not use the execution trace but relies on the memory of parameter and feedback pairs.

“We implement the state-of-the-art LLM optimizer OPRO [13] as a baseline; in comparison with OptoPrime, OPRO does not use the execution trace but relies on the memory of parameter and feedback pairs.”

• cites [13]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The Trace framework leverages a workflow's structure and can incorporate rich feedback beyond scores, extending AutoDiff to complicated, non-differentiable computational workflows.

“The Trace framework a) leverages a workflow's structure and b) can incorporate rich feedback beyond scores, extending AutoDiff to complicated, non-differentiable computational workflows.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Not all details matter in optimization, so exposing every low-level operator can make the graph unnecessarily cluttered.

“On the other hand, not all details matter in optimization, so exposing every low-level operator can make the graph unnecessarily cluttered.”

Classified as probably not a substantial claim.

The experimental design makes the comparison fair by allowing each optimizer to access the same LLM APIs around the same time.

“This experimental design makes the comparison fair by allowing each optimizer to access the same LLM APIs around the same time, and showcases the flexibility of Trace framework to optimize any computational workflow.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Different abstraction choices trade off the graph complexity and the description needed for each operator.

“Different abstraction choices trade off the graph complexity and the description needed for each operator.”

Classified as probably not a substantial claim.

Only allowing the change to one part of a workflow but not others can be limiting for building self-adapting agents that can modify their own behavior.

“However, for building self-adapting agents that can modify their own behavior, only allowing the change to one part of a workflow but not others can be limiting.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In Trace, the chosen propagator (MSP) maximally preserves information for a general computation graph.

“In Trace, we chose a specific propagator (MSP), which maximally preserves information for a general computation graph.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The best representation depends on the application and OPTO optimizer at hand.

“The best representation depends on the application and OPTO optimizer at hand.”

May combine two claims.

TextGrad propagates text feedback, whereas Trace propagates minimal subgraphs.

“In comparison, TextGrad propagates text feedback, whereas Trace propagates minimal subgraphs (see Section 3.3).”

• cites [24]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Providing details of all operators in g through the context ω will require manually crafting a context for every workflow.

“However, this will require manually crafting a context for every workflow.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In theory Trace can run asynchronously so long as the overall graph does not end up with cycles.

“Also, Trace cannot convert all computational workflows into OPTO problems, e.g., stateful functions that modify their state in place cannot be represented as a DAG without modification, and distributed/parallel computing workflows are incompatible with the current implementation (though in theory Trace can run in an asynchronous way so long as the overall graph does not end up with cycles).”

May combine two claims.

Trace is able to learn a sophisticated control logic in dozens of interactions, working on both training and held-out testing initial conditions.

“The experimental results here are quite impressive, showing that Trace is able to learn a sophisticated control logic in dozens of interactions, not only working on the training initial conditions but also on held-out testing ones too.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

For a single problem, there is no difference in principle; one can choose to provide details of all operators in g through the context ω.

“For a single problem, there is no difference in principle; one can choose to provide details of all operators in g through the context ω.”

Classified as probably not a substantial claim. May combine two claims.

Abstracting everything into a single operator makes a trivial graph but requires more descriptions to faithfully capture the workflow.

“Abstracting everything into a single operator makes a trivial graph but requires more descriptions to faithfully capture the workflow.”

Probably not a substantial claim: may not pass S3 (load-bearing).

SAMMO represents the prompt parameter itself as a program to enable more efficient black-box search through the space of programs.

“SAMMO represents the prompt parameter itself as a program so as to enable more efficient black-box search through the space of programs.”

• cites [14]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

[37] frames LLM systems as a graph where nodes are operations and edges are messages/connections, and optimizes for the connection on edges by REINFORCE using scalar reward feedback and prompts by LLMs.

“[37] frames LLM systems as graph where nodes are operations and edges are messages/connections. (Note that this is different from the DAG used in Trace; here nodes are messages and edges are input-output of operators) and optimizes for the connection on edges (binary variables) by REINFORCE using scalar reward feedback and prompts by LLMs .”

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

DSPy can optimize directly the prompts or the few-shot examples to include using scalar reward feedback.

“DSPy [22] can optimize directly the prompts or the few-shot examples to include using scalar reward feedback.”

• cites [22]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

SAMMO is an example for prompts that uses additional graph structure to make the optimizer efficient.

“SAMMO [14] is an example for prompts that uses additional graph structure to make the optimizer efficient.”

• cites [14]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

OptoPrime has difficulty handling parameters or nodes that cannot be compactly represented in text, which prevents it from optimizing neural network weights or reasoning with large, stateful objects like a database.

“It has difficulty in handling parameters or nodes that cannot be compactly represented in text, which prevents it from optimizing neural network weights, or reasoning with large, stateful objects like a database.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Recent works use LLMs to optimize numerical hyperparameters as an alternate to Bayesian optimization.

“Recent works like [43,44] use LLMs to optimize numerical hyperparameters, as an alternate to Bayesian optimization.”

• cites [43]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [44]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The gradient can be computed in a forward mode, and there are techniques of Checkpointing and Truncated Back-Propagation approximation for efficiency.

“For example, the gradient can be computed in a forward mode (Forward Mode Differentiation) as well, and there are also techniques of Checkpointing [46] and Truncated Back-Propagation approximation [47] for efficiency.”

• cites [46]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [47]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Backpropagation (Backward Mode Differentiation) is not the only AutoDiff algorithm.

“Nonetheless, we highlight that backpropagation (Backward Mode Differentiation) is not the only AutoDiff algorithm.”

Probably not a substantial claim: may not pass S3 (load-bearing).

These frameworks support tracing of the workflow to aid in profiling, debugging and visualization.

“They support tracing of the workflow to aid in profiling, debugging and visualization.”

May combine two claims.

These LLM-based optimizers, including OPRO, often propose only principles of how prompts should be designed and require crafting problem-specific prompts.

“In addition, these LLM-based optimizers, including OPRO [13], often propose only principles of how prompts should be designed and requires crafting problem specific prompts (as opposed to a single optimization prompt that can be applied to different problems).”

• cites [13]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

These algorithms focus on optimizing prompts.

“These algorithms focus on optimizing prompts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There is a huge and fast growing literature on using LLMs as optimizers to improve prompts or codes.

“There is a huge and fast growing literature on using LLMs as optimizers to improve prompts [12,11,[38][39][40] or codes [41,15,42,38].”

• cites [12,11,[38][39][40]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [41]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [15]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [42]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [38]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Through the OPTO framing, Trace supports joint optimization of all parameters (prompts, hyperparameters, codes) with rich feedback, and is agnostic to graph structures.

“In contrast to these works, through the OPTO framing, Trace supports joint optimization of all parameters (prompts, hyperparameters, codes) with rich feedback, and is agnostic to graph structures (e.g., changing these parameters can dynamically change the graph structure and connections between nodes).”

Probably not a substantial claim: may not pass S3 (load-bearing).

OPTO can be viewed as a special case of Learning from Language Feedback (LLF) setup defined in [23] with observations as the trace feedback.

“On the other hand, OPTO can be also viewed as a special case of Learning from Language Feedback (LLF) setup defined in [23] with observations as the trace feedback.”

• cites [23]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

This is a framing of a meta LLF problem.

“This is a framing of a meta LLF problem.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Our design of Trace is inspired by back-propagation and the ease of use of the AutoDiff framework PyTorch.

“Our design of Trace is inspired by back-propagation and the ease of use of the AutoDiff framework PyTorch [19].”

• cites [19]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Deep Language Networks view all of the prompts in an LLM workflow as tunable parameters and jointly optimizes them.

“Deep Language Networks [36] view all of the prompts in an LLM workflow as tunable parameters and jointly optimizes them.”

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

These LLM-based optimizers do not consider optimizing prompts or codes as one component of a bigger workflow, which is harder and requires the right credit assignment.

“They do not consider optimizing prompts or codes as one component of a bigger workflow (e.g.implementing an autonomous agent), which is harder and requires the right credit assignment.”

May combine two claims.

[37]'s prompt optimization part does not take output feedback, but uses an LLM to self-check whether the prompt meets the need of generating desired functions the user specified.

“Their prompt optimization part also does not take output feedback, but simply use an LLM to self-check whether the prompt meets the need of generating desired functions the user specified.”

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Back-propagation has been shown to be a very effective tool in optimizing differential computational workflows.

“Back-propagation has been shown to be a very effective tool in optimizing differential computational workflows.”

May combine two claims.

Trace is designed to be general-purpose and agnostic to the underlying frameworks of computational workflows users choose.

“Moreover, Trace is designed to be general-purpose and agnostic to the underlying frameworks of computational workflows users choose.”

Classified as probably not a substantial claim.

Deep Language Networks discovered that optimizing each parameter in isolation produces subpar results.

“They discovered that optimizing each parameter in isolation instead produces subpar results.”

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Retroformer uses another small language model to provide suggestions/feedback to improve the behavior of an actor LLM, where the small LM is tuned by offline RL.

“Retroformer [35] uses another small language model (LM) to provide suggestions/feedback (i.e.changing prompts) to improve the behavior of an actor LLM, where the small LM is tuned by offline RL.”

• cites [35]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

If there exists a latent loss function that the feedback f adheres to, those OPTO instances can be written as partial monitoring game.

“If there exists a latent loss function that the feedback f adheres to (e.g. as in [23]), those OPTO instances can be written as partial monitoring game.”

• cites [23]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The main difference between Trace and the aforementioned hyperparameter optimization work is the representation of the problem.

“The main difference between Trace and the aforementioned work is the representation of the problem.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Trace can also be applied to optimize trivial OPTO problems where the returned graph has just a single node of the parameter.

“Trace can also be applied to optimize trivial OPTO problems where the returned graph has just a single node of the parameter (which are the scenarios considered by these works).”

Classified as probably not a substantial claim.

[37] optimizes each component separately without considering each other.

“They optimize each component separately without considering each other; for example, the prompts are optimized individually without considering the graph topology or how they are used down the road.”

• cites [37]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

There are multiple efforts to optimize the computational graph of LLM workflows, which is a special case of the OPTO problem.

“There are multiple efforts to optimize the computational graph of LLM workflows, which is a special case of the OPTO problem.”

May combine two claims.

Frameworks such as LangChain, Semantic Kernels, AutoGen, and DSPy allow composing computational workflows and provide hand-engineered optimizers to tune an LLM's context using scalar feedback with black-box search techniques.

“Frameworks such as LangChain [29], Semantic Kernels, AutoGen [5], DSPy [22] allow composing computational workflows and provide handengineered optimizers to tune an LLM's context (i.e.prompt templates, few shot examples, or tool libraries) using scalar feedback with black-box search techniques.”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [5]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [22]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims. The quote only partly states this claim.

Trace achieves efficient optimization as the graph becomes nontrivial and for diverse applications by using the abstract OPTO problem framing.

“Trace achieves this by using the abstract OPTO problem framing.”

Probably not a substantial claim: may not pass S3 (load-bearing).

OPTO admits a more general notion of feedback f than partial monitoring games.

“However OPTO admits a more general notion of feedback f , and we discuss solution concepts for them in Appendix F.”

May combine two claims.

Trace uses tracing for automatic optimization and constructs a different representation of the computational graph suited for that purpose.

“In contrast, Trace uses tracing for automatic optimization, and constructs a different representation of the computational graph which is suited for that purpose.”

Probably not a substantial claim: may not pass S4 (based).

The verification of a solution depends only on the output of computation.

“That is, the verification of a solution depends only on the output of computation.”

May combine two claims.

Without the execution trace, tabular RL has exponential complexity due to the size of the policy space; using the execution trace, tabular RL can be solved approximately in polynomial time.

“Without the execution trace (i.e., not seeing the Markovian structure and trajectories), the problem has an exponential complexity (due to the size of the policy space) and we know by using the execution trace here, tabular RL can be solved approximately in polynomial time [49].”

• cites [49]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

The formulation of the Trace Oracle abstracts the details of a computational workflow, so problems from different domains can be framed in the unified framework.

“The formulation of the Trace Oracle abstracts the details of a computational workflow, so problems from different domains can be framed in the unified framework.”

May combine two claims.

OPRO is suboptimal not due to a token limit but instead a lack of information, which is captured and represented using Trace.

“OPRO is suboptimal not due to a token limit but instead a lack of information, which is captured and represented using Trace.”

Classified as probably not a substantial claim.

Using information in the execution trace is the key to unlock efficient OPTO.

“From these observations, we conjecture using information in the execution trace is the key to unlock efficient OPTO.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

OPTO covers a wide range of complicated optimization problems.

“We show that OPTO covers a wide range of complicated optimization problems.”

Probably not a substantial claim: may not pass S2 (checkable).

This abstraction allows designing the computational tool Trace for various applications.

“This abstraction allows us to design the computational tool Trace for various applications.”

Probably not a substantial claim: may not pass S4 (based).

Search problems can be NP-hard.

“However search problems can be NP-hard.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The plots can roughly be interpreted as how long it takes to find a working policy.

“Therefore, the plots can roughly be interpreted as how long it takes to find a working policy.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The generality of OPTO raises fundamental questions such as if OPTO is well defined and when OPTO can be efficiently solved.

“However, the generality of OPTO also raises some fundamental questions, such as if OPTO is well defined and when OPTO can be efficiently solved.”

May combine two claims.

Without the execution trace, training neural networks is a complex black-box optimization with a loss value and no gradients; an execution trace allows implementation of back-propagation to compute gradients at the parameters.

“Without the execution trace, we have a complex black-box optimization with a loss value, without gradients, whereas an execution trace allows implementation of back-propagation to compute the gradients at the parameters.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The rollout length in the RL problem can be randomly determined.

“Similarly, the rollout length of in the RL problem can be randomly determined.”

Probably not a substantial claim: may not pass S2 (checkable).

Classical mathematical optimization problems have a problem definition which itself is the solution concept.

“Classical mathematical optimization problems have a problem definition which itself is the solution concept.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There are clearly problems where no solution exists (no parameter in Θ can be verified by the verification oracle).

“There are clearly problems where no solution exists (that is, no parameter in Θ can be verified by the verification oracle).”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

These questions stem from the generality of the context ω and the output feedback f in OPTO, since they can be anything describable as texts.

“These questions stem from its generality of the context ω and the output feedback f in OPTO, since e.g. they can be anything descriable texts.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In a minimization problem, the solution concept is to find the minimum of an objective function; in an equilibrium problem, the solution concept is finding a point/set satisfying an equilibrium inequality.

“For example, in a minimization problem, it is clear we want to find the minimum of an objective function; even for problems as abstract and general as an equilibrium problem, the problem setup clearly states the solution concept of finding a point/set satisfying an equilibrium inequality [48].”

• cites [48]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

In an OPTO problem, by varying the context ω, the desired parameter can change from one extreme to another.

“By contrast, in a OPTO problem (Θ, ω, T ), by varying the context ω, the desired parameter can change from one extreme to another.”

Probably not a substantial claim: may not pass S4 (based).

OPTO is efficiently solvable when the context and the trace feedback provide information to construct a corrective search direction.

“More precisely, we conjecture that OPTO is efficiently solvable when the context and the trace feedback need to provide information to construct a corrective search direction.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Trace abstracted away the different data types and enabled direct update and optimization of them.

“Trace abstracted away the different data types and enabled direct update and optimization of them.”

Classified as probably not a substantial claim.

The verification oracle in Axiom 1 is not limited to just algorithms.

“Notice the verification oracle in Axiom 1 is not limited to just algorithms.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

This assumption rules out problems where the feedback is adversary to the context, and makes solving OPTO a well-defined search problem.

“This assumption would rule out problems, e.g., where the feedback is adversary to the context, and makes solving OPTO is a well-defined search problem.”

• AssumptionThe authors state this as an assumption: For an OPTO problem, there exists at least one parameter that the verification oracle can verify as a solution.

May combine two claims.

This error signal is caught by Trace and can provide early feedback to efficiently improve the parameters, since the graph is truncated at the error.

“This error signal would be caught by Trace and can then provide early feedback to efficiently improve the parameters, since the graph is truncated at the error.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Fully characterizing the properties of OPTO is beyond the scope of this paper and would require years of future research.

“Fully characterize the properties of OPTO, due to its generality, is beyond the scope of this paper and would require years of future research to come.”

Probably not a substantial claim: may not pass S3 (load-bearing).

PSO appears bad in traffic control because 10 iterations is insufficient for its convergence.

“PSO appears bad because 10 iterations is insufficient for its convergence.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Out of 23 Big-Bench Hard tasks, 14 require a multiple-choice answer, 4 require yes/no, 1 requires True/False, 1 requires valid/invalid, and 3 require answers that contain words or numbers.

“Out of 23 tasks, 14 tasks require a multiple-choice answer with options provided in the question.4 of them require yes/no. 1 task requires True/False, while 1 task requires valid/invalid.And the 3 remaining tasks require answers that contain words or”

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Verifiability is only defined with respect to the context ω and the output feedback f, not the execution trace g.

“Lastly we note the verifiability is only defined with respect to the context ω and the output feedback f , not the execution trace g.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

One common pattern of classical optimization problems is that the solution concept can be described as conditions on feedback that the parameter should satisfy.

“One common pattern of these problems is that the solution concept is something that can be described as conditions on feedback that the parameter should satisfy.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The length of the execution trace depends on how far the code executes.

“The length of the execution trace, e.g., in the debugging example above depends on how far the code executes.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The computational graph g returned by the Trace Oracle T may have different graph structures.

“As mentioned, the computational graph g returned by the Trace Oracle T may have different graph structures.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Means and standard errors are computed over 10 random seeds.

“Means and standard errors are computed over 10 random seeds.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Trace allows us to unify prompt optimization and code generation, which enables the creation of agents capable of fast learning.

“Trace allows us to unify prompt optimization and code generation, which enables the creation of agents capable of fast learning.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

We currently do not have algorithms intelligent enough to process the wide range of contexts and feedback that OPTO allows.

“This is intentional because we currently do not have algorithms that are intelligent enough to process the wide range of contexts and feedback that OPTO allows.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A multi-output operator can always be modeled by a single-output operator and single-output indexers.

“A multi-output operator can always be modeled by a single-output operator and single-output indexers.”

Classified as probably not a substantial claim.

The difference in computation costs between the two strategies can be large, especially when not all operators in the graph are as complex as querying LLM.

“The difference in terms of computation costs between the two strategies can be large, especially when not all operators in the graph are as complex as querying LLM.”

Classified as probably not a substantial claim.

The paper extends OPRO to use parameter-feedback pairs for the more general setup of OPTO.

“Since we're interested in the more general setup of OPTO, we extend it to use parameter-feedback pairs.”

• AssumptionRests on an assumption: The original version of OPRO uses parameter-score pairs.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Supplementary material provides the exact code that was run to produce all results, and only requires user to supply an OpenAI API key.

“Supplementary material additionally provides the exact code that was run to produce all results, and only requires user to supply an OpenAI API key.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

All of the experiments described in the paper are reproducible with the supplied code.

“All of the experiments described in the paper are reproducible with the supplied code.”

Classified as probably not a substantial claim. May combine two claims.

To build a self-adapting agent with Trace, one only needs to annotate some empty functions (reason, act) and set up an optimizer following PyTorch semantics.

“To build a self-adapting agent with Trace, we only need to annotate some empty functions (reason, act) and set up an optimizer following PyTorch semantics.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The optimizer used in Table 2 is GPT-4o-2024-08-06, and the student model is GPT-35-turbo-1106.

“The optimizer is GPT-4o-2024-08-06, and the student model is GPT-35-turbo-1106.”

Probably not a substantial claim: may not pass S4 (based).

The bundle functionality of Trace is more flexible in representing computations to an LLM than pre-defined templates used in other optimization frameworks.

“The bundle functionality of Trace is more flexible in representing computations to an LLM than pre-defined templates used in other optimization frameworks.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

All of the code to implement Trace and OptoPrime are provided in the supplementary material.

“All of the code to implement Trace and OptoPrime are provided in the supplementary material.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In each iteration, one episode of rollout (10 steps) is performed, and then the policy is updated.

“In each iteration (x-axis), one episode of rollout (10 steps) is performed, and then the policy is updated.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The DAG-based design of Trace can be adapted to other programming languages also, just like AutoDiff libraries that are available in C++.

“The DAG-based design of Trace can be adapted to other programming languages also, just like AutoDiff libraries that are available in C++.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

The mean and standard error of the success rate over 10 seeds are shown for the Sawyer manipulator experiments.

“The mean and standard error of the success rate over 10 seeds are shown.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The experiment time reported in Table 2 is in minutes, involving training, validation and testing by running TextGrad's original pipeline; the time of GSM8K experiment is omitted as the experiment time (>8hrs) is determined primarily by the evaluation not optimization.

“The experiment time reported is in minutes (the time involves not just training but also validation and testing by running TextGrad's original pipeline); the time of GSM8K experiment is omitted as the experiment time (>8hrs) is determined primarily by the evaluation not optimization.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

All experiments include standard error bars from ≥ 10 replications.

“All experiments include standard error bars from ≥ 10 replications.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The results in Table 2 show the mean and the standard error of success rate of the last iterate computed by 5 seeds.

“The results show the mean and the standard error of success rate of the last iterate computed by 5 seeds.”

May combine two claims.

Running numerical optimization experiments, or optimizing code alongwith prompts and hyper-parameters can be challenging to set up in libraries that restrict variables to strings.

“consequently running numerical optimization experiments, or optimizing code alongwith prompts and hyper-parameters can be challenging to set up.”