CoChat
Sign Up
PreprintCited by 69Topic ModelingEvolutionary Algorithms and ApplicationsNatural Language Processing Techniques

Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution

Open access
Fernando, Chrisantha, Banarse, Dylan, Michalewski, Henryk, Osindero, Simon, Rocktäschel, Tim·2023·arXiv
DOI 10.48550/arxiv.2309.16797arXiv 2309.16797
Open this paper in CoChat — free accountOpen PDF

Abstract

Popular prompt strategies like Chain-of-Thought Prompting can dramatically improve the reasoning abilities of Large Language Models (LLMs) in various domains. However, such hand-crafted prompt-strategies are often sub-optimal. In this paper, we present Promptbreeder, a general-purpose self-referential self-improvement mechanism that evolves and adapts prompts for a given domain. Driven by an LLM, Promptbreeder mutates a population of task-prompts, and subsequently evaluates them for fitness on a training set. Crucially, the mutation of these task-prompts is governed by mutation-prompts that the LLM generates and improves throughout evolution in a self-referential way. That is, Promptbreeder is not just improving task-prompts, but it is also improving the mutationprompts that improve these task-prompts. Promptbreeder outperforms state-of-the-art prompt strategies such as Chain-of-Thought and Plan-and-Solve Prompting on commonly used arithmetic and commonsense reasoning benchmarks. Furthermore, Promptbreeder is able to evolve intricate task-prompts for the challenging problem of hate speech classification.

AI snapshot

Written from the paper’s full text.

TL;DR

Promptbreeder is a self-referential evolutionary system that uses an LLM to evolve task-prompts and the mutation-prompts that mutate them, without parameter updates. It outperforms state-of-the-art prompt strategies on arithmetic, commonsense, and hate-speech benchmarks.

Key findings

  • On PaLM 2-L, Promptbreeder (PB) zero-shot scores were MultiArith 99.7, SingleEq 96.4, AddSub 87.8, SVAMP 90.2, SQA 71.8, CSQA 85.4, AQuA-RAT 62.2, and GSM8K 83.9; few-shot scores were 100.0, 98.9, 87.1, 93.7, 80.2, 85.9, 64.6, and 83.5, respectively. It surpasses Plan-and-Solve PS+ PaLM 2-L on all datasets except AddSub, and beats OPRO’s 80.2% on GSM8K with 83.9% zero-shot.
  • On ETHOS hate speech classification, PB evolved two sequential prompts scoring 89%, versus 80% for the hand-designed prompt “Determine whether a text contains hate speech.” With the under-specified problem “Solve the Problem,” PB scored 81.6%; removing the Lamarckian operator dropped it to 64.6%, and removing several initialization/hypermutation components dropped it to 68.7%.
  • Ablations showed removing any self-referential operator was harmful under nearly all circumstances, with initial re-description of task-prompts giving the greatest benefit; one exception was that randomly drawing mutation-prompts upon initialization hurt GSM8K. Mutation-operator improvement rates on GSM8K were Zero-order Hyper-Mutation 42%, Lineage Based Mutation 26%, First-order Hyper-Mutation 23%, EDA Rank and Index 12.7%, Direct Mutation 12%, EDA Mutation 10.7%, and Lamarckian Mutation 6.3%.
  • On the 24 APE instruction-induction tasks, PB matched or surpassed APE on 21 out of 24 tasks by up to 21%.

Methods

  • Promptbreeder runs a binary tournament genetic algorithm over a population of 50 units, typically for 20–30 generations (or 1–2k fitness evaluations, described as 20–40 generations for populations of 50). Each unit contains typically two task-prompts, a mutation-prompt, and in the few-shot case a set of correct workings out; fitness is accuracy on a randomly sampled batch of 100 training Q&A pairs.
  • The LLM acts as the mutation operator: a task-prompt is mutated via P′ = LLM(M + P), and a mutation-prompt is hypermutated via M′ = LLM(H + M). Nine mutation operators in five classes are used: direct mutation, estimation-of-distribution mutation, hypermutation, Lamarckian mutation, and prompt crossover/context shuffling.
  • Main experiments use PaLM 2-L and compare against Chain-of-Thought, Zero-shot CoT, Program-of-Thoughts, Auto-CoT, OPRO, APE, Plan-and-Solve PS/PS+, and Few-Manual-CoT on GSM8K, SVAMP, MultiArith, AddSub, AQuA-RAT, SingleEq, CSQA, StrategyQA, ETHOS, and instruction-induction tasks. For datasets marked with asterisks, half the examples were used for training and accuracy reported on the remaining test set.

Limitations

  • Stated: PB’s prompting topology remains fixed—it adapts prompt content but not the prompting algorithm itself. It also does not invent new auxiliary ways of evaluating mutants; only the externally specified fitness function is used throughout. In few-shot evolution, contexts dominate and task-prompts often drift into nonsense, becoming less critically determining of fitness than the evolved contexts. Self-referential recursive prompting can diverge, derail, or get stuck in attractors if unconstrained.
  • Stated: PB is limited compared with human thought, which involves intonation, imagery, and multimodal processes, and the paper leaves open how to develop more complex thought strategies.
  • Evident from the text: some comparisons are not fully head-to-head because baselines use different underlying LLMs (e.g., text-davinci-003 versus PaLM 2-L), and several baseline entries in Table 1 are missing. The paper also states that comparisons with PoT, PS, and Auto-CoT controls using their model are not provided because PS and PS+ were the best prompts in Plan-and-Solve.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2309.16797

Cites

0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

10 of 14 cited sources checked · 7 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    The specific way a prompt is phrased can have a dramatic effect on its utility.

    “Since the specific way a prompt is phrased can have a dramatic effect on its utility (Madaan & Yazdanbakhsh, 2022), it raises the question of whether prompt engineering can be automated.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Madaan & Yazdanbakhsh, 2022 source
  • Premisecitation

    LLMs are effective at generating mutations from examples.

    “Promptbreeder makes use of the observation by Lehman et al. (2022), Meyerson et al. (2023) and Chen et al. (2023) that LLMs are effective at generating mutations from examples.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Lehman et al., 2022 source
    • cites Meyerson et al., 2023 source

      Section 4.1 reports that an LLM reliably generated novel, valid offspring from as few as three examples, supporting the claim that examples can elicit effective variation. But Section 2.2 defines mutation as generating from one parent and crossover as generating from multiple parents; this paper tests the latter, not mutation specifically. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Chen et al., 2023 source
  • Premisecitation

    Promptbreeder makes use of the observation that LLMs can be used to generate variations of input text.

    “Promptbreeder makes use of the observation that LLMs can be used to generate variations of input text (Lehman et al., 2022;Meyerson et al., 2023;Chen et al., 2023).”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Lehman et al., 2022 source
    • cites Meyerson et al., 2023 source
    • cites Chen et al., 2023 source
  • Findingown data

    Over 2000 evaluations, the fitness increases considerably.

    “Over 2000 evaluations, the fitness increases considerably.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Adding random mutation prompts brings performance back up to 73.7%.

    “Adding back online mutation increases performance back to 70.4% and adding random mutation prompts brings this back up to 73.7%.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 15
  • May combine two claims. 19
  • The quote only partly states this claim. 3

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Promptbreeder is an example of a system transitioning from

    “As Promptbreeder is proposing mutated prompts to itself, it is an example of a system transitioning from”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Jiang et al., 2022 source

      The cited paper discusses open-ended exploration and systems that generate learning challenges (Introduction, p. 4; §4.2.1, p. 10), but the passages do not address Promptbreeder or mutated prompts. Searches for those terms found no supporting passage. The citing sentence also ends at “transitioning from,” so the proposed transition cannot be assessed. (LLM, conf=0.74)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Different prompt strategies can have a significant impact on a model's reasoning abilities.

    “For example, different prompt strategies 1 can have a significant impact on a model's reasoning abilities (Wei et al., 2022;Nye et al., 2021;Zhou et al., 2022;Wang et al., 2022;Zhou et al., 2023;Wang et al., 2023b), multimodal processing abilities (Yang et al., 2023b;Wang et al., 2023d), or tool use abilities (Yao et al., 2022;Schick et al., 2023).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Wei et al., 2022 source
    • cites Nye et al., 2021 source
    • cites Zhou et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Pryzant et al. use a workingout-conditioned error detection and error fixing prompt to improve the task specification prompt, a self-referential process similar to our use of the Lamarckian operator.

    “Pryzant et al use a workingout-conditioned error detection and error fixing prompt to improve the task specification prompt, a self-referential process similar to our use of the Lamarckian operator.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Pryzant et al., 2023 source

      Section 2.2.1 (page unavailable) describes using a prompt to identify problems behind observed errors, then a second prompt to edit the task prompt in a recursive feedback loop. The appendix’s “Generating gradients” section shows that these prompts use the current prompt and examples it gets wrong; our copy does not establish that they are conditioned on a “workingout.” (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    In most cases all the mutation operators have a positive impact on fitness, with the Random Initial Prompts having the largest positive impact across all datasets.

    “The results in Figure 4 show that in most cases all the mutation operators have a positive impact on fitness, with the Random Initial Prompts having the largest positive impact across all datasets.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Promptbreeder evolved a prompt strategy scoring 89% on ETHOS, an improvement over the hand-designed prompt scoring 80%.

    “Promptbreeder was able to evolve a prompt strategy consisting of two sequentially applied relatively long prompts (see Appendix J.1) that scored 89% on ETHOS-an improvement over the hand-designed prompt”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Removing any self-referential operator is harmful under nearly all circumstances, with the greatest benefit being the initial re-description of task-prompts upon initialization.

    “Removing any self-referential operator is harmful under nearly all circumstances, the greatest benefit being the initial re-description of task-prompts upon initialization.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    If the output of the LLM is simply fed back into itself with no other context, these failure cases are observed, with higher sampling temperatures favouring escape from attractors.

    “If the output of the LLM is simply fed back into itself with no other context, then we observe these failure cases with higher sampling temperatures favouring escape from attractors.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Promptbreeder achieved a score of 81.6% on the ETHOS hate speech detection dataset with the under-specified problem specification 'Solve the Problem'.

    “Promptbreeder achieved a score of 81.6%.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Promptbreeder surpasses OPRO's 80.2% on GSM8K with 83.9% in the zero-shot setting using the prompt

    “At the time of its release, OPRO achieved a score of 80.2% via the optimized zero-shot prompt”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    All mutation operators are important for Promptbreeder to work, including hypermutation operators which lead to self-referential self-improvement.

    “It demonstrates that all mutation operators are important for Promptbreeder to work, including hypermutation operators which lead to self-referential self-improvement.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    In the few-shot evolution case, the contexts dominate, and often the task-prompts drift into nonsense; they are less critically determining of fitness than the evolved contexts.

    “We find that in the few-shot evolution case, the contexts dominate, and often the task-prompts drift into nonsense.They are less critically determining of fitness than the evolved contexts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Promptbreeder was able to match or surpass the APE results on 21 out of 24 Instruction Induction tasks up to 21%, despite using an LLM that is not instruction tuned unlike text-davinci-002.

    “Unlike text-davinci-002 our LLM is not instruction tuned and yet Promptbreeder was able to match or surpass the APE results on 21 out of 24 tasks up to 21%.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The performance of PS+ is improved by using PaLM 2-L as the underlying LLM on all datasets except ADDSUB compared to text-davinci-003 results in the original paper.

    “Note that the performance of PS+ is improved by using PaLM 2-L (Anil et al., 2023) as the underlying LLM (PS+ PaLM 2-L ) on all datasets except ADDSUB compared to text-davinci-003 results in the original paper.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On all other datasets, zero-shot Promptbreeder accuracy is higher than PS+, with further improvement in the few-shot case when examples of discovered solutions are included.

    “On all other datasets, zero-shot PB accuracy is higher than PS+, with further improvement in the few-shot case when examples of discovered solutions are included with the prompts.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Promptbreeder can be seen as a mechanism to extract knowledge that LLMs already encode through a diversity of causal processes that generate prompt strategies and mutation prompts, which influence the workings out generated by the LLM at inference time.

    “Promptbreeder can be seen as a mechanism to extract this knowledge through a diversity of causal processes that generate prompt strategies as well as mutation prompts used to create variations of prompt strategies, which in turn influence the the workings out generated by the LLM at inference time .”
    Quote in the paperSubstantial claim
    • Rests on an assumption: LLMs already encode knowledge about a vast array of problems.

  • May combine two claims.

    Premisecitation

    Automatic Prompt Engineer (APE) found diminishing returns to further selection rounds as quality stabilizes after three rounds, and consequently abandoned iterative APE.

    “However, Zhou et al. found”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Zhou et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    Promptbreeder uses LLMs to self-referentially improve mutation-prompts and is able to evolve contexts as well.

    “In contrast to the two approaches above, Promptbreeder uses LLMs to self-referentially improve mutation-prompts, and it is able to evolve contexts as well.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Three diversity maintenance methods are used in cases where the system gets trapped on a local optimum: random character strings appended to the front of the prompt, fitness sharing based on BERT similarity between prompt embeddings, and sampling temperature of the mutant producing LLM initialized uniformly from 1.0 to 2.0 and mutated by addition of a uniform random number in the range -0.2, 0.2.

    “Three diversity maintenance methods are used in cases where the system gets trapped on a local optimum: 1) Random character strings (typically of length 50) are appended into the front of the prompt before it is passed into the LLM.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationreasoning

    Promptbreeder has several pathways that facilitate self-referential improvement, including initialization of prompts and mutation prompts from LLM parameters, direct and EDA mutation of offspring prompts, hyper mutation of mutation prompts, inference, and Lamarckian mutation.

    “Promptbreeder has several pathways that facilitate this self-referential improvement: (i) Initial prompts are a function of the LLM parameters (Initialization Phase).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The richer the set of pathways to facilitate self-referential improvement, the more self-referential the LLM's interaction with itself is.

    “The richer the set of pathways to facilitate this, the more self-referential the LLMs interaction with itself is.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    Promptbreeder invents new ways of generating mutants, but it does not invent new auxiliary ways of evaluating them; only the externally specified fitness function is used throughout.

    “Promptbreeder invents new ways of generating mutants, but it does not invent new (auxiliary) ways of evaluating them (as in Jaderberg et al. (2017b))-only the externally specified fitness function is used throughout.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Promptbreeder remains limited compared to the open-endedness of human thought processes.

    “PB remains limited compared to the open-endedness of human thought processes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Promptbreeder improves not only task-prompts but also the mutation-prompts that govern how task-prompts are modified, in a self-referential way.

    “Crucially, the mutation of these task-prompts is governed by mutation-prompts that the LLM generates and improves throughout evolution in a self-referential way.That is, Promptbreeder is not just improving task-prompts, but it is also improving the mutation-prompts that improve these task-prompts.”
    • Also stated in: § 6 CONCLUSION AND FUTURE WORK · plain
      “PB is self-referential in that it not only evolves task-prompts, but it also evolves mutation-prompts that govern the way PB modifies task-prompts.Thus, it is not only improving prompts but it also improves the way it is improving prompts.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    If the process of recursive prompting is unconstrained and uncontrolled, it can diverge (derailment) or get stuck in an attractor.

    “If the process is unconstrained and uncontrolled then it can diverge (derailment) or get stuck in an attractor.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Where datasets were not provided with a training/test split (MultiArith, AddSub, SingleEQ and SVAMP), the dataset was split into two equal training and test sets before the experiments were conducted.

    “Where datasets were not provided with a training/test split (MultiArith, AddSub, SingleEQ and SVAMP) the dataset was split into two equal training and test sets before the experiments were conducted.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Comparison with PoT, PS and Auto-CoT controls using our model is not provided because PS and PS+ were the best prompts in Plan-and-Solve.

    “Comparison with PoT, PS and Auto-CoT controls using our model is not provided because PS and PS+ were the best prompts in Plan-and-Solve.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Plan-and-Solve

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Since these approaches directly modify parameters of a model, it is unclear how to scale them to the increasing number of parameters in modern LLMs.

    “However, since these approaches directly modify parameters of a model, it is unclear how to scale them to the increasing number of parameters in modern LLMs.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The experiments were run until the training fitness appeared to plateau, and then the fittest individual from the whole evolutionary run was evaluated against the test set.

    “The experiments were run until the training fitness appeared to plateau.At this point the fittest individual from the whole of the evolutionary run was evaluated against the test set.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Promptbreeder's substrate of self-referential self-improvement is natural language, avoiding costly parameter updates altogether.

    “In contrast, for Promptbreeder the substrate of selfreferential self-improvement is natural language, avoiding costly parameter updates altogether.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The authors investigated the following mutation operators and mechanisms: random initial prompts, random initial mutation prompts, and prompts from context (Lamarckian).

    “We investigated the following mutation operators and mechanisms:”
    • Also stated in: § Additional information · plain
      “• Random initial prompts”
    • Also stated in: § Additional information · plain
      “• Prompts from context (Lamarckian)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The authors performed ablation to measure the impact of various self-referential components of Promptbreeder.

    “We performed ablation to measure the impact of various self-referential components of Promptbreeder.”
    Quote in the paperNot a substantial claim
31 of 31

  • The authors state this as an assumption.

    The LLM can use the signal from the lineage of elite prompts to generate better prompts.

    “The rationale for this operator is that we expect the signal of improving genotype prompts may be stronger than the signal from prompts in the current population since they provide a gradient of bad to good prompts that could be followed (assuming this signal can be used by the LLM).”

  • The paper relies on this without support.

    The signal of improving genotype prompts is stronger than the signal from prompts in the current population.

    “The rationale for this operator is that we expect the signal of improving genotype prompts may be stronger than the signal from prompts in the current population since they provide a gradient of bad to good prompts that could be followed (assuming this signal can be used by the LLM).”

  • The paper relies on this without support.

    LLMs already encode knowledge about a vast array of problems.

    “LLMs already encode knowledge about a vast array of problems.”

  • The paper relies on this without support.

    The problem specification 'Solve the Problem' is under-specified, in contrast to the standard problem specification 'Determine whether a text contains hate speech'.

    “with the under-specified problem specification”
4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • PaLM 2-L does not improve PS+ on all but AddSub

    Minor

    The PaLM 2-L PS+ result is also lower on SQA (50.1 versus 65.4) and AQuA-RAT (39.4 versus 46.0), while SingleEq is unchanged at 94.7. AddSub is not the only exception.

  • Few-shot results do not always improve on zero-shot results

    Minor

    The few-shot PB results are lower than zero-shot PB on AddSub (87.1 versus 87.8) and GSM8K (83.5 versus 83.9), contrary to the stated further improvement.

2 of 2

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 5

  1. Inman Harvey. The microbial genetic algorithm. In Advances in Artificial Life. Darwin Meets von Neumann: 10th European Conference, ECAL 2009, Budapest, Hungary, September 13-16, 2009, Revised Selected Papers, Part II 10, pp. 126-133. Springer, 2011.
  2. Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Du- mas Ang. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585-597, 2015. doi: 10.1162/tacl a 00160. URL https: //aclanthology.org/Q15-1042. · DOI
  3. Joel Lehman and Kenneth O. Stanley. Abandoning Objectives: Evolution Through the Search for Novelty Alone. Evolutionary Computation, 19(2):189-223, June 2011b. ISSN 1063-6560. doi: 10.1162/EVCO a 00025. · DOI
  4. J. Schmidhuber. A 'Self-Referential' Weight Matrix. In Stan Gielen and Bert Kappen (eds.), ICANN '93, pp. 446-450, London, 1993. Springer. ISBN 978-1-4471-2063-6. doi: 10.1007/ 978-1-4471-2063-6 107. · DOI
  5. OpenReview.net, 2023. URL https://openreview.net/pdf?id=92gvk82DE-.

Cited by

Sorted by citations

Related

Authors
Fernando, C., Banarse, D., Michalewski, H., Osindero, S., & Rocktäschel, T.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2309.16797
arXiv
2309.16797
OpenAlex
W4387293261
Cited by
31
References
0
Citations by year
2026: 3 · 2025: 17 · 2024: 8 · 2023: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, chain-of-thought prompting, arithmetic reasoning, commonsense reasoning, hate speech detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-24.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-24.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review