Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory

Classified as probably not a substantial claim.

Contemporary LLMs such as GPT-4 can exhibit substantial improvements in reasoning and generation capability when additional compute is devoted to inference-time strategies.

“It is now widely known and accepted that contemporary LLMs such as GPT-4 can exhibit substantial improvements in reasoning and generation capability when additional compute is devoted to inference-time strategies (e.g., chain-ofthought prompting (Wei et al., 2022; Kojima et al., 2022; Zhou et al., 2022), tree-of-thought expansions (Yao et al., 2023; Long, 2023), minimum Bayes risk decoding (Suzgun”

• cites Wei et al., 2022 sourceNot checked — no full text of the source
• cites Yao et al., 2023 sourceSupported
• cites Wang et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

In Math Equation Balancer, GPT-4o and Claude 3.5 Sonnet reached near-perfect accuracy of 98-100% under DC, whereas their baselines stagnated around 45-50%.

“In tasks prone to arithmetic mistakes, such as balancing equations, DC enabled GPT-4o and Claude to reach near-perfect accuracy by recalling previously validated code, whereas their baselines stagnated around 50%.”

• Also stated in: § 1. Introduction · p. 2 · plain

  “We saw a similar pattern in Math Equation Balancer, where GPT-4o and Claude soared from 45-50% to 98-100% by 'recalling' a straightforward code-based approach instead of manually fumbling with numeric manipulations.”

The quote only partly states this claim.

Claude 3.5 Sonnet's accuracy on AIME 2025 increased by 30 percentage points under DC-Cu, from 6.7% to 36.7%.

“Likewise, it gained 30% accuracy on AIME 2025.”

• Also stated in: § Abstract · p. 1 · plain

  “Claude 3.5 Sonnet gaining 27% on AIME 2024 and 30% on AIME 2025 under DC-Cu.”

May combine two claims.

DC increases performance and reduces repetitive mistakes across multiple challenging benchmarks.

“We tested DC across multiple challenging benchmarks and observed that it increases performance and reduces repetitive mistakes.”

May combine two claims.

Smaller models such as GPT-4o-mini benefit from DC in limited amounts because they generate too few correct solutions, leaving the memory populated with flawed or incomplete strategies, and they struggle to refine stored content.

“We found that smaller models, such as GPT-4o-mini, benefit from DC in limited amounts. These models generate too few correct solutions in these challenging tasks in the first place, leaving the memory populated with flawed or incomplete strategies. Worse, they struggle to refine stored content.”

May combine two claims.

DC-Cu outperformed MV, reaching 50.0% on AIME 2024 and 36.7% on AIME 2025.

“In contrast, DC-Cu outperformed MV, reaching 50.0% on AIME 2024 and 36.7% on AIME 2025.”

The quote only partly states this claim.

On AIME 2025, GPT-4o-mini showed a minor improvement, peaking at 16.7% under DC-RS.

“AIME 2025 showed a minor improvement, peaking at 16.7% under DC-RS.”

May combine two claims.

GPT-4o managed a baseline of 20.0% but fell to 6.7% using FH, in contrast to 40.0% with DC-RS.

“Similarly, GPT-4o managed a baseline of 20.0% but fell to 6.7% using FH, in direct contrast to 40.0% with DC-RS.”

May combine two claims.

GPT-4o discovered a Python-based brute-force solution, stored it, and subsequently retrieved it for subsequent puzzles.

“Early in the test sequence, the model discovered a Python-based brute-force solution, stored it, and subsequently retrieved it for subsequent puzzles.”

May combine two claims.

Sonnet under FH reached 26.7% accuracy in 2024 questions, while DC-based methods hit 50.0%.

“As shown in Table 2, Sonnet under FH reached 26.7% accuracy in 2024 questions, while DC-based methods hit 50.0%.”

May combine two claims.

GPT-4o-mini showed even smaller gains, with some variants leading to slight declines in performance.

“That said, GPT-4o-mini (Table 3) showed even smaller gains, with some variants leading to slight declines in performance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-4o's near-complete reliance on Python scripts for Game of 24 exemplifies a shift toward code generation.

“GPT-4o's near-complete reliance on Python scripts for Game of 24 exemplifies this shift.”

May combine two claims.

On AIME 2024, DC-∅ provided a 20.0% boost for GPT-4o-mini, but both DC-Cu and DC-RS performed worse than baseline.

“On AIME 2024, DC-∅ provided a 20.0% boost, but both DC-Cu and DC-RS performed worse than baseline.”

May combine two claims.

Claude 3.5 Sonnet and GPT-4o showed notable gains across multiple tasks under DC, while Claude 3.5 Haiku and GPT-4o-mini showed more limited and inconsistent gains.

“While Claude 3.5 Sonnet and GPT-4o showed notable gains across multiple tasks under DC, their smaller counterparts, Claude 3.5 Haiku and GPT-4o-mini, showed more limited and inconsistent gains.”

May combine two claims.

On AIME 2024, Claude Sonnet averaged 370 tokens under BL, 494 under DC-∅, 1035 under DC-RS, and 1831 under DC-Cu.

“On AIME 2024, Claude Sonnet averaged 370 tokens under BL, 494 under DC-∅ , 1035 under DC-RS, and 1831 under DC-Cu.”

May combine two claims.

Models quickly learned an algorithmic or Python-based balancing routine, stored it in external memory, and repeatedly retrieved it, achieving exceptional consistency once the core method was established.

“As observed in Game of 24, the models quickly learned an algorithmic or Python-based balancing routine, stored it in external memory, and repeatedly retrieved it, achieving exceptional consistency once the core method was established.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The solutions were far too verbose and long.

“The solutions were far too verbose and long.”

May combine two claims.

DC-Cu achieved the highest accuracy scores on AIME 2024 and AIME 2025.

“DC-Cu, where the model curates memory after processing the input and does not involve a retrieval stage, also proved potent in recent exam sets, achieving highest accuracy scores in AIME 2024 and 2025.”

May combine two claims.

GPT-4o generated, stored, and iteratively refined a Python function that tested permutations of numbers and operations, allowing it to solve each instance of Game of 24 with high accuracy.

“It generated, stored, and iteratively refined a Python function that tested permutations of numbers and operations, allowing it to solve each instance of Game of 24 with high accuracy.”

May combine two claims.

This shift to structured code reuse resulted in a dramatic performance increase from 10% to 99% accuracy, eliminating arithmetic errors and redundant problem-solving efforts.

“This shift to structured code reuse resulted in a dramatic performance increase from 10% to 99% accuracy, eliminating arithmetic errors and redundant problem-solving efforts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In ablation experiments, we observed mixed results.

“In our ablation experiments, we observed mixed results.”

May combine two claims.

Both DC-Cu and DC-RS show high stability in memory content after the first few iterations in Game of 24.

“While both DC-Cu and DC-RS show high stability after the first few iterations”

May combine two claims.

DC-Cu updates the memory after processing an input query rather than refining it before generating a response, so the model lacks the opportunity to incorporate new insights from the current query while reasoning through its solution.

“First , it updates the memory after processing an input query, rather than refining it before generating a response. This means the model lacks the opportunity to incorporate new insights from the current query while reasoning through its solution.”

May combine two claims.

DC-Cu does not store or revisit past input-output pairs unless explicitly retained in memory, preventing the model from directly retrieving and leveraging historical responses.

“Second , DCCu does not store or revisit past input-output pairs unless explicitly retained in memory. This omission prevents the model from directly retrieving and leveraging historical responses, which can be particularly valuable in benchmarks covering diverse topics or domains (e.g., GPQA-Diamond).”

May combine two claims.

DC-RS enhances the model's adaptability and reasoning efficiency by refining the memory before responding and retrieving prior cases when needed.

“By refining the memory before responding and retrieving prior cases when needed, DC-RS enhances the model's adaptability and reasoning efficiency.”

Classified as probably not a substantial claim. May combine two claims.

The selected datasets include algorithmic, logical, and domain-specific reasoning tasks, each chosen to stress-test the model's ability to refine its reasoning over time.

“Overall, the selected datasets include algorithmic, logical, and domain-specific reasoning tasks, each chosen to stresstest the model's ability to refine its reasoning over time.”

• AssumptionRests on an assumption: The selected benchmarks are challenging tasks where contemporary state-of-the-art LLMs still face limitations and are not near saturation.

May combine two claims.

GPT-4o discovered a Python brute-force solver for Game of 24 early in the test sequence, stored it, and then retrieved it for subsequent queries, avoiding manual arithmetic.

“Early in the test sequence, the model discovered that an efficient Python brute-force solver eliminated all manual guesswork. Once this snippet was stored, GPT-4o simply retrieved it for subsequent queries, avoiding manual arithmetic entirely.”

• Also stated in: § 4.1. DC enables test-time learning and reduces repetitive errors · p. 5 · plain

  “Early in the task sequence, GPT-4o discovered a reliable, Python-based brute-force method to solve Game of 24 and later on recognized the repetitive structure of the problem. The model then encoded this approach into its memory. Once established, GPT-4o consistently retrieved and applied the more or less same Python solution for subsequent examples, leading to rapid and accurate results.”

May combine two claims.

DC's memory is self-curated, focusing on concise, transferable snippets rather than entire transcripts, thereby facilitating meta-learning and avoiding context ballooning.

“Crucially, DC's memory is self-curated, focusing on concise, transferable snippets rather than entire transcripts, thereby facilitating meta-learning and avoiding context ballooning.”

• Also stated in: § 1. Introduction · p. 2 · plain

  “Under DC, memory is carefully curated, focusing on succinct, useful, and transferable knowledge over raw transcripts. This prevents ballooning context lengths (Liu et al., 2024a) and helps ensure that repeated retrieval remains tractable.”

• cites Liu et al., 2024aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

DC can amplify the strengths of models that can already produce high-quality outputs, but cannot fix foundational gaps in reasoning.

“DC can amplify the strengths of models that can already produce high-quality outputs, but not fix foundational gaps in reasoning.”

Probably not a substantial claim: may not pass S3 (load-bearing).

GPQA-Diamond contains 198 expert-validated questions across natural sciences, including biology, chemistry, and physics.

“GPQA-Diamond contains 198 expertvalidated questions across natural sciences, including biology, chemistry, and physics.”

• cites Rein et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Full-History Appending can exceed context-window limits and include redundant or low-value information.

“FH can exceed context-window limits and include redundant or low-value information, but nonetheless, it provides a useful comparison for methods that actively curate content.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Dynamic Retrieval can help the model see relevant input-output pairs but does not directly codify abstract or generalized solutions.

“DR can help the model see relevant input-output pairs but not directly codify any abstract or generalized solutions.”

May combine two claims.

Smaller models generate correct solutions less reliably, leading to a sparse or low-quality memory repository.

“Smaller models, such as GPT-4o-mini and Claude 3.5 Haiku, generate correct solutions less reliably, leading to a sparse or low-quality memory repository.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Smaller models struggle with long-context understanding/generation and memory retrieval, leading to inefficient or irrelevant memory usage.

“Smaller models struggle with long-context understanding/generation and memory retrieval, leading to inefficient or irrelevant memory usage.”

May combine two claims.

DC requires memory curation after each query, but it optimizes efficiency over time by reducing redundant computation and token usage.

“Although DC requires memory curation after each query, it optimizes efficiency over time by reducing redundant computation and token usage.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Most large LLMs excel at processing lengthy inputs but struggle to generate comparably long and well-organized outputs.

“Most large LLMs excel at processing lengthy inputs but struggle to generate comparably long 12 and well-organized outputs.”

• cites Liu et al., 2024bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Claude temporarily stored and retrieved compact 'reference guides' on engineering and physics principles, which might have proved beneficial for thematically similar questions.

“Our examination of the curated memory entries shows that Claude temporarily stored and retrieved compact 'reference guides' on engineering and physics principles, which might have proved beneficial for thematically similar questions.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Smaller models often fail to retrieve the most relevant past solutions or misapply retrieved knowledge to new problems.

“smaller models often fail to retrieve the most relevant past solutions or misapply retrieved knowledge to new problems.”

May combine two claims.

Unlike MV, which passively aggregates outputs, DC actively refines knowledge over time, eliminating errors and improving solution quality.

“Unlike MV, which passively aggregates outputs, DC actively refines knowledge over time, eliminating errors and improving solution quality.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Curriculum-style learning may potentially bootstrap performance.

“This suggests that curriculum-style learning (Bengio et al., 2009), where simpler or archetypal problems are presented first to build a repository of valid heuristics, may potentially bootstrap performance.”

• cites Bengio et al., 2009 sourceNot checked — no full text of the source

May combine two claims.

Excessive uncurated input-output pairs can overwhelm the model's context window, dilute crucial insights, hamper retrieval efficiency, and significantly increase inference costs over time.

“Excessive uncurated inputoutput pairs can not only overwhelm the model's context window, dilute crucial insights and hamper retrieval efficiency, but also significantly increase inference costs over time.”

May combine two claims.

DC's selective memory curation ensures that problem-solving tips or code snippets remain readily accessible without clutter, thus facilitating more robust and consistent improvements across consecutive queries.

“On the other hand, DC's selective memory curation ensures that problem-solving tips or code snippets remain readily accessible without clutter, thus facilitating more robust and consistent improvements across consecutive queries.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Larger models can sometimes produce higher-quality strategies that could benefit smaller models if the memory is transferred.

“We also observed that larger models, such as Claude 3.5 Sonnet and GPT-4o, can sometimes produce higher-quality strategies that, in principle, could benefit smaller models if the memory is transferred.”

Probably not a substantial claim: may not pass S3 (load-bearing).

DC has potential to nurture efficient tool-usage: the capacity to recognize when external tools are more robust than internally verbalized chain-of-thought calculations.

“This inclination toward automation illustrates DC's potential to nurture efficient tool-usage: the capacity to recognize when external tools (e.g., Python, symbolic math engines, or dedicated solvers) are more robust than internally verbalized chain-of-thought calculations.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The initial cost of discovering a robust approach and curating it remains non-trivial.

“By encoding and reusing well-established techniques (e.g., Python-based solving for Game of 24), models can bypass repeated rediscovery of the same strategies. This significantly cuts down reasoning overhead and token usage in subsequent queries, though the initial cost of discovering a robust approach and curating it remains non-trivial.”

May combine two claims.

DC remains fully external and training-free, requiring no gradient-based adaptation.

“Moreover, unlike methods that assume the model can be retrained or finetuned to incorporate memory items, we remain fully external and training-free; this aligns with 'plug-andplay' usage principle, in which an off-the-shelf model is augmented by an external memory that it reads from and writes to, but does not require any gradient-based adaptation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

DC can help transfer learned strategies within clusters of similar questions.

“DC can help transfer learned strategies within these clusters”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The memory curator ensures the memory remains concise, generalizable, and action-oriented, continuously improving test-time reasoning.

“This ensures the memory (cheatsheet) remains concise, generalizable, and action-oriented, continuously improving test-time reasoning.”

May combine two claims.

While the compute for the first few tasks may be higher, future tasks become simpler because the system can consult and adapt previously curated knowledge.

“Thus, while the compute for the first few tasks may be higher, future tasks become simpler because the system can consult and adapt previously curated knowledge.”

May combine two claims.

DC does not require a new training loop for each batch or scenario; instead, the memory itself is updated to reflect newly found solutions, errors, or strategies without touching the model weights.

“Furthermore, DC does not require a new training loop for each batch or scenario; instead, the memory itself is updated to reflect newly found solutions, errors, or strategies without touching the model weights.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The same structured prompt is used in all non-BL methods, including DC-∅, DR, FH, DC-Cu, and DC-RS.

“Notably, this same structured prompt is used in all non-BL methods, including DC-∅ , DR, FH, DC-Cu, and DC-RS.”

Probably not a substantial claim: may not pass S2 (checkable).

Without careful curation, erroneous heuristics may also spread.

“but without careful curation, erroneous heuristics may also spread”

Classified as probably not a substantial claim.

DC emphasizes selectively storing the most relevant insights and heuristics.

“While these works reveal the many strategies for harnessing memory or feedback, DC emphasizes selectively storing the most relevant insights and heuristics.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Modern retrieval-augmented LLM approaches generally consult an external corpus of documents to improve factuality and reduce hallucination.

“Modern retrieval-augmented LLM approaches generally consult an external corpus of documents (i.e., a knowledge base) to improve factuality and reduce hallucination (Lewis et al., 2020; Lazaridou et al., 2023; Vu et al., 2023; Zhang et al., 2024b)”

• cites Lewis et al., 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lazaridou et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Vu et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

DC spreads the computational benefit of extra compute across multiple interactions, effectively learning to use tools more consistently and reliably over time.

“Thus, DC not only invests extra compute in a single session but spreads that computational benefit across multiple interactions, effectively learning to use tools more consistently and reliably over time.”

May combine two claims.

DC's memory reduces repetitive mistakes and consolidates and codifies robust solution strategies, effectively amortizing or sharing the cost of initial reflection across future tasks.

“This memory not only reduces repetitive mistakes, but also consolidates and codifies robust solution strategies-effectively amortizing or 'sharing' the cost of initial reflection across future tasks.”

Probably not a substantial claim: may not pass S4 (based).

The memory curation step selectively compiles relevant solutions, heuristics, expansions, or code blocks into a form that can be reused for upcoming queries.

“The memory curation step selectively compiles relevant solutions, heuristics, expansions, or code blocks into a form that can be reused for upcoming queries.”

Classified as probably not a substantial claim.

The DC framework focuses explicitly on storing generalizable heuristics, solutions, or meta-level insights that can be repeatedly retrieved and applied across tasks, not just to correct a single solution.

“Our proposed DC framework differs by focusing explicitly on storing generalizable heuristics, solutions, or meta-level insights that can be repeatedly retrieved and applied across tasks, not just to correct a single solution.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The memory curator selectively retains high-value insights, refines existing strategies, and organizes memory efficiently.

“The curator selectively retains high-value insights, refines existing strategies, and organizes memory efficiently.”

Probably not a substantial claim: may not pass S4 (based).

The retrieval corpus in modern retrieval-augmented LLM approaches is almost always fixed prior to inference and does not evolve over time.

“but the retrieval corpus is almost always fixed prior to inference and does not evolve over time.”