CoChat
Sign Up
PreprintCited by 1AI-based Problem Solving and PlanningSpeech and dialogue systemsRecommender Systems and Techniques

FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents

Open access
Wang, Ben, Zhou, Kang, Guo, Lifan, Chen, Feng, Zhang, Chi·2026·arXiv
DOI 10.48550/arxiv.2608.04095arXiv 2608.04095
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language model (LLM) agents are increasingly used as personalized assistants in high-stakes domains such as financial advising, yet it remains unclear whether they can maintain and update an individualized user model over long horizons. Existing personalized-memory benchmarks primarily test factual retention or rely on weakly constrained model-generated trajectories, leaving event-driven preference adaptation underexplored. We introduce FinPerMA, an event-grounded benchmark that evaluates personalized memory against frozen longitudinal investor trajectories. Its generation pipeline combines deterministic, theory-informed impact rules, controlled LLM narration, and automated quality screening; a Post-Shock checkpoint isolates whether an agent has integrated a material event into its persistent user model. On 2,994 questions from 276 personas, seven frontier LLMs and up to seven memory configurations remain far from saturated: no full-context configuration exceeds approximately 0.47 overall accuracy or approximately 39% on multiple-choice questions. Attribution analysis shows that summary-based memory often preserves factual details while losing the preference signals needed for personalization; simple retrieval can therefore outperform purpose-built memory systems, with the gap widening after shocks.

AI snapshot

Written from the paper’s full text.

TL;DR

FinPerMA is an event-grounded benchmark that tests whether LLM agents maintain and update a personalized user model across frozen longitudinal investor trajectories, using a deterministic theory-informed Impact Model to generate auditable preference changes. Across 2,994 questions from 276 personas, seven frontier LLMs and up to seven memory configurations remain far from saturated, with no full-context configuration exceeding ≈0.47 overall accuracy or ≈39% on MCQs.

Key findings

  • Persona history lifts every backbone well above its parametric floor: overall accuracy rises from 0.18–0.27 (no_memory) to 0.41–0.47 (full_context), a 1.5–2.4× jump; Qwen3.8 leads on overall accuracy (0.469) and open-ended quality (0.878). MCQ accuracy stays near chance without history (0.17–0.26) and no full-context MCQ score exceeds ≈39%.
  • Under a fixed backbone (Qwen3.7-Max), retrieval recovers ≈88% of the lower-to-upper-bound gap (bge_rag 88.2%, naive_rag 87.0%) while structured/profile systems recover 54–79% (memobase 79.1%, mem0 72.4%, memOS 54.3%). bge_rag reaches this with 1.40k context tokens versus 12.8k for full_context, a 12× marginal-efficiency edge; mem0 is the most token-efficient system.
  • The full_context–mem0 gap widens from 8.0 points in-timeline (53.4 vs 45.4) to 13.0 points at Post-Shock (51.9 vs 38.9), while a zero-memory control stays at the ~17% floor. memOS matches full_context on factual recall (27.7 vs 27.1) but drops sharply on preference tracking (31.9 vs 52.9) and bias identification (32.6 vs 54.0).
  • No backbone leads on all seven capabilities; the lead rotates across Claude (factual recall, causal reasoning), Qwen3.8 (preference tracking, situational adaptation), and Gemini (cross-domain transfer). The top error modes — zero-memory default, single-signal over-inference, and temporal misalignment — account for ≈66% of errors, and all models score 4.5–11.2 points lower on anti-typical than typical personas.

Methods

  • Design: a five-stage pipeline (persona synthesis → event timeline → three-layer Impact Model → dialogue synthesis → question generation) producing a frozen v8gold corpus of (persona, timeline, dialogues, questions) tuples. Layer 1 deterministically builds an ImpactConstraint from five sub-modules (BIT classification, event-shock coefficients, Big-Five multiplier, experience-history aggregation, symbolic memory-reference anchor); Layer 2 has an LLM narrate a candidate reaction under that constraint; Layer 3 applies hard checks (risk direction, magnitude bounds with 20% tolerance, forbidden biases, evolution-type consistency) with up to three retries and a template fallback.
  • Data: ~300 synthesized personas (276 evaluated), each with 5–8 events drawn from 97 dated 2020–2026 events (17 macro, 64 industry, 16 personal) with a minimum four-week inter-event spacing; personas are sampled from distributions calibrated to Shenzhen Stock Exchange and Asset Management Association of China retail-investor surveys and classified into four Behavioral Investor Types with each type ≥20% of the corpus. The benchmark contains 2,994 questions (2,494 six-option MCQs, 83.3%; 500 open-ended, 16.7%) over a 3×4 dimension (Memory Fidelity, Preference Reasoning, Context Adaptation) × checkpoint (Zero-memory, In-timeline, Post-distraction, Post-Shock) grid; personas contribute 3–19 questions (median 11).
  • Analysis: seven frontier LLMs evaluated under no_memory and full_context references, with a seven-way memory-system comparison fixed on Qwen3.7-Max (BM25 and BGE-M3 retrieval; mem0, memOS, memobase). MCQ correctness is deterministic option-content matching; open-ended correctness is majority vote from a three-way cross-vendor judge ensemble, which also returns PAS, BIA, and MemFid scores. Metrics include %UB (share of the no_memory–full_context gap recovered), Ctx (mean context tokens per query), and η1k (accuracy gain per additional 1k tokens).

Limitations

  • Stated: FinPerMA uses synthetic personas and rule-guided preference changes, so benchmark accuracy does not establish performance with real investors; timelines contain only five to eight events and focus on personal finance; the memory-system comparison uses one seed per configuration and unmatched native token budgets. The authors call for longer histories, repeated runs, other personalization domains, and human review where privacy permits.
  • Stated: the fixed-backbone memory comparison uses native, unmatched token budgets, so its ranking describes operating points rather than controlled architecture quality.
  • Evident from the text: the seven-way memory-system comparison is run only on Qwen3.7-Max, so architecture conclusions rest on a single backbone; the authors also note that the low score on one capability is partly a control-set artifact (76% of such MCQs are zero-memory by design), and that Claude-Opus-4-8 and GPT-5.6-sol full_context results use n=2959 because 35 queries were dropped by a gateway content-safety filter.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2608.04095

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

11 of 24 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Existing benchmarks emphasize factual retention, long-context recall, or explicit information updates.

    “(G1) Limited event-conditioned state change.Existing benchmarks emphasize factual retention, long-context recall, or explicit information updates.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Jiang et al. 2025 source

      Section 5.1 (p. 9) describes prior benchmarks centered on retrieving facts and evaluating memory over long contexts, supporting the first two parts of the claim. Section 5.2 (p. 10) contrasts PERSONAMEM with earlier personalization benchmarks, but does not establish that those benchmarks emphasize explicit information updates; the passages about preference updates describe PERSONAMEM itself. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Maharana et al. 2024 source

      The Introduction (p. 2) describes LOCOMO’s own long-term recall and event-understanding tasks, while its discussion of conventional evaluation describes response overlap, consistency, contradiction, and empathy. The passages found when searching for prior benchmarks and information updates do not establish that existing benchmarks emphasize factual retention, long-context recall, or explicit information updates. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites Wu et al. 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisenone given

    Existing financial LLM resources such as FinGPT and FinBen do not center this longitudinal personalized-memory problem.

    “Existing financial LLM resources such as FinGPT (Yang, Liu, and Wang 2023) and FinBen (Xie et al. 2024) do not center this longitudinal personalized-memory problem.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Yang, Liu, and Wang 2023 source
    • cites Xie et al. 2024 source
  • Findingown data

    Summary-based memory often preserves factual details while losing the preference signals needed for personalization.

    “Attribution analysis shows that summary-based memory often preserves factual details while losing the preference signals needed for personalization; simple retrieval can therefore outperform purpose-built memory systems, with the gap widening after shocks.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.
  • Findingown data

    mem0 is the most token-efficient system.

    “bge_rag reaches the 88% level with 1.40k rather than 12.8k context tokens, giving a 12× marginal-efficiency edge, while mem0 is the most token-efficient system.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.
  • Findingnone given

    We curate 97 dated events spanning 17 macro, 64 industry, and 16 personal events.

    “We curate 97 dated events spanning 17 macro, 64 industry, and 16 personal events.”
    Quote in the paperSubstantial claim
  • May combine two claims. 33
  • Classified as probably not a substantial claim. 21

  • May combine two claims.

    Premisenone given

    Existing personalized-memory benchmarks primarily test factual retention or rely on weakly constrained model-generated trajectories, leaving event-driven preference adaptation underexplored.

    “Existing personalized-memory benchmarks primarily test factual retention or rely on weakly constrained model-generated trajectories, leaving event-driven preference adaptation underexplored.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Jiang et al. 2025 source

      Section 5.1 (p. 9) describes prior long-context benchmarks as testing fact retrieval and other general memory abilities, and says some use LLM-generated sessions. Section 5.2 (p. 10) distinguishes PERSONAMEM by its evolving preferences grounded in temporal events. The paper does not characterize earlier generated trajectories as weakly constrained, so that part of the claim is unsupported. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Maharana et al. 2024 source

      The Introduction (p. 2) says conventional dialogue-evaluation metrics are poorly suited to assessing long-term context, and Related Work (p. 3) identifies challenges in reasoning about past events. The paper does not establish that personalized-memory benchmarks primarily test factual retention or use weakly constrained model-generated trajectories, and it does not specifically identify event-driven preference adaptation as an underexplored benchmark target. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites Wu et al. 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisereasoning

    Existing benchmarks provide less control over how a user's latent preferences change after consequential events, making it difficult to separate recalling the past from reasoning with an updated user state.

    “They provide less control over how a user's latent preferences change after consequential events, making it difficult to separate recalling the past from reasoning with an updated user state.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Jiang et al. 2025 source

      Section 5.1 (p. 9) says earlier dialogue benchmarks focus on general memory, whereas PERSONAMEM evaluates fine-grained personalization. The benchmark overview (p. 3) describes preferences changing after life events, and Section 4.2 (p. 7) distinguishes recalling changes from using the latest preference. The paper does not explicitly establish that existing benchmarks offer less control over those changes or that this makes the two abilities difficult to separate. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites Maharana et al. 2024 source

      The Introduction (pp. 1–2) says earlier evaluations used limited conversational contexts and distinguishes recalling past context from understanding causal and temporal connections. The event-generation prompt (p. 15) also describes life events that can change a person's state of mind or personality. But the paper does not say existing benchmarks offer less control over changes in latent preferences, or that this specifically makes recall difficult to separate from reasoning about an updated user state. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites Wu et al. 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    LongMemEval tests retention and updates at the fact level, whereas LoCoMo tests continuity over very long conversations. Neither isolates whether an agent integrates a consequential event into its persistent user model.

    “LongMemEval tests retention and updates at the fact level (Wu et al. 2025), whereas LoCoMo tests continuity over very long conversations (Maharana et al. 2024).Neither isolates whether an agent integrates a consequential event into its persistent user model.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Wu et al. 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Maharana et al. 2024 source

      The Introduction (p. 2) and Conclusion (p. 9) support the description of LOCOMO as evaluating memory and continuity across very long conversations. Section 4.2 (p. 6) specifies an event-summarization task, but the paper does not establish the further claim that LOCOMO fails to isolate whether an agent incorporates a consequential event into a persistent user model. The LongMemEval description is attributed to a different citation and is not judged here. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductcitation

    Open-ended correctness is determined by majority vote from a three-way cross-vendor judge ensemble; judge-produced auxiliary scores are used for preference alignment, bias identification, and memory fidelity.

    “Open-ended correctness is determined by majority vote from a three-way cross-vendor judge ensemble (Zheng et al. 2023); judge-produced auxiliary scores are used for preference alignment, bias identification, and memory fidelity.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Zheng et al. 2023 source

      Section 3.1 (p. 4) describes LLM judges evaluating open-ended answers, and §4.1 (p. 7) uses LLM and human judges. But the paper does not describe determining correctness by majority vote of a three-way cross-vendor judge ensemble; Appendix D.3 (p. 26) uses majority vote for human judgments instead. The citation is attached to the first clause, so the auxiliary-scores clause is outside this verdict. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingcitation

    All models score 4.5-11.2 points lower on anti-typical than typical personas, indicating a fallback to type-level stereotypes when individual evidence conflicts with a familiar profile.

    “All models score 4.5-11.2points lower on anti-typical than typical personas, indicating a fallback to type-level stereotypes when individual evidence conflicts with a familiar profile (Jiang et al. 2025).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Jiang et al. 2025 source

      The error analysis (Appendix F, p. 23) says GPT-4o often chose broadly reasonable responses over more personalized ones. It does not report a 4.5–11.2-point gap between anti-typical and typical personas, evaluate that comparison across all models, or identify type-level stereotypes as the cause. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Findingown data

    On 2,994 questions from 276 personas, seven frontier LLMs and up to seven memory configurations remain far from saturated: no full-context configuration exceeds ≈ 0.47 overall accuracy or ≈ 39% on multiple-choice questions.

    “On 2,994 questions from 276 personas, seven frontier LLMs and up to seven memory configurations remain far from saturated: no full-context configuration exceeds ≈ 0.47 overall accuracy or ≈ 39% on multiple-choice questions.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Its timelines contain five to eight events and focus on personal finance.

    “Its timelines contain five to eight events and focus on personal finance, while the memory-system comparison uses one seed per configuration and unmatched native token budgets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Across the frozen corpus, personas contribute 3-19 questions (median 11) over the 3×4 dimension-checkpoint grid.

    “Across the frozen corpus, personas contribute 3-19 questions (median 11) over the 3×4 dimension-checkpoint grid.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Qwen3.7-Max full_context-mem0 gap widens from 8.0 in-timeline (53.4 vs 45.4) to 13.0 at Post-Shock (51.9 vs 38.9), while a zero-memory control stays at the ∼17% floor.

    “By checkpoint, the Qwen3.7-Maxfull_context-mem0 gap widens from 8.0 in-timeline (53.4 vs 45.4) to 13.0 at Post-Shock (51.9 vs 38.9), while a zero-memory control stays at the ∼ 17% floor.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Findingown data

    This low value is partly a control-set artifact (76% of such MCQs are zero-memory by design); once removed, situational adaptation and cross-domain transfer are the hardest capabilities.

    “This low value is partly a control-set artifact (76% of such MCQs are zero-memory by design); once removed, situational adaptation and cross-domain transfer are the hardest capabilities (Fig. 4a).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Findingown data

    Under a fixed backbone, retrieval approaches the full-context upper bound with far fewer tokens, whereas profile-based consolidation often retains facts but loses preference cues.

    “Under a fixed backbone, retrieval approaches the full-context upper bound with far fewer tokens, whereas profile-based consolidation often retains facts but loses preference cues.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Each event records its type, date, affected assets or domains, region, summary, typical responses, and a severity score on a 1-5 ordinal scale.

    “Each event records its type, date, affected assets or domains, region, summary, typical responses, and a severity score on a 1-5 ordinal scale.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The benchmark contains 2,494 six-option MCQs (83.3%) and 500 open-ended questions (16.7%).

    “The benchmark contains 2,494 six-option MCQs (83.3%) and 500 open-ended questions (16.7%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Qwen3.8 leads on both overall accuracy (0.469) and open-ended quality (0.878).

    “Qwen3.8 leads on both overall accuracy (0.469) and open-ended quality (0.878).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Findingown data

    No backbone leads on all seven capabilities; the lead rotates across Claude (factual recall, causal reasoning), Qwen3.8 (preference tracking, situational adaptation), and Gemini (cross-domain transfer).

    “No backbone leads on all seven capabilities (Fig. 4a): the lead rotates across Claude (factual recall, causal reasoning), Qwen3.8 (preference tracking, situational adaptation), and Gemini (cross-domain transfer).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Findingown data

    memos matches full_context on factual recall (27.7 vs 27.1) yet falls sharply on inference targets (preference tracking 31.9 vs 52.9; bias identification 32.6 vs 54.0).

    “By capability, memos matches full_context on factual recall (27.7 vs 27.1) yet falls sharply on inference targets (preference tracking 31.9 vs 52.9; bias identification 32.6 vs 54.0).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Findingown data

    These modes account for ≈66% of errors and isolate recall, integration, and recency as separate bottlenecks.

    “Together, these modes account for ≈ 66% of errors and isolate recall, integration, and recency as separate bottlenecks.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Findingown data

    MCQ accuracy is especially memory-dependent: without history, it remains near chance (0.17-0.26), and no full-context MCQ score exceeds ≈ 39%, so FinPerMA remains far from saturated.

    “MCQ accuracy is especially memory-dependent.Without history, it remains near chance (0.17-0.26), and no full-context MCQ score exceeds ≈ 39%, so FinPerMA remains far from saturated.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Distractors are designed for comparable structure and plausibility; empirically, the median absolute length difference from the correct option is five characters, while 52.9% fall within five characters.

    “Distractors are designed for comparable structure and plausibility; empirically, the median absolute length difference from the correct option is five characters, while 52.9% fall within five characters.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The top error is zero-memory default, followed by single-signal over-inference and temporal misalignment.

    “The top error is zero-memory default, followed by single-signal over-inference and temporal misalignment.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Findingown data

    The gain is largest where the parametric floor is weakest (Qwen3.7-Max, 2.4×) and smallest for the strongest priors (Claude-Opus-4-8, 1.5×).

    “The gain is largest where the parametric floor is weakest (Qwen3.7-Max,2.4×) and smallest for the strongest priors (Claude-Opus-4-8, 1.5×);”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Findingown data

    bge_rag reaches the 88% level with 1.40k rather than 12.8k context tokens, giving a 12× marginal-efficiency edge.

    “bge_rag reaches the 88% level with 1.40k rather than 12.8k context tokens, giving a 12× marginal-efficiency edge, while mem0 is the most token-efficient system.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Findingown data

    Both retrievers recover ≈88% of the lower-to-upper-bound gap, while the three structured/profile memory systems recover 54-79%.

    “On the complete Qwen3.7-Maxgrid (Panel B), both retrievers recover ≈88% of the lower-to-upper-bound gap, while the three structured/profile memory systems recover 54-79%.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Findingown data

    Persona history lifts every model far above its parametric floor: overall accuracy rises from 0.18-0.27 without memory to 0.41-0.47 under full context, a 1.5-2.4× jump.

    “Persona history lifts every model far above its parametric floor (Table 1, Panel A): overall accuracy rises from 0.18-0.27without memory to 0.41-0.47under full context, a 1.5-2.4×jump.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Findingown data

    Full context improves every backbone, confirming that persona-specific evidence matters, yet the benchmark remains far from saturated.

    “Full context improves every backbone, confirming that persona-specific evidence matters, yet the benchmark remains far from saturated.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Interpretationreasoning

    Simple retrieval can outperform purpose-built memory systems, with the gap widening after shocks.

    “Attribution analysis shows that summary-based memory often preserves factual details while losing the preference signals needed for personalization; simple retrieval can therefore outperform purpose-built memory systems, with the gap widening after shocks.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    Following the framing of retrieval-augmented and constraint-augmented generation, we insert the constraint into the prompt as a hard boundary rather than a soft suggestion.

    “Following the framing of retrieval-augmented and constraint-augmented generation (Lewis et al. 2020), we insert the constraint into the prompt as a hard boundary rather than a soft suggestion:”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Lewis et al. 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductreasoning

    The frozen v8gold corpus makes all systems face the same trajectories, so comparisons do not depend on regenerating labels for each evaluated model.

    “The frozen v8gold corpus makes all systems face the same trajectories, so comparisons do not depend on regenerating labels for each evaluated model.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The Impact Model is not an empirically estimated model of human behavior.

    “It is not an empirically estimated model of human behavior; its purpose is to produce reproducible, internally consistent trajectories for comparing memory systems.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    MCQ correctness is computed deterministically by comparing the selected option's content with the gold option after construction-time shuffling.

    “MCQ correctness is computed deterministically by comparing the selected option's content with the gold option after construction-time shuffling.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The gap widens precisely when new evidence should revise the stored user state, making Post-Shock a distinct test of memory updating.

    “The gap widens precisely when new evidence should revise the stored user state, making Post-Shock a distinct test of memory updating.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    This symbolic mechanism is deliberately interpretable and auditable; no embedding similarity or temperature parameter is used in v8gold.

    “This symbolic mechanism is deliberately interpretable and auditable; no embedding similarity or temperature parameter is used in v8gold.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Given fixed (P, E, M), this object is deterministic: the same inputs across independent runs produce the same output.

    “Given fixed (P, E, M), this object is deterministic: the same inputs across independent runs produce the same output.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The difference concentrates on open-ended questions: retrieval preserves verbatim evidence that summary/profile memory may discard.

    “The difference concentrates on open-ended questions: retrieval preserves verbatim evidence that summary/profile memory may discard.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Big-Five personality traits' association with risk tolerance has been documented in individual-investor studies.

    “Psychometric variables comprise (i) Big-Five personality traits, whose association with risk tolerance has been documented in individual-investor studies (Durand, Newby, and Sanghani 2008);”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Durand, Newby, and Sanghani 2008 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The memory-system comparison uses one seed per configuration and unmatched native token budgets.

    “Its timelines contain five to eight events and focus on personal finance, while the memory-system comparison uses one seed per configuration and unmatched native token budgets.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Layer 3 enforces the checks implemented in the released validator: risk-direction consistency; risk-delta bounds with a 20% tolerance; exclusion of biases_forbidden; and evolution-type consistency for shocks.

    “Layer 3 enforces the checks implemented in the released validator: riskdirection consistency; risk-delta bounds with a 20% tolerance; exclusion of biases_forbidden; and evolution-type consistency for shocks.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    This is a deterministic benchmark operationalization rather than an empirically estimated structural coefficient.

    “This is a deterministic benchmark operationalization rather than an empirically estimated structural coefficient.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    On Qwen3.7-Max we additionally compare five memory frameworks: BM25 and BGE-M3 retrieval and three structured/profile systems (Mem0, MemOS, Memobase), all on the frozen v8gold set (2,994 questions, single seed).

    “On Qwen3.7-Max we additionally compare five memory frameworks: BM25 and BGE-M3 retrieval and three structured/profile systems (Mem0 (Chhikara et al. 2025), MemOS (Li et al. 2025), Memobase), all on the frozen v8gold set (2,994 questions, single seed).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Chhikara et al. 2025 source
    • cites Li et al. 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    MCQ answer positions are uniformly shuffled with a fixed seed at construction time.

    “MCQ answer positions are uniformly shuffled with a fixed seed at construction time.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The challenge is not simply storing more text, but preserving evidence that can revise the current user state.

    “The challenge is therefore not simply storing more text, but preserving evidence that can revise the current user state.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

  • May combine two claims.

    Study design and conductreasoning

    FinPerMA uses synthetic personas and rule-guided preference changes, so benchmark accuracy does not establish performance with real investors.

    “FinPerMA uses synthetic personas and rule-guided preference changes, so benchmark accuracy does not establish performance with real investors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    To ensure balanced coverage, rejection sampling is applied so that each BIT type occupies ≥ 20% of the corpus.

    “To ensure balanced coverage, rejection sampling is applied so that each BIT type occupies ≥ 20% of the corpus.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Failed hard checks trigger retry feedback, up to three attempts; if all attempts fail, a template fallback is emitted with confidence= 0.2 and layer3_fallback=True.

    “Failed hard checks trigger retry feedback, up to three attempts; if all attempts fail, a template fallback is emitted with confidence= 0.2 and layer3_fallback=True.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The Impact Model's purpose is to produce reproducible, internally consistent trajectories for comparing memory systems.

    “It is not an empirically estimated model of human behavior; its purpose is to produce reproducible, internally consistent trajectories for comparing memory systems.”
    Quote in the paperSubstantial claim
45 of 45

  • The paper relies on this without support.

    The synthetic investor trajectories generated by the Impact Model are a valid proxy for real longitudinal investor behavior for evaluating personalized memory.

    “FinPerMA uses synthetic personas and rule-guided preference changes, so benchmark accuracy does not establish performance with real investors.”

  • The paper relies on this without support.

    Real events with verifiable timestamps, including the 2020 COVID crash, the 2022 rate-hike cycle, and the 2023 SVB collapse, supply external timeline anchors.

    “Real events with verifiable timestamps, including the 2020 COVID crash, the 2022 rate-hike cycle, and the 2023 SVB collapse, supply external timeline anchors.”

  • The paper relies on this without support.

    The seven evaluated LLM backbones and memory systems are representative of current frontier systems.

    “We evaluate seven recent frontier LLMs representative of systems used in Chinese-market financial-advisory settings: Qwen3.8,Qwen3.7-Max,DeepSeek-V4-Pro, GLM-5.2,GPT-5.6-sol,Gemini-3.1-pro,and Claude-Opus-4-8.”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • No-memory accuracy range omits Claude

    Minor

    The stated no-memory range ends at 0.27, but Claude-Opus-4-8 has a no-memory accuracy of 31.2% (0.312) in Table 1.

  • Full-context accuracy range omits GPT-5.6-sol

    Minor

    The stated full-context range starts at 0.41, but Table 1 reports 40.1% (0.401) for GPT-5.6-sol.

  • Pipeline stage count differs

    Minor

    The methodology says the pipeline has five sequential stages, whereas Figure 1 labels Quality Control as stage 6.

  • Retry limit conflicts with three-attempt algorithm

    Minor

    Figure 2 allows up to three retries, but the methodology specifies up to three attempts total. Algorithm 1 also runs the narrator at most three times, permitting only two retries after an initial failure.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; Yih, W.-t.; Rocktäschel, T.; Riedel, S.; and Kiela, D. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020).
  2. Pompian, M. M. 2012. Behavioral Finance and Investor Types: Managing Behavior to Make Better Investment Decisions. John Wiley & Sons. ISBN 9781118011508.

Cited by

Sorted by citations

Authors
Wang, B., Zhou, K., Guo, L., Chen, F., & Zhang, C.
Affiliations
Alibaba Group (China) (CN)
ORCID
Zhou, Kang
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2608.04095
arXiv
2608.04095
OpenAlex
W7196960074
Semantic Scholar
f82f58a5cc90dd55c6940ce7d7701bb0ebaa9090
Cited by
0
References
16
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
LLM agents, benchmark, financial advice, preference adaptation, memory retrieval, attribution analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review