CoChat
Sign Up
PreprintCited by 3Information Retrieval and Search BehaviorTopic ModelingSpeech and dialogue systems

Same Ranking, Different Winner: How Scoring Targets Shape LLM Memory Benchmarks

Open access
Panthi, Sugam, Abdelfattah, Rabab·2026·arXiv
DOI 10.48550/arxiv.2605.24060arXiv 2605.24060
Open this paper in CoChat — free accountOpen PDF

Abstract

Conversational-memory systems increasingly transform dialogue history into facts, summaries, timelines, and other source-linked descendants, so a single source turn can coexist with several derived memories in the same retrieval index. This raises an underspecified evaluation question: which stored form should receive retrieval credit? We show that this scoring-target choice is often left implicit and can materially change benchmark conclusions. We present TIAP, a fixed-output audit that rescores saved ranked outputs under three targets -- Raw, Source, and Canonical -- without rerunning retrieval. On LoCoMo and LongMemEval-S, switching only the credited target changes nDCG on 83.4--94.0 percent of shared queries, flips target orderings on Mem0 and MemoryOS transfer runs, and reverses parser-density recommendations. A 1,902-case semantic audit further shows that relaxed source-linked credit is fully justified only 29.2 percent of the time, despite high rubric reliability in a validation subset. These results reveal target noninvariance: conclusions about memory architectures can silently flip with a single benchmark-design choice. Conversational-memory papers should therefore define and report the scoring target explicitly.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper shows that in conversational-memory benchmarks, changing only which stored memory form receives retrieval credit—Raw, Source, or Canonical—can change scores, system rankings, and design recommendations even when ranked outputs are fixed. It introduces TIAP, a fixed-output audit, and reports target noninvariance across LoCoMo, LongMemEval-S, and transfer runs on Mem0 and MemoryOS.

Key findings

  • Switching only the scoring target changes nDCG on 83.4%–94.0% of shared Raw-vs-Canonical queries across LoCoMo and LongMemEval-S native runs.
  • In a 1,902-case semantic audit of contested credits, five-model majority voting found only 29.2% fully support the query, 39.6% are partial, and 31.2% do not support it.
  • Parser-density recommendations flip by target: for lexical F1 vs F5, Raw prefers the sparser F1 store, Source prefers the denser F5 store, and Canonical returns to F1.
  • Transfer runs on Mem0 and MemoryOS remain target-sensitive but show nonmonotonic target orderings depending on benchmark and metric; BEAM-1M nDCG@60 moves from 0.055 under Raw to 0.097 under Source and 0.197 under Canonical on the full 625-query run.

Methods

  • TIAP constructs Raw, Source, and Canonical scoring targets from source-linked lineage metadata, then rescores the same saved top-k = 60 ranked traces without rerunning retrieval; shared-query subsets control for differing target coverage.
  • Main benchmarks are LoCoMo and LongMemEval-S, with BEAM-1M as a supplementary stress test; native retrievers are lexical search, all-MiniLM-L6-v2, BGE-M3, and mxbai-embed-large, plus transfer runs on Mem0 and MemoryOS.
  • The semantic audit labels all 1,902 raw-miss/source-hit/canonical-hit disagreement cases using five frontier LLMs; the rubric was validated on a 115-case human-annotated subset with Fleiss’ κ = 0.83 under a binary mapping.

Limitations

  • The paper states the native headline matrix is concentrated in one parser-derived store construction, and the Mem0/MemoryOS runs, while broadening evidence, are still a small architectural sample.
  • The semantic audit’s human validation subset is small (115 cases, one primary annotator with a 36-case adjudicated overlap), so prevalence estimates should be interpreted accordingly.
  • Canonical coverage is not random: canonical-covered queries have higher Raw nDCG than uncovered queries, so full-set Raw scores are not directly comparable to Canonical scores on the covered subset; TIAP also treats target membership as binary and does not assign graded credit or compose support across multiple retrieved facts.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2605.24060

Cites

24 of 35 references verified0 citation errors · 11 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

7 of 10 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured

  • Premisenone given

    Conversational-memory systems increasingly transform dialogue history into facts, summaries, timelines, and other source-linked descendants, so a single source turn can coexist with several derived memories in the same retrieval index.

    “Conversational-memory systems increasingly transform dialogue history into facts, summaries, timelines, and other source-linked descendants, so a single source turn can coexist with several derived memories in the same retrieval index.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Chhikara et al., 2025 source
    • cites Kang et al., 2025 source

      The Memory Storage and Update sections (pp. 3–5) describe dialogue pages, summaries of related pages, and facts and traits extracted from past interactions. The Retrieval section (p. 5) describes retrieving these forms of memory from different storage tiers. The paper does not establish timelines, explicit source links, or several derivatives of one turn coexisting in the same retrieval index. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites Rasmussen et al., 2025 source

      Sections 2 and 2.1 (pp. 2–3) describe Zep retaining conversation messages while deriving facts and entity information, with links that let semantic artifacts be traced to source episodes. Sections 2.3 and 3.1 (p. 4) describe summaries and retrieval of facts, entities, and communities. The paper does not establish an increasing trend across conversational-memory systems, nor does it describe timelines as separate derived memories or show a source turn and all its descendants coexisting in a single retrieval index. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
  • Premisecitation

    LoCoMo annotates turn IDs on QA gold answers.

    “LoCoMo annotates turn IDs on QA gold answers (Maharana et al., 2024).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Maharana et al., 2024 source
  • Findingown data

    Switching only the credited target flips target orderings on Mem0 and MemoryOS transfer runs.

    “flips target orderings on Mem0 and MemoryOS transfer runs”
    • Also stated in: § 1 Introduction · plain
      “Transfer runs with Mem0 and MemoryOS remain target sensitive but show different target orderings”
    Quote in the paperSubstantial claim
  • Findingreasoning

    Target coverage can differ across Raw, Source, and Canonical scoring because not every source turn produces a transformed serving memory.

    “Target coverage can differ across Raw, Source, and Canonical scoring because not every source turn produces a transformed serving memory.”
    • Also stated in: § 4.2 Target Mappings · plain
      “Canonical coverage can differ from Raw coverage because not every source turn produces a transformed serving memory.”
    Quote in the paperSubstantial claim
  • Findingown data

    A 1,902-case semantic audit shows that relaxed source-linked credit is fully justified only 29.2% of the time.

    “A 1,902-case semantic audit further shows that relaxed source-linked credit is fully justified only 29.2% of the time, despite high rubric reliability in a validation subset.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 11
  • May combine two claims. 31
  • The quote only partly states this claim. 1

  • Classified as probably not a substantial claim.

    Premisecitation

    Ong et al. (2025) observe that existing long-term conversation datasets lack gold mappings between dialogue contexts and memories.

    “Ong et al. (2025) observe that existing long-term conversation datasets lack gold mappings between dialogue contexts and memories.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Ong et al. (2025) source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    LoCoMo evaluates long-term conversational memory and annotates turn IDs for QA gold answers.

    “LoCoMo(Maharana et al., 2024)evaluates long term conversational memory and annotates turn IDs for QA gold answers.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Maharana et al., 2024 source

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Rasmussen et al. (2025) maintain episodic edges linking extracted facts to source episodes but state that these connections are not examined in their experiments.

    “Rasmussen et al. (2025) maintain episodic edges linking extracted facts to source episodes but state that these connections are not examined in their experiments.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Rasmussen et al. (2025) source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    LoCoMo reports lexical automatic metrics for answer quality and retrieval recall over annotated evidence turns.

    “It reports lexical automatic metrics for answer quality and retrieval recall over annotated evidence turns.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Maharana et al., 2024 source

  • May combine two claims.

    Findingown data

    The transfer systems have zero targetless queries for all three targets, so full run values are also shared subset values.

    “Unlike the native parser runs, these transfer systems have zero targetless queries for all three targets, so full run values are also shared subset values.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Target choice interacts with benchmark structure and memory architecture; target orderings are non-monotonic for both Mem0 and MemoryOS, depending on the benchmark.

    “Instead, target choice interacts with benchmark structure and memory architecture.In transfer runs, target orderings are non-monotonic for both Mem0 and MemoryOS, depending on the benchmark.”
    • Also stated in: § 5.5 Target Sensitivity Transfers Across Memory Architectures · boosted
      “The transfer runs remain strongly target-sensitive across Recall@60, MRR, and nDCG@60, but the target ordering is not uniform (Table 7).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    The semantic audit covers all 1,902 raw_miss / source_hit / canonical_hit disagreement cases extracted from the eight native runs.

    “The semantic audit covers all 1,902 raw_miss / source_hit / canonical_hit disagreement cases extracted from the eight native runs.”
    • Also stated in: § 5.4 Relaxed Credit Is Only Partially Reliable · plain
      “we audit all 1,902 raw_miss / source_hit / canonical_hit disagreement cases across the eight native runs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On LoCoMo, Raw best predicts answer F1, while on LongMemEval-S, Source and Canonical align better with the judge-based scorer.

    “On LoCoMo, Raw best predicts answer F1, while on LongMemEval-S, Source and Canonical align better with the judge-based scorer (Appendix E).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The effect is large, affecting 83.4% to 94.0% of shared queries, surviving transfer to Mem0 and MemoryOS, and not being removed by post-hoc aggregation.

    “We show across two primary benchmarks, one supplementary stress test, and two external memory architectures that the effect is large, affecting 83.4% to 94.0% of shared queries, surviving transfer to Mem0 and MemoryOS, and not being removed by post-hoc aggregation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Across shared queries, Raw versus Canonical rescoring changes nDCG for 83.6% to 93.2% of queries within major LoCoMo categories and for 77.1% to 100.0% of queries within LongMemEval-S categories.

    “Across shared queries, Raw versus Canonical rescoring changes nDCG for 83.6% to 93.2% of queries within major LoCoMo categories and for 77.1% to 100.0% of queries within LongMemEval-S categories.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Queries with a Canonical target are easier under Raw retrieval than queries without Canonical coverage: Raw nDCG is higher by a mean run-level gap of +0.086 with bootstrap 95% CI [+0.049, +0.123], while answer F1 changes much less, with a mean gap of +0.008 and CI [+0.002, +0.014].

    “Queries with a Canonical target are easier under Raw retrieval than queries without Canonical coverage: Raw nDCG is higher by a mean run-level gap of +0.086 with bootstrap 95% CI [+0.049, +0.123], while answer F1 changes much less, with a mean gap of +0.008 and CI [+0.002, +0.014].”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On saved traces, Recall@k plateaus after approximately k = 40, with less than six percentage points of additional recall from k = 40 to k = 60 across retriever and dataset combinations.

    “On saved traces, Recall@k plateaus after approximately k = 40, with less than six percentage points of additional recall from k = 40 to k = 60 across retriever and dataset combinations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On the shared subset with n = 408, the ordering persists, with 0.071 < 0.134 < 0.197.

    “On the shared subset with n = 408, the ordering persists, with 0.071 < 0.134 < 0.197.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On LoCoMo, 33.1% of relaxed credit is fully justified, compared with 21.3% on LongMemEval-S.

    “On LoCoMo, 33.1% of relaxed credit is fully justified, compared with 21.3% 13).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The scoring-target choice in conversational-memory evaluation is often left implicit and can materially change benchmark conclusions.

    “We show that this scoring-target choice is often left implicit and can materially change benchmark conclusions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On full LoCoMo runs, Mem0 follows the same ordering across all three metrics, with Source above Canonical and Canonical above Raw.

    “On full LoCoMo runs, Mem0 follows the same ordering across all three metrics, with Source above Canonical and Canonical above Raw.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    On the canonical-covered shared subset with n = 408, the same Raw < Source < Canonical ordering persists.

    “On the canonical-covered shared subset with n = 408, the same ordering persists.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Mean binary agreement across all ten model pairs is 90.9%, and mean three-class agreement is 85.9%.

    “Mean binary agreement across all ten model pairs is 90.9%, and mean three-class agreement is 85.9%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On LongMemEval-S, the aggregation function determines which ordering emerges: geometric mean and minimum reproduce Raw, while arithmetic mean reproduces Source or Canonical.

    “On LongMemEval-S, the aggregation function determines which ordering emerges: geometric mean and minimum reproduce Raw, while arithmetic mean reproduces Source or Canonical.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Raw versus Canonical comparisons show the highest change rates (83.4%-94.0%), while Raw versus Source rates are lower but still substantial (31.3%-66.2%).

    “Raw versus Canonical comparisons show the highest change rates (83.4%-94.0%),while Raw versus Source rates are lower but still substantial (31.3%-66.2%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Agreement filtering fails: restricting evaluation to queries where all three targets agree on binary hit status retains 72% to 85% of queries, but nDCG values still diverge, with mean per-query spreads between 0.16 and 0.30.

    “Agreement filtering also fails.Restricting evaluation to queries where all three targets agree on binary hit status retains 72% to 85% of queries, but nDCG values still diverge, with mean per-query spreads between 0.16 and 0.30.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Retrieval units are described at the architectural level in most audited papers, but only LoCoMo explicitly labels and separately evaluates distinct unit types.

    “Retrieval units are described at the architectural level in most papers, but only LoCoMo explicitly labels and separately evaluates distinct unit types.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The majority-vote distribution is 29.2% supports, 39.6% partial, and 31.2% does not support.

    “The majority-vote distribution is 29.2% supports, 39.6% partial, and 31.2%does not support.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    TIAP currently treats target membership as binary once a retrieved item falls inside a target set; it does not assign graded credit to partially sufficient evidence or compose support across multiple retrieved facts.

    “TIAP currently treats target membership as binary once a retrieved item falls inside a target set.It does not assign graded credit to partially sufficient evidence or compose support across multiple retrieved facts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    TIAP repeats target comparisons on shared query subsets when coverage differs, including a query only when all compared targets have at least one eligible stored ID.

    “TIAP therefore repeats target comparisons on shared query subsets when coverage differs.A query is included only when all compared targets have at least one eligible stored ID, ensuring that score changes reflect the credited target set rather than changes in the evaluated query population.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The Raw / Source / Canonical basis is a minimal audit basis for the source-linked transformed-memory systems studied, not a complete taxonomy of conversational memory.

    “The Raw / Source / Canonical basis used here is a minimal audit basis for the source-linked transformed-memory systems we study.”
    • Also stated in: § 7 Limitations · plain
      “We therefore treat this basis as an instantiation for this setting, not as a complete taxonomy of conversational memory.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    This is a benchmark-design problem rather than a system-tuning one: the audit varies the evaluation definition while holding system output fixed.

    “This is a benchmark-design problem rather than a system-tuning one: unlike hyperparameter sensitivity analysis, which varies a system knob while holding evaluation fixed, our audit varies the evaluation definition while holding system output fixed.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The five-model majority vote reduces single-model bias, but the human validation subset is still small (115 cases, one primary annotator with a 36-case adjudicated overlap), so prevalence estimates should be interpreted accordingly.

    “The five-model majority vote reduces single-model bias, but the human validation subset is still small (115 cases, one primary annotator with a 36-case adjudicated overlap), so prevalence estimates should be interpreted accordingly.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Graph-native memories may require node-, edge-, or path-level target definitions beyond the Raw/Source/Canonical basis used here.

    “Graph-native memories may require node-, edge-, or path-level target definitions beyond the Raw/Source/Canonical basis used here.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Binary target membership should be read as an audit baseline rather than a complete model of evidential sufficiency.

    “Our semantic audit surfaces exactly this boundary, so binary target membership should be read as an audit baseline rather than a complete model of evidential sufficiency.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Queries, reference answers, and evidence labels are held out and used only at evaluation time, never provided to the parser that produces transformed memories, so they cannot leak into the memory store.

    “Across all benchmarks, queries, reference answers, and evidence labels are held out and used only at evaluation time.They are never provided to the parser that produces transformed memories, so they cannot leak into the memory store.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The audit is distinct from classical IR evaluation-sensitivity analysis, which varies relevance judgments or changes the retrieval unit globally rather than crediting system-generated descendants of a single source turn.

    “It is also distinct from classical IR evaluation-sensitivity analysis, which varies relevance judgments or changes the retrieval unit globally rather than crediting system-generated descendants of a single source turn; we develop this contrast in Section 2.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Target choice changes both absolute retrieval scores and the relative evaluation landscape across datasets and retrievers.

    “Thus, target choice changes both absolute retrieval scores and the relative evaluation landscape across datasets and retrievers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Source versus Canonical target choice matters less when the answer is a simple single-session fact, but becomes more consequential when a query requires aggregation, update tracking, or reasoning across sessions.

    “This suggests that Source versus Canonical target choice matters less when the answer is a simple single-session fact, but becomes more consequential when a query requires aggregation, update tracking, or reasoning across sessions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    A more abstract target is not always better, because Raw and Canonical trade places depending on where the relevant detail survives and which metric is used.

    “A more abstract target is not always better, because Raw and Canonical trade places depending on where the relevant detail survives and which metric is used.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Of the 1,902-case semantic audit, 1,876 cases received valid labels from all five models; 26 cases returned empty responses from GPT-5.5 or DeepSeek V4 Pro after retries.

    “Of these cases, 1,876 received valid labels from all five models; 26 cases returned empty responses from GPT-5.5 or DeepSeek V4 Pro after retries.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The difference in answer alignment between LoCoMo and LongMemEval-S is consistent with their answer scorers: LoCoMo's lexical scoring rewards exact answer wording more directly, whereas LongMemEval-S's judge-based scoring is more tolerant of paraphrased or transformed evidence.

    “This difference is consistent with the benchmarks' answer scorers: LoCoMo's lexical scoring rewards exact answer wording more directly, whereas LongMemEval-S's judge-based scoring is more tolerant of paraphrased or transformed evidence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    BEAM includes a source_chat_ids field in its question schema, although some published examples leave this field empty.

    “BEAM includes a source_chat_ids field in its question schema, although some published examples leave this field empty (Tavakoli et al., 2026).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Tavakoli et al., 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductown data

    Source-linked lineage is recoverable from evidence at the level of source conversation turns or sessions, allowing construction of the Raw, Source, and Canonical target distinction.

    “This makes source linked lineage recoverable and allows us to construct the Raw, Source, and Canonical target distinction.”
    Quote in the paperSubstantial claim
39 of 39

  • The paper relies on this without support.

    In most transformed memory stores, the Raw target set and the Canonical target set are disjoint: the former contains the original evidence turn, while the latter contains transformed serving memories derived from it.

    “In most transformed memory stores, T raw (q, S) and T can (q, S) are disjoint: the former contains the original evidence turn, while the latter contains transformed serving memories derived from it.”

  • The authors state this as an assumption.

    The audit assumes stable source anchors s(q), recoverable lineage from retrieved IDs back to those anchors, enumerable target sets over the descendants, and exported fixed ranked traces.

    “The audit also assumes stable source anchors s(q), recoverable lineage from retrieved IDs back to those anchors, enumerable target sets over the descendants, and exported fixed ranked traces.”

  • The paper relies on this without support.

    A benchmark query is typically anchored to one source turn, while the evaluated store may contain several retrieval units derived from that same evidence.

    “A benchmark query is typically anchored to one source turn, while the evaluated store may contain several retrieval units derived from that same evidence.”
3 of 3

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Panthi, S., & Abdelfattah, R.
Affiliations
University of Southern Mississippi (US)
ORCID
Abdelfattah, Rabab
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2605.24060
arXiv
2605.24060
OpenAlex
W7162457387
Semantic Scholar
9f40dfa03e1110be6c88aaca0313a31a850b3ea1
Cited by
0
References
32
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
conversational memory, NDCG, LongMemEval, LoCoMo, benchmark design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review