CoChat
Sign Up
PreprintCited by 30Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack

Open access
Kuratov, Yuri, Bulatov, Aydar, Anokhin, Petr, Rodkin, Ivan, Sorokin, Dmitry, Sorokin, Artyom, Burtsev, Mikhail·2024·arXiv
DOI 10.48550/arxiv.2406.10149arXiv 2406.10149
Open this paper in CoChat — free accountOpen PDF

Abstract

In recent years, the input context sizes of large language models (LLMs) have increased dramatically. However, existing evaluation methods have not kept pace, failing to comprehensively assess the efficiency of models in handling long contexts. To bridge this gap, we introduce the BABILong benchmark, designed to test language models' ability to reason across facts distributed in extremely long documents. BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets. These tasks are challenging on their own, and even more demanding when the required facts are scattered across long natural text. Our evaluations show that popular LLMs effectively utilize only 10-20\% of the context and their performance declines sharply with increased reasoning complexity. Among alternatives to in-context reasoning, Retrieval-Augmented Generation methods achieve a modest 60\% accuracy on single-fact question answering, independent of context length. Among context extension methods, the highest performance is demonstrated by recurrent memory transformers after fine-tuning, enabling the processing of lengths up to 50 million tokens. The BABILong benchmark is extendable to any length to support the evaluation of new upcoming models with increased capabilities, and we provide splits up to 10 million token lengths.

Verified 2026-10-11 · 1 reference not checked yet

Integrity signals

Registry

DataCite · 10.48550/arxiv.2406.10149

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

12 of 58 cited sources checked · 9 supported · 0 flagged · accuracy not yet measured

  • Premisereasoning

    Existing evaluation methods have not kept pace with the increasing context sizes of large language models, failing to comprehensively assess their efficiency in handling long contexts.

    “However, existing evaluation methods have not kept pace, failing to comprehensively assess the efficiency of models in handling long contexts.”
    • Also stated in: § 1 Introduction · plain
      “Despite these advances in models capabilities, the benchmarks used to evaluate them have not kept pace.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Bai et al., 2023

      The Introduction says that, unlike for short contexts, no multi-task benchmark existed for a multi-aspect evaluation of long-context understanding. Related Work says existing metrics and benchmarks may not reflect real-world performance or cover enough task types. These passages support the gap in comprehensive evaluation, but the paper frames that gap around understanding ability, not the efficiency of models in handling long contexts. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites An et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: The original default needle-in-a-haystack setup evaluates model predictions with an LLM (GPT-3.5-turbo) on a 1-10 scale, using the same single needle for each position and document length.
  • Premisecitation

    Models are capable of processing hundreds of thousands and millions of tokens.

    “while models are capable of hundreds of thousands and millions of tokens (Rodkin et al., 2024;Reid et al., 2024;Bulatov et al., 2024;Anthropic, 2024;Liu et al., 2024a;Gu & Dao, 2023;OpenAI, 2023a).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Rodkin et al., 2024 source
    • cites Reid et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Bulatov et al., 2024 source
  • Premisecitation

    Current benchmarks such as Longbench and L-Eval scale only up to 40,000 tokens.

    “current benchmarks such as Longbench (Bai et al., 2023) and L-Eval An et al. (2023) scale only up to 40,000 tokens”
    • Also stated in: § Additional information · plain
      “current benchmarks such as Longbench (Bai et al., 2023) and L-Eval An et al. (2023) scale only up to 40,000 tokens”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Bai et al., 2023

      The Conclusion (section 5) says LongBench includes data lengths from thousands to tens of thousands of tokens, supporting the general scale attributed to LongBench. It does not establish 40,000 tokens as an upper limit. The separate claim about L-Eval is not assessed against this citation. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites An et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingcitation

    BABILong has a higher correlation with MMLU on short contexts (0K) than RULER.

    “BABILong has a higher correlation with MMLU on short contexts (0K) than RULER (Hsieh et al., 2024).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Hsieh et al., 2024

      The Experiments & Results section (p. 6) evaluates RULER at context lengths beginning at 4K, while the Task Correlation Analysis (p. 19) compares tasks within RULER. Searches for BABILong and MMLU found no passage comparing either benchmark’s correlation with MMLU at 0K; BABILong appears only in the references (p. 13). (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Findingcitation

    ARMT successfully scales to 50 million tokens.

    “While ARMT successfully scales even further, reaching up to 50 million tokens (Rodkin et al., 2024).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Rodkin et al., 2024 source
  • Classified as probably not a substantial claim. 84
  • May combine two claims. 63

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Works like RWKV, S4, and Mamba focus on advancing recurrent networks to reach high parallelism levels available to Transformers while retaining the linear complexity of RNN.

    “Works like RWKV (Peng et al., 2023a), S4 (Gu et al., 2021), Mamba (Gu & Dao, 2023), take another approach and focus on advancing recurrent networks to reach high parallelism levels available to Transformers while retaining the linear complexity of RNN.”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Peng et al., 2023a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Gu et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Gu & Dao, 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Big Bird, Longformer, and LongNet help extend context length for Transformers by switching from full self-attention to sparse self-attention mechanisms with linear complexity.

    “Big Bird (Zaheer et al., 2020), Longformer (Beltagy et al., 2020), LongNet (Ding et al., 2023) help extend context length for Transformers by switching from full self-attention to sparse self-attention mechanisms with linear complexity.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Zaheer et al., 2020 source
    • cites Beltagy et al., 2020 source
    • cites Ding et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The authors follow the recommended Datasheets for Datasets form.

    “We follow recommended Datasheets for Datasets form (Gebru et al., 2021).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Gebru et al., 2021 source

  • May combine two claims.

    Findingnone given

    The new Phi-3.5-MoE performs similarly to Phi-3-medium, but with only 6.6B active parameters compared to 14B in Phi-3-medium.

    “The new Phi-3.5-MoEperforms similarly to Phi-3-medium, but with only 6.6B active parameters compared to 14B in Phi-3-medium.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Phi-3-mini drops significantly from 64K to 128K, reaching less than 10%, while Phi-3-medium maintains 30% at 128K.

    “Phi-3-mini drops significantly from 64K to 128K, reaching less than 10%, while Phi-3-medium maintains 30% at 128K.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Qwen-2.5 models outperform LLama-3.1 of similar size and achieve the best results of all evaluated open LLMs on BABILong.

    “Qwen-2.5 models outperform LLama-3.1 of similar size and achieve the best results of all evaluated open LLMs on BABILong.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Retrieval performance using sentence chunks is superior to that of 512-token segments on QA1, with a notable decrease in accuracy observed already after 16k token context length.

    “The findings from the QA1 task, depicted in Figure 3a, indicate that retrieval performance using sentence chunks is superior to that of 512-token segments, with a notable decrease in accuracy observed already after 16k token context length.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    Yarn fails to extend to longer contexts despite showing stable results in long-context language modeling.

    “Yarn fails to extend to longer contexts despite showing stable results in long-context language modeling (Peng et al., 2023b).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Peng et al., 2023b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingnone given

    Jamba-v1 does not have a drop at 128K and shows 34% on this length.

    “Jamba-v1 does not have drop at 128K and shows 34% on this length.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Command-R and Phi-3-medium are the most robust to longer contexts, but start to lose performance more sharply at 128K.

    “Command-R and Phi-3-medium are the most robust to longer contexts, but start to lose performance more sharply at 128K.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    LLama-3.1-8B matches GPT-4o-mini, while LLama-3.1-70B outperforms GPT-4 on longer contexts.

    “LLama-3.1 models show significant improvement: LLama-3.1-8Bmatches GPT-4o-mini, while LLama-3.1-70Boutperforms GPT-4 on longer contexts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Activation Beacon performed better than Yarn context extension method for Mistral 7B, but still achieved low results (< 40%) on 32K contexts.

    “Activation Beacon performed better than Yarn context extension method for Mistral 7B, but still achieved low results (< 40%) on 32K contexts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 and LLama-3.1-70b perform well up to 16K tokens on QA1, while Qwen-2.5-70b and Gemini Pro 1.5 perform well up to 64K.

    “except for GPT-4 and LLama-3.1-70b, which perform well up to 16K, as well as Qwen-2.5-70band Gemini Pro 1.5 up to 64K.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Finetuned RMT, ARMT, and Mamba achieve outstanding scores across most sequence lengths, significantly outperforming LLMs despite having up to 100 times fewer parameters.

    “Finetuned RMT, ARMT, and Mamba achieve outstanding scores across most sequence lengths, significantly outperforming LLMs despite having up to 100 times ewer parameters.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Evaluation time grows linearly with context length.

    “As shown in Table 3, evaluation time grows linearly with context length.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    BABILong offers algorithmically adaptable document length and facts placement, and includes predefined sets of bins ranging from 0k to 10M tokens.

    “The BABILong benchmark offers algorithmically adaptable document length and facts placement, includes predefined sets of bins ranging from 0k to 10M tokens.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The best correlated RULER lengths with MMLU are 64K and the average of all lengths (<=128K).

    “The best correlated RULER lengths with MMLU are 64K and the average of all lengths (<=128K).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    QA4 is significantly more challenging, and only 5 models can reach this level of performance.

    “QA4 is significantly more challenging, and only 5 models can reach this level of performance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Recurrent Memory models greatly outperform retrieval-augmented models and are able to process sequences up to 10M and 50M tokens much faster than Mamba.

    “Recurrent Memory models greatly outperform retrieval-augmented models and are able to process sequences up to 10M and 50M tokens much faster than Mamba.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Mamba has an advantage on medium-length sequences, but recurrent memory models (RMT and ARMT) excel in processing much larger sequences up to 10M tokens.

    “Mamba has an advantage on medium-length sequences, but recurrent memory models (RMT and ARMT) excel in processing much larger sequences up to 10M tokens.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On QA4 and QA5, retrieval allows matching and even surpassing weaker competitors on longer context sizes.

    “On QA4 and QA5 retrieval allows to match and even surpass weaker competitors on longer context sizes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The RAG pipeline with GPT-4-turbo shows scalable but weak performance on BABILong for sentence embeddings and poor scalability with chunk embeddings.

    “The RAG pipeline with GPT-4-turbo shows scalable but weak performance on BABILong for sentence embeddings and poor scalability with chunk embeddings (see Fig. 3a).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    Recent open-source models LLama-3.1 and Qwen-2.5 show the best performance of pre-trained LLMs, with LLama-3.1 using pre-training and supervised fine-tuning on long context data up to 128K tokens.

    “The recent open-source models LLama-3.1 and Qwen-2.5 show the best performance of pre-trained LLMs, with LLama-3.1 using pre-training and supervised fine-tuning on long context data up to 128K tokens.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The proposed benchmark includes 20 diverse tasks, ranging from simple needle-in-a-haystack scenarios with distractor facts to more complex tasks requiring counting, logical reasoning, or spatial reasoning.

    “The proposed benchmark includes 20 diverse tasks, ranging from simple”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Fine-tuning improves the performance of GPT-3.5-Turbo and Mistral-7B, but their context lengths remain limited to 16K and 32K, respectively.

    “Fine-tuning improves the performance of GPT-3.5-Turbo and Mistral-7B, but their context lengths remain limited to 16K and 32K, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    BABILong shows high correlation on short contexts with MMLU and diverges from it as lengths increase.

    “Compared to other benchmarks, BABILong shows high correlation on short contexts with MMLU and diverges from it as lengths increases.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    RMT scales to context sizes of 1 million, 10 million, and 11.1 million tokens.

    “Surprisingly, with context sizes scaling to 1 million, 10 million tokens, and even 11.1 million tokens, which is over 600 times of the training length.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Comparing correlations of BABILong with MMLU at the most correlated RULER lengths shows much lower values: 0.928 vs. 0.435 and 0.910 vs. 0.455.

    “Comparing the correlations of BABILong with MMLU at the most correlated RULER lengths (<=128K and 64K) shows much lower values: 0.928 vs. 0.435 and 0.910 vs. 0.455, respectively.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The low-variety issue is partially mitigated by generating distractor facts using the same vocabularies.

    “This issue is partially mitigated by generating distractor facts using the same vocabularies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Mamba is hard to infer on lengths more than 128K tokens, while RMT and ARMT enable processing of lengths up to 50 million tokens.

    “However, Mamba is hard to infer on lengths more than 128K tokens, while RMT and ARMT enables the processing of lengths up to 50 million tokens.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Fine-tuning dramatically improves performance for longer contexts, making scores uniform across all input sizes.

    “Fine-tuning dramatically improves performance for longer contexts making scores uniform across all input sizes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    BABILong provides more tasks and more natural and deceptive mixing of information into background documents than existing approaches.

    “Compared to existing approaches it provides more tasks and more natural and deceptive mixing of information into background documents.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    BABILong is solvable by recurrent approaches like RMT.

    “However, BABILong is solvable by this type of models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    RULER maintains a high correlation with MMLU regardless of task length, with an even higher correlation at 64K, while BABILong's correlation with MMLU decreases with length.

    “However, RULER maintains a high correlation regardless of task length, with an even higher correlation at 64K, while BABILong's correlation with MMLU decreases with length.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    QA1 and QA5 are the easiest tasks, with most models achieving over 70% accuracy for the 0k split.

    “QA1 and QA5 are the easiest, with most models achieving over 70% accuracy for the 0k split.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Lengths beyond 10M tokens can be generated and models were tested up to 50M tokens.

    “Lengths beyond 10M tokens could be generated and we test models up to 50M tokens.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    BABILong consists of 20 diverse tasks covering multi-hop tracing, aggregation over needles, deduction and induction, time, positional, and size reasoning, and path finding.

    “BABILong consists of diverse set of 20 tasks that cover different capabilities including multi-hop tracing, aggregation over needles and extending them with basic deduction and induction, time, positional, and size reasoning, and path finding.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    For the evaluation set with 100 samples per task and per length, 100 test samples were randomly sampled from the full test set.

    “For evaluation set with 100 samples per task and per length, we randomly sampled 100 test samples from full test set.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The facts in BABILong are about fictional people, places, animals, and items.

    “The facts are about fictional people, places, animals, and items.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    GPT-4 effectively uses only about 10% of the full 128K window.

    “GPT-4 effectively uses only about 10% of the full 128K window.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    An extended version of BABILong with 60,000 samples, covering five tasks and offering 1,000 samples per split instead of 100, is available.

    “An extended version with 60,000 samples, covering five tasks and offering 1,000 samples per split instead of 100, is also available”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The released pre-generated version of BABILong includes 13,000 samples, divided into 13 context length splits across 10 tasks.

    “The released pre-generated version includes 13,000 samples, divided into 13 context length splits across 10 tasks”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets.

    “BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Changes in memory mainly occur when the model meets a new fact, which indicates model adaptation to distinguishing and storing facts in memory.

    “Changes in memory mainly occurs when the model meets a new fact, which indicates model adaptation to distinguishing and storing facts in memory.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The BABILong dataset is generative, offering an unlimited number of possible instances.

    “The BABILong dataset is generative, offering an unlimited number of possible instances.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The number of tokens for tokenizers of different models may differ for samples in the same split.

    “As shown in Table 5, the number of tokens for tokenizers of different models may differ for samples in the same split.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The comparison remains fair despite tokenizer differences between models.

  • May combine two claims.

    Findingown data

    Small LMs, ARMT & RMT with GPT-2 (137M) and Mamba (130M) fine-tuned for the task are able to solve it, with recurrent memory transformers scoring well up to record 50 000 000 tokens.

    “Small LMs, ARMT & RMT with GPT-2 (137M) and Mamba (130M) fine-tuned for the task are able to solve it, with recurrent memory transformers scoring well up to record 50 000 000 tokens.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Each sample in the BABILong dataset is assigned a label, which is the answer to the corresponding question.

    “Yes, each sample in the BABILong dataset is assigned a label, which is the answer to the corresponding question.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    BABILong inherits train and test splits from the bAbI dataset.

    “We inherit train and test splits from the bAbI (Weston et al., 2016) dataset.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Weston et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The released pre-generated version of BABILong is available on Hugging Face.

    “and is available on Hugging Face: https://huggingface.co/datasets/RMT-team/babilong.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The question in a BABILong sample can be added to either the beginning or the end of the resulting text sequence.

    “The question can be added to either the beginning or the end of the resulting text sequence.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingcitation

    The test set of BABILong combines sentences of books from the PG-19 test split with all test samples from bAbI tasks.

    “The test set of BABILong combines sentences of books from the PG-19 (Rae et al., 2020) test split with all test samples from bAbI (Weston et al., 2016) tasks.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Rae et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Weston et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    In the train split, all train samples from bAbI and randomly sampled texts from the PG-19 train split are used.

    “In train split, we use all train samples from bAbi and randomly sampled texts from PG-19 train split.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingcitation

    Each sample is a text document, combined from PG-19 books and facts and questions from the bAbI dataset.

    “Each sample is a text document, combined from PG-19 books (Rae et al., 2020) and facts and questions from the bAbI dataset (Weston et al., 2016).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Rae et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Weston et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    In the absence of a fact in the input, the memory remains similar to its initial states, but the introduction of a fact leads to a visible change in the memory state.

    “In the absence of fact in input, the memory remains similar to its initial states, but the introduction of fact leads to visible change in the memory state.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The BABILong benchmark uses background texts to hide facts in them.

    “The BABILong benchmark uses background texts to hide facts in them.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Background texts from PG-19 for the training set are randomly sampled.

    “Background texts from PG-19 for the training set are randomly sampled.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    For the test sets, background texts are fixed and pre-generated test splits are provided.

    “For the test sets, we fix the background texts and provide pre-generated test splits”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The BABILong dataset is already available.

    “The BABILong dataset is already available (see Section A).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    BABILong evaluation data is hosted on HuggingFace Datasets and code is hosted on GitHub.

    “Yes, we use HuggingFace Datasets to host evaluation data and GitHub for code (see Section A).”
    • Also stated in: § Additional information · plain
      “We use HuggingFace Datasets to host evaluation data and Croissant metadata, GitHub for code and data generation (see Section A).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The authors of the dataset are supporting, hosting, and maintaining the dataset.

    “The authors of the dataset.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingcitation

    The data was directly derived from PG-19 and bAbI by mixing sentences of these two datasets.

    “The data was directly derived from PG-19 (Rae et al., 2020) and bAbI (Weston et al., 2016) by mixing sentences of these two datasets.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Rae et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Weston et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Test sets of BABILong are self-contained and hosted on HuggingFace.

    “Test sets are self-contained and hosted on HuggingFace (see Section A).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Data from PG-19 corpora is under Apache 2.0 License and data from bAbI dataset is under BSD License.

    “We use data from PG-19 corpora (Rae et al., 2020) (Apache 2.0 License) and bAbI dataset (Weston et al., 2016) (BSD License).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Rae et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Weston et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    No export controls or other regulatory restrictions apply to the dataset or to individual instances.

    “No.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The authors are not aware of any third parties imposing IP-based or other restrictions on the data.

    “We are not aware of it.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The BABILong dataset does not relate to people.

    “No.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Code is provided to generate train data of arbitrary lengths.

    “We provide code to generate train data of arbitrary lengths (see Section A).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Texts from PG-19 may contain mentions of the same entities, places or items that are used in facts from bAbI.

    “Texts from PG-19 (Rae et al., 2020) may contain mentions of the same enities, places or items that are used in facts from bAbI (Weston et al., 2016).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Rae et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Weston et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    PG-19 texts may contain instances of offensive, insulting, or threatening content, or content that might cause anxiety.

    “it is possible that they contain instances of offensive, insulting, or threatening content, or content that might cause anxiety.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rae et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Train data relies on bAbI and PG-19 datasets, both of which are available online.

    “Train data relies on bAbI (Weston et al., 2016) and PG-19 datasets both of which are available online.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Weston et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Our code is released under Apache 2.0 License.

    “Our code is released under Apache 2.0 License.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    BABILong was first uploaded on February 16, 2024.

    “BABILong was first uploaded on February 16, 2024.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    All data was collected using the software developed in this paper, which is available on GitHub.

    “All data was collected using the software developed in this paper, which is available on GitHub: https://github.com/ booydar/babilong.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The BABILong dataset does not contain confidential data.

    “No.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    BABILong includes 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets.

    “BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets”
    • Also stated in: § 1 Introduction · plain
      “BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets, that were designed as prerequisites for any system that aims to be capable of conversing with a human (Weston et al., 2016).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Weston et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: The brevity and similarity of the task sentences enable the model to distinguish them from seemingly close background text with the assistance of few-shot examples.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Generated benchmarks such as bAbI and BABILong are immune to data contamination.

    “Generated benchmarks, such as bAbI and BABILong, are immune to this type of contamination.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    BABILong allows construction of tasks of almost arbitrary length, with predefined splits up to 10 million tokens.

    “In this way, BABILong allows the construction of tasks of almost arbitrary length, in order to adapt them to the evaluation of new, more powerful models in an extensible and controllable way.We provide sets of predefined lengths with splits up to 10 million tokens”
    • Also stated in: § Abstract · plain
      “The BABILong benchmark is extendable to any length to support the evaluation of new upcoming models with increased capabilities, and we provide splits up to 10 million token lengths.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The difference in distributions enables the scalability of BABILong to large amounts of diverse noise.

  • May combine two claims.

    Premisecitation

    The widely used needle-in-a-haystack task is overly simplistic, and novel long-context models often achieve perfect performance on it.

    “However, the widespread use of this approach has highlighted its limitations -it is overly simplistic, and novel long context models often achieve perfect performance, as usually demonstrated by fully green heatmaps (Reid et al., 2024;Cohere, 2024;Liu et al., 2024a;Wang et al., 2024c).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Reid et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Cohere, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Liu et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    This is the first combination of a recurrent transformer with a trainable retrieval mechanism.

    “As far as we know, this is the first combination of a recurrent transformer with a trainable retrieval mechanism.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    RMT is used in the experiments because of its simplicity, plug-and-play compatibility with pretrained transformer-based language models, and promising scaling capabilities.

    “We use RMT in our experiments because of its simplicity, plug-and-play compatibility with pretrained transformer-based language models, and promising scaling capabilities (Bulatov et al., 2024).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Bulatov et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Activation Beacon compresses activations from prior segments using separate parameters and integrates a sliding window mechanism, handling up to 400k tokens.

    “Activation Beacon (Zhang et al., 2024a) compresses activations from prior segments using separate parameters and integrates a sliding window mechanism, handling up to 400k tokens.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhang et al., 2024a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Study design and conductnone given

    The prompts selected for LLM experiments could be suboptimal.

    “We tried several prompts experiments with LLMs, but the ones that we selected could be suboptimal.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    BABILong can detect differences in models behavior starting from lengths as small as 2K tokens, while RULER requires lengths of at least 128K tokens to show significant differentiation from MMLU.

    “These results show that BABILong can detect differences in models behavior starting from lengths as small as 2K tokens, while RULER requires lengths of at least 128K tokens to show significant differentiation from relatively short MMLU benchmark.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    In GPT-4 and LLama-3 with RAG experiments, the retrieval component was not optimized.

    “In GPT-4 and LLama-3 with RAG experiments, we do not optimize the retrieval component.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    A random seed was fixed for sampling background texts from PG19 for the test set, but the seed was not fixed during training to avoid overfitting to specific sampled texts.

    “We fix a random seed used to sample background texts from PG19 for the test set.However, the seed was not fixed to avoid overfitting to specific sampled texts during training.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    For each curriculum step n, the number of segments was chosen randomly from 1 to n for every batch to prevent overfitting to a certain context size.

    “For each curriculum step n we chose the number of segments randomly from 1 to n for every batch to prevent overfitting to a certain context size.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    These works show promising results on long sequences but are still lagging behind the best transformer models in natural language processing tasks.

    “These works show promising results on long sequences but are still lagging behind the best transformer models in natural language processing tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    For reproducibility, dataset splits were pre-generated for several fixed lengths: 0k, 4k, 8k, 16k, 32k, 64k, 128k, 512k, 1M and 10M tokens.

    “For reproducibility, we pre-generate dataset splits for several fixed lengths: 0k (tasks with no distractor sentences), 4k, 8k, 16k, 32k, 64k, 128k, 512k, 1M and 10M tokens.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    To evaluate recurrent models on BABILong, the full test set was used for sequences up to 1M tokens, and for 10M tokens results are averaged over 100 samples.

    “To evaluate recurrent models on the BABILong benchmark we use the full test set for sequences up to 1M tokens, for 10M tokens we only provide results on averaged over 100 samples.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    In the experiments, only PG19 and Wiki were tried as background text sources.

    “In our experiments, we only tried PG19 and Wiki as background text sources.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: PG19 and Wiki contain natural narratives and facts about people similar to bAbI tasks.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Facts in BABILong can be generated making it leak-proof for future LLMs.

    “Facts in BABILong could be generated making it leak-proof for future LLMs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    For each experiment, three runs were conducted with different memory initializations and dataset shuffles.

    “For each experiment we conducted three runs with different memory intitalizations and dataset shuffles.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premiseown data

    BABILong provides information on which facts are relevant.

    “Additionally, BABILong provides information on which facts are relevant, which can be used for supervision or more detailed metrics and analysis of systems.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    PG-19 is a collection of free books from Project Gutenberg published before 1919.

    “PG-19 is a collection of books published before 1919.”
    • Also stated in: § Additional information · plain
      “PG-19 corpora is a collection of free books from Project Gutenberg 13 published before 1919.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rae et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    The model learned to distinguish important facts from background text and preserve them in memory until a question appears.

    “This indicates that the model learned to distinguish important facts from the background text and preserving them in memory until a question appears.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    RMT learns to detect and store relevant facts using memory.

    “RMT learns to detect and store relevant facts using memory.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Contributions can be made via Pull Requests on GitHub and HuggingFace datasets.

    “Yes, contributions can be made via Pull Requests on GitHub and HuggingFace datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The BABILong dataset uses data from PG-19 and bAbI, both of which are available online independently.

    “The BABILong dataset uses data from PG-19 (Rae et al., 2020) and bAbI (Weston et al., 2016), both of which are available online independently.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Rae et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Weston et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    The dataset can be used for various tasks beyond long-context evaluation, and its usage is not restricted to a specific set of tasks.

    “The dataset can be used for various tasks beyond long-context evaluation, and we do not restrict its usage to a specific set of tasks.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The reasoning tasks embedded within the texts are designed to challenge specific reasoning abilities in LLMs based on the bAbI dataset.

    “The reasoning tasks embedded within the texts are designed to challenge specific reasoning abilities in LLMs based on the bAbI dataset (Weston et al., 2016).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Weston et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisenone given

    Facts relevant for the question are hidden inside a larger background text from PG19.

    “Facts relevant for the question are hidden inside a larger background texts from PG19.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    BABILong contains multiple facts distributed over long texts that need to be combined to get the correct answer.

    “BABILong contains multiple facts distributed over long texts that need to be combined to get the correct answer.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The authors' code is released under the Apache 2.0 License.

    “Our code is released under the Apache 2.0 License.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The authors provide code that generates BABILong data from PG-19 and bAbI datasets on-the-fly.

    “Yes, we provide code that generates BABILong data from PG-19 (Rae et al., 2020) and bAbI (Weston et al., 2016) datasets on-the-fly (see Section A).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Rae et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Weston et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The authors release the predictions of all models used in their study to ensure that all reported results can be reproduced and verified.

    “Additionally, we release the predictions of all models used in our study to ensure that all reported results can be reproduced and verified: https://github.com/booydar/babilong/tree/ predictions_06_2024.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    BABILong uses texts from the PG-19 corpus, which consists of books published before 1919.

    “The BABILong dataset uses texts from the PG-19 corpus (Rae et al., 2020), which consists of books published before 1919.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rae et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Code for generating data and evaluating models is available at https://github.com/booydar/babilong.

    “Code for generating data and evaluating models is available at https://github.com/booydar/ babilong.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The authors use data from the PG-19 corpora (Apache 2.0 License) and the bAbI dataset (BSD License).

    “We use data from the PG-19 corpora (Rae et al., 2020) (Apache 2.0 License 7 ) and the bAbI dataset (Weston et al., 2016) (BSD License 8 ).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Rae et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Weston et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Some possible applications include training and/or evaluating multi-hop question-answering systems or retrieval systems.

    “Some possible applications include training and/or evaluating multi-hop question-answering systems or retrieval systems, as BABILong contains multiple facts distributed over long texts that need to be combined to get the correct answer.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The authors also provide pre-generated evaluation data hosted on HuggingFace datasets.

    “We also provide pre-generated evaluation data hosted on HuggingFace datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    A binary metric was adopted for evaluating retrieval accuracy, where the criterion was the presence or absence of relevant facts within the retrieved text chunks.

    “We adopted a binary metric for evaluating retrieval accuracy, where the criterion was the presence or absence of relevant facts (singular or multiple, based on the specific task) within the retrieved text chunks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The authors' code includes data generation, metrics, and the evaluation pipeline used to benchmark models.

    “Our code includes data generation, metrics, and the evaluation pipeline used to benchmark models.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    The tasks differ by the number of facts and the task complexity, testing the ability for multiple reasoning aspects.

    “The tasks differ by the number of facts and the task complexity, testing the ability for multiple reasoning aspects.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    This method ensures controlled testing conditions but may not accurately reflect the type of reasoning required in real-world scenarios.

    “This method ensures controlled testing conditions but may not accurately reflect the type of reasoning required in real-world scenarios.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The PG-19 dataset can be replaced with other sources of text, such as Wikipedia.

    “Additionally, the PG-19 dataset can be replaced with other sources of text, such as Wikipedia.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The evaluation sets include 100 samples per length and per task, with lengths from 0k (no background text from PG-19) to 10 million tokens, and 1000 samples per length and per task with lengths from 0k to 128k tokens.

    “The evaluation sets include 100 samples per length and per task, with lengths from 0k (no background text from PG-19) to 10 million tokens: https://huggingface.co/datasets/RMT-team/babilong and 1000 samples per length and per task with lengths from 0k to 128k tokens: https://huggingface. co/datasets/RMT-team/RMT-team/babilong-1k-samples.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The croissant metadata for both evaluation sets is provided by HuggingFace.

    “The croissant metadata for both evaluation sets is provided by HuggingFace:”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The bAbI dataset was released in 2015.

    “The bAbI dataset was released in 2015.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Weston et al., 2016

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Interference between similar facts in the background text and facts from bAbI can make the benchmark more difficult.

    “Interference between similar facts in the background text and facts from bAbI can make the benchmark more difficult.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    The obtained sequence lengths were validated to match the desired values.

    “The obtained sequence lengths were validated to match the desired values.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    PG-19 was released in 2019.

    “The PG-19 corpora contains books published before 1919 and it was released in 2019.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rae et al., 2020

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

122 of 122

  • The paper relies on this without support.

    The original default needle-in-a-haystack setup evaluates model predictions with an LLM (GPT-3.5-turbo) on a 1-10 scale, using the same single needle for each position and document length.

    “Another major drawback of the original default setup 1 is that model predictions are evaluated and scored by an LLM (GPT-3.5-turbo) on a scale of 1 to 10, with the same single needle used for each position and document length.”

  • The paper relies on this without support.

    The brevity and similarity of the task sentences enable the model to distinguish them from seemingly close background text with the assistance of few-shot examples.

    “The brevity and similarity of the task sentences also enable the model distinguish them from seemingly close background text with the assistance of few-shot examples.”

  • The paper relies on this without support.

    The difference in distributions enables the scalability of BABILong to large amounts of diverse noise.

    “This difference in distributions enables the scalability of BABILong to large amounts of diverse noise.”

  • The paper relies on this without support.

    PG19 and Wiki contain natural narratives and facts about people similar to bAbI tasks.

    “PG19 and Wiki were chosen because they contain natural narratives and facts about people, in a way similar to bAbI tasks.”

  • The paper relies on this without support.

    The comparison remains fair despite tokenizer differences between models.

    “However considering the trade-off between the sequence length and embedding layer size, we believe the comparison remains fair.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Long-context benchmarks test single-pass attention over a given context, not the write-and-retrieve loop of a persistent memory.

    “Long-context benchmarks (RULER, Hsieh et al., 2024; BABILong, Kuratov et al., 2024) test single-pass attention over a given context, not the write-and-retrieve loop of a persistent memory.”

    From When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents (2026) · cites this paper as Kuratov et al., 2024

    Section 2 describes BABILong as testing reasoning over a supplied long context. But Appendix H, “Recurrent Memory Transformer Analysis” (page unavailable), explicitly analyzes a model on BABILong writing facts to memory and reading them back to answer a question. The paper therefore does not support the claim that BABILong tests only single-pass attention rather than a write-and-retrieve loop. (LLM, conf=0.88)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • BABILong includes tasks that perform poorly (e.g., the counting task achieves 28% accuracy) even without any irrelevant background text (0K).

    “For instance, BABILong includes tasks that perform poorly (e.g., the counting task achieves 28% accuracy) even without any irrelevant background text (0K) (Kuratov et al., 2024).”

    From NoLiMa: Long-Context Evaluation Beyond Literal Matching (2025) · cites this paper as Kuratov et al., 2024

    Section 2 says the first ten tasks show mixed LLM performance without distractor text, and Appendix F identifies QA7 as among the most challenging tasks. Appendix F also defines 0K as having no distractor sentences. The passages do not establish the specific figure of 28% accuracy for the counting task. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • In BABILong (QA1) and RULER, Llama 3.1 70B achieves effective lengths of 16K and 32K, respectively.

    “For comparison, in other benchmarks with similar settings, such as BABILong (QA1) (Kuratov et al., 2024) and RULER (Hsieh et al., 2024), Llama 3.1 70B achieves effective lengths of 16K 5 and 32K, respectively.”

    From NoLiMa: Long-Context Evaluation Beyond Literal Matching (2025) · cites this paper as Kuratov et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • L-Eval's stated length limit conflicts with its reported average

    Minor

    The introduction says L-Eval reaches only 40,000 tokens, but the related-work section reports average example lengths as high as 60,000 tokens. An average cannot exceed the stated maximum.

  • QA4 table exceeds the stated count of successful models

    Minor

    The text says only five models exceed 70% accuracy on QA4 at 0K. The QA4 0K row contains ten scores above 70%, including six before the fine-tuned models at the end of the row.

  • A QA3 example answer names the wrong reference location

    Minor

    The example asks where the football was before the bedroom, but its answer describes where it was before the kitchen. The location given, garden, matches the bedroom question; the answer's reference to the kitchen does not.

  • The listed preset lengths do not match the released split count

    Minor

    The dataset-statistics section lists ten pre-generated lengths, while the dataset description says the released version has 13 context-length splits across the same ten tasks. The text does not identify the other three lengths.

4 of 4

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 5

  1. [11]11. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 47e288629a6996a17ce50b90a056a0e1-Paper-Conference.pdf.
  2. [41]41. Li, D., Shao, R., Xie, A., Sheng, Y., Zheng, L., Gonzalez, J., Stoica, I., Ma, X., and Zhang, H. How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023a.
  3. [47]47. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.
  4. [76]76. Voelker, A., Kajić, I., and Eliasmith, C. Legendre memory units: Continuous-time representation in recurrent neural networks. Advances in neural information processing systems, 32, 2019.
  5. [86]86. Curran As- sociates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/ file/c8512d142a2d849725f31a9a7a361ab9-Paper.pdf.

Cited by

Sorted by citations

Related

Authors
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., & Burtsev, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
106519-106554
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2406.10149
arXiv
2406.10149
OpenAlex
W4415795958
Semantic Scholar
b47507f12a8b7d26f78a484e32a08d61d3f87358
Cited by
19
References
0
Citations by year
2026: 11 · 2025: 7 · 2024: 1
FWCI
5.49
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Analytical Center for the Government of the Russian Federation (70-2021-00138, 000000D730324P540002)
  • Moscow Institute of Physics and Technology (70-2021-00138)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
long-context reasoning, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review